Digital image optimization for ophthalmic surgery

Abstract

The present disclosure provides a digital image optimization system comprising a camera comprising at least one sensor that detects light reflected from an eye and sends a signal corresponding to the detected light to a processor. The system further comprises an image processing system comprising the processor and executing instructions for producing a digital image of the eye. The image processing system also executes instructions for applying a digital image optimization algorithm comprising a trained machine learning model to the digital image of the eye to generate an optimized digital image of the eye. The system further comprises a digital display that displays the optimized digital image of the eye.

Description

Technical Field

[0001] This disclosure relates to ophthalmic surgery and surgical equipment, and more specifically, to a digital image optimization system and associated methods for improving digital visualization in ophthalmic surgery. Background Technology

[0002] Ophthalmic surgery is surgery performed on the eye or any part of the eye. Every year, ophthalmic surgery saves and improves the vision of tens of thousands of patients. However, given the sensitivity of vision to even small changes in the eye and the tiny, fragile nature of many eye structures, ophthalmic surgery is challenging to perform, and even the reduction of small or unusual surgical errors or a slight improvement in the accuracy of surgical technique can make a huge difference in a patient's postoperative vision.

[0003] A type of ophthalmic surgery (vitretinal surgery) encompasses a variety of delicate procedures involving the internal parts of the eye, such as the vitreous fluid, retina, and vitreoretinal membrane. Different vitreoretinal surgical procedures (sometimes using lasers) are used to improve visual perception in the treatment of many eye diseases, including epiretinal membranes, diabetic retinopathy, vitreous hemorrhage, macular holes, retinal detachment, and complications from cataract surgery, among others.

[0004] During ophthalmic surgeries, such as vitreoretinal surgery, ophthalmologists typically use non-electro-optical surgical microscopes with eyepieces to view magnified images of the eye undergoing surgery. More recently, vitreoretinal surgeons may use eyepiece-less digital imaging systems to aid visualization during vitreoretinal surgery. These systems can include three-dimensional (3D) high dynamic range (“HDR”) camera systems with a pair of two-dimensional (2D) complementary metal-oxide-semiconductor (CMOS) single-chip or three-chip sensors, allowing surgeons to view the retina on a display screen using polarized glasses, digital eyepieces, or a head-mounted display. The display screen eliminates the need for eyepieces to observe the surgery and allows others in the operating room to see the procedure just like the surgeon. Compared to traditional optical analog surgical microscopes, this system also allows for improved digital images at high magnification and increased depth of field, thus improving visualization of the eye. Summary of the Invention

[0005] This disclosure provides a digital image optimization system and associated method for improving digital visualization in ophthalmic surgery. The digital image optimization system includes a camera comprising at least one sensor that detects light reflected from the eye and sends a signal corresponding to the detected light to a processor. The digital image optimization system also includes an image processing system containing the processor. The image processing system executes instructions for generating a digital image of the eye, and executes instructions for applying a digital image optimization algorithm, including a trained machine learning model, to the digital image of the eye to generate an optimized digital image of the eye. The digital image optimization system also includes a digital display for displaying the optimized digital image of the eye.

[0006] The digital image optimization system and its usage method may include the following additional features: i) the digital image optimization algorithm can at least partially reduce instrument glare, optical aberrations, vitreous haze, or any combination thereof in the digital image of the eye to generate an optimized digital image of the eye; ii) the contrast, sharpness, clarity, dynamic range, or any combination thereof of the optimized digital image of the eye can be equal to or higher than the contrast, sharpness, clarity, dynamic range, or any combination thereof of the digital image of the eye; iii) the noise, distortion, vignetting, or any combination thereof of the optimized digital image of the eye can be equal to or lower than the noise, distortion, vignetting, or any combination thereof of the digital image of the eye; iv) the digital image optimization algorithm can utilize interpolation, multiple exposure image denoising, or a combination thereof to generate the optimized digital image of the eye; v) the trained machine learning model may have been trained using multiple training images from a red enhancement library; vi) the trained machine learning model may have been trained using multiple digital images of the eye taken at the start of surgery and multiple corresponding digital images of the eye taken at the end of surgery; vii) the digital image optimization system may be Components of a 3D visualization system (Novartis AG Corp., Switzerland).

[0007] This disclosure further provides a digital image optimization system including a camera with at least one sensor that detects light reflected from the contralateral eye and sends a signal corresponding to the detected light to a processor. The at least one sensor also detects light reflected from the surgical eye and sends a signal corresponding to the detected light to the processor. The digital image optimization system further includes an image processing system containing the processor. The image processing system executes instructions for generating a template digital image of the contralateral eye, executes instructions for generating a digital mirror image of the template digital image, executes instructions for generating a working digital image of the surgical eye, and executes instructions for aligning the digital mirror image with the working digital image to generate a retina arcade alignment template. The digital image optimization system also includes a digital display showing the working digital image and the retina arcade alignment template.

[0008] The digital image optimization system and its method of use may include the following additional features: i) the retinal arch alignment template can be displayed as an overlay on the working digital image; ii) the template digital image may include eye structures, which are the retina of the contralateral eye, the central retinal vein of the contralateral eye, the retinal arch of the contralateral eye, the optic disc of the contralateral eye, or any combination thereof; iii) the working digital image may include eye structures, which are the retina of the operated eye, the central retinal vein of the operated eye, the retinal arch of the operated eye, the optic disc of the operated eye, or any combination thereof. Any combination thereof; iv) The working digital image may include the field of view of the surgical eye equivalent to that of the template digital image of the contralateral eye; v) Aligning the digital mirror image with the working digital image may include aligning eye structures, which are the optic disc of the contralateral eye and the optic disc of the surgical eye, the central retinal vein of the contralateral eye and the central retinal vein of the surgical eye, the arterial-venous intersection of the contralateral eye and the arterial-venous intersection of the surgical eye, the branch veins of the contralateral eye and the branch veins of the surgical eye, or any combination thereof; vi) The digital image optimization system may be Components of a 3D visualization system (Novartis AG, Switzerland).

[0009] This disclosure further provides a digital image optimization system comprising an endoscope including optical fibers. The digital image optimization system also includes a camera including at least one sensor that detects light reflected from the interior of the eye and propagating through the optical fibers. The sensor sends a signal corresponding to the detected light to a processor. The digital image optimization system also includes an image processing system comprising the processor. The image processing system executes instructions for generating endoscopic digital images of the eye, and executes instructions for applying an endoscopic digital image optimization algorithm, including a trained machine learning model, to the digital images of the eye to generate an optimized digital image of the eye. The digital image optimization system also includes a digital display showing the optimized endoscopic digital image of the eye.

[0010] The digital image optimization system and its usage method may include the following additional features: i) the endoscopic digital image optimization algorithm can utilize interpolation, multiple exposure image denoising, or a combination thereof to generate optimized endoscopic digital images of the eye; ii) the machine learning model may have been trained using multiple training images, including multiple endoscopic digital images of the eye successfully optimized using image processing methods; iii) the endoscopic digital image optimization algorithm can utilize single-image super-resolution to generate optimized endoscopic digital images of the eye; iv) the single-image super-resolution can use a very deep super-resolution neural network; v) the endoscopic digital image optimization algorithm can perform the following operations on the endoscopic digital images of the eye: at least partially reduce noise, increase resolution, improve digital image quality, or any combination thereof; vi) the digital image optimization system may be Components of a 3D visualization system (Novartis AG, Switzerland).

[0011] This disclosure further provides an optimization network including a digital image optimization system. The digital image optimization system includes a camera, which includes at least one sensor that detects light reflected from the eye and sends a signal corresponding to the detected light to a first processor. The digital image optimization system further includes a first image processing system, which includes the first processor and executes instructions for generating a digital image of the eye. The digital image optimization system further includes a digital display. The optimization network also includes a communication network communicatively coupled to the digital image optimization system and the digital image management system. The optimization network sends the digital image of the eye to the digital image management system. The optimization network also includes the digital image management system. The digital image management system includes a second image processing system, which includes a second processor and a trained machine learning model. The second image processing system executes instructions for processing the digital image of the eye using the trained machine learning model to compute an optimized digital image of the eye, and sends the optimized digital image of the eye to the digital display. The digital display displays the optimized digital image of the eye.

[0012] The optimized network and its usage method may include the following additional features: i) the second image processing system may be stored locally as part of the first image processing system; ii) the second image processing system may run on a server; iii) the second image processing system may provide cloud-based digital image optimization services; iv) the trained machine learning model may have been trained using multiple training images; v) the multiple training images may include multiple digital images of the eye taken at the start of the surgery and multiple paired digital images of the same eye taken at the end of the surgery; vi) the multiple training images may include multiple endoscopic digital images of the eye successfully optimized using the image processing method; The 3D visualization system (Novartis AG, Switzerland) captures the plurality of training images; vii) the communication network can be used to upload the plurality of training images to the digital image management system; viii) the trained machine learning model may have been trained using an integrated architecture including a convolutional neural network and at least one long short-term memory unit; ix) the digital image optimization system may be Components of a 3D visualization system (Novartis AG, Switzerland).

[0013] This disclosure further provides a digital timeout system including an identification tag affixed to a patient, the identification tag including machine-readable information. The digital timeout system also includes a camera including at least one sensor that acquires machine-readable information and sends a signal corresponding to the machine-readable information to a processor. The digital timeout system also includes an image processing system including a processor and processing the machine-readable information to determine patient data. The image processing system also determines the differences, consistency, or combinations thereof between the patient data and information provided to a surgeon, operating room staff, or a combination thereof.

[0014] The digital timeout system and its method of use may include the following additional features: i) the digital timeout system may further include a digital display showing the differences, similarities, or combinations thereof between patient data and information provided to the surgeon, operating room staff, or a combination thereof; ii) the identification tag may be affixed above the patient's eye; iii) the identification tag may be a removable decal tattoo, patch, sticker, tape, or any combination thereof; iv) the identification tag may further include a locator; v) the machine-readable information may be a one-dimensional barcode, multi-dimensional barcode, quick-response code, symbol, or any combination thereof; vi) the digital timeout system may be... The system comprises a 3D visualization system (Novartis AG, Switzerland); vii) the patient data may include information associated with the patient, such as patient-associated identification information, patient-associated medical information, index information, or any combination thereof; viii) the camera may detect light reflected from the eye and send a signal corresponding to the detected light to a first processor, and the image processing system may execute instructions for generating a digital image of the eye; ix) the system may include a communication network communicatively coupled to the digital timeout system and the digital image management system. The communication network may send the digital image of the eye to the digital image management system. The digital image management system may include a second image processing system, which may include a second processor and a trained machine learning model. The machine-readable data may include index information associated with the digital image of the eye and stored in the digital image management system; x) the trained machine learning model may have been trained using multiple training images including the digital image of the eye. The multiple training images may have been selected based on the index information associated with the digital image of the eye.

[0015] This disclosure further provides a method for optimizing digital images of an eye, the method comprising: capturing a digital image of the eye with a camera; applying a digital image optimization algorithm to generate an optimized digital image of the eye using an image processing system; and displaying the optimized digital image of the eye on a digital display. The digital image optimization algorithm can at least partially reduce instrument glare, optical aberrations, vitreous haze, or any combination thereof in the digital image of the eye to generate the optimized digital image of the eye. The digital image optimization algorithm may be a trained machine learning model. The trained machine learning model may have been trained using multiple digital images of the eye captured at the beginning of surgery and multiple corresponding digital images of the eye captured at the end of surgery. The camera may be… Components of a 3D visualization system (Novartis AG, Switzerland).

[0016] This disclosure further provides a method for generating a digital image optimization algorithm, the method comprising: capturing multiple input digital images of the eye at the start of surgery; capturing multiple output digital images of the eye at the end of surgery; training a machine learning model using the multiple input digital images and the multiple output digital images of the eye as training images; and using the trained machine learning model as a digital image optimization algorithm.

[0017] This disclosure further provides a method for optimizing digital images of the eye, the method comprising: capturing a digital image of the contralateral eye; capturing a digital image of the surgical eye; generating a mirror image of the digital image of the contralateral eye; aligning the mirror image of the digital image of the contralateral eye with the digital image of the surgical eye to generate a retinal arch alignment template; displaying the retinal arch alignment template as an overlay of the digital image of the surgical eye on a digital display; and performing ophthalmic surgery on the surgical eye using the retinal arch alignment template. The mirror image of the digital image of the contralateral eye can be aligned with the digital image of the surgical eye by aligning the optic disc, the central retinal vein, the arterial-venous intersection, branch veins, or any combination thereof.

[0018] This disclosure further provides a method for optimizing endoscopic digital images of the eye, the method comprising: capturing endoscopic digital images of the eye using an endoscope; applying an endoscopic digital image optimization algorithm to generate an optimized endoscopic digital image of the eye; and displaying the optimized endoscopic digital image of the eye on a digital display. The endoscopic digital image optimization algorithm may use interpolation, multiple exposure image denoising, machine learning, deep learning, or any combination thereof.

[0019] This disclosure further provides a method for generating an optimization algorithm for endoscopic digital images, the method comprising: capturing multiple endoscopic digital images of the eye; optimizing the multiple endoscopic digital images of the eye using an image processing method to generate multiple optimized endoscopic digital images of the eye; training a machine learning model using the multiple endoscopic digital images of the eye as input images and using the multiple optimized endoscopic digital images of the eye as output images; and using the trained machine learning model as an optimization algorithm for endoscopic digital images.

[0020] This disclosure further provides a method for training a machine learning model, the method comprising: inputting multiple training images, including multiple input images and multiple corresponding output images, into an ensemble architecture using an image processing system; generating feature representations of the multiple input images using a convolutional neural network; calculating multiple hypothesis differences between multiple predicted optimized digital images of the eye and the multiple corresponding output images using the ensemble architecture; calculating a loss between the multiple hypothesis differences and the multiple differences calculated between the multiple input images and the multiple corresponding output images; evaluating whether the total loss is less than a predefined threshold; updating the parameters of the ensemble architecture if the total loss is not less than the predefined threshold; and storing the parameters of the ensemble architecture and ending training if the total loss is less than the predefined threshold. The loss can be calculated using a loss function. The loss function can be a smoothed L1 loss function. The multiple hypothesis differences can be calculated using the feature representations of the multiple input images. Once the machine learning model is trained, the image processing system can be run on a server. The image processing system can provide cloud-based digital image optimization services.

[0021] This disclosure further provides a method for performing a digital timeout, the method comprising: using a camera to acquire machine-readable information printed on an identification tag affixed to a patient; using an image processing system to process the machine-readable information; determining the correctness of the patient's identity based on the machine-readable information; stopping a planned medical procedure if the patient's identity is incorrect; determining the correctness of the planned medical procedure based on the machine-readable information if the patient's identity is correct; stopping the planned medical procedure if the planned medical procedure is incorrect; and executing the planned medical procedure if the planned medical procedure is correct.

[0022] This disclosure further provides a method for optimizing digital images of an eye, the method comprising: capturing a digital image of the eye using a digital image optimization system; associating the digital image of the eye with processed machine-readable information; training a machine learning model using the digital image of the eye; processing the digital image of the eye using the trained machine learning model; generating an optimized digital image of the eye using an image processing system; and displaying the optimized digital image of the eye on a digital display. The digital image of the eye can be selected based on the processed machine-readable information to be included in a set of training images.

[0023] Unless explicitly mutually exclusive, aspects of digital image optimization systems and their methods of application can be combined with each other. Furthermore, unless explicitly mutually exclusive, additional features of the aforementioned digital image optimization systems and their associated methods can also be combined with each other. Attached Figure Description

[0024] To gain a more thorough understanding of this disclosure and its features and advantages, reference is now made to the following description taken in conjunction with accompanying drawings, which are not to scale, and in which the same reference numerals indicate the same features, and in these drawings:

[0025] Figure 1 It is a schematic diagram of a digital image optimization system that includes a camera, an image processing system, and a digital display.

[0026] Figures 2A to 2D This is a schematic diagram of the training images;

[0027] Figure 3 This is a flowchart illustrating methods for optimizing digital images of the eye to improve digital visualization in ophthalmic surgery;

[0028] Figure 4 This is a flowchart illustrating a method for using machine learning models to generate digital image optimization algorithms to improve digital visualization in ophthalmic surgery;

[0029] Figure 5 It is a schematic diagram of a digital image optimization system that includes a camera, an image processing system, and a digital display.

[0030] Figure 6 This is a flowchart illustrating a method for improving digital visualization of ophthalmic surgery by optimizing digital images of the eye using a retinal arch alignment template;

[0031] Figure 7 It is a schematic diagram of a digital image optimization system that includes an endoscope, optical fiber, fiber optic light source, camera, image processing system and digital display;

[0032] Figure 8It is a flowchart illustrating a method for optimizing endoscopic digital images of the eye to improve digital visualization in ophthalmic surgery;

[0033] Figure 9 This is a flowchart illustrating a method for using machine learning models to generate optimized algorithms for endoscopic digital images to improve digital visualization in ophthalmic surgery;

[0034] Figure 10 It is a schematic diagram of an optimization network that includes a digital image optimization system, a communication network, and a digital image management system;

[0035] Figure 11 This is an illustration of an example solution for determining an optimized digital image of the eye;

[0036] Figure 12 This is a diagram illustrating an example ensemble architecture for a machine learning model;

[0037] Figure 13 It is a flowchart illustrating the method for training machine learning models;

[0038] Figure 14 It is a schematic diagram of a digital timeout system that includes identification tags, cameras, image processing systems, and digital displays;

[0039] Figure 15 This is a flowchart illustrating the method for implementing digital timeouts;

[0040] Figure 16 This is a flowchart illustrating a method for optimizing digital images of the eye used in ophthalmic surgery;

[0041] Figure 17 It is a schematic diagram of a computer system including a digital image optimization system; and

[0042] Figure 18 A to Figure 18 C is a schematic diagram of a medical system that includes a digital image optimization system. Detailed Implementation

[0043] This disclosure provides a digital image optimization system and associated methods for improving digital visualization in ophthalmic surgery.

[0044] Ophthalmic surgeons face unique challenges when visualizing the eye. In particular, aberrations during medical procedures can degrade the quality of digital images of the eye. For example, any view obtained through a patient's pupil is affected by optical aberrations, which can distort the digital image of the eye. Optical aberrations can be caused by eye diseases or previous surgeries resulting in corneal asphericity or intraocular lens implants, leading to aberrations in the image seen by the surgeon. Vitreous haze can also degrade digital image quality. Additionally, stainless steel instruments inserted into the eye can produce glare, reducing the image quality when viewing the eye during surgery. In other examples, low-resolution digital images (such as endoscopic digital images of approximately 30,000 pixels) can be displayed on high-resolution digital displays with over one million pixels. This can result in poor image quality for endoscopic digital images.

[0045] Poor image quality can interfere with a surgeon's ability to visualize the inside of the eye and make surgery more difficult. In analog systems, methods for correcting aberrations are very limited, and many are simply not correctable. Digital visualization systems do allow for a variety of corrective measures, which can improve the digital image of the eye presented to the surgeon and other personnel assisting in ophthalmic surgery. However, current systems and methods may not utilize optimization algorithms, which can include machine learning models, to improve the quality of digital images of the eye. Furthermore, current systems and methods may not utilize identity tagging to classify digital images of the eye for use in optimization algorithms. In such cases, the image quality of the digital image of the eye may be degraded. For example, the contrast, sharpness, clarity, and dynamic range of the digital image of the eye may decrease, while noise, distortion, and vignetting may increase. Degraded image quality can increase the risk of complications during ophthalmic surgery.

[0046] The digital image optimization system and method disclosed herein can provide faster, safer, and more efficient medical procedures by improving the digital visualization of ophthalmic surgery. The digital image optimization system and method disclosed herein can improve the digital visualization of ophthalmic surgery by providing optimized digital images that, compared to digital images provided by other systems and methods, have reduced instrument glare, reduced optical aberrations, reduced vitreous haze, or any combination thereof. Compared to current systems and methods, the digital image optimization system and method disclosed herein can improve the digital visualization of ophthalmic surgery by providing a retinal arch alignment template for aligning retinal attachment during surgery. The digital image optimization system and method disclosed herein can improve the digital visualization of ophthalmic surgery by providing optimized digital images as super-resolution digital images. Compared to current systems and methods, the system and method disclosed herein can improve the digital visualization of ophthalmic surgery by including a machine learning model for optimizing digital images of the eye. The digital image optimization system and method disclosed herein can improve the digital visualization of ophthalmic surgery by providing training images for training the machine learning model. The training images can be multiple digital images of multiple eyes in multiple medical procedures. The digital image optimization system and method disclosed herein can improve the digital visualization of ophthalmic surgery by providing cloud-based digital image optimization services. The digital image optimization system and method disclosed herein can also improve the digital visualization of ophthalmic surgery by providing digital timeouts. Digital timeouts may include a computational check that matches patient-associated medical information with information provided to the surgeon, operating room staff, or a combination thereof.

[0047] The systems and methods disclosed herein can improve digital visualization in ophthalmic surgery by providing a digital image optimization system that can improve the contrast, sharpness, clarity, and dynamic range of digital images of the eye and reduce noise, distortion, and vignetting in the digital images of the eye. The digital image optimization system can be a component of a digital-assisted vitreoretinal surgery (“DAVS”) system, or it can be... Components of a 3D visualization system (Novartis AG, Switzerland). For example, digital images of the eye can be captured using a camera or endoscope. The systems and methods disclosed herein can use machine learning models to optimize digital images of the eye. The systems and methods disclosed herein can use a retinal arch alignment template to align digital images of the same eye captured at different stages during surgery and provide optimized digital images for retinal reattachment after vitrectomy. The systems and methods disclosed herein can also provide an optimization network that can allow cloud-based digital image optimization services. The optimization network can be a component of a digitally assisted vitrectomy (“DAVS”) system, or it can be... A component of a 3D visualization system (Novartis AG, Switzerland). The systems and methods disclosed herein can also provide a digital timeout system that includes machine-readable information about digital images of the eye. This machine-readable information can be used to save records for storing digital images of the eye in an optimized network, and for selecting training images to train a machine learning model. The digital timeout system can be a component of a digital-assisted vitrectomy (“DAVS”) system, or it can be... Components of a 3D visualization system (Novartis AG, Switzerland).

[0048] Now for reference Figure 1 The digital image optimization system 100 may include a camera 150, an image processing system 170, and a digital display 190. The image processing system 170 can provide optimized digital images of the eye 101. The camera 150 can capture digital images of the eye 101. The digital images of the eye 101 captured by the camera 150 may include a field of view 160. The field of view 160 may include a close-up view of the eye 101 and may include a higher magnification view of the macula, vitreous humor, or other areas of the eye 101. The field of view 160 may also include a surgical instrument 105. The surgical instrument 105 may be a cannula or another surgical tool used in ophthalmic surgery.

[0049] Camera 150 can be a digital camera, an HDR camera, a 3D camera, or any combination thereof. Camera 150 can be... A component of a 3D visualization system (Novartis AG, Switzerland). Camera 150 may also be a camera coupled to a microscope. Camera 150 may replace the eyepiece on the microscope and may be a fifth-generation image acquisition module (ICM5) 3D surgical camera. Camera 150 may be configured to provide a stereoscopic digital image (not shown) of eye 101. Camera 150 may include lens 121. Lens 121 may also be an optomechanical focusing lens, a manual focusing lens, or a combination thereof. Camera 150 may also include at least one image sensor 152, which may be a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor. Camera 150 may be a monochrome camera or a color camera, and at least one image sensor 152 may be a monochrome image sensor or a color image sensor. At least one image sensor 152 may be an image sensor with a color filter array (e.g., a Bayer filter) or an image sensor without a color filter array.

[0050] The digital image optimization system 100 may include a visible light illumination source 155. The visible light illumination source 155 may be a visible light illumination source for the camera 150. The visible light illumination source 155 may be an internal illuminator. The visible light illumination source 155 may include a xenon source, a white LED light source, or any other suitable visible light source. The visible light illumination source 155 can illuminate the internal structures of the eye 101.

[0051] Light emitted by visible light source 155 and reflected from surgical instrument 105 may cause instrument glare 110. Instrument glare 110 may also be caused by light emitted from other light sources. Instrument glare 110 may appear as noise or distortion in a digital image of the eye 101 captured by camera 150.

[0052] Light emitted by the visible light source 155 and reflected from the interior of the eye 101 may also be affected by optical aberrations 120. Optical aberrations 120 may be caused by a cloudy cornea or a cloudy lens, the cloudy cornea possibly caused by prolonged medical procedures or eye defects, and the cloudy lens possibly caused by cataracts or blood in the eye. Optical aberrations 120 may also be caused by other features of the eye 101, surgical settings, or combinations thereof. Optical aberrations 120 reduce the contrast, sharpness, clarity, and dynamic range of digital images of the eye 101 captured by the camera 150.

[0053] Light emitted by the visible light source 155 and reflected from the eye 101 may be affected by vitreous haze 130, which may appear as haze or fog in a digital image of the eye 101 taken by the camera 150.

[0054] The digital image captured by camera 150 can be processed by image processing system 170. Image processing system 170 may include processor 180. Camera 150 may use at least one image sensor 152 to detect light reflected from the outside of eye 101 into lens 121, and the at least one image sensor may send a signal corresponding to the detected light to processor 180. Processor 180 may execute instructions for generating a digital image of eye 101. Image processing system 170 may also include memory medium 181. The digital image of eye 101 may be stored in memory medium 181.

[0055] Processor 180 can execute instructions for applying a digital image optimization algorithm to a digital image of eye 101 to produce an optimized digital image of eye 191. The digital image optimization algorithm can be an algorithm that is a trained machine learning model. A machine learning model can allow the development of digital image optimization algorithms that have learned and improved from experience without explicit programming, as will be discussed. In one example, a "red enhancement" library of digital images from past surgeries can be used as training images to train the machine learning model. The red enhancement library can contain images of poor red refraction and bright red refraction in the eye. By training on the red enhancement library, the machine learning model can be able to predict the optimized digital image of eye 191. Alternatively, the digital image optimization algorithm can be an algorithm that utilizes another image enhancement technique (e.g., interpolation or multiple exposure image denoising).

[0056] A digital image optimization algorithm can at least partially reduce instrument glare 110, optical aberration 120, vitreous haze 130, or any combination thereof, in a digital image of eye 101 captured by camera 150 to produce an optimized digital image of eye 191. Compared to a digital image of eye 101 captured by camera 150 without the application of the digital optimization algorithm, the optimized digital image of eye 191 can have less instrument glare 110, less optical aberration 120, less vitreous haze 130, or any combination thereof.

[0057] The processor 180 may include, for example, a field-programmable gate array (FPGA), a microprocessor, a microcontroller, a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or any other digital or analog circuit system configured to interpret and / or execute program instructions and / or process data.

[0058] Processor 180 may include any physical means capable of storing and / or executing instructions. Processor 180 may execute processor instructions to implement at least a portion of one or more systems, one or more flowcharts, one or more processes, and / or one or more methods described herein. For example, processor 180 may execute instructions for generating a digital image of eye 101. Processor 180 may be configured to receive instructions from memory medium 181. In one example, processor 180 may include memory medium 181. In another example, memory medium 181 may be external to processor 180. Memory medium 181 may store instructions. Instructions stored by memory medium 181 may be executed by processor 180 and may be configured, encoded, decoded, and / or encoded using instructions that are at least a portion of one or more systems, one or more flowcharts, one or more methods, and / or one or more processes described herein. Memory medium 181 may store instructions that can be executed by processor 180 to apply digital image optimization algorithms to a digital image of eye 101 captured by camera 150.

[0059] An FPGA can be configured, encoded, and / or coded to implement at least a portion of one or more systems, flowcharts, processes, and / or methods described herein. For example, an FPGA can be configured, encoded, and / or coded to generate a digital image of eye 101. An ASIC can be configured to implement at least a portion of one or more systems, flowcharts, processes, and / or methods described herein. For example, an ASIC can be configured, encoded, and / or coded to generate a digital image of eye 101. A DSP can be configured, encoded, and / or coded to implement at least a portion of one or more systems, flowcharts, processes, and / or methods described herein. For example, a DSP can be configured, encoded, and / or coded to generate a digital image of eye 101.

[0060] A single device may include a processor 180 and an image processing system 170, or the processor 180 may be separate from the image processing system 170. In one example, a single computer system may include the processor 180 and the image processing system 170. In another example, the device may include an integrated circuit that may include the processor 180 and the image processing system 170. Alternatively, the processor 180 and the image processing system 170 may be incorporated into a surgical console.

[0061] Processor 180 can interpret and / or execute program instructions stored in memory and / or process data stored in memory medium 181. Memory medium 181 can be configured, in part or in whole, as application memory, system memory, or both. Memory medium 181 can include any system, apparatus, or device configured to hold and / or accommodate one or more memory devices. Each memory device can include any system, module, or device (e.g., a computer-readable medium) configured to retain program instructions and / or data for a period of time. The described one or more servers, electronic devices, or other machines can include one or more similar processors or memories that can store and execute program instructions for implementing the functions of the associated machine.

[0062] The digital image optimization system 100 may include a digital display 190. The digital display 190 may include any type of screen or projector capable of displaying a digital image of the eye 101 at a sufficient resolution for use in ophthalmic surgery. For example, the display may include any type of screen or projector used in conjunction with ophthalmic surgery, including the type of display used in conventional vitreoretinal surgery systems for presenting digital images. The digital display 190 may display an optimized digital image of the eye 191. This can improve the digital visualization of ophthalmic surgery by providing a digital image of the eye with less instrument glare 110, less optical aberration 120, less vitreous fog 130, or any combination thereof, compared to an unoptimized digital image of the eye 101. The digital display 190 may display a digital image that is a combination of the digital image of the eye 101 and the optimized digital image of the eye 191. The combined digital image may be generated by a processor 180. The ratio of the digital image of the eye 101 and the optimized digital image of the eye 191 displayed in the combined image may be controlled by the surgeon (e.g., using a slider controlled by a controller 192). In this way, surgeons can control the extent to which optimized digital images are used for visualization during surgery.

[0063] Digital display 190 can display a single image or two images for stereoscopic viewing. Digital display 190 can be a digital display, screen, head-up display, head-mounted display, or any combination thereof, and may also include multiple displays. Digital display 190 can be a flat panel display or an ultra-high-definition 3D flat panel display. Digital display 190 can be a 3D organic light-emitting diode (OLED) surgical display. The image displayed on digital display 190 can be viewed through a pair of passive circularly polarized glasses. Digital display 190 can be a picture-in-picture display. Digital display 190 can be a component of a digitally assisted vitreoretinal surgery (“DAVS”) system, or can be... Components of a 3D visualization system (Novartis AG, Switzerland).

[0064] Digital display 190 can display an optimized digital image of eye 191 generated by processor 180 or another processor, as well as other information generated by processor 180 or another processor. This information may include graphical or textual information such as surgical parameters, surgical mode, flow rate, intraocular pressure, endoscopic video, OCT images, warnings, digital images, color-coded information, or augmented reality information. Processor 180 can reformat video produced using camera 150 for display on digital display 190, which can be viewed using circularly polarized glasses, digital eyepieces, or a head-mounted display.

[0065] Machine learning models can be used to generate digital optimization algorithms. As will be discussed, multiple training images can be used to train the machine learning model. Training images can include digital images taken at the beginning of the surgery and digital images taken at the end of the surgery, such as... Figures 2A to 2D The digital images taken at the start of surgery provide the worst visualization of the eye. This is likely because the digital images include instrument glare 110, optical aberrations 120, vitreous fog 130, or any combination thereof. Digital images of the same eye taken at the end of surgery provide the best visualization of the eye. This is likely because many sources of noise or distortion, such as instrument glare 110, optical aberrations 120, vitreous fog 130, or any combination thereof, have been removed. Therefore, training a machine learning model on multiple eyes using multiple digital images taken at the start of surgery and multiple corresponding digital images of the eye taken at the end of surgery can generate a digital optimization algorithm for predicting the digital images taken at the end of surgery based on the input digital images taken at the start of surgery. If desired, the digital images of the eye taken at the start of surgery can be aligned with the digital images of the eye taken at the end of surgery using the optic disc, retinal arch, or a combination thereof. This provides a register for aligning and comparing the images.

[0066] For example, the digital image 201 of the eye at the start of surgery could be a digital image of the eye 101 taken by camera 150 at the start of surgery, such as... Figure 2AThe digital image 201 of the eye at the start of surgery may include, for example, instrument glare 110 caused by light emitted from visible light source 155 and reflected from surgical instruments 105. Instrument glare 110 may appear as noise or distortion in the digital image 201 of the eye at the start of surgery. The digital image 201 of the eye at the start of surgery may also include optical aberrations 120 caused by light emitted from visible light source 155 and reflected from the interior of the eye 101. Optical aberrations 120 may be caused, for example, by an opaque lens 220. Optical aberrations 120 reduce the contrast, sharpness, clarity, and dynamic range of the digital image 201 of the eye at the start of surgery.

[0067] The digital image 202 of the eye at the end of the surgery can be a digital image of the eye 101 taken by camera 150 at the end of the surgery, such as... Figure 2B As depicted. At the end of the surgery, the surgical instruments 105 may have been at least partially removed from the eye 101. Therefore, the digital image 202 of the eye at the end of the surgery may have less instrument glare 110, or may not have instrument glare 110. At the end of the surgery, the cloudy lens 220 may be at least partially replaced by an artificial lens 221. Therefore, the digital image 202 of the eye at the end of the surgery may have less optical aberration 120, or may not have optical aberration 120. Compared to the digital image 201 of the eye at the beginning of the surgery, the digital image 202 of the eye at the end of the surgery may have improved contrast, sharpness, clarity, and dynamic range, and may have reduced noise, distortion, and vignetting.

[0068] In another example, the digital image 203 of the eye at the start of surgery could be a digital image of the eye 101 taken by camera 150 at the start of surgery, such as... Figure 2C The digital image 203 of the eye at the start of surgery may include vitreous fog 130 caused by light emitted from visible light source 155 and reflected from the vitreous body 210. Vitreous fog 130 may appear as noise or distortion in the digital image 203 of the eye at the start of surgery. The digital image 203 of the eye at the start of surgery may also include optical aberrations 120 caused by light emitted from visible light source 155 and reflected from the interior of the eye 101. Optical aberrations 120 may be caused by blood 211. Optical aberrations 120 reduce the contrast, sharpness, clarity, and dynamic range of the digital image 203 of the eye at the start of surgery.

[0069] The digital image 204 of the eye at the end of the surgery can be a digital image of the eye 101 taken by camera 150 at the end of the surgery, such as... Figure 2DAs depicted. At the end of the surgery, the vitreous humor 210 may have been at least partially removed from the eye 101. Therefore, the digital image 204 of the eye at the end of the surgery may include less vitreous fog 130, or may not include vitreous fog 130. At the end of the surgery, blood 211 may also have been at least partially removed. Therefore, the digital image 204 of the eye at the end of the surgery may include less optical aberration 120, or may not include optical aberration 120. Compared to the digital image 203 of the eye at the beginning of the surgery, the digital image 204 of the eye at the end of the surgery may have improved contrast, sharpness, clarity, and dynamic range, and may have reduced noise, distortion, and vignetting.

[0070] Multiple digital images of the eye at the start of surgery and multiple digital images of the same eye at the end of surgery can be used as input and output images, respectively, in a set of training images to train a machine learning model. For example, the input image could be digital image 201 of the eye at the start of surgery, and the output image could be digital image 202 of the eye at the end of surgery. As will be discussed, the machine learning model can be trained based on extracting features from the input images and learning the relationship function between the extracted features and the differences between the input and output images. The trained machine learning model can then be used as a digital image optimization algorithm.

[0071] A digital image optimization algorithm can use memory medium 181 to generate processor instructions to be executed on processor 180 to optimize a digital image of the eye at the start of surgery (e.g., digital image 201 or digital image 203 of the eye at the start of surgery). Processor 180 can execute instructions to generate an optimized digital image of eye 191. The optimized digital image of eye 191 can have improved image quality comparable to that of the digital image of the eye at the end of surgery, for example, comparable to that of digital image 202 or digital image 204 of the eye at the end of surgery. Compared to the digital image of eye 101, the optimized digital image of eye 191 can have improved contrast, improved sharpness, improved clarity, improved dynamic range, reduced noise, reduced distortion, reduced vignetting, or any combination thereof.

[0072] Figure 3A flowchart of a method for optimizing digital images of the eye to improve digital visualization during ophthalmic surgery is presented. In step 300, a digital image of the eye is captured. The digital image of the eye can be captured by a camera such as camera 150. The digital image of the eye can be captured at the start of surgery and can be a digital image (such as digital image 201 of the eye at the start of surgery), or the digital image of the eye can be captured during surgery. The digital image of the eye may have instrument glare (such as instrument glare 110), optical aberrations (such as optical aberration 120), vitreous fog (such as vitreous fog 130), or any combination thereof. In step 310, a digital image optimization algorithm is applied to the digital image of the eye to generate an optimized digital image of the eye, such as an optimized digital image of eye 191. The digital image optimization algorithm can be a trained machine learning model. In step 320, the optimized digital image of the eye is displayed on a digital display (such as digital display 190).

[0073] Figure 4 A flowchart is presented outlining a method for using a machine learning model to generate a digital image optimization algorithm to improve the digital visualization of ophthalmic surgery. In step 400, an input digital image of the eye is captured at the start of the surgery, such as digital image 201 of the eye at the start of surgery captured by camera 150. In step 410, an output digital image of the same eye is captured at the end of the surgery, such as digital image 202 of the eye at the end of surgery captured by camera 150. In step 420, the input digital image of the eye at the start of the surgery and the output digital image of the eye at the end of the surgery are used, respectively, as input and output images in a set of training images to train the machine learning model. In step 430, steps 400-420 are repeated for multiple input and output images from multiple patients. These images can be collected from numerous surgeons and ophthalmology hospitals. Multiple input and output images can be used to train the machine learning model to predict the output image (e.g., digital image of the eye at the end of surgery) when only the input image (e.g., digital image of the eye at the start of surgery) is provided. In step 440, the trained machine learning model can be used as a digital image optimization algorithm to improve the digital images of the eye.

[0074] In an alternative example, the digital image optimization system 500 can improve the digital visualization of ophthalmic surgery by providing a retinal arch alignment template, such as... Figure 5The retinal arch can include the vascular architecture of the retina. The retinal arch can include the temporal arch vessels. By aligning the retinal arch, digital images of the same eye taken at different stages during surgery can be aligned. Alternatively, alignment of a digital mirror image of the retinal arch of the contralateral eye 502 (the healthy eye not undergoing surgery) with a digital image of the surgical eye 501 of the same patient can provide a retinal arch alignment template during surgery. This can provide an optimized digital image of the surgical eye. The contralateral eye can be the right eye, and the surgical eye can be the left eye of the same patient. Alternatively, the contralateral eye can be the right eye, and the surgical eye can be the left eye of the same patient.

[0075] For example, during vitrectomy, digital visualization can be improved by overlaying and aligning a digital mirror image of the contralateral eye's retinal arch with a digital image of the operated eye to provide a retinal arch alignment template for retinal reattachment. This can at least partially prevent ciliary body distortion following retinal detachment correction during vitrectomy. The pattern of the contralateral eye's retinal arch is likely to be most similar to that of the operated eye compared to any other eye. Figure 5 As shown, alignment can be achieved during surgery using a digital visualization system. This visualization system can be either 2D or 3D.

[0076] The digital image optimization system 500 may include a camera 550, an image processing system 570, and a digital display 590. The camera 550 may be a digital camera, an HDR camera, a 3D camera, or any combination thereof. A component of a 3D visualization system (Novartis AG, Switzerland). Camera 550 may also be a camera coupled to a microscope. Camera 550 may replace the eyepiece on the microscope and may be a fifth-generation image acquisition module (ICM5) 3D surgical camera. Camera 550 may be configured to provide stereoscopic digital images of the surgical eye 501 or the contralateral eye 502 (not shown). Camera 550 may include lens 521. Lens 521 may also be an optomechanical focusing lens, a manual focusing lens, or any combination thereof. Camera 550 may also include at least one image sensor 552, which may be a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor. Camera 550 may be a monochrome camera or a color camera, and at least one image sensor 552 may be a monochrome image sensor or a color image sensor. At least one image sensor 552 may be an image sensor with a color filter array (e.g., a Bayer filter) or an image sensor without a color filter array. Alternatively, the images captured by camera 550 may be optical coherence tomography (OCT) images, multispectral / hyperspectral images, ultrasound images, or any combination thereof.

[0077] Digital images of the surgical eye 501 or the contralateral eye 502 captured by camera 550 may include a field of view 560. The field of view 560 may include a close-up view of the surgical eye 501 or the contralateral eye 502, and may include a higher magnification view of the retinal arch or other areas of the surgical eye 501 or the contralateral eye 502.

[0078] Before surgery, camera 550 can capture a template digital image 592 of the fundus of the contralateral eye 502. The template digital image 592 may include digital images of the retina of the contralateral eye 510, the central retinal vein of the contralateral eye 512, the retinal arch of the contralateral eye 515, the optic disc of the contralateral eye 517, or any combination thereof. During surgery, camera 550 can capture a working digital image 591 of the fundus of the surgical eye 501. The working digital image 591 may have a visual field 560 of the surgical eye 501 that is equivalent to the visual field 560 of the template digital image 592 of the contralateral eye 502. The working digital image 591 may include digital images of the retina of the surgical eye 509, the central retinal vein of the surgical eye 511, the retinal arch of the surgical eye 514, the optic disc of the surgical eye 516, or any combination thereof.

[0079] Digital images captured by camera 550 can be processed by image processing system 570. Image processing system 570 may include processor 580. Camera 550 may use at least one image sensor 552 to detect light reflected from the interior of the surgical eye 501 or the contralateral eye 502 into lens 521, and the at least one image sensor may send a signal corresponding to the detected light to processor 580. Processor 580 may execute instructions for generating a working digital image 591 of surgical eye 501, a template digital image 592 of contralateral eye 502, or a combination thereof. Image processing system 570 may also include memory medium 581. Working digital image 591, template digital image 592, or a combination thereof may be stored in memory medium 581.

[0080] Processor 580 can execute instructions for creating a digital mirror image 593 for creating a template digital image 592. Processor 580 can execute instructions for aligning the digital mirror image 593 with a working digital image 591 to generate a retinal arch alignment template 594. Processor 580 can execute instructions for aligning the digital mirror image 593 with the working digital image 591 by aligning the optic disc of the contralateral eye 517 with the optic disc of the surgical eye 516, by aligning the central retinal vein of the contralateral eye 512 with the central retinal vein of the surgical eye 511, or combinations thereof. Processor 580 can also execute instructions for aligning the digital mirror image 593 with the working digital image 591 by aligning arterial-venous intersections (not shown), branch veins (not shown), or combinations thereof. Processor 580 may be similar to processor 180.

[0081] Digital display 590 may include any type of screen or projector capable of displaying digital images of the surgical eye 501 at a sufficient resolution for use in ophthalmic surgery. For example, the display may include any type of screen or projector used in conjunction with ophthalmic surgery, including the type of display used in conventional vitreoretinal surgery systems that presents digital images. Digital display 590 may be similar to digital display 190.

[0082] Digital display 590 can display a working digital image 591, a template digital image 592, a digital mirror image 593, a retinal arch alignment template 594, or any combination thereof. Digital display 590 can display the working digital image 591 and the superimposed retinal arch alignment template 594. This can improve the digital visualization of ophthalmic surgery by providing the retinal arch alignment template 594 for retinal reattachment. The working digital image 591 superimposed with the retinal arch alignment template 594 can represent an optimized digital image of the eye compared to the working digital image 591 alone. Digital display 590 can display a digital image that is a combined digital image of the retinal arch alignment template 594 and the working digital image 591. The combined digital image can be generated by processor 580. The ratio of the working digital image 591 and the retinal arch alignment template 594 displayed in the combined image can be controlled by the surgeon (e.g., using a slider controlled by controller 595). In this way, the surgeon can control the extent to which the retinal arch alignment template 594 is used for visualization during surgery.

[0083] Digital display 590 can display a single image or two images for stereoscopic viewing. Digital display 590 can be a digital display, screen, head-up display, head-mounted display, or any combination thereof, and may also include multiple displays. Digital display 590 can be a flat panel display or an ultra-high-definition 3D flat panel display. Digital display 590 can be a 3D organic light-emitting diode (OLED) surgical display. The image displayed on digital display 590 can be viewed through a pair of passive circularly polarized glasses. Digital display 590 can be a component of a digitally assisted vitreoretinal surgery (“DAVS”) system, or can be... Components of a 3D visualization system (Novartis AG, Switzerland).

[0084] Alternatively, the digital display 590 can be a picture-in-picture display and can display at least two digital images simultaneously. For example, the digital display 590 can simultaneously display a working digital image 591 and a retinal arch alignment template 594. The working digital image 591 can be displayed as the main image, and the retinal arch alignment template 594 can be displayed at the insertion position. Alternatively, the working digital image 591 can be displayed as the main image, and the working digital image 591 overlaid with the retinal arch alignment template 594 can be displayed at the insertion position.

[0085] Digital display 590 can display working digital images 591, template digital images 592, digital mirror images 593, retinal arch alignment templates 594, or any combination thereof, generated by processor 580 or another processor, as well as other information generated by processor 580 or another processor. This information may include graphical or textual information such as surgical parameters, surgical mode, flow rate, intraocular pressure, endoscopic video, OCT images, warnings, digital images, color-coded information, or augmented reality information. Processor 580 can reformat video produced using camera 550 for display on digital display 590, which can be viewed with circularly polarized glasses, digital eyepieces, or a head-mounted display.

[0086] The digital image optimization system 500 can be used to optimize digital images of the eye by providing a retinal arch alignment template 594 for retinal reattachment after vitrectomy. The image processing system 570 can detect any misalignment of the retina during reattachment using the retinal arch alignment template 594. Misalignment of the retina can be adjusted using, for example, a soft-tipped cannula. The image processing system 570 can also be used to quantify the degree of misalignment. The degree of misalignment can be quantified mathematically, graphically, or in combination thereof to inform the surgeon's assessment of retinal reattachment. The digital image optimization system 500 can further use mathematical information, graphical information, or a combination thereof to suggest surgical alignment movements to the surgeon.

[0087] Figure 6 A flowchart is presented of a method for improving digital visualization of ophthalmic surgery by optimizing digital images of the eye using a retinal arch alignment template. In step 600, a digital image of the contralateral eye is captured prior to surgery, such as template digital image 592. In step 610, a digital image of the operated eye is captured during surgery, such as working digital image 591. In step 620, a mirror image of the digital image of the contralateral eye is generated, such as digital mirror image 593. In step 630, the mirror image of the contralateral eye is aligned with the digital image of the operated eye using features of the retinal arch to generate a retinal arch alignment template, such as retinal arch alignment template 594. For example, the digital image can be aligned by aligning the optic disc, central retinal vein, arterial-venous intersection, branch veins, or any combination thereof. The digital images can be aligned in 2D or 3D. In step 640, the retinal arch alignment template is displayed as an overlay on the digital image of the operated eye. In step 650, ophthalmic surgery is performed on the operated eye using the retinal arch alignment template. Examples of ophthalmic surgeries that will benefit from improved digital visualization using retinal arch alignment templates include retinal reattachment after vitrectomy.

[0088] In another alternative example, the digital image optimization system 700 can improve the digital visualization of ophthalmic surgery by providing super-resolution optimization of endoscopic digital images of the eye, such as... Figure 7 As depicted. This can improve the digital visualization of ophthalmic surgery by increasing the resolution of low-resolution digital images (e.g., endoscopic digital images of approximately 30,000 pixels). This can allow low-resolution digital images to be displayed on high-resolution digital displays (e.g., digital displays with more than one million pixels) with improved image quality.

[0089] Now for reference Figure 7 The digital image optimization system 700 may include an endoscope 715, an optical fiber 740, an optical fiber light source 741, a camera 750, an image processing system 770, and a digital display 790. The image processing system 770 may further include a processor 780 and a memory medium 781. The digital image optimization system 700 can provide optimized endoscopic digital images 791 of the eye 101 to improve the digital visualization of ophthalmic surgery.

[0090] An endoscope 715 can be inserted into the eye 101. The endoscope 715 can be positioned such that a desired field of view 760 of the interior of the eye 101 is captured in an optimized endoscopic digital image 791. An optical fiber 740 can be located within the endoscope 715 and can extend to the tip of the endoscope 715. The optical fiber 740 may include approximately 30,000 imaging fibers. Alternatively, the optical fiber 740 may include any suitable number of imaging fibers to provide the desired optimized endoscopic digital image 791 of the eye 101 using the endoscope 715.

[0091] Fiber optic cable 740 can be connected to fiber optic light source 741. Fiber optic light source 741 can be a laser source, narrowband laser source, broadband laser source, supercontinuum laser source, incandescent bulb, halogen bulb, metal halide bulb, xenon bulb, mercury vapor bulb, light-emitting diode (LED), laser engine, other suitable light source, or any combination thereof. Light reflected from the interior of eye 101 can propagate through the image fiber in fiber optic cable 740 and can be detected by camera 750. In addition to camera 750 (not shown), digital image optimization system 700 may also include an eyepiece for endoscope 715.

[0092] Camera 750 may include at least one camera sensor 752. The at least one camera sensor 752 may be a complementary metal-oxide-semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor. Camera 750 may be a monochrome camera or a color camera, and the at least one camera sensor 752 may be a monochrome image sensor or a color image sensor. The at least one camera sensor 752 may use light propagated by optical fiber 740 to capture digital images, which may be light reflected from the interior of eye 101. The at least one camera sensor 752 may capture digital images of eye 101, which may be endoscopic digital images of eye 101.

[0093] The optical fiber 740, optical fiber light source 741, camera 750, and at least one camera sensor 752 can be controlled by a control device 742. For example, the control device 742 can adjust the intensity of the optical fiber light source 741, the sensitivity of the at least one camera sensor 752, or any combination thereof. Although Figure 7 A single endoscope 715 in a digital image optimization system 700 is shown, but the digital image optimization system 700 may include multiple endoscopes 715, optical fibers 740, cameras 750, and camera sensors 752. In this case, multiple endoscopes 715 can be inserted into multiple locations in the eye 101 to provide multiple endoscopic digital images of the eye.

[0094] The digital image optimization system 700 may include an image processing system 770. Digital images captured by at least one camera sensor 752 can be processed by the image processing system 770. The image processing system 770 may include a processor 780. The camera 750 may use at least one camera image sensor 752 to detect light reflected from the interior of the eye 101 and propagated by an optical fiber 740, and send a signal corresponding to the detected light to the processor 780. The processor 780 may execute instructions for generating endoscopic digital images of the eye 101.

[0095] An endoscopic digital image of the eye 101 captured by camera 750 can be processed by image processing system 770. Image processing system 770 may also include memory medium 781. The endoscopic digital image of the eye 101 can be stored in memory medium 781. Processor 780 can execute instructions for applying an endoscopic digital image optimization algorithm to the endoscopic digital image of the eye 101 to generate an optimized endoscopic digital image 791 of the eye. The optimized endoscopic digital image 791 of the eye can be a super-resolution digital image of the eye. A super-resolution digital image can be generated by enlarging a low-resolution endoscopic digital image of the eye to generate a high-resolution endoscopic digital image of the eye. Examples of methods that can be used to generate the optimized endoscopic digital image 791 of the eye include, but are not limited to, interpolation, multiple exposure image denoising, machine learning, deep learning, or any combination thereof, as will be described. For example, using an interpolation method can generate the optimized endoscopic digital image 791 of the eye faster than using a machine learning method. For example, compared to using interpolation methods, machine learning methods can generate improved, optimized digital endoscopic images of the eye 791.

[0096] An endoscopic digital image optimization algorithm can use interpolation to generate an optimized endoscopic digital image 791 of the eye. For example, the algorithm can generate an optimized endoscopic digital image 791 of the eye 101 by replacing lower-quality object pixels in the endoscopic digital image of the eye 101 with neighboring pixels that have higher contrast, higher sharpness, higher clarity, or any combination thereof. Lower-quality object pixels can be pixels that do not display the expected color, pixels that permanently display a specific color, pixels that are permanently white, pixels that are permanently black, pixels that are permanently displayed as "hot pixels," pixels that are bad pixels, or any combination thereof. Object pixels can be modified to optimize pixel image quality without any noticeable image distortion because the modified area is typically small. Examples of interpolation methods that can be included in the endoscopic digital image optimization algorithm include, but are not limited to, nearest neighbor interpolation, linear interpolation, bilinear interpolation, bicubic interpolation, anisotropic filtering, or any combination thereof.

[0097] In another example, the endoscopic digital image optimization algorithm can use multi-exposure image denoising to generate an optimized endoscopic digital image 791 of the eye. The surgeon can dynamically scan the endoscope 715 over the region of interest, allowing multiple endoscopic digital images of the eye 101 to be combined to give a single optimized endoscopic digital image 791 of the eye.

[0098] Endoscopic digital image optimization algorithms can be trained machine learning models. As will be discussed, a machine learning model can be trained using multiple training images, which can be multiple endoscopic digital images of the eye that have been successfully optimized using image processing methods (such as one of the imaging processing methods described above). For example, if a surgeon successfully performs surgery using optimized endoscopic digital images of the eye, the optimization of the endoscopic digital images of the eye can be considered successful.

[0099] In a further example, the endoscopic digital image optimization algorithm can use deep learning methods to generate optimized endoscopic digital images of the eye 791. For example, the endoscopic digital image optimization algorithm can utilize single-image super-resolution (SISR) using deep learning. SISR can be a process of generating a high-resolution digital image from a single low-resolution digital image. In one example, a very deep super-resolution (VDSR) neural network can be used for SISR. The VDSR neural network can learn the mapping between low-resolution and high-resolution digital images. Typically, low-resolution and high-resolution digital images of the same object may have similar image content. The low-resolution digital image may differ from the high-resolution digital image in high-frequency details. The pixel value difference between a high-resolution image and a low-resolution image of the same object that has been magnified to match the size of the high-resolution image can be called a residual image. The VDSR neural network can learn to estimate the residual image from a set of training images of a pair of low-resolution and high-resolution endoscopic digital images of the eye. By adding the estimated residual image to a magnified low-resolution endoscopic digital image of the eye, a high-resolution endoscopic digital image of the eye can be reconstructed from the low-resolution endoscopic digital image of the eye. By using a VDSR neural network, SISR can recover a high-resolution endoscopic digital image of eye 101 from a low-resolution endoscopic digital image of eye 101. The high-resolution endoscopic digital image of eye 101 can be an optimized endoscopic digital image 791 of the eye. Alternatively, the endoscopic digital image optimization algorithm can use other suitable neural networks to perform SISR.

[0100] The endoscopic digital image optimization algorithm can perform the following operations on the endoscopic digital image of the eye 101 captured by camera 750: at least partially reduce noise, increase resolution, improve digital image quality, or any combination thereof. For example, compared to the endoscopic digital image of the eye 101 captured by camera 750 without the application of the endoscopic digital optimization algorithm, the optimized endoscopic digital image 791 of the eye can have increased contrast, sharpness, clarity, dynamic range, or any combination thereof. Although the digital image optimization system 700 includes an endoscope 715, a super-resolution digital image of the eye can also be provided to the digital image of the eye captured by a camera (e.g., a camera such as camera 150 or camera 550).

[0101] The optimized endoscopic digital image 791 can be displayed on a digital display 790. The digital display 790 can include any type of screen or projector capable of displaying the optimized endoscopic digital image 791 at a sufficient resolution for ophthalmic surgery. For example, the display can include any type of screen or projector used in conjunction with ophthalmic surgery, including the type of display used in conventional vitreoretinal surgery systems to present digital images. The digital display 790 can simultaneously display the optimized endoscopic digital image 791 of eye 101 and another digital image, such as an optimized digital image 191 of the eye. The digital display 790 can display a digital image that is a combination of the endoscopic digital image of eye 101 and the optimized endoscopic digital image 791. The combined digital image can be generated by a processor 780. The ratio of the endoscopic digital image of eye 101 and the optimized endoscopic digital image 791 of the eye displayed in the combined image can be controlled by the surgeon (e.g., using a slider controlled by a controller 792). In this way, surgeons can control the extent to which optimized endoscopic digital images 791 are used for visualization during surgery.

[0102] The digital display 790 can display a single image or two images for stereoscopic viewing. The digital display 790 can be a digital display, a screen, a heads-up display, a head-mounted display, or any combination thereof, and may also include multiple displays. The digital display 790 can be a flat panel display or an ultra-high-definition 3D flat panel display. The digital display 790 can be a 3D organic light-emitting diode (OLED) surgical display. The image displayed on the digital display 790 can be viewed through a pair of passive circularly polarized glasses. The digital display 790 can be a component of a digitally assisted vitreoretinal surgery (“DAVS”) system, or can be... Components of a 3D visualization system (Novartis AG, Switzerland).

[0103] Alternatively, the digital display 790 can be a picture-in-picture display and can display at least two digital images simultaneously. For example, the digital display 790 can simultaneously display an optimized endoscopic digital image 791 and another digital image of the eye 101, such as an optimized digital image 191 of the eye. The optimized endoscopic digital image 791 can be displayed as the main image, while the other digital image of the eye 101 can be displayed in the insertion position. Alternatively, the other digital image of the eye 101 can be displayed as the main image, while the optimized endoscopic digital image 791 can be displayed in the insertion position.

[0104] The digital display 790 can display optimized endoscopic digital images 791 generated by the processor 780 or another processor, as well as other information generated by the processor 780 or another processor. This information may include graphical or textual information such as surgical parameters, surgical mode, flow rate, intraocular pressure, endoscopic video, OCT images, warnings, digital images, color-coded information, or augmented reality information. The processor 780 can reformat video produced using the camera 750 for display on the digital display 790, which can be viewed using circularly polarized glasses, a digital eyepiece, or a head-mounted display.

[0105] Figure 8 A flowchart is presented of a method for improving the digital visualization of ophthalmic surgery by optimizing endoscopic digital images of the eye using an endoscopic digital image optimization algorithm. In step 800, an endoscopic digital image of the patient's eye is captured using an endoscope (e.g., endoscope 715). In step 810, the endoscopic digital image optimization algorithm is applied to the endoscopic digital image of the eye to generate an optimized endoscopic digital image of the eye, such as an optimized endoscopic digital image of eye 791. The endoscopic digital image optimization algorithm may use interpolation, multiple exposure image denoising, machine learning, deep learning, or any combination thereof. In step 820, the optimized endoscopic digital image of the eye is displayed on a digital display (e.g., digital display 790).

[0106] Figure 9A flowchart is presented outlining a method for using a machine learning model to generate an endoscopic digital image optimization algorithm to improve the digital visualization of ophthalmic surgery. In step 900, an endoscopic digital image of the patient's eye is captured, such as an endoscopic digital image of eye 101 captured by endoscope 715. In step 910, the endoscopic digital image of the eye is optimized using imaging processing methods such as interpolation, multiple exposure image denoising, or a combination thereof, to generate an optimized endoscopic digital image of the eye, such as optimized endoscopic digital image 791. The optimization of the endoscopic digital image of the eye can be considered successful if the surgeon successfully completes the surgery using the optimized endoscopic digital image of the eye. In step 920, the endoscopic digital image of the eye and the optimized endoscopic digital image of the eye are used, respectively, as input and output images in a set of training images to train the machine learning model. In step 930, steps 900-920 are repeated for multiple input and output images of endoscopic digital images from multiple patients. These images can be collected from numerous surgeons and ophthalmic hospitals. Multiple input and output images can be used to train a machine learning model to predict output images (e.g., optimized endoscopic digital images of the eye) when only input images (e.g., endoscopic digital images of the eye) are provided. In step 940, the trained machine learning model can be used as an endoscopic digital image optimization algorithm to improve endoscopic digital images of the eye.

[0107] like Figure 10 The digital image optimization system 100, digital image optimization system 500, and digital image optimization system 700 described herein can be used in conjunction with an optimization network 1000. The optimization network 1000 may further include a communication network 1010 and a digital image management system 1015. Figure 10 An example is shown of generating an optimized digital image 1191 of the eye from a digital image of the eye 101 using a digital image management system 1015. Although Figure 10 The optimized network 1000 includes a camera 150, but the optimized network 1000 may alternatively include an endoscope, such as endoscope 715, or any other suitable ophthalmic visualization system. The digital image management system 1015 can communicate with a local image processing system, such as image processing system 170, via communication network 1010. Alternatively, the digital image management system 1015 may be part of or include a local image processing system (such as image processing system 170).

[0108] The digital image management system 1015 can use multiple training images of the eye 101 to train a machine learning model. For example, the digital image management system 1015 can use multiple digital images of the eye taken at the beginning of surgery and multiple pairs of digital images of the same eye taken at the end of surgery (e.g., digital image 201 of the eye at the beginning of surgery and digital image 202 of the eye at the end of surgery) to train the machine learning model offline. Alternatively, the digital image management system 1015 can use multiple endoscopic digital images of the eye that have been successfully optimized using image processing methods to train the machine learning model offline. Training images used to train the machine learning model can be uploaded to the digital image management system 1015 via a communication network 1010. Images used to train the machine learning model can be used with a digitally assisted vitreoretinal surgery system (e.g., [missing information]). The 3D visualization system (Novartis AG, Switzerland) is used to capture images. Images used to train the machine learning model can be taken by multiple surgeons. Images used to train the machine learning model can be images of multiple patients in multiple locations. Images used to train the machine learning model can have similar resolution, similar quality, be taken with similar cameras, be taken during similar medical procedures, or any combination thereof.

[0109] During surgery, the digital image management system 1015 can use a trained machine learning model to process digital images of the eye 101 to produce an optimized digital image 1191 of the eye. Compared to a digital image of the eye 101 taken by camera 150 without processing by the digital image management system 1015, the optimized digital image 1191 of the eye may have less instrument glare, less optical aberration, less vitreous haze, or any combination thereof. Compared to a digital image of the eye 101 taken by camera 150 without processing by the digital image management system 1015, the optimized digital image 1191 of the eye may have increased contrast, sharpness, clarity, dynamic range, or any combination thereof.

[0110] Figure 11An example solution for determining an optimized digital image of the eye is shown, which may be an optimized digital image of the eye 1191. A digital image management system 1015 may include an image processing system 1170, which may further include a processor 1180, a memory medium 1181, and a machine learning model 1025. The image processing system 1170 determines the optimized digital image of the eye based on the digital image of the eye 101 by calculating a difference based on information in the digital image of the eye 101 and combining that difference with the digital image of the eye 101. The difference may be calculated based on the raw pixel data of the digital image of the eye 101. For example, the digital image of the eye 101 may be captured by the digital image optimization system 100 at the start of surgery. The digital image of the eye 101 may include instrument glare, optical aberrations, vitreous fog, or any combination thereof. The image processing system 1170 may calculate the difference based on the raw pixel data of the digital image of the eye 101. The difference may be calculated using the machine learning model 1025. The image processing system 1170 can further determine an optimized digital image of the eye 101 by combining the digital image of the eye 101 with the difference.

[0111] like Figure 11 As shown, the digital image optimization system 100 can capture a digital image of eye 101 and send it to the image processing system 1170. The digital image optimization system 100 can use a communication network (such as communication network 1010) to send the digital image of eye 101 to the image processing system 1170. The image processing system 1170 can be included in the digital image management system 1015. Therefore, the image processing system 1170 can process the digital image of the eye captured from a remote ophthalmic visualization system to produce an optimized digital image. Alternatively, once trained, the image processing system 1170 can be stored locally on the ophthalmic visualization system, for example, as the image processing system 170 within the digital image optimization system 100. In this example, the image processing system 1170 can optimize the digital image of eye 101 locally.

[0112] Image processing system 1170 can use machine learning model 1025 to calculate the difference between the digital images of the eye 101. For example... Figure 11As depicted, a machine learning model 1025 can be trained using multiple training images 1030. Training images 1030 for multiple eyes 1101 can be obtained. Each set of training images 1030 includes an input image and an output image, where the input image can be, for example, a digital image 201 of the eye at the start of surgery, and the output image can be, for example, a digital image 202 of the eye at the end of surgery. The image processing system 1170 can access the training images 1030 to train the machine learning model 1025. The machine learning model 1025 can be trained based on: (1) extracting features from the input images, and (2) learning a relational function between the extracted features and the differences between the input and output images. Therefore, the machine learning model 1025 can be used to predict the output image when only the input image is provided. Alternatively, any suitable training images 1030 can be obtained in any suitable manner to train the machine learning model 1025 to optimize the digital images of the eyes.

[0113] Figure 12An example ensemble architecture 1200 of a machine learning model 1025 is shown for computing differences based on information in digital images of the eye, for example, for computing the difference between a digital image 201 of the eye at the start of surgery and a digital image 202 of the eye at the end of surgery. The ensemble architecture 1200 may include a convolutional neural network (CNN) 1210 and at least one long short-term memory unit (LSTM) 1220. The CNN 1210 may include a class of deep feedforward artificial neural networks capable of efficiently analyzing images. The CNN 1210 may use a variant of a multilayer perceptron designed to require minimal preprocessing. A multilayer perceptron may also be referred to as a shift-invariant or spatially invariant artificial neural network due to its shared weight architecture and translation-invariant properties. The CNN 1210 may include multiple sequential layers. At least one LSTM unit 1220 may include a recurrent neural network, which may be used as a building block or hidden layer block for a eventually larger recurrent neural network. The LSTM unit 1220 itself may be a recurrent network because it may contain recurrent connections similar to those in a conventional recurrent neural network. A convolutional neural network 1210 and at least one long short-term memory (LSM) unit 1220 can be integrated together. Integration can be achieved by inserting at least one LSM unit 1220 after the last layer of a plurality of sequential layers of the convolutional neural network 1210. The convolutional neural network 1210 may include a final sequential layer as a fully connected layer 1211. At least one LSM unit 1220 can be inserted after the fully connected layer 1211. At least one LSM unit 1220 may correspond to raw pixel data of a digital image of the eye, such as image width, image length, color channel data, or any combination thereof. The output of at least one LSM unit 1220 may be an output vector 1230. The image processing system 1170 can use the output vector 1230 to compute differences based on information in the digital image of the eye. Alternatively, the image processing system 1170 can use any suitable architecture to train the machine learning model 1025.

[0114] Image processing system 1170 can generate feature representations of multiple training images 1030 by processing them using convolutional neural network 1210. Convolutional neural network 1210 can be based on a pre-trained classification network. For example, the pre-trained classification network may include a residual network. Image processing system 1170 can implement convolutional neural network 1210 by modifying the last fully connected layer 1211 of the pre-trained classification network. Image processing system 1170 can use the output of convolutional neural network 1210 as the generated feature representation of the multiple training images 1030. Examples of feature representations can be numerical vectors. Image processing system 1170 can also use the output of convolutional neural network 1210 as input to at least one long short-term memory unit 1220. Alternatively, any suitable convolutional neural network can be used in any suitable manner to generate feature representations.

[0115] Image processing system 1170 can learn a relationship function between the feature representations of multiple training images 1030 and the differences between input and output image pairs of training images 1030. The learning of the relationship function can be based on machine learning ensemble architecture 1200 and a specific loss function. The relationship function may include a mapping function. The input to the mapping function may include the feature representation of the input image and the difference between the input image and the corresponding output image. The output of the mapping function may include the difference between the hypothesized eye prediction and the corresponding output image. The difference may include values ​​corresponding to the original pixel data of the digital image, such as image width, image length, color channel data, or any combination thereof. The mapping function may further be based on a specific loss function. For example, the loss function may be a smoothed L1 loss function. For any variable x, the smoothed L1 loss function can be defined as:

[0116]

[0117] As an alternative, the image processing system 1170 can learn any suitable relational function in any suitable manner.

[0118] Figure 13A flowchart illustrating a method for training a machine learning model (e.g., machine learning model 1025) is presented. In step 1300, an image processing system (e.g., image processing system 1170) may input multiple training images (e.g., training image 1030) into an integrated architecture (e.g., integrated architecture 1200). The multiple training images may include multiple input images and multiple corresponding output images. The input images may be, for example, a digital image 201 of the eye at the start of surgery, and the output images may be, for example, a digital image 202 of the eye at the end of surgery. In step 1310, the image processing system may use a convolutional neural network (e.g., convolutional neural network 1210) to generate feature representations of the multiple input images of the multiple training images. In step 1320, the image processing system may use the integrated architecture to compute multiple hypothesis differences between multiple predicted optimized digital images of the eye and their corresponding output images. The multiple hypothesis differences may be computed using the feature representations of the multiple corresponding input images. In step 1330, the image processing system may compute a loss between the multiple hypothesis differences and the multiple differences computed between the multiple input images and the multiple corresponding output images. The loss can be calculated using a loss function, which may include a smoothed L1 loss function. In step 1340, the image processing system can evaluate whether the total loss is less than a predefined threshold. If the total loss is not less than the predefined threshold, the image processing system can proceed to step 1350. In step 1350, the image processing system can update the parameters of the ensemble architecture. The loop of steps 1310 to 1340 can then be repeated. If the total loss is less than the predefined threshold, the image processing system can proceed to step 1360. In step 1360, the image processing system can store the parameters of the ensemble architecture and end training. This may be because the machine learning model has converged. Alternatively, any suitable method used to train the machine learning model to optimize digital images of the eye can be used.

[0119] Image processing system 1170 can be trained on a system associated with digital image management system 1015. Once trained, image processing system 1170 can run on a server and can be used to provide cloud-based digital image optimization services. Alternatively, once trained, image processing system 1170 can be distributed to a local ophthalmic visualization system. This can include local digital image optimization systems such as digital image optimization system 100, digital image optimization system 500, digital image optimization system 700, or any combination thereof.

[0120] For example, a trained image processing system 1170 can receive digital images of the eye. These digital images can be captured by a camera such as camera 150 or an endoscope such as endoscope 715. The image processing system 1170 can compute an optimized digital image of the eye by processing the digital images of the eye using a trained machine learning model 1025. The image processing system 1170 can first generate a feature representation of the digital image of the eye by processing the digital images of the eye using the trained machine learning model 1025 (e.g., using a convolutional neural network 1210). Then, the image processing system 1170 can process the generated feature representation and the digital image of the eye using a trained machine learning model (e.g., using at least one long short-term memory unit 1220). The image processing system 1170 can then output the difference between the digital image of the eye and a predicted optimized digital image of the eye. The image processing system 1170 can use this difference to compute the optimized digital image of the eye from the digital images of the eye. Alternatively, the image processing system 1170 can compute an optimized digital image of the eye for any suitable digital image of the eye.

[0121] like Figure 14 The depicted digital image optimization system 100, digital image optimization system 500, digital image optimization system 700, and optimization network 1000 can be used in conjunction with the digital timeout system 1400. The digital timeout system 1400 may include an identification tag 1405, a camera 1450, an image processing system 1470, a digital display 1490, a communication network 1010, and a digital image management system 1015. The image processing system 1470 may further include a processor 1480 and a memory medium 1481.

[0122] Camera 1450 can be a camera such as camera 150. Camera 1450 can be a component of a digitally assisted vitreoretinal surgery (“DAVS”) system, or it can be... A component of a 3D visualization system (Novartis AG, Switzerland). Digital display 1490 can be a display such as digital display 190. Digital display 1490 can be a component of a digitally assisted vitreoretinal surgery (“DAVS”) system, or can be... Components of a 3D visualization system (Novartis AG, Switzerland).

[0123] Identification tag 1405 may be affixed to patient 1410. Identification tag 1405 may be affixed above patient 1410's eye 101. Identification tag 1405 may be affixed above patient 1410's eye 101 where surgery will be performed. Alternatively, identification tag 1405 may be affixed to patient 1410 at any location convenient for surgery. Identification tag 1405 may be a removable decal tattoo, patch, sticker, tape, or any combination thereof. Identification tag 1405 may also be any other suitable mark for affixing to patient 1410 such that it will not fall off, become soiled, be misapplied, or any combination thereof.

[0124] The identification tag 1405 can be black with white text. Alternatively, the identification tag 1405 can be white with black text. The identification tag 1405 may also include a locator 1430, which may be, for example, a crosshair. The camera 1450 can recognize the locator 1430 and position itself at a specific location relative to the locator 1430. The camera 1450 can be a robot-controlled camera. The camera 1450 can move in six degrees of freedom. The locator 1430 may allow the camera 1450 to automatically position itself above the patient 1410's eye 101.

[0125] The identification tag 1405 may include machine-readable information 1420 associated with the patient 1410. For example, the machine-readable information 1420 may include a machine-readable label, such as a barcode. The barcode may be a one-dimensional barcode. The barcode may be a multi-dimensional barcode. The barcode may be a Quick Response (QR) code 1425. In another example, the machine-readable information 1420 may be or include symbols. Symbols may be or include letters of a written language (e.g., English, German, French, Chinese, Russian, etc.) that can be processed via an Optical Character Recognition (OCR) process and / or system.

[0126] Machine-readable information 1420 can be used to store information associated with patient 1410. Information associated with patient 1410 may include identification information associated with patient 1410. Identification information may include one or more of the following: name, date of birth, and government identity. Information associated with patient 1410 may also include medical information associated with patient 1410. Medical information associated with patient 1410 may include one or more of the following: medical procedures, medication lists, patient 1410's doctor, patient 1410 as part of a medical procedure, allergies to one or more drugs, diagnoses, and patient 1410's orientation toward medical procedures. Medical information associated with patient 1410 may include the eye to be operated on, such as the right or left eye. In another instance, information associated with patient 1410 may include indexing information. Indexing information may be used to index into a database and / or storage device storing information associated with patient 1410. Indexing information may be used as a key or for creating a key that can be used to retrieve information associated with patient 1410 from the database and / or storage device storing information associated with patient 1410.

[0127] Machine-readable information 1420 can be printed on any necessary supporting media format on the identification tag 1405, such as paper, cloth, plastic card, sticker, or any combination thereof. Machine-readable information 1420 may include barcodes and / or custom optical codes and / or markings. Camera 1450 can capture a digital image of the identification tag 1405 and send the signal to processor 1480. Camera 1450 can also capture a digital image of the patient 1410's eye 101 to be operated on and send the signal to processor 1480. Camera 1450 can acquire machine-readable information 1420. Machine-readable information 1420 can be processed by image processing system 1470 using processor 1480. The digital image of the patient 1410's eye 101 can also be processed by image processing system 1470 using processor 1480. Camera 1450 can automatically acquire machine-readable information 1420 by recognizing the identification tag 1405 in the operating room. Once the camera 1450 acquires machine-readable information 1420, the machine-readable information 1420 can initiate automatic white balance color calibration of the camera 1450.

[0128] Processing machine-readable information 1420 may include processing one or more images to determine information stored via the machine-readable information 1420. For example, the one or more images may include one or more barcodes having information associated with a medical procedure. The one or more barcodes may include the information described herein, and processing the machine-readable information 1420 may include retrieving the information associated with the medical procedure based on the machine-readable information 1420.

[0129] Processor 1480 can be used to process machine-readable information 1420 to determine patient data. In one example, machine-readable information 1420 may include patient data. In another example, processing machine-readable information 1420 to determine patient data may include retrieving one or more portions of patient data from a storage device. The patient data retrieved from the storage device may include information about planned medical procedures, image data from previous ophthalmic surgeries or examinations, diagnostic data from previous ophthalmic surgeries or examinations, information about previous eye conditions, or any combination thereof. The retrieved patient data may be displayed on digital display 1490. The retrieved patient data may be displayed on digital display 1490 as a picture-in-picture display. Machine-readable information 1420 may include encrypted information. For example, processing machine-readable information 1420 to determine patient data may include: decrypting information stored via machine-readable information 1420 to determine patient data.

[0130] Machine-readable information 1420 can be read at the start of surgery. Machine-readable information 1420 can be read at the end of surgery. Alternatively, machine-readable information 1420 can be read at any necessary time during surgery. Reading of machine-readable information 1420 can trigger a digital timeout. The digital timeout can provide a computational check to match the information associated with patient 1410 with the information provided to the surgeon, operating room staff, or a combination thereof. The digital timeout can be used to detect any errors in the information provided to the surgeon, operating room staff, or a combination thereof compared to the information associated with patient 1410. For example, the digital timeout can provide a surgical equivalent that performs parity checks on the patient's identity, the planned medical procedure, the eye to be operated on, or any combination thereof.

[0131] The digital timeout can confirm the identity of patient 1410. During the digital timeout, image processing system 1470 can compare the identity information associated with patient 1410 stored in machine-readable information 1420 with information provided to the surgeon, operating room staff, or a combination thereof. During the digital timeout, image processing system 1470 can use machine-readable information 1420 to retrieve an image of the patient's eye and compare that image with a digital image of patient 1410's eye 101 captured by camera 1450. This can confirm the identity of patient 1410.

[0132] A digital timeout can confirm a medical procedure planned for patient 1410. During the digital timeout, image processing system 1470 can compare the identity of the eye to be operated on, provided in the medical information associated with patient 1410, with the identity of the eye to be operated on, provided to the surgeon, operating room staff, or a combination thereof. During the digital timeout, image processing system 1470 can compare the medical procedure to be performed with the planned medical procedure recorded in the medical information associated with patient 1410.

[0133] Image processing system 1470 can determine the discrepancies between information associated with patient 1410 provided by machine-readable information 1420 and information provided to surgeons, operating room staff, or a combination thereof. Image processing system 1470 can determine the consistency between information associated with patient 1410 and information provided to surgeons, operating room staff, or a combination thereof. Image processing system 1470 can determine the discrepancies, consistency, or combination thereof by displaying information associated with patient 1410 and information provided to surgeons, operating room staff, or a combination thereof on digital display 1490. Alternatively, image processing system 1470 can use any suitable means to report the discrepancies, consistency, or combination thereof between information associated with patient 1410 and information provided to surgeons, operating room staff, or a combination thereof. If a discrepancy exists between information associated with patient 1410 and information provided to surgeons, operating room staff, or a combination thereof, digital timeout system 1400 can prevent the surgeon from using medical equipment, such as camera 1450.

[0134] Machine-readable information 1420 can also be used in an optimized network (e.g., optimized network 1000). Machine-readable information 1420 may include indexing information that can be associated with or stored digitally with a corresponding patient digital image of eye 101. For example, the indexing information provided by machine-readable information 1420 may be digitally stored in a digital image management system 1015 along with a corresponding digital image of the patient 1410's eye 101. The digital image management system 1015 can generate an optimized digital image 1491 of the eye from the digital image of eye 101. The digital image management system 1015 can communicate with a local image processing system, such as image processing system 1470, via communication network 1010.

[0135] The digital image management system 1015 may include an image processing system, such as image processing system 1170, which can train a machine learning model, such as machine learning model 1025, using patient digital images of eyes 101 from a set of training images (such as training image 1030). Information associated with patient 1410 provided by machine-readable information 1420 can provide additional information about the patient digital images of eyes 101. This may be useful for record-keeping of digital images of eyes stored in an optimized network. The information associated with patient 1410 provided by machine-readable information 1420 can also be used to select training images for training the machine learning model. For example, information such as gender, age, eye condition, or any combination thereof can be used to filter the digital images of eyes selected as training images to provide a customized image processing system to compute optimized image 1491 of the eye.

[0136] Figure 15 A flowchart illustrating the method for executing a digital timeout is presented. In step 1500, machine-readable information (such as machine-readable information 1420 printed on identification tag 1405) may be acquired by a camera (such as camera 1450). In step 1510, the machine-readable information is processed, for example, by image processing system 1470. In step 1520, the correctness of the patient's identity is determined based on the machine-readable information compared to information provided to the surgeon, operating room staff, or a combination thereof. The correctness of the patient's identity can be determined by comparing the identity information associated with the patient (such as the identity information associated with patient 1410) with information provided to the surgeon, operating room staff, or a combination thereof (which may be, for example, a list of patients scheduled for medical procedures that day). In step 1530, if the patient's identity is incorrect, the planned medical procedure is stopped. In step 1540, if the patient's identity is correct, the correctness of the planned medical procedure is determined based on the machine-readable information compared to the information provided to the surgeon, operating room staff, or a combination thereof. The correctness of a planned medical procedure can be determined by comparing the patient's associated medical information (such as the medical information associated with patient 1410) with information provided to the surgeon, operating room staff, or a combination thereof (which may be, for example, a list of planned medical procedures for the day). In step 1550, if the planned medical procedure is incorrect, the planned medical procedure is stopped. In step 1560, if the planned medical procedure is correct, the planned medical procedure is performed.

[0137] Figure 16A flowchart of a method for optimizing digital images of an eye for ophthalmic surgery is presented. In step 1600, a digital image of the eye, such as a digital image of eye 101, is captured by a digital image optimization system. The digital image optimization system may be, for example, digital image optimization system 100, and the digital image of the eye may be captured, for example, by camera 150. In another example, the digital image optimization system may be digital image optimization system 500, and the digital image of the eye may be captured by camera 550. Alternatively, the digital image optimization system may be digital image optimization system 700, and the digital image of the eye may be captured by an endoscope (such as endoscope 715). In step 1610, the digital image of the eye is associated with processed machine-readable information (such as machine-readable information 1420). The machine-readable information may have been processed when the digital image of the eye was captured during a digital timeout (e.g., a digital timeout initiated using digital timeout system 1400). The machine-readable information may contain information associated with the patient in the digital image of the eye. In step 1620, a digital image of the eye is used to train a machine learning model, such as machine learning model 1025 in optimization network 1000. For example, the digital image of the eye may be included as input to multiple training images (such as training image 1030). The digital image of the eye may have already been selected for inclusion in training image 1030 based on processed machine-readable information. In step 1630, the digital image of the eye is processed by the trained machine learning model (such as trained machine learning model 1025). In an alternative example, if the machine learning model has already been trained, step 1630 may directly follow step 1600 or step 1610. In step 1640, an optimized digital image of the eye is generated by an image processing system (e.g., image processing system 1170). In step 1650, the optimized digital image of the eye is displayed on a digital display. For example, the optimized digital image may be displayed on digital display 190, digital display 590, digital display 790, or digital display 1490.

[0138] As depicted in Figure 1700, digital image optimization system 100, digital image optimization system 500, digital image optimization system 700, optimization network 1000, or digital timeout system 1400 can be used in conjunction with computer system 1700. Computer system 1700 may include processor 1710, volatile memory medium 1720, non-volatile memory medium 1730, and input / output (I / O) devices 1740. Volatile memory medium 1720, non-volatile memory medium 1730, and I / O devices 1740 can be communicatively coupled to processor 1710.

[0139] The term "memory medium" can mean "memory," "storage device," "memory apparatus," "computer-readable medium," and / or "tangible computer-readable storage medium." For example, memory media can include, but is not limited to, the following storage media: direct access storage devices (including hard disk drives), sequential access storage devices (such as magnetic tape drives), optical discs (CDs), random access memory (RAM), read-only memory (ROM), CD-ROMs, DVDs, electrically erasable programmable read-only memory (EEPROM), flash memory, non-volatile media, and / or any combination thereof. Figure 17 As shown, the non-volatile memory medium 1730 may include processor instructions 1732. Processor instructions 1732 may be executed by processor 1710. In one example, one or more portions of processor instructions 1732 may be executed via the non-volatile memory medium 1730. In another example, one or more portions of processor instructions 1732 may be executed via the volatile memory medium 1720. One or more portions of processor instructions 1732 may be transferred to the volatile memory medium 1720.

[0140] Processor 1710 can execute processor instructions 1732 to implement at least a portion of one or more systems, one or more flowcharts, one or more processes, and / or one or more methods described herein. For example, processor instructions 1732 can be configured, encoded, and / or encoded using instructions according to at least a portion of one or more systems, one or more flowcharts, one or more methods, and / or one or more processes described herein. Although processor 1710 is shown as a single processor, processor 1710 may be or include multiple processors. One or more of storage media and memory media can be software products, program products, and / or articles of manufacture. For example, software products, program products, and / or articles of manufacture can be configured, encoded, and / or encoded using instructions executable by a processor according to at least a portion of one or more systems, one or more flowcharts, one or more methods, and / or one or more processes described herein.

[0141] Processor 1710 may include any suitable system, apparatus, or device operable to interpret and execute program instructions, process data, or both stored in a memory medium and / or received via a network. Processor 1710 may further include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or other circuit systems configured to interpret and execute program instructions, process data, or both.

[0142] I / O device 1740 may include any one or more tools that allow, permit, and / or enable a user to interact with computer system 1700 and its associated components by facilitating input from and output to the user. Facilitating input from the user may allow the user to manipulate and / or control computer system 1700, and facilitating output to the user may allow computer system 1700 to indicate the effects of the user's manipulation and / or control. For example, I / O device 1740 may allow the user to input data, instructions, or both into computer system 1700 and otherwise manipulate and / or control computer system 1700 and its associated components. I / O devices may include user interface devices such as a keyboard, mouse, touchscreen, joystick, handheld lens, tool tracking device, coordinate input device, or any other I / O device suitable for use with the system.

[0143] I / O device 1740 may include one or more buses, one or more serial devices, and / or one or more network interfaces, which may facilitate and / or permit processor 1710 to implement at least a portion of one or more systems, processes, and / or methods described herein. In one example, I / O device 1740 may include a storage interface that may facilitate and / or permit processor 1710 to communicate with external memory. The storage interface may include one or more of a Universal Serial Bus (USB) interface, a SATA (Serial ATA) interface, a PATA (Parallel ATA) interface, and a Small Computer System Interface (SCSI). In a second example, I / O device 1740 may include a network interface that may facilitate and / or permit processor 1710 to communicate with a network. I / O device 1740 may include one or more of a wireless network interface and a wired network interface. In a third example, I / O device 1740 may include one or more of a Peripheral Component Interconnect (PCI) interface, a PCI Express (PCIe) interface, a Serial Peripheral Interconnect (SPI) interface, and an Internal Integrated Circuit (I2C) interface. In the fourth example, I / O device 1740 may include circuitry that allows processor 1710 to communicate data with one or more sensors. In the fifth example, I / O device 1740 may facilitate and / or allow processor 1710 to communicate data with one or more of the following: display 1750 and digital image optimization system 100. Figure 17 As shown, I / O device 1740 can be coupled to network 1770. For example, I / O device 1740 may include a network interface. In another example, network 1770 may be or include network 1010.

[0144] Network 1770 may include a wired network, a wireless network, an optical network, or any combination thereof. Network 1770 may include and / or be coupled to various types of communication networks. For example, network 1770 may include and / or be coupled to a local area network (LAN), a wide area network (WAN), the Internet, a public switched telephone network (PSTN), a cellular telephone network, a satellite telephone network, or any combination thereof. A WAN may include a private WAN, a corporate WAN, a public WAN, or any combination thereof.

[0145] although Figure 17 Computer system 1700 is shown as being external to digital image optimization system 100, but digital image optimization system 100 may include computer system 1700. For example, processor 1710 may be or include processor 180.

[0146] Figure 18 A to Figure 18 C shows an example of a medical system 1800. (Example) Figure 18 As shown in Figure A, the medical system 1800 may include a digital image optimization system 100. Alternatively, the medical system 1800 may include a digital image optimization system 500, a digital image optimization system 700, an optimization network 1000, a digital timeout system 1400, or any combination thereof. Figure 18 As shown in Figure B, the medical system 1800 may include a digital image optimization system 100 and a computer system 1700. The digital image optimization system 100 may be communicatively coupled to the computer system 1700. Figure 18 As shown in Figure C, the medical system 1800 may include a digital image optimization system 100, which may include a computer system 1700.

[0147] Unless explicitly mutually exclusive, digital image optimization systems 100, 500, 700, optimization network 1000, digital timeout system 1400, computer system 1700, medical system 1800, and their components can be combined with other elements of the visualization tools and systems described herein. For example, digital image optimization system 500 can be combined with digital timeout system 1400 and can be used with other optimization systems, visualization systems, computer systems, and medical systems described herein.

[0148] The subject matter disclosed above should be considered illustrative rather than limiting, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments falling within the true spirit and scope of this disclosure. For example, while digital image optimization systems are most often needed to improve the digital visualization of ophthalmic surgery, the systems and methods described herein may also be employed if they are useful in another procedure (such as a purely diagnostic procedure not otherwise considered surgery).

Claims

1. A digital image optimization system, comprising: The camera includes at least one sensor operable to detect light reflected from the eye and send a signal corresponding to the detected light to a processor; An image processing system, the image processing system including the processor and operable to: Execute instructions for generating a digital image of the eye; and Instructions are executed to apply a digital image optimization algorithm, including a trained machine learning model, to digital images of the eye to generate an optimized digital image of the eye, wherein the trained machine learning model is trained using multiple digital images of the eye taken at the start of the surgery and multiple corresponding digital images of the eye taken at the end of the surgery. as well as A digital display operable to display an optimized digital image of the eye.

2. The digital image optimization system as described in claim 1, wherein, The digital image optimization algorithm at least partially reduces instrument glare, optical aberrations, vitreous fog, or any combination thereof in the digital image of the eye to generate an optimized digital image of the eye.

3. The digital image optimization system as described in claim 1, wherein, The contrast, sharpness, clarity, dynamic range, or any combination thereof of the optimized digital image of the eye are equal to or greater than the contrast, sharpness, clarity, dynamic range, or any combination thereof of the digital image of the eye.

4. The digital image optimization system as described in claim 1, wherein, The noise, distortion, vignetting, or any combination thereof in the optimized digital image of the eye is equal to or lower than the noise, distortion, vignetting, or any combination thereof in the digital image of the eye.

5. The digital image optimization system as described in claim 1, wherein, The digital image optimization algorithm uses interpolation, multiple exposure image denoising, or a combination thereof to generate an optimized digital image of the eye.

6. The digital image optimization system as described in claim 1, wherein, The trained machine learning model was trained using multiple training images from the red enhancement library.

7. A digital image optimization system, comprising: An endoscope, the endoscope including optical fibers; The camera includes at least one sensor operable to detect light reflected from inside the eye and propagating through the optical fiber, and to send a signal corresponding to the detected light to a processor; An image processing system, the image processing system including the processor and operable to: Execute instructions for generating endoscopic digital images of the eye; and Instructions are executed to apply an endoscopic digital image optimization algorithm, including a trained machine learning model, to an endoscopic digital image of the eye to generate an optimized endoscopic digital image of the eye, wherein the trained machine learning model is trained using multiple digital images of the eye taken at the start of the surgery and multiple corresponding digital images of the eye taken at the end of the surgery. as well as A digital display operable to display optimized endoscopic digital images of the eye.

Images