Digital perceptual system for ophthalmic surgery

Abstract

Certain embodiments provide a method for performing an ophthalmic surgical procedure. The method includes taking and preparing pre-surgical data and intra-surgical data associated with an eye of a patient for further processing. In some embodiments, the method further includes integrating the pre-surgical data and the intra-surgical data to generate context-sensitive data for further processing. The method further includes classifying and annotating the pre-surgical data, the intra-surgical data, and the context-sensitive data. The method further includes extracting one or more actionable reasones from the pre-surgical data, the intra-surgical data, the context-sensitive data, and the classified and annotated data. The method further includes triggering one or more actions on the imaging system or surgical system based on the one or more actionable reasoning.

Description

Background Art

[0001] Various diseases or conditions associated with the eye can be treated through ophthalmic surgery. Examples of ophthalmic surgery include vitreoretinal surgery, cataract surgery, glaucoma surgery, laser eye surgery (LASIK), and the like.

[0002] Vitreoretinal surgery is a type of eye surgery that treats problems with the retina or vitreous. Vitreoretinal surgery can be performed to treat conditions such as diabetic traction retinal detachment, diabetic vitreous hemorrhage, macular holes, retinal detachment, epiretinal membranes, and many other ophthalmic conditions. Cataract surgery involves emulsifying the patient's lens with an ultrasonic handpiece and aspirating it from the eye. An intraocular lens (IOL) is then implanted in the posterior lens capsule of the eye. During vitreoretinal, cataract, and other types of surgery mentioned above and known to those of ordinary skill in the art, various defects may negatively impact the outcome, efficiency, and effectiveness of the surgery, as well as the ease with which the surgeon can perform the surgery, and in some cases damage the patient's optical anatomy. Summary of the invention

[0003] The present disclosure generally relates to methods and apparatus for performing ophthalmic surgical procedures. In certain embodiments, the method includes ingesting and preparing preoperative data and intraoperative data associated with a patient's eye for further processing. In certain embodiments, the method further includes integrating the preoperative data and intraoperative data to generate context-sensitive data for further processing. The method also includes classifying and annotating the preoperative data, intraoperative data, and context-sensitive data. The method also includes extracting one or more actionable inferences from the preoperative data, intraoperative data, context-sensitive data, and the classified and annotated data. The method further includes triggering one or more actions on an imaging system or a surgical system based on one or more actionable inferences.

[0004] The following description and the associated drawings set forth in detail certain illustrative features of the one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The drawings depict only examples of certain embodiments of the present disclosure and therefore should not be considered limiting the scope of the disclosure.

[0006] Figure 1 An example of a digital perception system (hereinafter referred to as a digital perception system) configured with digital perception technology according to some embodiments is shown.

[0007] Figure 2 According to some embodiments, Figure 1 Example operations used by the number-aware system.

[0008] Figure 3 At least some implementations according to some embodiments are shown. Figure 1 An example computing device that implements one or more functions of a digitally aware system.

[0009] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION

[0010] Although features of the present invention may be discussed with respect to certain embodiments and drawings below, all embodiments of the present invention may include one or more advantageous features discussed herein. In other words, although one or more embodiments may be discussed as having certain advantageous features, one or more of such features may also be used according to various other embodiments discussed herein. In a similar manner, although exemplary embodiments may be discussed below as device, apparatus, or method embodiments, it should be understood that such exemplary embodiments may be implemented in various devices, apparatuses, and methods.

[0011] As described above, during vitreoretinal, cataract, and other types of surgeries mentioned above and known to those of ordinary skill in the art, various deficiencies may negatively impact the outcome, efficiency, and effectiveness of the surgery, as well as the ease with which the surgeon can perform the surgery, and in some cases cause damage to the patient's optical anatomy, etc. For example, existing surgical systems or platforms (hereinafter referred to as systems) are technically deficient when it comes to providing surgical image guidance, surgical patient monitoring, virtual assistance, and other automated operations that can improve surgical outcomes, reduce the likelihood of physically harming the patient's eye, and increase surgical efficiency and effectiveness, etc. Various types of deficiencies with respect to existing surgical systems are described below.

[0012] Image-guided defects

[0013] For example, existing surgical systems are technically unable to identify toric IOL markings (e.g., three laser dots) to improve lens alignment during IOL implantation as part of cataract surgery. In another example, existing surgical systems are technically unable to track incision location during cataract surgery to facilitate easy insertion of an IOL delivery device into the incision for the purpose of IOL injection. In another example, existing surgical systems are technically unable to identify residual lens fragments after phacoemulsification to enable efficient removal of lens material fragments. In yet another example, existing surgical systems are technically unable to provide image guidance during MIGS (minimally invasive glaucoma surgery) implantation. For the sake of brevity, additional examples of image guidance deficiencies associated with existing surgical systems are omitted.

[0014] Patient Monitoring Deficiencies

[0015] During cataract surgery, surgical instruments may rupture the capsular bag, and currently, existing surgical systems are technically unable to accurately detect whether or when the surgical instruments are too close to the capsular bag. In another example, during retinal surgery, surgical instruments may rupture retinal tissue, and currently, existing surgical systems are technically unable to accurately detect whether or when the surgical instruments are too close to the retinal tissue. In yet another example, existing surgical systems are technically unable to assess the cataract grade to suggest the focal setting for phacoemulsification. In yet another example, existing surgical systems are technically unable to monitor and map the path of the vitrectomy device (i.e., vitrectomy) in the eye and suggest the focal area for residual vitreous removal. In another example, existing surgical systems are technically unable to monitor the condition of the patient's eyes during surgery to warn the surgeon of any unexpected conditions. For the sake of brevity, additional examples of patient monitoring defects associated with existing surgical systems are omitted.

[0016] Virtual Assistance Disadvantages

[0017] For example, existing surgical systems are not technically capable of monitoring image quality and adjusting (or suggesting adjustments to) device settings to improve viewing quality. In another example, existing surgical systems are not technically capable of adjusting lighting settings to improve viewing and detection quality of ocular features during surgery. In yet another example, existing surgical systems are not technically capable of automating the glide path of a robotic arm during a surgical procedure. Additional examples of virtual assistance deficiencies associated with existing surgical systems are omitted for brevity.

[0018] Automation flaws

[0019] For example, existing surgical systems are technically incapable of annotating surgical videos for billing purposes and generating teaching / training aids. In another example, existing surgical systems are technically incapable of automatically generating billing codes for surgeries based on the complexity of the surgery. For the sake of brevity, additional examples of automation deficiencies associated with existing surgical systems are omitted.

[0020] Digital sensing surgery system

[0021] The embodiments of this article describe a digital sensing surgery system that provides a technical solution to the above-mentioned technical problems and deficiencies.

[0022] The digital perceptual surgical system described in this article has at least four key technical capabilities, including: (i) the ability to analyze data in real time, (ii) the ability to process multi-model data (i.e., data generated and / or received in different formats simultaneously, such as surgical videos, digital data, voice data, text, images, signals, etc.), (iii) the ability to simultaneously process data received from a single source or from multiple sources (e.g., images captured by a camera, internal sensor data, voice recordings from a microphone), and (iv) the ability to use the received and processed data related to the surgical procedure, surgical instruments, the state of the patient or his or her eyes to reason and control the surgical equipment.

[0023] As described herein, digital sensing technology can provide intelligent functions for surgical systems. Intelligent functions for eye surgery systems can take various forms in operating rooms (ORs), such as image-guided operations, patient monitoring, virtual assistance to surgeons, and / or service automation. Many improvements to existing surgical systems are produced by combining the intelligent functions described by the embodiments of this article. The improved surgical system described herein can assist surgeons in performing surgical tasks with higher accuracy, efficiency, and / or safety, thereby ultimately leading to better surgical results for each patient.

[0024] Figure 1 An example of a digital perception system 100 (hereinafter referred to as a digital perception system) configured with digital perception technology according to some embodiments is shown. As shown, the digital perception system 100 includes various systems, such as one or more pre-operative (hereinafter "pre-operative") imaging systems 110, one or more surgical systems 112, one or more intra-operative (hereinafter "intra-operative") imaging systems 114, and post-operative (hereinafter "post-operative") imaging systems 116.

[0025] The preoperative imaging system 110, surgical system 112, intraoperative imaging system 114, and postoperative imaging system 116 can be co-located or located in a variety of locations, including diagnostic clinics, surgical clinics, hospitals, and other locations. Whether co-located or located in different locations, the systems 110, 112, 114, and 116 can each generate data that can be transmitted to other systems 110, 112, 114, and 116, computing system(s) 120, and / or to databases 130 and 135 as part of input data 102 via one or more networks (e.g., a local area network, a wide area network, and / or the Internet).

[0026] The preoperative imaging system 110 may refer to any number of diagnostic systems that may be used to obtain multidimensional images and / or ophthalmic anatomy measurements prior to surgery in a clinic, such as an optical coherence tomography (OCT) system, a rotational camera (e.g., a Scheimpflug camera), a magnetic resonance imaging (MRI) system, a keratometer, an ophthalmometer, an optical biometer, a topographer, a retinal camera, any type of intraoperative optical measurement system (such as an intraoperative aberrometer and / or any other type of optical measurement / imaging system). Examples of OCT systems are described in more detail in U.S. Patent No. 9,618,322, which discloses “Process for Optical Coherence Tomography and Apparatus for Optical Coherence Tomography” and U.S. Patent Application Publication No. 2018 / 0104100, which discloses “Optical Coherence Tomography Cross View Image,” each of which is incorporated herein by reference in its entirety.

[0027] Surgical system 112 may refer to any number of systems for performing various ophthalmic surgical procedures. As examples, surgical system 112 may include a console for performing vitreoretinal surgery (e.g., a Constellation console manufactured by Alcon Inc., Switzerland), cataract surgery (e.g., a Centrurion console manufactured by Alcon Inc., Switzerland), and many other systems for performing various ophthalmic surgeries as known to those of ordinary skill in the art. Note that, herein, the term "system" also includes the terms "console" and "device."

[0028] The intraoperative imaging system 114 may include any system that can obtain imaging data or video data and measurements associated with a patient's eye during a surgical procedure. For example, an example of an intraoperative imaging system 114 for cataract surgery is a system with a Verifeye TM Ora TM (Alcon, Switzerland), which is used to provide intraoperative measurements of the eye, including one or more of corneal curvature, axial length, white-to-white diameter of the cornea, etc. Other types of intraoperative systems for generating and providing intraoperative data may include digital microscopes, such as three-dimensional stereo digital microscopes (e.g., Visualization system (Alcon, Switzerland). Various other intraoperative imaging systems may also be used, as known to those of ordinary skill in the art.

[0029] The postoperative imaging system 116 may refer to any number of diagnostic systems that may be used in the clinic postoperatively to obtain multi-dimensional images and / or measurements of ophthalmic anatomy. The postoperative imaging system 116 may be the same as the preoperative imaging system 110 described above.

[0030] The input data 102 includes preoperative data 104, intraoperative data 106, and postoperative data 108. The preoperative data 104 may include information about the patient, including data that may be received from a database 135 (e.g., a database such as an electronic medical record (EMR) database for storing patient history information), and data about the patient's eye generated and provided by the preoperative imaging system 110. For example, the preoperative data 104 may include patient history information, including one or more relevant physiological measurements of the patient that are not directly related to the eye, such as one or more of age, height, weight, body mass index, genetic makeup, ethnicity, race, gender, blood pressure, other demographic and health-related information, etc. In some examples, the patient's medical history may further include one or more relevant risk factors and / or a family history of one or more of these risk factors, the one or more relevant risk factors including a history of smoking, diabetes, heart disease, other underlying conditions prior to surgery, etc.

[0031] The data about the patient's eye generated and provided by the preoperative imaging system 110 may include one or more preoperative measurements and images and any measurements or other types of information extracted from one or more preoperative images. As an example, the preoperative images may include images of one or more optical components of the eye (e.g., retina, vitreous body, lens, cornea, etc.). As known to those of ordinary skill in the art, the preoperative measurements may include the patient's axial length, corneal curvature, anterior chamber depth, white-to-white diameter of the cornea, lens thickness, effective lens position, and measurements related to retinal diseases and other conditions.

[0032] Intraoperative data 106 may include any information obtained or generated during or as a result of a surgical procedure on a patient. For example, intraoperative data 106 may include data input (e.g., by a user) into surgical system 112 and intraoperative imaging system 114 or generated and provided (e.g., automatically) by these systems, which may be present in an operating room during a surgical procedure on a patient. In particular, such intraoperative imaging data may include one or more intraoperative images and / or measurements, including images and / or measurements of the eye obtained while performing the procedure.

[0033] Examples of intraoperative data 106 include surgical videos and images captured by a digital microscope and images captured by a surgical microscope, surgical system data including system parameters, activation settings, and UI / UX / control states set by the surgeon or staff, other data modalities relevant to the surgeon interacting with the system (such as voice commands, gesture-based commands, or commands that can be received by tracking the surgeon's eye gaze), patient monitoring information (such as the patient's eye position obtained by systems other than the surgical microscope), data obtained from sensors embedded in the surgical / imaging system, surgical procedure-specific data associated with the patient's optical components (such as the cornea, cataract, vitreoretinal components, MIGS-related components) (e.g., details related to cataract surgery, including incision location, IOL type, injector type, lighting settings, etc.).

[0034] The postoperative data 108 may include one or more postoperative measurements and images, and any measurements or other information extracted from one or more postoperative images. The postoperative data 108 may also include patient outcome data, including postoperative satisfaction scores. The patient outcome data may also be related to treatment efficacy and / or treatment-related safety endpoints. The postoperative data 108 may be particularly important for algorithm training and continued improvement of the performance of the digital perception system 100.

[0035] Computing system(s) 120 may refer to one or more co-located or disparate systems that execute instruction layers, illustrated as detection layer 121, integration layer 122, annotation layer 123, inference layer 124, activation layer 125. Computing system(s) 120 also executes model trainer 126 and one or more machine learning models 127. In certain embodiments, computing system(s) 120 may be cloud-based (e.g., private or public), or locally deployed ("on-prem"), or a combination thereof.

[0036] In some embodiments, when there are multiple computing systems 120, different instructions (e.g., instruction layers 121 to 125, model trainer 126, and ML model 127) may be executed by different computing systems 120. For example, one of the multiple computing systems 120 may be configured to execute the detection layer 121, while another of the multiple computing systems 120 may execute the ML model 127. In another example, one of the multiple computing systems 120 may be configured to execute the detection layer 121, while another of the multiple computing systems 120 may be configured to execute the integration layer 122. In some embodiments, one or more instruction layers 121 to 125, model trainer 126, and ML model 127 may be executed by multiple computing systems 120 in a distributed and decentralized manner. In some embodiments, one or more of the computing systems 120 may be or include one or more of the imaging systems 110, 114, and 116 and the surgical system 112, respectively, for obtaining ophthalmic information or performing ophthalmic surgery as described above.

[0037] During surgery, instruction layers 121 - 125 and ML model 127 may be executed to acquire input data 102 for a particular patient on whom surgery is being performed and to provide a particular output, such as output 104 .

[0038] For example, the detection layer 121 is configured to ingest the input data 102 or any portion thereof and prepare the input data for further processing. The integration layer 122 integrates the intraoperative data 106 with the preoperative data 104 to generate context-sensitive information for further processing. The annotation layer 123 can be configured to classify and annotate the data generated by the detection layer 121 and the integration layer 122 using one or more of the ML models 127. The reasoning layer 124 can be configured with algorithms designed to extract one or more actionable inferences from the data generated by the detection layer 121, the integration layer 122, and the annotation layer 123. In other words, the data generated by the detection layer 121, the integration layer 122, and the annotation layer 123 are used as input to the reasoning layer 124. The activation layer 125 can be configured with algorithms designed to trigger a set of defined downstream events based on the output from the reasoning layer 124. An example output of the activation layer 125 is shown as output 140.

[0039] The model trainer 126 includes or refers to one or more AI-based learning algorithms (hereinafter referred to as "AI-based algorithms"), which are configured to train the ML model 127 using a training data set stored in a database (e.g., database 130). Examples of AI-based algorithms are optimization algorithms such as gradient descent, stochastic gradient descent, nonlinear conjugate gradient, etc.

[0040] In some embodiments, the trained ML model 127 refers to, for example, a function with weights and parameters that can be used by one or more layers 121 to 125 to make predictions and determinations. Various ML models 127 can be trained for different layers 121 to 125 and used by these layers for different purposes. Example ML models can include different types of neural networks, such as long short-term memory (LSTM) networks, 3D convolutional networks, deep neural networks, or many other types of neural networks, or other machine learning models, etc.

[0041] The database 130 may refer to a database or storage server configured to store the input data 102 associated with each patient and a training data set used by the model trainer 126 to train the ML model 127. The training data set may include population-based data as well as personalized data.

[0042] As shown, outputs 140 are categorized into a variety of different outputs, including image guidance 141, patient monitoring 142, control system parameters 143, virtual assistance 144, service automation 145, etc. As described above, outputs 140 can be triggered by (multiple) computing systems 140 (such as activation layer 125). Any of the above types of outputs 140 can be executed on one or more of the imaging systems 110, 114 and 116 and the surgical system 112. Figure 3 (multiple) activation applications 328) provide or cause to be provided by the one or more software applications.

[0043] Image guidance 141 refers to a set of operations provided for guiding surgical operations. Examples of operations based on image guidance include: identifying toric IOL (intraocular lens) markers (e.g., three laser dots) on an image to improve lens alignment during implantation in cataract surgery, tracking incision locations during cataract surgery to facilitate easy placement of a delivery device for IOL injection, identifying residual lens fragments after phacoemulsification to enable efficient removal of lens material fragments, image guidance during MIGS implantation, etc.

[0044] Patient monitoring 142 refers to a set of operations that are performed (e.g., automatically) to monitor aspects of a surgical procedure. Examples of patient monitoring operations include: detecting the position of surgical instruments relative to various tissues or optical components of the eye during cataract surgery to avoid the risk of capsular bag rupture, detecting the position of surgical instruments relative to various tissues or optical components of the eye during vitreoretinal surgery to avoid the risk of surgical instruments rupturing retinal tissue, assessing the cataract grade of a cataractous lens during cataract surgery to suggest an optimal power setting for performing phacoemulsification, monitoring and mapping the path of a vitrectomy cutter device (vitrectomy cutter) in the eye and suggesting a focal area for residual vitreous removal, monitoring the condition of the patient's eye during surgery to alert the surgeon of any unexpected conditions, etc.

[0045] The control system parameters 143 refer to system parameters determined and output by the activation layer 125 for reconfiguring and / or controlling / changing the operation of one or more of the imaging systems 110 , 114 , and 116 and the surgical system 112 .

[0046] Virtual assistance 144 refers to a set of operations performed to provide virtual assistance to the surgeon, including: automatically monitoring image quality and suggesting adjustments (or automatically adjusting) system settings to improve viewing quality (e.g., suggesting automatic white balance for a 3D visualization system), adjusting lighting settings to improve viewing and detection quality of eye features during surgery, automating the sliding path of an automated robotic arm during a particular surgical procedure, etc.

[0047] Service Automation 145 refers to a set of operations used to automatically provide certain tasks associated with surgical procedures, including: automatically annotating surgical videos for billing and generating teaching / training aids, automatically generating billing codes for billing based on the complexity of the surgical procedure, and automatically processing surgical videos for teaching / training purposes.

[0048] Figure 2 Operations 200 are shown for use by a digital perception system (e.g., digital perception system 100) to provide surgical image guidance, surgical patient monitoring, virtual assistance, and other automated operations that may improve surgical outcomes, reduce the likelihood of physically harming a patient's eye, and increase surgical efficiency and effectiveness, etc., in accordance with certain embodiments. Operations 200 may be performed by one or more of computing system(s) 120, one or more of imaging systems 110, 114, and 116, and surgical system 112, or any combination thereof.

[0049] At operation 210, the digital perception system generates or obtains the pre-operative data 104 and the intra-operative data 106. For example, as described above, one or more imaging systems 110 and 114 may generate the pre-operative data 104 and the intra-operative data 106.

[0050] At operation 220, the digital perception system ingests and prepares the preoperative data 104 and the intraoperative data 106 for further processing. For example, operation 220 can be performed by the detection layer 121. As an example, the detection layer 121 can be configured with one or more machine learning models trained to identify "toric points" on the IOL. In such an example, the detection layer 121 can take as input the raw surgical video feed provided by the intraoperative imaging system 114, such as a digital camera, and output an "annotated surgical video" with the "toric IOL points" identified and marked. This "annotated surgical video" with the identified and marked "toric IOL points" can be used to provide image guidance to the surgeon for toric IOL alignment purposes.

[0051] In another example, the detection layer 121 can be configured with one or more machine learning models as follows, which are trained to identify specific key points (landmarks) in the eye corresponding to the areas where the MIGS device needs to be placed based on the action mode (MoA) of the MIGS device. Identifying these specific key points in the eye is particularly useful in MIGS surgery. For example, before the surgeon performs a MIGS surgery in the operating room, based on the surgeon's indication that a MIGS surgery is to be performed, the detection layer 121 can be configured to (e.g., automatically) perform the following tasks: identify all relevant key points in the eye and generate an "annotated surgical video" to feed as input to the integration layer 122. The additional layers 122 to 125 then process and operate on the annotated surgical video to provide image guidance for the MIGS surgery.

[0052] At operation 230, the digital perception system integrates the preoperative data with the intraoperative data to generate context-sensitive data for further processing. In some embodiments, operation 230 may be performed by the integration layer 122. In some embodiments, the integration layer 122 may integrate the preoperative data with the intraoperative data by associating the preoperative data with the intraoperative data based on corresponding timestamps of the preoperative data and the intraoperative data.

[0053] To continue with the example use case described above regarding providing image guidance for MIGS implantation, the integration layer 122 can combine two or more of the outputs from the detection layer 121, each output identifying (i) ocular key points, (ii) the MIGS device model being implanted, and (iii) the instrument being used for the MIGS implantation, to generate a comprehensive view of the ocular surgical video image that (i) highlights ocular key points appropriate for a given MIGS device model and (ii) overlays an optimal path for MIGS device implantation.

[0054] In another example, the integration layer 122 can queue and automatically load preoperative diagnostic images into an intraoperative surgical video stream, allowing the surgeon to view the preoperative diagnostic images and the video stream side-by-side during different stages of the surgical procedure. The integration layer 121 can queue and load preoperative images according to the surgical stage of the surgery, ensuring that the images are loaded into the correct video stream at the correct stage of the surgery.

[0055] At operation 240, the digital perception system classifies and annotates the preoperative data and intraoperative data (e.g., received from the detection layer 121) and the context-sensitive data (e.g., received from the integration layer 122) using one or more trained machine learning models. In some embodiments, operation 240 is performed by the annotation layer 123. For example, when processing a continuous stream of video data, the annotation layer 123 can use various machine learning models such as neural networks (e.g., ML model 127) to perform feature extraction on the video data and predict the surgical step currently occurring or being performed.

[0056] For example, the annotation layer 123 can perform feature extraction on each video frame using a two-dimensional convolutional neural network (2D-CNN), such as Visual Geometry Group (VGG), Inception, or Visual Transformer (called ViT). The features of each frame are fed to an RNN (Recurrent Neural Network) that processes sequential data to continuously predict surgical step labels (using an RNN, such as a unidirectional LSTM). The surgical step label refers to a label that identifies the surgical step being performed in real time.

[0057] In another example, the annotation layer 123 can perform 3D feature extraction from each video segment including multiple frames by using, for example, a 3D-CNN. The features of each video segment can then be fed to a dense FC (fully connected) network to predict surgical step labels.

[0058] In yet another example, the annotation layer 123 may perform feature extraction from each video frame (as described above), but the features in each frame may be used directly to predict surgical step labels instead of feeding the features of each frame to the RNN. This approach is simpler than feeding the features of each frame to the RNN, but is less resource intensive.

[0059] At operation 250, the digital perception system extracts one or more actionable inferences from the preoperative data, intraoperative data, context-sensitive data, and classified and annotated data. In some embodiments, operation 250 is performed by the inference layer 124. For example, the inference layer 124 can use one or more ML models (e.g., ML model 127) to make determinations or predictions, which can then be used to trigger one or more actions (e.g., actions performed by the activation layer 125) to provide output 140. As examples, these determinations or predictions can include: determining the distance between the instrument tip and a specific key point in the patient's eye, determining image contrast, color, and defocus based on specific image quality metrics, and detecting changes in tasks or surgical steps within an ongoing surgical procedure.

[0060] At operation 260, the digital perception system triggers a set of defined downstream events based on the output of operation 250. In some embodiments, operation 260 is performed by activation layer 125 based on the output from reasoning layer 124. As described above, the output of activation layer 125 can take a variety of forms, examples of which are provided above as Figure 1 Related outputs 140. Additional examples of actions that may be triggered by the activation layer 125 include: flashing a color code on a heads-up display of a 3D visualization system (e.g., the NGENUITY system provided by Alcon of Switzerland) based on the inferred proximity of the surgical instrument to specific key points in the patient's eye; sending a push notification to the surgeon to receive updated device display settings to correct suboptimal image quality metric measurements; pushing a log file to record the surgical procedure with representative snapshots, text descriptions; and automatic billing, etc.

[0061] Figure 3 An example computing system 300 is shown that at least partially implements one or more functions of a digital perception system, such as digital perception system 100. Computing system 300 may be Figure 1 Any of the imaging systems 110 , 114 , 116 , the surgical system 112 , and the computing system 120 .

[0062] As shown, the computing system 300 includes a central processing unit (CPU) 302, one or more I / O device interfaces 304 that can allow various I / O devices 314 (e.g., keyboard, display, mouse device, pen input, etc.) to be connected to the computing system 300, and a network 390 through which the computing system 300 is connected (which can be a local network, an intranet, the Internet, or any other group of computing systems communicatively connected to each other, as described with reference to FIG. 1 ). Figure 1 The network interface 306, memory 308, storage device 310 and interconnect 312 described above.

[0063] Where computing system 300 is an imaging system (e.g., imaging system 110, 114, or 116), computing system 300 may further include one or more optical components for obtaining ophthalmic imaging of a patient's eye and any other components known to one of ordinary skill in the art. Where computing system 300 is a surgical system (e.g., surgical system 112), computing system 300 may further include many other components known to one of ordinary skill in the art for performing the above-described ophthalmic imaging. Figure 1 Ophthalmic surgeries as described and known to those of ordinary skill in the art.

[0064] CPU 302 can retrieve and execute programming instructions stored in memory 308. Similarly, CPU 302 can retrieve and store application data residing in memory 308. Interconnect 312 transmits programming instructions and application data between CPU 302, I / O device interface 304, network interface 306, memory 308, and storage device 310. CPU 302 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc.

[0065] The memory 308 represents a volatile memory such as a random access memory and / or a non-volatile memory such as a non-volatile random access memory, a phase change random access memory, etc. As shown, the memory 608 includes a detection layer 321, an integration layer 322, an annotation layer 323, an inference layer 324, an activation layer 325, a model trainer 326, an ML model 327, and (multiple) activation applications 328. The detection layer 321, the integration layer 322, the annotation layer 323, the inference layer 324, the activation layer 325, the model trainer 326, the ML model 327 are similar or identical to the detection layer 121, the integration layer 122, the annotation layer 123, the inference layer 124, the activation layer 125, the model trainer 126, and the ML model 127. Note that all instructions, modules, layers, and applications in memory 208 are shown in dashed boxes to indicate that they are optional, because one or more of these instructions, modules, layers, and applications may be executed by computing system 300, while others may not be executed by the computing system, depending on the functionality of computing system 300. For example, where computing system 300 is an imaging system (e.g., one of imaging systems 110, 114, or 116) or a surgical system (e.g., surgical system 112), in some embodiments, memory 308 may store activation application 328 (to trigger one or more actions based on output 140) but not model trainer 326. Where computing system 300 is a server system (e.g., not an imaging system or a surgical system) configured to train ML model 327, in some embodiments, memory 308 may store model trainer 326 but not activation application 328.

[0066] Storage device 310 may be a non-volatile memory such as a disk drive, a solid-state drive, or a collection of storage devices distributed across multiple storage systems. Storage device 310 may optionally store input data 330 (e.g., similar or identical to input data 102) and training data set 332. As described above, training data set 330 may be used by model trainer 326 to train ML model 327. Training data set 330 may also be stored in an external storage device such as a database (e.g., database 130).

[0067] Additional considerations

[0068] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments are clear to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, without departing from the scope of the present disclosure, the function and arrangement of the elements discussed may be changed. Various examples may appropriately omit, replace or add various programs or components. In addition, the features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover devices or methods practiced using other structures, functions, or structures and functions of various aspects of the present disclosure set forth herein in addition to or different from those set forth herein. It should be understood that any aspect of the present disclosure disclosed herein may be embodied by one or more elements of the claims.

[0069] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. For example, "at least one of a, b, or c" is intended to cover any combination of a, b, c, ab, ac, bc, and abc, as well as multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc or any other order of a, b, and c).

[0070] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, etc. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Also, "determining" may include resolving, selecting, choosing, establishing, etc.

[0071] The method disclosed herein includes one or more steps or actions for implementing the method. Without departing from the scope of the claims, the method steps and / or actions can be interchangeable with each other. In other words, unless a specific step or action sequence is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. Further, the various operations of the above-mentioned method can be performed by any suitable device that can perform the corresponding function. These devices may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Generally, in the case of operations shown in the figure, those operations can have corresponding corresponding devices with similar numbers plus functional components.

[0072] The various illustrative logical blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general purpose processor may be a microprocessor, but in an alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0073] The processing system can be implemented with a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnecting buses and bridges. The bus may link together various circuits including a processor, a machine-readable medium, and an input / output device. A user interface (e.g., a keyboard, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and therefore will not be further described. The processor may be implemented with one or more general and / or special processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuit systems that can execute software. Those skilled in the art will recognize how to best implement the described functions of the processing system according to the specific application and the overall design constraints imposed on the entire system.

[0074] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Software should be broadly interpreted as instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or other. Computer-readable media include both computer storage media and communication media (such as any medium that facilitates the transfer of computer programs from one place to another). The processor may be responsible for managing the bus and general processing, including the execution of software modules stored on a computer-readable storage medium. A computer-readable storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated with the processor. For example, a computer-readable medium may include a transmission line, a carrier modulated by data, and / or a computer-readable storage medium on which instructions separated from a wireless node are stored, all of which may be accessed by a processor via a bus interface. Alternatively or in addition, a computer-readable medium or any portion thereof may be integrated into a processor, such as a cache and / or a general register file. For example, examples of machine-readable storage media may include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage media, or any combination thereof. Machine-readable media may be embodied in a computer program product.

[0075] A software module may include a single instruction or multiple instructions, and may be distributed over several different code segments, between different programs, and across multiple storage media. A computer-readable medium may include multiple software modules. A software module includes instructions that, when executed by a device such as a processor, cause a processing system to perform various functions. A software module may include a transmission module and a reception module. Each software module may exist in a single storage device, or may be distributed across multiple storage devices. For example, when a triggering event occurs, a software module may be loaded from a hard drive into a RAM. During the execution of a software module, a processor may load some instructions into a cache to increase access speed. Then, one or more cache lines may be loaded into a general register file for execution by the processor. When referring to the function of a software module, it should be understood that such function is implemented by the processor when executing instructions from the software module.

[0076] The following claims are not intended to be limited to the embodiments shown herein, but are given the full scope consistent with the language of the claims. In the claims, unless otherwise specified, references to singular elements are not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. According to the provisions of 35 U.S.C. § 112 (f), the elements of any claim will not be interpreted unless the phrase "device for..." is used to expressly describe these elements, or in the case of method claims, the phrase "step for..." is used to describe these elements. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, whether or not such disclosure is explicitly described in the claims, the content disclosed herein is not intended to be committed to the public.

Claims

1. An ophthalmic imaging or surgical system comprising: a memory including executable instructions; as well as a processor in data communication with the memory and configured to execute the instructions to cause the ophthalmic imaging or surgical system to: ingesting and preparing pre-operative and intra-operative data associated with the patient's eye for further processing; integrating the pre-operative data and the intra-operative data to generate context-sensitive data for further processing; classifying and annotating the pre-operative data, the intra-operative data, and the context-sensitive data; extracting one or more actionable inferences from the pre-operative data, the intra-operative data, the context-sensitive data, and the classified and annotated data; as well as One or more actions are performed at the ophthalmic imaging system or surgical system based on the one or more actionable inferences.

2. The ophthalmic imaging or surgical system of claim 8, wherein: The processor is further configured to cause the ophthalmic imaging or surgical system to generate at least a portion of the intra-operative data.

3. The ophthalmic imaging or surgical system of claim 8, wherein: Integrating the pre-operative data and the intra-operative data is based on a timestamp associated with the pre-operative data and a timestamp associated with the intra-operative data.

4. The ophthalmic imaging or surgical system of claim 8, wherein: The classification and annotation further includes performing feature extraction on the pre-operative data, the intra-operative data, and the context-sensitive data using one or more trained machine learning models.

5. The ophthalmic imaging or surgical system of claim 8, wherein: The one or more actionable inferences include: determining the distance between the instrument tip and a specific critical point in the patient's eye, Determine image contrast, color, and defocus based on specific image quality metrics, or Detecting changes in tasks or surgical steps within an ongoing surgical procedure.

6. The ophthalmic imaging or surgical system of claim 8, wherein: The one or more actions include: Provide image guidance; Provide patient monitoring; or Offer virtual help.

7. The ophthalmic imaging or surgical system of claim 13, wherein: Providing image guidance includes flashing a code on a heads-up display of a 3D visualization system based on an inferred proximity of a surgical instrument to a particular key point in the patient's eye.

Images