Technology for automatically tracking surgical procedures

Abstract

A particular embodiment provides a method for performing an ophthalmic surgical procedure. The method includes collecting and preparing preoperative and intraoperative data related to a patient's eye for further processing. In a particular embodiment, the method further includes integrating the preoperative and intraoperative data to generate context-dependent data for further processing. The method also includes classifying and annotating the preoperative data, intraoperative data, and context-dependent data. The method also includes extracting one or more actionable inferences from the preoperative data, intraoperative data, context-dependent data, and the classified and annotated data. The method further includes triggering one or more actions on an imaging system or surgical system based on one or more actionable inferences.

Description

Technical Field

[0001] The present invention relates to a technique for automatically tracking surgical procedures.

Background Art

[0002] Various diseases or symptoms related to the eye can be treated by ophthalmic surgical procedures. Examples of ophthalmic surgical procedures include vitreoretinal surgery, cataract surgery, glaucoma surgery, laser eye surgery (LASIK), and the like.

[0003] Vitreoretinal surgery is a type of ophthalmic surgery that treats problems of the retina or vitreous. Vitreoretinal surgery can be performed to treat symptoms such as diabetic traction retinal detachment, diabetic vitreous hemorrhage, macular hole, retinal detachment, epiretinal membrane, and many other ophthalmic symptoms. Cataract surgery involves emulsifying the patient's lens with an ultrasonic handpiece and aspirating it from the eye. Then, an intraocular lens (IOL) is implanted into the lens capsule of the eye. During the above-mentioned vitreoretinal, cataract, and other types of surgeries known to those skilled in the art, various problems can, as a result of the surgery, have an adverse impact on the efficiency, effectiveness, and ease of performing the surgery by the surgeon, and in more specific cases, can cause harm to the patient's optical anatomical structure and the like.

Summary of the Invention

Means for Solving the Problems

[0004] One aspect of the present disclosure provides a method for automatically tracking surgical procedures performed on an anatomical object of a patient. The method includes: acquiring imaging data of an anatomical object of a patient captured during a period; automatically generating a unique identifier corresponding to at least one surgical procedure performed on the anatomical object during the period based on the imaging data; automatically providing a visual representation on a user interface associated with at least one surgical procedure and the generated unique identifier; automatically transmitting the unique identifier to a second device for automated processing upon receiving confirmation from a user to verify the unique identifier; and automatically overwriting the unique identifier and transmitting a second unique identifier provided by the user via the user interface to the second device for automated processing upon receiving user input that is inconsistent with or rejected by the unique identifier.

[0005] Other embodiments provide an apparatus that is operable, configured, or otherwise adapted to perform the methods described above and the methods described elsewhere in this specification; a non-temporary computer-readable medium containing instructions that, when executed by the processor of the apparatus, cause the apparatus to perform the methods described above and the methods described elsewhere in this specification; a computer program product embodied on a computer-readable storage medium containing code for performing the methods described above and the methods described elsewhere in this specification; and an apparatus that includes means for performing the methods described above and the methods described elsewhere in this specification. For example, the apparatus may include a processing system, a device comprising a processing system, or processing systems cooperating on one or more networks.

[0006] The following description and attached drawings illustrate specific features for illustrative purposes only.

[0007] The accompanying drawings illustrate only examples of specific embodiments of the present disclosure and should therefore not be considered to limit the scope of the present disclosure. [Brief explanation of the drawing]

[0008] [Figure 1]Examples of digital recognition systems (hereinafter referred to as "digital recognition systems") comprising digital recognition technology, according to several embodiments, are shown below. [Figure 2] This describes exemplary operation for use by ophthalmic surgical systems, such as digital recognition systems, according to a specific embodiment. [Figure 3A-3B] This shows multiple image frames of imaging data representing various surgical procedures performed on a patient's eye according to a specific embodiment. [Figure 3C-3D] This shows multiple image frames of imaging data representing various surgical procedures performed on a patient's eye according to a specific embodiment. [Figure 4] An exemplary computing device according to a specific embodiment is shown. [Modes for carrying out the invention]

[0009] For the purpose of facilitating understanding, the same reference numerals are used wherever possible to indicate identical elements common to the drawings. Elements and features of a particular embodiment are intended to be advantageously incorporated into other embodiments without further mention.

[0010] Features of the present invention may be discussed below with reference to specific embodiments and drawings, but all embodiments of the present invention may include one or more of the advantageous features discussed herein. In other words, one or more embodiments may be discussed as having certain advantageous features, and one or more such features may be used in accordance with various other embodiments discussed herein. Similarly, exemplary embodiments may be discussed below as apparatus, apparatus, or methods, and it should be understood that such exemplary embodiments can be implemented in various apparatus, apparatus, and methods.

[0011] Digital Recognition Surgical System Certain embodiments of this specification describe, among other things, a digital recognition surgical system configured to automatically track one or more surgical procedures performed on a patient, including identifying one or more surgical procedures, capturing imaging data (e.g., photographic and / or video data) of one or more surgical procedures, generating one or more unique identifiers associated with one or more surgical procedures, transmitting information including photographic evidence and one or more unique identifiers to an EMR database, and / or other operations as described in more detail below.

[0012] The digital recognition surgical systems described herein have at least four key technical capabilities, including (1) the ability to analyze data in real time; (2) the ability to process multi-model data (i.e., data generated and / or received simultaneously in different formats such as surgical video, numerical data, audio data, text, images, and signals); (3) the ability to process data received simultaneously from a single source or multiple sources (e.g., images captured by a camera, internal sensor data, and audio recordings from a microphone); and (4) the ability to perform inferences using the received and processed data with respect to the state or stage of a surgical procedure, the instrumentation of surgical instruments, the state of the patient or the patient's eyes, and the control of surgical instruments.

[0013] Digital recognition technologies as described herein can provide smart functions for surgical systems. Smart functions for ophthalmic surgical systems can take multiple forms in the operating room (OR), such as image-guided operation, patient monitoring, virtual assistants for surgeons, and / or service automation. Incorporating the smart functions described in the embodiments herein results in many improvements over existing surgical systems. The improved surgical systems described herein can help surgeons perform surgical tasks with greater precision, efficiency, and / or safety, ultimately leading to better surgical outcomes for each patient.

[0014] Figure 1 shows examples of a digital recognition system 100 (hereinafter referred to as the "digital recognition system") comprising digital recognition technology according to several embodiments. As shown, the digital recognition system 100 includes various systems such as one or more pre-operative (hereinafter referred to as "pre-operative") imaging systems 110, one or more surgical systems 112, one or more intra-operative (hereinafter referred to as "intra-operative") imaging systems 114, and one or more post-operative (hereinafter referred to as "post-operative") imaging systems 116.

[0015] The preoperative imaging system 110, the surgical system 112, the intraoperative imaging system 114, and the postoperative imaging system 116 may be located in the same location, including a diagnostic clinic, a surgical clinic, a hospital, and other locations, or they may be located in various locations. Regardless of whether they are located in the same location or various locations, systems 110, 112, 114, and 116 may each generate data that can be communicated and used as part of input data 102 on one or more networks (e.g., a local area network, a wide area network, and / or the Internet) to other systems 110, 112, 114, and 116, computing devices 120, and / or databases 130, 135.

[0016] The preoperative imaging system 110 may refer to any number of diagnostic systems, such as optical coherence tomography (OCT) systems, rotating cameras (e.g., Scheinproof cameras), magnetic resonance imaging (MRI) systems, keratometers, corneal curvature meters, optical biometers, topographers, retinal cameras, digital microscopes, and / or any other types of optical measurement / imaging systems, which may be used before surgery, for example in a hospital, to obtain multidimensional images and / or measurements of the anatomical structure of the eye. Examples of OCT systems are described in detail in U.S. Patent No. 9,618,322 disclosing “Process for Optical Coherence Tomography and Apparatus for Optical Coherence Tomography” and U.S. Patent Application Publication No. 2018 / 0104100 disclosing “Optical Coherence Tomography Cross View Image,” both of which are incorporated herein by reference in their entirety.

[0017] The surgical system 112 may refer to any number of systems for performing various ophthalmic surgical procedures. For example, the surgical system 112 may include a console for performing vitreoretinal surgery (e.g., the Constellation console manufactured by Alcon, Switzerland), a console for performing cataract surgery (e.g., the Centurion console manufactured by Alcon, Switzerland), and many other systems used for performing various ophthalmic surgeries as known to those skilled in the art. Note that in this specification, the term “system” also includes the terms “console” and “device.”

[0018] The intraoperative imaging system 114 may include any system capable of obtaining imaging data (e.g., video data, image frames, metadata, etc.) and measurements related to the patient's eye during the surgical procedure. An example of an intraoperative imaging system 114 used for cataract surgery is the Ora® with Verifeye® (Alcon, Switzerland), which is used to provide intraoperative measurements of the eye, including one or more of the following: corneal curvature, axial length of the eye, and corneal white-to-white diameter. Other types of intraoperative systems used to generate and provide intraoperative data may include a digital microscope containing one or more digital cameras, such as a three-dimensional stereoscopic digital microscope (e.g., the NGENUITY® 3D Visualization System (Alcon, Switzerland)). Various other intraoperative imaging systems, as known to those skilled in the art, may also be used.

[0019] The postoperative imaging system 116 may refer to any number of diagnostic systems that can be used in the hospital after surgery to obtain multidimensional images and / or measurements of the anatomical structure of the eye. The postoperative imaging system 116 may be the same as the preoperative imaging system 110 described above.

[0020] The input data 102 includes preoperative data 104, intraoperative data 106, and postoperative data 108. The preoperative data 104 may include information about the patient, including data that can be received from a database 135 (for example, a database such as an electronic medical record (EMR) database for storing patient treatment history information) and data generated and provided by the preoperative imaging system 110 regarding the patient's eyes. For example, the preoperative data 104 may include patient treatment history information, including one or more relevant physiological measurements of the patient that are not directly related to the eyes, such as age, height, weight, body mass index, genetic predisposition, race, ethnicity, sex, blood pressure, other demographic and health-related information, and / or similar. In some examples, the patient's treatment history may further include one or more relevant risk factors, and / or a family history of one or more of these risk factors, including smoking history, diabetes, heart disease, other underlying conditions, past surgeries, and others. In some embodiments, patient treatment history information may include information about the patient's medication history, such as a list of medications the patient is currently taking, a list of medications the patient has taken in the past, a list of medications the patient is allergic to, and so on.

[0021] The data generated and provided by the preoperative imaging system 110 relating to the patient's eye may include one or more preoperative measurements and images, as well as any measurements or other types of information extracted from one or more preoperative images. For example, the preoperative images may include images of one or more optical components of the eye (e.g., retina, vitreous humor, lens, cornea, etc.). The preoperative measurements may include the axial length of the patient's eye, corneal curvature, anterior chamber depth, corneal diameter from white to white, lens thickness, effective lens position, and measurements related to retinal diseases and other conditions as known to those skilled in the art.

[0022] Intraoperative data 106 may include any information obtained or generated during or as a result of a surgical procedure on a patient. For example, intraoperative data 106 may include data (e.g., automatically) generated and provided by a surgical system 112 and an intraoperative imaging system 114 that are (e.g., entered by a user) input or may be present in an operating room during a patient's surgical procedure. In particular, such intraoperative imaging data may include one or more intraoperative images and / or measurements, including images of the eye and / or measurements obtained while the procedure is being performed.

[0023] Examples of intraoperative data 106 include imaging data such as surgical videos (e.g., video data) and images (e.g., image frames) captured by a digital microscope / digital camera. In some embodiments, the imaging data may include video / image data of one or more surgical procedures performed on a patient's eye. Intraoperative data 106 may also include surgical system data, including images captured by a surgical microscope, system parameters, active settings, and UI / UX / control states set by a surgeon or staff. Intraoperative data 106 may also include other data modalities related to a surgeon interacting with the system, such as voice commands, gesture-based commands, or commands that can be received by tracking the surgeon's line of sight, and patient monitoring information (e.g., the position of the patient's eye obtained by a system other than the surgical microscope). In some cases, intraoperative data 106 may also include specific machine settings of the surgical system 112. For example, in the case of a cataract surgery, the machine settings may include fluid settings, intraocular pressure settings, phaco energy settings, any state of any device connected to the surgical system 112, laser output settings, laser intensity settings, total laser energy application settings, ultrasonic frequency, ultrasonic power level, various other settings or states of the console used for cataract surgery, and the like. In another example, in the case of a vitreoretinal procedure, machine settings related to the operation of a vitrectomy probe, aspiration / perfusion operations, etc. can be recorded and used as part of the intraoperative data 106.

[0024] In some embodiments, the voice commands may include commands by the surgeon, such as requests for specific medical instruments or drugs to the intraoperative imaging system 114, the surgical system 112 (e.g., the console), any other system in the operating room, the nurse, or the medical staff. In some embodiments, the intraoperative data 106 may include data obtained from sensors embedded in the intraoperative imaging system 114 and / or the surgical system 112, for example, indicating the status or use of a specific medical instrument or system (e.g., the Alcon Centurion Phaco system or the Alcon Verion Image Guidance system) during a surgical procedure. For example, a surgical console used for cataract surgery can be configured to include sensors for indicating whether a phacoemulsification probe is being used by the surgeon. In some embodiments, the intraoperative data 106 includes inventory records associated with the patient's surgical procedure, indicating records of medical instruments or medications prepared for use during the surgical procedure. The inventory records can be complied as a result of data generated by the intraoperative imaging system 114 and / or the surgical system 112 throughout the surgery. In some embodiments, the intraoperative data 106 includes surgical procedure specific data related to the patient's optical components, such as the cornea, cataract, vitreoretinal components, MIGS-related components (e.g., details related to cataract procedures including incision location, IOL type, injector type, illumination settings, etc.). Additional details regarding the input data 102, such as the intraoperative data 106, are described below with respect to FIG. 2.

[0025] The postoperative data 108 may include one or more postoperative measurements and images and any measurements or other information extracted from the one or more postoperative images. The postoperative data 108 may also include patient outcome data including postoperative satisfaction scores. The patient outcome data may also relate to the effectiveness of the treatment and / or safety endpoints related to the treatment. The postoperative data 108 may be particularly important for algorithm training and for continuously improving the performance of the digital recognition system 100.

[0026] Computing device 120 refers to one or more systems located in the same location or not, which execute layers of instructions, indicated as a detection layer 121, an integration layer 122, an annotation layer 123, an inference layer 124, and an initiation layer 125. Computing device 120 also executes a model trainer 126 and one or more machine learning (ML) / artificial intelligence (AI) models 127. In certain embodiments, computing device 120 may be cloud-based (e.g., private or public cloud), on-premises ("on-premise"), or a combination thereof.

[0027] In certain embodiments, when there are multiple computing devices 120, different computing devices 120 can execute different instructions (e.g., instruction layers 121-125, model trainer 126, and ML / AI model 127). For example, one of the multiple computing devices 120 may be configured to execute detection layer 121, and another of the multiple computing devices 120 may execute ML / AI model 127. In another example, one of the multiple computing devices 120 may be configured to execute detection layer 121, and another of the multiple computing devices 120 may be configured to execute integration layer 122. In certain embodiments, one or more instruction layers 121-125, model trainer 126, and ML / AI model 127 may be executed by the multiple computing devices 120 in a distributed and decentralized manner. In certain embodiments, one or more computing devices 120 may be or include one or more of the imaging systems 110, 114, 116 and surgical systems 112, which are used to acquire eye information or perform ophthalmic surgical procedures, as described above.

[0028] During surgery, the command layers 121-125 and the ML / AI model 127 may take input data 102 from a specific patient undergoing surgery and execute to provide specific outputs such as output 140.

[0029] For example, the detection layer 121 is configured to collect input data 102 or any portion thereof and prepare the input data for further processing. For example, the detection layer 121 receives various input data 102 from various sources (e.g., preoperative imaging system 110, surgical system 112, intraoperative imaging system 114, postoperative imaging system 116, database 135, etc.) and prepares various input data, which may be in different formats, for collection by the integration layer 122. As an example, the format in which input data is received from the surgical console may differ from the format in which imaging data is received from the intraoperative imaging system. Therefore, the detection layer 121 can change the format of various input data so that the integration layer 122 can collect all types of input data. For example, in some embodiments, preoperative data 104 may contain only image data, while intraoperative data 106 may have both intraoperative image data and text entries. The detection layer 121 can be configured to integrate data formats by, for example, tagging text entries with intraoperative image data and sending the tagged intraoperative image data to the integration layer 122. Another example of input data 102 is a UI menu selection provided (for example, by the user) on the UI of the surgical system 112. The UI menu selection can be converted into a procedure identifier used by the inference layer 124 below.

[0030] The integration layer 122 integrates various input data provided by the detection layer 121 (e.g., intraoperative data 106, preoperative data 104, etc.) to generate context-dependent information for further processing. In certain embodiments, the integration layer 122 can take various forms of intraoperative data 106 and integrate them together. For example, the integration layer 122 can integrate imaging data provided by an intraoperative digital microscope with other input data generated by a surgical console during surgery. In certain embodiments, integrating various input data together can be done by associating different streams of input data with associated timestamps. For example, timestamped data generated by a sensor embedded in the surgical console indicating the use of a phacoprobe may be integrated with timestamped video or image data generated by a digital microscope. The timestamps may be provided as part of metadata associated with various streams of input data.

[0031] In certain embodiments, the integration layer 122 may integrate preoperative data 104 with intraoperative data 106. For example, the integration layer 122 may integrate a patient's preoperative image data with a patient's intraoperative image data based on metadata that may indicate the patient's identity. The integration layer 122 may convert the preoperative image data to a scale that matches the intraoperative view so that the preoperative image data can be seamlessly overlaid on the intraoperative view.

[0032] The annotation layer 123 can be configured to use one or more ML / AI models 127 to classify and annotate the data generated by the detection layer 121 and / or the integration layer 122. For example, in some embodiments, the annotation layer 123 can be configured to acquire imaging data from the detection layer 121 of one or more surgical procedures performed on a patient's eye. The annotation layer 123 can then be configured (e.g., trained) to automatically label one or more objects (e.g., surgical instruments, consoles, optical components of the eye, drugs, etc.) in each image frame. The annotation layer 123 can be trained to label objects in image frames using, for example, an annotation training dataset having manually labeled images. In some embodiments, the annotation layer 123 processes a continuous flow of imaging data, taking into account not only current or instantaneous imaging data but also previous imaging data.

[0033] The inference layer 124 may consist of an algorithm designed to extract one or more feasible inferences from the data generated by the detection layer 121, the integration layer 122, and / or the annotation layer 123. In other words, the data generated by the detection layer 121, the integration layer 122, and / or the annotation layer 123 is used as input to the inference layer 124. For example, in some embodiments, the inference layer 124 may be configured (e.g., trained) to identify surgical procedures based on objects identified in the imaging data by the annotation layer 123 and any other inputs described above (e.g., by generating unique identifiers for each, such as a claim code). In some embodiments, surgical procedures may be identified in many ways. For example, in some embodiments, the inference layer 124 may be configured to generate a unique identifier associated with the surgical procedure, such as a claim code. In some embodiments, the inference layer 124 may be configured to generate a complexity level associated with the surgical procedure, or a flag indicating that the surgical procedure is a complex case. In some embodiments, as described herein, the output of the inference layer 124 (e.g., billing code, complexity level, flags, etc.) may be generated to help automatically track surgical procedures performed on a patient (e.g., to reduce the burden of record-keeping on surgeons or staff) and to reduce the likelihood of surgical procedures for a patient being inappropriately billed.

[0034] In some embodiments, some or all of the functions of the annotation layer 123 and the inference layer 124 can be combined. For example, in some embodiments, rather than using a first set of one or more ML / AI models to identify one or more objects in each image frame, and then using a second set of one or more ML / AI models to associate each unique identifier with one or more objects, one or more ML / AI models 127 may be trained to take the outputs of the detection layer 121 and / or integration layer 122 and directly generate unique identifiers associated with one or more surgical procedures performed in the imaging data. As an example, in such embodiments, the ML / AI model 127 (e.g., a deep learning model) may be trained using a training dataset containing multiple training data entries or records, each containing multiple data points from the input data 102, and labels identifying the surgical procedures (e.g., each unique identifier such as a billing code).

[0035] The activation layer 125 can consist of an algorithm designed to trigger a defined set of downstream events based on the output from the inference layer 124. An exemplary output from the activation layer 125 is shown as output 140, which is described in more detail below.

[0036] The model trainer 126 includes or refers to one or more AI-based learning algorithms (hereinafter referred to as "AI-based algorithms") configured to train the ML / AI model 127 using training datasets stored in a database (e.g., database 130). Examples of AI-based algorithms are optimization algorithms such as gradient descent, stochastic gradient descent, and nonlinear conjugate gradient.

[0037] In certain embodiments, the trained ML / AI model 127 refers to a function having, for example, weights and parameters, which can be used by one or more layers 121-125 to make predictions and decisions based on the input data 102. Various ML / AI models 127 may be trained for different instruction layers 121-125 and used by different instruction layers 121-125 for different purposes. Exemplary ML models may include different types of neural networks such as long-short-term memory (LSTM) networks, 3D convolutional networks, deep neural networks, or many other types of neural networks or other machine learning or AI models. Further details regarding the instruction layers 121-125, the model trainer 126, and the ML / AI model 127 are described below with reference to Figure 2.

[0038] The database 130 may refer to a database or storage server configured to store input data 102 associated with each patient, and training datasets used by the model trainer 126 to train the ML / AI model 127. The training datasets may include population-based data and personalized data.

[0039] As shown, output 140 is categorized into many different outputs, including a procedure identifier 141, a representative image / video 142, timestamp information 143, and visual aids 144. As described above, output 140 can be triggered by computing devices 120 such as an annotation layer 123, an inference layer 124, and an activation layer 125. Any of the above types of output 140 can be provided or made to be provided by one or more software applications running on one or more of the imaging systems 110, 114, 116, the surgical system 112, or the visualization system (e.g., NGENUITY® 3D visualization system).

[0040] The procedure identifier 141 refers to a unique identifier generated at least partially based on intraoperative data 106 (e.g., imaging data) corresponding to at least one surgical procedure performed on the patient's eye. In some embodiments, the unique identifier may include, for example, a billing code corresponding to the surgical procedure.

[0041] A representative image / video 142 refers to a representative image frame or representative video clip from intraoperative data 106 that demonstrates at least one surgical procedure performed on the eye or at least a particular segment thereof. In some embodiments, the representative image / video 142 may be used to document a claim event associated with at least one surgical procedure performed on the eye.

[0042] The timestamp information 143 refers to the timestamp corresponding to a representative image frame or representative video clip contained in the representative image / video 142. For example, the timestamp information 143 indicates the time when the representative image frame or representative video clip was generated. The timestamp information 143 may also indicate the time when the corresponding surgical procedure or segment thereof was performed.

[0043] Visual assistance 144 refers to a visual representation associated with at least one surgical procedure and a unique identifier corresponding to at least one surgical procedure, which may be provided on the user interface to provide visual assistance to a user (e.g., a surgeon, nurse, or medical staff). In some embodiments, the user interface may allow the user to confirm or reject the unique identifier corresponding to at least one surgical procedure. In some embodiments, the user interface may allow the user to select a second unique identifier corresponding to at least one surgical procedure. In some embodiments, the user interface may allow the user to confirm or reject a representative image frame or representative video clip contained in a representative image / video 142. In some embodiments, if a representative image frame or representative video clip is rejected by the user, the user interface may allow the user to select another representative image frame or another representative video clip. Additional details regarding output 140 are described below with reference to Figure 2.

[0044] Aspects related to automated tracking of surgical procedures Modern ophthalmic surgeries, such as cataract surgery, have few associated complications. Although relatively rare, these complications may require the surgeon to perform one or more additional surgical procedures, which can be time-consuming, complex, and expensive. For example, during cataract surgery and placement of an IOL into the patient's eye, the surgeon may discover loosening of the patient's lens capsule, which can affect the long-term stability of the IOL. Accordingly, the surgeon may perform an additional surgical procedure to insert a capsule dilation ring (CTR) to stabilize the lens capsule. Another example of a complication that can occur during cataract surgery is associated with intraoperative hypotonia syndrome, in which the patient's iris does not chemically dilate, often associated with medications used to treat benign prostatic hyperplasia. In such cases, the surgeon may perform another surgical procedure to manually open the patient's iris that does not chemically dilate by inserting an iris hook or maluzin ring.

[0045] These complications and additional surgical procedures represent deviations from the expected surgical protocol, require considerable time to perform, and increase the cost of the procedure. Due to these complications and additional surgical procedures, the insurance company may provide additional reimbursement. To receive additional reimbursement, you must submit the claim code associated with the additional surgical procedure to your insurance company. In the case of cataract surgery, if no complications occur, the claim code used for compensation may be labeled as "Standard Cataract." However, if complications occur and additional reimbursement is required, the claim code for this additional reimbursement may be labeled as "Complicated Cataract."

[0046] To claim this additional reimbursement for a "standard cataract" surgery, the surgeon (or their staff) may need to perform additional record-keeping to provide the insurance company with the necessary documentation to obtain compensation for one or more additional surgical procedures performed during a "complex cataract" surgery. While the above example concerns cataract surgery, it should be understood that any deviation from a standard surgical protocol can lead to additional record-keeping for obtaining additional compensation for any type of surgery. However, these additional record-keeping steps are often time-consuming, cumbersome, and prone to bureaucratic errors.

[0047] For example, in many cases, additional record keeping must involve tracking all additional surgical procedures performed on a patient, capturing photographic evidence of all additional surgical procedures, and tracking timestamps related to when all additional surgical procedures were performed and when the photographic evidence was taken. In some embodiments, because all of this additional record keeping must be performed, the surgeon may have to continuously interrupt surgeries to ensure that all necessary information (e.g., a list of surgical procedures performed, photographic evidence, timestamps, etc.) is collected, which can negatively impact surgical efficiency. In some embodiments, if the surgeon waits until the completion of the surgery, certain information (e.g., claim code) may be forgotten and not properly documented, which could lead to the denial of claims for additional reimbursement from insurance companies. In addition, in some embodiments, this additional record keeping may lead to patients being charged additional fees to compensate for the extra time required to perform the additional record keeping.

[0048] However, as discussed, existing systems are not configured to automatically track surgical procedures. In particular, existing systems are unable to receive input data from various input sources, integrate them to make the input data suitable for annotation and inference, annotate the input data, and / or output unique identifiers associated with surgical procedures or segments thereof.

[0049] Accordingly, aspects of the present disclosure provide techniques for automatically tracking surgical procedures performed on patients. For example, in some embodiments, such techniques may include the use of a digital recognition system (e.g., digital recognition system 100) for automatically generating a unique identifier corresponding to at least one surgical procedure performed on a patient's eye, based at least in part on imaging data. In some embodiments, these techniques may include training one or more artificial intelligence and / or ML models to take input data (e.g., input data 102 including image data such as image frames and video data) and output a unique identifier associated with a surgical procedure. Various techniques may be used to train such ML models. For example, in some embodiments, one or more ML / AI models may be trained to take input data related to a patient's surgery based on historical patient population data and output a unique identifier (e.g., billing code) for at least one surgical procedure performed as part of the patient's surgery. In certain embodiments, historical patient population data refers to or may include a training dataset comprising multiple training data entries or records, each of which includes various types of input data related to the patient's past surgery (e.g., video and / or image data related to various surgical procedures), as well as labels related to the surgical procedures performed on the patient. For example, the labels may represent a unique identifier corresponding to the surgical procedure (e.g., a billing code).

[0050] In addition, in some embodiments, video evidence (e.g., image frames or video clips) demonstrating the performance of at least one surgical procedure can be selected and included in the patient-associated medical record along with timestamp information and a generated unique identifier. The medical record can then be automatically transmitted to an EMR database associated with the patient's insurance company.

[0051] Therefore, these technologies provide technical solutions to technical problems associated with existing systems, such as the inability of existing systems to automatically track surgical procedures using input data from various input sources (e.g., video data). As an additional benefit, these technologies can also reduce the amount of time and effort associated with performing additional record-keeping to document surgical procedures performed on patients. By reducing the amount of time and effort associated with performing additional record-keeping, surgeons can focus their attention on performing the surgery rather than on record-keeping, thereby increasing the efficiency of the surgery and reducing the potential for patient injury. Furthermore, reducing the amount of time and effort associated with performing additional record-keeping can lead to a reduction in the fees paid to patients. In addition, these technologies can reduce the occurrence of bureaucratic errors when performing additional record-keeping, thereby reducing the likelihood of insurance companies denying requests for additional reimbursement.

[0052] Figure 2 shows an operation 200 used by an ophthalmic surgical system, such as a digital recognition system (e.g., digital recognition system 100), to automatically track surgical procedures performed on a patient's eye or any other anatomical object. Operation 200 may be performed by at least one processor of one or more computing devices 120, one or more imaging systems 110, 114, 116, and surgical system 112, or any combination thereof.

[0053] As shown, operation 200 begins in step 210 by the digital recognition system acquiring input data (e.g., input data 102) related to a surgical procedure performed on the patient's eye. As discussed, input data 102 includes data points acquired from various systems and databases (e.g., systems 110-116, database 135, etc.). As an example, input data 102 may include, for example, imaging data captured during the surgical procedure by the intraoperative imaging system 114, surgical instrument and / or usage information for at least one surgical procedure, voice command information associated with at least one surgical procedure (e.g., requests for specific surgical instruments and / or drugs), surgical instrument and / or drug preparation information for at least one surgical procedure, overall flow or progress information associated with at least one surgical procedure, and the target surgical site for at least one surgical procedure (e.g., retina vs. lens). In certain embodiments, input data 102 acquired and processed in the detection layer 121 may be further processed by the integration layer 122. The operation of integration layer 122 has been described above, and for the sake of brevity, it is omitted here.

[0054] Subsequently, in step 220, the digital recognition system automatically generates a unique identifier corresponding to the surgical procedure performed on the eye. In some embodiments, the unique identifier can be generated by the annotation layer 123 and / or inference layer 124 based on the input data 102 received by the detection layer 121 and / or further processed by the integration layer 122.

[0055] In some embodiments, the annotation layer 123 and / or inference layer 124 may use one or more ML / AI models (e.g., ML / AI model 127 shown in Figure 1) to take input data such as imaging data and provide one or more outputs (e.g., outputs 141-144 shown in Figure 1) based on the input data. For example, in some embodiments, one or more ML / AI models may be trained to generate and output unique identifiers (e.g., procedure identifiers 141 shown in Figure 1) for one or more surgical procedures performed, so as reflected in the input data 102. In some embodiments, one or more ML / AI models may be trained to identify surgical procedures performed on a patient's eye, so as reflected in the input data 102, and to output unique identifiers for the surgical procedures identified in the input data 102. As described above, the input data 102 that can be used by one or more ML / AI models to identify surgical procedures may include various types of information such as preoperative data 104, intraoperative data 106, and postoperative data 108. In some embodiments, preoperative data 104 may include one or more relevant physiological measurements of the patient (e.g., age, height, weight, body mass index, genetic makeup, race, ethnicity, sex, blood pressure, other demographic and health-related information, and / or similar), relevant risk factors (e.g., smoking history, diabetes, heart disease, other underlying conditions, past surgeries and / or similar, and / or a family history of one or more of these risk factors), the patient's medication history (e.g., current medications, past medications, allergies to medications and similar), etc. Intraoperative data 106 may include information such as imaging data, surgical system data (e.g., system parameters, active settings, and UI / UX / control status), other data modalities (e.g., voice commands, gesture-based commands, or commands that can be received by tracking the surgeon's gaze), patient monitoring information (e.g., eye position), machine settings (e.g., fluid settings, intraocular pressure settings, phaco energy settings, laser power settings, laser intensity settings, total laser energy application settings, etc.). Postoperative data 108 may include postoperative images, measurements, patient outcome data (e.g., satisfaction scores), treatment efficacy, treatment-related safety endpoints, and similar data.

[0056] As described above, one or more ML / AI models can be used to identify surgical procedures performed on a patient's eye based on various input data 102, such as the patient's medication history. In particular, some medications can increase the likelihood of certain symptoms occurring that necessitate specific surgical procedures. For example, intraoperative hypotonia syndrome, in which the iris of a patient's eye does not chemically dilate, is often associated with medications used to treat benign prostatic hyperplasia (BPH). In such cases, the surgeon may perform a surgical procedure to manually open the patient's iris, which does not chemically dilate, by inserting an iris hook or malugin ring. Therefore, in some embodiments, if the patient's medication history includes indications of medications used to treat BPH, one or more ML / AI models can use this information to more accurately predict whether a malugin ring insertion procedure has been performed on the patient. For example, one or more ML / AI models can combine this medication history with imaging data (e.g., surgical instruments or implants identified in the imaging data, as described below in relation to Figures 3A-3D) to predict whether a malugin ring insertion procedure has been performed on the patient.

[0057] In some embodiments, a model trainer of a digital recognition system (e.g., model trainer 126) can be configured to train one or more ML / AI models using a training dataset. In some embodiments, the training dataset may associate various types of information related to various surgical procedures (e.g., input data 102) with corresponding unique identifiers. For example, the training dataset may include multiple data entries or records associated with various past patients, each data record including various types of input data associated with the past patient's surgery (e.g., video and / or image data related to various surgical procedures), as well as labels associated with the surgical procedures performed on the past patient. For example, the labels may represent a unique identifier corresponding to the surgical procedure (e.g., a billing code).

[0058] As discussed, various types of information related to various surgical procedures performed on a patient in the past may include, for example, past imaging data of various surgical procedures, past information on the use of surgical instruments during various surgical procedures, past voice command information during various surgical procedures (e.g., requests for specific surgical instruments and / or drugs), past information on the preparation of surgical instruments and / or drugs for various surgical procedures, the overall flow or progression of various surgical procedures in the past, and past target surgical sites for various surgical procedures.

[0059] To ensure that the ML / AI model makes accurate predictions, the model trainer of the digital recognition system runs many samples in the corresponding training dataset until the prediction error is minimized. For example, in an embodiment in which the model is trained to identify surgical procedures performed on a patient (e.g., phacoemulsification, vitrectomy, stabilization of the lens capsule, manual iris dehiscence, etc.), the model trainer runs many samples in the corresponding training dataset to generate one or more unique identifiers (i.e., Y^) associated with various surgical procedures represented in the training dataset. The model trainer is configured to train one or more ML / AI models based on the resulting error (i.e., YY^) for each sample, which points to the difference between the unique identifier (i.e., Y^) predicted by the ML model and the actual unique identifier (i.e., Y) associated with the surgical procedure, as shown by each training sample in the training dataset. In other words, the model trainer can adjust the weights of the ML / AI models to minimize the error (or deviation) between the predicted unique identifier and the actual unique identifier for various surgical procedures in the training dataset. The model trainer can similarly train other ML models described herein. For example, in a particular embodiment, the model trainer can train one or more ML / AI models to identify objects in video and / or image data.

[0060] By running many samples through one or more ML / AI models and continuously adjusting the weights, beyond a certain point in time, one or more ML / AI models will be able to make extremely accurate predictions with very low error rates. At that point, one or more ML / AI models can be ready to be deployed to take an input set for the current patient (e.g., input data such as imaging data, voice commands, surgical instrument usage, target surgical site, etc.) and generate one of the outputs described above (e.g., predicting a unique identifier for one or more procedures performed on the current patient, or an object identified in video / image data).

[0061] Hereafter, with reference to Figures 3A-3D, exemplary examples of ML / AI models trained to make predictions about surgical procedures performed on a patient will be described. As described above, one or more ML / AI models may be trained to identify surgical procedures reflected in the imaging data acquired in step 210. For example, in some embodiments, one or more ML / AI models of the digital recognition system may be configured to automatically identify at least one of one or more surgical instruments and surgical products used on the patient's eye in the imaging data. In some embodiments, the digital recognition system may then be configured to automatically identify at least one surgical procedure based on the surgical instrument or product identified in the imaging data.

[0062] In some embodiments, to automatically identify surgical procedures, the ML / AI model may be further configured to generate confidence scores associated with surgical procedures in the imaging data. In some embodiments, the confidence score indicates a level of confidence that the identification of the surgical procedure performed in the imaging data is correct. In some embodiments, the digital recognition system may be configured to identify a surgical procedure based on whether the confidence score is above a confidence threshold. In some embodiments, as described below, the confidence score may be determined based on or taking into account one or more surgical instruments or surgical products used on the patient's eye in the imaging data. In some embodiments, as described below, the confidence score may take into account other information related to the surgical procedure being performed, such as the target surgical site on the patient's eye.

[0063] Figures 3A-3D show multiple image frames of the imaging data, illustrating various surgical procedures performed on the patient's eye during cataract surgery, and corresponding confidence scores generated by one or more ML / AI models 127 of the digital recognition system 100. For example, Figure 3A shows the first image frame 301 of the imaging data. As shown, the first image frame 301 is simply a view of the patient's eye without any surgical instruments. Therefore, the surgical instruments are not identified by one or more ML / AI models, and one or more ML / AI models generate the highest confidence score associated with the view of the patient's eye. For example, as shown in Figure 3A, one or more ML / AI models can generate multiple confidence scores associated with different surgical procedures. For example, one or more ML / AI models can generate a first confidence score 302A (e.g., 0.5929) related to the view of the patient's eye, a second confidence score 302B (e.g., 0.3332) related to the viscoelastic injection procedure, and a third confidence score 302C (e.g., 0.022) related to the interstitial fluid replacement procedure. Since the first confidence score 302A is higher than the second and third confidence scores 302B and 302C, one or more ML / AI models can identify in Figure 3A that the viscoelastic injection procedure or interstitial fluid replacement procedure was performed only on the view of the patient's eye, and not on the patient's eye itself.

[0064] Subsequently, as shown in the second image frame 304 of Figure 3B, an IOL injector 305 appears in the field of view, which can be identified by one or more ML / AI models. The IOL injector 305 can be used by a surgeon to perform an IOL injection procedure in which an IOL is injected into the patient's eye. Therefore, based at least partially on the identified IOL injector 305, one or more ML / AI models can predict that an IOL injection procedure is being performed in the second image frame 304. For example, one or more ML / AI models can generate a confidence score 306 (e.g., 0.9781) associated with the IOL injection procedure based at least partially on the identified IOL injector 305. Then, one or more ML / AI models can predict that an IOL injection procedure is being performed in the second image frame 304 because the confidence score 306 (e.g., 0.9781) associated with the IOL injection procedure is greater than a confidence threshold (e.g., 0.50 or 50%). In this example, 50% is used as the confidence threshold, but you can also choose and use other confidence thresholds.

[0065] In some embodiments, information about the target surgical site in which one or more surgical instruments or surgical products are used can also be used by one or more ML / AI models to generate a confidence score 306 and identify the surgical procedure performed on the eye. For example, as shown in Figure 3B, one or more ML / AI models can identify that the IOL injector 305 is being used at a target surgical site 303 on the patient's eye, such as the corneal margin of the patient's eye. This additional information related to the target surgical site 303 can be used to improve the confidence score 306, thereby enabling one or more ML / AI models to more accurately predict that the IOL injection procedure is being performed in the second image frame 304.

[0066] Subsequently, as shown in the third image frame 307 of Figure 3C, after the IOL is injected into the patient's eye, a Sinskey hook 308 appears in the visual field, which can be identified by one or more ML / AI models. The Sinskey hook 308 can be used by a surgeon to perform an IOL dilation procedure. Therefore, based at least partially on the Sinskey hook 308, one or more ML / AI models predict that an IOL dilation procedure has been performed in the third image frame 307. For example, one or more ML / AI models can generate a confidence score 309 (e.g., 0.7851) associated with an IOL dilation procedure based at least partially on the identified Sinskey hook 308. Then, one or more ML / AI models can predict that an IOL dilation procedure has been performed in the third image frame 307 because the confidence score 309 (e.g., 0.7851) associated with an IOL dilation procedure is greater than the confidence threshold (e.g., 0.50 or 50%).

[0067] In some embodiments, in addition to the identified Sinskey-Hook 308, one or more ML / AI models can generate a confidence score 309 based on additional input information such as voice command information (e.g., requests to Sinskey-Hook 308), surgical instrument and / or drug preparation information in the patient's medical record (e.g., indicating that Sinskey-Hook 308 is prepared for surgery), and additional input information such as the overall flow or progress of the cataract surgery (e.g., additional intraoperative data 106), and predict that an IOL expansion procedure is being performed in the third image frame 307. For example, cataract surgery may generally have a known procedure flow indicating that an IOL expansion procedure is performed after an IOL injection procedure. Therefore, after identifying the performance of the IOL injection procedure, one or more ML / AI models can generate a confidence score 309 based on the overall procedure flow of the cataract surgery and then predict or identify that an IOL expansion procedure is being performed in the third image frame 307.

[0068] Following the IOL augmentation procedure, a Sinskey hook 311 appears in the field of view, as shown in the fourth image frame 310 of Figure 3D, and this can be identified by one or more ML / AI models. In addition, one or more ML / AI models can identify a malugin ring 312 in the fourth image frame 310. In some embodiments, the Sinskey hook 311 can be used by a surgeon to perform a malugin ring removal procedure to remove the malugin ring 312 from the patient's eye after the IOL has been placed. Thus, based at least partially on the identified Sinskey hook 311 and the malugin ring 312, one or more ML / AI models predict that a malugin ring removal procedure has been performed in the fourth image frame 310. For example, one or more ML / AI models can generate a confidence score 313 (e.g., 0.5335) associated with a malugin ring removal procedure based at least partially on the identified Sinskey hook 311. Subsequently, one or more ML / AI models can predict that the malusin ring removal procedure was performed in the fourth image frame 310 because the confidence score 313 (e.g., 0.5335) associated with the malusin ring removal procedure is greater than the confidence threshold (e.g., 0.50 or 50%).

[0069] In some embodiments, one or more ML / AI models may include deep learning (DL) models that can be trained to predict surgical procedures based on video and / or image data, such as multiple image frames of imaging data shown in Figures 3A, 3B, 3C, and 3D. The DL models can implicitly identify instruments used in the video / image data and encode features of the identified instruments. In some embodiments, the DL models can identify other features in the video / image data and be trained to make predictions about the surgical procedure being performed, such as the movement of specific instruments identified in the video / image data, time series of actions in the video / image data, the presence of specific objects in the eye (e.g., IOLs and IOL tactile parts), and time series of the current procedure relative to a preceding procedure (e.g., after IOL injection).

[0070] In some embodiments, one or more ML / AI models may include ML models configured to make predictions about the surgical procedure being performed based on combinations of features detected in video and / or image data (e.g., multiple image frames of the imaging data shown in Figures 3A, 3B, 3C, and 3D). In some embodiments, combinations of features that can be used by the ML model may include the presence of an IOL, the presence of a particular instrument, the movement of an instrument, or a time series of the current procedure relative to a preceding procedure (e.g., after IOL injection), and these features can be used to make predictions about the surgical procedure being performed.

[0071] Once a surgical procedure is identified using the techniques described above, the digital recognition system can generate a unique identifier corresponding to the surgical procedure, as described above with respect to step 220 in Figure 2. Subsequently, in step 230, the digital recognition system automatically provides a visual representation (e.g., visual aid 144 shown in Figure 1) associated with the surgical procedure and the generated unique identifier on the digital recognition system or on a user interface associated with the digital recognition system. In some embodiments, to ensure that the correct unique identifier is generated for a surgical procedure, the visual representation provided (e.g., displayed) on the user interface allows the user of the digital recognition system to confirm or reject the unique identifier generated for at least one surgical procedure.

[0072] Subsequently, in some embodiments, as shown in step 240 of Figure 2, upon receiving confirmation from a user (e.g., a surgeon or their medical staff) that the unique identifier is correct (e.g., confirming that the unique identifier is correct), the digital recognition system can automatically transmit the unique identifier to another system, such as an EMR system (e.g., database 135), for automated processing. In some embodiments, as shown in step 250 of Figure 2, upon receiving user input that contradicts or rejects the unique identifier, the digital recognition system can automatically overwrite the unique identifier and transmit a second unique identifier provided by the user via the user interface to the EMR for automated processing.

[0073] In some embodiments, in addition to transmitting a unique identifier to the EMR system, other information associated with the identified surgery may also be provided to the EMR system. For example, in some embodiments, the digital recognition system may be configured to select and output at least one representative image frame and / or video clip (e.g., representative image / video 142 shown in Figure 1) that demonstrate the execution of a surgical procedure. In some embodiments, the digital recognition system may select a representative image frame or video clip based on a confidence score, such as the confidence score shown in Figures 3A-3D, where the confidence score is greater than or equal to a confidence threshold. For example, the confidence score may indicate the confidence level at which the surgical procedure is demonstrated in the selected representative image frame or video clip. The digital recognition system can then automatically transmit at least one representative image frame and / or video clip demonstrating the execution of the surgical procedure to the EMR system. In some embodiments, the digital recognition system may report several aggregate metrics, such as the time taken for each surgical procedure, to the EMR or directly to the surgeon.

[0074] In some embodiments, the digital recognition system can generate and transmit patient records to the EMR system, such that it automatically transmits a unique identifier, as well as at least one representative image frame and / or video clip, to the EMR system. The patient records may include a representation of the identified surgical procedure, a unique identifier corresponding to the surgical procedure, and at least one representative image frame and / or video clip demonstrating the performance of the surgical procedure. In some embodiments, the digital recognition system may also be configured to output and include in the patient records a timestamp (e.g., timestamp information 143 shown in Figure 1) indicating when the representative image frame or video clip was taken.

[0075] In some embodiments, the digital recognition system can also be configured to allow the user to confirm or reject a representative image frame or video clip. For example, in some embodiments, the digital recognition system can transmit a representative image frame or video clip to the EMR system only if the user confirms that the representative image frame or video clip is correct. In other cases, upon receiving user input that contradicts or rejects the representative image frame or video clip, the digital recognition system can be configured to automatically overwrite the representative image frame or video clip, allowing the user to find or select a second representative image frame or video clip. For example, the digital recognition system can transmit a second representative image frame or video clip, provided by the user via a user interface for automated processing, to a second device.

[0076] Figure 4 shows an exemplary digital recognition system 400 that at least partially performs one or more functions of a digital recognition system (e.g., digital recognition system 100), such as operation 200 shown in Figure 2. The digital recognition system 400 may be any one of the imaging systems 110, 114, 116, surgical system 112, and computing device 120 of Figure 1.

[0077] As shown, the digital recognition system 400 includes a central processing unit (CPU) 402, one or more I / O device interfaces 404 that allow various I / O devices 414 (e.g., keyboard, display, mouse device, pen input, etc.) to be connected to the digital recognition system 400, a network interface 406 for connecting the digital recognition system 400 to a network 490 (which may be a local network, intranet, internet, or any other group of computing systems connected to communicate with each other, as described in relation to Figure 1), memory 408, storage 410, and an interconnection unit 412.

[0078] If the digital recognition system 400 is an imaging system (e.g., imaging systems 110, 114, or 116), the digital recognition system 400 may further include one or more optical components for acquiring an ophthalmic image of the patient's eye and any other components known to those skilled in the art. If the digital recognition system 400 is a surgical system (e.g., surgical system 112), the digital recognition system 400 may further include many other components known to those skilled in the art for performing ophthalmic surgery as described above with respect to Figure 1 and known to those skilled in the art.

[0079] The CPU 402 can retrieve and execute programming instructions stored in memory 408. Similarly, the CPU 402 can retrieve and store application data residing in memory 408. The interconnection unit 412 transmits programming instructions and application data between the CPU 402, the I / O device interface 404, the network interface 406, the memory 408, and the storage 410. The CPU 402 is included as a representative of a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc.

[0080] Memory 408 represents volatile memory such as random access memory, and / or non-volatile memory such as non-volatile random access memory or phase-change random access memory. As shown, memory 408 includes a detection layer 421, an integration layer 422, an annotation layer 423, an inference layer 424, a startup layer 425, a model trainer 426, an ML / AI model 427, and a startup application 428. The functions of the detection layer 421, an integration layer 422, an annotation layer 423, an inference layer 424, a startup layer 425, a model trainer 426, and an ML / AI model 427 are similar to or identical to the functions of the detection layer 121, an integration layer 122, an annotation layer 123, an inference layer 124, a startup layer 125, a model trainer 126, and an ML / AI model 127. All instructions, modules, layers, and applications in memory 408 are shown in dashed boxes to indicate their optionality, so it should be noted that, depending on the function of the digital recognition system 400, one or more instructions, modules, layers, and applications may be executed by the digital recognition system 400 while others do not. For example, if the digital recognition system 400 is an imaging system (e.g., one of imaging systems 110, 114, or 116) or a surgical system (e.g., surgical system 112), then in certain embodiments, memory 408 may store a startup application 428 instead of a model trainer 426 (to trigger one or more actions based on output 140). If the digital recognition system 400 is a server system configured to train an ML / AI model 427 (e.g., neither an imaging system nor a surgical system), then in certain embodiments, memory 408 may store a model trainer 426 instead of a startup application 428.

[0081] The storage 410 may be non-volatile memory such as a disk drive, a solid-state drive, or a group of storage devices distributed across multiple storage systems. The storage 410 may optionally store input data 430 (e.g., similar to or identical to input data 102) and training dataset 432. The training dataset 432 may be used by the model trainer 426 to train the ML / AI model 427 as described above. The training dataset 432 may also be stored in external storage such as a database (e.g., database 130).

[0082] In some embodiments, one or more of the detection layer 421, integration layer 422, annotation layer 423, inference layer 424, invocation layer 425, model trainer 426, ML / AI model 427, and invocation application 428 contained within the memory 408 may include programming instructions for performing the operation 200 shown in Figure 2.

[0083] For example, in some embodiments, based on a programming instruction, the CPU 402 may cause the digital recognition system 400 to acquire imaging data of the patient's eye captured during a period. In some embodiments, the CPU 402 may further cause the digital recognition system 400 to automatically generate a unique identifier corresponding to at least one surgical procedure performed on the patient's eye during a period, based on the imaging data. In some embodiments, the CPU 402 may further cause the digital recognition system 400 to automatically provide a visual representation on a user interface (e.g., an I / O device 414 such as a display) associated with the at least one surgical procedure and the generated unique identifier. In some embodiments, the CPU 402 may further cause the digital recognition system 400 to automatically transmit the unique identifier to a second device for automated processing upon receiving confirmation from a user confirming the unique identifier. In some embodiments, the CPU 402 may further cause the digital recognition system 400 to automatically overwrite the unique identifier and transmit a second unique identifier provided by the user via the user interface to the second device for automated processing upon receiving user input that contradicts or rejects the unique identifier.

[0084] In some embodiments, the I / O device 414 of the digital recognition system 400 may include a digital camera configured to capture imaging data of the patient's eye during a period and to output the imaging data to the CPU 402.

[0085] In some embodiments, the CPU 402 can further cause the digital recognition system 400 to automatically identify at least one of one or more surgical instruments used on the patient's eye in the imaging data and at least one of one or more surgical products used on the patient's eye in the imaging data.

[0086] In some embodiments, the CPU 402 can further cause the digital recognition system 400 to automatically identify at least one surgical procedure performed on the patient's eye based on at least one of one or more surgical instruments and at least one of one or more surgical products identified in the imaging data.

[0087] In some embodiments, to automatically identify at least one surgical procedure performed on a patient's eye, the CPU 402 can cause the digital recognition system 400 to generate a confidence score for at least one surgical procedure performed on a patient's eye in the imaging data, based on one or more surgical instruments or surgical products identified in the imaging data, the confidence score indicating the confidence level regarding the correct identification of at least one surgical procedure. In addition, the CPU 402 can cause the digital recognition system 400 to automatically identify at least one surgical procedure performed on a patient's eye based on the confidence score.

[0088] In some embodiments, the CPU 402 may further cause the digital recognition system 400 to automatically identify at least one surgical procedure based on the fact that the confidence score is greater than or equal to the confidence threshold for at least a threshold time period.

[0089] In some embodiments, the CPU 402 can further cause the digital recognition system 400 to automatically identify, in the imaging data, the target surgical site of the eye where one or more surgical instruments or one or more surgical products are used. In some embodiments, the CPU 402 can further cause the digital recognition system 400 to automatically identify, based on the target surgical site of the patient's eye identified in the imaging data, at least one surgical procedure performed on the patient's eye.

[0090] In some embodiments, the CPU 402 can further cause the digital recognition system 400 to automatically identify at least one surgical procedure based on an ML / AI model 427 trained to identify surgical procedures performed in the imaging data.

[0091] In some embodiments, the CPU 402 can further cause the digital recognition system 400 to automatically generate a unique identifier corresponding to at least one surgical procedure, based on an ML / AI model 427 trained to generate a unique identifier for the surgical procedure performed in the imaging data.

[0092] In some embodiments, the CPU 402 may cause the digital recognition system 400 to acquire a training dataset 432. In some embodiments, the training dataset 432 associates historical imaging data related to various surgical procedures with corresponding unique identifiers. In some embodiments, the CPU 402 may cause the model trainer 426 of the digital recognition system 400 to train an ML / AI model 427 based on the training dataset to generate a unique identifier corresponding to at least one surgical procedure based on the imaging data.

[0093] In some embodiments, the imaging data includes multiple image frames captured during the period. In some embodiments, the CPU 402 can cause the digital recognition system 400 to select at least one representative image frame and / or representative video clip that demonstrate the performance of at least one surgical procedure. In some embodiments, the CPU 402 can cause the digital recognition system 400 to select at least one representative image frame and / or representative video clip based on the confidence score being equal to or greater than a confidence threshold. In some embodiments, the confidence score indicates the confidence level regarding the demonstration of at least one surgical procedure in the selected representative image frame or representative video clip.

[0094] In some embodiments, the CPU 402 can further cause the digital recognition system 400 to automatically transmit to a second device at least one representative image frame and representative video clip demonstrating the performance of at least one surgical procedure.

[0095] In some embodiments, the CPU 402 can cause the digital recognition system 400 to further generate a patient record by automatically transmitting a unique identifier, as well as at least one representative image frame and / or video clip, to a second device. In some embodiments, the patient record includes a display of at least one surgical procedure, a unique identifier corresponding to at least one surgical procedure, and at least one representative image frame and / or video clip demonstrating the performance of at least one surgical procedure.

[0096] In some embodiments, the CPU 402 may cause the digital recognition system 400 to further include a timestamp in the patient record indicating when a representative image frame or representative video clip was captured.

[0097] In some embodiments, the visual representation provided on a user interface (e.g., an I / O device 414 such as a display) further includes a selected representative image frame or representative video clip. In such cases, the CPU 402 may further cause the digital recognition system 400 to automatically transmit the representative image frame or representative video clip to a second device for automatic processing upon receiving confirmation from the user to verify a unique identifier, in order to automatically transmit the representative image frame or representative video clip to a second device. In some embodiments, the CPU 402 may further cause the digital recognition system 400 to automatically overwrite the representative image frame or representative video clip upon receiving user input that contradicts or rejects the representative image frame or representative video clip, and transmit a second representative image frame or representative video clip provided by the user via the user interface to the second device for automatic processing.

[0098] Additional exemplary embodiments Additional implementation examples are described in the following numbered embodiments.

[0099] Embodiment 1: An ophthalmic surgical system for automatically tracking surgical procedures, comprising a memory containing executable instructions and a processor configured to execute the executable instructions, wherein the processor is configured to cause the ophthalmic surgical system to acquire one or more imaging data of: the anatomical structures of a patient observed during at least one surgical procedure; one or more surgical instruments observed during at least one surgical procedure; and one or more imaging data of surgical implants observed during at least one surgical procedure; to automatically generate a unique identifier corresponding to at least one aspect of at least one surgical procedure based on the imaging data; and to automatically transmit the unique identifier to a second device for processing.

[0100] Embodiment 2: The ophthalmic surgery system according to Embodiment 1, wherein the imaging data includes at least one of video data and an image frame.

[0101] Embodiment 3: The ophthalmic surgical system according to Embodiment 1, further configured to cause the ophthalmic surgical system to automatically store in memory at least one surgical procedure and a visual representation associated with a generated unique identifier.

[0102] Embodiment 4: The ophthalmic surgery system according to Embodiment 1, further comprising a digital camera configured to capture imaging data of the patient's anatomical structures during a period and to output the imaging data to at least one processor.

[0103] Embodiment 5: The ophthalmic surgical system according to Embodiment 1, further configured to cause the ophthalmic surgical system to automatically identify at least one surgical procedure based on imaging data.

[0104] Embodiment 6: The ophthalmic surgical system according to Embodiment 5, further configured so that the processor causes the ophthalmic surgical system to automatically identify at least one of one or more surgical instruments and at least one of one or more surgical implants in the imaging data in order to identify at least one surgical procedure.

[0105] Embodiment 7: The ophthalmic surgical system according to Embodiment 6, further configured to cause the ophthalmic surgical system to automatically identify at least one surgical procedure based on at least one of one or more surgical instruments and at least one of one or more surgical products identified in the imaging data.

[0106] Embodiment 8: The ophthalmic surgical system according to Embodiment 7, wherein at least one processor is configured to generate a confidence score for at least one surgical procedure based on one or more surgical instruments or one or more surgical products identified in the imaging data, wherein the confidence score indicates a confidence level regarding the correctness of the identification of at least one surgical procedure, and to cause the ophthalmic surgical system to automatically identify at least one surgical procedure based on the confidence score.

[0107] Embodiment 9: The ophthalmic surgical system according to Embodiment 8, further configured to cause the ophthalmic surgical system to automatically identify at least one surgical procedure based on the fact that the confidence score is greater than or equal to a confidence threshold for at least a threshold time period.

[0108] Embodiment 10: The ophthalmic surgical system according to Embodiment 6, further configured to cause the ophthalmic surgical system to automatically identify, in imaging data, a target surgical site for an anatomical object in which one or more surgical instruments or one or more surgical products are used, and to automatically identify, in addition to the target surgical site for the anatomical object identified in the imaging data, at least one surgical procedure.

[0109] Embodiment 11: The ophthalmic surgery system according to Embodiment 5, wherein at least one processor is configured to cause the ophthalmic surgery system to automatically identify at least one surgical procedure based on a machine learning (ML) model trained to identify surgical procedures performed in imaging data.

[0110] Embodiment 12: The ophthalmic surgery system according to Embodiment 1, wherein at least one processor is configured to cause the ophthalmic surgery system to automatically generate a unique identifier corresponding to at least one surgical procedure, based on a machine learning (ML) model trained to generate a unique identifier for a surgical procedure performed in imaging data.

[0111] Embodiment 13: The ophthalmic surgery system according to Embodiment 12, wherein at least one processor is configured to cause the ophthalmic surgery system to acquire a training dataset, wherein the training dataset associates historical imaging data related to various surgical procedures with corresponding unique identifiers, and to train an ML model based on the training dataset to generate unique identifiers corresponding to at least one surgical procedure based on the imaging data.

[0112] Embodiment 14: The ophthalmic surgical system according to Embodiment 1, wherein the imaging data includes a plurality of image frames captured during a period, and at least one processor is further configured to cause the ophthalmic surgical system to select at least one representative image frame and at least one representative video clip that demonstrates the performance of at least one surgical procedure.

[0113] Embodiment 15: The ophthalmic surgery system according to Embodiment 14, wherein at least one processor is configured to select at least one representative image frame and / or representative video clip based on the confidence score being greater than or equal to a confidence threshold, and the confidence score indicates a confidence level that at least one surgical procedure is demonstrated in the selected representative image frame or representative video clip.

[0114] Embodiment 16: The ophthalmic surgical system according to Embodiment 14, further configured to cause the ophthalmic surgical system to automatically transmit to a second device at least one representative image frame and at least one representative video clip demonstrating the performance of at least one surgical procedure.

[0115] Embodiment 17: The ophthalmic surgery system according to Embodiment 16, wherein at least one processor is further configured to cause the ophthalmic surgery system to generate a patient record in order to automatically transmit a unique identifier, and at least one of a representative image frame and a representative video clip, to a second device, the patient record comprising a display of at least one surgical procedure, a unique identifier corresponding to at least one surgical procedure, and at least one of a representative image frame and a representative video clip demonstrating the performance of at least one surgical procedure.

[0116] Embodiment 18: The ophthalmic surgical system according to Embodiment 17, further configured with at least one processor to cause the device to include a timestamp indicating when a representative image frame or representative video clip was captured in the patient record.

[0117] Embodiment 19: The ophthalmic surgery system according to Embodiment 16, further configured to cause the ophthalmic surgery system to store in memory a visual representation associated with at least one surgical procedure and a generated unique identifier, wherein the visual representation stored in memory further includes a selected representative image frame or representative video clip, and the representative image frame or representative video clip is to be automatically transmitted to a second device, wherein the at least one processor is further configured to automatically transmit the representative image frame or representative video clip to the second device for automatic processing upon receiving confirmation from a user confirming the unique identifier, and to automatically overwrite the representative image frame or representative video clip and transmit to the second device a second representative image frame or representative video clip provided by the user via a user interface for automatic processing upon receiving user input that contradicts or rejects the representative image frame or representative video clip.

[0118] Additional matters The above description is provided so that a person skilled in the art can implement the various embodiments described herein. Various modifications to these embodiments will be obvious to a person skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes to the function and arrangement of the elements discussed may be made without departing from the scope of this disclosure. Various procedures or components may be omitted, substituted or added as needed in various examples. Also, features described in some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be performed using any number of embodiments described herein. Furthermore, the scope of this disclosure shall include, in addition to or otherwise, similar apparatuses or methods implemented using structures, functions, or structures and functions, in addition to the various embodiments of the disclosure described herein. It should be understood that any embodiment of the disclosure disclosed herein can be realized by one or more elements described in the claims.

[0119] As used herein, the phrase “at least one” of the enumerated items refers to any combination of those items, including a single component. For example, “at least one of a, b, or c” includes not only a, b, c, ab, ac, bc, and abc, but also any combination of multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0120] As used herein, the term "decision" encompasses a wide range of actions. For example, "decision" may include calculation, computation, processing, derivation, investigation, referencing (e.g., referencing a table, database, or other data structure), confirmation, etc. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "decision" may include resolving, selecting, choosing, establishing, etc.

[0121] The methods disclosed herein include one or more steps or operations to implement the method. The steps and / or operations of the method are interchangeable without departing from the claims. In other words, unless a specific order of steps or operations is specified, the order and / or use of specific steps and / or operations may be changed without departing from the claims. Furthermore, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding function. These means may include, but are not limited to, various hardware and / or software elements and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations are shown in the drawings, these operations may include corresponding means-plus-function elements with similar numbering.

[0122] The various exemplary logic blocks, modules, and circuits described in connection with this disclosure may be implemented or run by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic element (PLD), discrete gate or transistor logic, discrete hardware elements, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, multiple microprocessors, a combination of a DSP core and one or more microprocessors working together, or any other such configuration.

[0123] The processing system may be implemented using a bus architecture. The bus may include any number of interconnection buses and bridges, depending on the specific application and overall design constraints of the processing system. The bus can interconnect various circuits, in particular, including processors, machine-readable media, and input / output devices. User interfaces (e.g., keypads, displays, mice, joysticks, etc.) may also be connected to the bus. The bus can also connect various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, which are known in the art and will not be described further. The processor may be implemented with one or more general-purpose and / or dedicated processors. Several examples include microprocessors, microcontrollers, DSP processors, and other circuits capable of running software. Those skilled in the art will recognize how to optimally implement the described functions of the processing system, depending on the specific application and the overall design constraints imposed on the system as a whole.

[0124] If implemented in software, the functions described above may be stored or transmitted as one or more instructions or code on a computer-readable medium. Software should be interpreted broadly as instructions, data, or any combination thereof, regardless of the terminology used, including software, firmware, middleware, microcode, hardware description language, etc. Computer-readable medium includes both computer storage media and communication media, such as any medium that facilitates the transfer of computer programs from one location to another. The processor may be responsible for managing bus and general operations, including the execution of software modules stored on the computer-readable storage medium. The computer-readable storage medium may be coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be built into the processor. For example, the computer-readable medium may include computer-readable storage media in which instructions are stored separately from transmission lines, data-modulated carriers, and / or wireless nodes, all of which may be accessed by the processor via a bus interface. Alternatively or additionally, computer-readable media, or any portion thereof, may be incorporated into the processor, as in the case of caches and / or general-purpose register files. Examples of machine-readable storage media include, for example, RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage media or any combination thereof. Machine-readable media may be implemented in computer program products.

[0125] A software module may contain a single instruction or a number of instructions, and may be distributed across several different code sections, across different programs, and across multiple storage media. A computer-readable medium may contain a number of software modules. A software module contains instructions that, when executed by a device such as a processor, cause a processing system to perform various functions. A software module may include a send module and a receive module. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, when a trigger event occurs, a software module may be loaded from the hard drive into RAM. While a software module is executing, the processor may load some of the instructions into a cache to increase access speed. Then, one or more cache lines may be loaded into a general-purpose register file for execution by the processor. When referring to the functionality of a software module, it will be understood that such functionality is realized by the processor when executing instructions from that software module.

[0126] The following claims are not limited to the embodiments described herein and shall be subject to the full scope consistent with the language of the claims. Where an element is referred to in the singular in a claim, it shall mean "one or more" and not "just one" unless otherwise specifically declared. Unless otherwise specifically stated, the term "several" means one or more. No element of a claim shall be construed under Section 112(f) of the United States Patent Act unless it is expressly described using the phrase "means for..." or, in the case of a method claim, the phrase "steps to...". All structural and functional equivalents of elements of various forms described throughout this disclosure, which are known to those skilled in the art or which may become publicly known thereafter, are expressly incorporated herein by reference and are incorporated into the claims. Furthermore, nothing disclosed herein is intended to be dedicated to the public, whether such disclosure is expressly enumerated in the claims or not.

Claims

1. An ophthalmic surgical system for automatically tracking surgical procedures performed on anatomical objects of patients, Memory containing executable instructions, A processor configured to execute the aforementioned executable instructions and The processor includes, To acquire imaging data of the anatomical objects of the patient captured during the period, Based on the imaging data, a unique identifier is automatically generated corresponding to at least one surgical procedure performed on the anatomical object during the period. To automatically provide the user interface with the visual representation associated with the at least one surgical procedure and the generated unique identifier, Upon receiving confirmation from the user to verify the aforementioned unique identifier, the unique identifier is automatically transmitted to the second device for automated processing. Upon receiving user input that is inconsistent with or rejected by the aforementioned unique identifier, the unique identifier is automatically overwritten, and a second unique identifier provided by the user via the user interface is transmitted to the second device for automated processing. An ophthalmic surgical system configured to perform an ophthalmic surgical procedure.

2. The aforementioned ophthalmic surgical system, During the aforementioned period, the imaging data of the anatomical object of the patient is captured, Outputting the aforementioned imaging data to at least one processor The ophthalmic surgery system according to claim 1, further comprising a digital camera configured to perform the following.

3. The aforementioned at least one processor, One or more surgical instruments used on the anatomical object in the imaging data, and One or more surgical products used for the anatomical object in the imaging data. The ophthalmic surgical system according to claim 1, further configured to cause the ophthalmic surgical system to automatically identify at least one of the following.

4. The ophthalmic surgical system according to claim 3, wherein the at least one processor is further configured to cause the ophthalmic surgical system to automatically identify the at least one surgical procedure performed on the anatomical object based on the one or more surgical instruments identified in the imaging data and at least one of the one or more surgical products identified in the imaging data.

5. In order to automatically identify the at least one surgical procedure performed on the anatomical object, the at least one processor, To generate a confidence score for the at least one surgical procedure performed on the anatomical object in the imaging data, based on the one or more surgical instruments or the one or more surgical products identified in the imaging data, wherein the confidence score indicates a level of confidence that the identification of the at least one surgical procedure is correct. Based on the confidence score, the system automatically identifies the at least one surgical procedure performed on the anatomical object. The ophthalmic surgical system according to claim 4, configured to perform the ophthalmic surgical procedure.

6. The ophthalmic surgical system according to claim 5, further configured to cause the ophthalmic surgical system to automatically identify the at least one surgical procedure based on the fact that the confidence score is greater than or equal to a confidence threshold for at least a threshold time period.

7. The aforementioned at least one processor, In the aforementioned imaging data, the target surgical site of the anatomical object in which one or more surgical instruments or one or more surgical products are used is automatically identified. Based on the target surgical site of the anatomical object identified in the imaging data, the system automatically identifies the at least one surgical procedure performed on the anatomical object. The ophthalmic surgical system according to claim 4, further configured to cause the ophthalmic surgical system to perform the procedure.

8. The ophthalmic surgery system according to claim 4, wherein the at least one processor is configured to cause the ophthalmic surgery system to automatically identify the at least one surgical procedure based on a machine learning (ML) model trained to identify the surgical procedures performed in the imaging data.

9. The ophthalmic surgery system according to claim 1, wherein the at least one processor is configured to cause the ophthalmic surgery system to automatically generate the unique identifier corresponding to the at least one surgical procedure based on a machine learning (ML) model trained to generate a unique identifier for the surgical procedure performed in the imaging data.

10. The aforementioned at least one processor, The acquisition of a training dataset, wherein the training dataset associates historical imaging data related to various surgical procedures with corresponding unique identifiers. The ML model is trained based on the training dataset to generate the unique identifier corresponding to the at least one surgical procedure based on the imaging data. The ophthalmic surgical system according to claim 9, configured to perform the ophthalmic surgical procedure.

11. The imaging data includes a plurality of image frames captured during the period, The ophthalmic surgical system according to claim 1, wherein the at least one processor is further configured to cause the ophthalmic surgical system to select at least one representative image frame and a representative video clip that demonstrate the performance of the at least one surgical procedure.

12. The ophthalmic surgery system according to claim 11, wherein the at least one processor is configured to select at least one of the representative image frame and the representative video clip based on the confidence score being greater than or equal to a confidence threshold, and the confidence score indicates a level of confidence that the at least one surgical procedure is demonstrated in the selected representative image frame or representative video clip.

13. The ophthalmic surgery system according to claim 11, wherein the at least one processor is further configured to cause the ophthalmic surgery system to automatically transmit to the second device at least one of the representative image frames and the representative video clips demonstrating the execution of the at least one surgical procedure.

14. In order to automatically transmit the unique identifier, as well as at least one of the representative image frame and the representative video clip, to the second device, the at least one processor is further configured to cause the ophthalmic surgery system to generate a patient record. The ophthalmic surgery system according to claim 13, wherein the patient record includes a display of the at least one surgical procedure, the unique identifier corresponding to the at least one surgical procedure, and at least one of the representative image frame and the representative video clip demonstrating the performance of the at least one surgical procedure.

15. The ophthalmic surgery system according to claim 14, wherein the at least one processor is further configured to cause the device to include a timestamp indicating when the representative image frame or representative video clip was captured in the patient record.

16. The visual representation provided on the user interface further includes the selected representative image frame or representative video clip, In order to automatically transmit the representative image frame or representative video clip to the second device, at least one processor, Upon receiving confirmation from the user to verify the unique identifier, the system automatically transmits the representative image frame or representative video clip to the second device for automated processing. Upon receiving user input that contradicts or rejects the aforementioned representative image frame or representative video clip, the system automatically overwrites the representative image frame or representative video clip and transmits a second representative image frame or representative video clip provided by the user via the user interface to the second device for automatic processing. The apparatus according to claim 13, further configured to perform the following:

Images