Combined source algorithm models

Abstract

A system for identifying and classifying a target in an image obtained during a. medical procedure can comprise a medical device, and processing circuitry connected to the medical device. The processing circuitry can be configured to receive an image of a surgical site and analyze the image of the surgical site to locate a target in the image. The processing circuitry can further cause light to be emitted toward, the target from a light source associated with the medical device. At least a portion of the emitted light scattered by or reflected from the target can be collected at an optical splitter, and the collected light can be analyzed using a spectrometer coupled to the optical splitter to identify or classify the target. The processing circuitry can provide an output to a user based on at least one of the identification or classification of the target.

Description

COMBINED SOURCE ALGORITHM MODELS PRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 515,896, filed July 27, 2023, and U.S. Provisional Patent Application Serial No. 63 / 652,256, filed May 28, 2024, the contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to determining a characteristic of a target during a medical procedure. BACKGROUND

[0003] During a medical procedure, such as a lithotripsy procedure, a user such as a physician or a robot may interact with one or more targets of various and distinct types, sizes, or other characteristics such as material composition. The target can include a calculus such as a stone (e.g., a kidney stone or a gall stone), a tumor, a piece of tissue, or the like, in a patient’s body. The target can be located in a medium (e.g., a liquid medium such as water or saline), and the physician can use a medical device such as a scope, (e.g., an endoscope) or a surgical laser for surgical procedures on the targets.

[0004] Characteristics of the target can vary depending on the type of procedure and can change during the procedure. One or more characteristics of the target can impact how to begin the medical procedure (e.g., what type of medical instruments to use, what types of lasers to use, or the like) and conditions during the procedure, including changes to the target, can impact how a physician proceeds during the procedure. SUMMARY

[0005] It is desirable to know one or more characteristics of anatomical targets within a patient, such as a tumor or a stone, when performing either a diagnostic or therapeutic procedure. For example, a user attempting to visualize a cancerous tumor needs to determine the dimensions (e.g., height and width) of the tumor to determine whether the tumor can be surgically removed, to determine a baseline Attorney Docket No.5409.873WO1 1 Client Reference No. GAP24019-URKT-WO1measurement for the tumor to monitor change in the size of the tumor after cancer treatment has begun, or the like. Similarly, a user performing an endoscopic procedure such as stone or tissue ablation or removal may want to determine the size of the stone or piece of tissue to be removed, such as to decide if the target is small enough to remove using the scope, such as removal through a ureter or through an access sheath with a particular inner diameter, or whether the target should be reduced further before removal. During laser procedures such as laser lithotripsy, the physician may want to know the location of the scope within the body of the patient. For example, the physician may want to know the distance between the tip of the scope and the target. The distance between the tip of the scope and the target is helpful to determine, for example, the amount or intensity of ablation energy (e.g., laser energy, radiation, ultrasonic energy, or the like) to deliver to the target.

[0006] Furthermore, as targets change during a procedure, the scope may need to be repositioned, the intensity of the ablation energy may need to be adjusted (or stopped entirely). For example, the composition of the target may change during the procedure (e.g., the material composition of a kidney stone may change as the stone is ablated) or tissue being contacted by laser energy may change from target tissue to non-target tissue. Similarly, a physician may need to be aware of when ablation energy may be contacting non-target tissue or objects.

[0007] A model can be used to calculate, predict, or suggest how to conduct a surgical procedure or how to proceed during the surgical procedure. For example, the model can be used to predict how much ablation energy, such as from a laser or an acoustic transducer, is necessary or appropriate to ablate or reduce a kidney stone. The model can base the prediction at least in part on endoscopic video or images collected from an imaging device such as a camera or optical spectrum measurements of the target or spectral analysis of signals from the surgical scene. For example, the endoscopic video can be used to locate one or more targets in a surgical scene. Then, a response signal from a target, such as light reflected from, scattered by, or otherwise emitted from the target (e.g., in response to illumination of the target) can be collected by an optical sensor or light detector and analyzed by a device such as a spectrometer to identify or classify the target. The accuracy or precision of the model can be enhanced or increased by using data or measurements of one or more Attorney Docket No.5409.873WO1 2 Client Reference No. GAP24019-URKT-WO1parameters, such as temperature or pressure, at the surgical site, collected by one or more sensors that can be included on or connected to a medical device such as an endoscope.

[0008] A system for identifying and classifying a target in an image obtained during a medical procedure can comprise a medical device and processing circuitry. The processing circuitry can be configured or configurable to receive an image of a surgical site and analyze the image of the surgical site to locate a target in the image of the surgical site. The processing circuitry can further cause an electromagnetic signal such as light to be emitted toward the target from a light source associated with (e.g., included on, connected to, coupled to, or the like) the medical device. The electromagnetic signal can be a signal at an optical or visible wavelength (e.g., in a human-visible range between 100 nanometers (nm) and 1 millimeter (mm)) or a signal in a non-optical or non-visible wavelength. The non-visible signal can include ultraviolet light, infrared light, x- rays, microwaves, gamma rays, or the like. At least a portion of the emitted light reflected back from, or scattered by the target can be collected at an optical detector or other similar light collector and passed to an optical sensor such as a spectrometer. The spectrometer can analyze the collected light to at least one of identify or classify the target. Based on at least one of the identification or classification of the target, the processing circuitry can provide an output to a user, such as a physician.

[0009] Thus during a procedure, the system can use an endoscopic image (e.g., an image from a camera or other imaging sensor on or coupled to the endoscope) in conjunction with spectral analysis of the surgical scene to identify a target or to classify or categorize the target by type. The endoscopic image(s) can be used to look at the target (e.g., a stone) in large scale and provide a gross estimate of the target's characteristics, whereas the spectroscopic data can be used to look at the target in a small scale and provide more accurate target characteristics.

[0010] The medical device can include an endoscope and the image of the surgical site can be captured by an endoscopic camera. The processing circuitry can compare one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site. The processing circuitry can calibrate or otherwise adjust one or more colors on the image of the surgical site based on the spectral analysis of the surgical site to Attorney Docket No.5409.873WO1 3 Client Reference No. GAP24019-URKT-WO1create a calibrated endoscopic image of the surgical site. The calibrated endoscopic image of the surgical site can be output to a graphical user interface such as a computer screen or monitor, a tablet, or the like. The calibrated endoscopic image can be annotated to include, for example, information about the target, information about the surgical site, identification of non-target anatomy, information about a change to the target, or information about a change to the surgical site.

[0011] In an example, the processing circuitry can receive data from one or more additional sources such as historical data pertaining to a patient undergoing the medical procedure, historical data pertaining to a prior medical procedure undergone by members of a demographic class to which the patient belongs, or real-time data obtained from a sensor coupled to or connected to the medical device. The historical data can be retrieved from a database connected to the processing circuitry. The historical data pertaining to the patient can include information about a prior medical procedure, a pre-existing condition of the patient, or a physical characteristic of the patient. The sensor coupled to the medical device can include at least one of a temperature sensor, a pressure sensor, or a flow rate sensor. The output can include a warning or a recommendation. The warning can include at least one of a visual warning, an audible warning, or a haptic warning such as when ablation energy is contacting non-target anatomy. The recommendation can include a recommendation regarding a subsequent medical procedure such as a type of medical scope to use during the subsequent medical procedure, a type and intensity of laser to use during the subsequent medical procedure, a type and amount of anesthesia to use during the subsequent medical procedure, or a risk during the subsequent medical procedure.

[0012] The image of the surgical site can be analyzed using a first algorithm and identification or classification of the target can be made using a second algorithm. The output from the first algorithm and the output from the second algorithm can be used as inputs to a recommendation algorithm.

[0013] A potential advantage to using endoscopic images along with spectrum measurements (optical or non-optical spectrum measurements) and / or sensor data is that targets can be more accurately and precisely identified, and adjustments can be made in real-time based on changes in the target or Attorney Docket No.5409.873WO1 4 Client Reference No. GAP24019-URKT-WO1conditions at the surgical site during the procedure. For example, when the target is a stone, knowing whether the stone is formed from a soft or a hard tissue can impact how the stone is ablated. This increased accuracy and any adjustments made during the procedure can lead to more quicker and efficient procedures, greater comfort to the patient post-procedure, reduce the need for subsequent procedures, and provide helpful information to the physician when a subsequent procedure is necessary. Other examples of target characteristics that can determine how the current procedure should proceed or what subsequent procedure(s) may need to be performed later can include being able to identify whether a target is a prostate capsule or an adenoma, or whether the target is cancerous (or potentially or likely cancerous) or benign. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.

[0015] FIG. 1 illustrates an example of a lithotripsy system including an optical detector.

[0016] FIG. 2 illustrates a flowchart for using multiple algorithms to generate an output based on multiple data inputs.

[0017] FIG. 3 illustrates a method for identifying and classifying a target in an image during a medical procedure.

[0018] FIG. 4 is a block diagram of an example of a machine upon which any one or more of the techniques (e.g., methodologies) discussed herein can perform.

[0019] FIG. 5 shows a schematic diagram of an exemplary computer-based clinical decision support system (CDSS). DETAILED DESCRIPTION

[0020] A system using a combined source model to determine one or more target characteristics can include an imaging device. The imaging device can be Attorney Docket No.5409.873WO1 5 Client Reference No. GAP24019-URKT-WO1coupled to the distal tip of a medical scope such as an endoscope. The system can further include a light sensor or optical sensor such as a spectrometer. The spectrometer can be used to detect and analyze a signal from a target at or near the distal tip of the medical scope. The detected signal can include light reflected or otherwise scattered or emitted from the target, such as in response to an electromagnetic signal (optical or non-optical) emitted toward the target (e.g., illumination of the target with visible light, an infrared signal emitted toward the target, or the like). The system can also include a processing circuitry and memory. The memory can include instructions that, when executed by the processing circuitry, can cause the processing circuitry to identify or classify the target of a medical procedure. For example, during a lithotripsy procedure, the system can use an image from a camera or other imaging sensor in conjunction with spectral analysis of the surgical scene to identify a target or to classify or categorize the target by type. In such an example, the endoscopic images can be used to look at the target (e.g., a stone) in large scale and provide a gross estimate of the target's characteristics, whereas the spectroscopic data can be used to look at the target in a small scale and provide more accurate target characteristics. For example, the system can classify the target as tissue, stone, muscle tissue, endothelial tissue, or as a prostate capsule or prostate adenoma. In addition, the system can classify the target as normal or as abnormal (e.g., as cancerous). In an example, the endoscopic video and the spectral analysis of light signals from the surgical scene, including targets in the surgical scene can be synchronized such that spectral analysis can be performed at the same time (or substantially the same time) that the endoscopic video is collected.

[0021] The identification and classification of the target can be made by one or more algorithms. An algorithm can include an artificial intelligence (AI) or machine learning (ML) or other algorithm (e.g., a non-AI or non-ML deterministic algorithm) or process. Additionally, or alternatively, a portion of the identification and / or classification of the target or a recommendation can be made using a hardware-based feedback loop or feedback control. In an example, a single algorithm can be used to identify and classify the target (or determine a characteristic of the target). Alternatively, multiple algorithms can be used independently of one another. Their outputs can be passed to another model (e.g., a recommendation algorithm) such as to aggregate the outputs and make a Attorney Docket No.5409.873WO1 6 Client Reference No. GAP24019-URKT-WO1prediction or a recommendation based on the analysis by the other algorithms. For example, imaging data from the imaging sensor (e.g., endoscopic video or still frames, acoustic imaging, ultrasound imaging, or the like) can be delivered to a first algorithm for detecting or identifying the target. Spectra captured from a signal from light from the target (e.g., endoscopic light reflecting back along a surgical fiber, emission of fluorescence from the target in response to illumination of the target, or the like) can be delivered to the same algorithm or to a second, different algorithm such as to help classify the target (e.g., determine whether the target is a stone or tumor) or determine a characteristic, such as material composition or size of the target. When a second algorithm classifies the target or determines a characteristic of the target, results from the first algorithm and the second algorithm can be passed to a third algorithm to aggregate the outputs of the first and second algorithms, such as to make a prediction or recommendation, alert the user, or the like. For example, the third algorithm can make a prediction about the treatment location, a prediction about the target, make a recommendation for a future procedure, or the like.

[0022] In an example, the model can apply weighting factors so that the outputs of the first algorithm and the second algorithm can be weighed, scored, ranked, prioritized, or the like differently. For example, when the data from the imaging sensor passed to the first algorithm is a blurry endoscopic image, the output from the second algorithm (to which the spectra captured from the signal from light from the target is passed) can be weighed or ranked higher than the output of the first algorithm.

[0023] The prediction(s) can be used to provide insight to the user about the treatment site or to suggest an optimal treatment configuration. The treatment configuration can include a recommendation for a setting (or an adjustment to a setting) of the endoscope or of a treatment device being used by the endoscope, such as a laser or other ablation member. For example, the system can recommend or suggest an appropriate laser setting or to adjust a current setting of the laser (e.g., whether to increase, decrease, or terminate the laser radiation) for ablation of the target. Additionally or alternatively, the system can recommend or suggest to the user to increase or decrease irrigation or suction of the surgical site. Thus, rather than implementing the system to make sequential decisions from different algorithm models, both endoscopic image data and Attorney Docket No.5409.873WO1 7 Client Reference No. GAP24019-URKT-WO1spectral measurements can be delivered to one or more algorithms to deliver determinations (predictions or recommendations) in one or more steps.

[0024] In an example, data input into the algorithm models can include one or more inputs from other one or more measurements obtained by one or more additional sensors. The additional sensors can be located at or near the treatment site or coupled to the medical device and can include a temperature sensor, pressure sensor, flow rate sensor, or the like. Thus, examples of these other measurements can include temperature or pressure at the anatomical site, irrigation flow rate, irrigation temperature, suction flow rate, temperature of suction fluid, or the like.

[0025] Additionally, or alternatively, other data or factors such as anatomical or demographic information about the patient and / or the medical procedure can be passed to the one or more algorithms and used to create the prediction or recommendation. Such other data can include information about the patient. The information about the patient can include information about the medical procedure to be performed on the patient such as the anatomical treatment site. The information about the patient can also include information such as the age, sex, or ethnicity of the patient or preexisting conditions of the patient. The information about the patient can further include information about prior procedures conducted in the target anatomy, pre-operative imaging data (e.g., from a CT scan that diagnosed the patient to have stones) or information derived therefrom (e.g., information about the number of stones, stone size, stone location, or the like), tissue biopsy results, prior stone composition results, or the like. The other data can additionally include information about the medical instruments to be used during the procedure (e.g., procedural configuration data) such as the use of an access sheath, size of laser fiber, endoscope working channel or insertion tube size, endoscopic light source type, or the age of the light source.

[0026] Some or all of these factors can influence or change the algorithm models' suggestions for laser setting, irrigation fluid flow rate and temperature, fiber size, or the like. The predictions, suggestions, or recommendations made by the algorithms can be adjusted as target type, target composition, or conditions at the treatment site are identified or change during the procedure and can contribute to when the algorithm models suggest pausing laser emission or Attorney Docket No.5409.873WO1 8 Client Reference No. GAP24019-URKT-WO1adjusting a position or other setting of the scope. In an example, the system may identify a blood vessel as a non-target and, in response, change one or more settings such as stopping laser emission until the laser fiber is moved away from the blood vessel. In another example, in poor visibility conditions at the surgical field (e.g., caused by smoke, a large concentration of dust particles, or the like) the system may cause or recommend that the laser stop firing and to engage irrigation or suction to help clear the field). Potential advantages of the system described herein include more efficient treatment procedures, shorter treatment durations, and more efficient treatment outcomes. In the present disclosure, that ablation energy includes laser or acoustic ablation or any similar form of energy used to ablate a target or tissue and that the terms, laser, ablation, and acoustic energy can be used as alternates for each other.

[0027] FIG. 1 illustrates an example of a lithotripsy system including an optical detector. The system 100 can include a surgical laser 102 and a graphical user interface 104 (GUI). The graphical user interface 104 can include a touchscreen or other input mechanism (e.g., a button, switch, or other similar actuation member on the handle of the endoscope) configured to operate or control the surgical laser 102. The surgical laser 102 can include one or more laser sources configured to emit laser radiation. As shown in the dashed box in the example of FIG. 1, the light sources can include an ablation laser 106 and / or an illumination source 108. The illumination source 108 can include a probe laser, a Light Emitting Diode (LED), a Xenon-based light source, or any similar source of visible light. The ablation laser 106 can emit infrared radiation while the illumination source 108 can emit an aiming beam or an illumination beam of visible light to show where the tip of the scope (and therefore where the ablation energy from the ablation laser 106) is aimed. Additionally, or alternatively, the illumination source 108 can be used to illuminate a target 126. The target 126 can be a piece of tissue, debris, or an object, such as a kidney stone which is to be ablated, a tumor, a prostate capsule, or the like. The emitted light 128 from the ablation laser 106 or the illumination source 108, can be emitted through an optical fiber 116 such as can be connected to a surgical fiber 118 via an optical connector 120. In an example, the structure of the surgical fiber 118 can be the same or different from that of the optical fiber 116. The surgical fiber 118 can be located wholly or partially outside the surgical laser 102. The emitted light 128 Attorney Docket No.5409.873WO1 9 Client Reference No. GAP24019-URKT-WO1can thus be emitted from illumination source 108, through the optical fiber 116, the optical connector 120, and the surgical fiber 118, to a distal end of the surgical fiber 118. The distal end of the surgical fiber 118 can be inserted into a scope 124, such as an endoscope, a ureteroscope, laryngoscope, or the like. In an example, at least a portion of the emitted light 128 emitted from the distal end of the surgical fiber 118 and the scope 124 can be reflected off of, scattered by, or the like, a target 126 (reflected light 130) through a medium between the tip of the scope 124 and the target 126. The scope 124 can include an imaging device 122, such as a camera to capture video or images of the surgical scene, including the target 126. The medium can be any medium such as air, water, saline, carbon dioxide, or the like.

[0028] The surgical laser 102 can further include or couple to an optical component such as optical splitter 110, configured to collect at least a portion of the reflected light 130 passing through the aperture of the surgical fiber 118. In an example, the optical splitter 110 can be replaced with a dedicated fiber configured to collect at least a portion of the reflected light 130. The portion of reflected light 130 collected by the optical splitter 110 or dedicated fiber can be sent to a processor 112 in connection with or coupled to the surgical laser 102. An optical detector 132 (e.g., a spectrometer or other similar light detector or optical sensor) can be located between the optical splitter 110 and the processor 112, so that spectral analysis of the reflected light 130 can be performed in order to determine one or more characteristics of the target 126.

[0029] The processor 112 and / or optical detector 132 can analyze the portion of the reflected light 130 collected by the optical splitter 110 (or receive an analysis by a spectrometer connected or coupled to the processor 112 and / or the optical detector 132), to analyze the reflected light 130. The surgical laser 102 can optionally or additionally include controller 114 circuitry communicatively coupled to the processor 112. The processor 112 or the controller 114 can determine one or more characteristics of the reflected light 130 signal that is specific to the illumination source 108 such as a type of light source, a wavelength, an optical component, a pulse width, an optical configuration, or an age of the illumination source 108. While the example of FIG. 1 includes a single illumination source, it is understood that the system 100 can include multiple illumination sources of different types (e.g., an LED source, a Xenon- Attorney Docket No.5409.873WO1 10 Client Reference No. GAP24019-URKT-WO1based source, a broadband illumination source, or the like). Furthermore, while the illumination source 108 is shown to be a part or component of the surgical laser 102, the illumination source may be an external component of the laser system 100 (e.g. (external to the surgical laser 102) such as by being attached to the scope 124. It is also noted that the system 100 can include one or more other components such as a fluid pump, a suction unit, a smoke evaluator, a gas insufflator, or any component that may be needed or used during a laser procedure.

[0030] FIG. 2 illustrates a flowchart for using multiple algorithms to generate an output based on multiple data inputs. As shown in FIG. 2, data from multiple data sources can be used as inputs to multiple algorithms. The algorithms can be trained-learning algorithms, or non-trained deterministic algorithms, or a combination thereof. For example, data from a first data source 200 can be input into a first algorithm 206. The first data source 200 can include data from an imaging sensor such as a camera or video source capable of taking video or still images. The data in the first data source 200 can include still frames or video of a surgical site captured from a camera included in, coupled to, or the like, a medical device or medical scope such as an endoscope. The first data source 200 can further include red-green-blue (RGB) intensities from the endoscopic camera. The first algorithm 206 can perform analysis of the image or video from the first data source 200, for example, image segmentation, or the like, to locate a target (or a potential target) in the image.

[0031] Data from a second data source 202 can be input into a second algorithm 208. The second data source 202 can include a signal (e.g., light) collected from a target. For example, a light source included on, couple to, or the like, the medical device (e.g., a Xenon light source or any source of visible or non-visible light) can be emitted at the surgical scene such as toward the target identified in the image from the second data source 202. Light reflected off, scattered by, or the like, the target can be collected by an optical splitter included in, coupled to, or otherwise associated with the medical device.

[0032] An optical sensor such as a spectrometer can analyze the spectra of the light collected from the target to identify or classify the target. In an example, the spectral analysis can be used to confirm identification of a potential target in the image(s) from the first data source 200. Additionally, or alternatively, the Attorney Docket No.5409.873WO1 11 Client Reference No. GAP24019-URKT-WO1spectral analysis can be used to determine a characteristic or composition of a target identified in the image(s) from the first data source 200. For example, when a kidney stone is identified in an endoscopic image, light can be used to illuminate the kidney stone. Based at least in part on spectral analysis of light reflected off the stone, the composition of the stone (e.g., whether the stone is formed from uric acid or calcium oxalate), the size of the stone, or the like can be determined by the second algorithm 208. In an example, the morphology of the target acquired in the endoscopic image can help or aid in the spectral analysis. For example, the gross morphology or structure of a target such as a stone can be a better predictor of certain types of stones than spectral analysis alone. For example, uric acid stones typically present with smooth surface features, while brushite and calcium based stones present as having thin, sharp, protrusions (e.g., protruding crystal-like structures). Thus, the system can use the morphology of the stone (e.g., surface features of the stone, such as whether the stone is smooth or textured, the size of the stone, or the like) in conjunction with the spectral analysis to identify the type of stone, the composition of the stone, etc.

[0033] Data from a third data source 204 can be input into a third algorithm 210. The data from the third data source 204 can include data from one or more sensors included in or connected to the medical device such as a temperature sensor, a pressure sensor, a flow rate sensor, or the like. Additionally, or alternatively, data from the third data source 204 can include data specific to the patient, such as data from a patient file. The data from the patient file can include data such as the age, sex, weight, or other relevant physical characteristics of the patient. The data from the patient file can further include information regarding the medical history of the patient such as prior medical procedures, pre-existing conditions, allergies, blood type, blood antibodies, reaction to anesthesia, or the like. The data from the patient file can also include clinical record reports that may include information such as strictures noted, irregular anatomy tissue calcifications, or the like that may need to be treated or checked during the medical procedure or in a follow up procedure. In an example, the historical data can include data regarding people in the same or similar or substantially the same or similar demographic class with the patient Attorney Docket No.5409.873WO1 12 Client Reference No. GAP24019-URKT-WO1(e.g., people of the similar age, similar sex, or the like), or with similar medical condition(s) as the patient.

[0034] Outputs from the first algorithm 206, second algorithm 208, and third algorithm 210 can be used as inputs to a fourth algorithm 212. The fourth algorithm 212 can be a recommendation algorithm to analyze the outputs of the first algorithm 206, second algorithm 208, and third algorithm 210 to generate an output 214. For example, based on the endoscopic video or images, the first algorithm 206 can generate an endoscopic image of the surgical site with targets identified and the second algorithm 208 can provide analysis of the targets based on the spectral analysis (as discussed above). Based on the outputs from the first algorithm 206 and the second algorithm 208 the fourth algorithm 212 can compare the colors on the endoscopic image with the colors determined during spectral analysis and calibrate or adjust the colors on the endoscopic image to create a calibrated endoscopic image of the surgical site. The calibrated endoscopic image can represent a "true color" image of the surgical site.

[0035] In an example, the output 214 can include an annotated version of the calibrated endoscopic image of the surgical site. The annotated endoscopic image can identify targets in the endoscopic image and provide details about the targets such as size or composition. In another example, the calibrated endoscopic image can indicate non-targets, such as non-target tissue (e.g., muscle) that should not be contacted with laser or other ablation energy. The annotation can include information about a change to the target or a change to the surgical site. The change to the target or surgical site can be determined by comparing a first image obtained earlier in the same procedure, such as a change in the size or composition of a kidney stone as it is being ablated with a later image. Alternatively, the change can be determined from a first image obtained during an earlier, separate procedure with an image obtained during the current procedure. For example, the system can identify new targets during the current procedure that were not present in a prior procedure, indicate that a mass has increased or decreased in size, or the like. The annotated image can be displayed on a graphical user interface such as a monitor in the procedure room.

[0036] The output 214 can include a warning or an alert. For example, during a laser ablation procedure when one of the algorithms determines a change in composition of the target has occurred, that the laser is contacting non-target Attorney Docket No.5409.873WO1 13 Client Reference No. GAP24019-URKT-WO1tissue, or any other condition that can affect whether laser ablation should continue (or continue at the current intensity) or whether the tip of the scope should be moved or repositioned, the fourth algorithm 212 can cause an alert to be output. The alert can be a visual alert such as the endoscopic light blinking or changing color, can be an audible alert such as a beep, or can be a haptic alert such as the handle of the endoscope vibrating. The alert can also be a message displayed on the graphical user interface. In an example, output 214 can cause a controller, such as controller 114, to adjust or stop the ablation energy, or can cause an actuator connected to the endoscope to reposition the tip of the scope based on a determination by the fourth algorithm 212.

[0037] FIG. 3 illustrates a method for identifying and classifying a target in an image during a medical procedure. The method 300 can include or comprise a number of Operations or Steps. The Operations described herein are examples only, and the method can omit one or more of the listed Operations, can repeat Operations, can include other Operations, or can execute the Operations concurrently, substantially simultaneously, or in another order, as appropriate or desired.

[0038] At 302, the method 300 can include receiving an image of a surgical site. The image can be captured from an imaging device, sensor, or apparatus connected to a medical device, for example, a camera located at or near the tip of an endoscope. At 304, the method 300 can include analyzing an image of the surgical site to locate a target (or targets) in the image of the surgical site. The analysis can be performed by a trained learning algorithm (e.g., an artificial intelligence or machine learning algorithm) to identify potential targets such as kidney stones in the surgical site. The analysis can include a computer vision technique such as image segmentation analysis, or any similar technique capable of object detection.

[0039] At 306, the method 300 can include emitting a signal (e.g., visible or non-visible light, a laser radiation, or the like) toward a target identified in the image received at 302. For example, the endoscope can include or be coupled to a source of visible light such as a Xenon-based light source or any similar light source capable of illuminating the surgical scene.

[0040] At 308, the method 300 can include collecting at least a portion of a signal reflected or scattered by the target in response to the signal emitted in 306 Attorney Docket No.5409.873WO1 14 Client Reference No. GAP24019-URKT-WO1(hereinafter "response signal"), and 310 can include analyzing the collected response signal. The analysis of the response signal can include comparing one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site. In an example, when the scene is illuminated with an illumination light, an optical sensor or optical detector such as a spectrometer can be used to analyze the spectra of the illumination light or the response signal. The light analyzed by the spectrometer can include light reflected off of or scattered by tissue, targets, or other objects in the surgical scene. The analysis can further include calibrating or adjusting the one or more colors on the image of the surgical site based on or using the spectral analysis of the surgical site to create a calibrated endoscopic image (e.g., a true color image) of the surgical site.

[0041] At 312, the method 300 can include providing an output. In an example, the output can include a warning, such as a visual, audible, or haptic warning. The warning can be in response to a portion of the scope (e.g., the tip) being in an undesirable position or laser energy being directed toward non-target tissue, or the like. Additionally, or alternatively, the output can include a prediction or a recommendation. The recommendation can include a recommendation concerning a subsequent medical procedure. Such a recommendation can include a type of medical scope to use during the subsequent medical procedure, a type and intensity of ablation energy (e.g., laser radiation) to use during the subsequent medical procedure, a type or amount of anesthesia to use during the subsequent medical procedure, or the like. The recommendation can also include information about a risk to the patient in a subsequent medical procedure such as a risk of blood pressure dropping, blood type antibody allergies, or the like. The prediction can include whether a target is likely to be cancerous, or the like.

[0042] The method 300 can be a computer-implemented method that uses or employs different algorithms for carrying out the Operations of the method. For example, the image of the surgical site can be analyzed using a first algorithm and identification or classification of the target can be made using a second algorithm. In such an example, an output from the first algorithm (the analyzed image of the surgical site) and an output of the second algorithm (the identification or classification of the target) can be used as inputs to a Attorney Docket No.5409.873WO1 15 Client Reference No. GAP24019-URKT-WO1recommendation algorithm. The recommendation algorithm can, in turn, produce the output at 312. These algorithms can include an artificial intelligence (AI) or machine learning (ML) or other algorithm (e.g., a non-AI or non-ML deterministic algorithm) or process. Additionally, or alternatively, portions of the method 300 can be performed using a hardware-based feedback loop or feedback control. The hardware-based feedback loop or feedback control, the AI or ML algorithms, and the non-AI or non-ML algorithms can be used separately or in conjunction with each other as appropriate or desired.

[0043] FIG. 4 is a block diagram of an example of a machine 400 that can be used to help perform one or more of the techniques (e.g., methodologies) discussed herein. The machine 400 can operate as a standalone device or can be connected (e.g., networked) to other machines. For example, the machine 400 can be included in or connected to the medical device (e.g., an endoscope, a surgical laser, a surgical fiber, or the like) or can be included on or connected to one or more sensors can include components of the system. Additionally, or alternatively, the machine 400 can operate one or more of the algorithms discussed above, or the computer-based clinical decision support system (CDSS) discussed below with respect to FIG. 5. In a networked deployment, the machine 400 can operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 400 can act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 400 can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” can include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0044] Examples, as described herein, can include, or can operate by, logic or a number of components, or mechanisms. Circuit sets are a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership can be flexible over time and Attorney Docket No.5409.873WO1 16 Client Reference No. GAP24019-URKT-WO1underlying hardware variability. Circuit sets include members that can, alone or in combination, perform specified operations when operating. In an example, hardware of the circuit set can be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set can include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively coupled to the other components of the circuit set member when the device is operating. In an example, any of the physical components can be used in more than one member of more than one circuit set. For example, under operation, execution units can be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0045] Machine 400 (e.g., computer system) can include a hardware processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, field programmable gate array (FPGA), or any combination thereof), a main memory 404 and a static memory 406, some or all of which can communicate with each other via an interlink (e.g., bus) 430. The machine 400 can further include a display unit 410, an alphanumeric input device 412 (e.g., a keyboard), and a user interface (UI) navigation device 414 (e.g., a mouse). In an example, the display unit 410, input device 412 and UI navigation device 414 can be a touch screen display. The machine 400 can additionally include a storage device 408 (e.g., drive unit), a signal generation device 418 (e.g., a speaker), a network interface device 420, and one or more sensors 416, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. The machine 400 can include an output controller 428, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or Attorney Docket No.5409.873WO1 17 Client Reference No. GAP24019-URKT-WO1wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0046] The storage device 408 can include a machine readable medium 422 (e.g., a non-transitory medium) on which is stored one or more sets of data structures or instructions 424 (e.g., software) embodying or used by any one or more of the techniques or functions described herein. The instructions 424 can also reside, completely or at least partially, within the main memory 404, within static memory 406, or within the hardware processor 402 during execution thereof by the machine 400. In an example, one or any combination of the hardware processor 402, the main memory 404, the static memory 406, or the storage device 408 can constitute machine readable media.

[0047] While the machine readable medium 422 is illustrated as a single medium, the term "machine readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 424. The term “machine readable medium” can include any non-transitory medium that is capable of storing, encoding, or carrying instructions for execution by the machine 400 and that cause the machine 400 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non- limiting machine readable medium examples can include solid-state memories, and optical and magnetic media. In an example, a massed machine readable medium comprises a machine readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed machine-readable media are not transitory propagating signals. Specific examples of massed machine readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD- ROM disks.

[0048] FIG. 5 shows a schematic diagram of an exemplary computer-based clinical decision support system (CDSS) 500 that is configured to locate targets Attorney Docket No.5409.873WO1 18 Client Reference No. GAP24019-URKT-WO1in an image and identify or classify targets based on a spectral analysis of the image. The CDSS 500 can be further configured to output an alert, a recommendation, or a prediction. The recommendation can include recommended changed settings (e.g., changed laser settings, changed scope position, or the like) or a recommendation for a subsequent medical procedure. The recommendation for the subsequent medical procedure or the changed settings can be based on information about a target being treated, a piece or portion of tissue being treated, patient information, conditions encountered during the medical procedure, or a combination thereof. In various embodiments, the CDSS 500 can include an input interface 502 through which one or more inputs such as an image of the surgical site, which are specific to a patient, are provided as input features to an artificial intelligence (AI) model 504, a processor, such as processor 402, can perform an inference operation in which the information regarding the target, a piece or portion of tissue, the patient, the medical procedure, or any other appropriate or desired inputs are applied to the AI model to generate the warning, recommendation, or prediction and a user interface (UI) through which the warning, recommendation, or prediction can be communicated to a user, e.g., a clinician.

[0049] In some embodiments, the input interface 502 can be a direct data link between the CDSS 500 and one or more medical devices or sensors that generate at least some of the input features. For example, the input interface 502 can transmit information about the target, such as composition or density of the target, directly to the CDSS 500 during a therapeutic and / or diagnostic medical procedure. The information about the target can be determined using spectral analysis of a signal from the target, as discussed above. Additionally, or alternatively, the input interface 502 can be a classical user interface that facilitates interaction between a user and the CDSS 500. For example, the input interface 502 can facilitate a user interface through which the user can manually information about the patient or information about the medical procedure. Additionally, or alternatively, the input interface 502 can provide the CDSS 500 with access to an electronic patient record from which one or more input features can be extracted. The patient record can be stored in a database 506 coupled to the CDSS 500. In any of these cases, the input interface 502 can be configured to Attorney Docket No.5409.873WO1 19 Client Reference No. GAP24019-URKT-WO1collect one or more of the following input features in association with a specific patient on or before a time at which the CDSS 500 is used to assess:

[0050] An image of the surgical site 510, which can include regarding target location, information regarding the size of the target, or information about non- target tissue can be obtained from an imaging sensor such as an endoscopic camera.

[0051] Information from spectral analysis of the surgical site 512. The information can include target characteristics and can be used in conjunction with the image of the target captured by the imaging sensor of the scope;

[0052] An environmental condition measured by one or more sensors at or near the scope;

[0053] A location of the laser fiber or scope;

[0054] Information about the medical procedure;

[0055] An indication of non-target tissue;

[0056] Whether a target or tissue is normal or abnormal;

[0057] Information about the patient, such as patient history; and / or

[0058] Information about a condition of the patient during the medical procedure.

[0059] Based on one or more of the above input features, the processor 402 can perform an inference operation using the AI model to generate an output. The output can include an alert or warning. The output can include a change in one or more laser settings to be implemented or a recommended change in one or more laser settings to be proposed to the user. The output can include a recommendation for a subsequent medical procedure. The AI model can provide a computer system the ability to perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. The AI model can explore the study and construction of algorithms (e.g., machine-learning algorithms) that can learn from existing data and make predictions about new data. Such algorithms operate by building an AI model from example training data in order to make data-driven predictions or decisions expressed as outputs or assessments.

[0060] Two example modes for machine learning (ML) include: supervised ML and unsupervised ML. Supervised ML uses prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between Attorney Docket No.5409.873WO1 20 Client Reference No. GAP24019-URKT-WO1the inputs and the outputs. The goal of supervised ML is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs so that the ML model can implement the same relationships when given inputs to generate the corresponding outputs. Unsupervised ML is the training of an ML algorithm using information that is neither classified nor labeled and allowing the algorithm to act on that information without guidance. Unsupervised ML is useful in exploratory analysis because it can automatically identify structure in data.

[0061] Certain tasks for supervised ML can include classification problems and regression problems. Classification problems, also referred to as categorization problems, can aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms can aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of supervised-ML algorithms include Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM).

[0062] Some tasks for unsupervised ML can include clustering, representation learning, and density estimation. Some examples of unsupervised-ML algorithms are K-means clustering, principal component analysis, and autoencoders.

[0063] Another type of ML is federated learning (also referred to as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to centralized machine-learning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a common, robust machine learning model without sharing data, thus allowing to address critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0064] In some examples, the AI model can be trained continuously or periodically prior to performance of the inference operation by the processor 402. Then, during the inference operation, the patient specific input features provided to the AI model can be propagated from an input layer, through one or Attorney Docket No.5409.873WO1 21 Client Reference No. GAP24019-URKT-WO1more hidden layers, and ultimately to an output layer that corresponds to the changes to one or more of the laser settings, one or more predictions or recommendations, identification and / or classification of the target, or the like. For example, based on the target characteristics, the CDSS 500 can identify the target as a kidney stone formed of a specific material such as uric acid. Based on the identification and classification of the stone, the CDSS can recommend a setting for an ablation energy source such as a laser. Throughout the procedure, the characteristics of the stone can change (e.g., a change in size, a change in material composition, etc.) Based on the characteristics of the stone and a determined change in the target characteristics (e.g., a change in composition of the stone), the CDSS 500 can recommend a change in the laser settings such as laser intensity and / or frequency can be propagated to the output layer, a change to irrigation and / or suction settings, or make any other recommendation relevant to the procedure. In another example, the CDSS can make a prediction such as the amount of time required to finish the procedure, or that the temperature or pressure at the treatment site can increase within a certain period of time. Additionally, or alternatively, the system can make a recommendation to be implemented for a subsequent medical procedure. Such a recommendation can be based, at least in part on, a condition or occurrence during the current medical procedure. For example, if a patient response negatively to a type of anesthetic used during the procedure, the system can recommend a different type of anesthesia to be used in a subsequent procedure.

[0065] During or subsequent to the inference operation, the recommendation can be communicated to the user via a user interface (UI) and / or cause the processor 402 to automatically adjust a setting (e.g., a laser setting, an irrigation flow rate, a suction rate, or the like) of the medical device. Attorney Docket No.5409.873WO1 22 Client Reference No. GAP24019-URKT-WO1ADDITIONAL NOTES AND EXAMPLES:

[0066] Example 1 is a system for identifying and classifying a target in an image obtained during a medical procedure, the system comprising: a medical device; and processing circuitry, connected to the medical device, the processing circuitry to: receive an image of a surgical site; analyze the image of the surgical site to locate a target in the image of the surgical site; cause light to be emitted toward the target from a light source associated with the medical device; collect at least a portion of the emitted light reflected back from or scattered by the target at an optical splitter associated with the medical device; analyze the collected light using an optical sensor coupled to the optical splitter to at least one of identify or classify the target; and based on at least one of the identification or the classification of the target, provide an output to a user.

[0067] In Example 2, the subject matter of Example 1 optionally includes subject matter wherein the medical device is an endoscope and the image of the surgical site is captured by an endoscopic camera, and wherein the processing circuitry is to: compare one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site; and calibrate or otherwise adjust the one or more colors on the image of the surgical site based on spectral analysis of the surgical site to create a calibrated endoscopic image of the surgical site.

[0068] In Example 3, the subject matter of Example 2 optionally includes subject matter wherein the processing circuitry is to: output the calibrated endoscopic image of the surgical site on a graphical user interface.

[0069] In Example 4, the subject matter of any one or more of Examples 2–3 optionally include subject matter wherein the processing circuitry is to: annotate the calibrated endoscopic image of the surgical site.

[0070] In Example 5, the subject matter of Example 4 optionally includes subject matter wherein to annotate the calibrated endoscopic image of the surgical site includes annotating the calibrated endoscopic image of the surgical site with at least one of information about the target, information about the surgical site, identification of non-target anatomy, information about a change to the target, or information about a change to the surgical site.

[0071] In Example 6, the subject matter of any one or more of Examples 1–5 optionally include subject matter wherein the image of the surgical site is Attorney Docket No.5409.873WO1 23 Client Reference No. GAP24019-URKT-WO1analyzed using a first algorithm and identification or classification of the target is made using a second algorithm, wherein an output form the first algorithm and an output of the second algorithm are used as inputs to a recommendation algorithm.

[0072] In Example 7, the subject matter of Example 6 optionally includes subject matter wherein the processing circuitry is to: receive data from one or more additional sources; and input the data from the one or more additional sources into the recommendation algorithm, and wherein the output is based at least in part on an analysis of the data from the one or more additional sources.

[0073] In Example 8, the subject matter of Example 7 optionally includes subject matter wherein the processing circuitry is to: cause an adjustment of the medical device based at least in part on one or more of the data from the one or more additional sources or the identification or classification of the target.

[0074] In Example 9, the subject matter of any one or more of Examples 7–8 optionally include subject matter wherein the data from the one or more additional sources includes historical data pertaining to a patient undergoing the medical procedure, historical data pertaining to a prior medical procedure undergone by member of a demographic class to which the patient belongs, or real-time data obtained from a sensor coupled to or connected to the medical device.

[0075] In Example 10, the subject matter of Example 9 optionally includes subject matter wherein the historical data pertaining to the patient includes information about a prior medical procedure, a pre-existing condition of the patient, or a physical characteristic of the patient.

[0076] In Example 11, the subject matter of any one or more of Examples 1– 10 optionally include subject matter wherein the output includes a recommendation, wherein the recommendation includes a recommendation regarding a subsequent medical procedure, and wherein the recommendation regarding a subsequent medical procedure includes one or more of: a type of medical scope to use during the subsequent medical procedure, a type and intensity of ablation energy to use during the subsequent medical procedure, a type and amount of anesthesia to use during the subsequent medical procedure, or a risk during the subsequent medical procedure. Attorney Docket No.5409.873WO1 24 Client Reference No. GAP24019-URKT-WO1

[0077] In Example 12, the subject matter of any one or more of Examples 1– 11 optionally include subject matter wherein the processing circuitry is further to: determine one or more morphological characteristics of the target; and use the determined one or more morphological characteristics in conjunction with the analysis of the collected light by the optical sensor to at least one of identify or classify the target.

[0078] Example 13 is a computer implemented method for identifying and classifying a target in an image obtained during a medical procedure, comprising: receiving an image of a surgical site from an optical sensor connected to a medical device used during the medical procedure; analyzing the image using a first algorithm; emitting a signal toward a target from a light source connected to the medical device; collecting at least a portion of a response signal reflected back from or scattered by the target in response to the emitted signal at an optical splitter connected to the medical device; analyzing the collected response signal using an optical sensor coupled to the optical splitter to at least one of identify or classify the target; and providing an output to a user based at least on part on the identification or classification of the target.

[0079] In Example 14, the subject matter of Example 13 optionally includes subject matter comparing one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site; and calibrating or otherwise adjusting the one or more colors on the image of the surgical site based on spectral analysis of the surgical site to create a calibrated endoscopic image of the surgical site.

[0080] In Example 15, the subject matter of Example 14 optionally includes subject matter annotating the calibrated endoscopic image of the surgical site, wherein annotating the calibrated endoscopic image of the surgical site includes annotating the calibrated endoscopic image of the surgical site with at least one of information about the target, information about the surgical site, identification of non-target anatomy, information about a change to the target, or information about a change to the surgical site; and outputting the calibrated endoscopic image of the surgical site on a graphical user interface.

[0081] In Example 16, the subject matter of any one or more of Examples 13– 15 optionally include subject matter wherein the image of the surgical site is analyzed using a first algorithm and identification or classification of the target Attorney Docket No.5409.873WO1 25 Client Reference No. GAP24019-URKT-WO1is made using a second algorithm, wherein an output form the first algorithm and an output of the second algorithm are used as inputs to a recommendation algorithm, and wherein the method further comprises: receiving data from one or more additional sources, wherein the data from the one or more additional sources includes historical data pertaining to a patient undergoing the medical procedure, historical data pertaining to a prior medical procedure undergone by member of a demographic class to which the patient belongs, or real-time data obtained from a sensor coupled to or connected to the medical device; inputting the data from the one or more additional sources into the recommendation algorithm, and wherein the output is based at least in part on an analysis of the data from the one or more additional sources; and causing an adjustment of the medical device based at least in part on one or more of the data from the one or more additional sources or the identification or classification of the target.

[0082] Example 17 is a non-transitory computer-readable medium with instructions stored thereon, that when performed by a processor of a computing device connected to a medical device, cause the processor to: receive an image of a surgical site; analyze the image of the surgical site to locate a target in the image of the surgical site; cause light to be emitted toward the target from a light source associated with the medical device; collect at least a portion of the emitted light reflected back from or scattered by the target at an optical splitter associated with the medical device; analyze the collected light using an optical sensor coupled to the optical splitter to at least one of identify or classify the target; and based on at least one of the identification or the classification of the target, provide an output to a user.

[0083] In Example 18, the subject matter of Example 17 optionally includes subject matter wherein the instructions cause the processor to: compare one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site; calibrate or otherwise adjust the one or more colors on the image of the surgical site based on spectral analysis of the surgical site to create a calibrated endoscopic image of the surgical site; annotate the calibrated endoscopic image of the surgical site; output the calibrated endoscopic image of the surgical site on a graphical user interface; receive data from one or more additional sources; and input the data from the one or more additional sources into a recommendation algorithm, and wherein the output is Attorney Docket No.5409.873WO1 26 Client Reference No. GAP24019-URKT-WO1based at least in part on an analysis of the data from the one or more additional sources.

[0084] In Example 19, the subject matter of Example 18 optionally includes subject matter wherein the image of the surgical site is analyzed using a first algorithm and identification or classification of the target is made using a second algorithm, wherein an output form the first algorithm and an output of the second algorithm are used as inputs to a recommendation algorithm.

[0085] In Example 20, the subject matter of Example 19 optionally includes subject matter wherein the output includes a recommendation regarding a subsequent medical procedure, and wherein the recommendation regarding a subsequent medical procedure includes one or more of: a type of medical scope to use during the subsequent medical procedure, a type and intensity of ablation energy to use during the subsequent medical procedure, a type and amount of anesthesia to use during the subsequent medical procedure, or a risk during the subsequent medical procedure.

[0086] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments that can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0087] All publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls. Attorney Docket No.5409.873WO1 27 Client Reference No. GAP24019-URKT-WO1

[0088] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the term “and / or” is used to refer to a nonexclusive or, such that “A and / or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0089] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) can be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is to allow the reader to quickly ascertain the nature of the technical disclosure and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features can be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter can lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment. The scope of the embodiments should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. Attorney Docket No.5409.873WO1 28 Client Reference No. GAP24019-URKT-WO1

Claims

CLAIMS WHAT IS CLAIMED IS:

1. A system for identifying and classifying a target in an image obtained during a medical procedure, the system comprising: a medical device; and processing circuitry, connected to the medical device, the processing circuitry to: receive an image of a surgical site; analyze the image of the surgical site to locate a target in the image of the surgical site; cause light to be emitted toward the target from a light source associated with the medical device; collect at least a portion of the emitted light reflected back from or scattered by the target at an optical splitter associated with the medical device; analyze the collected light using an optical sensor coupled to the optical splitter to at least one of identify or classify the target; and based on at least one of the identification or the classification of the target, provide an output to a user.

2. The system of claim 1, wherein the medical device is an endoscope and the image of the surgical site is captured by an endoscopic camera, and wherein the processing circuitry is to: compare one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site; and calibrate or otherwise adjust the one or more colors on the image of the surgical site based on spectral analysis of the surgical site to create a calibrated endoscopic image of the surgical site.

3. The system of claim 2, wherein the processing circuitry is to: output the calibrated endoscopic image of the surgical site on a graphical user interface. Attorney Docket No.5409.873WO1 29 Client Reference No. GAP24019-URKT-WO14. The system of claim 2, wherein the processing circuitry is to: annotate the calibrated endoscopic image of the surgical site.

5. The system of claim 4, wherein to annotate the calibrated endoscopic image of the surgical site includes annotating the calibrated endoscopic image of the surgical site with at least one of information about the target, information about the surgical site, identification of non-target anatomy, information about a change to the target, or information about a change to the surgical site.

6. The system of claim 1, wherein the image of the surgical site is analyzed using a first algorithm and identification or classification of the target is made using a second algorithm, wherein an output form the first algorithm and an output of the second algorithm are used as inputs to a recommendation algorithm.

7. The system of claim 6, wherein the processing circuitry is to: receive data from one or more additional sources; and input the data from the one or more additional sources into the recommendation algorithm, and wherein the output is based at least in part on an analysis of the data from the one or more additional sources.

8. The system of claim 7 , wherein the processing circuitry is to: cause an adjustment of the medical device based at least in part on one or more of the data from the one or more additional sources or the identification or classification of the target.

9. The system of claim 7, wherein the data from the one or more additional sources includes historical data pertaining to a patient undergoing the medical procedure, historical data pertaining to a prior medical procedure undergone by member of a demographic class to which the patient belongs, or real-time data obtained from a sensor coupled to or connected to the medical device. Attorney Docket No.5409.873WO1 30 Client Reference No. GAP24019-URKT-WO110. The system of claim 9, wherein the historical data pertaining to the patient includes information about a prior medical procedure, a pre-existing condition of the patient, or a physical characteristic of the patient.

11. The system of claim 1, wherein the output includes a recommendation, wherein the recommendation includes a recommendation regarding a subsequent medical procedure, and wherein the recommendation regarding a subsequent medical procedure includes one or more of: a type of medical scope to use during the subsequent medical procedure, a type and intensity of ablation energy to use during the subsequent medical procedure, a type and amount of anesthesia to use during the subsequent medical procedure, or a risk during the subsequent medical procedure.

12. The system of claim 1, wherein the processing circuitry is further to: determine one or more morphological characteristics of the target; and use the determined one or more morphological characteristics in conjunction with the analysis of the collected light by the optical sensor to at least one of identify or classify the target .

13. A computer implemented method for identifying and classifying a target in an image obtained during a medical procedure, comprising: receiving an image of a surgical site from an optical sensor connected to a medical device used during the medical procedure; analyzing the image using a first algorithm; emitting a signal toward a target from a light source connected to the medical device; collecting at least a portion of a response signal reflected back from or scattered by the target in response to the emitted signal at an optical splitter connected to the medical device; analyzing the collected response signal using an optical sensor coupled to the optical splitter to at least one of identify or classify the target; and providing an output to a user based at least on part on the identification or classification of the target. Attorney Docket No.5409.873WO1 31 Client Reference No. GAP24019-URKT-WO114. The method of claim 13, comprising: comparing one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site; and calibrating or otherwise adjusting the one or more colors on the image of the surgical site based on spectral analysis of the surgical site to create a calibrated endoscopic image of the surgical site.

15. The method of claim 14, comprising: annotating the calibrated endoscopic image of the surgical site, wherein annotating the calibrated endoscopic image of the surgical site includes annotating the calibrated endoscopic image of the surgical site with at least one of information about the target, information about the surgical site, identification of non-target anatomy, information about a change to the target, or information about a change to the surgical site; and outputting the calibrated endoscopic image of the surgical site on a graphical user interface.

16. The method of claim 13, wherein the image of the surgical site is analyzed using a first algorithm and identification or classification of the target is made using a second algorithm, wherein an output form the first algorithm and an output of the second algorithm are used as inputs to a recommendation algorithm, and wherein the method further comprises: receiving data from one or more additional sources, wherein the data from the one or more additional sources includes historical data pertaining to a patient undergoing the medical procedure, historical data pertaining to a prior medical procedure undergone by member of a demographic class to which the patient belongs, or real-time data obtained from a sensor coupled to or connected to the medical device; inputting the data from the one or more additional sources into the recommendation algorithm, and wherein the output is based at least in part on an analysis of the data from the one or more additional sources; and causing an adjustment of the medical device based at least in part on one or more of the data from the one or more additional sources or the identification or classification of the target. Attorney Docket No.5409.873WO1 32 Client Reference No. GAP24019-URKT-WO117. A non-transitory computer-readable medium with instructions stored thereon, that when performed by a processor of a computing device connected to a medical device, cause the processor to: receive an image of a surgical site; analyze the image of the surgical site to locate a target in the image of the surgical site; cause light to be emitted toward the target from a light source associated with the medical device; collect at least a portion of the emitted light reflected back from or scattered by the target at an optical splitter associated with the medical device; analyze the collected light using an optical sensor coupled to the optical splitter to at least one of identify or classify the target; and based on at least one of the identification or the classification of the target, provide an output to a user.

18. The non-transitory computer-readable medium of claim 17, wherein the instructions cause the processor to: compare one or more colors on the image of the surgical site with one or more colors determined during spectral analysis of the surgical site; calibrate or otherwise adjust the one or more colors on the image of the surgical site based on spectral analysis of the surgical site to create a calibrated endoscopic image of the surgical site; annotate the calibrated endoscopic image of the surgical site; output the calibrated endoscopic image of the surgical site on a graphical user interface; receive data from one or more additional sources; and input the data from the one or more additional sources into a recommendation algorithm, and wherein the output is based at least in part on an analysis of the data from the one or more additional sources.

19. The non-transitory computer-readable medium of claim 18, wherein the image of the surgical site is analyzed using a first algorithm and identification or Attorney Docket No.5409.873WO1 33 Client Reference No. GAP24019-URKT-WO1classification of the target is made using a second algorithm, wherein an output form the first algorithm and an output of the second algorithm are used as inputs to a recommendation algorithm.

20. The non-transitory computer-readable medium of claim 19, wherein the output includes a recommendation regarding a subsequent medical procedure, and wherein the recommendation regarding a subsequent medical procedure includes one or more of: a type of medical scope to use during the subsequent medical procedure, a type and intensity of ablation energy to use during the subsequent medical procedure, a type and amount of anesthesia to use during the subsequent medical procedure, or a risk during the subsequent medical procedure. Attorney Docket No.5409.873WO1 34 Client Reference No. GAP24019-URKT-WO1

Images