Bodily lumen injury prevention system

Abstract

Injury to a bodily lumen, such as a ureter, during a surgical procedure is prevented by tracking the locations of surgical instruments relative to the location of the bodily lumen. Image data that depict the bodily lumen are received. A marker instrument is introduced into the bodily lumen. The marker instrument has fluorescent and non-fluorescent markers, where a geometry of each non-fluorescent marker is known. Based on the image data, the location of the bodily lumen is determined from measured fluorescence signals of the fluorescent markers and a position of the marker instrument is determined by calculating a distance between the non-fluorescent markers and the imaging system. A location of a second surgical instrument relative to the first surgical instrument is then determined based on a calculated separation distance between the position of the first surgical instrument and a position of the second surgical instrument. Surgical navigation data are generated and output with the computer system. The surgical navigation data may be generated at least in part based on the calculated separation distance.

Description

BODILY LUMEN INJURY PREVENTION SYSTEMBACKGROUNDCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Application Serial No. 63 / 642,043, filed May 3, 2024, the entirety of which is incorporated herein by reference.

[0002] Medical errors are the third leading cause of death, with more than 250,000 people in the United States dying every year due to medical mistakes. Iatrogenic injury is a serious complication that can occur during various abdominal or pelvic operations. The overall incidence of significant ureteral injury in gynecologic and colorectal surgeries is 1.7% and 1.9%, respectively. The risk of injury is higher in cases with a complex history of prior surgery, radiation, inflammatory conditions, or mass obscuring anatomic planes. In such cases, ureteral identification is more challenging, even when traditional measures such as ureteral stenting are employed to improve the ability to localize the ureter intraoperatively.

[0003] Furthermore, the postoperative course can be challenging due to the common need for urinary diversion using different methods such as ureteral stents, nephrostomy tubes, or urethral catheters, which can cause more pain, irritative urinary tract symptoms, and predisposition to urinary infections, among other complications that significantly impact a patient's quality of life, especially with late detection of injury. Eight to fifty-seven percent of all ureteral injuries are recognized late, and the interval from injury to recognition is important and guides management, reflecting the importance of intraoperative detection. All these factors, among others, signify the severe impact on patient safety and subsequent quality of life, as seen from prolonged hospitalization, the need for more surgical interventions, among many other issues.SUMMARY OF THE DISCLOSURE

[0004] It is an aspect of the present disclosure to provide a method for tracking a surgical instrument. The method includes receiving image data with a computer system, where the image data includes an image acquired with an imaging system. The image depicts a subject containing one or more bodily lumens into which one or more first surgical instruments have been introduced. The first surgical instrument includes a fluorescent portion and a non-fluorescent portion, where a geometry of the non-fluorescent portion is known. A location of the bodily lumen is determined based on a measured fluorescence signal of the fluorescent portion of the first surgical instrument in the image data. A position of the first surgical instrument is determined by calculating, from theimage data, a distance between the non-fluorescent portion of the first surgical instrument and the imaging system. A location of a second surgical instrument relative to the first surgical instrument is then determined based on measuring a separation distance between the position of the first surgical instrument and a position of the second surgical instrument. Surgical navigation data are then output with the computer system, thereby providing guidance of the second surgical instrument relative to the bodily lumen. The surgical navigation data are generated based on the separation distance. Other embodiments of this aspect include corresponding systems (e.g., computer systems), programs, algorithms, and / or modules, each configured to perform the steps of the methods.

[0005] It is an aspect of the present disclosure to provide a stent for insertion into a bodily lumen. The stent includes a flexible body extending along a length from a first end to a second end. The flexible body includes a plurality of fluorescent portions and a plurality of non- fluorescent portions. The plurality of non-fluorescent portions are uniformly spaced apart along the length of the flexible body and interleaved with the plurality of fluorescent portions. Each of the plurality of non-fluorescent portions have a common geometry. The stent also includes a lumen extending through the flexible body from the first end to the second end.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 A is an example of a marker instrument that can be inserted into a bodily lumen to provide visualization and localization of the bodily lumen during a surgical procedure.

[0007] FIG. IB shows a section of the marker instrument shown in FIG. 1A.

[0008] FIG. 2 illustrates a push catheter with alternating fluorescent and non-fluorescent markers along its length.

[0009] FIG. 3 is a flowchart of an example method for providing surgical navigation of one or more surgical instruments to prevent injury to a bodily lumen, such as a ureter, during a surgical procedure.

[0010] FIG. 4 is a block diagram of an example system for preventing injury to a bodily lumen, such as a ureter, during a surgical procedure.

[0011] FIG. 5 is a block diagram of example components that can implement the system of FIG. 4.

[0012] FIG. 6 shows a sectional view of a subject with a ureteral stent positioned within a ureter between a kidney and bladder.

[0013] FIGS. 7A-7D illustrate different views of a surgical site during a procedure, showing instrument positioning and surgical display outputs.DETAILED DESCRIPTION

[0014] Described here are systems and methods for preventing injury to a bodily lumen, or other anatomical structure, of a subject during laparoscopic and / or robotic surgical procedures. As one non-limiting example, the bodily lumen may be a ureter. Alternatively, the bodily lumen may be a blood vessel, a renal tubule, or other bodily lumen. The subject may be a human subject undergoing a surgical procedure in a clinical setting, or may be an animal subject undergoing a surgical procedure in a veterinary setting.

[0015] In general, the systems and methods described in the present disclosure utilize an artificial intelligence (Al)-based technology to assist in intraoperatively preventing and detecting injury to a bodily lumen, which will help prevent iatrogenic injury to the bodily lumen. Visualization and / or localization of the bodily lumen can be enhanced using a fluorescent material to highlight the location of the bodily lumen in image data acquired from the subject. Based on measured fluorescence signals generated in response to exciting the fluorescent material with light from a light source, the location of the bodily lumen can be determined and visually enhanced to provide feedback to a surgeon to prevent injury to the bodily lumen during the surgical procedure.

[0016] As one example, the fluorescent material can be part of a surgical instrument introduced into the bodily lumen during the surgical procedure. In these instances, during the surgical procedure, a first surgical instrument is inserted into the bodily lumen while a second surgical instrument is used by the surgeon to perform a tissue dissection, tissue ablation, or other surgical procedure. The first surgical instrument may be, for example, a stent. Images of the field- of-view containing the bodily lumen are obtained during the surgical procedure. A fluorescent marker on the first surgical instrument is detected in the images and used to visualize the location of the bodily lumen during the surgical procedure. A non-fluorescent marker on the first surgical instrument is detected in the images and used to determine a location of the first surgical instrument relative to an external reference. The external reference may be the imaging system used to obtain the images, the second surgical instrument being used during the surgical procedure, or some other suitable external reference with a known position. The location of the non-fluorescent marker provides accurate guidance of the second surgical instrument relative to the bodily lumen, such that the bodily lumen can be avoided during the surgical procedure, thereby preventing iatrogenic injury to the subject.

[0017] As another example, the fluorescent material used to enhance visualization of the bodily lumen can be a fluorescent dye that is administered to the subject during the surgicalprocedure. In these instances, the fluorescent dye is administered to the subject and allowed to flow to the bodily lumen, such that the fluorescent dye is present in the bodily lumen while image data are acquired from the subject. The fluorescent dye is detected in the images and used to visualize the location of the bodily lumen during the surgical procedure.

[0018] The form of feedback and machine response provided by the disclosed systems and methods can be controlled by the surgeon. The location of surgical instruments relative to the bodily lumen will be continuously monitored during the surgical procedure based in part on the enhanced visualization of the bodily lumen described above. When a surgical instrument is determined as coming within a threshold distance (e.g., a critical safety distance) from the bodily lumen, an alert can be generated to the users, the movement of a robotic surgical system can be halted, or the like. Advantageously, the disclosed systems and methods can be used during laparoscopic surgery independent of any robotic platform.

[0019] In some applications, the systems and methods can be used with a robotic surgery platform. In these instances, the position of the bodily lumen can be determined from images of the surgical field. For instance, the location of the bodily lumen can be visualized based on fluorescence signals imaged from the fluorescent markers of the marker instrument and / or from a fluorescent dye administered to the subject. When a marker instrument is being used, the location of the marker instrument can be calculated based on images of the non-fluorescent markers of the marker instrument. Because the position of surgical instruments coupled to the robotic surgery system can be precisely known and tracked, the separation distance between those surgical tools and the position of the bodily lumen determined from the location of the marker instrument and / or bodily lumen can be calculated and used to prevent injury to the bodily lumen. For example, the calculated separation distance can be used to generate surgical navigation data to provide feedback to a user when a surgical instrument gets too close to the bodily lumen. The feedback data may be provide visual feedback (e.g., a visual indication of how close the surgical instrument is to the bodily lumen), haptic feedback, or the like. As one example, the visual feedback may be provided on a display in the operating room, or may be presented to a user via an augmented reality and / or extended reality (AR / XR) system. Additionally or alternatively, the surgical navigation data can include control signals sent to the robotic surgery system to control operation of the surgical instruments. For example, when a surgical instrument is determined to be within a safety margin around the bodily lumen, the robotic surgery system can halt further movement of the surgical instrument towards the bodily lumen.

[0020] In some other applications, the disclosed systems and methods can be used within a laparoscopic procedure. In these instances, the precise location of the laparoscopic surgicalinstrument(s) may not be precisely known. Here, a relative distance between a laparoscopic surgical instrument and the bodily lumen can be determined and used to prevent injury to the bodily lumen. For example, a tracking marker can be coupled to the laparoscopic instrument and used to monitor the relative position of the laparoscopic surgical instrument within the surgical field. The tracking marker may be, for example, a fluorescent marker that generates fluorescence signals when excited by light from a light source. These fluorescence signals allow for visualization of the laparoscopic surgical instrument within the surgical field. Images of the surgical field can thus be processed to determine the relative distance between the bodily lumen (e.g., as determined from the position of the marker instrument calculated from the images of the non-fluorescent marker) and the laparoscopic surgical instrument. For instance, the depth of the laparoscopic surgical instrument may not be precisely determinable based solely on the images of the surgical field, but the relative distance of the laparoscopic surgical instrument to the bodily lumen can be determined and used as a safety measure to prevent injury to the bodily lumen.

[0021] As in the robotic surgery case, surgical navigation data are generated by processing the images of the surgical field. The surgical navigation data can include the relative distance discussed above. Based on the relative distance, feedback data can be generated and presented to a user. The feedback data may include visual feedback (e.g., a visual indication of how close the surgical instrument is to the bodily lumen), haptic feedback, or the like. As one example, the visual feedback may be provided on a display in the operating room, or may be presented to a user via an AR / XR system.

[0022] It will be appreciated that the systems and methods can be used in a combination of robotic surgery and laparoscopic surgery applications, such as robotic surgery with laparoscopic assistance. The disclosed systems and methods can also be used with an AR / XR system to provide feedback to a user during the procedure, or may be used without such a system. In the latter instances, feedback can be provided to a user through other means, such as a display monitor in the operating room, or the like.

[0023] As described above, the systems and methods described in the present disclosure utilize the visualization and localization of the at-risk bodily lumen (e.g., a ureter) based on imaging the subject while a surgical instrument is arranged within the bodily lumen. This marker instrument may be a stent, catheter, or the like. An example of such a marker instrument is illustrated in FIGS. 1A and IB.

[0024] FIGS. 1A and IB show an example surgical instrument 100 that is constructed as a stent. The surgical instrument 100 includes a flexible body 112 (e.g., a flexible shaft) extending along a length from a first end 114 to a second end 116. A lumen 118 extends through the flexiblebody 112 along the length of the surgical instrument 100. The surgical instrument 100 includes one or more fluorescent markers 120 and one or more non-fluorescent markers 122 arranged on, or otherwise coupled to, an outer surface of the flexible body 112.

[0025] As an example, the fluorescent marker(s) 120 can include one or more portions of the flexible body 112 coated in a fluorescent coating, or otherwise composed of a material that generates fluorescence when excited with light from a light source. For instance, the fluorescent marker(s) 120 may include portions of the flexible body 112 that are impregnated with a fluorescent dye, or the like, such that those portions of the flexible body 112 may fluoresce when excited with light from a light source.

[0026] The non-fluorescent marker(s) 122 can include one or more portions of the flexible body 112 that are not coated in a fluorescent coating, or are otherwise not composed of a fluorescent material. Each non-fluorescent marker 122 has a known geometric profde (e.g., a known size, a known shape, etc.), such that the precise location of the non-fluorescent marker 122 may be determined from images of the subject in which the non-fluorescent marker 122 is visible.

[0027] The non-fluorescent marker(s) 122 are generally arranged along the length of the flexible body 112. As one example, the non-fluorescent marker(s) 122 may be portions of the flexible body 112 that are not coated in, or composed of, a fluorescent material. As another example, a non-fluorescent marker 122 can include an annular body having an inner diameter that receives the flexible body 112 of the surgical instrument 100, such that the annular body of the non-fluorescent marker 122 can be arranged over the surface of the flexible body 112. In these instances, the non-fluorescent markers 122 can be composed of a plastic or other biocompatible material. In some cases, the non-fluorescent markers 122 may comprise a plurality of annular shaped markers each having a central aperture, wherein the flexible body 112 extends through the central aperture of each of the plurality of annular shaped markers.

[0028] As will be described, the location of the non-fluorescent markers 122 can be determined based on a focal length equation to estimate the distance between the non-fluorescent markers 122 and the imaging system used to obtain the imaged of the subject.

[0029] As a non-limiting example, the non-fluorescent markers 122 can have a width of 1 cm. Alternatively, the non-fluorescent markers 122 can have other widths. When more than one surgical instrument 100 may be used during a surgical procedure to prevent injury to more than one bodily lumen (e.g., when two surgical instruments are used, one for each ureter), the non- fluorescent markers 122 on each surgical instrument 100 can be different to allow for differentiation between the different bodily lumens. As an example, the non-fluorescent markers 122 used on one surgical instrument 100 may have a geometric profile that is different from thegeometric profile of the non-fluorescent markers 122 used on the other surgical instrument. For instance, the non-fluorescent markers 122 on one surgical instrument 100 may have a first width and the non-fluorescent markers 122 on the other surgical instrument 100 may have a second width that is different from the first width. Additionally or alternatively, the non-fluorescent markers 122 may be arranged differently (e.g., with different spacings between non-fluorescent markers 122) along the length of the flexible bodies 112 of the different surgical instruments 100.

[0030] In some implementations, the surgical instrument 100 includes a plurality of non- fluorescent markers 122 arranged along the length of the flexible body 112, such that the entire length of the bodily lumen can be covered when the surgical instrument 100 is fully inserted into the bodily lumen. The non-fluorescent markers 122 can be arranged along the length of the flexible body 112 with uniform spacing between each non-fluorescent marker 122. As a non-limiting example, the non-fluorescent markers 122 can be spaced apart by 1 cm, 2 cm, 3 cm, 4 cm, or the like. Additionally or alternatively, the non-fluorescent markers 122 can be spaced apart by other distances, such as 0.5 cm, 1.0 cm, 1.5 cm, 2.0 cm, 2.5 cm, 3.0 cm, 3.5 cm, 4.0 cm, or the like.

[0031] FIG. 2 illustrates an example surgical instrument 100 comprising a push catheter 200. The push catheter 200 includes a flexible body 202 that extends along a length from a first end 114 to a second end 116. The flexible body 202 incorporates alternating non-fluorescent markers 204, 210, 218, 224 and fluorescent markers 206, 214, 220 or portions, similar to the arrangement seen in the surgical instrument 100.

[0032] The push catheter 200 may include a flexible body 202 that extends along a length from a first end 114 to a second end 116. In some cases, the flexible body 202 may be composed of a biocompatible material. For example, the flexible body 202 may be made of silicone, polyurethane, or other suitable polymers. In some implementations, the flexible body 202 may incorporate a braided reinforcement layer to enhance pushability and torque transmission while maintaining flexibility. The first end 114 of the push catheter 200 may include a connector for attaching to external devices or tubing. This connector could be a Luer lock fitting, allowing for secure connections to syringes, drainage bags, or other medical equipment. The second end 116, or distal end 208, of the push catheter 200 may be designed with an atraumatic shape to minimize tissue damage during insertion and navigation. For example, the distal end 208 could be rounded or tapered, and may be made of a softer durometer material compared to the rest of the flexible body 202.

[0033] The push catheter 200 incorporates a plurality of non-fluorescent markers 204, 210, 218, 224 and fluorescent markers 206, 214, 220 arranged along the length of the flexible body 202. A non-fluorescent marker 204 is positioned near one end of the flexible body 202, followed by afluorescent marker 206. Moving along the flexible body 202 toward a distal end 208, there is another non-fluorescent marker 210. A fluorescent marker length 212 separates adjacent non- fluorescent markers 204, 210. The fluorescent marker length 212 may be a portion of the flexible body 202 that is coated with or impregnated with a fluorescent material. For example, the fluorescent material may be a biocompatible fluorescent dye or compound that emits light when excited by an appropriate wavelength.

[0034] The push catheter 200 comprises a fluorescent marker 214, followed by another fluorescent marker length 216. A non-fluorescent marker 218 is positioned further along the flexible body 202, followed by a fluorescent marker 220. Near the opposite end of the flexible body 202, a non-fluorescent marker 224 is located. The non-fluorescent markers 204, 210, 218, 224 and fluorescent markers 206, 214, 220 are arranged in an alternating pattern along the length of the flexible body 202, with the fluorescent marker lengths 212, 216 providing spacing between adjacent non-fluorescent markers 204, 210, 218, 224.

[0035] The non-fluorescent markers 204, 210, 218, 224 may have distinct geometric profiles that serve different functions along the push catheter 200. For example, non-fluorescent markers 210 and 218 may comprise a plurality of thin stripe patterns with varying spacing arrangements that encode positional information along the length of the flexible body 202. In contrast, non- fluorescent marker 224 may be configured as a solid band with a wider profile at the distal end 208, providing a visual reference point for the terminal portion of the push catheter 200. Similarly, non-fluorescent marker 204 may be implemented as a solid marker with a distinctive width that demarcates the functional section of the flexible body 202. In some cases, the non-fluorescent markers 204, 210, 218, 224 may be manufactured using radio-opaque materials such as barium sulfate-impregnated polymers, allowing for visualization under multiple imaging modalities. These varied geometric configurations may enable precise three-dimensional localization of the push catheter 200 within a bodily lumen during a surgical procedure through triangulation calculations based on the known dimensions and relative positions of the markers.

[0036] The fluorescent markers 206, 214, 220 and fluorescent marker lengths 212, 216 may be designed to provide continuous visibility of the push catheter 200 when excited by an appropriate light source. The fluorescent material used may be selected based on its emission wavelength and intensity to ensure visibility during surgical procedures. For instance, the fluorescent material may emit light in the near-infrared spectrum, which may penetrate tissue more effectively than visible light.

[0037] The arrangement of non-fluorescent markers 204, 210, 218, 224 and fluorescent markers 206, 214, 220 along the flexible body 202 may serve multiple purposes. The fluorescentportions may allow for continuous visualization of the push catheter 200 within a bodily lumen, while the non-fluorescent markers 204, 210, 218, 224 may provide reference points for precise distance measurements. This combination may enable accurate localization of the push catheter 200 and, by extension, the bodily lumen in which the push catheter 200 is inserted.

[0038] A lumen may extend through the flexible body 202 from the first end to the second end, similar to the lumen described in relation to the surgical instrument 100. This lumen may allow for fluid flow or the insertion of other instruments through the push catheter 200.

[0039] The push catheter 200 may be inserted into a bodily lumen, such as a ureter, using standard medical insertion techniques. For example, the push catheter 200 may be guided into position using a guidewire and fluoroscopic imaging. Once in place, the alternating pattern of fluorescent markers 206, 214, 220 and non-fluorescent markers 204, 210, 218, 224 may allow for continuous monitoring of the bodily lumen's position during surgical procedures.

[0040] The push catheter 200 may be adapted for various medical procedures and anatomical contexts, designed for use in different bodily lumens. In some cases, the push catheter 200 may be designed for ureteral applications, featuring a flexible body 202 optimized for navigating the urinary tract, specifically the ureter. In other implementations, the push catheter 200 may be configured for cardiovascular procedures, with a flexible body 202 incorporating features for navigating blood vessels such as arteries and veins. The push catheter 200 may also be adapted for gastrointestinal applications, potentially featuring a longer flexible body 202 for reaching deep into the digestive tract, including the esophagus, stomach, and intestines. In some cases, the push catheter 200 may be designed for neurological interventions, with an extremely thin and flexible body 202 for use in delicate cerebral vasculature, such as cerebral arteries and veins. Additionally, the push catheter 200 may be configured for pulmonary procedures, with a flexible body 202 designed to navigate bronchial passages within the lungs. Each of these variations of the push catheter 200 may incorporate specific design elements tailored to their intended use in different bodily lumens, while maintaining the alternating non-fluorescent markers 204, 210, 218, 224 and fluorescent portions along the flexible body 202.

[0041] The push catheter 200 may be adapted for any suitable surgical site. In various surgical procedures performed near different bodily lumens, the fluorescent portions and non-fluorescent markers 204, 210, 218, 224 of the push catheter 200 may be visualized to aid in navigation and prevent injury to the lumen. Some examples include: for ureteral applications, during pelvic or abdominal surgeries, the alternating pattern of fluorescent portions and non-fluorescent markers 204, 210, 218, 224 may be visible through the thin walls of the ureter, allowing surgeons to identify and avoid the ureter while operating on nearby structures such as the ovaries or colon. Incardiovascular procedures, such as thoracic surgeries, the push catheter 200 within major blood vessels may be detectable through the vessel walls, helping surgeons navigate around vasculature while operating on lung tissue or mediastinal structures. For gastrointestinal applications, during laparoscopic abdominal surgeries, the fluorescent and non-fluorescent pattern of the push catheter 200 within the intestines may be visible through the intestinal walls, aiding in the identification and preservation of bowel structures while addressing pathologies in the abdominal cavity. In neurological interventions, such as brain tumor resections, the extremely thin push catheter 200 within cerebral arteries may be detectable through the vessel walls, assisting neurosurgeons in preserving blood supply while removing adjacent tumor tissue. For pulmonary procedures, during thoracoscopic surgeries, the push catheter 200 within bronchial passages may be visible through the bronchial walls, helping thoracic surgeons identify and protect airways while operating on lung parenchyma or pleural structures.

[0042] Referring now to FIG. 3, a flowchart is illustrated as setting forth the steps of an example method for providing surgical navigation of one or more surgical instruments to prevent injury to a bodily lumen, such as a ureter, during a surgical procedure.

[0043] The method includes receiving image data with a computer system, as indicated at step 302. Receiving the image data may include retrieving such data from a memory or other suitable data storage device or medium. Additionally or alternatively, receiving the image data may include acquiring such data with an imaging system and transferring or otherwise communicating the data to the computer system, which may be a part of the imaging system.

[0044] In general, the image data include images acquired with an imaging system, which may be an optical imaging system such as a camera. The image data may include a time-series of image frames acquired in real-time, such as a video stream. The image data depict a field-of-view in a subject undergoing a surgical procedure. The field-of-view is selected to include one or more bodily lumens in the subject that are at risk of injury during the surgical procedure.

[0045] In some embodiments, the image data are acquired from the subject while a first surgical instrument (e.g., a marker instrument) is arranged within an at-risk bodily lumen of the subject, such as a ureter. As described above, when there is more than one bodily lumen that is at risk of injury, more than one marker instruments may be used to detect each bodily lumen. The image data are acquired while exciting the fluorescent marker(s) in the first surgical instrument using light from a light source. As a result, the image data include fluorescence signals indicating the location of the first surgical instrument within the bodily lumen.

[0046] In some other embodiments, the image data are acquired from the subject after a fluorescent dye has been administered to the subject. In these instances, the image data acreacquired while the fluorescent dye is present in the bodily lumen, such that the image data include fluorescence signals indicating the location of the bodily lumen.

[0047] In some instances, the fluorescence signals are acquired using the same imaging system as the image data. Additionally or alternatively, the image data may include fluorescence signal data received from a separate fluorescence imaging system used to excite and record fluorescence signals from the fluorescent marker(s) and / or fluorescent dye. For example, a laparoscopic fluorescence imaging system may be used to separately excite and record fluorescence signals from the fluorescent marker(s) and / or fluorescent dye.

[0048] The location of the bodily lumen is then determined in the image data, as indicated at step 304. As an example, the location of the bodily lumen can be determined based on detecting fluorescence signals received from the fluorescent markers of the first surgical instrument (e.g., the marker instrument) and / or from the fluorescent dye administered to the subject. The location of the bodily lumen can thus be determined based on the fluorescence signals in the image data. In some instances, determining the location of the bodily lumen may include generating a visual indication of the fluorescence signals and overlaying that visual indication on the image data, which may be presented to a user via an AR / XR system, on a display in the operating room, or the like.

[0049] The position of the bodily lumen can thus be determined by processing the image data. In some instances, the image data can be processed using an Al and / or machine learning (ML) model to segment the image data (e.g., to segment the fluorescence signals from other image data) and / or detect and track the location of the fluorescent marker(s) of the first surgical instrument and / or fluorescent dye administered to the subject. Examples of AI / ML models include computer vision models, and the like. As a non-limiting example, the AI / ML model may include a neural network, such as a convolutional neural network, that has been trained on training data to segment, track, or otherwise detect fluorescence signals from other signals in image data.

[0050] Additionally, when a first surgical instrument (i.e., a marker instrument) has been inserted into the bodily lumen, a position of the first surgical instrument is determined from the image data, as indicated at step 306. In instances when a marker instrument is not introduced into the bodily lumen of the subject, the position of the bodily lumen may be estimated from the fluorescence signals in the image data alone. In such instances, step 306 may be skipped.

[0051] By determining the position of the first surgical instrument, the position of the bodily lumen can be more precisely known and used to determine a safety margin around the bodily lumen. As described above, the position of the first surgical instrument can be determined based on images of the non-fluorescent markers of the first surgical instrument. Because the non-fluorescent markers have a known geometric profile, the size and shape of the non-fluorescent markers in the image data can be used to calculate the position of the first surgical instrument within the surgical field.

[0052] As a non-limiting example, the position of the first surgical instrument can be determined based on a focal length equation associated with the imaging system used to acquire the image data. For instance, the distance,fromthe imaging system to the non-fluorescent marker can be calculated as:

[0053] where is the actual length of the non-fluorescent marker, which is known from the f geometric profile of the non-fluorescent marker; is the focal length of the camera lens of the imaging system; is the size of the non-fluorescent marker in the image data, measured in pixels; .® js t e width of the non-fluorescent marker in the image data, measured in pixels; andis the width of the image in pixels. By calculating the distance between the imaging system and each non-fluorescent marker, the position of each non-fluorescent marker within the surgical field can be determined.

[0054] Using the determined location of the bodily lumen and / or the calculated position of the first surgical instrument, surgical navigation data can be generated by further processing the image data, as indicated at step 308. The surgical navigation data may include a distance calculated between the first surgical instrument (as determined by the position of the non-fluorescent marker(s)) and the second surgical instrument being used during the surgical procedure. Additionally or alternatively, the surgical navigation data may include a distance calculated between the bodily lumen (as determined by the position of the bodily lumen estimated from the fluorescence signals in the image data) and the second surgical instrument being used during the surgical procedure.

[0055] In some embodiments, the distance can be a distance calculated between a known position of the second surgical instrument and the determined position of the first surgical instrument (or bodily lumen). For example, during a surgical procedure using a robotic surgery system, the position of the second surgical instrument, which is coupled to the robotic surgery system, can be known and tracked by the robotic surgery system. Using this known position, the distance between the first and second surgical instruments (or the distance between the estimated position of the bodily lumen and the known position of the second surgical instrument) can be calculated and used to track the separation between the second surgical instrument and the bodilylumen. In these instances, the surgical navigation data may include control signals for controlling the operation of the robotic surgery system. For example, the surgical navigation data may include control signals that halt the movement of the robotic surgery system when the separation distance falls below an acceptable safety margin (i.e., when the second surgical instrument becomes too close to the bodily lumen).

[0056] In some other embodiments, the distance can be a relative distance that is estimated based on a distance between the precise location of the first surgical instrument and a relative location of the second surgical instrument, or between an estimated position of the bodily lumen and a relative location of the second surgical instrument. For instance, when the second surgical instrument is a laparoscopic instrument without a precisely known location within the surgical field, the relative position of the second surgical instrument can be determined. As an example, a fluorescent marker (e.g., fluorescent tape, a fluorescent band, a fluorescent coating, etc.) can be coupled to the second surgical instrument and the relative position of the second surgical instrument can be determined by detecting fluorescence signals from the fluorescent markers. Because the fluorescence signals may not provide accurate spatial information about the location of the marker within the surgical field, the position of the second surgical instrument in these instances may only be a relative distance. The distance may therefore be a relative distance that is based on the calculated distance between fluorescence signals in the imaging plane of the image data. The position of the second surgical instrument can be determined by processing the image data, which in some instances may include processing the image data using an AI / ML model to segment the image data and / or detect and track the location of the fluorescent marker(s) of the second surgical instrument.

[0057] As another example, the surgical navigation data can include one or more visual overlays that can be displayed to the surgeon or other user together with the image data. For instance, the surgical navigation data can include the fluorescence signals detected from the fluorescent markers of the marker instruments and / or fluorescent dye arranged in any at-risk bodily lumens during the surgical procedure. The fluorescence signals can be overlaid on the image data to provide visualization of the at-risk bodily lumens during the surgical procedure. The surgical navigation data may include changes to how the fluorescence signals are visualized based on the distance between the first surgical instrument (and / or bodily lumen) and the second surgical instrument. For example, the color of the fluorescence signals as depicted as an overlay on the image data (or presented to a user via an AR / XR system) can be changed as the distance between the second surgical instrument and the bodily lumen decreases. As a non-limiting example, the color of the fluorescence signals can gradually be shifted from green to blue as the distance between the second surgical instrument and the bodily lumen decreases. For instance, the color ofthe color overlay on the image data (or presented to a user via an AR / XR system) can be gradually adjusted from a first color end of a color gradient to a second color end of a color gradient as the separation distance decreases, and the color can be gradually adjusted from the second color end of the color gradient to the first color end of the color gradient as the separation distance increases. In the example above, the color gradient may be a color gradient from green to blue, where the first color end can be green and the second color end can be blue, such that a green color indicates when a surgical instrument is farther from the bodily lumen and a blue color indicates when a surgical instrument is closer to the bodily lumen. This color variation in the fluorescence signals can provide additional visual feedback to the surgeon or other user, which can further prevent injury to the bodily lumen.

[0058] The surgical navigation data can then be displayed to a user, stored for later use or further processing, or otherwise output, as indicated at step 310. As described above, in some instances the surgical navigation data can be output to an AR / XR system to provide visual feedback to a user during a surgical procedure. Additionally or alternatively, the surgical navigation data can be output to a display in the operating room. In some other instances, the surgical navigation data can include instructions or control signals for controlling the operation of a robotic surgery system. In these case, the instructions or control signals can be sent to the robotic surgery system to control movement of the second surgical instrument by the robotic surgery system.

[0059] Step 312 may involve monitoring the surgical procedure by analyzing signal patterns emitted from the first surgical instrument. The first surgical instrument may be configured with segments that may emit or respond to signals in distinct ways, such as fluorescent portions that may emit light when excited by an external light source. These fluorescent portions may be interspersed with non-fluorescent markers along the length of the instrument, and in some implementations, the fluorescent portions may emit different intensities or wavelengths of light depending on their position. The surgical procedure may be continuously monitored by analyzing the signal patterns from the first surgical instrument using image processing techniques, spectral analysis, or machine learning algorithms. For example, a convolutional neural network may be implemented to process the image data and identify changes in the fluorescence patterns. Sudden increases in signal intensity from specific segments of the first surgical instrument may be detected, which may indicate potential injury to the bodily lumen, such as when a tear may occur in the tissue surrounding the bodily lumen, causing a previously covered segment of the first surgical instrument to become exposed.

[0060] Normal signal variations may be differentiated from those indicative of potential injury based on factors such as the magnitude of the signal change, the rate of change, or the specific location along the first surgical instrument where the change may occur. In some implementations, threshold-based detection algorithms may identify potential injuries by comparing the detected signal intensities to predetermined threshold values, potentially triggering an alert or other response when the thresholds may be exceeded. These threshold values may be dynamically adjusted based on factors such as the specific type of surgical procedure being performed or the individual patient's anatomy. Temporal analysis of the signal patterns may also be incorporated, tracking changes in signal intensity over time using techniques such as moving averages or exponential smoothing, which may help distinguish between transient signal fluctuations and sustained changes that may require intervention.

[0061] Alerts or notifications may be generated based on the detected signal patterns. These alerts may be visual, auditory, or haptic, and may be tailored to the specific needs of the surgical team. For instance, a visual alert may be displayed on a surgical monitor, potentially highlighting the area of concern and providing relevant information about the nature of the potential injury. Auditory alerts may be generated using digital signal processing techniques to create distinct tones or verbal warnings through the operating room audio system. Haptic feedback may be delivered through vibration motors embedded in wearable devices worn by the surgical team or through force feedback mechanisms in robotic surgical systems. In some examples, an output indicative of a potential injury may be combined with the navigation data described above. For instance, if the navigation data comprises a color overlay of a bodily lumen that displayed on a screen, the alert may comprise changing the color to a specific alert color (e.g., red), flashing the overlay where the injury may have occurred, etc.

[0062] FIG. 4 shows an example of a system 400 for avoiding injury to a bodily lumen (e.g., a ureter) during a surgical procedure in accordance with some embodiments described in the present disclosure. As shown in FIG. 4, a computing device 450 may receive one or more types of data (e.g., image data, fluorescence signal data) from data source 402. In some embodiments, computing device 450 may execute at least a portion of a bodily lumen injury prevention system 404 to determine the location of surgical instruments relative to an at-risk bodily lumen from data received from the data source 402. In some cases, computing device 450 may be a component of an augmented reality laparoscopic surgical system, where computing device 450 processes image data from the surgical field and generates visual overlays that may be presented to a surgeon through a heads-up display. The augmented reality system may include wearable display devices that overlay surgical navigation data onto the surgeon's field of view, allowing the surgeon to visualize the location of bodily lumens and surgical instruments in real-time during the procedure.

[0063] Additionally or alternatively, in some embodiments, the computing device 450 may communicate information about data received from the data source 402 to a server 452 over a communication network 454, which may execute at least a portion of the bodily lumen injury prevention system 404. In such embodiments, the server 452 may return information to the computing device 450 (and / or any other suitable computing device) indicative of an output of the bodily lumen injury prevention system 404. For example, the server 452 may perform computationally intensive image processing tasks, such as fluorescence signal detection and depth estimation calculations, while the computing device 450 may handle the real-time display of processed information to the surgical team through the augmented reality system.

[0064] In some embodiments, the bodily lumen injury prevention system 404 may incorporate one or more trained machine learning models for processing image data and detecting anatomical structures. These machine learning models may include convolutional neural networks (CNNs) trained on datasets of surgical images containing fluorescent markers and non-fluorescent markers. The training process may involve supervised learning using labeled images where the positions of bodily lumens, surgical instruments, and markers are annotated by medical professionals. The models may be trained using techniques such as transfer learning, where pre-trained networks on general image recognition tasks are fine-tuned for the specific task of surgical instrument and bodily lumen detection. In some cases, the machine learning models may include segmentation networks such as U-Net architectures that can precisely delineate the boundaries of bodily lumens in the image data. The models may also incorporate recurrent neural network components for temporal tracking of surgical instruments across sequential image frames, allowing for continuous monitoring of relative positions during a surgical procedure.

[0065] The trained machine learning models within the bodily lumen injury prevention system404 may support several key functions during surgical procedures. For example, a fluorescence detection model may process image data to identify and track the fluorescent portions of a marker instrument inserted into a bodily lumen, enabling real-time visualization of the bodily lumen's location even when the lumen itself may be obscured by surrounding tissue. A depth estimation model may analyze the geometric properties of non-fluorescent markers visible in the image data, calculating the precise distance between the imaging system and the marker instrument based on the apparent size of the markers with known dimensions. This calculation may utilize the focal length equation described above. Additionally, a proximity warning model may continuously evaluate the separation distance between surgical instruments and the bodily lumen, generating appropriate visual feedback by adjusting the color gradient of overlays displayed to the surgeon as instruments approach critical safety thresholds. In some implementations, these models may operate in concert to provide comprehensive spatial awareness during minimally invasiveprocedures, with inference times optimized to maintain real-time performance on standard surgical computing hardware.

[0066] In some embodiments, computing device 450 and / or server 452 may be any suitable computing device or combination of devices, such as a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable computer, a server computer, a virtual machine being executed by a physical computing device, and so on. The computing device 450 and / or server 452 may also reconstruct images from the data.

[0067] In some embodiments, data source 402 may be any suitable source of data (e.g., measurement data, images reconstructed from measurement data, processed image data), such as an imaging system, another computing device (e.g., a server storing measurement data, images reconstructed from measurement data, processed image data), and so on. In some embodiments, data source 402 may be local to computing device 450. For example, data source 402 may be incorporated with computing device 450 (e.g., computing device 450 may be configured as part of a device for measuring, recording, estimating, acquiring, or otherwise collecting or storing data). As another example, data source 402 may be connected to computing device 450 by a cable, a direct wireless link, and so on. Additionally or alternatively, in some embodiments, data source 402 may be located locally and / or remotely from computing device 450, and may communicate data to computing device 450 (and / or server 452) via a communication network (e.g., communication network 454). In some cases, data source 402 may be a component of a laparoscopic surgical system, such as a robotic surgical system, where data source 402 captures and transmits image data from the surgical field to computing device 450 for processing and analysis. For instance, data source 402 may include a laparoscopic camera system that provides real-time video feeds of the surgical site, allowing the bodily lumen injury prevention system 404 to track the positions of surgical instruments and bodily lumens during the procedure.

[0068] In some embodiments, communication network 454 may be any suitable communication network or combination of communication networks. For example, communication network 454 may include a Wi-Fi network (which may include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, etc., complying with any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX, etc.), other types of wireless network, a wired network, and so on. In some embodiments, communication network 454 may be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. Communications links shown in FIG. 4 may each be any suitablecommunications link or combination of communications links, such as wired links, fiber optic links, Wi-Fi links, Bluetooth links, cellular links, and so on.

[0069] Referring now to FIG. 5, an example of hardware 500 that can be used to implement data source 402, computing device 450, and server 452 in accordance with some embodiments of the systems and methods described in the present disclosure is shown.

[0070] As shown in FIG. 5, in some embodiments, computing device 450 can include a processor 502, a display 504, one or more inputs 506, one or more communication systems 508, and / or memory 510. In some embodiments, processor 502 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), and so on. In some embodiments, display 504 can include any suitable display devices, such as a liquid crystal display (LCD) screen, a light-emitting diode (LED) display, an organic LED (OLED) display, an electrophoretic display (e.g., an "e-ink" display), a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 506 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0071] In some embodiments, communications systems 508 can include any suitable hardware, firmware, and / or software for communicating information over communication network 454 and / or any other suitable communication networks. For example, communications systems 508 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 508 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0072] In some embodiments, memory 510 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 502 to present content using display 504, to communicate with server 452 via communications system(s) 508, and so on. Memory 510 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 510 can include random-access memory (RAM), read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), other forms of volatile memory, other forms of non-volatile memory, one or more forms of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 510 can have encoded thereon, or otherwise stored therein, a computer program for controlling operation of computing device 450. In such embodiments, processor 502 can execute at least a portion of the computer program to present content (e.g.,images, user interfaces, graphics, tables), receive content from server 452, transmit information to server 452, and so on. For example, the processor 502 and the memory 510 can be configured to perform the methods described herein (e.g., the method of FIG. 3).

[0073] In some embodiments, server 452 can include a processor 512, a display 514, one or more inputs 516, one or more communications systems 518, and / or memory 520. In some embodiments, processor 512 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, and so on. In some embodiments, display 514 can include any suitable display devices, such as an LCD screen, LED display, OLED display, electrophoretic display, a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 516 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0074] In some embodiments, communications systems 518 can include any suitable hardware, firmware, and / or software for communicating information over communication network 454 and / or any other suitable communication networks. For example, communications systems 518 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 518 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0075] In some embodiments, memory 520 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 512 to present content using display 514, to communicate with one or more computing devices 450, and so on. Memory 520 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 520 can include RAM, ROM, EPROM, EEPROM, other types of volatile memory, other types of non-volatile memory, one or more types of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 520 can have encoded thereon a server program for controlling operation of server 452. In such embodiments, processor 512 can execute at least a portion of the server program to transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 450, receive information and / or content from one or more computing devices 450, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone), and so on.

[0076] In some embodiments, the server 452 is configured to perform the methods described in the present disclosure. For example, the processor 512 and memory 520 can be configured to perform the methods described herein (e.g., the method of FIG. 3).

[0077] In some embodiments, data source 402 can include a processor 522, one or more data acquisition systems 524, one or more communications systems 526, and / or memory 528. In some embodiments, processor 522 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, and so on. In some embodiments, the one or more data acquisition systems 524 are generally configured to acquire data, images, or both, and can include an imaging system, which may be an optical imaging system (e.g., a camera system). Additionally or alternatively, in some embodiments, the one or more data acquisition systems 524 can include any suitable hardware, firmware, and / or software for coupling to and / or controlling operations of the imaging system. In some embodiments, one or more portions of the data acquisition system(s) 524 can be removable and / or replaceable. For example, data acquisition systems 524 may include a laparoscopic scope that captures images of the surgical field during a minimally invasive procedure. The laparoscopic scope may include a light emitter to stimulate fluorescence from fluorescent markers on surgical instruments or from fluorescent dye administered to the subject. In some cases, the light emitter may emit light at specific wavelengths, such as near-infrared wavelengths between 700 nm and 900 nm, to excite the fluorescent material and generate fluorescence signals that can be detected by the imaging system. The laparoscopic scope may also include optical filters to selectively capture the fluorescence emission while blocking the excitation light, thereby enhancing the visibility of the fluorescent signals in the captured images.

[0078] Note that, although not shown, data source 402 can include any suitable inputs and / or outputs. For example, data source 402 can include input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, a trackpad, a trackball, and so on. As another example, data source 402 can include any suitable display devices, such as an LCD screen, an LED display, an OLED display, an electrophoretic display, a computer monitor, a touchscreen, a television, etc., one or more speakers, and so on.

[0079] In some embodiments, communications systems 526 can include any suitable hardware, firmware, and / or software for communicating information to computing device 450 (and, in some embodiments, over communication network 454 and / or any other suitable communication networks). For example, communications systems 526 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 526 can include hardware, firmware, and / or software that can be used to establish a wired connection using any suitable port and / or communication standard(e.g., VGA, DVI video, USB, RS-232, etc.), Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0080] In some embodiments, memory 528 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 522 to control the one or more data acquisition systems 524, and / or receive data from the one or more data acquisition systems 524; to generate images from data; present content (e.g., data, images, a user interface) using a display; communicate with one or more computing devices 450; and so on. Memory 528 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 528 can include RAM, ROM, EPROM, EEPROM, other types of volatile memory, other types of non-volatile memory, one or more types of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 528 can have encoded thereon, or otherwise stored therein, a program for controlling operation of data source 402. In such embodiments, processor 522 can execute at least a portion of the program to generate images, transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 450, receive information and / or content from one or more computing devices 450, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone, etc.), and so on.

[0081] In some embodiments, any suitable computer-readable media can be used for storing instructions for performing the functions and / or processes described herein. For example, in some embodiments, computer-readable media can be transitory or non-transitory. For example, non- transitory computer-readable media can include media such as magnetic media (e.g., hard disks, floppy disks), optical media (e.g., compact discs, digital video discs, Blu-ray discs), semiconductor media (e.g., RAM, flash memory, EPROM, EEPROM), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and / or any suitable tangible media. As another example, transitory computer-readable media can include signals on networks, in wires, conductors, optical fibers, circuits, or any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and / or any suitable intangible media.

[0082] As discussed above, implementations of the described technology may be applied to a wide variety of surgical instruments and surgical uses. FIGS. 6 and 7A-D illustrate aspects of an example application of the described technology with respect to a ureteral stent. Of course, any or all of the aspects described may be employed in any implementation.

[0083] FIG. 6 illustrates a section view of a subject 600 showing anatomical structures and a ureteral stent 604. The figure depicts a kidney 602 connected to a bladder 612 via a ureter 608. Aureteral stent 604 is positioned within the ureter 608, extending from the kidney 602 to the bladder 612. For example, the ureteral stent 604 may comprise a double pigtail ureter stent or any other similar instrument.

[0084] The ureteral stent 604 includes multiple non-fluorescent markers 606 spaced along its length. Between the non-fluorescent markers 606 are fluorescent portions 610. The non- fluorescent markers 606 and fluorescent portions 610 alternate along the length of the ureteral stent 604, creating a pattern that extends from where the stent enters the kidney 602 to where it terminates in the bladder 612.

[0085] The non-fluorescent markers 606 appear as dark bands around the ureteral stent 604, while the fluorescent portions 610 comprise the sections between these markers. This arrangement allows for visualization and tracking of the stent's position within the ureter 608 during surgical procedures.

[0086] In some cases, the lengths of the fluorescent portions 610 may vary along the length of the ureteral stent 604. For example, the fluorescent portions 610 near the kidney 602 may be longer than those near the bladder 612, or vice versa. Alternatively, the lengths of the fluorescent portions 610 may be uniform along the entire length of the ureteral stent 604. The variation or uniformity in the lengths of the fluorescent portions 610 may be selected based on factors such as the specific anatomy of the subject 600, the type of surgical procedure being performed, or the desired level of visibility in different regions of the ureter 608.

[0087] The ureteral stent 604 may be inserted into the ureter 608 using standard medical insertion techniques. For example, the ureteral stent 604 may be guided into position using a guidewire and fluoroscopic imaging. Once in place, the alternating pattern of fluorescent portions 610 and non-fluorescent markers 606 may allow for continuous monitoring of the ureter's position during subsequent surgical procedures. The placement of the ureteral stent 604 within the urinary system, as shown in FIG. 6, may assist in navigation during surgery. For instance, the fluorescent portions 610 may enhance visibility of the ureter 608, even when obscured by surrounding tissue. The non-fluorescent markers 606 may serve as reference points for precise distance measurements, enabling accurate localization of the ureteral stent 604 and, by extension, the ureter 608.

[0088] FIGS. 7A-7D illustrate different views of a surgical site 700 during a surgical procedure. The surgical site 700 may be any anatomical location where a bodily lumen may be at risk of injury during surgery. In some cases, the surgical site 700 may be in the pelvic region, where the ureter 716 may be at risk during gynecological procedures such as hysterectomies or ovarian surgeries. In other cases, the surgical site 700 may be in the abdominal cavity, where the ureter 716 may be at risk during colorectal surgeries, such as colon resections or rectal procedures.The surgical site 700 may also be in the thoracic region for cardiovascular procedures where blood vessels may be at risk, in the neurological region for procedures involving cerebral vasculature, or in the pulmonary region for bronchial procedures. The surgical procedure may be any type of procedure, such as a single or multiple incision minimally invasive procedure, a natural orifice procedure, an open procedure, a robotic procedure, or a laparoscopic procedure. In the illustrated example, the ureteral stent 702 has been inserted into the ureter 716 prior to surgery so that the ureteral stent 702 is already present when the operation begins, allowing for immediate visualization and monitoring of the ureter 716 location throughout the procedure without additional steps for stent placement during the operation.

[0089] FIG. 7A shows a close-up view of the surgical site 700. The ureteral stent 702 may be inserted into a ureter 716 and may include fluorescent portions 712 alternating with non- fluorescent markers along its length. Non-fluorescent markers 708, 714, and 718 may be positioned along the ureteral stent 702, alternating with the fluorescent portions 712. The non- fluorescent markers 708, 714, and 718 may have known geometric profiles, such as specific widths or shapes, that enable distance calculations. In some cases, the non-fluorescent markers 708, 714, and 718 may serve as reference points for precise distance measurements during surgical procedures. The arrangement of non-fluorescent markers 708, 714, and 718 and fluorescent portions 712 may allow for continuous monitoring of the ureteral stent 702 position within the ureter 716, providing visual reference points along the length of the ureter 716 for surgical navigation.

[0090] The scope 706 may include a light emitter 704 and a camera 710 positioned to view the surgical site 700. The light emitter 704 may include one or more light-emitting diodes (LEDs) or laser diodes that emit light at wavelengths between 700 nm and 900 nm to excite the fluorescent portions 712 of the ureteral stent 702. The camera 710 may capture the fluorescence signals emitted by the fluorescent portions 712 that pass through the ureteral tissue. In some cases, the scope 706 may be near-infrared (NIRF) compatible, allowing for deeper tissue penetration and improved visualization. The camera 710 may include optical filters to selectively capture the fluorescence emission while blocking the excitation light. The non-fluorescent markers 708, 714, and 718 may appear as dimmed regions or pixels in the image data captured by the camera 710, creating a distinctive pattern of bright fluorescent regions separated by dark non-fluorescent markers.

[0091] The image data captured by the camera 710 may be processed to generate surgical navigation data. The non-fluorescent markers 708, 714, and 718 may be detected in the image data and used to determine the position of the ureteral stent 702 within the surgical field. In some cases, the different configurations of non-fluorescent markers 708, 714, and 718 may encode position ordepth information along the length of the ureteral stent 702. For example, the spacing between non-fluorescent markers 708 and 714 may differ from the spacing between non-fluorescent markers 714 and 718, providing positional encoding along the ureteral stent 702. The location of a second surgical instrument relative to the ureteral stent 702 may be determined based on measuring a separation distance between the position of the ureteral stent 702 and a position of the second surgical instrument.

[0092] In other examples, the non-fluorescent markers may be uniform and / or uniformly spaced, such as illustrated with respect to FIGS. 1A-1B, where the non-fluorescent markers 122 are uniformly spaced apart along the length of the flexible body 112 and interleaved with the fluorescent markers 120. The uniformly spaced non-fluorescent markers 122 may be used to determine the position of the first surgical instrument within the bodily lumen by analyzing the apparent size of each non-fluorescent marker 122 in the image data. For example, the distance between the imaging system and each non-fluorescent marker 122 may be calculated using a focal length equation that compares the known physical size of the non-fluorescent marker 122 with its apparent size in pixels within the image data. The uniform spacing between adjacent non- fluorescent markers 122 may provide reference points for depth calculation, as markers that appear closer together in the image may indicate portions of the first surgical instrument that are farther from the imaging system. In some cases, the computer system may process the image data to identify all visible non-fluorescent markers 122 and calculate their relative positions in three- dimensional space based on their uniform geometry and spacing, thereby enabling precise localization of the first surgical instrument within the bodily lumen.

[0093] The position data of the ureteral stent 702 obtained from the non-fluorescent markers 708, 714, and 718 may be used to determine the distance and direction from the stent to a second surgical instrument. In some cases, the computer system may calculate a three-dimensional coordinate for each visible non-fluorescent marker on the ureteral stent 702 within the surgical field. These coordinates may then be compared to the known position of a second surgical instrument, such as a robotic arm or laparoscopic tool, to determine the relative distance and orientation between the two. The system may continuously update these calculations in real-time as the surgical instruments move within the field, providing dynamic spatial awareness to the surgical team. This spatial information may be used to generate visual overlays or auditory alerts when the second surgical instrument approaches a predetermined safety threshold distance from the ureteral stent 702. The location of the second surgical instrument relative to the first surgical instrument may be determined by measuring a separation distance between the position of the first surgical instrument and a position of the second surgical instrument. In some cases, the separation distance may be calculated using a triangulation algorithm that processes the three-dimensionalcoordinates of both instruments within the surgical field. The separation distance may be continuously monitored and updated (e.g., at a rate of 30+ frames per second) to provide real-time feedback during the surgical procedure.

[0094] FIGS. 7B-7D illustrate examples of surgical displays with visual navigation data overlaid on the ureter 716. For example, the surgical display 720 may be a display 504 or display 514 of a system as described with respect to FIG. 5. FIG. 7B shows a surgical display 720 where a surgical instrument 722 approaches the ureter 716, with a first distance indicator 724a representing the separation between the instrument and a surgical target 725. FIG. 7C depicts the surgical display 720 as the surgical instrument 722 moves closer to the ureter 716, with an updated second distance indicator 724b. FIG. 7D illustrates the surgical display 720 during a procedure where a tear 726 has occurred, resulting in a brighter signal 728 being detected in the affected area.

[0095] The surgical navigation data may be output to a heads-up display of an augmented reality system, allowing surgeons to view the spatial information without looking away from the surgical site 700. In robotic surgery applications, the navigation data may be used to generate control signals for the robotic system, potentially halting or adjusting the movement of surgical instruments when they approach critical safety thresholds around the ureter 716. The surgical navigation data may include trajectory prediction information derived from analyzing the movement patterns of the surgical instrument 722 relative to the ureteral stent 702. For example, the system may calculate a projected path of the surgical instrument 722 based on its current velocity and direction, and provide visual indicators of potential intersection points with the ureter 716. In some cases, the navigation data may include tissue deformation metrics that estimate how surgical manipulation may affect the position of the ureter 716. Additionally, the navigation data may include depth maps constructed from the known geometric profiles of the non-fluorescent markers 708, 714, and 718, providing three-dimensional spatial awareness beyond simple distance measurements. The system may generate these depth maps by triangulating the positions of multiple non-fluorescent markers visible in the image data and calculating their relative distances from the camera 710 using focal length equations.

[0096] In some cases, the system may include a safety margin threshold that defines a minimum allowable separation distance between the surgical instrument 722 and the ureter 716. This safety margin may be a predetermined, user-programmable, or automatically calculated distance, such as 5 mm, 10 mm, or any other suitable distance based on the specific surgical procedure and anatomical considerations. The system may continuously compare the calculated separation distance between the surgical instrument 722 and the ureter 716 to the defined safety margin threshold. When the separation distance falls below the safety margin threshold, the systemmay automatically generate a control signal to stop the movement of the surgical instrument 722, thereby preventing potential injury to the ureter 716.

[0097] FIG. 7B shows a view as displayed on a surgical display 720, where a surgical instrument 722 approaches the ureter 716. A first distance indicator 724a may be shown representing a first separation distance between the surgical instrument 722 and a surgical target 725. The surgical target 725 may be the ureter 716 or a specific portion of the ureteral stent 702. The first distance indicator 724a may be generated by a bodily lumen injury prevention system 404 based on calculations performed using the known geometry of the non-fluorescent markers 708, 714, and 718 and the focal length of the camera 710.

[0098] FIG. 7C depicts another view on the surgical display 720 where the surgical instrument 722 has moved closer to the ureter 716. A second distance indicator 724b may be shown representing a closer separation distance between the surgical instrument 722 and the surgical target 725 than 724a. The second distance indicator 724b may be updated in real-time as the surgical instrument 722 moves within the surgical site 700, providing continuous feedback about the proximity of the surgical instrument 722 to the ureter 716.

[0099] The distance indicators 724a and 724b may be part of a visual feedback system that provides a visual indication of the separation distance between the surgical instrument 722 and the ureter 716. In some cases, this visual feedback 724a, 724b may be presented as a color overlay representing the bodily lumen (e.g., ureter) enveloping the fluorescence signals from the fluorescent portions 712 of the ureteral stent 702. The color of the color overlay may change based on the separation distance. For example, the color may gradually adjust along a gradient from a first color to a second color as the separation distance decreases, and vice versa as the separation distance increases. The color gradient may range from green (indicating a safe distance) to blue (indicating close proximity) to red (indicating potential danger of contact).

[0100] In some implementations, the surgical navigation data, including the visual feedback, may be output to a heads-up display of an augmented reality system. This may allow the surgeon to view the separation distance information without looking away from the surgical site 700. The augmented reality system may include a wearable display device that overlays the distance indicators 724a and 724b onto the surgeon's field of view. The wearable display device may communicate with the computing device 450 or server 452 via the communication network 454 to receive the surgical navigation data in real-time.

[0101] The surgical instrument 722 may be coupled to a robotic surgery system in some cases. The surgical navigation data may then comprise instructions for controlling the robotic system. For example, the instructions may include commands to stop the movement of the surgicalinstrument 722 when the separation distance falls below a predetermined safety margin around the ureter 716. Additionally, the system may cause the robotic surgery system to move the surgical instrument 722 more slowly as it approaches the ureter 716, providing an additional layer of safety. The robotic surgery system may include actuators and control mechanisms that respond to the instructions generated by the bodily lumen injury prevention system 404 based on the calculated separation distance.

[0102] In some implementations, the surgical instrument 722 may have a fluorescent marker coupled to it. The position of the surgical instrument 722 may be determined based on measured fluorescence signals from this marker, allowing for precise tracking of the instrument's location relative to the ureter 716. The fluorescent marker on the surgical instrument 722 may emit light at a different wavelength than the fluorescent portions 712 of the ureteral stent 702, enabling the bodily lumen injury prevention system 404 to distinguish between the two sources of fluorescence in the image data captured by the camera 710.

[0103] FIG. 7D illustrates a view on the surgical display 720 showing the surgical site 700 during a surgical procedure where an injury 726 (e.g., a tear) has occurred in the tissue surrounding the ureter 716. This tear 726 may result in a brighter signal 728 being detected in that area because of the cut exposing the stent 702. In some examples, the ureteral stent 702 may be configured with segment-specific signal-emitting or signal-responsive features, and the bodily lumen injury prevention system 404 may be programmed to analyze signal patterns from the ureteral stent 702. The brighter signal 728 may be detected by the camera 710 and processed by the processor 502 or processor 512 to identify potential tissue damage.

[0104] An injury such as the tear 726 may be identified based on the detection of sudden bursts of signal from a specific segment of the ureteral stent 702 upon exposure to the light emitter 704, wherein these signal bursts may be distinguishable from segments of the ureteral stent 702 that remain covered by ureteral tissue. The location of the injury may be determined based on its proximity to a specific non-fluorescent marker, such as non-fluorescent marker 714. The bodily lumen injury prevention system 404 may analyze the intensity, pattern, and location of the brighter signal 728 to determine the nature and severity of the tear 726.

[0105] The fluorescent portions 712 of the ureteral stent 702 may be visible between the non- fluorescent markers 708, 714, and 718, providing visual reference points along the length of the ureter 716. This arrangement may allow for monitoring of the separation distance between the surgical instrument 722 and the ureter 716 during the surgical procedure and detection of injuries. The non-fluorescent markers 708, 714, and 718 may have a known geometric profile, such as a specific width or shape, that enables the bodily lumen injury prevention system 404 to calculate 1the distance between the ureteral stent 702 and the camera 710 using the focal length equation described previously.

[0106] In some cases, the bodily lumen injury prevention system 404 may employ machine learning algorithms to analyze the signal patterns from the ureteral stent 702. These algorithms may be trained to distinguish between normal variations in signal intensity and those indicative of potential injury. For example, a convolutional neural network may be used to process the image data from the camera 710 and identify changes in the fluorescence patterns that may suggest tissue damage. The convolutional neural network may be trained on a dataset of images showing both normal and injured tissue, with annotations indicating the presence and location of injuries.

[0107] The system may also incorporate temporal analysis of the signal patterns, tracking changes in signal intensity over time. This may help distinguish between transient signal fluctuations and sustained changes that may require intervention. For instance, the system may use techniques such as moving averages or exponential smoothing to identify trends in the signal intensity that may indicate progressive tissue damage. The temporal analysis may be performed by the processor 502 or processor 512 using data stored in the memory 510 or memory 520, which may include a time series of image frames captured by the camera 710.

[0108] In addition to visual feedback, the system may generate auditory or haptic alerts based on the detected signal patterns. These alerts may be tailored to the specific needs of the surgical team. For example, an auditory alert may be generated using digital signal processing techniques to create distinct tones or verbal warnings through the operating room audio system when the surgical instrument 722 approaches the safety margin around the ureter 716. The auditory alerts may vary in pitch, volume, or pattern depending on the proximity of the surgical instrument 722 to the ureter 716, providing intuitive feedback to the surgical team without requiring visual attention to the surgical display 720.

[0109] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms “component,” “system,” “module,” “framework,” and the like are intended to encompass part or all of computer-related systems that include hardware, software, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a processor device, a process being executed (or executable) by a processor device, an object, an executable, a thread of execution, a computer program, or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components (or system, module, and so on) may reside within a process or thread of execution, may be localized on one computer, may be distributed between twoor more computers or other processor devices, or may be included within another component (or system, module, and so on).

[0110] In some implementations, devices or systems disclosed herein can be utilized or installed using methods embodying aspects of the disclosure. Correspondingly, description herein of particular features, capabilities, or intended purposes of a device or system is generally intended to inherently include disclosure of a method of using such features for the intended purposes, a method of implementing such capabilities, and a method of installing disclosed (or otherwise known) components to support these purposes or capabilities. Similarly, unless otherwise indicated or limited, discussion herein of any method of manufacturing or using a particular device or system, including installing the device or system, is intended to inherently include disclosure, as embodiments of the disclosure, of the utilized features and implemented capabilities of such device or system.[0U1] The present disclosure has described one or more preferred embodiments, and it should be appreciated that many equivalents, alternatives, variations, and modifications, aside from those expressly stated, are possible and within the scope of the invention.

Claims

CLAIMS1. A method for tracking a surgical instrument, the method comprising: receiving image data with a computer system, wherein the image data comprise an image acquired with an imaging system, wherein the image depicts a subject containing a bodily lumen into which a first surgical instrument has been introduced, the first surgical instrument having a fluorescent portion and a non-fluorescent portion, wherein a geometry of the non-fluorescent portion is known; determining, with the computer system, a location of the bodily lumen based on a measured fluorescence signal of the fluorescent portion of the first surgical instrument in the image data; determining, with the computer system, a position of the first surgical instrument by calculating, from the image data, a distance between the non-fluorescent portion of the first surgical instrument and the imaging system; determining, with the computer system, a location of a second surgical instrument relative to the first surgical instrument based on measuring a separation distance between the position of the first surgical instrument and a position of the second surgical instrument; and outputting surgical navigation data with the computer system thereby providing guidance of the second surgical instrument relative to the bodily lumen, wherein the surgical navigation data are generated based on the separation distance.

2. The method of claim 1, wherein the distance between the non-fluorescent portion of the first surgical instrument and the imaging system is determined based on an apparent size of the non-fluorescent portion of the first surgical instrument in the image data.

3. The method of claim 2, wherein the apparent size of the non-fluorescent portion of the first surgical instrument in the image data is calculated based on a focal length of a camera of the imaging system and the known geometry of the non-fluorescent portion of the first surgical instrument.

4. The method of claim 1, wherein the surgical navigation data comprise visual feedback data output with the computer system.

5. The method of claim 4, wherein the visual feedback data comprise a color overlay representing the fluorescence signals of the fluorescent portion of the first surgical instrument.

6. The method of claim 5, wherein outputting the surgical navigation data comprises changing a color of the color overlay based on the separation distance.

7. The method of claim 6, wherein changing the color of the color overlay based on the separation distance comprises gradually adjusting the color from a first color end of a color gradient to a second color end of a color gradient as the separation distance decreases, and gradually adjusting the color from the second color end of the color gradient to the first color end of the color gradient as the separation distance increases.

8. The method of claim 1, wherein the surgical navigation data comprise a visual indication of the separation distance.

9. The method of claim 1, wherein outputting the surgical navigation data comprises outputting the surgical navigation data to a heads up display of an augmented reality system.

10. The method of claim 1, wherein the second surgical instrument is coupled to a robotic surgery system and the surgical navigation data comprise instructions for controlling operation of the robotic surgery system.

11. The method of claim 10, wherein the instructions comprise stopping movement of the second surgical instrument by the robotic surgery system when the separation distance falls below a safety margin around the bodily lumen.

12. The method of claim 10, wherein the separation distance is measured between the determined position of the first surgical instrument and a known position of the second surgical instrument received from the robotic surgery system.

13. The method of claim 1, wherein the separation distance is measured between the determined position of the first surgical instrument and a relative position of the second surgical instrument.

14. The method of claim 13, wherein the relative position of the second surgical instrument is determined from the image data.

15. The method of claim 14, wherein the second surgical instrument has a fluorescent marker coupled thereto and the relative position of the second surgical instrument is determined from the image data based on measured fluorescence signals from the fluorescent marker in the image data.

16. The method of claim 1, further comprising: determining, with the computer system, a location of a potential injury to the bodily lumen based on a change of a measured fluorescence signal of the fluorescent portion of the first surgical instrument in the image data; outputting injury data indicative of the potential injury with the computer system.

17. A method for tracking a surgical instrument, the method comprising: receiving image data with a computer system, wherein the image data comprise an image acquired with an imaging system, wherein the image depicts a subject containing a bodily lumen into which a fluorescent dye has been administered; estimating, with the computer system, a location of the bodily lumen based on a measured fluorescence signal of the fluorescent dye in the image data; estimating, with the computer system, a location of a surgical instrument relative to the bodily lumen based on measuring a separation distance between the estimated position of the bodily lumen and a position of the surgical instrument; and outputting surgical navigation data with the computer system thereby providing guidance of the surgical instrument relative to the bodily lumen, wherein the surgical navigation data are generated based on the separation distance.

18. A stent for insertion into a bodily lumen, comprising: a flexible body extending along a length from a first end to a second end, the flexible body comprising a plurality of fluorescent portions and a plurality of non-fluorescent portions, wherein the plurality of non-fluorescent portions are uniformly spaced apart along the length of the flexible body and interleaved with the plurality of fluorescent portions, wherein each of the plurality of non-fluorescent portions have a common geometry; anda lumen extending through the flexible body from the first end to the second end.

19. The stent of claim 18, wherein the plurality of fluorescent portions comprise portions of the flexible body that are coated with a fluorescent material and the plurality of non-fluorescent portions comprise portions of the flexible body that are not coated with the fluorescent material.

20. The stent of claim 18, wherein the plurality of fluorescent portions comprise portions of the flexible body that are impregnated with a fluorescent material and the plurality of non- fluorescent portions comprise portions of the flexible body that are not impregnated with the fluorescent material.

21. The stent of claim 18, wherein the plurality of non-fluorescent portions comprise a plurality of annular shaped markers each having a central aperture, wherein the flexible body extends through the central aperture of each of the plurality of annular shaped markers.

Images