2D image registration

Abstract

The present disclosure relates to systems, devices and methods for enhancing two-dimensional images.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 295,516, entitled "TWO-DIMENSIONAL IMAGE REGISTRATION," filed on December 31, 2021, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] Various medical procedures involve the use of one or more devices configured to penetrate the human anatomy to reach a treatment site. Certain operational processes may include locating the medical instrument within a patient and visualizing an area of ​​interest within the patient. To do so, many medical instruments may include sensors to track the instrument's position and may include vision capabilities, such as the use of an embedded camera or compatible with a visual probe. [Brief description of the drawings]

[0003] Various embodiments are illustrated in the accompanying drawings for purposes of illustration and should not be construed as limiting the scope of the present disclosure in any way. It should be noted that various features of different disclosed embodiments may be combined to form further embodiments that are part of this disclosure. Throughout the drawings, reference numerals may be reused to indicate correspondence between referenced elements. [Figure 1] FIG. 1 is a block diagram illustrating an exemplary two-dimensional image registration system for performing various medical procedures according to aspects of the present disclosure. [Diagram 2] FIG. 1 illustrates an augmented two-dimensional image in accordance with one or more embodiments. [Diagram 3] 1 is a flowchart illustrating a method for adding tool position information to a two-dimensional image, according to an exemplary embodiment. [Figure 4] 1 is a diagram showing an example of a two-dimensional image with segmented data in accordance with an exemplary embodiment; [Diagram 5]1A-1C are diagrams illustrating exemplary enhanced non-contrast images, in accordance with exemplary embodiments; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The directions provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. Although certain exemplary embodiments are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Thus, claims that may arise from this specification are not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described in sequence as multiple separate operations in a manner that may be helpful in understanding a particular embodiment, however, the order of description should not be construed to imply that these operations are order dependent. Furthermore, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are realized by any particular embodiment. Thus, for example, various embodiments may be made in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0005] overview The present disclosure relates to systems, devices and methods for augmenting a two-dimensional image with three-dimensional data from a position sensor or any other suitable three-dimensional system data, such as robotic data (e.g., insert commands, retract commands, articulation, etc.).

[0006] 2D Image Registration System FIG. 1 is a block diagram illustrating an exemplary two-dimensional image registration system 100 for performing various medical procedures according to aspects of the present disclosure. The two-dimensional image registration system 100 includes a robotic system 110 configured to engage and / or control a medical instrument 120 to perform a procedure on a patient 130. The two-dimensional image registration system 100 also includes a control system 140 configured to interface with the robotic system 110, provide information regarding the procedure, and / or perform various other operations. For example, the control system 140 may include a display(s) 142 that presents certain information to assist a physician 160. The display(s) 142 may be a monitor, a screen, a television, virtual reality hardware, augmented reality hardware, a three-dimensional imaging device (e.g., a hologram device), etc., or a combination thereof. The two-dimensional image registration system 100 may include a platform 150 configured to hold the patient 130. System 100 may further include an electromagnetic (EM) field generator 180, which may be held by one or more robotic arms 112 of robotic system 110 or may be a stand-alone device. In an embodiment, two-dimensional image registration system 100 may also include an imaging device 190, which may be integrated into a C-arm and / or configured to provide imaging during a procedure, such as a fluoroscopy-type procedure.

[0007] In some implementations, the two-dimensional image registration system 100 may be used to perform a percutaneous procedure. For example, if the patient 130 has a kidney stone that is too large to remove through the urinary tract, the physician 160 may perform a procedure to remove the kidney stone through a percutaneous access point on the patient 130. To illustrate, the physician 160 may interact with the control system 140 to control the robotic system 110 to advance and navigate the medical instrument 120 (e.g., a scope) from the urethra through the bladder, up the ureter, and into the kidney where the stone is located. The control system 140 may provide information about the medical instrument 120 via the display(s) 142, such as real-time images captured therewith, to assist the physician 160 in navigating the medical instrument 120.

[0008] Upon reaching the location of the kidney stone (e.g., within the calyx), the medical instrument 120 may be used to designate / tag a target location (e.g., a desired point for accessing the kidney) for a medical instrument 170 (e.g., a needle) to percutaneously access the kidney. To minimize damage to the kidney and / or surrounding anatomical structures, the physician 160 may designate a particular papilla as the target location for entry into the kidney with the medical instrument 170. However, other target locations may be designated or determined. To assist the physician in driving the medical instrument 170 through the particular papilla and into the patient 130, the control system 140 may provide a visualization interface 144, which may include a rendering of two-dimensional image data augmented based on three-dimensional data from systems such as position sensor data, robotic data, image data, etc. As will be described in more detail, the visualization interface 144 may provide the operator with information to aid in driving the medical instrument 170 to the target location. An example of a visualization interface 200 according to one embodiment is shown in FIG. 2. For example, the visualization interface 200 may include visual indicators that map the location of the instrument to a location in the two-dimensional image. For example, the operator may "tag" or otherwise mark or designate a particular location of the instrument as a desired location. For example, in kidney stone removal, the operator may move the endoscope to different calyces and tag those locations when the endoscope reaches the desired calyx. The visualization interface 200 includes tagged locations 210a, 210b, 210c. The visualization interface may also render a visual indicator representing the current location of the endoscope, such as by a current location indicator 220. Additionally, it may be useful for the operator to understand where the scope has moved during the course of the procedure. This is indicated by a tracking indicator 230, which provides a visual indicator of where the endoscope was relative to the anatomy. It will be appreciated that the location of the scope's location in the two-dimensional image may use the registration techniques described herein to map a location in the position sensor space to a location in the two-dimensional image.

[0009] 1 , once the instrument 170 reaches the target, the physician 160 may use the medical instrument 170 and / or another medical instrument to remove the kidney stone from the patient 130. One such instrument may be a percutaneous catheter. The percutaneous catheter, like the instrument 120, may be an instrument with steering capabilities, but in some embodiments may not have a dedicated camera or position sensor. Some embodiments may use an augmented visualization interface 144 to render augmented images that are useful for navigating the percutaneous catheter within the anatomy.

[0010] Although the above percutaneous and / or other procedures have been discussed in the context of using a medical instrument 120, in some implementations, a percutaneous procedure may be performed without the assistance of a medical instrument 120. Additionally, the two-dimensional image registration system 100 may be used to perform a variety of other procedures.

[0011] Additionally, while many embodiments are described as the physician 160 using the medical instrument 170, the medical instrument 170 may alternatively be used by components of the two-dimensional image registration system 100. For example, the medical instrument 170 may be held / manipulated by the robotic system 110 (e.g., one or more robotic arms 112), and the techniques discussed herein may be implemented to control the robotic system 110 to insert the medical instrument 170 in the proper pose (or aspect of the pose, such as orientation or placement) to reach the target location.

[0012] 1, medical instrument 120 is implemented as a scope and medical instrument 170 is implemented as a needle. Thus, for ease of discussion, medical instrument 120 will be referred to as "the scope 120" or "the lumen-based medical instrument 120" and medical instrument 170 will be referred to as "the needle 170" or "the percutaneous medical instrument 170". However, medical instrument 120 and medical instrument 170 may each be implemented as any suitable type of medical instrument, including, for example, a scope (sometimes referred to as an "endoscope"), a needle, a catheter, a guidewire, a nephrolithotomy device, a basket retrieval device, a forceps, a vacuum, a needle, a scalpel, an imaging probe, a jaw, a scissors, a grasper, a needle holder, a microdissection instrument, a staple applier, a tacker, an aspirating / irrigating tool, a clip applier, or the like. In some embodiments, the medical instrument is a steerable device, while in other embodiments, the medical instrument is a non-steerable device. In some embodiments, a surgical tool refers to a device configured to pierce or be inserted through a body structure, such as a needle, scalpel, guidewire, etc. However, a surgical tool may refer to other types of medical instruments.

[0013] In some embodiments, the medical instrument, such as the scope 120 and / or the needle 170, includes a sensor configured to generate sensor data that can be transmitted to another device. In examples, the sensor data can be indicative of and / or can be used to determine the position / orientation of the medical instrument. For example, the sensor can include an electromagnetic (EM) sensor having a coil of conductive material, where an EM field generator, such as EM field generator 180, can provide an EM field that is detected by the EM sensor on the medical instrument. The magnetic field can induce small currents in the coil of the EM sensor, which can be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator. Additionally, the medical instrument can include other types of sensors configured to generate sensor data, such as any one or more of a camera, range sensor, radar device, shape-sensing fiber, accelerometer, gyroscope, satellite-based positioning sensor (e.g., Global Positioning System (GPS)), radio frequency transceiver, etc. In some embodiments, the sensor is positioned on the distal end of the medical instrument, while in other embodiments the sensor is positioned at another location on the medical instrument. In some embodiments, the sensor on the medical instrument may provide sensor data to the control system 140, which may implement one or more localization techniques to determine / track the position and / or orientation of the medical instrument.

[0014] In some embodiments, the two-dimensional image registration system 100 may record or otherwise track run-time data generated during a medical procedure. This run-time data may be referred to as system data. For example, the two-dimensional image registration system 100 may track or otherwise record sensor readings (e.g., sensor data) from instruments (e.g., the scope 120 and needle 170) in a data store 145A (e.g., a computer storage system such as a computer-readable memory, a database, a file system, etc.). In addition to the sensor data, the two-dimensional image registration system 100 may store other types of system data in the data store 145. For example, in the context of FIG. 1 , system data may further include time series data of video images captured by scope 120, status of robotic system 110, command data from I / O device(s) (e.g., I / O device(s) 146 described below), audio data (which may be captured by an audio capture device embedded within the two-dimensional image registration system, such as a microphone on the medical instrument, a robotic arm, or other location within two-dimensional image registration system 100), imaging data from an external (relative to the patient) imaging device (an RGB camera, a LIDAR imaging sensor, a fluoroscopic imaging sensor, etc.), and imaging device 190, etc.

[0015] 1, the control system 140 includes an augmentation module 141, which may be control circuitry configured to operate on the two-dimensional image data stored in the system data and case data store 145 to generate an augmented representation of the two-dimensional image data with the three-dimensional system data. As described in more detail below, the augmentation module 141 may use machine learning techniques to segment the two-dimensional image data according to anatomical structures or instruments present in the two-dimensional image. In some embodiments, once the two-dimensional image data is segmented, the augmentation module 141 may augment the two-dimensional image data with three-dimensional data from the system 100.

[0016] The terms "scope" or "endoscope" are used herein according to their broad and ordinary meaning and may refer to any type of elongated medical instrument having imaging, viewing, and / or capture capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, and / or space of the body. For example, references herein to a scope or endoscope may refer to a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidney), a bronchoscope (e.g., for accessing the airways such as the bronchi), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), a borescope, etc.

[0017] The scope may comprise a tubular and / or flexible medical instrument configured to be inserted into a patient's anatomy to capture images of the anatomy. In some embodiments, the scope may house wires and / or optical fibers to transfer signals between the optical assembly and the distal end of the scope, and the scope may include an imaging device, such as an optical camera. The camera / imaging device may be used to capture images of an internal anatomical space, such as a target calyx / papilla of the liver. The scope may be further configured to house optical fibers to carry light from a proximally located light source, such as a light emitting diode, to the distal end of the scope. The distal end of the scope may include a port for a light source to illuminate the anatomical space when the camera / imaging device is used. In some embodiments, the scope is configured to be controlled by a robotic system, such as the robotic system 110. The imaging device may comprise optical fibers, fiber arrays, and / or lenses. The optical components may move with the tip of the scope, such that movement of the tip of the scope results in a change in the image captured by the imaging device.

[0018] The scope may be articulatable, such as with respect to at least a distal portion of the scope, so that the scope may be maneuvered within the human anatomy. In some embodiments, the scope is configured to be articulated, for example, with 5 or 6 degrees of freedom, including, for example, X, Y, Z coordinate translation, as well as pitch, yaw, and roll. The scope's position sensor(s) may similarly have similar degrees of freedom with respect to the position information they create / provide. The scope may include telescopic parts, such as an inner leader portion and an outer sheath portion, which may be manipulated to telescopically extend the scope. The scope may, in some cases, comprise a rigid or flexible tube and may be sized to pass through an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device. In some embodiments, the scope includes a working channel for deploying medical instruments (e.g., lithotriptor, basket device, forceps, etc.), irrigation, and / or suction to the working area at the distal end of the scope.

[0019] The robotic system 110 may be configured to at least partially facilitate the performance of a medical procedure. The robotic system 110 may be arranged in a variety of ways depending on the particular procedure. The robotic system 110 may include one or more robotic arms 112 configured to engage and / or control the scope 120 to perform the procedure. As shown, each robotic arm 112 may include multiple arm segments coupled to articulations, thereby providing multiple degrees of mobility. In the example of FIG. 1, the robotic system 110 is positioned adjacent a leg of the patient 130, and the robotic arm 112 is actuated to engage and position the scope 120 for access to an access point, such as the urethra, of the patient 130. Once the robotic system 110 is properly positioned, the scope 120 may be inserted into the patient 130 robotically using the robotic arm 112, manually by the physician 160, or a combination thereof. The robotic arm 112 may also be connected to an EM field generator 180 , which may be positioned near the treatment site, such as within close proximity to the kidneys of the patient 130 .

[0020] The robotic system 110 may also include a support structure 114 coupled to the one or more robotic arms 112. The support structure 114 may include control electronics / circuitry, one or more power sources, one or more pneumatics, one or more light sources, one or more actuators (e.g., motors to move the one or more robotic arms 112), memory / data storage, and / or one or more communication interfaces. In some embodiments, the support structure 114 includes Input / Output (I / O) device(s) 116 configured to receive inputs, such as user inputs for controlling the robotic system 110, and / or provide output, such as a Graphical User Interface (GUI), information about the robotic system 110, information about a procedure, etc. The I / O device(s) 116 may include a display, a touch screen, a touch pad, a projector, a mouse, a keyboard, a microphone, a speaker, etc. In some embodiments, the robotic system 110 is mobile (e.g., the support structure 114 includes wheels) so that the robotic system 110 can be positioned in an appropriate or desired location for a procedure. In other embodiments, the robotic system 110 is a fixed system. Additionally, in some embodiments, the robotic system 112 is integrated into the table 150.

[0021] The robotic system 110 may be coupled to any of the components of the two-dimensional image registration system 100, such as the control system 140, the stage 150, the EM field generator 180, the scope 120, and / or the needle 170. In some embodiments, the robotic system is communicatively coupled to the control system 140. In one example, the robotic system 110 may be configured to receive control signals from the control system 140 to perform operations, such as positioning the robotic arm 112 in a particular manner, manipulating the scope 120, etc. In response, the robotic system 110 may control the components of the robotic system 110 to perform the operations. In another example, the robotic system 110 may be configured to receive images from the scope 120 depicting the internal anatomy of the patient 130 and / or transmit the images to the control system 140, which may then be displayed on the display(s) 142. Additionally, in some embodiments, the robotic system 110 is coupled to components of the two-dimensional image registration system 100, such as the control system 140, in a manner that allows it to receive fluids, optics, power, etc. therefrom. Exemplary details of the robotic system 110 are discussed in further detail below with reference to FIG.

[0022] Control system 140 may be configured to provide a variety of functions to assist in the performance of a medical procedure. In some embodiments, control system 140 may be coupled to robotic system 110 and operate in cooperation with robotic system 110 to perform a medical procedure on patient 130. For example, control system 140 may communicate with robotic system 110 via a wireless or wired connection (e.g., to control robotic system 110 and / or scope 120, to receive image(s) captured by scope 120, etc.), provide fluid to robotic system 110 via one or more fluid channels, provide power to robotic system 110 via one or more electrical connections, provide optics to robotic system 110 via one or more optical fibers or other components, etc. Additionally, in some embodiments, control system 140 may communicate with needle 170 and / or scope 170 to receive sensor data from needle 170 and / or endoscope 120 (via robotic system 110 and / or directly from needle 170 and / or endoscope 120). Additionally, in some embodiments, control system 140 may be in communication with table 150 to position table 150 in a particular orientation or otherwise control table 150. Additionally, in some embodiments, control system 140 may be in communication with EM field generator 180 to control the generation of the EM field around patient 130.

[0023] The control system 140 includes various I / O devices configured to assist the physician 160 or other person in performing a medical procedure. In this example, the control system 140 includes I / O device(s) 146 used by the physician 160 or other user to control the scope 120, such as to navigate the scope 120 within the patient 130. For example, the physician 160 may provide input via the I / O device(s) 146, in response to which the control system 140 may send control signals to the robotic system 110 to operate the scope 120. Although the I / O device(s) 146 are illustrated as controllers in the example of FIG. 1, the I / O device(s) 146 may be implemented as various types of I / O devices, such as a touch screen, a touch pad, a mouse, a keyboard, a surgeon's or physician's console, virtual reality hardware, augmented hardware, a microphone, a speaker, a haptic device, etc.

[0024] As also shown in FIG. 1 , the control system 140 may include display(s) 142 to provide various information regarding the procedure. As described above, the display(s) 142 may present a visualization interface 144 to assist the physician 160 in the percutaneous access procedure (e.g., maneuvering the needle 170 toward the target site). The display(s) 142 may also provide information regarding the scope 120 (e.g., via the visualization interface 144 and / or another interface). For example, the control system 140 may receive real-time images captured by the scope 120 and display the real-time images via the display(s) 142. Additionally or alternatively, the control system 140 may receive signals (e.g., analog, digital, electrical, acoustic / sonic, pneumatic, tactile, hydraulic, etc.) from medical monitors and / or sensors associated with the patient 130, while the display(s) 142 may present information regarding the health or environment of the patient 130. Such information may include information displayed via a medical monitor, including, for example, heart rate (e.g., electrocardiogram (ECG), Heart Rate Variability (HRV), etc.), blood pressure / rate, muscle biosignals (e.g., electromyogram (EMG)), body temperature, blood oxygen saturation (e.g., saturation of percutaneous oxygen (SpO2)), CO2, brain waves (e.g., electroencephalogram (EEG)), environmental and / or local or core body temperature, etc.

[0025] To facilitate the functioning of control system 140, control system 140 may include various components (which may be referred to as "subsystems"). For example, control system 140 may include control electronics / circuitry, as well as one or more power sources, pneumatics, light sources, actuators, memory / data storage devices, and / or communication interfaces. In some embodiments, control system 140 includes control circuitry, including a computer-based control system configured to store executable instructions that, when executed, cause various operations to be implemented. In some embodiments, as shown in FIG. 1, control system 140 is mobile, while in other embodiments, control system 140 is a stationary system. Although various functions and components are discussed as being implemented by control system 140, any of the functions and / or components may be integrated into and / or performed by other systems and / or devices, such as robotic system 110, platform 150, and / or EM generator 180 (or even scope 120 and / or needle 170). Exemplary details of the control system 140 are discussed in further detail below with reference to FIG.

[0026] Imaging device 190 may be configured to capture / generate one or more images of patient 130 during a procedure, such as one or more X-ray or CT images. In an embodiment, images from imaging device 190 may be provided in real time to view anatomical structures and / or medical instruments, such as scope 120 and / or needle 170, within patient 130 to assist physician 160 in performing a procedure. Imaging device 190 may be used to perform fluoroscopy (e.g., with a contrast agent within patient 130) or another type of imaging technique.

[0027] The various components of the two-dimensional image registration system 100 may be communicatively coupled to one another over a network, which may include a wireless network and / or a wired network. Exemplary networks include one or more Personal Area Networks (PANs), Local Area Networks (LANs), Wide Area Networks (WANs), Internet Area Networks (IANs), cellular networks, the Internet, etc. Additionally, in some embodiments, the components of the two-dimensional image registration system 100 are connected for data communication, fluid / gas exchange, power exchange, etc., via one or more supporting cables, tubes, etc.

[0028] Although various techniques and systems are discussed as being implemented as robotically-assisted procedures (e.g., procedures that at least partially use the two-dimensional image registration system 100), these techniques and systems may be implemented in other procedures, such as fully robotic medical procedures, human-only procedures (e.g., procedures that do not include a robotic system), etc. For example, the two-dimensional image registration system 100 may be used to perform a procedure (e.g., a fully robotic procedure) without a physician holding / manipulating medical instruments. That is, medical instruments used during a procedure, such as the scope 120 and needle 170, may each be held / controlled by components of the two-dimensional image registration system 100, such as the robotic arm(s) 112 of the robotic system 110.

[0029] Two-dimensional image registration method and operation Details of the methods and operations of an exemplary two-dimensional image registration system are now described. The methods and operations disclosed herein are described with reference to the two-dimensional image registration system 100 shown in Figure 1. However, it should be understood that the methods and operations may be performed by any of the components discussed herein, alone or in combination.

[0030] As described above, the two-dimensional image registration system may register the coordinate frame of a two-dimensional image (e.g., an image acquired from a fluoroscope) with the three-dimensional data of the robotic system. In some cases, the two-dimensional image used for registration may contain additional information that is difficult or desirable to obtain at a later stage of the medical procedure. This includes fluoroscopic images acquired with contrast, which may identify anatomical structures that are not visible, or at least are poorly visible, in non-contrast fluoroscopic images. Once registered, the two-dimensional image registration system may render the two-dimensional image using information derived from the three-dimensional data of the robotic system. In this manner, the two-dimensional image registration system may provide the operator with a registered two-dimensional image with information regarding the current position of the instrument, even if there is a time lag between when the two-dimensional image is acquired and when the position of the instrument is determined.

[0031] FIG. 3 is a flow chart illustrating a method 300 of adding positional information of an instrument to a two-dimensional image, according to an exemplary embodiment. As used herein, “positional information” may refer to any suitable component or combination of position or orientation. As FIG. 3 illustrates, the method 300 may begin at block 310, where a two-dimensional image registration system acquires two-dimensional image data generated by one or more imaging devices of the two-dimensional image registration system. For example, during a medical procedure, a fluoroscope may capture an image including a representation of the patient's anatomy. In some cases, a contrast agent may be introduced into the patient to increase the details of the anatomy captured by the fluoroscope. This may be the case for a pelvic image, which allows the fluoroscope to acquire the internal morphology of the kidney, such as the ureter, renal pelvis, or calyx.

[0032] At block 320, the two-dimensional image registration system may identify a first segment in the two-dimensional image data as corresponding to a portion of an anatomical structure. By way of example only and not limitation, if the two-dimensional image is a renal pelvis image, the portion of the anatomical structure may correspond to at least one of a ureter, a renal pelvis, or a renal calyx. However, the portion of the anatomical structure may refer to any suitable portion of the anatomical structure for an anatomical structure other than a kidney.

[0033] It should be appreciated that the two-dimensional image registration system may identify additional segments other than the first segment at block 320. For example, the two-dimensional image registration system may identify multiple segments within a pyelogram, each of the multiple segments corresponding to a different portion of an anatomical structure, such as a ureter, a renal pelvis, or a renal calyx.

[0034] The two-dimensional image registration system may identify segments from the two-dimensional image data according to a variety of techniques. For example, in one embodiment, the two-dimensional image registration system may utilize a neural network (e.g., a convolutional neural network with a U-net architecture) that has learned consistent intensity patterns that define the anatomical structure of interest. As another example, the two-dimensional image registration system may provide a user interface that receives user input on the boundaries of some of the anatomical structures. Such user input may in some cases define the boundaries themselves, or in other cases correct or otherwise modify the segments that were automatically generated by the two-dimensional image registration system based on the neural network processing.

[0035] 4 is a diagram illustrating an example of a two-dimensional image with segmentation data 400, according to an example embodiment. The segmentation data may include a segmentation 402 for identifying the renal ureter and a segmentation 404 for identifying the renal pelvis and calyces.

[0036] Referring back to FIG. 3, in block 330, the two-dimensional image registration system may acquire position sensor data of the instrument. The position sensor data may indicate the position of the instrument moving within the anatomical structure over a first time period. For example, the position sensor data may be sensor data derived from EM sensors (Electromagnetic sensors), shape sensing fibers, accelerometers, magnetometers, gyroscopes, etc. It should be understood that such position sensor data may be expressed according to a coordinate frame different from that of the imaging device. In relation to block 330, the first time period may be when the system first begins collecting position data for the acquisition of the two-dimensional image data. In another example, the first time period may correspond to a time period during which an operator drives and "tags" a particular anatomical feature.

[0037] In some embodiments, the two-dimensional image registration system may acquire additional data regarding the position of the instrument. For example, the two-dimensional image registration system may acquire robotic data (e.g., kinematic data derived from commanded movements of a robotic arm). Alternatively or additionally, the two-dimensional image registration system may acquire tagged data. The tagged data may refer to automatic / user-initiated data that designates a particular location in the position sensor space having a determinable anatomical structure. For example, an operator may drive an instrument scope to touch a particular renal calyx, and in response to input from the operator, the system may tag the location identified as the particular renal calyx in the position sensor space.

[0038] In block 340, the two-dimensional image registration system uses the position sensor data and the first segment to determine a transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame. As used herein, a transformation may be data or logic that maps one coordinate frame to another. For block 340, the transformation may map positions from the position sensor coordinate frame to the two-dimensional image data coordinate frame. Once the two-dimensional image registration system has completed block 340 and the position sensor and two-dimensional image coordinate frames are aligned with each other, the two-dimensional image registration system can map positions from the instrument to the two-dimensional image.

[0039] As will be explained in more detail below, the first step for automatic registration is to move the two-dimensional image and the three-dimensional position sensor data into the same dimension. To move the two-dimensional data into the three-dimensional dimension, the system 100 may add a dummy dimension to the two-dimensional image, but the problem then becomes aligning the three-dimensional position sensor data with the three-dimensional plane that represents the two-dimensional image.

[0040] To move the position sensor data to the second dimension, the system determines the angle at which the two-dimensional image was taken. One solution is to obtain the angle explicitly from the imaging device. An alternative solution is to assume that the two-dimensional image was taken from a "visibility" angle, and thus orient the position sensor data to the "visibility" angle, i.e., orient the EM points according to their eigenvalues. Once the registration dimensions are unified, the position sensor data is registered to the two-dimensional image by combining AI-based pyelogram annotation, rigid and non-rigid alignment of 3D point clouds (e.g., coherent point drift), image filters for highlighting and segmenting tubular structures (e.g., Frangi filter algorithm), and any other suitable techniques.

[0041] With regard to determining the "visible" angle, the system may use any number of techniques. For example, some systems may look at a pre-op CT and see the angle of the renal plane relative to the bed. Assume the angulation is 10 degrees anterior. In an atypical supine position, the patient may be tilted 15 degrees to expose the flank. The viewing angle may then be a function of those, e.g., 10+15=25 degrees. Other systems may instead find the principal axis in the position sensor trace to find the renal plane (essentially fitting a plane to the position sensor data). The system may know where the bed is relative to the position sensor space (the cart is parallel to the bed and the robot is holding the CFG).

[0042] Once the position sensor coordinate frame and the two-dimensional image coordinate frame are aligned, the two-dimensional image registration system may begin to augment the two-dimensional image with information regarding the position of the instrument. For example, in block 350, the two-dimensional image registration system may determine an updated position of the instrument based on additional position sensor data generated from the position sensor. In some embodiments, the additional position sensor data may be generated for a time period after the first time period or may reference an immediately preceding time period.

[0043] In block 360, the two-dimensional image registration system may display data indicative of the updated position within the two-dimensional image based on the transformation generated in block 340. As described above, the updated position may refer to the position of the instrument after the position sensor coordinate frame and the two-dimensional image are registered. In some cases, the updated position may refer to the most recent position of the instrument. Thus, at the end of block 360, the two-dimensional image registration system has augmented the two-dimensional image with the current position data of the instrument. This may be beneficial because the two-dimensional image may include additional details (e.g., contrast agent) that would not normally be present if the two-dimensional image were retaken at a time consistent with the instrument being in the updated position. Some embodiments may display the data indicative of the updated position as a model of the instrument. Other embodiments may display the data indicative of the updated position as an icon representing position information such as position and / or orientation.

[0044] It should be appreciated that positions other than the updated position may be displayed within the two-dimensional image based on the transformation generated at block 340. For example, some embodiments of the two-dimensional image registration system may cause the historical position of the instrument to be displayed within the two-dimensional image based on the transformation generated at block 340. For example, once the two-dimensional image registration system generates the transformation at block 340, the two-dimensional image registration system may cause some or all of the position sensor data acquired at block 330 to be displayed within the two-dimensional image. In this manner, the historical path of the instrument through the anatomy may be represented within the two-dimensional image.

[0045] The two-dimensional image registration system may represent the position data in the two-dimensional image using graphical icons in various ways. For example, the two-dimensional image registration system may represent the position data as discrete graphical icons, such as dots, squares, arrows, or any other graphical icon, possibly spaced according to frequency, whereby more closely spaced graphical icons represent instruments moving along a path at a slower speed. Additionally or alternatively, the two-dimensional image registration system may represent the position data as lines designating a path. In any of these embodiments, the two-dimensional image registration system may use different characteristics to distinguish different aspects of the procedure. For example, the two-dimensional image registration system may use a first type of graphical icon to designate a first instrument and a second, different type of graphical icon to designate a second instrument. In addition, the two-dimensional image registration system may use one type of graphical icon to represent the path of the instrument and another type of graphical icon to represent user or system driven events, such as the user tagging an anatomical structure or the instrument being in a particular state (e.g., lasing, taking a biopsy, delivering a therapeutic agent).

[0046] FIG. 2 provides an example embodiment showing one type of graphical icon (e.g., a dot) representing the path of the instrument and another type of graphical icon (e.g., a crosshair) representing the calyx tagged by the operator.

[0047] Alignment Method 300 describes in block 340 using the position sensor data and the first segment to generate a transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame. This process may be referred to as registration. An exemplary embodiment of registration will now be described in more detail. In the context of the embodiments described herein, registration may be defined as aligning the three-dimensional position sensor data with the two-dimensional fluoroscopic image data. Registration may include several steps, such as: (1) segmenting the anatomical structures shown in the two-dimensional image, (2) generating an anatomical structure level set from the segmentation results, and (3) aligning the three-dimensional position sensor data with the two-dimensional image using a level set-based distance map. To simplify the description of these steps, the embodiments are described in the context of renal anatomy, but it should be understood that any suitable anatomical structures may be segmented using these approaches. Furthermore, while the following description focuses on position sensor data, other embodiments may include any additional system data for identifying the position and positioning of the instrument, such as robotic data.

[0048] i. Segmentation Segmentation of kidney tissues may be based on machine learning methods used by the 2D image registration system. A database of kidney fluoroscopy with contrast is collected and manually annotated with all tissues of interest. These fluoroscopic images are normalized to compensate for intensity variations, noise, different resolutions, etc. An encoder-decoder neural network designed for pixel-wise image segmentation, referred to herein as a "segmentation network," is trained on the normalized images from the database to learn the appearance of kidney tissues.

[0049] A new fluoroscopy is obtained in preparation for the percutaneous nephropathy procedure. This fluoroscopy image is normalized and then processed by the previously trained segmentation network. The result of the segmentation network is a mask of the ureter, renal pelvis, and calyces generated for the new fluoroscopy image. The resulting segmentation mask is the same size as the new fluoroscopy.

[0050] ii. Level Set The kidney level set is generated from the kidney tissue segmentation. All pixels corresponding to the outer boundary of the kidney tissue will have a value of 0 on the level set. All pixels outside the segmented kidney tissue will have a negative value that encodes the negative distance to the nearest pixel that belongs to the kidney segmentation boundary. All pixels located within the kidney tissue segmentation will have a positive value that encodes the distance to the nearest pixel that belongs to the kidney segmentation boundary. The "deeper inside" a pixel is in the kidney, the higher its value in the level set.

[0051] iii. Alignment Alignment of the 3D sensor data with the 2D fluoroscopy uses a renal level set. The objective of the present invention is to position the 3D sensor data such that the sensor coordinate point passes through the pixel of the level set that has the highest sum. To achieve this, the 2D image registration system may perform several steps, including:

[0052] A. The two-dimensional image registration system may start with an initial guess where the sensor is currently located within the kidney. There are several ways this initial guess may be achieved. For example, the two-dimensional image registration system may set the initial guess to a known location within the anatomy, such as the lowest point of the ureter on fluoroscopy. Alternatively, the two-dimensional image registration system may prompt the operator to position the instrument at a known location within the anatomy.

[0053] B. The two-dimensional image registration system may then define a set of allowable transformations. The allowable translation of the sensor data across the two-dimensional image may be unlimited. The allowable in-plane and out-of-plane rotations of the sensor data are constrained using the standard positioning of the patient, robot, and fluoroscopic information. Note that the constrained rotations do not mean that the three-dimensional sensor data cannot be rotated, but rather the two-dimensional image registration system may have some limitations on the possible rotations, such that the three-dimensional sensor data is not rotated 180° during this procedure. Scaling is also constrained by the standard positioning of the depicted objects.

[0054] Using the initial guess, the 2D image registration system projects the 3D sensor data onto the 2D fluoroscopy, i.e., removing the dimension oriented along the fluoroscopy line. The 2D image registration system then calculates the sum of all projected sensor points across the kidney level set. The 2D image registration system then updates the initial guess according to the allowable transformations to improve the positioning of the projected 3D points. The update may be performed using a gradient descent algorithm that maximizes the sum of all projected sensor points.

[0055] Subsequent 2D image expansion Generating segmented data on anatomy from contrast-enhanced two-dimensional images may have additional applications for a procedure. For example, a two-dimensional image registration system may acquire two-dimensional images later in a procedure, but because these subsequent two-dimensional images may be taken without administering contrast to the patient, these subsequent two-dimensional images may lack the anatomical details found in the segmented two-dimensional images. At the same time, such non-contrast-enhanced two-dimensional images may have visible instruments. The two-dimensional image registration system may use the segmented anatomy to augment the non-contrast-enhanced two-dimensional images by overlaying the depicted instruments with anatomical details obtained from the anatomy segmentation. To do so, the two-dimensional image registration system may (1) segment the instruments (and component parts such as the scope tip) from the non-contrast images, (2) estimate the anatomical resolution of the fluoroscopy, and (3) register a previously acquired fluoroscopy with contrast to the segmented instruments of the fluoroscopy without contrast.

[0056] i. Instrument segmentation Segmentation of the scope and possibly its component parts (e.g., instrument tips) may involve methodologies similar to those described above for segmenting anatomical structures. For example, non-contrast 2D images may be processed by a neural network trained on a database of annotated 2D images that identifies instruments within the 2D images.

[0057] The result of the segmentation is an instrument mask, which provides the coordinates and orientation of the instrument tip.

[0058] ii. Estimation of anatomical resolution To estimate the resolution of the depicted structures in millimeters per pixel, the two-dimensional image registration system first determines the centerline of the instrument segmentation. For points along the centerline of the instrument, the two-dimensional image registration system finds the normal direction, i.e., the direction perpendicular to the centerline. The distance between the furthest points segmented as instruments along the normal direction is interpreted by the two-dimensional image registration system as the radius of the scope at the centerline point. By calculating the radius of the instrument for all centerline points, the two-dimensional image registration system determines the average radius of the instrument as it may be measured in pixels. By normalizing this average radius to the known radius of the instrument, the two-dimensional image registration system determines the anatomical-to-fluorescence resolution, i.e., how many millimeters of kidney tissue are within one pixel. In one embodiment, the two-dimensional image registration system may obtain the known radius of the instrument via calibration parameters sent to the control system when the instrument is docked to the robotic arm. In other embodiments, the two-dimensional image registration system may obtain the known radius based on an identification of the instrument received from the operator and a lookup table that maps the instrument to characteristics such as instrument measurements. In other embodiments, the 2D registration system can compare the size and shape of the instrument tip from known instrument characteristics and tip segmentation results. This information can be combined with the radius analysis to improve the accuracy of the resolution estimation.

[0059] iii. Alignment of segmented data The segmented instrument in the non-contrast image should fit inside the patient's anatomy. Considering tissue elasticity and the time elapsed between the contrast fluoroscopy acquisition and the non-contrast fluoroscopy with instrument acquisition, the instrument is expected to be positioned as inside the segmented anatomy derived from the contrast image as possible. In some embodiments, the two-dimensional image registration system may utilize the possible articulations of the instrument and the general shape of the anatomy (e.g., in the context of the kidney, ureter) to constrain the possible positions of the instrument within the anatomy. This positioning is obtained using a simplified version of the algorithm for the three-dimensional position sensor to two-dimensional image registration described above. The simplification comes from the fact that the segmented instrument is already two-dimensional, in contrast to the three-dimensional position sensor data. The two-dimensional image registration system may use this to constrain the allowable transformations. In particular, the two-dimensional image registration system may constrain the scaling based on 1) constraining out-of-plane rotations and 2) constraining in-plane rotations based on the assumption that the imaging device was not moved during the procedure. Based on this, a two-dimensional image registration system may ultimately involve translation as well as some small scaling and in-plane rotation.

[0060] iv. Enhanced non-contrast image rendering After registering the segmented instrument with the segmented anatomical structure, the two-dimensional image registration system may augment the non-contrast two-dimensional image with the previously acquired anatomical structure segmentation. This augmented non-contrast two-dimensional image is then rendered on a display device for the operator of the two-dimensional image registration system. The enhanced non-contrast two-dimensional image shows the operator where the anatomical boundaries are located relative to the instrument. Another potential advantage is that the non-contrast fluorescence analysis may improve the initial guess for the fluorescence registration that the two-dimensional image registration system described above. FIG. 5 is a diagram illustrating an exemplary augmented non-contrast image 500, according to an exemplary embodiment. The augmented non-contrast image 500 may include two-dimensional image data 502 with anatomical segmentation data 504 superimposed thereon. As described, the anatomical structure segmentation data 504 may be data derived from the contrasted two-dimensional image data where the system segments the anatomical structure.

[0061] Mounting system and terminology Implementations disclosed herein provide systems, methods, and devices for enhancing two-dimensional images. Various implementations described herein provide improved visualization of one or more medical instruments performing a medical procedure.

[0062] The two-dimensional image registration system 100 may include various other components. For example, the two-dimensional image registration system 100 may include one or more control circuits, power sources, pneumatics, light sources, actuators (e.g., motors for moving a robotic arm), memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory may store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to perform any of the operations discussed herein. For example, the memory may store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to receive inputs and / or control signals related to the operation of a robotic arm and, in response, control the robotic arm to position it in a particular arrangement.

[0063] The various components of the two-dimensional image registration system 100 may be electrically and / or communicatively coupled using certain connection circuits / devices / functions that may or may not be part of the control circuitry. For example, the connection feature(s) may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuitry of the two-dimensional image registration system 100. In some embodiments, two or more of the control circuitry, data storage / memory, communication interface, power supply unit(s), and / or input / output (I / O) component(s) may be electrically and / or communicatively coupled to one another.

[0064] The term "control circuitry" is used herein according to its broad and ordinary meaning and may refer to any collection of the following: one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connection circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field programmable gate arrays, programmable logic circuits, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any device that manipulates signals (analog and / or digital) based on hard-coding of circuitry and / or operational instructions. The control circuitry may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuits of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments in which the control circuitry includes a hardware state machine (and / or implements a software state machine) and includes analog, digital, and / or logic circuitry, the data storage device(s) / register(s) that store any associated operational instructions may be embedded within or external to the circuitry that includes the state machine, analog, digital, and / or logic circuitry.

[0065] The term "memory" is used herein according to its broad and ordinary meaning and may refer to any suitable or desirable type of computer-readable medium, including, for example, one or more volatile, non-volatile, removable, and / or non-removable data storage devices implemented using any technology, layout, and / or data structure(s) / protocol that contain any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0066] Computer-readable media that may be implemented with embodiments of the present disclosure include, but are not limited to, phase-change memory, Static Random-Access Memory (SRAM), Dynamic Random-Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disk Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transitory medium that may be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media may generally not include communication media such as modulated data signals and carrier waves. Accordingly, computer readable media should be understood to generally refer to non-transitory media.

[0067] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the algorithms or processes described herein may be performed in a different order, added, merged, or omitted entirely, and thus, in a particular embodiment, not all of the described acts or events are necessary to the performance of a process.

[0068] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not intended to imply that features, elements, and / or steps are generally required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like, are synonymous and used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z," is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is not generally intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0069] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the method of disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment(s). Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Thus, it is intended that the scope of the present disclosure should not be limited by the particular embodiments described above, but should be determined solely by a fair reading of the following claims.

[0070] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, component, operation, etc., do not necessarily indicate a priority or order of the element with respect to any other elements, but rather may generally distinguish the element from another element having a similar or identical name (apart from the use of the ordinal term). Note that, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0071] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0072] Spatially relative terms such as "outer", "inner", "upper", "lower", "below", "upper", "vertical", "horizontal", and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the figures. It should be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation, yet in the orientation depicted in the figures. For example, if the devices shown in the figures are inverted, a device positioned "below" or "beneath" another device may be positioned "above" the other device. Thus, the illustrative term "lower" may include both lower and upper positions. Devices may also be oriented in other directions, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0073] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater," and the like are intended to encompass the notion of equality. For example, "less" can mean not only "less" in the strict mathematical sense, but also "less than or equal to."

[0074] [Embodiment] (1) A method for augmenting a two-dimensional image using instrument position information, the method comprising: acquiring two-dimensional image data generated by one or more imaging devices of a medical system, the two-dimensional image data corresponding to a position sensor coordinate frame; identifying a first segment of an anatomical structure in the two-dimensional image data as corresponding to a portion of the anatomical structure, the two-dimensional image data corresponding to a two-dimensional image data coordinate frame; acquiring position sensor data of the instrument from a position sensor, the position sensor data indicating a position of the instrument moving within the anatomical structure over a first time period; determining a transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame using the position sensor data and the first segment of the anatomical structure; determining an updated position of the instrument using additional position sensor data generated from the position sensor; and and using the transformation to cause data indicative of the updated position to be displayed within the two-dimensional image. (2) The method of embodiment 1, wherein the one or more imaging devices include a fluoroscopy device. (3) The method of embodiment 1, wherein the anatomical structure includes a kidney. (4) The method of embodiment 3, wherein the portion is one of a ureter, a renal pelvis, or a renal calyx. (5) The method of embodiment 1, wherein the position sensor includes at least one of an electromagnetic sensor, a shape sensing fiber, a gyroscope, an accelerometer, or a magnetometer.

[0075] (6) The method of embodiment 1, wherein the two-dimensional imaging data is a renal pelvis image. (7) The method of embodiment 1, wherein the two-dimensional imaging data is acquired preoperatively. (8) The method of embodiment 7, wherein the position sensor data is acquired during surgery. (9) determining the transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame, generating a renal map using the position sensor data; and 2. The method of claim 1, further comprising registering the two-dimensional imaging data with the renal map. (10) acquiring angles associated with the one or more imaging devices relative to the anatomical structure; and 10. The method of claim 9, further comprising aligning the position sensor coordinate frame and the two-dimensional image data coordinate frame based on the angle.

[0076] (11) The method of embodiment 10, wherein obtaining the angle further comprises receiving the angle from the one or more imaging devices. (12) The method of embodiment 10, wherein obtaining the angle further comprises determining a desired angle of the one or more imaging devices. (13) The method of embodiment 1, wherein determining the transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame further comprises generating a three-dimensional representation of the two-dimensional image data. (14) acquiring non-contrast enhanced two-dimensional data from the one or more imaging devices at a time period after acquiring the two-dimensional image data; Determining an instrument shape; 2. The method of claim 1, further comprising: rendering the non-contrast enhanced two-dimensional data on a display device together with a representation of the first segment based on the instrument shape. (15) The method of embodiment 14, further comprising acquiring additional position sensor data for the instrument, the additional position sensor data indicating a position of the instrument moving within the anatomical structure over a second time period, and determining the instrument shape is based on the additional position sensor data.

[0077] (16) The method of embodiment 14, further comprising identifying a second segment in the non-contrast two-dimensional image data as corresponding to the instrument, and determining the instrument shape is based on the second segment. (17) A medical system capable of augmenting a two-dimensional image using position information of an instrument, the medical system comprising: one or more imaging devices that generate two-dimensional image data corresponding to a position sensor coordinate frame; A memory storing computer-executable instructions that, when executed by control circuitry, cause the control circuitry to: identifying a first segment of an anatomical structure in the two-dimensional image data as corresponding to a portion of the anatomical structure, the two-dimensional image data corresponding to a two-dimensional image data coordinate frame; acquiring position sensor data of the instrument from a position sensor, the position sensor data indicating a position of the instrument moving within the anatomical structure over a first time period; determining a transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame using the position sensor data and the first segment of the anatomical structure; determining an updated position of the instrument using additional position sensor data generated from the position sensor; and and displaying data indicative of the updated position within the two-dimensional image using the transformation. (18) The medical system of embodiment 17, wherein the one or more imaging devices include a fluoroscopy device. (19) The medical system of embodiment 17, wherein the anatomical structure includes a kidney. (20) The medical system of embodiment 19, wherein the portion is one of a ureter, a renal pelvis, or a renal calyx.

[0078] (21) The medical system of embodiment 17, wherein the position sensor includes at least one of an electromagnetic sensor, a shape sensing fiber, a gyroscope, an accelerometer, or a magnetometer. (22) The medical system described in embodiment 17, wherein the two-dimensional imaging data is a renal pelvis image. (23) The medical system of embodiment 17, wherein the two-dimensional imaging data is acquired preoperatively. (24) The medical system of embodiment 23, wherein the position sensor data is acquired during surgery. (25) determining the transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame, generating a renal map using the position sensor data; and 18. The medical system of claim 17, further comprising registering the two-dimensional imaging data with the renal map.

[0079] (26) The computer executable instructions further cause the control circuit to: obtaining an angle associated with the one or more imaging devices relative to the anatomical structure; 26. The medical system of claim 25, further comprising: aligning the position sensor coordinate frame and the two-dimensional image data coordinate frame based on the angle. (27) The medical system of embodiment 26, wherein obtaining the angle further comprises receiving the angle from the one or more imaging devices. (28) The medical system of embodiment 26, wherein obtaining the angle further comprises determining a desired angle of the one or more imaging devices. (29) The medical system of claim 17, wherein determining the transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame further comprises generating a three-dimensional representation of the two-dimensional image data. (30) The computer executable instructions further cause the control circuit to: acquiring non-contrast enhanced two-dimensional data from the one or more imaging devices at a time period after acquiring the two-dimensional image data; Determining an instrument shape; 18. The medical system of claim 17, further comprising: rendering the non-contrast two-dimensional data on a display device together with a representation of the first segment based on the instrument shape.

[0080] (31) The computer executable instructions further include causing the control circuit to: The medical system of embodiment 30, further comprising: acquiring additional position sensor data of the instrument, the additional position sensor data indicating a position of the instrument moving within the anatomical structure over a second period of time; and determining the instrument shape based on the additional position sensor data. (32) The computer executable instructions further cause the control circuit to: The medical system of embodiment 30, further comprising: identifying a second segment in the non-contrast two-dimensional image data as corresponding to the instrument; and determining the instrument shape based on the second segment. (33) A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed, causing a processor of a device to perform at least: acquiring two-dimensional image data generated by one or more imaging devices of a medical system, the two-dimensional image data corresponding to a position sensor coordinate frame; identifying a first segment of an anatomical structure in the two-dimensional image data as corresponding to a portion of the anatomical structure, the two-dimensional image data corresponding to a two-dimensional image data coordinate frame; acquiring position sensor data for an instrument from a position sensor, the position sensor data indicating a position of the instrument moving within the anatomical structure over a first time period; determining a transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame using the position sensor data and the first segment of the anatomical structure; determining an updated position of the instrument using additional position sensor data generated from the position sensor; and and causing data indicative of the updated position to be displayed within the two-dimensional image using the transformation. (34) The non-transitory computer-readable storage medium of embodiment 33, wherein the one or more imaging devices include a fluoroscopy device. (35) The non-transitory computer-readable storage medium of embodiment 33, wherein the anatomical structure includes a kidney.

[0081] (36) The non-transitory computer-readable storage medium of embodiment 35, wherein the portion is one of a ureter, a renal pelvis, or a renal calyx. (37) The non-transitory computer-readable storage medium of embodiment 33, wherein the position sensor includes at least one of an electromagnetic sensor, a shape-sensing fiber, a gyroscope, an accelerometer, or a magnetometer. (38) The non-transitory computer-readable storage medium of embodiment 33, wherein the two-dimensional imaging data is a renal pelvis image. (39) The non-transitory computer-readable storage medium of embodiment 33, wherein the two-dimensional imaging data is acquired pre-operatively. (40) The non-transitory computer-readable storage medium of embodiment 39, wherein the position sensor data is acquired during surgery.

[0082] (41) Determining the transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame comprises: generating a renal map using the position sensor data; and 34. The non-transitory computer-readable storage medium of embodiment 33, further comprising registering the two-dimensional imaging data with the renal map. (42) The instructions further include: obtaining an angle associated with the one or more imaging devices relative to the anatomical structure; 42. The non-transitory computer-readable storage medium of claim 41, further comprising: aligning the position sensor coordinate frame and the two-dimensional image data coordinate frame based on the angle. (43) The non-transitory computer-readable storage medium of embodiment 42, wherein obtaining the angle further comprises receiving the angle from the one or more imaging devices. (44) The non-transitory computer-readable storage medium of embodiment 42, wherein obtaining the angle further comprises determining a desired angle of the one or more imaging devices. (45) The non-transitory computer-readable storage medium of embodiment 33, wherein determining the transformation between the position sensor coordinate frame and the two-dimensional image data coordinate frame further comprises generating a three-dimensional representation of the two-dimensional image data.

[0083] (46) The instructions further include: acquiring non-contrast enhanced two-dimensional data from the one or more imaging devices at a time period after acquiring the two-dimensional image data; Determining an instrument shape; 34. The non-transitory computer-readable storage medium of embodiment 33, further comprising: rendering the non-contrast two-dimensional data on a display device together with a representation of the first segment based on the instrument shape. (47) The instructions further include: A non-transitory computer-readable storage medium as described in embodiment 46, wherein additional position sensor data of the instrument is acquired, the additional position sensor data indicating a position of the instrument moving within the anatomical structure over a second period of time, and determining the instrument shape is based on the additional position sensor data. (48) The non-transitory computer-readable storage medium of embodiment 46, wherein the instructions further cause the processor to identify a second segment in the non-contrast two-dimensional image data as corresponding to the instrument, and wherein determining the instrument shape is based on the second segment.

Claims

1. A method for enhancing an image using location information, said method comprising: acquiring, from an imaging device, a two-dimensional (2D) image of the anatomical structure associated with a 2D coordinate frame; using a neural network to identify segments of the anatomical structure in the 2D image; acquiring first position sensor data from a position sensor associated with an instrument positioned within the anatomical structure, the first position sensor data indicative of one or more first poses of the instrument in a three-dimensional (3D) coordinate frame; determining a transformation between the 3D coordinate frame and the 2D coordinate frame based on the segment of the anatomical structure and the one or more first poses of the instrument, the transformation mapping a pose of the instrument in the 3D coordinate frame to the 2D coordinate frame; acquiring second position sensor data from the position sensor indicative of a second pose of the instrument in the 3D coordinate frame; and mapping the second pose of the instrument onto the 2D image based on the transformation, such that the 2D image shows the second pose of the instrument relative to the segment of the anatomical structure.

2. Mapping the one or more first orientations of the instrument based on the transformation to history indicators on the 2D image, each of the history indicators representing a corresponding previous position and / or orientation of the instrument relative to the segment of the anatomical structure; 2. The method of claim 1, further comprising displaying an instrument indicator on the 2D image simultaneously with displaying the history indicator on the 2D image, the instrument indicator and the history indicator together representing a path of the instrument.

3. The method of claim 1 , wherein the neural network is a convolutional neural network having a U-Net architecture.

4. The method of claim 1, wherein the segment of the anatomical structure is a kidney; determining the transformation generating a renal map based on the first position sensor data; and registering the renal map with the 2D image.

5. The method of claim 1, wherein determining the transformation comprises: determining an angle between the imaging device and the anatomical structure; and aligning the 3D coordinate frame with the 2D coordinate frame based on the determined angle.

6. The method of claim 1, wherein determining the transformation includes generating a 3D representation of the 2D image based at least in part on inserting the 2D image as a plane in an artificial 3D space.

7. Obtaining a non-contrast image of the anatomical structure depicting the instrument positioned within the anatomical structure; determining a shape of the instrument in the non-contrast image; The method of claim 1 , further comprising: mapping the identified segments of the anatomical structure to the non-contrast image based at least in part on the shape of the instrument.

8. The method of claim 7, further comprising obtaining third position sensor data from the position sensor indicating a third orientation of the instrument in the 3D coordinate frame, wherein the shape of the instrument is determined based at least in part on the third orientation of the instrument.

9. 8. The method of claim 7, further comprising identifying segments of the instrument in the non-contrast image, wherein the shape of the instrument is determined based at least in part on the identified segments of the instrument in the non-contrast image.

10. A medical system, comprising: an imaging device configured to capture a two-dimensional (2D) image of the anatomical structure associated with a 2D coordinate frame; a control circuit, wherein the control circuit using a neural network to identify segments of the anatomical structure in the 2D image; acquiring first position sensor data from a position sensor associated with an instrument positioned within the anatomical structure, the first position sensor data indicative of one or more first poses of the instrument in a three-dimensional (3D) coordinate frame; determining a transformation between the 3D coordinate frame and the 2D coordinate frame based on the segment of the anatomical structure and the one or more first poses of the instrument, the transformation mapping a pose of the instrument in the 3D coordinate frame to the 2D coordinate frame; acquiring second position sensor data from the position sensor indicative of a second pose of the instrument in the 3D coordinate frame; and mapping the second pose of the instrument onto the 2D image based on the transformation, such that the 2D image shows the second pose of the instrument relative to the segment of the anatomical structure.

Images