Systems, devices, and methods for three-dimensional image registration

Abstract

The system includes a medical device and a computer-readable medium storing instructions that, when executed by a processor, cause operations to be performed, including receiving a three-dimensional (3D) image of an anatomical structure, processing the 3D image to extract a 3D model that identifies multiple anatomical sites of interest, receiving multiple images captured by an imaging device of the medical device that include a current image of the target site as the medical device advances through a body lumen to the target site, receiving spatial information of the medical device from a position sensing system that includes a transmitter / receiver in or on the medical device, processing the images and spatial information to generate a 3D surface map of the body lumen, registering the 3D model to the patient using the map and spatial information, and generating and displaying a graphical user interface that overlays a representation of the positions / trajectories of the multiple anatomical sites of interest on the current image.

Description

[Technical Field]

[0001] The present disclosure relates generally to systems, devices, and methods for three-dimensional (3D) image registration. More specifically, aspects of the present disclosure relate to devices, systems, and / or methods for 3D image registration that generate and display graphical user interfaces (GUIs) to facilitate medical procedures. [Background technology]

[0002] Endoscopic retrograde cholangiopancreatography (ERCP) is a procedure that utilizes endoscopy and fluoroscopy to diagnose and / or treat conditions in the biliary and pancreatic systems, such as strictures. During an exemplary ERCP, an endoscope can be inserted into the patient's mouth, down the esophagus, through the stomach, into the duodenum, and advanced to the papilla, where the common bile duct and pancreatic duct each open into the duodenum. Cannulation can then be performed by inserting a guidewire, catheter, and / or other device through the papilla into either the common bile duct or pancreatic duct. A contrast agent is then injected, allowing the operator to visualize the source of the stricture within the biliary or pancreatic system using fluoroscopy and on an x-ray image. The operator can then deliver and / or implement the appropriate treatment.

[0003] Cannulation during ERCP procedures is known to pose various challenges for gastroenterologists. Often, the inability to visualize the location or trajectory of the common bile duct and / or pancreatic duct makes cannulation difficult, making it difficult to insert guidewires, catheters, and / or other devices into the desired ducts (e.g., the bile duct rather than the pancreatic duct). Such difficulties resulting from the inability to visualize can extend the length of the procedure. Additionally, while most attempted ERCP procedures are successful in cannulation, some attempts fail, often necessitating the patient's referral to a tertiary care specialist. Even when successful, the act of cannulation can lead to post-ERCP pancreatitis if repeated attempts are made or excessive force is used. Summary of the Invention

[0004] A system for three-dimensional (3D) image registration to facilitate a medical procedure may include a medical device having a distal portion configured to be inserted into a body lumen of a patient during a medical procedure. The medical device may include an imaging device located at a distal tip of the distal portion and configured to capture multiple images of the body lumen as the medical device is inserted into and advanced through the body lumen to a target site. At least one of the multiple images may include a current image of the target site. The medical device may include a transmitting or receiving device of a position sensing system located at the distal tip, the position sensing system configured to determine a position or orientation of the distal tip. The system may include a non-transitory computer-readable medium storing multiple instructions that, when executed by a processor of a computing device, cause the processor to perform multiple operations. The operations may include receiving 3D images of the patient's anatomy imaged by an imaging system before the medical procedure and processing the 3D images to extract a 3D model identifying multiple anatomical sites within the anatomy, the multiple anatomical sites including one or more anatomical sites of interest for the medical procedure. The operations may also include receiving multiple images of the body lumen imaged by the imaging device during the medical procedure, receiving the position or orientation of the distal tip of the medical device from the position sensing system, and processing the multiple images of the body lumen and the position or orientation of the distal tip to generate a 3D surface map of at least a portion of the body lumen. The 3D surface map may include portions of the multiple anatomical sites that do not include the one or more anatomical sites of interest.The operations may also include registering the 3D model to the patient using the 3D surface map and the position or orientation of the distal tip; generating a graphical user interface (GUI) based on the registration that overlays a representation of the position or trajectory of the one or more anatomical structures of interest on the current image of the target region; and displaying the GUI on a display device.

[0005] In any of the example systems disclosed herein, the operations may include determining that one or more anatomical structure portions of the plurality of anatomical structure portions identified in the 3D model are incomplete and estimating the incomplete portions of the one or more anatomical structure portions of the plurality of anatomical structure portions. The operations may further include generating a prompt having instructions for an operator to confirm the estimated incomplete portions, and displaying the 3D model including the estimated incomplete portions and the prompt via the display device, where the prompt may be displayed in relation to the estimated incomplete portions of the 3D model. The operations may further include determining that additional image data is needed to generate the 3D surface map based on the number or type of anatomical structure sites in the portion of the plurality of anatomical structure sites included in the 3D surface map; generating a prompt having a plurality of instructions for an operator to move the medical device to one or more positions within the body lumen corresponding to one or more anatomical structure sites among the plurality of anatomical structure sites that are not included in the portion or are included but incomplete in the portion to acquire the additional image data via the imaging device of the medical device; and displaying the prompt using the display device.

[0006] In some embodiments, registering the 3D model to the patient may further include determining a transformation matrix and applying the transformation matrix to the 3D model to transform the 3D model. One or more of the portions of the anatomical structures included in the 3D surface map may be matched to corresponding anatomical structures in the 3D model, and based on the matching, an initial registration may be performed and a deformation compensation may be determined, and the transformation matrix may be determined based on the initial registration and the deformation compensation. The GUI may be generated using the transformed 3D model, and the representation of the position or the trajectory of the one or more anatomical regions of interest includes at least one of a portion of the transformed 3D model including the one or more anatomical regions of interest, a wireframe model of the one or more anatomical regions of interest, a representation of a centerline of the one or more anatomical regions of interest, a series of disks positioned perpendicular to the centerline of the one or more anatomical regions of interest, or a tubular structure of the one or more anatomical regions of interest.

[0007] In other aspects, the operations may include receiving spatial information about the patient from the position sensing system, the position sensing system including one or more transmitting or receiving devices located in a patch locally applied to the patient, and the 3D model may be registered to the patient using the 3D surface map, the spatial information about the medical device, and the spatial information about the patient. The operations may further include identifying one or more anatomical structures from the portion of the plurality of anatomical structures in the 3D surface map by providing the plurality of images as input to a machine learning model trained to predict the one or more anatomical structures present in each of the plurality of images. The operations may also include identifying one or more anatomical structures from the portion of the plurality of anatomical structures in the 3D surface map as part of the generation of the 3D surface map by mapping the geometry of the one or more anatomical structures as a 3D surface and identifying the anatomical structures based on the mapped geometry.

[0008] In a further aspect, the target site is a site for cannulation, and movement at the target site may be tracked during cannulation, and the GUI may be updated to deform the representation of the position or the trajectory of the one or more anatomical sites of interest overlaid on the current image of the target site to match the movement of the target site. Spatial information about a tool delivered to the target site via the medical device may be received from the position sensing system as the tool is advanced through at least one of the one or more anatomical sites of interest, the tool including one or more transmitting devices or one or more receiving devices of the position sensing system, and the GUI may be updated using the spatial information about the tool to depict a representation of the tool advancing through the representation of the position or the trajectory of the at least one of the one or more anatomical sites of interest overlaid on the current image of the target site.

[0009] In another aspect, in addition to the processing of the plurality of images to generate the 3D surface map, additional 3D images captured during a procedure may be received and processed after the medical device has reached the target site through the body lumen. Additionally, a determination may be made as to whether the image quality of the 3D images meets a predetermined threshold before processing the 3D images to extract the 3D model. The medical procedure may be an ERCP procedure. The target site may be a papilla for cannulation, and the one or more anatomical sites of interest may include at least the common bile duct and the pancreatic duct. A GUI generated using the registered 3D model may create an augmented reality image by overlaying the representation of the position or trajectory of the common bile duct and the pancreatic duct on the current image of the papilla.

[0010] In another example, a method for 3D image registration to facilitate a medical procedure may include receiving 3D images of a patient's anatomy and processing the 3D images to extract a 3D model identifying multiple anatomical sites within the anatomy, the multiple anatomical sites including one or more anatomical sites of interest for the medical procedure. The method may also include receiving multiple images of the patient's body lumen captured by an imaging device of the medical device as the medical device is inserted into the body lumen and advanced through the body lumen to a target site during the medical procedure. At least one of the multiple received images may include a current image of the target site. The method may further include receiving spatial information of the medical device from a position sensing system, the position sensing system may include a transmitting device or a receiving device located within or on the medical device, and processing the multiple images of the body lumen and the spatial information of the medical device to generate a 3D surface map of at least a portion of the body lumen. The 3D surface map may include a portion of the plurality of anatomical structures that does not include the one or more anatomical structures of interest. The method may further include registering the 3D model to the patient using the 3D surface map and the spatial information about the medical device. The registering may include determining a transformation matrix. The method may further include applying the transformation matrix to the 3D model to transform the 3D model, using the transformed 3D model to generate a GUI that overlays a representation of the position or trajectory of the one or more anatomical structures of interest on the current image of the target region, and displaying the GUI on a display device.

[0011] Any of the example methods disclosed herein may include any of the following features: spatial information about the patient may be received from the position sensing system, which may include one or more transmitting or receiving devices located in a patch locally applied to the patient; the 3D model may be registered to the patient using the 3D surface map, the spatial information about the medical device, and the spatial information about the patient; to register the 3D model to the patient, one or more of the portions of one or more anatomical structures included in the 3D surface map may be matched to corresponding anatomical structures in the 3D model; based on the matching, an initial registration may be performed, a deformation compensation may be determined, and the transformation matrix may be determined based on the initial registration and the deformation compensation.

[0012] In a further example, a method for 3D image registration to facilitate an ERCP procedure may include receiving 3D images of a patient's anatomy and processing the 3D images to extract a 3D model that identifies multiple anatomical sites within the anatomy, including multiple anatomical sites of the patient's upper gastrointestinal (GI) tract and biliary-pancreatic system, where at least the common bile duct and pancreatic duct are anatomical sites of interest for the ERCP procedure. The method may also include receiving multiple images of the upper GI tract as a medical device is inserted into the patient's mouth and travels through the patient's upper GI tract to a papilla during the ERCP procedure, the multiple images including a current image of the papilla captured by an imaging device of the medical device, the papilla being a target site for cannulation; receiving spatial information about the medical device from a position sensing system, the position sensing system including a transmitting device or a receiving device located within or on the medical device; and processing the multiple images and the spatial information about the medical device to generate a 3D surface map. The 3D surface map may include portions of a plurality of anatomical structures of the upper GI tract identified in the 3D model and the papilla. The method may also include registering the 3D model to the patient using the 3D surface map and spatial information about the medical device, generating a graphical user interface (GUI) based on the registration that overlays a representation of a position or trajectory of at least the common bile duct and the pancreatic duct on a current image of the papilla, and displaying the GUI on a display device to provide visual guidance for cannulation of the papilla.

[0013] Any of the example methods disclosed herein may include any of the following features: The movement of the papilla can be tracked as cannulation is performed, and the GUI can be updated to deform a representation of the position or trajectory of the common bile duct and pancreatic duct that overlays the current image of the papilla to match the movement of the papilla.

[0014] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the invention as claimed. As used herein, the terms "comprises," "comprising," "including," or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or device comprising a list of elements does not include only those elements, but may also include other elements not expressly listed or elements inherent to such process, method, article, or device. The term "exemplary" is used to mean "example" rather than "ideal." The term "distal" refers to a direction away from the operator / toward the target site, and the term "proximal" refers to a direction toward the operator. The term "approximately" or similar terms (e.g., "substantially") include values ​​of + / - 10% of the stated value. [Brief explanation of the drawings]

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several examples of the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Figure 1A-1B] 1A and 1B illustrate an exemplary environment in which three-dimensional (3D) image registration may be performed to facilitate a medical procedure. [Figure 2A] FIG. 2A illustrates an exemplary medical device for use in the exemplary environment of FIGS. 1A and 1B. [Figure 2B] FIG. 2B illustrates an exemplary medical device for use in the exemplary environment of FIGS. 1A and 1B. [Figure 3]FIG. 3 illustrates an exemplary process for 3D image registration. [Figure 4] FIG. 4 shows an exemplary process for extracting a 3D model for use in 3D image registration. [Figure 5] FIG. 5 illustrates an exemplary process for registering a 3D model to a patient. [Figure 6A] FIG. 6A shows an exemplary graphical user interface (GUI). [Figure 6B] FIG. 6B shows an exemplary graphical user interface (GUI). [Figure 6C] FIG. 6C illustrates an exemplary graphical user interface (GUI). [Figure 7] FIG. 7 illustrates an example of a computing device. DETAILED DESCRIPTION OF THE INVENTION

[0016] As briefly described above, during an exemplary endoscopic retrograde cholangiopancreatography (ERCP), an endoscope, such as a duodenoscope, can be inserted into a patient's mouth, down the esophagus, through the stomach, into the duodenum, and then to the papilla, where the common bile duct and pancreatic duct each open into the duodenum. Cannulation can then be performed by inserting a guidewire, catheter, and / or other device through the papilla and into the desired duct (e.g., either the common bile duct or the pancreatic duct). The endoscope's imaging device can capture multiple images that allow visualization of the papilla. However, due to the anatomical location of the papilla, the common bile duct and pancreatic duct cannot be seen in the endoscopic images. As a result, cannulating the papilla to insert a guidewire, catheter, and / or other device into the desired duct can be difficult due to the inability to visualize the location or trajectory of the common bile duct and pancreatic duct.

[0017] Due to a lack of visualization, multiple cannulation attempts may be made and / or the wrong duct may be inadvertently entered. Multiple cannulation attempts may increase the procedure's duration. Additionally, repeated cannulation attempts may irritate the tissue at the papilla, causing it to swell and / or become blocked. Swelling and / or blockage may prevent the pancreatic duct from properly draining, leading to fluid accumulation within the pancreas and / or potentially causing post-ERCP pancreatitis. Furthermore, if the wrong duct, such as the pancreatic duct, is inserted and contrast is injected into the duct for fluoroscopy, the contrast may irritate the pancreas, often causing post-ERCP pancreatitis. If multiple cannulation attempts are made and / or contrast is incorrectly injected into the pancreatic duct, the patient may be referred to a tertiary care specialist to prevent and / or manage post-ERCP pancreatitis, which can be very painful for the patient and costly to the healthcare system.

[0018] To reduce cannulation difficulty and decrease the number of patients developing post-ERCP pancreatitis, embodiments disclosed herein present systems, devices, and methods for 3D image registration that enable the generation of a graphical user interface (GUI) that overlays a representation of the location and / or trajectory of at least the common bile duct and pancreatic duct extending from the papilla onto an endoscopic image of the papilla to provide visual guidance for cannulation of the papilla.

[0019] 1A illustrates an exemplary environment 100 in which 3D image registration may be performed to facilitate a medical procedure. Environment 100 may include a medical device system 102, a position sensing system 104, one or more imaging systems 106, a data storage system 108, one or more displays 110, and a computing device 112, each of which communicates with one or more other components of environment 100 via a wired or wireless network, such as network 114. FIG. 1B illustrates exemplary components of medical device system 102 and position sensing system 104 within environment 100.

[0020] 1A and 1B simultaneously, the medical device system 102 may include a medical device 120 and a medical device (MD) controller 122. The medical device 120 may be used to perform a medical procedure. The medical device 120 may be an endoscope, which may be a specialized type of endoscope utilized for a medical procedure. For example, the medical device 120 may be a duodenoscope used to perform an ERCP procedure. In some examples, the medical device 120 may include one or more position sensing components of the position sensing system 104, such as one or more electromagnetic (EM) sensors 132 integrated into at least a distal portion (e.g., a distal tip) of the medical device 120, to enable the position and / or orientation of the medical device 120 to be tracked during the medical procedure. Additionally or alternatively, the medical device 120 may include a fiber optic shape sensor, an accelerometer, and / or a gyro sensor to assist in enabling estimation of the spatial position and / or orientation of the medical device 120 during a medical procedure.

[0021] The MD controller 122 can be a computing device communicatively coupled to the medical device 120 to send and receive signals from the medical device 120. For example, the MD controller can send signals to one or more illumination devices (see FIGS. 2A and 2B) of the medical device 120 to illuminate an area of ​​interest within a body lumen of the patient P. In addition, the MD controller 122 can receive image signals from one or more imaging devices (see FIGS. 2A and 2B) of the medical device 120. The MD controller 122 can have one or more applications (e.g., software programs) locally installed to perform image processing that can be executed to process the image signals to generate images (e.g., live images) for display on one or more of the one or more displays 110 communicatively coupled to the MD controller 122. For example, as the medical device 120 is inserted into the body lumen of the patient P and advanced through the body lumen toward a target site, multiple image signals from the one or more imaging devices can be received by the MD controller 122 and processed by the MD controller 122 to generate and display multiple corresponding images.

[0022] In some examples, the MD controller 122 may have one or more additional applications (e.g., software programs) installed locally to perform one or more operations related to 3D image registration (e.g., one or more operations described in FIG. 3). For example, the generated images may be processed to generate a 3D surface map of the body lumen, which is used as part of the registration process, as described in more detail below. In other examples, the images may be transmitted to another system and / or computing device within the environment 100 for processing, analysis, storage, display, etc.

[0023] One or more components of the MD controller 122, such as one of its applications, can generate or have generated one or more GUIs based on instructions / information stored in memory, instructions / information received from other components in the environment 100, and / or the like. One or more components of the MD controller 122 can also display the GUI using one of the one or more displays 110. The GUI can include images, text, input text boxes, selection controls, and / or the like.

[0024] The medical device system 102 may also include one or more tools 123 that can be inserted and / or delivered into a body lumen of the patient P using the medical device 120. The one or more tools may extend distally from the medical device 120 for use during a medical procedure. Exemplary one or more tools 123 used with the medical device 120 for an ERCP procedure may include tools for cannulation (e.g., sphincterotomes), cholangioscopes, catheters, balloons, stent delivery systems, forceps, baskets, nets, biopsy needles, and / or guidewires, among other similar tools for facilitating diagnosis and / or treatment. In some examples, the one or more tools 123 may also include multiple position-sensing components of the position sensing system 104, such as one or more EM sensors 133 of the one or more tools integrated therein, to enable tracking of the position and / or orientation of the one or more tools 123 during a medical procedure. For example, the tool's one or more EM sensors 133 may be disposed in or on at least the distal tip or distal portion of the one or more tools 123 to assist in tracking at least the position and / or orientation of the distal tip or distal portion of the one or more tools 123. Additionally or alternatively, the one or more tools 123 may include fiber optic shape sensors, accelerometers, and / or gyro sensors to assist in enabling estimation of the spatial position and / or orientation of the one or more tools 123 (e.g., the distal tip or distal portion of the one or more tools 123) during a medical procedure.

[0025] The position sensing system 104 may be a spatial tracking system for determining the position and / or orientation of one or more components of the medical device system 102 and / or other components of the environment 100 within and / or on the body of the patient P. The position sensing system 104 may incorporate any of several features described in U.S. Patent No. 10,782,114, issued September 22, 2020, which is incorporated herein by reference in its entirety. The position sensing system 104 may be an EM-based tracking system including a position sensing system (PSS) controller 130 communicatively coupled to one or more transmitting devices for generating electromagnetic fields and one or more receiving devices for detecting the generated electromagnetic fields. Determining the position and / or orientation of the medical device 120 and / or one or more tools 123 may be based on the strength of the electromagnetic fields detected by the receiving devices.

[0026] The one or more transmitting devices may include, for example, an external device 136 (e.g., an external field generator) adjacent to the patient P. The one or more transmitting devices may each include multiple elements for generating a magnetic field. For example, the one or more transmitting devices may each include one or more coils (e.g., solenoids) and one or more circuit elements that transmit current through the one or more coils. Thus, the one or more coils may generate a magnetic field.

[0027] The one or more receiving devices may include at least one or more EM sensors 132 disposed within or on the medical device 120 to help enable determination of the position and / or orientation of the medical device 120, as described in detail with reference to Figures 2A and 2B. The position and / or orientation of the medical device 120 may be used in conjunction with the above-mentioned 3D surface map generated from multiple images taken by the medical device 120 as part of the registration process.

[0028] In some examples, the one or more receiving devices may also include one or more tool EM sensors 133 disposed within or on the one or more tools 123 to help enable determination of the position and / or orientation of the one or more tools 123. The position and / or orientation of the one or more tools 123 can be used to track the one or more tools 123 as they extend distally from the medical device 120 and enter the anatomical structure during a medical procedure, which may help enable visual guidance of the advancement of the one or more tools 102 through the anatomical structure, as described in detail below.

[0029] In some embodiments, the environment 100 may also include a patch 134 that can be locally applied to the patient P at least during the medical procedure. In some aspects, the one or more receiving devices of the position sensing system 104 may further include one or more patch EM sensors 135 disposed within or on the patch 134. The one or more patch EM sensors 135 can help enable identification of the position and / or orientation of the patient P to determine whether the patient P moves during the medical procedure, including, for example, any respiratory motion that may affect registration. Thus, any movement of the patient P can be accounted for or compensated for in the registration process. As described in further detail below, in some examples, the patch 134 may be placed before the medical procedure for pre-operative 3D imaging, which can further facilitate registration. For example, the patch 134 may also include one or more radiopaque markers, MRI markers, or the like, that can be imaged in the pre-operative 3D images.

[0030] The one or more receiving devices (e.g., one or more patch EM sensors 135) may each include one or more magnetic field sensors. The magnetic field sensors may include, for example, magnetoresistive (MR) elements such as tunneling magnetoresistive (TMR) elements, anisotropic magnetoresistive sensing elements, giant magnetoresistive sensing elements, Hall effect sensing elements, colossal magnetoresistive sensing elements, extraordinary magnetoresistive sensing elements, or semiconductor magnetoresistive elements. Additionally or alternatively, the magnetic field sensors may include one or more inductive sensors (e.g., inductive coil sensors), planar coil sensors, spin Hall sensing elements (or other Hall sensing elements), or magnetic gradiometers. The magnetic field sensors of the one or more receiving devices may have any characteristics of magnetic field sensors known in the art (e.g., including TMR sensors). For example, the magnetic field sensors may include a pinned layer, a tunnel layer, and a free layer. The resistance may change when the free layer is aligned with the pinned layer.

[0031] In some examples, at least some of the one or more receiving devices, such as the one or more EM sensors 132 and / or the one or more patch EM sensors 135, may include multiple magnetic field sensors arranged in a two-axis, six-degree-of-freedom arrangement to enable measurements of x, y, z, roll, pitch, and yaw. For example, these one or more receiving devices may include three magnetic field sensors arranged in a dual-axis, six-degree-of-freedom arrangement (see FIG. 2B) to enable the positioning of the imaging plane to be determined in three dimensions based on the measurements of x, y, z, roll, pitch, and yaw. In such a configuration, two of the three magnetic field sensors may be oriented such that their primary sensing direction is aligned (approximately parallel) with the longitudinal axis of the respective device in which they are incorporated. A full Wheatstone bridge configuration may be utilized by two of the magnetic field sensors. A third magnetic field sensor may be positioned such that its primary sensing direction is transverse (e.g., approximately orthogonal / perpendicular) to the longitudinal axis. A half Wheatstone bridge configuration may be utilized by the third magnetic field sensor. The Wheatstone bridge may have any of the characteristics of a Wheatstone bridge known in the art.

[0032] The position sensing system 104 may have other configurations within the scope of the present disclosure. For example, a three-axis configuration may be utilized for the receiving device's magnetic field sensors, with each of the three magnetic field sensors positioned such that its primary sensing direction is aligned with a different axis (e.g., the primary sensing directions of the multiple magnetic field sensors are aligned orthogonal to one another). For example, a first magnetic field sensor may have a primary sensing direction of the X-axis, a second magnetic field sensor may have a primary sensing direction of the Y-axis, and a third magnetic field sensor may have a primary sensing direction of the Z-axis. In such a three-axis configuration, each of the magnetic field sensors may utilize a half-Wheatstone bridge configuration. In another example, only two magnetic field sensors may be utilized by the receiving device to measure six degrees of freedom, with each of the two magnetic field sensors having a half-Wheatstone bridge configuration (or a full Wheatstone bridge configuration). In a further example, two magnetic field sensors may be used to measure five degrees of freedom. In such an example, the position sensing system 104 may not be able to measure roll. In an additional example, a single magnetic field sensor may be implemented by the receiving device and use a half Wheatstone bridge to measure five degrees of freedom.

[0033] In the examples described above, external device 136 is a transmitting device and one or more EM sensors 132 and / or one or more patch EM sensors 135 are receiving devices. In other examples, one or more EM sensors 132 and / or one or more patch EM sensors 135 can be transmitting devices and external device 136 can be receiving devices.

[0034] The PSS controller 130 may be communicatively coupled to one or more transmitting devices and one or more receiving devices of the position sensing system 104. For example, as shown in FIG. 1B , the PSS controller 130 may be communicatively coupled to an external device 136, one or more EM sensors 132, and / or one or more patch EM sensors 135. The PSS controller 130 may send signals to the external device 136, for example, to initiate generation of a magnetic field and subsequently pause, stop, and / or resume generation of the magnetic field. In addition, the PSS controller 130 receives signals from the one or more EM sensors 132 and / or one or more patch EM sensors 135, which signals indicate the strength of the magnetic field (e.g., the voltage induced thereby) detected by the one or more EM sensors 132 and / or one or more patch EM sensors 135. The position and / or orientation of the medical device 120, specifically at least the distal tip of the medical device 120 (see FIGS. 2A and 2B ), may be determined based on the signals received from the one or more EM sensors 132. Additionally, the position and / or orientation of the patient P may be determined based on signals received from one or more patch EM sensors 135 to help identify any patient movement, including, for example, respiratory movement, that may affect registration and therefore be taken into account during registration.

[0035] The environment 100 may include, for example, one or more imaging systems 106 that capture multiple images of the patient P's anatomy. The environment may also include, for example, a data storage system 108 that stores images captured by the one or more imaging systems 106. The captured images may be three-dimensional (3D) images. Alternatively or additionally, the images may be two-dimensional (2D) images that can be reconstructed into 3D images using techniques known or that may become known in the art. At least one of the captured 3D images can be used to extract a 3D model that identifies multiple anatomical structures within the patient P's anatomy, including one or more anatomical structures of interest based on a medical procedure, such as the bile duct and pancreatic duct for an ERCP procedure. The extracted 3D model can be registered to the patient as part of a registration process. For example, the 3D model can be registered to the patient P using a 3D surface map generated from multiple images captured by the medical device 120 and the position and / or orientation of the medical device 120 determined by the position sensing system 104, as described in detail below.

[0036] The one or more imaging systems 106 may include one or more preoperative imaging systems. Exemplary preoperative imaging system modalities may include computed tomography (CT), magnetic resonance cholangiopancreatography (MRCP), ultrasound (US), or other similar three-dimensional (3D) imaging modalities. In some examples, preoperative imaging may be requested specifically in preparation for a medical procedure. For example, a gastroenterologist may schedule a patient P for an ERCP procedure and may also prescribe preoperative imaging in preparation for the ERCP procedure. In such examples, the patient P may be positioned in the same or similar position during imaging relative to how the patient will be positioned during the procedure. In other examples, preoperative images available for the patient P may be requested for diagnostic or other illustrative purposes. As a result, the patient P may not be positioned in the same or similar position during imaging relative to how the patient will be positioned during the procedure, which may be taken into account in the image registration process, described in detail below.

[0037] In some examples, the one or more imaging systems 106 may also include an intraoperative imaging system. The one or more intraoperative imaging systems may be in addition to and / or replace the one or more preoperative imaging systems. Exemplary modalities of an intraoperative imaging system may include non-3D imaging modalities that can be used to reconstruct 3D images, such as transabdominal US, endoscopic US, and / or fluoroscopy. Additionally or alternatively, exemplary modalities of an intraoperative imaging system may include intraoperative 3D imaging modalities, such as fluoroscopic cone beam CT, C-arm tomography, and / or digital tomosynthesis. In examples where the one or more imaging systems 106 include multiple intraoperative 3D imaging modalities, one or more additional 3D images can optionally be captured by one of the multiple intraoperative 3D imaging modalities during the medical procedure to enable generation of a 3D surface map. For example, the additional 3D images may be used in conjunction with multiple images taken by the medical device 120 to generate a 3D surface map, as described in detail below.

[0038] In some examples, each of the one or more imaging systems 106 may include and / or be associated with a computing device (e.g., distinct from a separate computing device 112). The computing device may include one or more applications (e.g., software programs) installed locally on the memory of the computing device, for example, to perform image processing that may be executed to generate images. In addition, one or more components of the computing device, such as one of the applications, may generate or have generated one or more GUIs based on instructions / information stored in memory, instructions / information received from other components in the environment 100, and / or the like. The one or more components of the computing device may display the GUI using one of the one or more displays 110. The GUI may include images, text, input text boxes, selection controls, and / or the like, and may enable operator interaction with images captured by individual modalities from the various aforementioned modalities. For example, the computing device may generate and display a 3D image from the image signals received from each modality, and an operator may utilize an application to manipulate the 3D image (e.g., rotate, zoom in, zoom out, annotate objects or anatomical structures, etc.).

[0039] In some examples, the computing device may have one or more additional applications installed locally to perform one or more operations related to 3D image registration (e.g., one or more operations described in FIG. 3 ). For example, the generated 3D images may be processed to extract a 3D model, and the 3D model may be registered to the patient P, as described in detail below. In other examples, the 3D images may be transmitted to another system and / or computing device within the environment 100 for processing, analysis, storage, display, etc. Additionally, one or more other components of the environment 100, such as the one or more imaging systems 106 and the data storage system 108, may be components or subsystems of a larger system, such as a picture archiving and communication (PAC) system.

[0040] The data storage system 108 may include a server system or a computer-readable memory, such as a hard drive, flash drive, disk, etc. The data storage system 108 includes an interface for exchanging data with and / or interacts with other systems, such as one or more of the other components of the environment 100. For example, the data storage system 108 may be configured to receive and store 3D images of the patient P generated by one or more of the one or more imaging systems 106. As another example, the data storage system 108 may be configured to receive and store multiple images of a body lumen of the patient P from the medical device system 102. As a further example, the data storage system 108 may be configured to receive spatial information about the medical device 120 and / or the patient P from the position sensing system 104.

[0041] The one or more displays 110 may be communicatively coupled to one or more other components of the environment 100 to receive and display data, including, for example, image data. As one example, the one or more displays 110 may receive and display 3D images captured by one of the one or more imaging systems 106 and processed to extract a 3D model. In addition, the one or more displays 110 may receive and display the extracted 3D model. As another example, the one or more displays 110 may receive multiple images captured by the medical device 120, for example, as the medical device 120 is inserted into a body lumen of the patient P and advanced through the body lumen toward a target site. In addition, the one or more displays 110 may receive and display a 3D surface map generated by processing the multiple images.

[0042] Additionally, once 3D image registration is complete, the one or more displays 110 may receive and display a GUI, described in detail below, that overlays a representation of the location or trajectory of one or more anatomical sites of interest for the medical procedure onto a current image of the target site (e.g., a current image of the target site, which is one of multiple images captured by the medical device 120). For example, in the case of an ERCP procedure, the GUI may overlay a representation of the location or trajectory of the common bile duct and pancreatic duct onto a current image of the papilla to provide visual guidance for cannulation into one of these ducts through the papilla. In some examples, the one or more displays 110 may be an interactive display and / or a display of a computing device configured to receive input from an operator to enable operator interaction with the image data.

[0043] Computing device 112 may be a standalone computing device configured to communicate with one or more of multiple other components of environment 100 via network 114. For example, computing device 112 may communicate with one or more of medical device system 102, position sensing system 104, imaging system 106, and / or data storage system 108 via network 114 to exchange information, including receiving image data and spatial data (e.g., position and / or orientation information). Computing device 112 may be, for example, a computer system such as a desktop computer, a laptop computer, a tablet, a smart mobile phone, a smart watch, or other electronic wearable.

[0044] In some examples, the computing device 112 may include one or more applications, e.g., programs, plug-ins, etc., installed locally on the memory of the computing device 112 to perform one or more operations related to 3D image registration based on the received information. For example, the 3D model can be registered to the patient using a 3D model extracted from 3D images captured by one of the one or more imaging systems 106, a 3D surface map generated from multiple images captured by the medical device 120, and spatial information of at least the medical device 120 identified by the position sensing system 104. In some examples, the computing device 112 may receive the 3D model and the 3D surface map from other respective components of the environment 100. In other examples, the computing device 112 may generate the 3D model and the 3D surface map.

[0045] Additionally, one or more components of computing device 112, such as one of those applications, may generate or have generated one or more GUIs based on instructions / information stored in memory, instructions / information received from other components in environment 100, and / or the like. Further, one or more components of computing device 112 may display a GUI using a display of computing device 112 or using one or more other displays (e.g., one or more displays 110). A GUI may include text, input text boxes, selection controls, and / or the like. A display may include a display with a touch screen or other input system (e.g., a mouse, keyboard, etc.) for an operator of computing device 112 to control functions of computing device 112.

[0046] For example, as shown in Figures 6A-6C, once the 3D model is registered to the patient, a GUI may be generated for display. The GUI may include a representation of the location and / or trajectory of one or more anatomical regions of interest for the medical procedure generated using the transformed 3D image based on the registration. The representation may be overlaid on a current image (e.g., one of multiple images) of the target region imaged by the medical device 120. Additionally, throughout the registration process, one or more prompts may be generated and displayed to the operator, for example, to ask the operator to confirm and / or correct automatic decisions or identifications made by the application.

[0047] Computing device 112 is described as including an application configured to perform one or more operations or steps of 3D image registration (e.g., one or more steps described in FIG. 3 below). Additionally or alternatively, one or more of the imaging system(s) 106's or associated computing devices and / or one or more other computing devices, such as MD controller 122, may include the same or similar applications for performing at least some (or all) of the operations of 3D image registration. In some examples, a particular application running on a single computing device of environment 100 may be configured to perform each of multiple steps (e.g., 3D model extraction, 3D surface map generation, registration, and GUI generation). In other examples, multiple applications running on the same computing device or across different computing devices of environment 100 may perform different operations. As a non-limiting specific example, a particular application may be configured to perform multiple operations related to 3D model extraction, while another application may be configured to perform multiple operations related to 3D surface map generation. Furthermore, additional applications may be configured to perform multiple operations related to registration and / or GUI generation.

[0048] The one or more applications executing on one or more components of environment 100 are described herein as local applications, e.g., installed on the memory of the respective components, such that a network connection (e.g., Internet access) is not required to enable communication with the remote server and applications to function. However, in other embodiments, the applications may be web-based applications accessible via a browser executing on the component, and the one or more applications may communicate with a remote server (not shown) over network 114. In such examples, one or more operations of 3D image registration may be performed by processing devices of the remote server.

[0049] As described above, one or more components of environment 100 may communicate via network 114. Network 114 may be an electronic network. Network 114 may include one or more wired and / or wireless networks, such as a wide area network (WAN), a local area network (LAN), a personal area network (PAN), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc.). In a specific, non-limiting illustrative example, multiple components of environment 100 may communicate and / or connect to network 114 via a universal serial bus (USB) or other similar local low-latency connection or direct wireless protocol.

[0050] In some embodiments, network 114 includes the Internet, and information and data provided between various systems occurs online. "Online" may mean connecting to or accessing source data or information from a location remote from other devices or networks coupled to the Internet. Alternatively, "online" may refer to connecting to or accessing an electronic network (wired or wireless) via a mobile communications network or device. The Internet is a global system of computer networks, a collection of networks through which a party at a particular computer or other device connected to the network can obtain information from any other computer or communicate with other parties at other computers or devices. Components of environment 100 can be connected via network 114 using one or more standard communication protocols so that the components can send and receive communications to and from each other via network 114, as described in more detail below.

[0051] 1A and 1B as separate components, it should be understood that a component or portion of a component in environment 100 may, in some embodiments, be integrated with or incorporated into one or more other components. In some embodiments, the operation or aspects of one or more of the components described above may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of environment 100 may be used.

[0052] While specific examples included throughout this disclosure demonstrate 3D image registration in environment 100 to facilitate an ERCP procedure, it should be understood that techniques according to the present disclosure can be adapted to register 3D models to other types of images besides endoscopic images captured by medical device 120. For example, a 3D model can be registered to a fluoroscopic image of a patient's anatomy captured during ERCP to enable navigation of the bile duct or pancreatic duct. Fluoroscopic images generally only show the bile duct and pancreatic duct for a very short period of time when contrast is injected. Using similar techniques described herein, a 3D model can be registered to the fluoroscopic image, and a GUI can be generated based on the registration that includes a representation of the bile duct and pancreatic duct overlaid on the fluoroscopic image. Additionally, techniques according to the present disclosure can be adapted to other types of endoscopic medical procedures, such as a percutaneous nephrolithotomy (PCNL) procedure, or any other procedure involving endoluminal access to structures that are elastic and deformed during the procedure. For example, to aid in the placement of a stent or other similar device to treat a cystic lesion in the pancreas, a 3D image including a portion of the stomach wall and the cystic lesion may be registered. A representation of the lesion may be overlaid on an endoscopic image of the stomach wall captured during the procedure at the approximate location where the lesion would appear if it were visible to guide the location of a penetration to gain access to the lesion for stent placement. It should also be understood that the above examples are merely illustrative. The techniques and technologies of the present disclosure may be adapted for any suitable activity.

[0053] 2A shows the exemplary medical device 120 of FIGS. 1A and 1B. The medical device 120 may include a handle 202 and an insertion portion 204. The medical device 120 may also include an umbilicus 206 for connecting the medical device 120 to sources of, for example, air, water, suction, power, etc., as well as imaging and / or display equipment such as an MD controller 122.

[0054] The insertion portion 204 includes a sheath or shaft 208, and the insertion portion 204 may also include a distal tip 210. FIG. 2B shows an exemplary distal tip assembly 211 positioned at the distal tip 210. Referring simultaneously to FIGS. 2A and 2B, the distal tip assembly 211 may include a substrate 240 (e.g., a rigid or flexible circuit board or other type of substrate) that may be at least partially disposed within or on an inner core (not shown) of the distal tip 210. In some examples, the substrate 240 is rigid and includes multiple layers.

[0055] As shown in FIGS. 2A and 2B , the distal tip 210 and distal tip assembly 211 include one or more imaging devices 212 (e.g., one or more cameras) for capturing images, and the distal tip assembly 211 may also include one or more illumination devices 214 (e.g., one or more light emitting diodes (LEDs) or optical fibers) for providing illumination and facilitating image capture and visualization. The distal tip 210 and distal tip assembly 211 are side-facing. That is, the imaging device 212 and illumination device 214 are oriented radially outward, perpendicularly, approximately perpendicularly, or otherwise laterally relative to the longitudinal axis of the shaft 208 and distal tip 210. However, the present disclosure also encompasses other configurations of the distal tip and distal tip assembly. For example, the distal tip and distal tip assembly may be “forward-facing” (i.e., facing distally).

[0056] The imaging device 212 and the illumination device 214 may be mounted to the substrate 240 by any suitable method, including, but not limited to, wire bonding, surface mount assembly, electromechanical assembly, and / or plated-through-hole techniques. While one imaging device 212 and two illumination devices 214 are shown in FIGS. 2A and 2B , any suitable number of imaging devices 212 and / or illumination devices 214 may be utilized. Alternatively, the imaging device 212 and the illumination device 214 may be combined in a single device. A conduit 242 may accommodate one or more wires or cables attached to the substrate 240 or multiple elements mounted on the substrate 240 for transmitting power and / or signals to and from the substrate 240 and / or multiple elements mounted on the substrate 240. The wires or cables extend through the shaft 208 and into the handle 202, and the wires or cables may be connected to the MD controller 122, for example, via the umbilicus 206. For example, the MD controller 122 may send signals for the plurality of lighting devices 214 to emit light and receive image signals from the imaging device 212 for processing and subsequent display. In some examples, the MD controller 122 may also initiate image capture by sending signals via wires or cables housed within a conduit to cause the imaging device 212 to capture an image.

[0057] The elements of the position sensing system 104 may also be disposed on the substrate 240 and mounted according to any of the techniques described above with respect to the imaging device 212 and the plurality of illumination devices 214. For example, at least one or more EM sensors 132, such as the first EM sensor 132 a, the second EM sensor 132 b, and the third EM sensor 132 c, may be disposed on the substrate 240. In some examples, the first EM sensor 132 a, the second EM sensor 132 b, and the third EM sensor 132 c may be oriented on the substrate as described in detail above with reference to FIGS. 1A and 1B. Any alternative number of EM sensors 132 may be utilized, and the illustrated three EM sensors 132 a, 132 b, 132 c are merely exemplary. 1B, EM sensors 132a, 132b, and 132c are receiving devices capable of measuring, for example, magnetic fields generated by external device 136, thereby enabling tracking of the position and / or orientation of distal tip 210. In other examples, EM sensors 132a, 132b, 132c may be transmitting devices capable of generating magnetic fields.

[0058] The inclusion of the imaging device 212 within the distal tip assembly 211 allows for the EM sensors 132a, 132b, 132c to be utilized to track the position and / or orientation of the distal tip 210, thereby enabling the position and / or orientation of the medical device 120 to be known relative to a 3D surface map generated from multiple images captured by the imaging device 212. This known position and / or orientation can be used with the 3D surface map as part of a registration process, as described in detail below with reference to FIG. 3. While FIG. 2B shows the EM sensors 132a, 132b, 132c included within the distal tip assembly 211 at the distal tip 210 of the medical device 120, in other examples, one or more other EM sensors may be positioned at other locations on the medical device 120, including within the shaft 208 and / or handle 202.

[0059] Other optional components of the position sensing system 104 may be mounted on the substrate 240, including a capacitor 244 and one or more diodes 246. The capacitor 244 may help reduce noise in the voltage supplying the position sensing system 104. For example, the capacitor 244 may function as a decoupling capacitor and act as a low-pass filter for any electromagnetic interference ("EMI") on the supply voltage. The one or more diodes 246 may help provide high-voltage protection, such as electrostatic discharge ("ESD") protection. The one or more diodes 246 may help prevent damage to the EM sensors 132a, 132b, 132c due to electrostatic discharge. The one or more diodes 246 may additionally or alternatively help protect aspects of the imaging device 212.

[0060] In some examples, the imaging device 212 (e.g., the camera and lens of the imaging device 212) may be calibrated to understand the transformation between the optical coordinate system in which images are captured by the imaging device 212 and the spatial coordinate system in which the position and / or orientation of the medical device 120 is determined by the position sensing system 104. The calibration may be performed using algorithms that are commonly known or that may become known in the art, such as a "hand-eye" calibration method. The calibration may be performed during the manufacturing process of the medical device 120.

[0061] The distal tip assembly 211 may also include an elevator 216 for changing the orientation of a tool (e.g., one of the one or more tools 123) inserted into the working channel of the medical device 120. The elevator 216 may alternatively be referred to as a swing stand, a pivot stand, a raising base, or any other suitable terminology. The elevator 216 may be pivotable, for example, using an actuation wire or another control element extending from the handle 202 through the shaft 208 to the elevator 216. The elevator 216 may be pivotable about an axis 217. The axis 217 may be rotatably held within the distal tip assembly 211.

[0062] The distal tip assembly 211 may also include components in addition to or in place of those described above. For example, the distal tip assembly 211 may also include additional or alternative illumination sources and / or additional or alternative imaging components (e.g., additional cameras). The distal tip assembly 211 may also include additional types of sensors, such as moisture sensors, temperature sensors, pressure sensors, or other types of sensors that may be useful during a medical procedure.

[0063] The distal portion of the shaft 208 connected to the distal tip 210 may have a steerable section 218. The steerable section 218 may be, for example, an articulation joint. The shaft 208 and the steerable section 218 may include a variety of structures known or that may become known in the art.

[0064] The handle 202 may have one or more actuator / control mechanisms 220. The control mechanisms 220 may provide control over the steerable section 218 or may allow for the provision of air, water, suction, etc. For example, the handle 202 may include control knobs 222, 224 for controlling the steerable section 218 left, right, up, and / or down. For example, one of the knobs 222, 224 may provide left / right control for the steerable section 218, and the other of the knobs 222, 224 may provide up / down control for the steerable section 218. The handle 202 may further include one or more locking mechanisms 226 (e.g., knobs or levers) for preventing steering of the steerable section 218 in at least one of an up direction, a down direction, a left direction, or a right direction. The handle 202 may include an elevator control lever 228. The elevator control lever 228 can raise and / or lower the elevator 216 using a connection between the lever 228 and an actuation wire (not shown) that extends from the lever 228 through the shaft 208 to the elevator 216. The port 230 can allow a tool (e.g., one of the one or more tools 123) to enter a working channel (not shown) of the medical device 120 through the port 230, pass through the shaft 208, and reach the distal tip 210. Although not shown, the handle 202 can include one or more valves, buttons, actuators, etc. to control the supply of air, water, suction, etc.

[0065] In use, an operator can insert at least a portion of the shaft 208 into a body lumen of a subject, such as a patient P. The distal tip 210 can reach a target site within the body lumen. For an ERCP procedure, the distal tip 210 can be inserted into the patient's mouth, down the esophagus, through the stomach, into the duodenum, and reach the papilla, which is the target site for cannulation. The operator can insert a cannulation tool (e.g., one of one or more tools 123, such as a sphincterotome) into the port 230 and pass the cannulation tool through the shaft 208 via the working channel to the distal tip 210. The cannulation tool can exit the working channel at the distal tip 210. The operator can use the elevator control lever 228 to raise the elevator 216 and angle the cannulation tool toward a desired location on the papilla. The operator can use the cannulation tool to perform cannulation. Using the systems, devices, and methods described herein, positioning of a cannulation tool relative to a desired location of the papilla for cannulation can be facilitated by the display of a GUI. For example, the GUI may include a representation of the position and / or trajectory of the bile duct and pancreatic duct overlaid on a current image of the papilla imaged by the imaging device 212 of the medical device 120.

[0066] 3 shows an example process 300 for 3D image registration. In some examples, process 300 may be performed by one or a combination of components of environment 100, such as a computing device of one of one or more imaging systems 106 or a computing device associated with that imaging system 106, MD controller 122, and / or a separate computing device 112, using one or more applications running thereon.

[0067] In step 302, process 300 may include receiving a 3D image of a patient's anatomy, such as patient P. The 3D image may be of a particular anatomy (e.g., including multiple particular anatomy regions) depending on the type of medical procedure. For example, if the medical procedure being performed is an ERCP procedure, the 3D image may be of the upper gastrointestinal (GI) tract, including the esophagus, stomach, duodenum, and biliary-pancreatic anatomy. In some examples, an operator may select a 3D image from among multiple 3D images to be used for 3D image registration.

[0068] Pre-operative imaging may be performed on patient P several weeks, days, or the same day as the medical procedure. In such an example, the 3D image may be a pre-operative 3D image captured by one of the pre-operative imaging systems of the one or more imaging systems 106, such as, for example, a CT image, an MRCP image, an US image, etc. The pre-operative 3D image may be received from one of the one or more imaging systems 106 that captured the 3D image or from the data storage system 108.

[0069] In other examples, intraoperative imaging may be performed on patient P during a medical procedure. In such examples, the 3D image may be an intraoperative 3D image captured by one of the intraoperative imaging systems of the one or more imaging systems 106. The intraoperative imaging system may include a non-3D imaging modality that may be used to reconstruct the 3D image, including transabdominal US, endoscopic US, or fluoroscopy. Additionally or alternatively, the intraoperative imaging system may include a 3D imaging modality such as fluoroscopic cone beam CT, C-arm tomography, and / or digital tomosynthesis.

[0070] In some examples, when preoperative and / or intraoperative imaging is performed, a contrast agent or other substance (e.g., secretin) may be administered to the patient P to help enhance the appearance of certain anatomical structures in the 3D image, such as the biliary pancreatic duct. Additionally, if preoperative and / or intraoperative imaging is specifically indicated for a medical procedure, the patient P may be positioned for imaging in the same or similar position as the patient P would be or is currently positioned in for the ERCP procedure. The same or similar position may facilitate registration. However, preoperative images are often ordered by a different physician and / or for a different purpose (e.g., for diagnostic purposes), and thus the patient P may be in a different position for preoperative imaging than when the patient P undergoes the medical procedure. Accordingly, the registration techniques described herein may adjust for or account for different positions present in the 3D image.

[0071] Optionally, patch 134, or at least some of its components, including radiopaque markers, MRI markers, or other similar markers, may be applied to patient P prior to pre-operative and / or intra-operative imaging. As a result, the 3D images received in step 302 may include markers of patch 134 within the 3D images, which may provide additional alignment features for registration.

[0072] In step 304, process 300 may include processing the 3D image to extract a 3D model that identifies multiple anatomical structures within the anatomical structure, including one or more anatomical structures of interest for the medical procedure. For example, if the medical procedure is an ERCP procedure, the 3D model may identify and separate (e.g., segment) multiple anatomical structures of at least the upper GI tract and biliary-pancreatic system, including the esophagus, stomach, pyloric sphincter, duodenum, bile duct (e.g., common bile duct, cystic duct, and hepatic duct), liver, pancreatic duct, and pancreas. The biliary-pancreatic system, particularly the bile duct and pancreatic duct, may be multiple anatomical structures of interest. In addition to the anatomical structures of the upper GI tract and biliary-pancreatic system, one or more other anatomical structures, such as ribs, spine, and / or other structures, may be identified that can be mapped to intraoperative images, including endoscopic and / or fluoroscopic images, captured by medical device 120 for use during registration, as described in detail below.

[0073] In some examples, the 3D image may be processed using manual segmentation, whereby an operator may manually identify and label each of the anatomical structures. For example, the 3D image may be displayed, for example, using one of the one or more displays 110, and the operator provides input via one of the one or more displays 110 and / or an associated computing device (e.g., a computing device of one of the imaging systems 116 or computing device 112) to manually label the anatomical structures. In other examples, the 3D image may be processed using computational methods commonly used in computer vision, machine learning, and / or other image processing techniques to separate multiple structures. The computational methods may be fully automatic and / or may be used in conjunction with operator input, for example, to confirm or correct / correct multiple anatomical features.

[0074] As a specific exemplary computational method, a computer vision model or machine learning model (hereinafter referred to as a “model”) may be trained and implemented to predict multiple anatomical structure locations present in a 3D image. The model may be trained by one of multiple components within environment 100 that may implement the model, such as a computing device of one or more imaging systems 106 or a computing device associated with the imaging system(s) 106, computing device 112, and / or MD controller 122. In other examples, the model may be trained by a third-party system, and the model may be provided to components within environment 100 that may implement the model for execution. To train the model, training data may be received and processed to generate (e.g., build) a trained model for predicting multiple anatomical structure locations present in a 3D image. The training data may include multiple 3D training images of a patient's anatomy. The multiple 3D training images may include multiple imaging modalities (e.g., CT images, MRCP images, US images, etc.). The training data may be generated, received, or otherwise acquired from internal and / or external resources. In some examples, the training data may also include multiple synthetic 3D training images of the patient's anatomy.

[0075] Generally, a model includes a set of variables, e.g., nodes, neurons, filters, etc., that are adjusted, e.g., weighted or biased, to different values ​​through the application of training data. In some examples, supervised, unsupervised, semi-supervised, and / or reinforcement learning processes may be implemented to train the model. In some embodiments, a portion of the training data may be withheld during training and / or used to validate the trained model.

[0076] When a supervised learning process is used, multiple labels or annotations corresponding to multiple 3D training images (e.g., multiple labels or annotations corresponding to training data) can facilitate the learning process by providing ground truth. For example, the multiple labels or annotations can indicate multiple anatomical structures present in the 3D images. Training may proceed by feeding 3D training images (e.g., samples) from the training data to a model, with the model having multiple variables set to initialized values, e.g., randomly, based on Gaussian noise, a pre-trained model, or the like. The model may output multiple predicted anatomical structures present for the samples. The output can be compared with the corresponding labels or annotations (e.g., ground truth) to determine an error, which can then be back-propagated through the model to adjust the values ​​of the multiple variables. This process can be repeated for multiple samples, at least until the determined loss or error is below a predetermined threshold. In some examples, some of the training data can be withheld and used to further validate or test the trained model.

[0077] In the case of an unsupervised learning process, the training data may not include pre-assigned labels or annotations to aid the learning process. Rather, the unsupervised learning process may include clustering, classification, or the like to identify naturally occurring patterns in the training data. K-means clustering or K-nearest neighbors may also be used, which may be supervised or unsupervised. A combination of K-nearest neighbor and unsupervised clustering techniques may also be used. In the case of semi-supervised learning, a combination of training data with pre-assigned labels or annotations and training data without pre-assigned labels or annotations may be used to train the model.

[0078] When reinforcement learning is employed, an agent (e.g., an algorithm) can be trained through trial and error to make decisions about anatomical structures contained in samples from training data. For example, upon making a decision, the agent can then receive feedback (e.g., a positive reward if the predicted anatomical structures are actually present in the sample) and adjust its next decision to maximize the reward, iterating until the loss function is optimized.

[0079] Once trained, the trained model may be stored and subsequently applied by one of the components of environment 100. For example, the trained model may receive as input data the 3D image received in step 302. The trained model may output predicted anatomical structures present (e.g., included) in the 3D image. A 3D model may be extracted based on the output that identifies (e.g., separates) the anatomical structures. The 3D model with the separated anatomical structures may then be displayed (e.g., using one of one or more displays 110). In some examples, operator input via touch or other input to one of the one or more displays 110 and / or associated computing devices (e.g., one of one or more imaging systems 106, MD controller 122, and / or one of computing devices 112) may be received as feedback on the prediction. For example, the operator may confirm and / or modify the identified anatomical structures. The feedback may be used to retrain the model, for example, by adjusting the values ​​of one or more variables of the model.

[0080] 4, in some examples, the 3D model extracted in step 304 may be incomplete. For example, one or more of the multiple anatomical structures may be incomplete. In such examples, an estimated model of the incomplete anatomical structure may be calculated.

[0081] By identifying (e.g., isolating and / or segmenting) anatomical structures, the extracted 3D model can visualize specific anatomical structures that are not normally visible during a medical procedure, such as the bile duct and pancreatic duct during an ERCP procedure. Additionally, the identified anatomical structures can serve as landmarks to facilitate accurate image fusion and registration, as described in detail below.

[0082] In step 306, the process 300 may include receiving multiple images of the body lumen of the patient P imaged by the imaging device 212 of the medical device 120 as the medical device 120 is inserted into and advanced through the body lumen toward a target site during a medical procedure. At least one of the multiple images may include a current image of the target site. For example, for an ERCP procedure, the medical device 120 may be inserted into the mouth of the patient P, down the esophagus, through the stomach, into the duodenum, and advanced to the papilla, which may be a target site for cannulation. Thus, at least one of the received multiple images may include a current image of the papilla. The multiple images may be received in real time as the medical device 120 is advanced toward the target site and displayed using one of the one or more displays 110, allowing the operator to view the body lumen and the target site as the medical device 120 is advanced toward the target site.

[0083] At step 308, the process 300 may include receiving spatial information about the medical device 120 from the position sensing system 104. In some examples, the spatial information about the medical device 120 may be received in real time as the medical device 120 advances toward the target site to enable correlation with multiple images captured by the imaging device 212 as the medical device 120 advances toward the target site. The spatial information of the medical device 120 may include the position and / or orientation of the medical device 120, and more specifically, at least the position and / or orientation of the distal tip 210 and / or distal assembly 211 of the medical device 120 as described and shown in FIGS. 2A and 2B . For example, the position and / or orientation of the distal tip 210 of the medical device 120 may be determined based on signals received from the EM sensors 132a, 132b, 132c indicative of the strength of (e.g., voltage induced thereby) a magnetic field generated by the external device 136 as detected by one or more patch EM sensors 135. The position and / or orientation of any other component of the medical device 120 may also be determined if an additional EM sensor 132 is disposed within that component (e.g., if the EM sensor 132 is positioned within the shaft 208 and / or handle 202).

[0084] Optionally, spatial information of the patient P may also be received from the position sensing system 104. For example, when patches 134 are applied locally to the patient P, the position and / or orientation of the patient P may be determined by the PSS controller 130 based on, for example, signals received from one or more patch EM sensors 135 indicative of the strength of the magnetic field (e.g., the voltage induced by the magnetic field) generated by the external device 136 as detected by the one or more patch EM sensors 135.

[0085] In step 310, the process 300 may include processing the multiple images of the body lumen and the spatial information of the medical device 120 to generate a 3D surface map of at least a portion of the body lumen. The 3D surface map may include portions of multiple anatomical structures identified (e.g., isolated or segmented) in the extracted 3D model that do not include the multiple anatomical structures of interest. Continuing with the example of an ERCP procedure, the 3D surface map may map the anatomical structures surrounding the upper GI tract and the biliary pancreatic duct (e.g., the duodenal wall and papilla). However, the biliary pancreatic duct itself is not mapped in the 3D surface map because the duodenal wall and papilla effectively block or prevent visualization of the biliary pancreatic duct, and therefore are not imaged in the multiple images.

[0086] As the medical device 120 advances toward the target site, images captured by the imaging device 212 of the medical device 120 and corresponding spatial tracking information about the medical device 120, including at least the position and / or orientation of the distal tip 210 of the medical device 120, may be used to generate a 3D surface map. The spatial tracking information may include at least the position and / or orientation of the distal tip 210 of the medical device 120 as determined by the position sensing system 104 using one or more EM sensors 132. In some examples, information from other positioning systems, such as light detection and ranging (LIDAR), ultrasound ranging (using pulse-echo or transmit-receive methods), stereoscopic cameras, fluoroscopic images, intraoperative 3D radiological images, structured light images, etc., may be utilized to generate the 3D surface map.

[0087] In any embodiment in which the environment 100 does not include the position sensing system 104 and / or other similar independent spatial tracking system, it may be necessary to determine the position and / or orientation of the imaging device 212 at the distal tip 210 of the medical device 120 with respect to multiple anatomical sites while simultaneously mapping the multiple anatomical sites as multiple 3D surfaces. In some examples, the position and / or orientation of the imaging device 212 may be determined by applying a Simultaneous Localization and Mapping (SLAM) algorithm. The SLAM algorithm may incorporate multiple inputs from multiple other sensors and / or imaging modalities. When a SLAM algorithm is utilized, multiple anatomical sites may be successively mapped to serve as multiple landmarks for use in at least registration as the medical device 120 navigates through a body lumen, such that the relative positions of each anatomical site are known (e.g., a continuous path between each of the multiple anatomical sites may be mapped).

[0088] In addition to (e.g., as part of or in conjunction with) generating the 3D surface map, multiple anatomical structures may be identified. For example, the identified multiple anatomical structures may correspond to multiple anatomical structures identified in the 3D model extracted in step 304 for use in registration. In some examples, the distinctiveness of the appearance and / or geometry of a given anatomical structure may determine which of the identified multiple anatomical structures can be utilized as part of the 3D surface map to serve as landmarks, for example, to facilitate image registration, as described below.

[0089] Continuing with the example of the medical procedure being an ERCP procedure, as the medical device 120 passes through the upper GI tract (e.g., down the esophagus, through the stomach, and into the duodenum), the esophagus, fundus, body, antrum, pylorus, pyloric sphincter, duodenal bulb, and / or duodenum may be identified as distinct anatomical regions. In some examples, distinct anatomical regions and regions within each region may be identified based on the unique appearance and / or morphological form of the region and / or structure. For example, the esophagus may be identified based on its long, straight, tubular form. Portions of the stomach may be identified by the appearance and / or shape of the gastric rugae or the more open bulbous shape of the fundus. The pyloric sphincter may be identified by the shape of a small opening at the end of the larger cavity volume. The duodenum may be identified by the ring-like folds surrounding its tubular wall or the appearance of the intestinal villi. Additionally, the identity of multiple anatomical structure sites may be inferred by their spatial proximity to other anatomical structures and / or by the temporal order in which multiple anatomical features appear during a medical procedure as the medical device 120 advances through a body lumen to the target site.

[0090] In other examples, different regions of the upper GI tract may be mapped as a single structure without separate classification. However, if the 3D surface map is incomplete, it may be useful to identify at least a portion of the mapped anatomical structures (e.g., structures that serve as landmarks) so that those portions of the anatomical structures can be matched to corresponding anatomical structures identified (e.g., separated or segmented) in the 3D model during registration. Additionally, if deformations are compensated for during registration, at least the anatomical structures in the target region (e.g., the nipple) may be separately identified.

[0091] The multiple anatomical structures may be identified using one or a combination of multiple identification techniques. For example, using a first identification technique, the multiple anatomical structures may be identified by providing multiple images (e.g., 2D image data) as input to a trained computer vision or machine learning model. For example, the computer vision or machine learning model may be trained and implemented to predict multiple anatomical structures within multiple images (e.g., within multiple 2D images) captured by the medical device 120 using a similar technique described above in step 304 for predicting multiple anatomical structures within multiple 3D images. However, in this embodiment, the training data includes multiple real and / or synthetic 2D endoscopic images rather than multiple real and / or synthetic 3D images such as CT images, MRCP images, or US images. The different anatomical forms or appearances of the multiple anatomical structures, their spatial proximity to other anatomical structures, and / or the temporal order in which the multiple anatomical structures appear during a medical procedure may be learned as part of the training. Thus, each received image may be provided as input data to the trained model, which may output one or more predicted anatomical structure locations within the image.

[0092] As previously mentioned, as the medical device 120 is advanced through the body lumen toward the target site, images of the body lumen and the target site may be displayed in real time by one of the one or more displays 110. In some examples, indications of predicted anatomical sites output by the trained model may be displayed in relation to the locations of the anatomical sites within the displayed images.

[0093] Additionally or alternatively, for example, when the medical device 120 reaches a known anatomical site and the anatomical site is visualized in a displayed image, a prompt may be generated and displayed requesting manual input from the operator to identify or confirm the anatomical site. For example, a trained model may be executed to predict an anatomical site such as the cardia of the stomach, and then a prompt may be generated and displayed instructing the operator to confirm that the anatomical site identified in the image is indeed the cardia. The prompt may be displayed adjacent to an indication of the predicted anatomical site of the cardia. In other examples, rather than employing a computer vision or machine learning model to automatically identify or predict multiple anatomical sites, the operator may instead be prompted to identify at least a portion of multiple anatomical sites displayed in the multiple images. The portion of the multiple anatomical sites may include landmarks that facilitate accurate image registration, as described in detail below.

[0094] Additionally or alternatively, a second identification technique may be used to identify multiple anatomical structures by first mapping the geometry of the anatomical structure site as multiple 3D surfaces (e.g., forming part of a 3D surface map) and then using the 3D surface map to identify the anatomical structure. For example, any multiple anatomical structure site identified by its geometry may be mapped as multiple 3D surfaces using multiple images captured by the medical device 120 and spatial information from one or more EM sensors 132, other optional sensors incorporated into the medical device 120, such as accelerometers, and / or information from other positioning systems, such as LIDAR (light detection and ranging), ultrasound ranging (using pulse-echo or transmit-receive methods), stereoscopic cameras, fluoroscopic images, intraoperative 3D radiological images, structured light images, etc.

[0095] In some cases, particularly when the imaging device 212 of the medical device 120 has a limited range of field of view, the spatial information (e.g., as determined by the position sensing system 104) and the geometry of the distal tip 210 of the medical device 120 may also be used to generate a contact map that estimates the geometry of the anatomical region when the position of travel of the distal tip 210 is restricted. For anatomical regions with easily deformable boundaries, the contact map can estimate the tissue walls or boundaries of the anatomical region by measuring the deceleration of the distal tip 210.

[0096] In some examples, at least one anatomical structure site, such as the papilla for an ERCP procedure, may be identified based on the medical procedure and may need to be included in the 3D surface map. In the case of a patient with pancreas divisum, the major and minor papilla may be identified separately during the ERCP procedure. Once the medical device 120 reaches the papilla, the papilla may be automatically identified using one or a combination of the identification techniques described above. Additionally, in some examples, the registration performed in step 312, described in detail below, may begin before the papilla is identified if the 3D surface map is being generated. In such examples, the operator may be guided to the approximate location of the papilla when registration is partially completed. For example, to prompt the operator to use one of the knobs 222, 224 to control the steerable section 218, multiple visual indicators, such as arrows or other similar directional graphical components, may be provided for display on the image captured by the imaging device 212 and displayed on one of the one or more displays 110. Additionally, once a nipple is identified, a visual indication of the location of the identified nipple may be provided for display on the current image being captured by imaging device 212 and displayed on one of the one or more displays 110. Alternatively, if the identified nipple is no longer in the field of view, one or more visual indicators similar to those described above may be provided for display instructing the operator to return to the location of the identified nipple. In some examples, the operator may be requested to confirm the automatically identified nipple with a prompt or notification.

[0097] In another embodiment, the operator may be prompted to manually identify the papilla by clicking, tapping, or otherwise selecting the papilla in the displayed image. In other examples, the application may wait until a cannulation tool (e.g., one of the one or more tools 123 inserted into the medical device 120 via the working channel and exiting the working channel at the distal tip 210) is inserted through the papilla to identify the papilla. For example, the application may identify the papilla based on where the cannulation tool intersects tissue in a current image of the target site and / or where the tip of the cannulation tool is located (e.g., the tool is fluoro-opaque) on a fluoroscopic image taken during the medical procedure.

[0098] In another example, as the 3D surface map is generated and multiple anatomical structures are identified, a mapping progress indicator may be generated and displayed. For example, the mapping progress indicator may include a percentage or a graphical component (e.g., a pie chart, a bar graph, etc.) that indicates or suggests the percentage or confidence of sufficient mapping data (e.g., portions of the multiple identified anatomical structures) that can be correlated to the 3D model. Additionally or alternatively, the 3D surface map itself in its current state may be displayed to inform the operator of the map's level of completeness and / or accuracy.

[0099] If there is insufficient image or other mapping data, a notification may be generated requesting additional data and / or indicating that registration accuracy may be unreliable if further data is not collected. As one example, image and / or mapping data may be insufficient if one or more landmarks (i.e., an insufficient number of landmarks) were not identified and / or if the image in which one or more of the landmarks were identified is blurred. In response, a notification may be generated that includes a prompt instructing the operator to move the medical device 120 to capture another image of the missing and / or blurred landmark(s). Additionally, the prompt may instruct the operator to confirm or manually identify the missing and / or blurred landmark(s). As another example, a notification may be generated that includes a prompt instructing the operator to move the medical device 120 to return to view the easiest landmarks to identify and / or have the operator confirm or manually identify these landmarks.

[0100] In some examples, if intraoperative 3D imaging is available, intraoperative 3D images (e.g., different from the 3D images received in step 302) may be captured during the medical procedure to generate a 3D surface map more quickly and / or accurately. The intraoperative 3D images may also help correct for or account for deformation or displacement due to differences in the patient's position or the presence of the medical device 120 within the body lumen. Capturing and utilizing intraoperative 3D images to supplement the mapping data may reduce the amount of time spent collecting data using the medical device 120, as only minimal 3D surface mapping data may be required to register a 3D model already corrected for deformation. This may result in a shorter overall procedure time while further improving accuracy. The intraoperative 3D images may be captured either before or after identifying the papilla when the medical device 120 (i.e., the distal tip 210) is positioned within the duodenum to accurately depict any deformation or displacement due to the presence of the medical device 120.

[0101] In step 312, the process 300 may include registering the 3D model to the patient using the 3D surface map and spatial information of the medical device 120. The registration may include determining a transformation matrix. The registration may begin while the 3D surface map is being generated but may not be completed until certain structural regions are mapped to the 3D surface map. The structural regions mapped may include the duodenum and / or the papilla if the medical procedure is an ERCP procedure. The spatial information of the medical device 120 may include the position and / or orientation of the medical device 120 relative to the 3D surface map. Additionally, if spatial information of the patient P is optionally received in addition to the spatial information of the medical device received from the position sensing system 104 in step 308, the spatial information of the patient P may also be used to register the 3D model to the patient P to account for any movement, including, for example, respiratory movement, of the patient P.

[0102] In some examples, the registration performed may be automatic registration using one or more algorithms. For example, as described in more detail with reference to FIG. 5 , automatic registration may include an initial registration based on aligning or matching anatomical structures in the 3D surface model to corresponding anatomical structures identified in the 3D model. For example, in the case of image registration performed for an ERCP procedure, the extracted 3D model may identify anatomical structures of the upper GI tract and the biliary-pancreatic system (among other structures). Similarly, with respect to image registration performed for an ERCP procedure, the 3D surface map may include anatomical structures of at least a portion of the upper GI tract and the papilla. The anatomical structures of at least a portion of the upper GI tract included in the 3D surface map may be aligned or otherwise matched to the corresponding anatomical structures of the upper GI tract in the 3D model. Based on the initial registration, a transformation matrix that compensates for the deformation and / or displacement of the multiple anatomical structures may then be calculated by weighting the susceptibility of multiple specific regions of the anatomy to known deformations and correlating the deformation or displacement from a specific anatomical structure to another anatomical structure. In some examples, spatial information received from the position sensing system 104 about the medical device 120 (and, optionally, the patient P) may be used to estimate the deformation and / or displacement.

[0103] In other examples, automatic registration may be performed using a computer vision or machine learning model trained and implemented to perform the registration. In further examples, registration may be manual registration and / or a combination of automatic and manual registration. Additionally, in any of these examples, the registration process may further include determining a confidence level and / or a percentage of registration accuracy.

[0104] The determined transformation matrix may be applied to the 3D model to transform the 3D model. The 3D model to which the transformation matrix is ​​applied may be the 3D model extracted in step 304. As a result, the transformed 3D model may take into account any deformations and / or displacements of multiple anatomical structure sites due to the patient's position, the presence of the medical device 120, and / or physiological function, for example, to align the anatomical structure in the 3D image with the patient's current anatomical structure.

[0105] Once the 3D model is registered to the patient, in step 314, process 300 may include generating a GUI that overlays a representation of the location or trajectory of one or more anatomical sites of interest onto a current image of the target site. For example, for an ERCP procedure, a representation of the bile duct (or at least the common bile duct of multiple bile ducts) and / or pancreatic duct extending from the papilla may be overlaid onto a live image of the papilla imaged by imaging device 212 of medical device 120. The representations of the bile duct and / or pancreatic duct may be overlaid at the approximate location where these ducts would appear if they were viewed through imaging device 212 (e.g., creating an augmented reality image).

[0106] In some examples, the representation may be a portion of the converted 3D model itself, including the multiple anatomical regions of interest. In other examples, the representation may be in the form of a wireframe model, a centerline, a series of discs positioned orthogonal to the centerline, a tubular structure, or the like, that may be generated using the converted 3D model. In additional examples, the overlay may indicate the approximate size (e.g., diameter) of the multiple anatomical regions of interest using various sizes, colors, and / or appearances. In further examples, the overlay may include multiple features to indicate a confidence interval or potential error in the alignment of the overlay (e.g., that may be determined as part of the registration process in step 312). Exemplary graphical user interfaces including the overlay are shown in Figures 6B and 6C below.

[0107] In step 316, the process 300 may include displaying a GUI on a display device. The display device may be one of the one or more displays 110 displaying a current image of the target site. Consequently, by displaying the GUI, a representation of the position or trajectory of one or more anatomical structures of interest may be overlaid on the current image of the target site. For example, the GUI may overlay a representation of the position or trajectory of the bile duct and / or pancreatic duct to be overlaid on the current image of the papilla. Based on the displayed GUI, the operator may confirm and / or adjust the position of the medical device 120 (e.g., extend or retract the distal tip 210 and / or manipulate one or more of the knobs 222, 224 to control the steerable section 218) and / or adjust a tool for performing the cannulation (e.g., one of the one or more tools 123 using the elevator 216), for example, to help ensure alignment with the particular vessel being cannulated.

[0108] After the GUI is displayed and cannulation facilitated by the GUI is initiated, the duodenal papilla or wall may be moved or displaced intentionally to access the papilla or incidentally as the cannulation tool is advanced through the papilla. In some embodiments, to compensate for that movement and / or displacement, the overlaid representation may be locally deformed to match the movement at the papilla to provide a more accurate estimate of the tube position and / or trajectory as it changes during the cannulation process. Movement at the papilla may be identified by tracking the position of the papilla using machine learning algorithms or other computer vision techniques. In some examples, if the operator repositions the medical device 120 during cannulation, fluoroscopic images captured by one of the one or more imaging systems 106 can provide additional context about how the shaft 208 and / or steerable section 218 moved to understand how the papilla and / or duodenal wall has changed. Additionally, in instances where the representation of the bile duct and / or pancreatic duct position or trajectory is not part of the 3D transformed image itself, but rather is in centerline form, among other similar instances, the representation may not require transformation (e.g., based on the implication that the type of representation conveys that the duct position and trajectory are approximate and not exact). Alternatively, the GUI may be removed from the display once the tip of the cannulation tool contacts the duodenal wall and / or enters the papilla.

[0109] In further embodiments, the GUI may be updated in real time as one or more other tools (e.g., from the one or more tools 123), such as a cholangioscope, catheter, balloon, stent delivery system, forceps, basket, net, biopsy needle, guidewire, etc., are advanced through one of these vessels after cannulation. For example, if spatial information (e.g., position and / or orientation) of the one or more tools 123 is tracked, the GUI may be updated in real time to overlay a representation of the one or more tools 123 advancing through a representation of the respective vessel. The position of the one or more tools 123 may be tracked using an EM-based tracking system, such as the position sensing system 104. For example, one or more tool EM sensors 133 similar to the one or more EM sensors 132 and one or more patch EM sensors 135 may be positioned, for example, at the distal tip of the one or more tools 123. Additionally or alternatively, the position of the one or more tools 123 may be tracked using a shape-sensing tracking system (through fiber optics), using accelerometers, and / or by fluoroscopic imaging.

[0110] In some examples, the lengths of the one or more tools 123 can be represented using the tracked spatial information and the known lengths of the one or more tools 123 to show the one or more tools 123 from a third person point of view (POV) as they advance through the desired canal. The positions and / or trajectories of the one or more tools 123 may be represented in an overlaid portion on the current image at the target site of the nipple. In other examples, the distal tips of the one or more tools 123 may be represented using the tracked spatial information to show the one or more tools 123 from a first person POV as they advance through the desired canal. For example, the positions and / or trajectories of the one or more tools 123 may be represented in an overlaid portion on the current image at the target site of the nipple. Additionally or alternatively, if one of the one or more tools 123 is a cholangioscope, a representation of the duct through which the cholangioscope is being advanced may be overlaid onto a current 2D image captured by the cholangioscope's imaging device (e.g., similar to imaging device 212 of medical device 120). Overlaying the representation of the duct may be useful to allow the operator to see where the duct leads past visual obstructions within the duct, such as stones, tumors, etc., that may otherwise prevent those portions of the duct from being visualized by the cholangioscope's imaging device.

[0111] The visualized and / or tracked locations of obstacles, stents, procedures, biopsy samples, tool paths, etc. may be recorded and referenced for follow-up procedures or imaging. For example, the recorded locations may be provided to data storage system 108 for storage in association with other images and information of patient P. This information may be used to plan future procedures, monitor patient outcomes, and / or explain procedures to patients, or may be included as part of research or academic disclosures.

[0112] Accordingly, some embodiments may be implemented for image registration. The process 300 described above is provided by way of example only and may include additional, fewer, different, or differently configured steps than those shown in FIG.

[0113] 4 shows an example process 400 for extracting a 3D model for use in image registration. Process 400 may be performed by one or more of the components of environment 100, for example, via an application running on the component configured to perform at least several operations associated with processing the 3D image to extract the 3D model. In some examples, process 400 may be performed by a computing device of one of one or more imaging systems 106 that captured the 3D image from which the 3D model can be extracted, or a computing device associated with that imaging system 106. In other examples, process 400 may be performed by MD controller 122 of medical device system 102. In further examples, process 400 may be performed by computing device 112. Process 400 may be used to perform at least a portion of step 304 of process 300 to extract the 3D model.

[0114] In step 402, process 400 may include receiving a 3D image of a patient's anatomy, such as patient P. The 3D image may be of a specific anatomical structure depending on the type of medical procedure (e.g., including multiple specific anatomical locations). The 3D image may be a pre-operative 3D image captured by one of the pre-operative imaging systems of the one or more imaging systems 106 prior to the medical procedure, or, if available, the 3D image may be an intra-operative 3D image captured by one of multiple intra-operative imaging systems of the multiple imaging systems during the medical procedure. This step 402 may be the same as or similar to step 302 of process 300, described in more detail above with reference to FIG. 3.

[0115] At decision step 404, process 400 may include determining whether the image quality of the 3D image received at step 402 meets a preset threshold. The preset threshold may be based on a minimum image quality level required to extract a 3D model from the 3D image so that multiple anatomical structures, including one or more anatomical structures of interest, can be identified (e.g., separated or segmented). If at decision step 404 it is determined that the image quality of the 3D image meets (e.g., is equal to or greater than) the preset threshold, the process may proceed to step 406.

[0116] Otherwise, if at decision step 404 it is determined that the image quality of the 3D image does not meet (e.g., is below) the preset threshold, process 400 returns to step 402, where another 3D image of the patient's anatomy may be received. In some examples, the other 3D image may be another pre-operative 3D image captured by one of the one or more imaging systems 106 and stored in data storage system 108. For example, the operator may select the other 3D image from multiple 3D images of patient P available in data storage system 108. In other examples, the operator may receive a notification that the 3D image is insufficient and may prescribe additional pre-operative imaging of patient P, and / or additional pre-operative imaging may be automatically prescribed. In a further example, 3D intra-operative imaging, if available, may be performed during the medical procedure to capture the other 3D image. The other 3D image may then be analyzed to determine whether the image quality threshold is met. Steps 402 and 404 may continue to be repeated until a 3D image determined to meet the minimum image quality level is received. Process 400 may then proceed to step 406.

[0117] In step 406, process 400 may include processing the 3D image to extract a 3D model. The 3D model may identify multiple anatomical sites within the anatomy, including one or more anatomical sites of interest for a medical procedure. This step 406 may be the same as or similar to step 304 of process 300, described in detail above with reference to FIG. 3 .

[0118] At determination step 408, process 400 may include determining whether the 3D model extracted at step 406 meets a predetermined completeness threshold. The predetermined completeness threshold may be based on a minimum number and / or type of identified complete anatomical structures, including a minimum number and / or type of complete anatomical structures that serve as landmarks for registration (e.g., to ensure accurate registration). For example, for an ERCP procedure, the predetermined completeness threshold may include at least a complete common bile duct and a complete pancreatic duct, among other types of complete anatomical structures. If the 3D model extracted at step 406 is determined to meet the completeness threshold at determination step 408 (e.g., is greater than or equal to the completeness threshold), process 400 may end at step 410. In some examples, the extracted 3D model may be transmitted to another system and / or computing device within environment 100 for processing, analysis, storage, display, etc. Otherwise, if the 3D model extracted in step 406 is determined in decision step 408 to not meet the completeness threshold (e.g., is less than the completeness threshold), process 400 may proceed to step 412.

[0119] At step 412, process 400 may include estimating one or more incomplete anatomical structures. Estimated models of multiple incomplete anatomical features may be calculated based on available anatomical structure information. As an example, the common bile duct and pancreatic duct identified in the extracted 3D model may be incomplete. For example, the last few millimeters of the common bile duct and pancreatic duct, where they enter the duodenum (e.g., short portions of these ducts), may not be visible on 3D CT and / or MRCP images because, for example, the papilla constricts around these portions of these ducts. The constriction of the papilla around these portions of these ducts may result in surrounding fluid acting as a contrast agent for the 3D CT and / or MRCP images. Because these portions of these ducts are not visible on the 3D CT and / or MRCP images, these portions of these ducts are not identified (e.g., separated or segmented) in the 3D model extracted from the 3D images. To complete the 3D model, imperfections in these ducts can be estimated or approximated by extrapolating the ducts based on their trajectories up to the imperfections to where they intersect with the duodenum.

[0120] In some examples, the operator may be asked to confirm whether the estimate of one or more incomplete anatomical structures is a reasonable estimate. For example, a 3D model including the estimated incomplete anatomical structures may be provided for display along with a prompt for confirmation. The prompt may be displayed (e.g., on one or more displays 110) in association with the estimated incomplete anatomical structures. Additionally or alternatively, the operator may be allowed to manually estimate the trajectory of the incomplete anatomical structures. For example, the operator may provide the estimated trajectory via touch or other input to one of the one or more displays 110 and / or associated computing devices (e.g., a computing device of one of the one or more imaging systems 106, the MD controller 122, and / or the computing device 112).

[0121] Accordingly, several particular embodiments may be implemented to process a 3D image to extract a 3D model. The process 400 described above is provided by way of example only and may include additional, fewer, different, or differently configured steps than those shown in FIG.

[0122] 5 shows an example process 500 for registering a 3D model to a patient. Process 500 may be performed by one of multiple components of environment 100, such as MD controller 122 or computing device 112 of medical device system 102, via an application running on that component configured to perform at least multiple operations related to registration. Process 500 may be used to perform at least a portion of step 312 of process 300 to register the 3D model, which may include determining a transformation matrix.

[0123] In step 502, process 500 may include matching one or more portions of the anatomical structure sites included in the 3D surface map generated in step 310 of process 300 with corresponding anatomical structure sites in the 3D model extracted in step 304 of process 300. Continuing with the example of an ERCP procedure, the extracted 3D model may identify or isolate anatomical structure sites of the upper GI tract and biliary-pancreatic system (among other structures), and the 3D surface map includes at least a portion of the anatomical structure sites of the upper GI tract and the papilla. In step 502, at least a portion of the anatomical structure sites of the upper GI tract included in the 3D surface map may be matched or aligned to the corresponding anatomical structure sites of the upper GI tract identified in the extracted 3D model.

[0124] At step 504, process 500 may include performing an initial registration based on the matching. In some examples, the initial registration may be rigid image registration using methods and / or processes commonly known or that may become known in the art. For example, multiple parameters of a transformation matrix may be identified to map multiple voxel locations in the 3D model to a 3D surface map. In some examples, the initial registration may require matching and / or alignment of a predetermined number (or types) of anatomical structures. For example, at least three anatomical structures may be matched or aligned.

[0125] The initial registration performed may assume that the anatomical regions are rigid structures. However, the anatomical regions are not rigid structures. The anatomical regions within the 3D surface map may be deformed and / or displaced. For example, this deformation and / or displacement may be based on how the patient P was positioned during the medical procedure relative to how the patient P was positioned when the 3D images used to extract the 3D model were captured, based on the introduction of the medical device 120 into the body lumen during the medical procedure, and / or based on other naturally occurring physiological functions of the anatomical structures, such as respiration. For example, the orientation of the duodenum and pylorus may appear rotated relative to the axis of the esophagus between the 3D surface map and the 3D model, which may occur if the 3D model images the patient in a different position than the patient's position during the ERCP procedure. As another example, the shape of an anatomical region such as the stomach may change significantly as its walls collapse or expand due to changes in internal pressure, but the common bile duct and pancreatic duct may not be deformed or displaced to the same extent.

[0126] To improve the accuracy of registration, deformations and / or displacements of anatomical regions may be compensated for. For example, in step 506, process 500 may include determining deformation compensation. The deformation compensation may be determined by weighting the susceptibility of certain regions of the anatomy to known deformations and correlating the deformations or displacements from certain anatomical regions to other anatomical regions.

[0127] For example, knowing that the shape of the stomach may be deformed or displaced to a significantly greater extent than the common bile duct and / or pancreatic duct, any deformation may be compensated for to a lesser extent (i.e., compared to the stomach) with respect to the transformation parameters corresponding to the common bile duct and / or pancreatic duct. For example, based on the assumption that a short portion of the common bile duct and / or pancreatic duct is elastic, while the remaining portions of each of these ducts are relatively fixed, the position or trajectory of the fixed portion of each of these ducts may be continuously derived (e.g., using a spline function).

[0128] As another example, other structures identified in the extracted model, such as the spine, ribs, etc., may inform susceptibility to deformation. For example, if another anatomical structure is located near a region of the upper GI tract, that region may be more rigid and less susceptible to deformation.

[0129] As a further example, if the orientation of the duodenum and pylorus appears rotated relative to the axis of the esophagus, then the transformation parameters corresponding to the biliary-pancreatic system may be adjusted accordingly, under the assumption that the biliary-pancreatic system is similarly rotated. In some examples, the assumption may be learned from a training set of patient images analyzed to identify representative locations of the biliary-pancreatic system based on a given orientation of the duodenum and pylorus relative to the axis of the esophagus.

[0130] In some examples, spatial information about the medical device 120 from the position sensing system 104 may be used to estimate deformation and / or displacement. For example, the position and / or orientation of the distal tip 210 and / or any other position of the medical device 120 where one or more EM sensors 132 may be positioned may be used to estimate deformation and / or displacement. Additionally or alternatively, the position of the patient P from one or more patch EM sensors 135 may be used to estimate deformation and / or displacement.

[0131] At step 508, process 500 may include determining a transformation matrix based on the initial registration and the deformation compensation. In some examples, the transformation matrix may first be based on multiple parameters of the initial registration, which may be adjusted or changed using the deformation compensation.

[0132] Accordingly, certain embodiments may be implemented to register a 3D model to a patient. The process 500 described above is provided merely as an example and may include additional, fewer, different, or differently configured steps than shown in FIG.

[0133] In this disclosure, various steps may be described as being performed or processed by one of multiple components from FIGS. 1A and 1B , such as a computing device of one of the one or more imaging systems 106 or a computing device associated with the imaging system(s) 106, the MD controller 122 of the medical device system 102, or the computing device 112. However, it should be understood that in various embodiments, various components of the environment 100 described above may execute multiple instructions or perform multiple steps, including those described above. Steps performed by a device are considered to be performed by a processor, actuator, or the like associated with that device. Furthermore, it should be understood that in various embodiments, various steps may be added, omitted, and / or rearranged in any suitable manner.

[0134] 6A-6C illustrate exemplary GUIs 600A, 600B, and 600C that are generated and displayed during a medical procedure, such as an ERCP procedure. A first GUI 600A, shown in FIG. 6A, displays a current image 602 of a papilla 604, imaged by the imaging device 212 of the medical device 120 during, for example, an ERCP procedure. The papilla 604 may be a target site for cannulation during the ERCP procedure. A second GUI 600B, shown in FIG. 6B, displays a portion of a transformed 3D model 610 representing the position and trajectory of a common bile duct 612 and a pancreatic duct 614, overlaid on the current image 602. Thus, the operator sees the common bile duct 612 and the pancreatic duct 614 in the approximate locations where these ducts would appear if viewed through the imaging device 212. Using the second GUI 600B as a guide, the operator of the medical device 120 can confirm and / or adjust the position of the medical device 120 (e.g., by manipulating one or more of the knobs 222, 224 to extend or retract the distal tip 210 and / or to control the steerable section 218) and / or the tool for performing the cannulation (e.g., one of the one or more tools 123 using the elevator 216), for example, to help ensure alignment with the particular vessel being cannulated.

[0135] In other examples, rather than overlaying a portion of the transformed 3D model itself onto the current image 602, representations of the positions and / or trajectories of these ducts may be overlaid in the form of a wireframe model generated using the transformed 3D model, a centerline, a series of disks positioned orthogonal to the centerline, a tubular structure, etc. The third GUI 600C shown in FIG. 6C displays exemplary centerline representations 620 of these ducts. For example, the first visual indicator 622 may be a centerline representation of the common bile duct, and the second visual indicator 624 may be a centerline representation of the pancreatic duct. The first visual indicator 622 and the second visual indicator 624 may be visually distinct from one another to highlight the different trajectories of the common bile duct and the pancreatic duct.

[0136] Although not shown in Figures 6A-6C, various other visual schemes may be used within the GUIs generated and displayed during a medical procedure. The GUIs 600A, 600B, 600C described above are provided by way of example only and may include additional steps, fewer steps, different or differently organized information and / or features relative to those shown in Figures 6A-6C.

[0137] FIG. 7 illustrates an example of a computer 700. FIG. 7 is a simplified functional block diagram of a computer 700 that may be configured as a device for performing the processes, steps, or operations shown in or described with respect to FIGS. 1-6C , according to exemplary embodiments of the present disclosure. For example, the computer 700 may be configured as one of the computing devices of one or more imaging systems 106 or associated with the imaging system 106, the MD controller 122 of the medical device system 102, the computing device 112, the PSS controller 130 of the position sensing system 104, and / or another device or component, according to exemplary embodiments of the present disclosure. In various embodiments, any of the systems herein may be or include the computer 700, including, for example, a data communication interface 720 for packet data communication. The computer 700 may communicate with one or more other computers using, for example, an electronic network 725 (e.g., via the data communication interface 720). The electronic network 725 may include a wired or wireless network similar to the network 114 shown in FIG. 1 .

[0138] The computer 700 may also include a central processing unit ("CPU") in the form of one or more processors 702 that execute program instructions 724. The program instructions 724 may include instructions for executing one or more applications related to 3D model extraction, 3D surface map generation, and / or 3D image registration and GUI generation on one of the imaging systems 106, medical device systems 102, or computing devices 112 (e.g., when the computer 700 is a computing device of or associated with one of the one or more imaging systems 106, medical device systems 102, or computing devices 112). The program instructions 724 may include instructions for performing one or more operations for determining position and / or orientation (e.g., when the computer 700 is a PSS controller 130 of the position sensing system 104). The computer 700 may include an internal communication bus 708. The computer may include a drive unit 706 (such as a read-only memory (ROM), hard disk drive (HDD), solid-state disk drive (SDD), etc.) that may store data on a computer-readable medium 722 (e.g., a non-transitory computer-readable medium), although the computer 700 may receive programming and data via network communications. The computer 700 may also have memory 704, such as a random-access memory (RAM), that stores instructions 724 for performing the techniques presented herein. It should be noted, however, that in some aspects the instructions 724 may be temporarily or permanently stored within other modules of the computer 700 (e.g., the processor 702 and / or the computer-readable medium 722).The computer 700 may also include user input and output devices 712 and / or a display 710 for connecting with input and / or output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Various system functions may be implemented in a distributed manner across several similar platforms to distribute the processing load. Alternatively, the system may be implemented by appropriate programming of a particular computer hardware platform.

[0139] Program aspects of the present technology can be thought of as "products" or "articles of manufacture" in the form of executable code and / or associated data typically carried on or embodied in some type of machine-readable medium. "Storage"-type media include any or all of the tangible memory of a computer, processor, or the like, or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc., capable of providing non-transitory storage for software programming at any time. All or portions of the software may sometimes be communicated via the Internet or various other telecommunications networks. Such communication may, for example, enable loading of the software from one computer or processor to another. Accordingly, other types of media capable of carrying software elements include optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline networks, and over various airlinks. Physical elements carrying such waves, such as wired or wireless links, optical links, etc., can also be considered software-bearing media. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0140] While the principles of the present disclosure have been described herein with reference to illustrative examples for particular applications, the disclosure is not limited thereto. Those skilled in the art and with access to the teachings provided herein will recognize that additional modifications, adaptations, and equivalent substitutions are all within the scope of the embodiments described herein. Accordingly, the present invention should not be deemed limited by the foregoing description.

Claims

1. A system for three-dimensional (3D) image registration to facilitate medical procedures, A medical device having a distal portion configured to be inserted into a patient's body lumen during a medical procedure, An imaging device located at the distal tip of the distal portion, configured to capture a plurality of images of the body lumen as the medical device is inserted into the body lumen and moves through the body lumen to a target site, wherein at least one of the plurality of images includes a current image of the target site, and the imaging device and A medical device comprising a transmitting device or receiving device of a position sensing system located at the distal tip, wherein the position sensing system is configured to determine the position or orientation of the distal tip, and the transmitting device or receiving device, A non-temporary computer-readable medium for storing multiple instructions, wherein, when the multiple instructions are executed by the processor of a computing device, the processor causes the processor to perform multiple operations, and the multiple operations are Receiving a 3D image of the patient's anatomical structure captured by an imaging system before the aforementioned medical procedure, The process involves processing the 3D image to extract a 3D model that identifies multiple anatomical structures within the anatomical structure, wherein the multiple anatomical structures include one or more anatomical structures of interest for the medical procedure. Receiving the plurality of images of the body lumen captured by the imaging device during the medical procedure, Receiving the position or orientation of the distal tip of the medical device from the position sensing system, The process involves processing the plurality of images of the body lumen and the position or orientation of the distal tip to generate a 3D surface map of at least a portion of the body lumen, wherein the 3D surface map includes portions of the plurality of anatomical structures that do not include one or more anatomical structures of interest. Registering the 3D model to the patient using the 3D surface map and the position or orientation of the distal tip, Based on the registration, a graphical user interface (GUI) is generated that overlays a representation of the location or trajectory of one or more anatomical structures of interest onto the current image of the target site. A system including displaying the GUI on a display device.

2. The aforementioned multiple operations are, Determining that one or more of the multiple anatomical structural sites identified in the 3D model are incomplete, The system according to claim 1, comprising estimating an incomplete portion of one or more of the aforementioned plurality of anatomical structural sites.

3. The aforementioned multiple operations are, To generate a prompt that has instructions for the operator to confirm the estimated incomplete portion, The system according to claim 2, comprising displaying the 3D model including the estimated incomplete portion and the prompt via the display device, wherein the prompt can be displayed in relation to the estimated incomplete portion of the 3D model.

4. The aforementioned multiple operations are, Based on the number or type of anatomical structures in a portion of the plurality of anatomical structures included in the 3D surface map, it is determined that additional image data is required to generate the 3D surface map. To acquire the additional image data via the imaging device of the medical device, the system generates prompts having multiple instructions to move the medical device to one or more locations within the body lumen corresponding to one or more anatomical structural sites among the plurality of anatomical structural sites that are not included in the portion or are included in the portion but are incomplete, The system according to any one of claims 1 to 3, further comprising displaying the prompt using the display device.

5. Registering the aforementioned 3D model to the patient is Determining the transformation matrix, The system according to any one of claims 1 to 3, further comprising applying the transformation matrix to the 3D model in order to transform the 3D model.

6. Registering the aforementioned 3D model to the patient is Matching one or more of the portions of the plurality of anatomical structural regions included in the 3D surface map to the corresponding plurality of anatomical structural regions in the 3D model, Based on the aforementioned matching, the initial registration is performed, Determining deformation compensation, The system according to claim 5, further comprising determining the transformation matrix based on the initial registration and the deformation compensation.

7. Generating the aforementioned GUI is This includes generating the GUI using the converted 3D model, The system according to claim 5, wherein the representation of the position or trajectory of the one or more anatomical structures of interest includes at least one of the following: a portion of the converted 3D model including the one or more anatomical structures of interest; a wireframe model of the one or more anatomical structures of interest; a representation of the centerline of the one or more anatomical structures of interest; a series of disks arranged perpendicular to the centerline of the one or more anatomical structures of interest; or a tubular structure of the one or more anatomical structures of interest.

8. The aforementioned multiple operations are, Receiving spatial information about the patient from the position sensing system, wherein the position sensing system includes one or more transmitting or receiving devices located within a patch applied locally to the patient. The system according to any one of claims 1 to 3, further comprising registering the 3D model to the patient using the 3D surface map, the spatial information of the medical device, and the spatial information of the patient.

9. The aforementioned multiple operations are, The system according to any one of claims 1 to 3, further comprising identifying one or more anatomical structures from a portion of the plurality of anatomical structures in the 3D surface map by providing the plurality of images as input to a machine learning model trained to predict the one or more anatomical structures present in each of the plurality of images.

10. The aforementioned multiple operations are, The system according to any one of claims 1 to 3, further comprising, as part of the generation of the 3D surface map, mapping the geometric shape of one or more anatomical structural sites as a 3D surface, and identifying the anatomical structures based on the mapped geometric shape, thereby identifying one or more anatomical structures from a portion of the plurality of anatomical structural sites in the 3D surface map.

11. The aforementioned target site is a site for cannulation, The aforementioned multiple operations are, Tracking the movement of the target area during cannulation, The system according to any one of claims 1 to 3, comprising updating the GUI to deform the representation of the position or trajectory of one or more anatomical structures of interest that are overlaid on the current image of the target site in order to match the movement of the target site.

12. The aforementioned multiple operations are, Receiving spatial information about a tool delivered to the target site via the medical device from the position sensing system as the tool is advanced through at least one of the one or more anatomical structures of interest, wherein the tool includes one or more transmitting devices or one or more receiving devices of the position sensing system. The system according to any one of claims 1 to 3, further comprising updating the GUI to depict a representation of the tool advancing through the representation of the position or trajectory of at least one of the anatomical structures of interest overlaid on the current image of the target site, using the spatial information of the tool.

13. The aforementioned multiple operations are, After the medical device reaches the target site through the body lumen, it receives additional 3D images captured during the procedure. The system according to any one of claims 1 to 3, further comprising processing the additional 3D image to generate the 3D surface map, in addition to the processing of the plurality of images.

14. The aforementioned multiple operations are, The system according to any one of claims 1 to 3, comprising determining whether the image quality of the 3D image satisfies a predetermined threshold before processing the 3D image and extracting the 3D model.

15. The medical procedure is an endoscopic retrograde cholangiopancreatography (ERCP) procedure, the target site is the papilla for cannulation, and the one or more anatomical sites of interest include at least the common bile duct and the pancreatic duct. Generating the aforementioned GUI is The system according to any one of claims 1 to 3, comprising creating an augmented reality image by using the registered 3D model to overlay the representation of the position or trajectory of the common bile duct and the pancreatic duct onto the current image of the papilla.

Images