System, apparatus and method for three-dimensional image registration

Abstract

A system includes a medical device and a computer readable medium storing instructions that, when executed by a processor, cause performing operations including receiving a three-dimensional (3D) image of an anatomical structure, processing the 3D image to extract a 3D model for identifying the anatomical structure of interest, and displaying the 3D model. Receiving an image captured by an imaging device of the medical device as the medical device navigates through a body cavity to a target site, including a current image of the target site, receiving spatial information of the medical device from a position sensing system including a transmitter / receiver in or on the medical device, the method includes receiving a current image, processing the image and spatial information to generate a 3D surface map of a body cavity, registering the 3D model to a patient using the map and spatial information, and generating and displaying a graphical user interface that overlays a representation of a position / trajectory of an anatomical structure of interest on the current image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 374,021, filed on August 31, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention generally relates to systems, devices and methods for three-dimensional (3D) image registration. More specifically, aspects of the present invention relate to devices, systems and / or methods for 3D image registration to generate and display a graphical user interface (GUI) to facilitate medical procedures. Background Art

[0004] Endoscopic retrograde cholangiopancreatography (ERCP) is a procedure that utilizes endoscopy and fluoroscopy to diagnose and / or treat diseases of the bile duct and pancreatic duct system, such as strictures. During an exemplary ERCP, an endoscope may be inserted into the patient's mouth, navigated along the esophagus through the stomach into the duodenum and to the papilla, where each of the common bile duct and pancreatic duct of the bile duct system enters the duodenum. Then, cannulation may be performed by inserting a guide wire, catheter, and / or other device through the papilla and into the common bile duct or pancreatic duct. Then, a contrast agent may be injected so that the operator can use fluoroscopy and visualize the cause of the stricture in the bile duct or pancreatic duct system on an x-ray image. Then, the operator may deliver and / or perform appropriate treatment.

[0005] It is known that intubation during ERCP surgery can cause various problems for gastroenterologists. Difficulty in intubation is usually due to the inability to visualize the position or trajectory of the common bile duct and / or pancreatic duct, making it difficult to insert a guide wire, catheter and / or other device into the required duct (e.g., into the bile duct rather than the pancreatic duct). This difficulty caused by lack of visualization may prolong the operation time. Additionally, although most attempted ERCP surgeries may be successfully intubated, failed attempts do occur, and patients usually need to be referred to tertiary care specialists. Even in the case of success, the action of intubation may also lead to post-ERCP pancreatitis when repeated attempts or excessive force are used. Summary of the invention

[0006] A system for three-dimensional (3D) image registration to facilitate medical surgery may include a medical device having a distal portion configured to be inserted into a body cavity of a patient during the medical surgery. The medical device may include an imaging device located at a distal tip of the distal portion and configured to capture multiple images of the body cavity when the medical device is inserted into the body cavity and navigated through the body cavity to a target site. At least one of the multiple images may include a current image of the target site. The medical device may also include a transmitter device or a receiver device of a position sensing system located at the distal tip, wherein the position sensing system is configured to determine the position or orientation of the distal tip. The system may also include a non-transitory computer-readable medium for storing instructions, which, when executed by a processor of a computing device, causes the processor to execute the instructions to perform operations. The operations may include receiving a 3D image of an anatomical structure of a patient captured by an imaging system before the medical surgery, and processing the 3D image to extract a 3D model, the 3D model identifying multiple anatomical structures in the anatomical structure, including one or more anatomical structures of interest for the medical surgery. The operation may also include receiving a plurality of images of a body cavity captured by an imaging device during a medical procedure; receiving a position or orientation of a distal tip of a medical device from a position sensing system; and processing the plurality of images of the body cavity and the position or orientation of the distal tip to generate a 3D surface map of at least a portion of the body cavity. The 3D surface map may include a portion of the plurality of anatomical structures that does not contain one or more anatomical structures of interest. The operation may also include registering the 3D model to the patient using the 3D surface map and the position or orientation of the distal tip; based on the registration, generating a graphical user interface (GUI) that overlays a representation of the position or trajectory of the one or more anatomical structures of interest on a current image of the target site; and displaying the GUI on a display device.

[0007] In any of the exemplary systems disclosed herein, the operation may also include determining that one or more of the multiple anatomical structures identified in the 3D model are incomplete; and estimating the incomplete portion of one or more of the multiple anatomical structures. The operation may also include generating a prompt having instructions for an operator to confirm the estimated incomplete portion; and displaying the 3D model via a display device, the 3D model including the estimated incomplete portion and the prompt, wherein the prompt may be displayed in association with the estimated incomplete portion of the 3D model. The operation may also include determining that additional image data is required to generate the 3D surface map based on the number or type of anatomical structures in the portion of the multiple anatomical structures included in the 3D surface map; generating a prompt having instructions for the operator to move the medical device to one or more positions in the body cavity, the one or more positions corresponding to one or more of the multiple anatomical structures that are not included in the portion or are included in the portion and are incomplete, to capture additional image data via an imaging device of the medical device; and displaying the prompt via the display device.

[0008] In some aspects, registering the 3D model to the patient may include determining a transformation matrix and applying the transformation matrix to the 3D model to transform the 3D model. One or more of the portions of the plurality of anatomical structures included in the 3D surface map may be matched to corresponding anatomical structures in the 3D model, an initial registration may be performed based on the matching, deformation compensation may be determined, and the transformation matrix may be determined based on the initial registration and deformation compensation. A GUI may be generated using the transformed 3D model, and a representation of the position or trajectory of the one or more anatomical structures of interest includes at least one of: a portion of the transformed 3D model including the one or more anatomical structures of interest, a wireframe model of the one or more anatomical structures of interest, a centerline representation for the one or more anatomical structures of interest, a series of disks positioned orthogonally to the centerline of the one or more anatomical structures of interest, or a tubular structure for the one or more anatomical structures of interest.

[0009] In other aspects, the operations may also include receiving spatial information of the patient from a position sensing system, wherein the position sensing system includes one or more transmitter devices or receiver devices located in a patch locally applied to the patient, and the 3D model may be registered to the patient using the 3D surface map, the spatial information of the medical device, and the spatial information of the patient. The operations may also include identifying one or more anatomical structures from portions of multiple anatomical structures in the 3D surface map by providing multiple images as input to a machine learning model, the machine learning model being trained to predict the one or more anatomical structures present in each of the multiple images. The operations may also include, as part of generating the 3D surface map, identifying one or more anatomical structures from portions of multiple anatomical structures in the 3D surface map by mapping the geometry of the one or more anatomical structures as a 3D surface, and identifying the anatomical structure based on the mapped geometry.

[0010] In further aspects, the target site can be a site for intubation, and when intubation occurs, movement at the target site can be tracked, and the GUI can be updated to deform a representation of the position or trajectory of the one or more anatomical structures of interest overlaid on a current image of the target site to match the movement of the target site. Spatial information of a tool delivered to the target site via a medical device can be received from a position sensing system as the tool is advanced through at least one of the one or more anatomical structures of interest, wherein the tool includes one or more transmitter devices or one or more receiver devices of the position sensing system, and the GUI can be updated using the spatial information of the tool to depict a representation of the tool advanced through a representation of the position or trajectory of at least one of the one or more anatomical structures of interest overlaid on a current image of the target site.

[0011] In other aspects, in addition to processing multiple images, additional 3D images captured intraoperatively after the medical device navigates through the body cavity to the target site may be received and processed to generate a 3D surface map. Additionally, it may be determined that the image quality of the 3D image meets a predefined threshold before processing the 3D image to extract the 3D model. The medical procedure may be an ERCP procedure. The target component may be a papilla for cannulation, and the one or more anatomical structures of interest may include at least the common bile duct and the pancreatic duct. A GUI generated using the registered 3D model may overlay a representation of the location or trajectory of the common bile duct and the pancreatic duct on a current image of the papilla to create an augmented reality image.

[0012] In other examples, a method for 3D image registration to facilitate a medical procedure may include receiving a 3D image of an anatomical structure of a patient, and processing the 3D image to extract a 3D model, the 3D model identifying a plurality of anatomical structures in the anatomical structure, including one or more anatomical structures of interest for the medical procedure. The method may also include receiving a plurality of images of a body cavity of the patient captured by an imaging device of the medical device when the medical device is inserted into the body cavity and navigated through the body cavity to a target site during the medical procedure. At least one of the plurality of images received may include a current image of the target site. The method may also include receiving spatial information of the medical device from a position sensing system, wherein the medical device may include a transmitter device or a receiver device located in or on the medical device; and processing the plurality of images of the body cavity and the spatial information of the medical device to generate a 3D surface map of at least a portion of the body cavity. The 3D surface map may include a portion of the plurality of anatomical structures that does not contain the one or more anatomical structures of interest. The method may also include registering the 3D model to the patient using the 3D surface map and the spatial information of the medical device. The registration may include determining a transformation matrix. The method may also include applying the transformation matrix to the 3D model to transform the 3D model; generating a GUI using the transformed 3D model to overlay a representation of the position or trajectory of one or more anatomical structures of interest on a current image of the target site; and displaying the GUI on a display device.

[0013] Any of the exemplary methods disclosed herein may include any of the following features. Spatial information of the patient may be received from a position sensing system, wherein the position sensing system may include one or more transmitter devices or receiver devices located in a patch locally applied to the patient. The 3D model is aligned to the patient using the 3D surface map, the spatial information of the medical device, and the spatial information of the patient. In order to align the 3D model to the patient, one or more of the portions of the plurality of anatomical structures included in the 3D surface map may be matched to corresponding anatomical structures in the 3D model, an initial alignment may be performed based on the matching, a deformation compensation may be determined, and a transformation matrix may be determined based on the initial alignment and the deformation compensation.

[0014] In another example, a method for 3D image registration to facilitate an ERCP procedure may include receiving a 3D image of an anatomical structure of a patient, and processing the 3D image to extract a 3D model, the 3D model identifying multiple anatomical structures in the anatomical structure, including the anatomical structure of the patient's upper gastrointestinal (GI) tract and the biliopancreatic tree, wherein at least the common bile duct and the pancreatic duct are anatomical structures of interest for the ERCP procedure. The method may also include receiving multiple images of the upper GI tract captured by an imaging device of a medical device while inserting the medical device into the patient's mouth and navigating through the patient's upper GI tract to the nipple during the ERCP procedure, including a current image of the nipple, the nipple being a target site for cannulation; receiving spatial information of the medical device from a position sensing system, the position sensing system including a transmitter device or a receiver device located in or on the medical device; and processing the multiple images and the spatial information of the medical device to generate a 3D surface map. The 3D surface map may include a portion of the multiple anatomical structures of the upper GI tract identified in the 3D model and the nipple. The method may also include registering the 3D model to the patient using the 3D surface map and spatial information of the medical device; generating a graphical user interface (GUI) based on the registration that overlays representations of the location or trajectory of at least the common bile duct and the pancreatic duct on a current image of the nipple; and displaying the GUI on a display device to provide visual guidance for nipple cannulation.

[0015] Any of the exemplary methods disclosed herein may include any of the following features. Movement of the papilla may be tracked as cannulation occurs. The GUI may be updated to deform a representation of the position or trajectory of the common bile duct and pancreatic duct overlaying a current image of the papilla to match the movement of the papilla.

[0016] It is to be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory of the invention as claimed, rather than restrictive descriptions. As used herein, the terms "include," "comprise," or any other variation thereof are intended to encompass non-exclusive contents, such that a process, method, article, or device comprising a list of elements may include not only those elements, but may include other elements that are not explicitly listed or are not inherent to such a process, method, article, or device. The term "exemplary" is used in the sense of "example" rather than "exemplary." The term "distal" refers to the direction away from the operator / towards the target site, and the term "proximal" refers to the direction toward the operator. The term "approximately" or similar terms (e.g., "substantially") include values ​​of ±10% of the specified value. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1A and Figure 1BAn exemplary environment is depicted in which three-dimensional (3D) image registration may be implemented to facilitate a medical procedure.

[0019] Figure 2A and Figure 2B Describes the use Figure 1A and Figure 1B An exemplary medical device in an exemplary environment.

[0020] Figure 3 An exemplary process for 3D image registration is depicted.

[0021] Figure 4 An exemplary process for extracting a 3D model for use in 3D image registration is depicted.

[0022] Figure 5 An exemplary process for registering a 3D model to a patient is depicted.

[0023] FIG. 6A to FIG. 6C An exemplary graphical user interface (GUI) is depicted.

[0024] Figure 7 An example of a computing device is depicted. DETAILED DESCRIPTION

[0025] As briefly described above, during an exemplary endoscopic retrograde cholangiopancreatography (ERCP), an endoscope, such as a duodenoscope, may be inserted into the patient's mouth and navigated along the esophagus through the stomach into the duodenum and to the papilla, where each of the common bile duct and the pancreatic duct enters the duodenum. Cannulation may then be performed by inserting a guidewire, catheter, and / or other device through the papilla and into the desired tube (e.g., the common bile duct or the pancreatic duct). The imaging device of the endoscope may capture images that enable visualization of the papilla. However, based on the anatomical structure of the papilla, the common bile duct and the pancreatic duct are not visible in the endoscopic image. As a result, since the location or trajectory of the common bile duct and the pancreatic duct cannot be visualized, it may be difficult to cannulate the papilla to insert the guidewire, catheter, and / or other device into the desired tube.

[0026] Due to the lack of visualization, multiple cannulation attempts may be performed and / or an incorrect tube may be inadvertently entered. Multiple cannulation attempts may increase the duration of the procedure. Additionally, repeated cannulation attempts may irritate the papillary tissue, which may cause tissue swelling and / or closure. The swelling and / or closure may prevent the pancreatic duct from draining properly, may cause fluid to accumulate in the pancreas and / or may cause post-ERCP pancreatitis. Furthermore, if an incorrect tube, such as the pancreatic duct, is entered and contrast is injected into the pancreatic duct to access fluoroscopy, the contrast may irritate the pancreas, often resulting in post-ERCP pancreatitis. In instances where multiple cannulation attempts are performed and / or contrast is incorrectly injected into the pancreatic duct, the patient may be referred to a tertiary care specialist to prevent and / or manage post-ERCP pancreatitis, which may be extremely distressing to the patient and expensive to the healthcare system.

[0027] In order to reduce the difficulty of intubation and reduce the number of patients suffering from post-ERCP pancreatitis, the embodiments disclosed herein propose systems, devices and methods for 3D image registration to enable generation of a graphical user interface (GUI) that overlays a representation of the position and / or trajectory of at least the common bile duct and pancreatic duct extending from the papilla on an endoscopic image of the papilla to provide visual guidance for papilla cannulation.

[0028] Figure 1A An exemplary environment 100 is depicted in which 3D image registration may be implemented to facilitate a medical procedure. Environment 100 may include a medical device system 102, a position sensing system 104, an imaging system 106, a data storage system 108, a display 110, and a computing device 112, each of which communicates with one or more other components of environment 100 via a wired or wireless network, such as network 114. Figure 1B Exemplary components of a medical device system 102 and a position sensing system 104 in environment 100 are depicted.

[0029] Also refer to Figure 1A and Figure 1B, the medical device system 102 may include a medical device 120 and a medical device (MD) controller 122. The medical device 120 may be used to perform a medical procedure. The medical device 120 may be an endoscope, and the endoscope may be a specialized type of endoscope for medical surgery. For example, the medical device 120 may be a duodenoscope for performing an ERCP procedure. In some examples, the medical device 120 may include one or more position sensing components of the position sensing system 104, such as an electromagnetic (EM) sensor 132, which is integrated in at least a distal portion (e.g., a distal tip) of the medical device 120 to enable tracking of the position and / or orientation of the medical device 120 during a medical procedure. Additionally or alternatively, the medical device 120 may include a fiber optic shape sensor, an accelerometer, and / or a gyroscope sensor to help estimate the spatial position and / or orientation of the medical device 120 during a medical procedure.

[0030] The MD controller 122 may be a computing device that is communicatively coupled to the medical device 120 to transmit and receive signals from the medical device 120. For example, the MD controller may transmit a signal to cause one or more lighting devices (see Figure 2A and Figure 2B ) to illuminate the region of interest in the body cavity of the patient P. Alternatively, the MD controller 122 may receive one or more imaging devices (see Figure 2A and Figure 2B ) receives image signals. The MD controller 122 may have one or more applications (e.g., software programs) installed locally for performing image processing, which may be performed to process image signals to generate images (e.g., real-time images) for display on one or more of the displays 110 communicatively coupled to the MD controller 122. For example, when the medical device 120 is inserted into a body cavity of a patient P and navigated therethrough toward a target site, multiple image signals may be received from one or more imaging devices and processed by the MD controller 122 to generate and display multiple corresponding images.

[0031] In some examples, MD controller 122 may have one or more additional applications (eg, software programs) installed locally to perform one or more operations associated with 3D image registration (eg, Figure 3 For example, the generated multiple images may be processed to generate a 3D surface map of the body cavity, where the 3D surface map is used as part of a registration process, as described in more detail below. In other examples, the images may be transmitted to another system and / or computing device within environment 100 for processing, analysis, storage, display, etc.

[0032] One or more components of MD controller 122, such as one of the applications, may generate or cause generation of one or more GUIs based on instructions / information stored in memory, instructions / information received from other components in environment 100, etc. One or more components of MD controller 122 may also cause a GUI to be displayed via one of displays 110. The GUI may include images, text, input text boxes, selection controls, etc.

[0033] The medical device system 102 may also include a tool 123 that can be inserted and / or delivered to a body cavity of a patient P via the medical device 120. One or more tools may extend distally from the medical device 120 for use during a medical procedure. Exemplary tools 123 used in conjunction with the medical device 120 for ERCP surgery may include tools for intubation (e.g., sphincterotomes), choledochoscopes, catheters, balloons, stent delivery systems, forceps, baskets, nets, biopsy needles and / or guidewires, and other similar tools for facilitating diagnosis and / or treatment. In some examples, the tool 123 may also include a position sensing component of the position sensing system 104 integrated therein, such as one or more tool EM sensors 133, to be able to track the position and / or orientation of the tool 123 during a medical procedure. For example, the tool EM sensor 133 may be located in or on at least the distal tip or distal portion of the tool 123 to help track at least one position and / or orientation of the distal tip or distal portion of the tool 123. Additionally or alternatively, the tool 123 may include a fiber optic shape sensor, an accelerometer, and / or a gyroscope sensor to assist in estimating the spatial position and / or orientation of the tool 123 (eg, a distal tip or distal portion of the tool 123) during a medical procedure.

[0034] The position sensing system 104 can be a spatial tracking system for determining the position and / or orientation of one or more components of the medical device system 102 and / or other components of the environment 100 in and / or on the body of the patient P. The position sensing system 104 can include any of the features described in U.S. Patent No. 10,782,114 issued on September 22, 2020, the entire contents of which are incorporated herein by reference. The position sensing system 104 can be an EM-based tracking system, which includes a position sensing system (PSS) controller 130, which is communicatively coupled to one or more transmitter devices to generate an electromagnetic field, and is communicatively coupled to one or more receiver devices for detecting the generated electromagnetic field. The position and / or orientation of the medical device 120 and / or tool 123 can be determined based on the strength of the field detected by the receiver device.

[0035] The one or more transmitter devices may include an external device 136 (e.g., an external field generator), for example, adjacent to the patient P. The one or more transmitter devices may each include an element for generating a magnetic field. For example, the one or more transmitter devices may each include one or more coils (e.g., solenoids) and one or more circuit elements that transmit current through the coils. Thus, the coils may generate a magnetic field.

[0036] The one or more receiver devices may include at least one or more EM sensors 132 located in or on the medical device 120 to help determine the location and / or orientation of the medical device 120, as shown in FIG. Figure 2A and Figure 2B As part of the registration process, the position and / or orientation of the medical device 120 may be used in conjunction with the above-described 3D surface map generated from the plurality of images captured by the medical device 120 .

[0037] In some examples, one or more receiver devices may also include one or more tool EM sensors 133 located in or on the tool 123 to help determine the position and / or orientation of the tool 123. The position and / or orientation of the tool 123 may be used to track the tool 123 as it extends distally from the medical device 120 and into the anatomical structure during a medical procedure, which may help advance the tool 123 through the anatomical structure to be visually guided, as described in detail below.

[0038] In some embodiments, the environment 100 may also include a patch 134 that may be locally applied to the patient P at least during the duration of the medical procedure. In some aspects, the one or more receiver devices of the position sensing system 104 may also include one or more patch EM sensors 135 located in or on the patch 134. The patch EM sensor 135 may help determine the position and / or orientation of the patient P to determine whether the patient P moves during the medical procedure, including any respiratory motion that may affect the registration. Thus, any movement of the patient P may be taken into account or compensated during the registration process. As discussed in further detail below, in some examples, the patch 134 may be placed prior to the medical procedure for preoperative 3D imaging, which may further facilitate registration. For example, the patch 134 may also include one or more radiopaque markers, MRI markers, etc., which may be captured in the preoperative 3D image.

[0039] One or more receiver devices (e.g., one or more patch EM sensors 135) may each include one or more magnetic field sensors. The magnetic field sensor may include, for example, a magnetoresistive (MR) element, such as a tunneling magnetoresistive (TMR) element, an anisotropic magnetoresistive sensing element, a giant magnetoresistive sensing element, a Hall effect sensing element, a giant magnetoresistive sensing element, an extraordinary magnetoresistive sensing element, or a semiconductor magnetoresistive element. Additionally or alternatively, the magnetic field sensor may include one or more inductive sensors (e.g., inductive coil sensors), planar coil sensors, spin Hall sensing elements (or other Hall sensing elements), or magnetic gradiometers. The magnetic field sensor of one or more receiver devices may have any properties of magnetic field sensors known in the art (including, for example, TMR sensors). For example, the magnetic field sensor may include a fixed layer, a tunnel layer, and a free layer. When the free layer is aligned with the fixed layer, the resistance may change.

[0040] In some examples, one or more receiver devices, such as at least a portion of EM sensor 132 and / or patch EM sensor 135, may include magnetic field sensors arranged in a dual-axis, six-degree-of-freedom arrangement to enable measurement of x, y, z, roll, pitch, and yaw. For example, these one or more receiver devices may include three magnetic field sensors arranged in a dual-axis, six-degree-of-freedom arrangement (see Figure 2B ) so that the location of the imaging plane can be determined in three dimensions based on measurements of x, y, z, roll, pitch and yaw. In this arrangement, two of the three magnetic field sensors may be oriented so that their primary sensing directions are aligned (approximately parallel) with the longitudinal axis of the respective device in which they are integrated. A full Wheatstone bridge configuration may be utilized by the two magnetic field sensors. The third magnetic field sensor may be arranged so that its primary sensing direction is transverse to (e.g., approximately orthogonal / perpendicular to) the longitudinal axis. A half Wheatstone bridge configuration may be utilized by the third magnetic field sensor. The Wheatstone bridge may have any of the characteristics of a Wheatstone bridge known in the art.

[0041] The position sensing system 104 may have other configurations within the scope of the present invention. For example, a three-axis configuration may be used for the magnetic field sensors of the receiver device, wherein each of the three magnetic field sensors is configured so that its primary sensing direction is aligned with a different axis (e.g., the primary sensing directions of the magnetic field sensors are aligned orthogonally to each other). For example, a first magnetic field sensor may have a primary sensing direction of the X axis, a second magnetic field sensor may have a primary sensing direction of the Y axis, and a third magnetic field sensor may have a primary sensing direction of the Z axis. In such a three-axis configuration, each of the magnetic field sensors may utilize a half-Wheatstone bridge configuration. In another example, the receiver device may measure six degrees of freedom using only two magnetic field sensors, wherein each of the two magnetic field sensors has a half-Wheatstone bridge configuration (or a full-Wheatstone bridge configuration). In another example, two magnetic field sensors may be used to measure five degrees of freedom. In such an example, the position sensing system 104 may not be able to measure rolling. In an additional example, a single magnetic field sensor may be implemented by the receiver device and five degrees of freedom may be measured using a half-Wheatstone bridge.

[0042] In the above examples, external device 136 is a transmitter device and EM sensor 132 and / or patch EM sensor 135 is a receiver device. In other examples, EM sensor 132 and / or patch EM sensor 135 may be a transmitter device and external device 136 may be a receiver device.

[0043] The PSS controller 130 may be communicatively coupled to one or more transmitter devices and one or more receiver devices of the position sensing system 104. For example, Figure 1B As shown, the PSS controller 130 may be communicatively coupled to the external device 136, the EM sensor 132, and / or the patch EM sensor 135. The PSS controller 130 may transmit a signal to the external device 136, for example, to initiate generation of a magnetic field, and subsequently pause, stop, and / or restart generation of the magnetic field. Additionally, the PSS controller 130 may receive a signal from the EM sensor 132 and / or the patch EM sensor 135 indicating the strength of the magnetic field detected by the EM sensor 132 and / or the patch EM sensor 135 (e.g., the voltage induced thereby). The medical device 120, specifically at least the distal tip of the medical device 120 (see Figure 2A and Figure 2B ) may be determined based on signals received from EM sensor 132. Additionally, the position and / or orientation of patient P may be determined based on signals received from patch EM sensor 135, for example, to help identify any patient movement, including respiratory motion, that may affect registration and therefore may need to be accounted for during registration.

[0044] The environment 100 may include an imaging system 106, for example, for capturing images of the anatomical structure of the patient P. The environment may also include a data storage system 108, for example, for storing images captured by the imaging system 106. The captured images may be three-dimensional (3D) images. Alternatively or additionally, the images may be two-dimensional (2D) images, which may be reconstructed into 3D images using techniques known or likely to become known in the art. At least one of the captured 3D images may be used to extract a 3D model that identifies multiple anatomical structures in the anatomical structure of the patient P, including one or more anatomical structures of interest based on medical procedures, such as the bile duct and pancreatic duct for ERCP procedures. As part of the registration process, the extracted 3D model may be registered to the patient. For example, the 3D model may be registered to the patient P using a 3D surface map generated from multiple images captured by the medical device 120 and the position and / or orientation of the medical device 120 determined by the position sensing system 104, as described in detail below.

[0045] The imaging system 106 may include one or more preoperative imaging systems. The modalities of exemplary preoperative imaging systems may include computed tomography (CT), magnetic resonance cholangiopancreatography (MRCP), ultrasound (US), or other similar three-dimensional (3D) imaging modalities. In some examples, preoperative imaging may be specifically ordered to prepare for a medical procedure. For example, a gastroenterologist may schedule a patient P for an ERCP procedure, and may also order preoperative imaging to prepare for the ERCP procedure. In such an example, the patient P may be positioned during imaging in the same or similar position as the patient during surgery. In other examples, preoperative images that may be used for patient P may have been ordered for diagnosis or other exemplary purposes. As a result, the patient P may not be positioned during imaging in the same or similar position as the patient during surgery, which may be taken into account during the image registration process, as detailed below.

[0046] In some examples, the imaging system 106 may also include an intraoperative imaging system. The intraoperative imaging system may be a supplement and / or alternative to one or more preoperative imaging systems. Exemplary modalities of the intraoperative imaging system may include non-3D imaging modalities that can be used to reconstruct 3D images, such as transabdominal US, endoscopic US, and / or fluoroscopy. Additionally or alternatively, exemplary modalities of the intraoperative imaging system may include intraoperative 3D imaging modalities, such as fluoroscopic cone beam CT, C-arm-based tomography, and / or digital tomosynthesis. In an example where the imaging system 106 includes an intraoperative 3D imaging modality, during a medical procedure, one or more additional 3D images are optionally captured by one of the intraoperative 3D imaging modalities to generate a 3D surface map. For example, additional 3D images may be used in conjunction with multiple images captured by the medical device 120 to generate a 3D surface map, as described in detail below.

[0047] In some examples, each of the imaging systems 106 may include and / or may be associated with a computing device (e.g., distinct from a separate computing device 112). The computing device may include one or more applications (e.g., software programs) installed locally, for example, on a memory of the computing device, for performing image processing executable to generate an image. Additionally, one or more components of the computing device, such as one of the applications, may generate or cause one or more GUIs to be generated based on instructions / information stored in the memory, instructions / information received from other components in the environment 100, and the like. One or more components of the computing device may cause a GUI to be displayed via one of the displays 110. The GUI may include images, text, input text boxes, selection controls, and the like, and may enable an operator to interact with images captured by a corresponding modality from the various aforementioned modalities. For example, the computing device may generate and display a 3D image based on an image signal received from the corresponding modality, and the operator may utilize the application to manipulate the 3D image (e.g., rotate, zoom in, zoom out, annotate an object or anatomical structure, and the like).

[0048] In some examples, the computing device may have one or more additional applications installed locally to perform one or more operations associated with 3D image registration (e.g., Figure 3 106 and / or the like). For example, the generated 3D images may be processed to extract a 3D model, wherein the 3D model is registered to the patient P, as described in detail below. In other examples, the 3D images may be transmitted to another system and / or computing device within the environment 100 for processing, analysis, storage, display, etc. Additionally, the imaging system 106 and one or more other components of the environment 100, such as the data storage system 108, may be components or subsystems of a larger system, such as a picture archiving and communication (PAC) system.

[0049] The data storage system 108 may include a server system or a computer-readable memory such as a hard drive, a flash drive, a disk, etc. The data storage system 108 includes and / or interacts with an interface to exchange data to other systems, for example, one or more of the other components of the environment 100. For example, the data storage system 108 may be configured to receive and store 3D images of the patient P generated by one or more of the imaging systems 106. As another example, the data storage system 108 may be configured to receive and store multiple images of a body cavity of the patient P from the medical device system 102. As another example, the data storage system 108 may be configured to receive spatial information of the medical device 120 and / or the patient P from the position sensing system 104.

[0050] The display 110 may be communicatively coupled to one or more other components of the environment 100, for example, to receive and display data, including image data. As an example, the display 110 may receive and display a 3D image captured by one of the imaging systems 106 and processed to extract a 3D model. Additionally, the display 110 may receive and display the extracted 3D model. As another example, the display 110 may receive a plurality of images captured by the medical device 120, for example, when the medical device 120 is inserted through a body cavity of the patient P and navigated toward a target site. Additionally, the display 110 may receive and display a 3D surface map generated by processing a plurality of images.

[0051] In addition, once the 3D image registration is complete, the display 110 can receive and display a GUI that overlays a representation of the location or trajectory of one or more anatomical structures of interest for the medical procedure on a current image of the target site (e.g., the current image of the target site is one of multiple images captured by the medical device 120), as described in detail below. For example, for an ERCP procedure, the GUI can overlay a representation of the location or trajectory of the common bile duct and pancreatic duct on a current image of the papilla to provide visual guidance for cannulation through the papilla and into one of the ducts. In some examples, the display 110 can be an interactive display and / or a display of a computing device that is configured to receive input from an operator to enable the operator to interact with the image data.

[0052] The computing device 112 may be a stand-alone computing device configured to communicate with one or more of the other components of the environment 100 across the network 114. For example, the computing device 112 may communicate with one or more of the medical device system 102, the position sensing system 104, the imaging system 106, and / or the data storage system 108 across the network 114 to exchange information, including receiving image data and spatial data (e.g., position and / or orientation information). The computing device 112 may be a computer system such as, for example, a desktop computer, a laptop computer, a tablet computer, a smartphone, a smart watch, or other electronic wearable device, etc.

[0053] In some examples, computing device 112 may include one or more applications, such as programs, plug-ins, etc., installed locally on a memory of computing device 112 to perform one or more operations associated with 3D image registration based on the received information. For example, the 3D model may be registered to the patient using a 3D model extracted from a 3D image captured by one of imaging systems 106, a 3D surface map generated from a plurality of images captured by medical device 120, and spatial information of at least medical device 120 determined by position sensing system 104. In some examples, computing device 112 may receive the 3D model and the 3D surface map from other corresponding components of environment 100. In other examples, computing device 112 may generate the 3D model and the 3D surface map.

[0054] Additionally, one or more components of computing device 112, such as one of the applications, may generate or cause to be generated one or more GUIs based on instructions / information stored in memory, instructions / information received from other components in environment 100, etc. Furthermore, one or more components of computing device 112 may cause a GUI to be displayed via a display of computing device 112 or via one or more other displays (e.g., display 110). The GUI may include text, input text boxes, selection controls, etc. The display may include a touch screen or a display with other input systems (e.g., a mouse, keyboard, etc.) for an operator of computing device 112 to control functions of computing device 112.

[0055] For example and FIG. 6A to FIG. 6C As shown, once the 3D model is registered to the patient, a GUI may be generated for display. The GUI may include a representation of the location and / or trajectory of one or more anatomical structures of interest for a medical procedure generated using the 3D image transformed based on the registration. The representation may be overlaid on a current image (e.g., one of multiple images) of the target site captured by the medical device 120. Additionally, throughout the registration process, one or more prompts may be generated and displayed to the operator, for example, requesting the operator to confirm and / or modify an automatic determination or identification made by the application.

[0056] The computing device 112 is described as including one or more operations or steps configured to perform 3D image registration (e.g., Figure 3In some examples, one application operating on a single computing device of environment 100 may be configured to perform each of the steps (e.g., 3D model extraction, 3D surface map generation, registration, and GUI generation). In other examples, multiple applications running on the same computing device of environment 100 or across different computing devices of environment 100 may perform different operations. As a non-limiting example, one application may be configured to perform operations related to 3D model extraction, and another application may be configured to perform operations related to 3D surface map generation. In addition, another application may be configured to perform operations related to registration and / or GUI generation.

[0057] One or more applications executed on one or more components of environment 100 are described herein as local applications that are installed, for example, on the memory of the corresponding component so that no network connection (e.g., Internet access) is required to communicate with a remote server and enable the application to run. However, in other embodiments, the application may be a web-based application accessible via a browser executed on the component, where the one or more applications may communicate with a remote server (not shown) via network 114. In such an example, one or more operations of 3D image registration may be performed by a processing device of the remote server.

[0058] As mentioned, one or more components of environment 100 may communicate via network 114. Network 114 may be an electronic network. Network 114 may include one or more wired and / or wireless networks, such as a wide area network (“WAN”), a local area network (“LAN”), a personal area network (“PAN”), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc.), etc. In one non-limiting illustrative example, components of environment 100 may communicate with and / or connect to network 114 via a universal serial bus (USB) or other similar local, low-latency connection or direct wireless protocol.

[0059] In some embodiments, network 114 includes the Internet, and the information and data provided between various systems occurs online. "Online" can mean connecting or accessing source data or information from a location away from other devices or networks connected to the Internet. Alternatively, "online" can refer to connecting or accessing an electronic network (wired or wireless) via a mobile communication network or device. The Internet is a global computer network system-a network of networks in which a party to a computer or other device connected to the network can obtain information from any other computer and communicate with parties to other computers or devices. The components of environment 100 can be connected via network 114 using one or more standard communication protocols so that the components can transmit and receive communications to each other across network 114, as discussed in more detail below.

[0060] Despite Figure 1A and Figure 1B 100 as separate components, it should be understood that in some embodiments, components or portions of components in environment 100 may be integrated with or incorporated into one or more other components. In some embodiments, operations or aspects of one or more of the components discussed above may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of environment 100 may be used.

[0061] Although the specific examples included in the present invention implement 3D image registration in environment 100 to facilitate ERCP surgery, it should be understood that the technology according to the present invention can be suitable for registering 3D models to other types of images other than endoscopic images captured by medical device 120. For example, the 3D model can be registered to a fluoroscopic image of the patient's anatomy captured during ERCP to facilitate navigation of the bile duct or pancreatic duct. Fluoroscopic images typically only show the bile duct and pancreatic duct in a very short period of time when contrast agent is injected. Using similar techniques described herein, the 3D model can be registered to the fluoroscopic image, and a GUI can be generated based on the registration, which includes a representation of the bile duct and pancreatic duct overlaid on the fluoroscopic image. Additionally, the technology according to the present invention can be suitable for other types of endoscopic medical procedures, such as percutaneous nephrolithotomy (PCNL) surgery or any other surgery involving intracavitary access to elastic and deformable structures during surgery. For example, in order to assist in the placement of a stent or other similar device to treat a cystic lesion of the pancreas, a 3D image including a portion of the stomach wall and the cystic lesion can be registered. A representation of the lesion can be overlaid on an endoscopic image of the stomach wall captured during surgery at the approximate location where the lesion would appear if the lesion were visible, to guide the penetration location to enter the lesion for stent placement. It should also be understood that the above examples are illustrative only. The techniques and technologies of the present invention can be adapted for any suitable activity.

[0062] Figure 2A Depicted Figure 1A and Figure 1B 1. The medical device 120 may include a handle 202 and an insertion portion 204. The medical device 120 may also include an umbilical tube 206 for connecting the medical device 120 to a source such as air, water, suction, electricity, etc., and for image processing and / or viewing purposes of a device such as an MD controller 122.

[0063] The insertion portion 204 can include a sheath or shaft 208 , and the insertion portion 204 can also include a distal tip 210 . Figure 2B An exemplary distal tip assembly 211 is depicted, which may be positioned at the distal tip 210. Figure 2A and Figure 2B , the distal tip assembly 211 may include a substrate 240 (e.g., a rigid or flexible circuit board or other type of board) that may be at least partially disposed within or on an inner core (not shown) of the distal tip 210. In an example, the substrate 240 is rigid and includes multiple layers.

[0064] like Figure 2A and Figure 2B As shown, the distal tip 210 and the distal tip assembly 211 may include one or more imaging devices 212 (e.g., one or more cameras) for capturing images, and the distal tip assembly 211 may also include one or more lighting devices 214 (e.g., one or more light emitting diodes (LEDs) or optical fibers) for providing illumination to facilitate image capture and visualization. The distal tip 210 and the distal tip assembly 211 may face sideways. That is, the orientation of the imaging device 212 and the lighting device 214 may be radially outward, vertical, approximately vertical, or otherwise transverse to the longitudinal axis of the shaft 208 and the distal tip 210. However, the present invention also covers other configurations of the distal tip and the distal tip assembly. For example, the distal tip and the distal tip assembly may "face forward" (i.e., face distally).

[0065] The imaging device 212 and the illumination device 214 may be mounted to the substrate 240 by any suitable method, including, but not limited to, wire bonding, surface mount assembly, electromechanical assembly, and / or plated through hole techniques. Figure 2A and Figure 2B206 , one imaging device 212 and two lighting devices 214 are depicted, but any suitable number of imaging devices 212 and / or lighting devices 214 may be utilized. Alternatively, the imaging device 212 and the lighting devices 214 may be combined into a single device. The conduit 242 may house one or more wires or cables that are attached to the base 240 or elements mounted on the base 240 to transmit power and / or signals to and from the base 240 and / or elements mounted on the base 240. The wires or cables may extend through the shaft 208 and into the handle 202, where the wires or cables may be connected to the MD controller 122, such as via the umbilicus 206. For example, the MD controller 122 may transmit a signal to illuminate the lighting device 214, and may receive an image signal from the imaging device 212 for processing and subsequent display. In some examples, the MD controller 122 may also initiate image capture by transmitting a signal via the wires or cables housed in the conduit to cause the imaging device 212 to capture an image.

[0066] Elements of the position sensing system 104 may also be disposed on the substrate 240 and may be mounted according to any of the techniques described above for the imaging device 212 and the lighting device 214. For example, at least one or more EM sensors 132, such as the first EM sensor 132a, the second EM sensor 132b, and the third EM sensor 132c, may be disposed on the substrate 240. In some examples, the first EM sensor 132a, the second EM sensor 132b, and the third EM sensor 132c may be oriented on the substrate as described above with reference to FIG. Figure 1A and Figure 1B Any alternative number of EM sensors 132 may be utilized, and the depicted three EM sensors 132a, 132b, 132c are merely exemplary. Figure 1B As described in detail, EM sensors 132a, 132b, and 132c may be receiver devices capable of measuring magnetic fields, such as those generated by external device 136, which may facilitate tracking the position and / or orientation of distal tip 210. In other examples, EM sensors 132a, 132b, 132c may be transmitter devices capable of generating magnetic fields.

[0067] Based on the inclusion of imaging device 212 in distal tip assembly 211, tracking the position and / or orientation of distal tip 210 using EM sensors 132a, 132b, 132c can make the position and / or orientation of medical device 120 known relative to a 3D surface map generated from images captured by imaging device 212. This known position and / or orientation can be used with the 3D surface map as part of the registration process, as described below with reference to Figure 3 discussed in detail. Figure 2BEM sensors 132a, 132b, 132c are shown included within distal tip assembly 211 at distal tip 210 of medical device 120, but in other examples, one or more other EM sensors may be positioned at other locations of medical device 120, including within shaft 208 and / or handle 202.

[0068] Other optional components of the position sensing system 104 may be mounted on the substrate 240, including a capacitor 244 and one or more diodes 246. The capacitor 244 may help reduce noise in the supply voltage for the position sensing system 104. For example, the capacitor 244 may be used as a decoupling capacitor, which acts as a low pass filter for any electromagnetic interference ("EMI") on the supply voltage. The one or more diodes 246 may help provide high voltage protection, such as electrostatic discharge ("ESD") protection. The one or more diodes 246 may help prevent electrostatic discharge from causing damage to the EM sensors 132a, 132b, 132c. The one or more diodes 246 may additionally or alternatively help provide protection for various aspects of the imaging device 212.

[0069] In some examples, imaging device 212 (e.g., a camera and lens of imaging device 212) may be calibrated to understand the transformation between an optical coordinate system in which images are captured by imaging device 212 and a spatial coordinate system in which the position and / or orientation of medical device 120 is determined by position sensing system 104. Calibration may be performed using algorithms that are or become known in the art, such as a "hand-eye" calibration method. Calibration may be performed during the manufacturing process of medical device 120.

[0070] The distal tip assembly 211 may also include an elevator 216 for changing the orientation of a tool (e.g., one of the tools 123) inserted into the working channel of the medical device 120. The elevator 216 may alternatively be referred to as a swing frame, a pivot frame, a lifting base, or any other suitable term. The elevator 216 may be pivoted via, for example, an actuation line or another control element extending from the handle 202 through the shaft 208 to the elevator 216. The elevator 216 may pivot around an axle 217. The axle 217 may be rotatably retained within the distal tip assembly 211.

[0071] In addition to or in lieu of the above-described components, the distal tip assembly 211 may also include components. For example, the distal tip assembly 211 may also include additional or alternative illumination sources and / or additional or alternative imaging components (e.g., additional cameras). The distal tip assembly 211 may also include additional types of sensors, such as humidity sensors, temperature sensors, pressure sensors, or other types of sensors that may be useful during a medical procedure.

[0072] The distal portion of the shaft 208 connected to the distal tip 210 may have a steerable section 218. The steerable section 218 may be, for example, an articulating joint. The shaft 208 and the steerable section 218 may include various structures that are or become known in the art.

[0073] The handle 202 may have one or more actuators / control mechanisms 220. The control mechanism 220 may provide control of the steerable section 218 or may allow for the provision of air, water, suction, etc. For example, the handle 202 may include control knobs 222, 224 for left, right, up, and / or down control of the steerable section 218. For example, one of the knobs 222, 224 may provide left / right control of the steerable section 218 and the other of the knobs 222, 224 may provide up / down control of the steerable section 218. The handle 202 may also include one or more locking mechanisms 226 (e.g., knobs or levers) to prevent the steerable section 218 from being steered in at least one of the up, down, left, or right directions. The handle 202 may include an elevator control lever 228. The elevator control lever 228 can raise and / or lower the elevator 216 via a connection between the lever 228 and an actuation wire (not shown) extending from the lever 228 through the shaft 208 to the elevator 216. The port 230 can allow a tool (e.g., one of the tools 123) to enter the working channel (not shown) of the medical device 120 through the port 230, through the shaft 208 to the distal tip 210. Although not shown, the handle 202 can also include one or more valves, buttons, actuators, etc. to control the supply of air, water, suction, etc.

[0074] In use, the operator can insert at least a portion of the shaft 208 into a subject, such as a body cavity of a patient P. The distal tip 210 can be navigated to a target site in the body cavity. For an ERCP procedure, the distal tip 210 can be inserted into the patient's mouth and navigated along the esophagus through the stomach into the duodenum and to the nipple, where the nipple is the target site for intubation. The operator can insert an intubation tool (e.g., one of the tools 123, such as a sphincterotomy) into the port 230 and allow the intubation tool to reach the distal tip 210 via the working channel through the shaft 208. The intubation tool can leave the working channel at the distal tip 210. The operator can use the elevator control rod 228 to raise the elevator 216 and angle the intubation tool toward the desired position of the nipple. The operator can use the intubation tool to perform intubation. Using the systems, devices, and methods described herein, the positioning of the intubation tool relative to the desired position of the nipple for intubation can be facilitated by displaying a GUI. For example, the GUI may include representations of the location and / or trajectory of the bile duct and pancreatic duct overlaid on a current image captured by the imaging device 212 of the medical device 120 .

[0075] Figure 3An exemplary process 300 for 3D image registration is depicted. In some examples, process 300 may be performed by a combination of one or more of the components of environment 100, such as a computing device of or associated with one of imaging systems 106, MD controller 122, and / or separate computing device 112, via one or more applications executed thereon.

[0076] At step 302, process 300 may include receiving a 3D image of the anatomy of a patient, such as patient P. The 3D image may be of a specific anatomy (e.g., including a specific anatomical structure), depending on the type of medical procedure. For example, if the medical procedure to be performed is an ERCP procedure, the 3D image may be a 3D image of the upper gastrointestinal (GI) tract, including the esophagus, stomach, duodenum, and pancreatic and bile duct anatomy. In some examples, an operator may select a 3D image from a plurality of 3D images to be used for 3D image registration.

[0077] Preoperative imaging may be performed on the patient P weeks, days, or the same day prior to surgery. In such an example, the 3D image may be a preoperative 3D image captured by one of the preoperative imaging systems of the imaging system 106, such as a CT image, an MRCP image, a US image, etc. The preoperative 3D image may be received from one of the imaging systems 106 that captured the 3D image or from the data storage system 108.

[0078] In other examples, intraoperative imaging may be performed on the patient P during surgery. In such examples, the 3D image may be an intraoperative 3D image captured by one of the intraoperative imaging systems of the imaging system 106. The intraoperative imaging system may include non-3D imaging modalities that may be used to reconstruct the 3D image, including transabdominal US, endoscopic US, or fluoroscopy. Additionally or alternatively, the intraoperative imaging system may include a 3D imaging modality, such as fluoroscopic cone beam CT, C-arm-based tomography, and / or digital tomosynthesis.

[0079] In some examples, when preoperative and / or intraoperative imaging is performed, a contrast agent or other substance (e.g., secretin) may be administered to patient P to help enhance the appearance of specific anatomical structures in the 3D image, such as the bile and pancreatic duct. Additionally, if preoperative and / or intraoperative imaging is ordered specifically for the procedure, patient P may be positioned in the same or similar imaging posture for the ERCP procedure that patient P will or is currently in. The same or similar posture may facilitate registration. However, preoperative images may often be ordered by different physicians and / or for different purposes (e.g., for diagnostic purposes), and therefore patient P may have a posture for preoperative imaging that is different from the posture in which patient P undergoes a medical procedure. Therefore, the registration techniques described herein may adjust for or take into account the different postures present in the 3D images.

[0080] Optionally, patch 134 or at least a portion of components of patch 134, including radiopaque markers, MRI markers, or other similar markers, may be applied to patient P prior to preoperative and / or intraoperative imaging. As a result, the 3D image received at step 302 may include a marker of patch 134 in the 3D image, which may provide an additional alignment feature for registration.

[0081] At step 304, process 300 may include processing the 3D image to extract a 3D model that identifies multiple anatomical structures in the anatomical structure, including one or more anatomical structures of interest to the medical procedure. For example, when the medical procedure is an ERCP procedure, the 3D model may identify and isolate (e.g., segment) at least the anatomical structures of the upper GI tract and the pancreatic and biliary trees, including the esophagus, stomach, pyloric sphincter, duodenum, bile ducts (e.g., common bile duct, cystic duct, and hepatic duct), liver, pancreatic duct, and pancreas. The pancreatic and biliary trees, in particular the bile ducts and pancreatic ducts, may be anatomical structures of interest. In addition to the anatomical structures of the upper GI tract and the pancreatic and biliary trees, one or more other anatomical structures may also be identified, such as ribs, spine, and / or other structures that may be associated with intraoperative images, including endoscopic images and / or fluoroscopic images captured by the medical device 120, for use during registration, as described in detail below.

[0082] In some examples, the 3D image may be processed via manual segmentation, whereby an operator may manually identify and mark each of the anatomical structures. For example, the 3D image may be displayed, for example, via one of displays 110, and the operator may provide input via one of displays 110 and / or an associated computing device (e.g., a computing device of one of imaging systems 116 or computing device 112) to manually mark the anatomical structures. In other examples, the 3D image may be processed via computational methods commonly used in computer vision, machine learning, and / or other image processing techniques for isolating structures. The computational methods may be fully automatic and / or may be used in conjunction with operator input, for example, to confirm or modify / correct anatomical features.

[0083] As an example computing method, a computer vision model or machine learning model (hereinafter referred to as a "model") may be trained and implemented to predict anatomical structures present in a 3D image. The model may be trained by one of the components of the environment 100 that may implement the model, such as a computing device of or associated with the imaging system 106, a computing device 112, and / or an MD controller 122. In other examples, the model may be trained by a third-party system, and the model may be provided to the component of the environment 100 that may implement the model for execution. In order to train the model, training data may be received and processed to generate (e.g., construct) a trained model for predicting anatomical structures present in a 3D image. The training data may include 3D training images of the patient's anatomical structure. The 3D training images may include multiple image modalities (e.g., CT images, MRCP images, US images, etc.). The training data may be generated, received, or otherwise obtained from internal and / or external resources. In some examples, the training data may also include a synthetic 3D training image of the patient's anatomical structure.

[0084] Typically, a model includes a set of variables, such as nodes, neurons, filters, etc., which are adjusted, such as weighted or biased to different values, via the application of training data. In some examples, supervised, unsupervised, semi-supervised, and / or reinforcement learning processes may be implemented to train the model. In some embodiments, a portion of the training data may be retained during training and / or used to validate the trained model.

[0085] When a supervised learning process is employed, labels or annotations corresponding to 3D training images (e.g., labels or annotations corresponding to training data) can facilitate the learning process by providing ground truth. For example, the labels or annotations may indicate the anatomical structures present in the 3D image. Training may proceed by feeding 3D training images (e.g., samples) from the training data into a model having variables set to initialization values, e.g., random, based on Gaussian noise, a pre-trained model, etc. The model may output a predicted anatomical structure present for the sample. The output may be compared with the corresponding label or annotation (e.g., ground truth) to determine an error, which may then be back-propagated through the model to adjust the value of the variable. The process may be repeated for multiple samples, at least until the determined loss or error is below a predefined threshold. In some examples, some of the training data may be retained and used to further validate or test the trained model.

[0086] For unsupervised learning processes, the training data may not include pre-assigned labels or annotations to assist the learning process. Conversely, the unsupervised learning process may include clustering, classification, etc. to identify naturally occurring patterns in the training data. K-means clustering or K-nearest neighbors may also be used, which may be supervised or unsupervised. A combination of K-nearest neighbors and unsupervised clustering techniques may also be used. For semi-supervised learning, a combination of training data with pre-assigned labels or annotations and training data without pre-assigned labels or annotations may be used to train the model.

[0087] When reinforcement learning is employed, an agent (e.g., an algorithm) may be trained to make decisions about anatomical structures included in a sample based on training data and errors obtained through trial and error. For example, upon making a decision, the agent may then receive feedback (e.g., a positive reward if the predicted anatomical structure is actually the anatomical structure present in the sample), adjust its next decision to maximize the reward, and repeat the process until the loss function is optimized.

[0088] Once trained, the trained model may be stored and subsequently applied by one of the components of environment 100. For example, the trained model may receive the 3D image received in step 302 as input data. The trained model may output predicted anatomical structures present (e.g., included) in the 3D image. The 3D model may be extracted based on the output of identifying (e.g., isolating) the anatomical structure. The 3D model with the isolated anatomical structure may then be displayed (e.g., via one of displays 110). In some examples, an operator input via touch or other input to one of displays 110 and / or an associated computing device (e.g., a computing device of one of imaging systems 106, MD controller 122, and / or computing device 112) may be received as feedback on the prediction. For example, the operator may confirm and / or correct the identified anatomical structure. For example, the model may be retrained using feedback by adjusting the values ​​of one or more variables of the model.

[0089] As referenced below Figure 4 As described in more detail, in some examples, the 3D model extracted at step 304 may be incomplete. For example, one or more of the multiple anatomical structures may be incomplete. In such examples, an estimated model of the incomplete anatomical structure may be calculated.

[0090] By identifying (e.g., isolating and / or segmenting) anatomical structures, the extracted 3D model can allow for visualization of certain anatomical structures that might not otherwise be visible during medical procedures, such as the bile duct and pancreatic duct during ERCP. Additionally, the identified anatomical structures can be used as landmarks to facilitate accurate image integration and registration, as described in detail below.

[0091] At step 306, process 300 may include receiving a plurality of images of a body cavity of patient P captured by imaging device 212 of medical device 120 as medical device 120 is inserted through the body cavity and navigated toward a target site during a medical procedure. At least one of the images may include a current image of the target site. For example, for an ERCP procedure, medical device 120 may be inserted into the patient's mouth and navigated along the esophagus through the stomach into the duodenum and to the nipple, where the nipple may be a target site for intubation. Thus, at least one of the received images may include a current image of the nipple. As medical device 120 navigates toward the target site, images may be received in real time and may be displayed via one of displays 110 so that an operator can visualize the body cavity and the target site as medical device 120 navigates toward the target site.

[0092] At step 308, process 300 may include receiving spatial information of medical device 120 from position sensing system 104. In some examples, as medical device 120 navigates toward a target site, spatial information of medical device 120 may be received in real time so as to correspond to images captured by imaging device 212 as medical device 120 navigates toward the target site. The spatial information of medical device 120 may include a position and / or orientation of medical device 120, and more specifically, a position and / or orientation of at least distal tip 210 and / or distal assembly 211 of medical device 120, such as Figure 2A and Figure 2B For example, the position and / or orientation of the distal tip 210 of the medical device 120 may be determined based on signals received from the EM sensors 132a, 132b, 132c indicating the strength of the magnetic field (e.g., the voltage induced by the magnetic field) generated by the external device 136 detected by the patch EM sensor 135. The position and / or orientation of any other components of the medical device 120 may also be determined if additional EM sensors 132 are provided therein (e.g., if the EM sensors 132 are positioned in the shaft 208 and / or the handle 202).

[0093] Optionally, spatial information of the patient P may also be received from the position sensing system 104. For example, if the patch 134 is locally applied to the patient P, the position and / or orientation of the patient P may be determined by the PSS controller 130, for example, based on a signal received from the patch EM sensor 135 indicating the strength of a magnetic field generated by the external device 136 (e.g., a voltage induced thereby) detected by the patch EM sensor 135.

[0094] At step 310, process 300 may include processing multiple images of the body cavity and spatial information of the medical device 120 to generate a 3D surface map of at least a portion of the body cavity. The 3D surface map may include a portion of the multiple anatomical structures identified (e.g., isolated or segmented) in the extracted 3D model that does not contain the anatomical structure of interest. Continuing with the example of an ERCP procedure, the 3D surface map may map the upper GI tract and the anatomical structures surrounding the bile and pancreatic duct (e.g., the duodenal wall and papilla). However, based on the duodenal wall and papilla effectively blocking or preventing visualization of the duct, the bile and pancreatic duct itself is not mapped in the 3D surface map because the bile and pancreatic duct is not visible to the imaging device 212 and is therefore not captured in the image.

[0095] As the medical device 120 navigates toward the target site, images captured by the imaging device 212 of the medical device 120 and spatial tracking information corresponding to the medical device 120 (including at least the position and / or orientation of the distal tip 210 of the medical device 120) may be used to generate a 3D surface map. The spatial tracking information may include at least the position and / or orientation of the distal tip 210 of the medical device 120, as determined by the position sensing system 104 using the EM sensor 132. In some examples, information from other positioning systems, such as light detection and ranging (LIDAR), ultrasonic distance measurement (via pulse echo or transmit-receive methods), stereo cameras, fluoroscopic images, intraoperative 3D radiographic images, structured light images, etc. may be used to generate a 3D surface map.

[0096] In any embodiment where the environment 100 does not include the position sensing system 104 and / or other similar independent spatial tracking systems, it may be desirable to identify the position and / or orientation of the imaging device 212 relative to the anatomical structure at the distal tip 210 of the medical device 120 while mapping the anatomical structure as a 3D surface. In some examples, the position and / or orientation of the imaging device 212 may be identified by applying a simultaneous localization and mapping (SLAM) algorithm. The SLAM algorithm may combine inputs from other sensors and / or imaging modalities. If a SLAM algorithm is utilized, then as the medical device 120 navigates through a body cavity, at least the anatomical structures used as landmarks for registration may be continuously mapped so that the relative position of each anatomical structure is known (e.g., a continuous path between each of the anatomical structures may be mapped).

[0097] In addition to generating a 3D surface map (e.g., as part of or in conjunction with it), anatomical structures may also be identified. For example, the identified anatomical structures may correspond to anatomical structures identified in the 3D model extracted in step 304 for use in registration. In some examples, the uniqueness of the appearance and / or geometry of a given anatomical structure may determine which of the identified anatomical structures may be used as part of the 3D surface map, to be used as marker points, for example, to facilitate image registration as described below.

[0098] Continuing the example of the medical procedure being an ERCP procedure, when the medical device 120 passes through the upper GI tract (e.g., along the esophagus through the stomach and into the duodenum), the esophagus, the fundus, the body of the stomach, the pyloric antrum, the pyloric sphincter, the duodenal bulb, and / or the duodenum can be identified as different anatomical structures. In some examples, different anatomical regions and anatomical structures within each region can be identified based on the unique appearance and / or morphological form of the region and / or structure. For example, the esophagus can be identified based on the long, straight, tubular form of the esophagus. The portion of the stomach can be identified by the appearance and / or shape of the gastric folds, or the more open spherical shape of the fundus. The pyloric sphincter can be identified by the small opening shape at the end of the larger spongy volume. The duodenum can be identified by the appearance of the annular folds or intestinal villi surrounding the tubular wall. Additionally, the identification of the anatomical structure can be inferred by the spatial proximity to other anatomical structures and / or by the temporal order in which the anatomical features appear when the medical device 120 navigates through the body cavity to the target site during surgery.

[0099] In other examples, different regions of the upper GI tract may be mapped as a single structure without separate classification. However, if the 3D surface map is incomplete, it may be useful to identify at least a portion of the mapped anatomical structures (e.g., structures used as landmarks) so that those portions of the anatomical structures can be matched with corresponding anatomical structures (e.g., isolated or segmented) in the 3D model identified during registration. Additionally, at least the anatomical structures at the target site (e.g., nipple) may be separately identified with compensation for deformation during registration.

[0100] The anatomical structure may be identified using a combination of one or more of the recognition techniques. For example, using a first recognition technique, the anatomical structure may be identified by providing an image (e.g., 2D image data) as input to a trained computer vision or machine learning model. For example, a computer vision or machine learning model may be trained and implemented using techniques similar to those used to predict anatomical structures in 3D images in step 304 above to predict anatomical structures in images captured by the medical device 120 (e.g., in 2D images). However, in this regard, the training data includes real and / or synthetic 2D endoscopic images, rather than real and / or synthetic 3D images, such as CT images, MRCP images, US images, etc. The different anatomical forms or appearances of the anatomical structures, the spatial proximity to other anatomical structures, and / or the temporal order in which the anatomical structures appear during the medical procedure may be learned as part of the training. Therefore, each image received may be provided as input data to the trained model, and the trained model may output one or more predicted anatomical structures in the image.

[0101] As previously discussed, as medical device 120 navigates through a body cavity toward a target site, images of the body cavity and target site may be displayed in real time by one of displays 110. In some examples, indications of predicted anatomical structures output by the trained model may be displayed in association with the locations of the anatomical structures in the displayed images.

[0102] Additionally or alternatively, a request for manual input from an operator to identify or confirm may be generated and displayed when the medical device 120 has reached a known anatomical structure and visualized in the displayed image. For example, a trained model may be executed to predict an anatomical structure, such as the cardia of the stomach, and then a prompt may be generated and displayed that guides the operator to confirm that the anatomical structure identified in the image is actually the cardia. The prompt may be displayed at an indication of the predicted anatomical structure adjacent to the cardia. In other examples, rather than using a computer vision or machine learning model to automatically identify or predict the anatomical structure, the operator may be prompted to identify at least a subset of the anatomical structures displayed in the image. The subset of anatomical structures may include marker points to facilitate accurate image registration, as discussed in detail below.

[0103] Additionally or alternatively, using a second identification technique, the anatomical structure may be identified by first mapping the geometry of the anatomical structure into a 3D surface (e.g., forming part of a 3D surface map), and then identifying the anatomical structure using the 3D surface map. For example, any anatomical structure identified by geometry may be mapped into a 3D surface using images captured by the medical device 120 and spatial information from the EM sensor 132, other optional sensors incorporated into the medical device 120, such as an accelerometer, and / or information from any positioning system, such as light detection and ranging (LIDAR), ultrasonic distance measurement (via pulse echo or transmit-receive methods), stereo cameras, fluoroscopic images, intraoperative 3D radiographic images, structured light images, etc.

[0104] In some cases, particularly when the imaging device 212 of the medical device 120 has a limited field of view, the spatial information (e.g., determined by the position sensing system 104) and geometry of the distal tip 210 of the medical device 120 may also be used to generate a contact map that estimates the geometry of the anatomical structure when the movement position of the distal tip 210 is limited. In anatomical structures with easily deformable boundaries, the contact map may rely on measuring the deceleration of the distal tip 210 to infer the walls or boundaries of the tissue of the anatomical structure.

[0105] In some cases, based on the medical procedure, it may be necessary to identify at least one anatomical structure and include it in the 3D surface map, such as a nipple for an ERCP procedure. For patients with pancreatic divisum, the major and minor nipples may be identified separately during the ERCP procedure. Once the medical device 120 reaches the nipple, a combination of one or more of the above-mentioned identification techniques may be used to automatically identify the nipple. Additionally, in some examples, the registration performed in step 312, described in detail below, may begin when the 3D surface map is generated and before the nipple is identified. In such an example, if the registration is partially completed, the operator may be guided to the proximal position of the nipple. For example, a visual indicator, such as an arrow or other similar directional graphic component, may be provided to be displayed on an image captured by the imaging device 212 and displayed on one of the displays 110 to prompt the operator, for example, to control the steerable section 218 using one of the knobs 222, 224. Additionally, once the nipple is identified, a visual indication of the position of the identified nipple may be provided to be displayed on the current image captured by the imaging device 212 and displayed on one of the displays 110. Alternatively, if the identified nipple is no longer in view, one or more visual indicators similar to those discussed above may be provided for display to instruct the operator to return to the location of the identified nipple. In some examples, the operator may be requested to confirm the automatically identified nipple via a prompt or notification.

[0106] In another aspect, the operator may be prompted to manually identify the nipple by clicking, tapping, or otherwise selecting the nipple in the displayed image. In other examples, the application may wait until a cannulation tool (e.g., one of the tools 123 that is inserted into the medical device 120 via the working channel and exits the working channel at the distal tip 210) is inserted through the nipple to identify the nipple. For example, the application may identify the nipple based on where the cannulation tool intersects tissue in the current image of the target site and / or where the tip of the cannulation tool (e.g., the tool is not fluoroscopic) is located on a fluoroscopic image captured during a medical procedure.

[0107] In other examples, as the 3D surface map is generated and anatomical structures are identified, a mapping progress indication may be generated and displayed. For example, the mapping progress indication may include a percentage or graphical component (e.g., a pie chart, a bar chart, etc.) that indicates or suggests a percentage or confidence level of sufficient mapping data (e.g., portions of anatomical structures identified) that may be associated with the 3D model. Additionally or alternatively, the 3D surface map itself may be displayed in its current state to inform the operator of the level of completeness and / or level of accuracy of the map.

[0108] If the image or other mapping data is insufficient, a notification may be generated requesting additional data and / or indicating that if no further data is collected, the confidence in the accuracy of the registration may be low. As an example, if one or more marker points are not identified (i.e., an insufficient number of marker points) and / or the image in which one or more of the marker points are identified is unclear, the image and / or mapping data may be insufficient. In response, a notification may be generated that includes a prompt to instruct the operator to move the medical device 120 to capture another image of the missing and / or unclear one or more marker points. Additionally, the prompt may instruct the operator to confirm or manually identify the missing and / or unclear marker points. As another example, a notification may be generated that includes a prompt to instruct the operator to move the medical device 120 to return to view the most easily identified marker points, and / or have the operator confirm or manually identify these marker points.

[0109] In some examples, where intraoperative 3D imaging is available, an intraoperative 3D image (e.g., different from the 3D image received in step 302) may be captured during the medical procedure to generate a 3D surface map more quickly and / or more accurately. The intraoperative 3D image may also help correct or explain deformations or displacements caused by differences in patient position or the presence of the medical device 120 in the body cavity. Capturing and using intraoperative 3D images to supplement mapping data may reduce the time spent collecting data with the medical device 120, because only minimal 3D surface mapping data may be required to align a 3D model that has been corrected for deformation. As a result, the duration of the entire surgery may also be reduced while further improving accuracy. The intraoperative 3D image may be taken when the medical device 120 (i.e., the distal tip 210) is positioned in the duodenum before or after identifying the nipple to accurately depict deformations or displacements caused by the presence of the medical device 120.

[0110] At step 312, process 300 may include registering the 3D model to the patient using the 3D surface map and the spatial information of the medical device 120. The registration may include determining a transformation matrix. The registration may begin when the 3D surface map is generated, but may not be completed until specific structures are mapped in the 3D surface map. When the medical procedure is an ERCP procedure, the mapped structures may include the duodenum and / or the nipple. The spatial information of the medical device 120 may include the position and / or orientation of the medical device 120 relative to the 3D surface map. Additionally, if spatial information of the patient P is optionally received in addition to the spatial information of the medical device received from the position sensing system 104 at step 308, the spatial information of the patient P may also be used to register the 3D model to the patient P, for example, to account for any movement, including respiratory movement of the patient P.

[0111] In some examples, the registration performed may be an automatic registration using one or more algorithms. Figure 5Described in more detail, automatic registration may include an initial registration based on the alignment or matching of anatomical structures in the 3D surface model with corresponding anatomical structures identified in the 3D model. For example, for image registration performed for an ERCP procedure, the extracted 3D model may identify the anatomical structures of the upper GI tract and the pancreatic and biliary trees (and other structures). Similarly, for image registration performed for an ERCP procedure, the 3D surface map may include at least a portion of the upper GI tract and the anatomical structures of the nipple. The anatomical structures of at least a portion of the upper GI tract included in the 3D surface map may be aligned or otherwise matched with corresponding anatomical structures of the upper GI tract in the 3D model. Based on the initial registration, a transformation matrix that compensates for deformation and / or displacement of the anatomical structure may then be calculated by weighting the known susceptibility of certain regions of the anatomical structure to deformation and associating deformation or displacement from one anatomical structure to another. In some examples, spatial information for the medical device 120 (and optionally, for the patient P) received from the position sensing system 104 may be used to estimate the deformation and / or displacement.

[0112] In other examples, automatic registration can be performed using a computer vision or machine learning model that is trained and implemented to perform the registration. In further examples, the registration can be a manual registration and / or a combination of automatic and manual registration. Additionally, in any of these examples, the registration process can also include determining a confidence level and / or a percentage of registration accuracy.

[0113] The determined transformation matrix may be applied to the 3D model to transform the 3D model. The 3D model to which the deformation matrix is ​​applied may be the 3D model extracted in step 304. As a result, the transformed 3D model may now account for any deformation and / or displacement of the anatomical structure due to the patient's position, the presence of the medical device 120, and / or physiological function, for example, to align the anatomical structure within the 3D image to the patient's current anatomical structure.

[0114] Once the 3D model is registered to the patient, the process 300 may include generating a GUI that overlays a representation of the location / trajectory of one or more anatomical structures of interest on the current image of the target site at step 314. For example, for an ERCP procedure, a representation of the bile duct (or at least the common bile duct of the bile duct) and / or the pancreatic duct extending from the papilla may be overlaid on a real-time image of the papilla captured by the imaging device 212 of the medical device 120. The representation of the bile duct and / or the pancreatic duct may be overlaid at the approximate location where the duct would appear if the duct were visible through the imaging device 212 (e.g., creating an augmented reality image).

[0115] In some examples, the representation can be part of the transformed 3D model itself, which includes the anatomical structure of interest. In other examples, the representation can be in the form of a wireframe model, a centerline, a series of disks positioned orthogonally to the centerline, a tubular structure, etc., which can be generated using the transformed 3D model. In additional examples, the overlay can use different sizes, colors, and / or appearances to indicate the approximate size (e.g., diameter) of the anatomical structure of interest. In further examples, the overlay may include features for indicating confidence intervals or potential errors in alignment in the overlay (e.g., which may be determined as part of the alignment process in step 312). An example graphical user interface including an overlay is shown below Figure 6B and Figure 6C middle.

[0116] At step 316, process 300 may include causing a GUI to be displayed on a display device. The display device may be one of displays 110 that displays a current image of the target site. As a result, the display of the GUI may cause a representation of the position or trajectory of one or more anatomical structures of interest to be overlaid on the current image of the target site. For example, the GUI may cause a representation of the position or trajectory of the bile duct and / or pancreatic duct to be overlaid on the current image of the nipple. Based on the displayed GUI, the operator may confirm and / or adjust the position of the medical device 120 (e.g., extend or retract the distal tip 210 and / or manipulate one or more of the knobs 222, 224 to control the steerable section 218) and / or adjust the tool used to perform the intubation (e.g., one of the tools 123 using the elevator 216), for example, to help ensure alignment with the tube of the patient to be intubated.

[0117] After the GUI is displayed and when intubation facilitated by the GUI is initiated, the nipple or duodenal wall may be intentionally moved or displaced in order to access the nipple, or accidentally moved or displaced when the intubation tool is advanced through the nipple. In some embodiments, to compensate for the movement and / or displacement, the overlaid representation may be locally deformed to match the movement at the nipple to provide a more accurate estimate of the tube position and / or trajectory as the tube position and / or trajectory changes during the intubation process. The movement at the nipple may be determined by tracking the position of the nipple using a machine learning algorithm or other computer vision technique. In some examples, if the operator repositions the medical device 120 during intubation, a fluoroscopic image captured by one of the imaging systems 106 may inform how the shaft 208 and / or steerable section 218 has moved as additional contextual information to understand how the nipple and / or duodenal wall has shifted. Additionally, in examples where the representation of the position or trajectory of the bile and / or pancreatic duct is not part of the 3D transformed image itself, but is in the form of a centerline, and other similar examples, the representation may not need to be deformed (e.g., based on the type of representation conveying an implication that the duct positions and trajectories are approximate rather than exact). Alternatively, the GUI may be removed from the display once the tip of the cannulation tool contacts the duodenal wall and / or enters the papilla.

[0118] In further embodiments, the GUI may be updated in real time when one or more other tools (e.g., from tools 123), such as a choledochoscope, a catheter, a balloon, a stent delivery system, a forceps, a basket, a mesh, a biopsy needle, a guidewire, etc., are advanced through one of the tubes after intubation. For example, if the spatial information (e.g., position and / or orientation) of the tool 123 is tracked, the GUI may be updated in real time to overlay representations of the one or more tools 123 advanced through the corresponding tube. An EM-based tracking system, such as a position sensing system 104, may be used to track the position of the tool 123. For example, a tool EM sensor 133 similar to the EM sensor 132 and the patch EM sensor 135 may be positioned, for example, at the distal tip of the tool 123. Additionally or alternatively, the position of the tool 123 may be tracked using a shape sensing tracking system (performed via optical fiber), an accelerometer, and / or by fluoroscopic imaging.

[0119] In some examples, the tracked spatial information and the known length of the tool 123 can be used to represent the length of the tool 123 to depict the tool 123 from a third person perspective (POV) when the tool 123 is advanced through the desired tube. The position and / or trajectory of the tool 123 can be represented in an overlay portion on the current image at the nipple target site. In other examples, the tracked spatial information can be used to represent the distal tip of the tool 123 to depict the tool 123 from a first person POV when the tool 123 is advanced through the desired tube. For example, the position and / or trajectory of the tool 123 can be represented in an overlay portion on the current image at the nipple target site. Additionally or alternatively, if one of the tools 123 is a choledochoscope, a representation of the tube through which the choledochoscope is being advanced can be overlaid onto a current 2D image captured by an imaging device of the choledochoscope (e.g., an imaging device 212 similar to the medical device 120). The representation of the overlaid tube may be helpful in allowing the operator to see where the tube is directed past visual obstructions in the tube, such as stones, tumors, etc., which may otherwise prevent those portions of the tube from being visualized by the imaging device of the choledochoscope.

[0120] The visualized and / or tracked locations of obstacles, stents, treatments, biopsy samples, tool paths, etc. may be recorded and referenced for subsequent surgery or imaging. For example, the recorded locations may be provided to the data storage system 108 for storage in association with other images and information of the patient P. This information may be used to plan future surgeries, monitor outcomes for the patient, and / or explain the surgery to the patient, or may be included as part of a research or academic publication.

[0121] Therefore, certain embodiments may be performed to perform image registration. The above process 300 is provided as an example only and may include Figure 3 There may be additional, fewer, different, or differently arranged steps than those depicted.

[0122] Figure 4 An exemplary process 400 for extracting a 3D model for image registration is depicted. Process 400 may be performed by one or more of the components of environment 100, for example, via an application executed thereon that is configured to perform at least operations associated with processing a 3D image to extract a 3D model. In some examples, process 400 may be performed by a computing device associated with one of imaging systems 106 that captures a 3D image from which a 3D model may be extracted. In other examples, process 400 may be performed by MD controller 122 of medical device system 102. In further examples, process 400 may be performed by computing device 112. Process 400 may be used to perform at least a portion of step 304 of process 300 to extract a 3D model.

[0123] At step 402, process 400 may include receiving a 3D image of the anatomy of a patient, such as patient P. The 3D image may be of a specific anatomy (e.g., including a specific anatomical structure), depending on the type of medical procedure. The 3D image may be a pre-operative 3D image captured by one of the pre-operative imaging systems of imaging system 106 prior to the medical procedure, or, if available, an intra-operative 3D image captured by one of the intra-operative imaging systems of the imaging system during the medical procedure. This step 402 may be similar to the step described above with reference to FIG. Figure 3 Step 302 of process 300 described in more detail is the same or similar.

[0124] At decision step 404, process 400 may include determining whether the image quality of the 3D image received at step 402 satisfies a predefined threshold. The predefined threshold may be based on a minimum image quality level required to extract a 3D model from the 3D image such that anatomical structures, including one or more anatomical structures of interest, may be identified (e.g., may be isolated or segmented). If, at decision step 404, it is determined that the image quality of the 3D image satisfies (e.g., is equal to or greater than) the predefined threshold, the process may proceed to step 406.

[0125] Otherwise, if it is determined at decision step 404 that the image quality of the 3D image does not meet (e.g., is below) a predefined threshold, process 400 may return to step 402, where another 3D image of the patient's anatomy is received. In some examples, the other 3D image may be another preoperative 3D image captured by one of the imaging systems 106 and stored in the data storage system 108. For example, the operator may select another 3D image from a plurality of 3D images of the patient P available in the data storage system 108. In other examples, the operator may receive a notification that the 3D image is insufficient and may order additional preoperative imaging for the patient P and / or may automatically order additional preoperative imaging. In a further example, if available, 3D intraoperative imaging may be performed during the medical procedure to capture another 3D image. The other 3D image may then be analyzed to determine whether the image quality threshold is met. Steps 402 and 404 may continue to repeat until a 3D image is received that is determined to meet a minimum image quality level. Then, process 400 may move to step 406.

[0126] At step 406, process 400 may include processing the 3D image to extract a 3D model. The 3D model may identify multiple anatomical structures in the anatomy, including one or more anatomical structures of interest for a medical procedure. This step 406 may be similar to the above-described anatomical structure. Figure 3 Step 304 of process 300 described in detail is the same or similar.

[0127] At decision step 408, process 400 may include determining whether the 3D model extracted at step 406 meets a predefined completeness threshold. The predefined completeness threshold may be based on a minimum number and / or type of complete anatomical structures identified, including a minimum number and / or type of complete anatomical structures used as registration landmarks (e.g., to ensure accurate registration). For example, for an ERCP procedure, the predefined completeness threshold may include at least a complete common bile duct and a complete pancreatic duct, as well as other types of complete anatomical structures. If it is determined at decision step 408 that the 3D model extracted at step 406 meets (e.g., is equal to or greater than) the completeness threshold, process 400 may end at step 410. In some examples, the extracted 3D model may be transmitted to another system and / or computing device within environment 100 for processing, analysis, storage, display, etc. Otherwise, if it is determined at decision step 408 that the 3D model extracted at step 406 does not meet (e.g., is less than) the completeness threshold, process 400 may continue to step 412.

[0128] In step 412, process 400 may include estimating one or more incomplete anatomical structures. An estimated model of incomplete anatomical features may be calculated based on available anatomical information. As an example, the common bile duct and pancreatic duct identified in the extracted 3D model may be incomplete. For example, the last few millimeters of the common bile duct and pancreatic duct where the tube flows into the duodenum (e.g., a short section of the tube) may not be visible on the 3D CT and / or MRCP image, for example, because the nipple is tightened around this section of the duct. The nipple tightened around this section of the duct may cause the surrounding fluid to act as a contrast agent for the 3D CT and / or MRCP image. Since this section of the duct is not visible on the 3D CT and / or MRCP image, it may not be possible to identify (e.g., isolate or segment) this section of the duct in the 3D model extracted from the 3D image. In order to complete the 3D model, the incomplete section of the duct may be estimated or approximately calculated by extrapolating the duct to the position where it intersects with the duodenum based on the trajectory of the duct before the incomplete section.

[0129] In some examples, the operator may be requested to confirm whether the estimate of one or more incomplete anatomical structures is a reasonable estimate. For example, a 3D model of the incomplete anatomical structure including the estimate may be provided for display with a prompt seeking confirmation. The prompt may be displayed in association with the estimated incomplete anatomical structure (e.g., on one or more displays 110). Additionally or alternatively, the operator may be enabled to manually estimate the trajectory of the incomplete anatomical structure. For example, the operator may provide the estimated trajectory to one of the displays 110 and / or an associated computing device (e.g., a computing device of one of the imaging systems 106, MD controller 122, and / or computing device 112) via touch or other input.

[0130] Therefore, certain embodiments may be performed to process 3D images to extract 3D models. The above process 400 is provided as an example only and may include the following steps: Figure 4 There may be additional, fewer, different, or differently arranged steps than those depicted.

[0131] Figure 5 An exemplary process 500 for registering a 3D model to a patient is depicted. The process 500 may be performed by one of the components of the environment 100, such as the MD controller 122 or the computing device 112 of the medical device system 102, via an application executed thereon, the application being configured to perform at least operations associated with registration. The process 500 may be used to perform at least a portion of step 312 of the process 300 to register the 3D model, wherein the registration includes determining a transformation matrix.

[0132] At step 502, process 500 may include matching one or more of the portions of the plurality of anatomical structures included in the 3D surface map generated in step 310 of process 300 to corresponding anatomical structures in the 3D model extracted in step 304 of process 300. Continuing with the example of an ERCP procedure, the extracted 3D model may identify or isolate anatomical structures of the upper GI tract and the pancreatic and biliary trees (and other structures), and the 3D surface map includes at least a portion of the anatomical structures of the upper GI tract and the papilla. At step 502, at least a portion of the anatomical structures of the upper GI tract included in the 3D surface map may be matched or aligned to corresponding anatomical structures of the upper GI tract identified in the extracted 3D model.

[0133] At step 504, process 500 may include performing an initial registration based on the matching. In some examples, the initial registration may be a rigid image registration using methods and / or processes that are known or become known in the art. For example, parameters of a transformation matrix may be identified to map voxel locations in a 3D model to a 3D surface map. In some examples, the initial registration may require matching and / or aligning a predefined number (or type) of anatomical structures. For example, at least three anatomical structures may be matched or aligned.

[0134] The initial registration performed may assume that the anatomical structures are rigid structures. However, anatomical structures are not rigid structures. The anatomical structures in the 3D surface map may deform and / or shift. For example, the deformation and / or shift may be based on the positioning of the patient P during the medical procedure relative to the positioning of the patient P when capturing the 3D images used to extract the 3D model, based on the introduction of the medical device 120 into the body cavity during the medical procedure, and / or based on other naturally occurring physiological functions of the anatomical structure, such as breathing. For example, the orientation of the duodenum and pylorus may appear to be rotated relative to the esophageal axis between the 3D surface map and the 3D model, which may occur when the 3D model is captured in a position different from the patient's position during the ERCP procedure. As another example, the shape of anatomical structures such as the stomach may change dramatically when the wall collapses or expands due to changes in internal pressure, but the common bile duct and pancreatic duct may not deform or shift to the same extent.

[0135] To improve the accuracy of the registration, deformation and / or displacement of the anatomical structure may be compensated. For example, at step 506, process 500 may include determining deformation compensation. Deformation compensation may be determined by weighting the known susceptibility of certain regions of the anatomical structure to deformation and associating deformation or displacement from one anatomical structure to another.

[0136] For example, knowing that the shape of the stomach may deform or shift with a significantly greater magnitude than the common bile duct and / or pancreatic duct, any deformation may be compensated for with a smaller magnitude for the transformation parameters corresponding to the common bile duct and / or pancreatic duct (i.e., compared to the stomach). For example, based on the assumption that short segments of the common bile duct and / or pancreatic duct are elastic, but the corresponding remaining portions of the ducts are relatively fixed, the positions or trajectories of the corresponding fixed portions of the ducts may be continuously derived (e.g., using a spline function).

[0137] As another example, other structures identified in the extracted model, such as the spine, ribs, etc., may inform susceptibility to deformation. For example, if a region of the upper GI tract has another anatomical structure nearby, that region may be more rigid and less susceptible to deformation.

[0138] As a further example, if the orientation of the duodenum and pylorus appears to be rotated relative to the esophageal axis, the transformation parameters corresponding to the biliopancreatic tree may be adjusted accordingly, assuming that the biliopancreatic tree will follow a similar rotation. In some examples, this assumption may be learned from a training set of patient images that are analyzed to identify a typical position of the biliopancreatic tree based on a given orientation of the duodenum and pylorus relative to the esophageal axis.

[0139] In some examples, spatial information of medical device 120 from position sensing system 104 may be used to estimate deformation and / or displacement. For example, the position and / or orientation of distal tip 210 and / or any other position of medical device 120 in which EM sensor 132 may be positioned may be used to estimate deformation and / or displacement. Additionally or alternatively, the position of patient P from patch EM sensor 135 may be used to estimate deformation and / or displacement.

[0140] At step 508, process 500 can include determining a transformation matrix based on the initial registration and the deformation compensation. In some examples, the transformation matrix can initially be based on parameters of the initial registration, which can then be adjusted or altered using deformation compensation.

[0141] Therefore, certain embodiments may be performed to register the 3D model to the patient. The above process 500 is provided as an example only and may include the following steps: Figure 5 There may be additional, fewer, different, or differently arranged steps than those depicted.

[0142] In the present invention, various steps can be described as Figure 1A and Figure 1B The instructions may be executed or performed by one of the components in the environment 100, such as one of the imaging systems 106 or a computing device associated therewith, the MD controller 122 of the medical device system 102, or the computing device 112. However, it should be understood that in various embodiments, the various components of the environment 100 discussed above may execute instructions or perform steps, including the steps discussed above. The steps performed by the device may be considered to be performed by a processor, actuator, etc. associated with the device. In addition, it should be understood that in various embodiments, various steps may be added, omitted, and / or rearranged in any suitable manner.

[0143] FIG. 6A to FIG. 6C Depicted are exemplary GUIs 600A, 600B, 600C generated and displayed during a medical procedure, such as an ERCP procedure. Fig. 6A The first GUI 600A shown in FIG. 6A displays a current image 602 of a nipple 604, captured, for example, by the imaging device 212 of the medical device 120 during an ERCP procedure. The nipple 604 may be a target site for cannulation during the ERCP procedure. Figure 6BThe second GUI 600B shown in FIG. 6A shows a portion of the transformed 3D model 610 representing the position and trajectory of the common bile duct 612 and the pancreatic duct 614, and the transformed 3D model 610 is overlaid on the current image 602. Thus, the operator can view the common bile duct 612 and the pancreatic duct 614 at the approximate location where the tubes would appear if the tubes were visible via the imaging device 212. Using the second GUI 600B as a guide, the operator of the medical device 120 can confirm and / or adjust the position of the medical device 120 (e.g., extending or retracting the distal tip 210 and / or manipulating one or more of the knobs 222, 224 to control the steerable section 218) and / or the tools used to perform the cannulation (e.g., one of the tools 123 using the elevator 216), for example, to help ensure alignment with the patient's tube to be cannulated.

[0144] In other examples, rather than overlaying a portion of the transformed 3D model itself over the current image 602, a representation of the position and / or trajectory of the tube generated using the transformed 3D model may be overlaid in the form of a wireframe model, a centerline, a series of disks orthogonal to the centerline, a tubular structure, etc. Figure 6C The third GUI 600C shown in FIG. 600C shows an exemplary centerline representation 620 of the ducts. For example, the first visual indicator 622 may be a centerline representation of the common bile duct, and the second visual indicator 624 may be a centerline representation of the pancreatic duct. The first visual indicator 622 and the second visual indicator 624 may be visually distinct from each other to emphasize the different trajectories of the common bile duct and the pancreatic duct.

[0145] Although FIG. 6A to FIG. 6C Although not shown in the figure, various other visual schemes may be employed in GUIs generated and displayed during medical procedures. The above-mentioned GUIs 600A, 600B, 600C are provided as examples only and may include the same FIG. 6A to FIG. 6C Additional, less, different, or differently arranged information and / or features than those depicted.

[0146] Figure 7 One example of a computer 700 is depicted. Figure 7 is a simplified functional block diagram of a computer 700 that may be configured to perform the operations described in FIGS. 1 to 2. Figure 6C1 , or a device for a process, step, or operation depicted in or related to the description thereof and according to an exemplary embodiment of the present invention. For example, the computer 700 may be configured as a computing device of or associated with one of the imaging systems 106, the MD controller 122 of the medical device system 102, the computing device 112, the PSS controller 130 of the position sensing system 104, and / or one of another devices or components according to an exemplary embodiment of the present invention. In various embodiments, any of the systems herein may be a computer 700 or include a computer 700, which includes, for example, a data communication interface 720 for packet data communications. The computer 700 may communicate with one or more other computers, for example, using an electronic network 725 (e.g., via the data communication interface 720). The electronic network 725 may include a wired or wireless network similar to the network 114 depicted in FIG. 1 .

[0147] The computer 700 may also include a central processing unit ("CPU") in the form of one or more processors 702 for executing program instructions 724. The program instructions 724 may include instructions for running one or more applications associated with 3D model extraction, 3D surface map generation, and / or 3D image registration and GUI generation on one of the imaging systems 106, the medical device system 102, or the computing device 112 (e.g., if the computer 700 is one of the imaging systems 106 or a computing device associated therewith, the medical device system 102, or the computing device 112). The program instructions 724 may include instructions for running one or more operations for position and / or orientation determination (e.g., if the computer 700 is the PSS controller 130 of the position sensing system 104). The computer 700 may include an internal communication bus 708. Although the computer 700 can receive programming and data via network communications, the computer may also include a drive unit 706 (such as a read-only memory (ROM), a hard disk drive (HDD), a solid-state disk drive (SDD), etc.), which can store data on a computer-readable medium 722 (e.g., a non-transitory computer-readable medium). The computer 700 may also have a memory 704 (such as a random access memory (RAM)) that stores instructions 724 for performing the techniques presented herein. However, it should be noted that in some aspects, the instructions 724 may be temporarily or permanently stored in other modules of the computer 700 (e.g., the processor 702 and / or the computer-readable medium 722). The computer 700 may also include a user input and output device 712 and / or a display 710 to connect to an input and / or output device, such as a keyboard, a mouse, a touch screen, a monitor, a display, etc. Various system functions may be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the system may be implemented by appropriate programming of one computer hardware platform.

[0148] The program aspects of the present technology may be considered as "products" or "manufactures", which typically take the form of executable code and / or associated data carried or embodied on or in a machine-readable medium. "Storage" type media include any or all of computers, processors, etc. or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc. (which can provide non-transient storage for software programming at any time). All or part of the software can sometimes communicate through the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another computer or processor. Therefore, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as across physical interfaces between local devices, through wired and optical fixed telephone line networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be regarded as media that carry software. As used herein, except for being limited to non-transient, tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0149] Although the principles of the present invention are described herein with reference to illustrative examples for specific applications, it should be understood that the present invention is not limited thereto. Personnel with ordinary skills in the art and access to the teachings provided herein will recognize that additional modifications, applications, and replacements of equivalents all fall within the scope of the examples described herein. Therefore, the present invention should not be considered to be limited by the foregoing description.

Claims

1. A system for three-dimensional (3D) image registration to facilitate medical procedures, the system comprising: A medical device having a distal portion configured to be inserted into a body cavity of a patient during a medical procedure, the medical device comprising: an imaging device at a distal tip of the distal portion, the imaging device configured to capture a plurality of images of the body cavity as the medical device is inserted into and navigated through the body cavity to a target site, at least one of the plurality of images comprising a current image of the target site; and a transmitter device or a receiver device of a position sensing system located at the distal tip, wherein the position sensing system is configured to determine the position or orientation of the distal tip; and A non-transitory computer-readable medium storing instructions that, when executed by a processor of a computing device, cause the processor to execute the instructions to perform operations comprising: receiving a 3D image of the patient's anatomy captured by an imaging system prior to the medical procedure; processing the 3D image to extract a 3D model that identifies a plurality of anatomical structures in the anatomy, including one or more anatomical structures of interest for the medical procedure; receiving a plurality of images of the body cavity captured by the imaging device during the medical procedure; receiving a position or orientation of a distal tip of the medical device from the position sensing system; processing the plurality of images of the body cavity and the position or the orientation of the distal tip to generate a 3D surface map of at least a portion of the body cavity, the 3D surface map including a portion of the plurality of anatomical structures that does not include the one or more anatomical structures of interest; registering the 3D model to a patient using the 3D surface map and the position or the orientation of the distal tip; Based on the registration, generating a graphical user interface (GUI) that overlays a representation of the location or trajectory of one or more anatomical structures of interest on the current image of the target site; and The GUI is displayed on a display device.

2. The system of claim 1, wherein the operations further comprise: determining that one or more of the plurality of anatomical structures identified in the 3D model is incomplete; as well as An incomplete portion of the one or more of the plurality of anatomical structures is estimated.

3. The system of claim 2, wherein the operations further comprise: generating a prompt having an instruction for an operator to confirm the incomplete portion of the estimate; as well as The 3D model including the estimated incomplete portion and the hint are displayed via the display device, wherein the hint is displayable in association with the estimated incomplete portion of the 3D model.

4. The system according to any one of claims 1 to 3, wherein the operation further comprises: determining that additional image data is needed to generate the 3D surface map based on the number or type of anatomical structures in the portion of the plurality of anatomical structures included in the 3D surface map; generating a prompt having instructions for an operator to move the medical device to one or more locations in the body cavity, the one or more locations corresponding to one or more of the plurality of anatomical structures that are not included in the portion or that are included in the portion and are incomplete, to capture the additional image data via the imaging device of the medical device; as well as The prompt is displayed via the display device.

5. The system of any one of claims 1 to 4, wherein registering the 3D model to the patient further comprises: Determine the transformation matrix; as well as The transformation matrix is ​​applied to the 3D model to transform the 3D model.

6. The system of claim 5, wherein registering the 3D model to the patient further comprises: matching one or more of the portions of the plurality of anatomical structures included in the 3D surface map to corresponding anatomical structures in the 3D model; performing an initial registration based on the matching; Determine deformation compensation; as well as The transformation matrix is ​​determined based on the initial registration and the deformation compensation.

7. A system according to any one of claims 5 or 6, wherein generating the GUI includes generating the GUI using a transformed 3D model, and the representation of the position or trajectory of the one or more anatomical structures of interest includes at least one of the following: a portion of the transformed 3D model including the one or more anatomical structures of interest, a wireframe model of the one or more anatomical structures of interest, a centerline representation for the one or more anatomical structures of interest, a series of disks positioned orthogonally to the centerline of the one or more anatomical structures of interest, or a tubular structure for the one or more anatomical structures of interest.

8. The system according to any one of claims 1 to 7, wherein the operations further comprise: receiving spatial information of the patient from the position sensing system, the position sensing system comprising one or more transmitter devices or receiver devices located in a patch topically applied to the patient; as well as The 3D model is registered to the patient using the 3D surface map, the spatial information of the medical device, and the spatial information of the patient.

9. The system according to any one of claims 1 to 8, wherein the operations further comprise: One or more anatomical structures are identified from the portion of the multiple anatomical structures in the 3D surface map by providing the multiple images as input to a machine learning model, the machine learning model being trained to predict the one or more anatomical structures present in each of the multiple images.

10. The system according to any one of claims 1 to 9, wherein the operations further comprise: As part of generating the 3D surface map, one or more anatomical structures are identified from the portion of the plurality of anatomical structures in the 3D surface map by mapping the geometry of the one or more anatomical structures as a 3D surface, and the anatomical structures are identified based on the mapped geometry.

11. The system of any one of claims 1 to 10, wherein the target site is a cannulation site, and wherein the operation further comprises: tracking movement at the target site as the cannulation occurs; as well as The GUI is updated to deform the representation of the position or trajectory of the one or more anatomical structures of interest overlaying the current image of the target site to match the movement of the target site.

12. The system according to any one of claims 1 to 11, wherein the operations further comprise: receiving spatial information of a tool delivered to the target site via the medical device from the position sensing system as the tool is advanced through at least one of the one or more anatomical structures of interest, wherein the tool includes one or more transmitter devices or one or more receiver devices of the position sensing system; as well as Using the spatial information of the tool, the GUI is updated to depict a representation of the tool advanced through the representation of the location or trajectory of the at least one of the one or more anatomical structures of interest overlaid on the current image of the target site.

13. The system according to any one of claims 1 to 12, wherein the operations further comprise: receiving additional 3D images captured intraoperatively after the medical device is navigated through the body cavity to the target site; In addition to processing the plurality of images, the additional 3D images are processed to generate the 3D surface map.

14. The system according to any one of claims 1 to 13, wherein the operations further comprise: Before processing the 3D image to extract the 3D model, it is determined whether the image quality of the 3D image meets a predefined threshold.

15. The system of any one of claims 1 to 14, wherein the medical procedure is an endoscopic retrograde cholangiopancreatography (ERCP) procedure, the target site is a papilla for cannulation, and the one or more anatomical structures of interest include at least a common bile duct and a pancreatic duct, and generating the GUI comprises: Using the registered 3D model, representations of the location or trajectory of the common bile duct and the pancreatic duct are overlaid on the current image of the papilla to create an augmented reality image.

Images