Systems and methods for registering instruments to images using point cloud data and endoscopic image data.

Abstract

This document discloses apparatus, systems, methods, and computer program products for combining position sensor data and endoscopic image data to improve the registration between (i) real patient anatomy within an anatomical region navigated by a medical device system and (ii) images of the anatomical region generated from preoperative and / or intraoperative imaging. When properly registered, the tracking position of the medical device system within the anatomical region can be mapped to the correct position within the anatomical model for use, for example, image-guided medical procedures. In some embodiments, this technology provides visual guidance, in the form of a virtual navigation image, at a viewpoint of the medical device system generated from within the anatomical model at a point corresponding to the location of the medical device system within the anatomical region after registration.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 994,205, filed March 24, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to systems, methods, and computer programs for registering instruments and image reference systems by combining point cloud data and endoscopic image data. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, the operator can insert minimally invasive medical instruments to reach target tissue sites. Minimally invasive medical instruments include devices such as therapeutic, diagnostic, biopsy, and surgical instruments. Medical instruments can be inserted into anatomical passageways and navigate towards regions of interest within the patient's anatomy. Image-aided navigation can be used with respect to the anatomical passageways. Improved systems and methods are needed to accurately perform image registration between medical instruments and anatomical passageways. Summary of the Invention

[0005] This document discloses devices, systems, methods, and computer program products for combining position sensor data (e.g., shape and / or electromagnetic sensor data) and endoscopic image data (e.g., video data, still images, etc.) to improve registration between (i) real patient anatomy (e.g., the patient's lung airways) within an anatomical region of the patient navigated by a medical device system as part of an image-guided medical procedure and (ii) images of an anatomical region (e.g., generated from preoperative and / or intraoperative imaging).

[0006] In some embodiments, a medical device system for use in an image-guided medical procedure includes a position sensor, an image capture device, a processor communicatively coupled to the position sensor and the image capture device, and a memory. The position sensor may be configured to generate position sensor data associated with one or more locations of the biomedical device within an anatomical region of the patient. The image capture device may be configured to capture first image data of patient anatomical structures within the anatomical region when the biomedical device is located within the anatomical region. The memory may store instructions that, when executed by the processor, cause the medical device system to perform operations including: (i) generating a point cloud of coordinate points at least partially based on the position sensor data; (ii) receiving second image data of the anatomical region, wherein the second image data is generated at least partially based on imaging of the anatomical region; (iii) generating a registration between at least a portion of the point cloud and at least a portion of the second image data; and / or (iv) updating the registration at least partially based on the first image data.

[0007] In these and other embodiments, a non-transient computer-readable medium may store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations including: (i) generating a point cloud of coordinate points based at least in part on location sensor data captured using a location sensor, wherein the location sensor data is associated with one or more locations of a biomedical device within the anatomical region of the patient; (ii) receiving first image data of a patient's anatomical structure captured using an image capture device located within the anatomical region; (iii) receiving second image data of the anatomical region, wherein the second image data is generated at least in part based on preoperative or intraoperative imaging of the anatomical region; (iv) generating a registration between at least a portion of the point cloud and at least a portion of the second image data; and / or (v) updating the registration at least in part based on the first image data.

[0008] In these and other embodiments, a method may include (i) generating a point cloud of coordinate points based at least in part on position sensor data captured using a position sensor of a robotic system, wherein the position sensor data is associated with one or more locations of a biomedical device within an anatomical region of the patient; (ii) receiving first image data of the patient's anatomical structures captured using an image capture device of the robotic system when the image capture device of the robotic system is located within the anatomical region; (iii) receiving second image data of the anatomical region, wherein the second image data is based at least in part on preoperative or intraoperative imaging of the anatomical region; (iv) generating a registration between at least a portion of the point cloud and at least a portion of the second image data; and / or (v) updating the registration based at least in part on a portion of the first image data.

[0009] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of the disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0010] Many aspects of this disclosure can be better understood with reference to the following accompanying drawings. The components in the drawings are not necessarily drawn to scale. Rather, the focus is on clearly illustrating the principles of this disclosure. The drawings should not be construed as limiting this disclosure to the specific embodiments depicted, but are for explanation and understanding only.

[0011] Figure 1 This is a schematic diagram of a robot or remotely operated medical system configured according to various embodiments of the present technology.

[0012] Figure 2 This is a schematic diagram of a manipulator assembly, medical device system, and imaging system configured according to various embodiments of the present technology.

[0013] Figure 3 It extends within the anatomical region of the patient according to various embodiments of this technology. Figure 2 A schematic diagram of a part of a medical device system.

[0014] Figure 4 The illustration shows multiple coordinate points configured according to various embodiments of the present technology to form a point cloud, which represents... Figure 3 The shape of this part of the medical device system.

[0015] Figure 5 The illustrations depict various embodiments of the present technology. Figure 3 Extending within the anatomical region Figure 3 The viewpoint of this part of the medical device system is a real navigation image of the actual patient anatomy.

[0016] Figure 6 The illustrations depict various embodiments of the present technology. Figure 3 When this part of the medical device system extends within the anatomical region Figure 3 Intraoperative images of a portion of the anatomical region.

[0017] Figure 7 This is a schematic diagram of a display system according to various embodiments of the present technology, the display system displaying a synthetic virtual navigation image, a virtual navigation image of a virtual patient anatomy, and a real navigation image of a real patient anatomy within the anatomical region, wherein the synthetic virtual navigation image, Figure 2 and Figure 3 The medical device system was registered to Figure 3 Anatomical model of the anatomical region.

[0018] Figure 8 This is a flowchart illustrating a method for registering an image of a patient's anatomical region with a point cloud of coordinate points using endoscopic image data, according to various embodiments of the present technology.

[0019] Figure 9 This is a schematic diagram of a real navigation image of the anatomical region of a patient and the actual patient anatomy within the anatomical region, as viewed from the perspective of a medical device system extending at different depths within the anatomical region, according to various embodiments of the present technology.

[0020] Figure 10 The illustration shows a realistic navigational image of a patient's anatomy as seen from the viewpoint of a medical device system extending within the patient's anatomical region, according to various embodiments of the present technology.

[0021] Figure 11A and Figure 11B The illustration shows the locations depicted within an anatomical model of the anatomical region from the viewpoint of the medical device system. Figure 10 A virtual navigation image of the anatomical structure of a virtual patient in the anatomical region, where points within the anatomical model of the anatomical region correspond to points of a medical device system extending within the anatomical region.

[0022] Figure 12 The illustration shows a virtual navigation image of the virtual patient anatomy of the anatomical region as viewed from the perspective of the medical device system, with points within the anatomical model depicting points in the anatomical region corresponding to points of the medical device system extending within the anatomical region.

[0023] Figures 13A-13C The illustration shows a real navigational image of the patient's anatomy within the anatomical region of Figure 11, viewed from the perspective of a medical device system extending at different depths within the anatomical region. Detailed Implementation

[0024] This disclosure relates to devices, systems, methods, and computer program products for registering images of a patient's anatomical region (i) as part of an image-guided medical procedure, navigating by a medical device system, between real patient anatomy (e.g., the patient's lung airways) and (ii) images of the anatomical region (e.g., generated from preoperative and / or intraoperative imaging). When fully registered, the tracking position of the medical device system within the anatomical region can be mapped to the correct position within an anatomical model of the anatomical region for guiding navigation of the medical device system throughout the anatomical region and / or for guiding interactions with subsurface structures within and / or near the anatomical region (e.g., for guiding biopsies and / or treatment of lung nodules). In particular, this technology provides visual guidance in the form of virtual navigation (e.g., fly-through) images of the viewpoint of the medical device system within the anatomical region generated from within the anatomical model at the location of the medical device system after registration.

[0025] In some embodiments, the step of registering a real patient anatomy to an anatomical model includes: (a) navigating a medical device system across the entire anatomical region of the patient; (b) generating a point cloud representing coordinates of points visited by the medical device system (e.g., a distal portion of the anatomical region); and (c) registering the point cloud (using an iterative nearest-point algorithm) to an image (e.g., a segmented CT image) of the anatomical region. In these and other embodiments, the technique uses an endoscope or other image capture device mounted to a distal portion (or another suitable location) of the medical device system to capture endoscopic image data (e.g., video data, still images, etc.) that includes a real navigation image of the real patient anatomy within the anatomical region. In these and other embodiments, the technique computes a virtual navigation image based at least in part on the registration. The virtual navigation image depicts a virtual patient anatomy of the anatomical region from the perspective of the distal portion (or another suitable location) of the medical device system.

[0026] In some embodiments, the degree to which a virtual navigation image of a virtual patient's anatomy matches a real navigation image of a real patient's anatomy of the anatomical region provides an indication of the degree of registration of the point cloud of coordinate points with the image of the anatomical region (e.g., with segmented CT images). Therefore, this technique utilizes information provided by both real and virtual navigation images to improve the registration of point clouds generated from data captured by a medical device system with preoperative and / or intraoperative images of the anatomical region. In the context of biopsy procedures, this technique thereby increases the accuracy of region-of-interest localization (e.g., tumor location estimation), which increases the probability of successfully navigating the patient's anatomical region and the probability of effectively diagnosing and treating the disease (e.g., effectively biopsiing or ablating small lung tumors).

[0027] A. Examples of robotic or remotely operated medical systems and associated devices, systems, and methods.

[0028] 1. Robots or remotely operated medical systems and related equipment and systems

[0029] Figure 1 This is a schematic diagram of a robotic or remotely operated medical system 100 (“Medical System 100”) configured according to various embodiments of the present technology. As shown, Medical System 100 includes a manipulator assembly 102, a medical device system 104, a master control assembly 106, and a control system 112. The manipulator assembly 102 supports and drives the medical device system 104 under the guidance of the master control assembly 106 and / or the control system 112 to perform various medical procedures on a patient 103 located on an operating table 107 in a surgical environment 101. In this regard, the master control assembly 106 typically includes one or more control devices operated by an operator 105 (e.g., a physician) to control the manipulator assembly 102. Additionally or alternatively, the control system 112 includes a computer processor 114 and at least one memory 116 for implementing control between the medical device system 104, the master control assembly 106, and / or other components of Medical System 100. The control system 112 may also include programmed instructions (e.g., a non-transient machine-readable medium storing these instructions) for implementing any one or more methods disclosed herein, including instructions for providing information to the display system 110 and / or instructions for processing data to register the medical device system 104 with an anatomical model of the patient 103 (as described in more detail below). The manipulator component 102 may be a remotely operated, non-remotely operated, or a hybrid of both. Therefore, all or part of the master control component 106 and / or all or part of the control system 112 may be located inside or outside the surgical environment 101.

[0030] To assist operator 105 in controlling manipulator assembly 102 and / or medical device system 104 during image-guided medical procedures, medical system 100 may further include position sensor system 108, endoscopic imaging system 109, imaging system 118, and / or virtual visualization system 115. In some embodiments, position sensor system 108 includes a point sensor system (e.g., an electromagnetic (EM) sensor system) and / or shape sensor system for capturing position sensor data (e.g., position, orientation, velocity, rate, posture, shape, etc.) of medical device system 104. In these and other embodiments, endoscopic imaging system 109 includes one or more image capture devices (not shown) that record endoscopic image data including concurrent or real-time images (e.g., video, still images, etc.) of patient anatomy. Images captured by endoscopic imaging system 109 may be two-dimensional or three-dimensional images of patient anatomy, for example, captured by an image capture device located within patient 103, and are referred to below as “real-time navigation images.”

[0031] In some embodiments, the medical device system 104 may include components of the position sensor system 108 and / or components of the endoscopic imaging system 109. For example, components of the position sensor system 108 and / or components of the endoscopic imaging system 109 may be integrally or removably coupled to the medical device system 104. Additionally or alternatively, the endoscopic imaging system 109 may include a separate endoscope (not shown) attached to a separate manipulator assembly (not shown), which may be used in conjunction with the medical device system 104 to image patient anatomy. The position sensor system 108 and / or the endoscopic imaging system 109 may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, such as one or more computer processors 114 of the control system 112.

[0032] The imaging system 118 of the medical system 100 can be positioned near the patient 103 within the surgical environment 101 to acquire real-time and / or near-real-time images of the patient 103 before, during, and / or after medical procedures. In some embodiments, the imaging system 118 includes a moving C-arm cone-beam computed tomography (CT) imaging system for generating three-dimensional images. For example, the imaging system 118 may include a DynaCT imaging system from Siemens Corporation, or other suitable imaging systems. In these and other embodiments, the imaging system 118 may include other imaging techniques, including magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques.

[0033] When the medical device system 104 is controlled during an image-guided medical procedure, the virtual visualization system 115 of the control system 112 provides navigation and / or anatomical interaction assistance to the operator 105. As described in more detail below, virtual navigation using the virtual visualization system 115 may be based at least in part on acquired preoperative or intraoperative datasets of the anatomical pathway of the patient 103 (e.g., at least in part on data generated by the position sensor system 108, the endoscopic imaging system 109, and / or the imaging system 118). In some embodiments, for example, the virtual visualization system 115 processes preoperative and / or intraoperative image data of the anatomical regions of the patient 103 captured by the imaging system 118 to generate an anatomical model (not shown) of the anatomical regions. Then, the virtual visualization system 115 registers the anatomical model to position sensor data generated by the position sensor system 108 and / or endoscopic image data generated by the endoscopic imaging system 109, in order to (i) map the tracking position, orientation, posture, shape, and / or movement of the medical device system 104 within the anatomical region to the correct position within the anatomical model, and / or (ii) determine the virtual patient anatomical structure of the anatomical region from the viewpoint of the medical device system 104 at the point in the anatomical model corresponding to the point of the medical device system 104 within the patient 103.

[0034] Display system 110 can display various images or representations of patient anatomy and / or medical device system 104 generated by position sensor system 108, endoscopic imaging system 109, imaging system 118, and / or virtual visualization system 115. In some embodiments, display system 110 and / or master control component 106 can be configured such that operator 105 can remotely control manipulator component 102, medical device system 104, master control component 106, and / or control system 112.

[0035] As described above, the manipulator assembly 102 drives the medical device system 104 under the guidance of the master control assembly 106 and / or the control system 112. In this regard, the manipulator assembly 102 may include selectable degrees of freedom of motion, which may be motorized and / or remotely operated, as well as selectable degrees of freedom of motion, which may be non-motorized and / or non-remotely operated. For example, the manipulator assembly 102 may include multiple actuators or motors (not shown) that drive inputs on the medical device system 104 in response to commands received from the control system 112. The actuators may include a drive system (not shown) that, when coupled to the medical device system 104, can advance the medical device system 104 into an anatomical opening created naturally or surgically. Other drive systems may move a distal portion (not shown) of the medical device system 104 with multiple degrees of freedom, which may include three linear degrees of freedom (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of freedom (e.g., rotation about the X, Y, Z Cartesian axes). Additionally or alternatively, the actuator may be used to actuate the articulated end effector of the medical device system 104 (e.g., for gripping tissue in the jaws of a biopsy device and / or for similar purposes).

[0036] Figure 2 It is located within the surgical environment 101 and is configured according to various embodiments of the present technology. Figure 1 The manipulator assembly 102, medical device system 104, and imaging system 118 are shown in the schematic diagram. Figure 2 As shown, the surgical environment 101 has a surgical reference frame (X). S Y S Z S In this system, patient 103 is located on operating table 107, and medical device system 104 has a medical device reference frame (X) within the surgical environment 101. M Y M Z M During medical procedures, patient 103 may be stationary within the surgical setting in a sense that total patient movement can be limited by sedation, restraint, and / or other means. In these and other embodiments, circulatory anatomical movements of patient 103 (including respiratory and cardiac movements) may continue unless patient 103 is instructed to hold his or her breath to temporarily suspend respiratory movements.

[0037] The manipulator assembly 102 includes an instrument holder 226 mounted to the insertion stage 228. In the illustrated embodiment, the insertion stage 228 is linear, while in other embodiments, the insertion stage 228 is curved or has a combination of curved and linear segments. In some embodiments, the insertion stage 228 is fixed within the surgical environment 101. Alternatively, the insertion stage 228 may be movable within the surgical environment 101 but has known locations within the surgical environment 101 (e.g., via a tracking sensor (not shown) or other tracking device). In these alternatives, a medical device reference frame (X... M Y M Z M ) relative to the surgical reference frame (X S Y S Z S () is fixed or otherwise known.

[0038] Figure 2 The medical device system 104 includes an elongated device 231, a medical device 232, a device body 235, at least a portion of a position sensor system 108, and at least a portion of an endoscopic imaging system 109. In some embodiments, the elongated device 231 is a flexible catheter or other biomedical device defining a channel or lumen 244. The channel 244 may be sized and shaped to receive the medical device 232 (e.g., via the proximal end 236 and / or device port (not shown) of the elongated device 231) and facilitate delivery of the medical device 232 to the distal portion 238 of the elongated device 231. The elongated device 231 is coupled to the device body 235, which in turn is coupled to and fixed relative to the device holder 226 of the manipulator assembly 102.

[0039] In operation, the manipulator assembly 102 can control the insertion movement (e.g., proximal and / or distal movement along axis A) of the elongated device 231 into the patient 103 via a natural or surgically created anatomical opening, to facilitate navigation of the elongated device 231 through an anatomical passage through the anatomical region of the patient 103 and / or to facilitate delivery of the distal portion 238 of the elongated device 231 to a target location within or near the patient 103. For example, the instrument holder 226 and / or insertion stage 228 may include actuators (not shown), such as servo motors, to facilitate control of movement of the instrument holder 226 along the insertion stage 228. Additionally or alternatively, in some embodiments, the manipulator assembly 102 can control movement of the distal portion 238 of the elongated device 231 in multiple directions, including yaw, pitch, and roll rotation (e.g., to navigate patient anatomy). For this purpose, the elongated device 231 may accommodate or include cables, linkages, and / or other steering controls (not shown) that the manipulator assembly 102 may use to controllably bend the distal portion 238 of the elongated device 231. For example, the elongated device 231 may accommodate at least four cables that the manipulator assembly 102 may use to provide (i) independent "up and down" control of the pitch of the distal portion 238 of the elongated device 231 and (ii) independent "left and right" control of the elongated device 231 for controlling the yaw of the distal portion 238 of the elongated device 231.

[0040] Medical device 232 of medical device system 104 can be used for medical procedures such as mapping anatomical access, surgery, biopsy, ablation, illumination, irrigation, and / or aspiration. Therefore, medical device 232 may include image capture probes, biopsy instruments, laser ablation fibers, and / or other surgical, diagnostic, and / or therapeutic tools. For example, medical device 232 may include an endoscope or other biomedical device having one or more image capture devices 247 located at a distal portion 237 of medical device 232 and / or at other points along medical device 232. In these embodiments, when medical device 232 is within the anatomical region of patient 103, image capture device 247 may capture one or more real navigation images or videos (e.g., a sequence of one or more real navigation image frames) of the anatomical access and / or other real patient anatomy.

[0041] As discussed above, the medical device 232 can be deployed to and / or delivered to a target location within the patient 103 via a channel 244 defined by the elongated device 231. In embodiments where the medical device 232 includes an endoscope or other biomedical device having an image capture device 247 on its distal portion 237, the image capture device 247 can be advanced to the distal portion 238 of the elongated device 231 before, during, and / or after the manipulator assembly 102 navigates the distal portion 238 of the elongated device 231 to the target location within the patient 103. In these embodiments, the medical device 232 can be used as a mapping instrument to capture realistic navigation images of anatomical channels and / or other real patient anatomy, and / or to assist an operator (not shown) in navigating the distal portion 238 of the elongated device 231 through the anatomical channel to the target location.

[0042] As another example, after the manipulator assembly 102 positions the distal portion 238 of the elongated device 231 near a target location within the patient 103, the medical device 232 can be advanced beyond the distal portion 238 of the elongated device 231 to perform a medical procedure at the target location. Continuing this example, after all or part of the medical procedure at the target location has been completed, the medical device 232 can be retracted into the elongated device 231, and additionally or alternatively, the medical device 232 can be removed from the proximal end 236 of the elongated device 231 or from another device port (not shown) along the elongated device 231.

[0043] like Figure 2 As shown, the position sensor system 108 of the medical device system 104 includes a shape sensor 233 and a position measuring device 239. In these and other embodiments, the position sensor system 108 may include other position sensors (e.g., accelerometers, rotary encoders, etc.) to be attached to or replace the shape sensor 233 and / or the position measuring device 239.

[0044] The shape sensor 233 of the position sensor system 108 includes an optical fiber extending within and aligned with an elongated device 231. In one embodiment, the optical fiber of the shape sensor 233 has a diameter of approximately 200 μm. In other embodiments, the diameter of the optical fiber may be larger or smaller. The optical fiber of the shape sensor 233 forms an optical fiber bend sensor, which is used to determine the shape, orientation, and / or posture of the elongated device 231. In some embodiments, an optical fiber with a fiber Bragg grating (FBG) can be used to provide strain measurements in one-dimensional or multi-dimensional structures. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions are described in more detail in U.S. Patent Application Publication No. 2006 / 0013523 (filed July 13, 2005) (disclosing an optical fiber position and shape sensing device and related methods thereof); U.S. Patent No. 7,781,724 (filed September 26, 2006) (disclosing an optical fiber position and shape sensing device and related methods thereof); U.S. Patent No. 7,772541 (filed March 12, 2008) (disclosing Rayleigh scattering-based optical fiber position and / or shape sensing); and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing an optical fiber bending sensor), the entire contents of which are incorporated herein by reference. In these and other embodiments, the sensor of this technology may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In these and other embodiments, the shape of the elongated device 231 may be determined using other techniques. For example, the history of the orientation of the distal portion 238 of the elongated device 231 can be used to reconstruct the shape of the elongated device 231 over time intervals.

[0045] In some embodiments, the shape sensor 233 is fixed at a proximal point 234 on the device body 235 of the medical device system 104. In operation, for example, the shape sensor 233 measures the medical device reference frame (X). M Y M Z M The shape sensor 233 is defined as the shape from the proximal point 234 to another point along the optical fiber (such as the distal portion 238 of the elongated device 231). The proximal point 234 of the shape sensor 233 may be movable together with the device body 235, but the location of the proximal point 234 may be known (e.g., via a tracking sensor (not shown) or other tracking device).

[0046] As the device body 235 moves along the insertion axis A on the insertion stage 228 of the manipulator assembly 102, the position measuring device 239 of the position sensor system 108 provides information about the position of the device body 235. In some embodiments, the position measuring device 239 includes a solver, an encoder, a potentiometer, and / or other sensors that determine the rotation and / or orientation of an actuator (not shown) that controls the movement of the device holder 226 of the manipulator assembly 102 and thus controls the movement of the device body 235 of the medical device system 104.

[0047] Figure 3 It extends within the anatomical region 350 (e.g., human lung) of patient 103 according to various embodiments of the present technology. Figure 2 A schematic diagram of a portion of the medical device system 104. Specifically, Figure 3 The illustration shows an elongated device 231 of a medical device system 104 extending within a branch anatomical passage 352 of an anatomical region 350. The anatomical passage 352 includes a trachea 354 and multiple bronchi 356.

[0048] like Figure 3 As shown, the elongated device 231 has a position, orientation, posture, and shape within the anatomical region 350, and all or part of it (attached to or replacing movement, such as velocity or rate) can be Figure 1 and Figure 2 Position sensor system 108 (e.g., via shape sensor 233 and / or position measuring device 239) Figure 2 The position sensor system 108 captures data as position sensor data to map the anatomical passage 352 of the anatomical region 350. Specifically, the position sensor system 108 can collect data from a medical device reference frame (X). M Y M Z M The medical device system 104 within the anatomical region 350 is used to map the anatomical passage 352. The position sensor data can be recorded at least partially as a set of two-dimensional or three-dimensional coordinate points. In an example where the anatomical region 350 is a human lung, the coordinate points can represent the location of the distal portion 238 of the elongated device 231 and / or other portions of the elongated device 231 as it is advanced through the trachea 354 and bronchus 356. In these and other embodiments, the set of coordinate points can represent the shape(s) of the elongated device 231(s) as it is advanced through the anatomical region 350. In these and other embodiments, the coordinate points can represent other portions of the medical device system 104 (e.g., medical device 232(s)). Figure 2 Location data.

[0049] Coordinate points can be combined to form a point cloud. For example, Figure 4The illustration shows a plurality of coordinate points 462 forming a point cloud 460 according to various embodiments of the present technology, when the elongated device 231 is located in the anatomical region 350 ( Figure 3 Within this timeframe, the point cloud represents... Figure 3 The shape of the elongated device 231. Specifically, when the elongated device 231 is in... Figure 3 At the indicated rest position, Figure 4 The point cloud 460 is composed of a position sensor system 108 ( Figure 2 Generates a union of all or a subset of the coordinates of 462 recorded.

[0050] In some embodiments, a point cloud (e.g., point cloud 460) may include a combination of all or a subset of coordinate points recorded by the position sensor system 108 during an image capture period of multiple shapes, positions, orientations, and / or poses of the elongated device 231 across the anatomical region 350. In these embodiments, the point cloud may include coordinate points captured by the position sensor system 108 representing multiple shapes of the elongated device 231 as it is advanced or moved across the patient's anatomy during the image capture period. Additionally or alternatively, since the configuration (including shape and location) of the elongated device 231 within the patient 103 can change due to anatomical motion during the image capture period, in some embodiments, when the elongated device 231 is passively moved within the patient 103, the point cloud may include multiple coordinate points 462 captured by the position sensor system 108 representing the shape of the elongated device 231. As described in more detail below, the point cloud of coordinate points captured by the position sensor system 108 may be registered to different models or datasets of the patient's anatomy.

[0051] Refer again Figure 2 The endoscopic imaging system 109 of the medical device system 104 includes one or more image capture devices 247, which are configured to capture images when the elongated device 231 and / or the medical device 232 are in the anatomical region of the patient 103 (e.g., Figure 3 Capturing real patient anatomical structures (e.g., within the anatomical region 350) Figure 3 One or more real navigation images of the anatomical access channel 352. For example, the endoscopic imaging system 109 may include an image capturing device 247 located at the distal portion 237 of the medical device 232. In these and other embodiments, the endoscopic imaging system 109 may include one or more image capturing devices (not shown) located at other points along the medical device 232 and / or along the elongated device 231 (e.g., at the distal portion 238 of the elongated device 231).

[0052] exist Figure 3 In the embodiment shown, medical device 232 ( Figure 2The image capturing device 247 is advanced to and located at the distal portion 238 of the elongated device 231. In this embodiment, when the elongated device 231 is navigated through the trachea 354 and bronchus 356 of the anatomical region 350, the image capturing device 247 can map the anatomical passage 352 by capturing a real navigation image of the anatomical passage 352.

[0053] Figure 5 It is via image capture device 247 ( Figure 3 ) captured Figure 3 An example of a real navigation image 570 (e.g., a still image, a video frame, etc.) of the patient's anatomical structures (e.g., one of the anatomical channels 352) in the anatomical region 350. As shown, the real navigation image 570 is displayed from the medical device 232 ( Figure 2 The viewpoint shows the branching points or carina 571 of the two anatomical passages 352 within the anatomical region 350. In this example, because the image capturing device 247 is located at the medical device 232 and the elongated device 231 respectively. Figure 3 The viewpoint of the true navigation image 570 is from the distal portion 237 of the medical device 232, so that the medical device 232 and the elongated device 231 are not visible in the true navigation image 570. In other embodiments, the image capturing device 247 may be located along the medical device 232 and / or along the elongated device 231. Figure 2 and Figure 3 Another point. In these embodiments, the endoscopic imaging system 109 ( Figure 2 A true navigation image can be captured from the corresponding viewpoint of the medical device 232 and / or the elongated device 231. A portion of the medical device 232 and / or the elongated device may be visible within these true navigation images, depending on the positions of the medical device 232 and the elongated device 231 relative to each other.

[0054] Refer again Figure 2 The real navigation images captured by the endoscopic imaging system 109 facilitate the navigation of the distal portion 238 of the elongated device 231 through the patient's anatomy (e.g., through...). Figure 3The anatomical channel 352) and / or delivery of the distal portion of the elongated device 231 to a target location within the patient 103. In these and other embodiments, the real navigation images captured by the endoscopic imaging system 109 can facilitate (i) navigating the distal portion 237 of the medical device 232 beyond the distal portion 238 of the elongated device 231, (ii) delivery of the distal portion 237 of the medical device 232 to a target location within the patient 103, and / or (iii) visualization of the patient's anatomy during medical procedures. In some embodiments, each real navigation image captured by the endoscopic imaging system 109 may be timestamped and / or recorded in a medical device reference frame (X). M Y M Z M The location is associated with the position in the image. As described in more detail below, the real navigation image captured by the endoscope imaging system 109 can therefore be used to improve the point cloud of coordinate points generated by the position sensor system 108 (e.g., Figure 4 The registration between the point cloud (460) and the image data captured by the imaging system 118.

[0055] like Figure 2 As shown, the imaging system 118 is positioned near the patient 103 to obtain images of the patient 103 (e.g., Figure 3 The imaging system 118 provides a three-dimensional image of the anatomical region 350. In some embodiments, the imaging system 118 includes one or more imaging techniques, including CT, MRI, fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar imaging techniques. The imaging system 118 is configured to generate image data of the patient's anatomy before, during, and / or after the elongated device 231 extends within the patient 103. Thus, the imaging system 118 can be configured to capture preoperative, intraoperative, and / or postoperative three-dimensional images of the patient's anatomy. In these and other embodiments, the imaging system 118 can provide real-time or near-real-time images of the patient's anatomy.

[0056] Figure 6 The illustration shows the imaging system 118 ( ) extending within the anatomical region 350 as the elongated device 231 of the medical device system 104 extends. Figure 2 Captured during the image capture period Figure 3 An example of intraoperative image data 680 of a portion 655 of the anatomical region 350. As shown, image data 680 includes a graphic element 682 representing the elongated device 231 and a graphic element 681 representing the anatomical passage 352 of the anatomical region 350.

[0057] All or part of the graphic elements 681 and 682 of image data 680 may be segmented and / or filtered to generate a virtual three-dimensional model of the anatomical passage 352 within a portion 655 of the anatomical region 350 (with or without the medical device system 104). In some embodiments, graphic elements 681 and 682 may additionally or alternatively be segmented and / or filtered to at least partially based on the imaging system 118 when the medical device system 104 is within the anatomical region 350. Figure 2 The captured images are used to generate an image point cloud (not shown) of the medical device system 104. During the segmentation process, pixels or voxels generated from image data 680 can be divided into segments or elements or identified to indicate that they share certain characteristics or calculated properties, such as color, density, intensity, and texture. The segments or elements can then be converted into an anatomical model and / or an image point cloud of the medical device system 104. Additionally or alternatively, segments or elements can be used to locate (e.g., calculate) and / or define a centerline or other points extending along the anatomical channel 352. The generated anatomical model and / or image point cloud can be two-dimensional or three-dimensional and can be displayed in an image reference frame (X). I Y I Z I Generated in ).

[0058] As mentioned above Figure 1 The healthcare system under discussion is 100 ( Figure 1 The display system 110 ( Figure 1 The patient's anatomy and / or various images or representations of the medical device system 104 can be displayed, at least in part, based on data captured and / or generated by the position sensor system 108, the endoscopic imaging system 109, the imaging system 118, and / or the virtual visualization system 115. In various embodiments, the images and / or representations can be used by the system to assist the operator 105. Figure 1 Image-guided medical procedures.

[0059] Figure 7 The display system 110 is based on various embodiments of the present technology. Figure 1 A schematic diagram of an example display 710 generated by [the diagram is missing here]. As shown, display 710 includes a real navigation image 770, a synthesized virtual navigation image 791 (also referred to as "synthetic virtual image 791"), and a virtual navigation image 792. The real navigation image 770 can be compared with [the diagram is missing here]. Figure 5 The actual navigation image 570 is essentially the same. Therefore, for example, the actual navigation image 770 can be generated by the endoscopic imaging system 109 ( Figure 2 ) capture and provide to the display system 110 ( Figure 1 The image is displayed on the display 710 in real-time or near real-time. In the illustrated embodiment, the real navigation image 770 illustrates the view from a distance of 232 from the medical device. Figure 2 The distal portion 237 of the true patient anatomy is viewed from a distal orientation viewpoint (e.g., the carina 771 marks the branching point of the two anatomical channels 352).

[0060] Figure 7 The synthesized virtual image 791 is displayed in the image reference frame (X). I Y I Z I ) and includes from the imaging system 118 ( Figure 2 ) captured Figure 3 The anatomical model 750 is generated from image data of the anatomical region 350. The anatomical model 750 is integrated with the position sensor system 108. Figure 2 The point cloud of coordinate points generated (e.g., Figure 4 The point cloud (460) was registered (i.e., dynamically referenced) to display the patient (103) within the anatomical model (750). Figure 2 Medical device systems 104 within ) (e.g., Figure 2 The elongated device 231) tracks the position, shape, posture, orientation, and / or movement of the representation 704. In some embodiments, the synthesized virtual image 791 is controlled by the control system 112 ( Figure 1 ) virtual visualization system 115 ( Figure 1 Generating a synthetic virtual image 791 involves using an image reference frame (X). I Y I Z I ) and surgical reference system (X S Y S Z S ) and / or medical device reference system (X M Y M Z M Registration. This registration can be performed using the coordinates of points in the point cloud captured by the position sensor system 108 (e.g., ...). Figure 4 Rigid and / or non-rigid transformations of the coordinate points 462 of the point cloud 460 are used to rotate, translate, or otherwise manipulate them to align the coordinate points with the anatomical model 750. Registration between the image and the surgical / instrument reference frame can be achieved, for example, by using point-based Iterative Closest Point (ICP) techniques, such as U.S. Provisional Patent Applications Nos. 62 / 205,440 and 62 / 205,433, the entire contents of which are incorporated herein by reference. In other embodiments, another point cloud registration technique may be used to achieve registration.

[0061] Based at least in part on registration, the virtual visualization system 115 can additionally or alternatively generate virtual navigation images (e.g., Figure 7 The virtual navigation image 792 includes a medical device system 104 located within the anatomical model 750. Figure 3 The virtual depiction of the patient's anatomy is shown from the viewpoint of a virtual camera on a 704-inch screen. Figure 7 In the illustrated embodiment, the virtual camera of the virtual navigation image 792 is located at the distal portion 737 of representation 704, such that (i) the virtual viewpoint of the virtual navigation image 792 is oriented distally away from the distal portion 737 of representation 704, and (ii) representation 704 is not visible within the virtual navigation image 792. In other embodiments, the virtual visualization system 115 may position the virtual camera at (a) another point along representation 704 and / or (b) in different orientations such that the virtual navigation image 792 has a corresponding virtual viewpoint. In some embodiments, based on the position and orientation of the virtual camera and the positions of the elongated device 231 and / or medical device 232 relative to each other within the patient 103, the virtual visualization system 115 may render a virtual representation (not shown) of at least a portion of the elongated device 231 and / or medical device 232 into the virtual navigation image 792.

[0062] In some embodiments, the virtual visualization system 115 can be used in conjunction with the image capture device 247 on the patient 103 ( Figure 2 The virtual camera is placed within the anatomical model 750, with its position and orientation corresponding to the location and orientation within the model. For example... Figure 7 As further shown, the virtual navigation image 792 is captured by the image capture device 247. Figure 2 The points captured in the real navigation image 770 are substantially the same as those in the virtual patient's anatomical structures, such as the carina 701, which marks the branching points of the two anatomical passages 752 of the anatomical model 750. Therefore, the virtual navigation image 792 in... Figure 3 The virtual navigation image 792 provides a rendering estimate of the patient's anatomical structures visible to the image capture device 247 at a given point within the anatomical region 350. Because the virtual navigation image 792 is at least partially based on the registration of point clouds generated by the position sensor system 108 and image data captured by the imaging system 118, the correspondence between the virtual navigation image 792 and the real navigation image 770 provides insight into the accuracy of the registration and can be used to improve the registration, as described in more detail below. Furthermore, the endoscopic imaging system 109 ( Figure 2 The captured real navigation image (e.g., real navigation image 770) can (a) provide information about the medical device system 104 within the patient 103. Figure 1 (a) information on the location and orientation of the anatomical region actually accessed by the medical device system, and / or (c) information on the portion of the anatomical region actually accessed by the medical device system 104 (e.g., branching points of anatomical passages), any one or more of which may be used to improve the accuracy of registration, as described in more detail below.

[0063] like Figure 7 As further shown, the virtual navigation image 792 may optionally include a navigation path overlay 799. In some embodiments, the navigation path overlay 799 is used to assist the operator 105 ( Figure 1 Navigation Medical Device System 104 Figure 1 The navigation path overlay 799 can be used to navigate through anatomical pathways across the anatomical region to target locations within the patient 103. For example, the navigation path overlay 799 can illustrate an "optimal" path through the anatomical region for the operator 105 to follow in order to deliver the distal portions 237 and / or 238 of the medical device 232 and / or elongated device 231 to the target locations within the patient 103, respectively. In some embodiments, the navigation path overlay 799 can be aligned with the centerline of the corresponding anatomical pathway or with another line along the bottom of the corresponding anatomical pathway (e.g., the bottom of the pathway).

[0064] 2. Related methods

[0065] Figure 8 This is a flowchart illustrating a method 800 for registering images of patient anatomy to a point cloud of coordinate points using endoscopic image data, according to various embodiments of the present technology. Method 800 is illustrated as a set of steps or processes 801-808, and is referred to hereinafter. Figure 7 and Figures 9-13C At least partially described. All or a subset of the steps of method 800 can be performed by various components or devices of a robot or remote operating system, such as... Figure 1 The system 100 shown or other suitable systems. For example, all or a subset of the steps of method 800 may be performed by components or devices of (i) the manipulator assembly 102, (ii) the medical device system 104, (iii) the master control assembly 106, (iv) the position sensor system 108, (v) the endoscopic imaging system 109, (vi) the display system 110, (vii) the control system 112, (viii) the virtual visualization system 115, and / or (ix) the imaging system 118. Additionally or alternatively, all or a subset of the steps of method 800 may be performed by an operator of system 100 (e.g., a physician, user, etc.). Furthermore, any one or more of the steps of method 800 may be performed in accordance with the discussion above.

[0066] At step 801, method 800 records position sensor data from the medical device system. In some embodiments, a position sensor system (e.g., Figure 1 and Figure 2The position sensor system 108) records position sensor data. Position sensor data may be recorded during a data acquisition period of the position sensor system. A data acquisition period may correspond to a time interval during which the shape sensor and / or one or more other position sensors of the position sensor system are activated to collect and record position sensor data. During the data acquisition period, the medical device system may be stationary, subject to commanded movement (e.g., operator-commanded propulsion or bending), and / or may be subject to passive movement (e.g., uncommanded movement but subject to anatomical movements arising from respiratory activity, cardiac activity, or other voluntary or involuntary patient movements).

[0067] As discussed in more detail above, when at least a portion of the medical device system is located within the patient's body, position sensor data provides positional information (shape, location, orientation, posture, movement, etc.) of the medical device system. For example, position sensor data may include shape data. In these and other embodiments, position sensor data may include elongated devices associated with the medical device system (e.g., Figure 1 and Figure 2 231) and / or medical devices (e.g., Figure 2 The location information associated with the distal end of the medical device 232 and / or along the elongated device and / or other points of the medical device. In some embodiments, the location sensor data may be recorded at least in part as a medical device reference frame (X). M Y M Z M One or more coordinate points in a two-dimensional or three-dimensional space, the medical device reference frame (X). M Y M Z M Surgical reference frame (X) relative to the surgical environment S Y S Z S It is known that ). In these and other embodiments, each coordinate point may be associated with a timestamp, which may be recorded as part of the location sensor data.

[0068] At step 802, method 800 generates a point cloud from the recorded position sensor data. In some embodiments, the point cloud is generated from a combination of all coordinate points or subsets of coordinate points recorded during the data acquisition period of the position sensor system at step 801. In these and other embodiments, the point cloud represents one or more shapes of the medical device system when it is stationary and / or actively or passively moving within a patient. The point cloud can be displayed in a medical device reference frame (X). M Y M Z M It can be generated in two or three dimensions.

[0069] At step 803, method 800 captures endoscopic image data of the patient's anatomy. In some embodiments, an endoscopic imaging system (e.g., Figure 1 and Figure 2 The medical device system 109 is used to capture endoscopic image data. Endoscopic image data can be captured during an image capture period of the endoscopic imaging system. The image capture period can correspond to the time during which at least one image capture device of the endoscopic imaging system 109 is activated to collect and record endoscopic image data. During the image capture period, the medical device system can be stationary, can be subject to commanded movement (e.g., operator-commanded propulsion or bending), and / or can be subject to passive movement (e.g., uncommanded movement but subject to anatomical movements from respiratory activity, cardiac activity, or other voluntary or involuntary patient movements).

[0070] As discussed in more detail above, endoscopic image data captures one or more images (e.g., still images, video, etc.) from the viewpoint of the medical device system. For example, the image capture device of the endoscopic imaging system can be mounted at a distal end of the medical device system (e.g., mounted to...). Figure 2 The distal portion 238 of the elongated device 231 and / or the distal portion 237 of the medical device 232. Furthermore, the image capturing device can be oriented such that the field of view of the image capturing device is substantially parallel to the axis defined by at least the distal portion of the medical device system and remote from the projection of the medical device system. In these embodiments, endoscopic image data can include one or more images of an object in front of (e.g., further away) the distal end of the medical device system. Thus, continuing this example, when the distal end of the medical device system is located within the anatomical region of the patient, the endoscopic image data can include one or more true navigation images of the patient's anatomy in front of (e.g., further away) the distal end of the medical device system. Other mounting locations and / or other orientations of the image capturing device of the endoscopic imaging system are, of course, possible and within the scope of this art. In some embodiments, each true navigation image of the endoscopic image data is associated with a timestamp that may be recorded as part of the endoscopic image data. Additionally or alternatively, when the image capturing device captures true navigation images, the position of the image capturing device can be known and in a medical device reference frame (X). M Y M Z M This is a portion of the data recorded as endoscopic image data.

[0071] At step 804, method 800 captures, receives, and / or processes image data of the patient and generates an anatomical model. In some embodiments, an imaging system (e.g., Figure 1 and Figure 2An imaging system (118) is used to capture image data. For example, a CT imaging system can be used to capture image data. Image data can be captured, received, and / or processed during the image capture period of the imaging system. The image capture period may correspond to the time period during which the imaging system is activated. In some embodiments, the image capture period may be preoperative, such that image data is captured, received, and / or processed before the medical device system is advanced into the patient. In these and other embodiments, the image capture period may be intraoperative, such that image data of the patient is captured, received, and / or processed while the medical device system is in the patient's body. In these embodiments, the medical device system may be stationary during the image capture period, may be subject to commanded movement (e.g., operator-commanded advancement or bending) during the image capture period, and / or may be passively moved during the image capture period (e.g., uncommanded movement but subject to anatomical movement from respiratory activity, cardiac activity, or other voluntary or involuntary patient movement). In yet another embodiment, the image capture period may be postoperative, such that image data of the patient is captured, received, and / or processed after the medical device system is removed from the patient. In some embodiments, image data can be captured, received, and / or processed in real-time or near real-time.

[0072] As discussed in more detail above, patient image data includes graphical elements representing the patient's anatomical features and (in the case of intraoperative image data) graphical elements representing the medical device system. A model of the patient's anatomical features is generated by segmenting and filtering the graphical elements included in the image data. During the segmentation process, pixels or voxels generated from the image data can be segmented into segments or elements and / or identified to indicate that they share certain characteristics or computed properties, such as color, density, intensity, and texture. In some embodiments, segmentation and filtering may be performed on less than all of the image data. The segments or elements associated with the patient's anatomical features are then converted into an anatomical model, which is displayed in an image reference frame (X). I Y I Z I Generated in ).

[0073] At step 805, method 800 generates one or more correspondences between endoscopic image data of the patient's anatomy captured at step 803 and patient image data captured, received, and / or processed at step 804, and / or updates the point cloud generated at step 802 based at least in part on one or more of these correspondences. For example, as discussed above, the image capture device of the endoscopic imaging system may be mounted to the distal portion of the medical device system and located within the patient's anatomical region. In these embodiments, the endoscopic image data captured at step 803 includes (i) an image of the actual patient anatomy near the distal end of the medical device system and (ii) an indication of the location of the distal portion of the medical device within the anatomical passage actually accessed by the medical device system. Therefore, when method 800 determines that the true navigation image of the patient's anatomy in the endoscopic image data captured at step 803 (e.g., a carina, which marks the branching points of two or more anatomical passages) partially matches the patient's image data captured, received, and / or processed at step 804, method 800 can generate a correspondence between the endoscopic image data of step 803 and the image data of step 804. This is because the true navigation image matching the patient's anatomy in the endoscopic image data is linked to the timestamp and image capture device in the medical device reference frame (X). M Y M Z M The known location within the medical device reference frame (X) is associated with the endoscopic image data in step 803 and the image data in step 804, so the correspondence generated provides the known location of the image capturing device in the medical device reference frame (X). M Y M Z M ) and image reference frame (X I Y I Z I The method 800 updates the point cloud generated in step 802 based at least in part on the generated correspondences between the known correspondences. For example, when the image capturing device captures a real navigation image of endoscopic image data that matches the image data of step 804, the method 800 may place the medical device reference frame (X) at and / or near a known location of the image capturing device. M Y M Z M One or more coordinate points from the coordinates are added to the point cloud in step 802.

[0074] At step 806, method 800 registers the point cloud generated at step 802 and / or updated at step 805 to the anatomical model generated at step 804. In some embodiments, registration involves aligning the medical device reference frame (X). M Y M ZM ) and / or surgical reference system (X S Y S Z S ) and image reference frame (X I Y I Z I Alignment. For example, medical device reference frame (X). M Y M Z M The point cloud obtained in steps 802 and / or 805 of the image can be registered to the image reference frame (X). I Y I Z I The registration can be performed by rotating, translating, or otherwise manipulating the coordinate points of the point cloud (e.g., coordinate points generated from position sensor data at steps 801 and 802 and / or added coordinate points generated at step 805 from the correspondence between the real navigation image in the endoscopic image data of step 803 and the image data of step 804) using rigid and / or non-rigid transformations to align the coordinate points with the anatomical model generated at 804. The transformation can be a six-degree-of-freedom transformation, allowing the point cloud to be translated or rotated on any or all of the X, Y, Z, pitch, roll, and yaw axes. In some embodiments, method 800 uses an Iterative Closest Point (ICP) algorithm to perform the registration. Specifically, method 800 can (i) calculate point-to-point correspondences between coordinate points in the point cloud and points within the anatomical model (e.g., on the centerline or at other locations), and (ii) calculate an optimal transformation to minimize the Euclidean distance between the corresponding points. In other embodiments, method 800 may use an alternative technique to perform the registration.

[0075] In some embodiments, method 800 may use the endoscopic image data captured at step 803 to improve the accuracy of the registration between the point cloud generated at step 802 and / or updated at step 805 and the anatomical model generated at step 804 and / or otherwise provide insight into the registration. For example, as discussed above with respect to step 805, method 800 may add one or more coordinate points at known locations of the image capture device where the patient anatomy in the real navigation image of the endoscopic image data from step 803 matches the patient anatomy captured in the image data from step 804. In some embodiments, the added coordinate points may be used in the ICP algorithm together with coordinate points generated from the position sensor data from steps 801 and / or 802 to compute the optimal transformation. In these and other embodiments, the added coordinate points may be weighted differently (e.g., more heavily or less heavily) in the computation compared to the coordinate points generated from the position sensor data from step 801. In these and other embodiments, the orientation alignment data captured by the correspondence at step 805 (e.g., information on how the patient's anatomy in the matching real navigation image of the endoscopic image data of step 803 must be transformed (e.g., translated, rotated, reflected, etc.) to be aligned with the corresponding portion of the patient's anatomy in the image data of step 804) can be fed as an additional error term minimized by the registration algorithm to further inform the registration between the point cloud and the image data of step 804.

[0076] In these and other embodiments, method 800 may use the endoscopic image data captured at step 803 to temporarily or partially improve the accuracy of the registration performed at step 806 and / or otherwise provide insight into the registration performed at step 806. For example, method 800 may use the coordinate points added at step 805 and / or the orientation alignment data captured through the correspondence at step 805 to improve the accuracy of only a portion of the registration performed at step 806 and / or otherwise provide insight into only a portion of the registration performed at step 806. Continuing this example, this portion of the registration performed at step 806 may correspond to a subset of coordinate points from steps 802 and / or 805 and / or points of the anatomical model generated at step 804, within a threshold distance of the coordinate points added at step 805 and / or within a threshold distance of the correspondence generated at step 805.

[0077] Alternatively, method 800 may perform registration (e.g., sparse point registration) only between (a) the coordinates of the endoscopic image data derived from step 803 and (b) the anatomical model generated at step 804. For example, Figure 9 Yes(i) Figure 3The illustrated patient's anatomical region 350 (e.g., lung) and (ii) are schematic diagrams of real navigation images 910-912 of the patient's anatomical structures within the anatomical region 350 captured by the image capture device 247 of the medical device system 104 as endoscopic image data at step 803. As shown, real navigation images 910-912 are images of branch points 920-922 of the anatomical passage 352 captured by the medical device system 104 as it navigates throughout the anatomical region 350. Branch points 920-922 of the anatomical region 350 are easily identifiable anatomical features in the real navigation images of the endoscopic image data at step 803 because each branch point 920-922 includes a bright ridge 915 of a protuberance in the central region of each real navigation image 910-912 and two or more openings (e.g., openings 916 and 917) of the anatomical passage 352. Therefore, in some embodiments, method 800 may (e.g., automatically) identify branch points 920-922 in the real navigation images 910-912, and / or other branch points in other real navigation images (not shown) of the endoscopic image data from step 803, and may record one or more coordinate points in the point cloud at points corresponding to the points of image capture device 247 when image capture device 247 captures each of the corresponding real navigation images 910-912. The point cloud may be a point cloud generated at step 802 and / or updated at step 805, and / or another point cloud. (Continue to refer to...) Figure 9 In the above example, method 800 may perform sparse point registration between (i) the anatomical model generated at step 804 and (ii) the coordinates of other real navigation images (not shown) in the endoscopic image data of step 803 in which the branch points of the anatomical passage 352 have been identified by method 800, derived only from real navigation images 910-912 and / or other real navigation images (not shown) in which method 800 has identified the branch points of the anatomical passage 352.

[0078] In these and other embodiments, method 800 may use the real navigation image of the endoscopic image data from step 803 to provide insight into the pathway taken by the medical device system as it navigates throughout the anatomical region. For example, after method 800 identifies a branch point in the real navigation image of the endoscopic image data from step 803, method 800 may use the real navigation image and / or one or more previously and / or subsequently captured real navigation images in the endoscopic image data to determine which anatomical pathway the medical device system took at the branch point as it navigated throughout the anatomical region.

[0079] For a more specific example, please refer to [link / reference]. Figure 9After method 800 identifies branch point 920 in real navigation image 910, method 800 can use real navigation image 910 and one or more real navigation images captured previously and / or subsequently in the endoscopic image data of step 803 to determine whether medical device system 104 travels through opening 916 or through opening 917. In this case, method 800 can determine that medical device system 104 travels through opening 917 of right anatomical passage 352. In other words, the endoscopic image data of step 803 can be used to estimate the specific path taken by medical device system 104 throughout anatomical region 350. Then, method 800 can use this information to instruct the ICP algorithm to register the data points of the point cloud (e.g., the point cloud of sparse points, the point cloud of step 802, and / or the point cloud of step 805) to a specific region of the anatomical model generated at step 804 (e.g., to the region corresponding to right anatomical passage 352 in real navigation image 910). Therefore, compared with naive ( Compared to the ICP algorithm, this information can be used to improve registration accuracy. The nascent ICP algorithm will register the data points of the point cloud to the nearest region of the anatomical model (e.g., the region corresponding to the left anatomical channel 352 of the real navigation image 910) in a different way, regardless of whether another less nearby region of the anatomical model (e.g., the region corresponding to the right anatomical channel 352 of the real navigation image 910) actually corresponds to a data point in the point cloud. This is expected to be particularly helpful in improving registration when the two regions are closely spaced from each other in the anatomical model.

[0080] When the medical device reference frame (X) is used M Y M Z M Registered to the image reference frame (X) I Y I Z I When an image is displayed to the operator on a display system, the operator can manipulate the medical device system more accurately through the patient's anatomy, observe the patient's anatomy from a distal perspective of the medical device system, and / or improve the efficiency and effectiveness of the target medical procedure. For example, in some embodiments, method 800 may display synthetic virtual images (e.g., Figure 7 The synthesized virtual image 791 includes an anatomical model generated at step 804, accompanied by a representation of a medical device system having position, shape, orientation, posture, and / or movement (e.g., velocity, rate, etc.) within the anatomical model, the position, shape, orientation, posture, and / or movement corresponding to the position, shape, orientation, posture, and / or movement of the medical device system within the patient's body. For example, the representation of the medical device system may be superimposed on the anatomical model.

[0081] In these and other embodiments, at least in part based on the performed registration, method 800 can calculate real-time and / or near-real-time virtual navigation images at locations within an anatomical model corresponding to locations of the image capture device of the medical device system within the patient (e.g., Figure 7 The virtual navigation image 792). For example, method 800 may calculate a virtual navigation image corresponding to the real navigation image in the endoscopic image data of step 803. In some embodiments, method 800 may select which of the real navigation images in the endoscopic image data to calculate the corresponding virtual navigation image. As discussed above, branch points of anatomical channels are patient anatomical structures that are easily identifiable in the real navigation images. Therefore, in some embodiments, method 800 may select only those real navigation images in which method 800 identifies branch points or other identifiable patient anatomical structures for which the corresponding virtual navigation image is calculated. In other embodiments, method 800 may use other selection criteria. Method 800 may display the calculated virtual navigation image and / or the real navigation image of the endoscopic image data of step 803 (e.g., ...) on a display of a display system. Figure 7 (Real navigation image 770).

[0082] At step 807, method 800 estimates and / or displays the registration error of the registration performed at step 806. For example, method 800 may calculate the inconsistency between (i) the known location of the image capture device associated with the actual navigation image of the endoscopic image data of step 803, which matches the image data of step 804, and (ii) the estimated location of the image capture device within the registration generated at step 806. After calculating the inconsistency, method 800 may display the estimated registration error on a display of the display system.

[0083] To clarify and understand the above concepts, please refer to... Figure 7 and Figure 9 Consider the following additional example. After registration is performed at step 806, method 800 can display the synthesized virtual image 791. Figure 7 The illustration shows the anatomical model 750 generated at step 804. Figure 7 ), accompanying medical device system 104 ( Figure 9 The representation of 704 () Figure 7 Then, method 800 can calculate the estimated registration error for the following parts: (i) the anatomical channel 752 of the synthesized virtual image 791 ( Figure 7 ) part 757 ( Figure 7 ), this part 757 ( Figure 7 ) corresponds to anatomical region 350 ( Figure 9 Anatomical passage 352 ( Figure 9 ) part 957 ( Figure 9 (ii) The anatomical passage 752 portion 758 of the synthesized virtual image 791 Figure 7 This part 758 ( Figure 7 ) corresponds to the portion 958 of the anatomical passage 352 of anatomical region 350. Figure 9 In this example, method 800 can display the estimated registration error by changing the color, pattern, and / or other visual indicators (e.g., digital display) of portions 757 and 758 within the synthetic virtual image 791, based on the magnitude of the corresponding registration error. For example, method 800 can color portion 757 of the anatomical model 750 in the synthetic virtual image 791 green to indicate that the estimated registration error at that portion 757 of the synthetic virtual image 791 is relatively small (e.g., indicating that the registration of the point cloud at that point with the image data of step 804 is well aligned with the correspondence between the portion of the endoscopic image data of step 803 and the image data of step 804 at that point). Conversely, method 800 may color a portion 758 of the anatomical model 750 in the synthesized virtual image 791 with different (e.g., darker, less intense, less bright) shades of green or different colors (e.g., yellow, orange, red, etc.), patterns, and / or visual indicators to indicate that the estimated registration error at portion 758 is relatively large (e.g., indicating that the registration of the point cloud with the image data of step 804 is not well aligned with the correspondence between the portion of the endoscopic image data of step 803 and the image data of step 804 at that point). In this way, method 800 may display color gradients, patterns, and / or other visual indicators (e.g., numerical displays) within the synthesized virtual image 791 to indicate the estimated registration error across the anatomical model 750. This can be useful, for example, for determining the optimal path through the patient's anatomy to a target point and / or for determining whether the current patient anatomy is aligned with the patient's preoperative imaging at the portion of interest in the anatomical model 750.

[0084] In these and other embodiments, method 800 can estimate and / or display registration errors in real time or near real time. For example, method 800 can estimate registration errors in real time or near real time for the current location of the image capture device of a medical device system within a patient. In this example, method 800 can calculate, at or near the current location of the image capture device, (i) the position of the image capture device associated with the actual navigation image of the endoscopic image data of step 803 that matches the image data of step 804, and (ii) the inconsistency between the estimated position of the image capture device within the registration performed in step 806.

[0085] After calculating the inconsistency, method 800 can display the estimated registration error on the display system's monitor in real time or near real time. For illustrative purposes, refer again... Figure 7 Method 800 can change the color, pattern, and / or other visual indicators of a portion 757 of the anatomical model 750 within the synthetic virtual image 791 at or near the current location of the image capture device (e.g., at or near the current location of the distal portion 737 of the representation 704 of the medical device system). Thus, during the period in which the patient breathes, a series of colors, shading, patterns, and / or other visual indicators can be used to display portion 757 to indicate changes in the magnitude of the estimated registration error during that period. In other words, method 800 can provide a temporary indication of when the registration of the point cloud at a given location with the image data of step 804 is well aligned with the correspondence between the portion of the endoscopic image data of step 803 and the image data of step 804 at that given location. This information can be useful, for example, for providing a temporary indication of where to gating the patient's breathing phase and performing a biopsy on target tissue during breath-holding.

[0086] In these and other embodiments, method 800 may alter the color, pattern, and / or other visual indicators of other information on the display to indicate the estimated registration error in real time, near real time, or otherwise. For example, method 800 may change the way virtual navigation images are displayed (e.g., Figure 7 The virtual patient anatomy in the virtual navigation image 792) and / or the navigation path overlay used to display the virtual navigation image (e.g., Figure 7 The navigation path is overlaid with colors, patterns and / or other visual indicators (799).

[0087] At step 808, method 800 updates the registration performed at step 806. In some embodiments, method 800 may update the registration by returning to step 801 and re-executing (e.g., iteratively) all or a subset of steps 801-807. In these and other embodiments, method 800 may update the registration performed at step 806 using the endoscopic image data captured at step 803. For example, method 800 may use one or more real navigation images from the endoscopic image data of step 803 to align the calculated virtual navigation image with the corresponding real navigation image from the endoscopic image data. For clarity and understanding, refer to Figures 10-11B Consider the following example. Figure 10This is a real navigation image 1030 capturing the actual patient anatomy in the endoscopic image data of step 803. The actual patient anatomy in the real navigation image 1030 includes the carina 1015, which marks the branching point of the two anatomical channels 352. The openings 1016 and 1017 of the anatomical channels 352 are visible in the real navigation image 1030.

[0088] In some embodiments, method 800 may compute a virtual navigation image at a location corresponding to the location of the image capture device when the image capture device captures the real navigation image 1030, at least in part based on the registration performed in step 806. For example, Figure 11A In the context of Figure 10 The virtual navigation image 1140, at least partially based on the virtual patient anatomy calculated by method 800, is constructed at locations corresponding to the points of the image capture device in the real navigation image 1030. The virtual patient anatomy in the virtual navigation image 1140 includes a ridge 1115, which marks the branch points of two virtual anatomical channels 1152. The openings 1116 and 1117 of the anatomical channels 1152 are visible in the virtual navigation image 1140. The branch points of the virtual patient anatomy in the virtual navigation image 1140 correspond to… Figure 10 Branch points of real patient anatomy in real navigation image 1030.

[0089] Let's refer to each other. Figure 10 and Figure 11A Method 800 can determine the virtual navigation image 1140 calculated in the calculation. Figure 11A The virtual patient anatomy and the real navigation image 1030 ( Figure 10 The actual patient anatomy in the virtual navigation image 1140 is misaligned. In other words, method 800 can determine that the registration performed in step 806 at the location of the image capture device associated with the actual navigation image 1030 is misaligned with the endoscopic image data of step 803. In these embodiments, method 800 can calculate a transformation to align the virtual navigation image 1140 with the actual navigation image 1030. For example, method 800 can determine that method 800 must translate forward the position of the virtual image capture device associated with the virtual navigation image 1140 and rotate it slightly counterclockwise to align the virtual navigation image 1140 with the actual navigation image 1030. Figure 11B After performing this transformation Figure 11AThe virtual navigation image 1141 shows the virtual patient's anatomical structure. In some embodiments, the calculated transformation can be used as an incremental (delta) registration matrix to update the registration performed at step 806 (e.g., ICP registration). This may involve changing the position of the coordinate points generated from the position sensor data of step 801 and recorded in the point cloud of steps 802 and / or 805 from the position corresponding to the virtual navigation image 1140. Figure 11A The location of the virtual image capture device associated with the virtual image is changed to correspond to the virtual navigation image 1141. Figure 11B The location of the associated virtual image capture device.

[0090] In these and other embodiments, method 800 (in step 808) can update the registration performed at step 806 by correcting for drift in the endoscopic image data distant from step 803. For clarity and understanding of this concept, refer to... Figures 12-13C Consider the following example. For example, Figure 12 It is a virtual navigation image 1250 of the virtual patient's anatomical structure 1271, and Figures 13A-13C The illustration shows a sequence of consecutive real navigation images 1360-1362 of the real patient anatomy 1371 captured in the endoscopic image data of step 803. In this example, method 800(i) identifies real navigation image 1360 ( Figure 13A The real patient anatomy 1371 in the image includes the branching points of two anatomical channels 352 and (ii) in the real navigation image 1360. Figure 13A The virtual navigation image 1250 is calculated at least in part based on the registration performed at step 806 at the points in the generated anatomical model (not shown) corresponding to the points of the image capture device in the image capture device. Figure 12 ).

[0091] Figure 12 The virtual patient anatomy 1271 in the virtual navigation image 1250 and Figure 13A The real patient anatomy 1371 in the real navigation image 1360 is not perfectly aligned. As mentioned above, each virtual navigation image (including Figure 12 The virtual navigation image 1250) and each real navigation image (including Figures 13A-13CThe real navigation images 1360-1362 are associated with timestamps indicating the time points at which the corresponding data portions (e.g., position sensor data from step 801 and / or endoscopic image data from step 803) were captured. Therefore, method 800 can search for real navigation images (including real navigation images 1360-1362) of the endoscopic image data from step 802 captured within a time period (e.g., timestamps indicating before, during, and / or after) surrounding the timestamps associated with the virtual navigation image 1250 to find a real navigation image that best matches the virtual navigation image 1250. In this example, Figure 13B Real navigation image 1361 and Figure 12 The virtual navigation image 1250 is optimally matched. Then, method 800 can calculate the difference between the timestamp of the optimally matched real navigation image 1361 and the timestamp of the virtual navigation image 1250, and use this difference as an incremental registration matrix to update the registration performed in step 806 (e.g., ICP registration). For example, method 800 can... Figure 12 The positions of one or more coordinate points in the point cloud of step 802 and / or step 805 corresponding to the virtual navigation image 1250 are changed to the position of the image capture device associated with the best-matching real navigation image 1361 of Figure 13.

[0092] While the above concepts have been illustrated and discussed in the context of matching the branching points of two anatomical pathways in a virtual navigation image with corresponding patient anatomy in a real navigation image, these concepts are particularly useful for points where the branching points are not visible in either the virtual or real navigation images. For example, the diameter of an anatomical pathway typically decreases as the medical device system navigates further. Therefore, the above concepts can be used to determine a real navigation image illustrating an anatomical pathway with a diameter that best matches the diameter of the anatomical pathway in the virtual navigation image. Thus, the best match can provide information about how far the medical device system inserts the anatomical pathway at a given point in time.

[0093] In some embodiments, method 800 temporarily or partially updates the registration performed at step 806 at step 808. For example, method 800 may update the registration performed at step 806 for a specific respiratory or cardiac stage. Continuing this example, method 800 may update the registration performed at step 806 differently for different respiratory or cardiac stages. As another example, method 800 may update only a portion of the registration performed at step 806 at step 808. Continuing this example, the updated portion of the registration may correspond to a subset of the coordinate points from steps 802 and / or 805 and / or the points of the anatomical model generated at step 804, within a threshold distance to coordinate points and / or anatomical model points corresponding to one or more real and / or virtual navigation images.

[0094] Some calculations performed in steps 801-808 above (e.g., matching between real and virtual navigation images) can be particularly resource-intensive. Therefore, as an extension of any or more of steps 801-808 above, method 800 can use the endoscopic image data captured at step 802 and / or other information available to method 800 to determine when to perform certain calculations of method 800. In some embodiments, method 800 can use input / output values ​​of the medical device system to identify when to perform registration calculations. For example, method 800 can use the distance traveled by the distal end of the medical device as an indicator of when to perform the calculation. As a more specific example, method 800 can anticipate that the patient's principal carina is located approximately a first distance distal to the medical device system at the initial insertion point. Therefore, method 800 can monitor and identify when the distal end of the medical device system has traveled the first distance from the initial insertion point to determine when to attempt to capture the principal carina in the endoscopic image data and / or when to attempt to generate a correspondence between the real navigation image of the endoscopic image data and (e.g., preoperative) image data of the patient's principal carina. Additionally or alternatively, method 800 may use the motion of the position sensor system and / or the registration performed in step 806 to estimate when the image capture device of the endoscopic imaging system may be close to the carina and may use the estimation to determine the correspondence between the actual navigation image that attempts to generate endoscopic image data captured in step 802 and the patient's (e.g., preoperative and / or intraoperative) image data captured, received and / or processed in step 804.

[0095] In these and other embodiments, method 800 may use the occurrence of other events to determine when to perform a calculation. For example, method 800 may perform a specific calculation each time the distal end or another part of the medical device system travels a threshold distance (e.g., each time the position of the distal end changes by a threshold amount). As another example, method 800 may perform a specific calculation after the orientation of the distal end of the medical device system has changed by a threshold amount. As yet another example, method 800 may periodically (e.g., according to set intervals and / or events) capture position sensor data and / or endoscopic image data and may wait to perform resource-intensive calculations until method 800 determines that the medical device system has undergone movement under a command (e.g., by an operator) and / or until another event occurs.

[0096] Although the steps of Method 800 are discussed and illustrated in a specific order, Figure 8The method 800 illustrated is not limited thereto. In other embodiments, method 800 may be performed in a different order. For example, step 804 may be performed before any of steps 801-803. In these and other embodiments, any step of method 800 may be performed before, during, and / or after any other step of method 800. Furthermore, those skilled in the art will recognize that the illustrated method 800 can be modified while still remaining in these and other embodiments of the present technology. In some embodiments, for example, Figure 8 One or more steps of the method 800 shown may be omitted and / or repeated.

[0097] B. Example

[0098] Several aspects of this technology are set forth in the following embodiments. Although several aspects of this technology are set forth in examples relating to systems, computer-readable media, and methods, any of these aspects of this technology may be similarly set forth in examples relating to any systems, computer-readable media, and methods in other embodiments.

[0099] 1. A medical device system for use in an image-guided medical procedure, the system comprising:

[0100] A position sensor configured to generate position sensor data associated with one or more locations of a biomedical device within the patient's anatomical region;

[0101] An image capture device configured to capture first image data of patient anatomical structures within the anatomical region when the biomedical device is located within the anatomical region;

[0102] A processor communicatively coupled to the position sensor and the image capture device; and

[0103] A memory for storing instructions that, when executed by the processor, cause the system to perform operations including:

[0104] A point cloud of coordinate points is generated, at least in part, based on the location sensor data.

[0105] Receive second image data of the anatomical region, wherein the second image data is generated at least in part based on imaging of the anatomical region.

[0106] Registration is generated between at least a portion of the point cloud and at least a portion of the second image data, and

[0107] The registration is updated at least in part based on the first image data.

[0108] 2. The system according to Example 1, wherein the operation further includes generating one or more correspondences by matching patient anatomical structures in one or more images of the first image data with patient anatomical structures in the anatomical region of the portion of the second image data.

[0109] 3. The system according to Example 2, wherein the patient anatomical structure in one or more images of the first image data and the patient anatomical structure in the anatomical region of the portion of the second image data are one or more branch points of anatomical channels in the anatomical region.

[0110] 4. The system according to Example 2 or Example 3, wherein the operation further includes adding one or more coordinate points to the point cloud at one or more locations, the one or more locations corresponding to one or more locations of the image capturing device within the anatomical region associated with the one or more images of the first image data.

[0111] 5. The system according to Example 4, wherein generating the registration includes weighting the one or more added coordinate points differently from other coordinate points of the point cloud generated from the position sensor data.

[0112] 6. The system according to Example 4 or Example 5, wherein the portion of the point cloud includes only the one or more added coordinate points.

[0113] 7. The system according to any one of Examples 2-6, wherein the operation further includes determining a transformation for aligning an image in one or more images of the first image data with a corresponding patient anatomical structure in the anatomical region of the portion of the second image data, and wherein generating the registration includes generating the registration at least in part based on the transformation.

[0114] 8. The system according to any one of Examples 2-7, wherein the operation further comprises determining at least a portion of a pathway taken by the biomedical device throughout the anatomical region based at least in part on the first image data, and wherein generating the registration comprises generating the registration between at least the portion of the point cloud and a segment of the anatomical region corresponding to the portion of the pathway.

[0115] 9. The system according to any one of Examples 2-8, wherein the operation further includes estimating the registration error between the correspondence in the one or more correspondences and the generated registration.

[0116] 10. The system according to Example 9, wherein the operation further includes coloring the display of the generated registration based at least in part on the magnitude of the estimated registration error.

[0117] 11. The system according to Example 10, wherein the operation further includes:

[0118] Real-time estimation of the registration error at the current location of the biomedical device within the anatomical region; and

[0119] The corresponding part of the display is colored.

[0120] 12. The system according to any one of Examples 1-11, wherein the operation further comprises:

[0121] Based at least in part on the generated registration, a virtual image of the patient's anatomical structure within the anatomical region is calculated from the perspective of the image capture device at the current position of the image capture device within the anatomical region; and

[0122] Determine a transformation for aligning the virtual image with the first image data corresponding to the current position of the image capture device.

[0123] 13. The system according to Example 12, wherein updating the registration includes updating the registration at least in part based on the determined transformation.

[0124] 14. The system according to Example 12 or Example 13, wherein determining the transformation includes:

[0125] The transformation is determined only for a portion of the generated registration within a threshold distance from the current point of the image capture device; or

[0126] The transformation is determined for the specific respiratory and / or cardiac phase of the patient.

[0127] 15. The system according to any one of Examples 1-14, wherein the operation further comprises:

[0128] Based at least in part on the generated registration, a virtual image of the patient's anatomical structure within the anatomical region is calculated from the viewpoint of the image capturing device at the current or previous location of the image capturing device within the anatomical region, wherein the virtual image is associated with a first timestamp; and

[0129] The first image data that best matches the virtual image is determined, wherein the image of the first image data is included in a group of two or more images of the first image data, and wherein each of the two or more images is associated with a timestamp occurring before, during, and / or after the first timestamp.

[0130] 16. The system according to Example 15, wherein the operation further includes determining a difference between (i) a timestamp associated with the image of the first image data that best matches the virtual image and (ii) the first timestamp, and wherein updating the registration includes updating the registration at least in part based on the determined difference.

[0131] 17. The system according to any one of Examples 1-16, wherein the operation further comprises:

[0132] Determine when the current position or orientation of the biomedical device changes the threshold amount; and

[0133] In response to the determination, a correspondence is generated by matching the patient's anatomical structures in the image of the first image data with the patient's anatomical structures in the portion of the anatomical region in the second image data.

[0134] 18. The system according to any one of Examples 1-17, wherein the operation further comprises:

[0135] The timing of the biomedical device's location at a first patient anatomical structure within the anatomical region is determined, at least in part, based on the generated registration; and

[0136] In response to the determination, a correspondence is generated by matching the first patient anatomical structure in the first image data with the first patient anatomical structure in the portion of the anatomical region in the second image data.

[0137] 19. The system according to any one of Examples 1-18, wherein the operation further comprises:

[0138] Determine when the biomedical device undergoes a commanded movement through an anatomical passageway across the anatomical region; and

[0139] In response to the determination, the registration is generated and / or updated.

[0140] 20. A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the computing system to perform operations including the following when executed by one or more processors of a computing system:

[0141] A point cloud of coordinate points is generated, at least in part, based on location sensor data captured using a location sensor, wherein the location sensor data is associated with one or more locations of a biomedical device within the patient's anatomical region;

[0142] Receive first image data of patient anatomical structures captured using an image capture device located within the anatomical region;

[0143] Receive second image data of the anatomical region, wherein the second image data is generated at least in part based on preoperative or intraoperative imaging of the anatomical region;

[0144] Registration is generated between at least a portion of the point cloud and at least a portion of the second image data; and

[0145] The registration is updated at least in part based on the first image data.

[0146] 21. The non-transient computer-readable medium according to Example 20, wherein the operation further includes generating one or more correspondences by matching patient anatomical structures in one or more images of the first image data with patient anatomical structures in the anatomical region of the portion of the second image data.

[0147] 22. The non-transient computer-readable medium according to Example 21, wherein the operation further includes adding one or more coordinate points to the point cloud at one or more locations, the one or more locations corresponding to one or more locations of the image capturing device within the anatomical region associated with the one or more images of the first image data.

[0148] 23. The non-transient computer-readable medium according to Example 22, wherein generating the registration includes weighting the one or more added coordinate points differently from other coordinate points of the point cloud generated from the position sensor data.

[0149] 24. The non-transient computer-readable medium according to any one of Examples 21-23, wherein the operation further includes determining a transformation for aligning an image in one or more images of the first image data with a corresponding patient anatomical structure in the anatomical region of the portion of the second image data, and wherein generating the registration includes generating the registration at least in part based on the transformation.

[0150] 25. The non-transient computer-readable medium according to any one of Examples 21-24, wherein the operation further comprises determining at least a portion of a pathway taken by the biomedical device throughout the anatomical region based at least in part on the first image data, and wherein generating the registration comprises generating the registration between at least the portion of the point cloud and a segment of the anatomical region corresponding to the portion of the pathway.

[0151] 26. The non-transient computer-readable medium according to any one of Examples 21-25, wherein said operation further comprises:

[0152] Estimate the registration error between the correspondence in one or more of the correspondences and the generated registration; and

[0153] The display of the generated registration is colored based at least in part on the magnitude of the estimated registration error.

[0154] 27. The non-transient computer-readable medium according to any one of Examples 21-26, wherein said operation further comprises:

[0155] Real-time estimation of (i) the registration error between the current location of the biomedical device within the anatomical region and (ii) the registration error between the correspondence in one or more correspondences and the generated registration; and

[0156] The corresponding portion of the generated registration is colored based at least in part on the magnitude of the estimated registration error.

[0157] 28. The non-transient computer-readable medium according to any one of Examples 20-27, wherein said operation further comprises:

[0158] Based at least in part on the generated registration, a virtual image of the patient's anatomical structure within the anatomical region is calculated from the perspective of the image capture device at the current position of the image capture device within the anatomical region; and

[0159] Determine a transformation for aligning the virtual image with the first image data corresponding to the current position of the image capture device.

[0160] 29. The non-transient computer-readable medium according to any one of Examples 20-28, wherein said operation further comprises:

[0161] Based at least in part on the generated registration, a virtual image of the patient's anatomical structure within the anatomical region is calculated from the viewpoint of the image capturing device at the current or previous location of the image capturing device within the anatomical region, wherein the virtual image is associated with a first timestamp; and

[0162] The first image data that best matches the virtual image is determined, wherein the image of the first image data is included in a group of two or more images of the first image data, and wherein each of the two or more images is associated with a timestamp occurring before, during, and / or after the first timestamp.

[0163] 30. The non-transient computer-readable medium according to Example 29, wherein the operation further includes determining a difference between (i) a timestamp associated with the image of the first image data that best matches the virtual image and (ii) the first timestamp, and wherein updating the registration includes updating the registration at least in part based on the determined difference.

[0164] 31. A method comprising:

[0165] A point cloud of coordinate points is generated, at least in part, based on position sensor data captured using position sensors of a robotic system, wherein the position sensor data is associated with one or more locations of a biomedical device within the anatomical region of the patient;

[0166] When the image capture device of the robotic system is located within the anatomical region, it receives first image data of the patient's anatomical structures captured using the image capture device;

[0167] Receive second image data of the anatomical region, wherein the second image data is at least partially based on preoperative or intraoperative imaging of the anatomical region;

[0168] Registration is generated between at least a portion of the point cloud and at least a portion of the second image data; and

[0169] The registration is updated based at least in part on a portion of the first image data.

[0170] C. in conclusion

[0171] The systems and methods described herein may be provided in the form of one or more tangible and non-transient machine-readable media (such as hard disk drives, hardware memory, etc.) on which instructions for execution by a processor or computer are recorded. The instruction set may include various commands instructing a computer or processor to perform specific operations, such as the methods and processes of the various embodiments described herein. The instruction set may be in the form of a software program or application. Computer storage media may include volatile and non-volatile media, as well as removable and non-removable media, for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media may include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technologies, CD-ROM, DVD or other optical storage devices, disk storage devices, or any other hardware media that can be used to store desired information and can be accessed by components of the system. Components of the system may communicate with each other via wired or wireless communication. Components may be separate from each other, or various combinations of components may be integrated together into a monitor or processor, or contained within a workstation having standard computer hardware (e.g., processors, circuit systems, logic circuits, memory, etc.). The system may include processing devices such as microprocessors, microcontrollers, integrated circuits, control units, storage media, and other hardware.

[0172] While numerous embodiments have been described above in the context of navigating and performing medical procedures within a patient's lungs, other applications and embodiments beyond those described herein are also within the scope of this technology. For example, unless otherwise stated or clearly stated from the context, the devices, systems, methods, and computer program products of this technology can be used in a variety of image-guided medical procedures, such as those performed on, within, or adjacent to a hollow patient anatomy, and more specifically, in procedures for mapping, biopsy, ablation, or otherwise treating tissue within and / or near a hollow patient anatomy. Thus, for example, the systems, devices, methods, and computer program products of this disclosure can be used in one or more medical procedures associated with other patient anatomy, such as a patient's bladder, urinary tract, GI system, and / or heart.

[0173] As used herein, the term "operator" should be understood to include any type of person who may be performing or assisting in a medical procedure, and therefore includes physicians, surgeons, doctors, nurses, medical technicians, other persons or users of the technologies disclosed herein, and any combination thereof. Additionally or alternatively, the term "patient" should be considered to include human and / or non-human (e.g., animal) patients to whom medical procedures are performed.

[0174] As will be understood from the foregoing, for illustrative purposes, specific embodiments of the present technology have been described herein, but well-known structures and functions have not been shown or described in detail to avoid unnecessarily obscuring the description of embodiments of the present technology. In the event of any conflict between any material incorporated herein by reference and this disclosure, this disclosure shall prevail. Where the context permits, singular or plural terms may also include plural or singular terms respectively. Furthermore, unless the word “or” is explicitly limited to referring only to a single item excluding other items when referring to a list of two or more items, its use in such a list shall be construed as including (a) any single item in the list, (b) all items in the list, or (c) any combination of items in the list. As used herein, the phrase “and / or” in “A and / or B” refers to A alone, B alone, and both A and B. Where the context permits, singular or plural terms may also include plural or singular terms respectively. Furthermore, the terms “comprising,” “including,” “having,” and “having” are used throughout to indicate that at least one or more of the listed features are included, such that any additional number of the same features and / or other types of additional features are not excluded.

[0175] Furthermore, as used herein, the term "substantially" refers to the complete or near-complete extent or degree of an action, characteristic, attribute, state, structure, item, or result. For example, a "substantially" closed object would mean that the object is either completely closed or nearly completely closed. In some cases, the exact permissible deviation from absolute completeness can depend on the specific circumstances. However, in general, a degree of near-completeness will have the exact same overall result as achieving absolute completeness and wholeness. When used in a negative sense, the use of "substantially" also applies to referring to the complete or near-complete lack of an action, characteristic, attribute, state, structure, item, or result.

[0176] The above detailed description of embodiments of this technology is not intended to be exhaustive or to limit the technology to the precise forms disclosed above. While specific embodiments and examples of the technology have been described above for illustrative purposes, various equivalent modifications can be made within the scope of this technology, as will be recognized by those skilled in the art. For example, although the steps are presented in a given order, alternative embodiments may perform the steps in a different order. As another example, the various components of the technology may be further divided into sub-components, and / or the various components and / or functions of the technology may be combined and / or integrated. Furthermore, although advantages associated with certain embodiments of this technology have been described in the context of those embodiments, other embodiments may also exhibit such advantages, and not all embodiments must exhibit such advantages to fall within the scope of this technology.

[0177] It should also be noted that other embodiments besides those disclosed herein are also within the scope of this technology. For example, embodiments of this technology may have different configurations, components, and / or programs than those shown or described herein. Furthermore, those skilled in the art will understand that these and other embodiments may be without certain configurations, components, and / or programs shown or described herein without departing from this technology. Therefore, this disclosure and associated technologies can cover other embodiments not explicitly shown or described herein.

Claims

1. A medical device system for use in an image-guided medical procedure, the system comprising: A position sensor configured to generate position sensor data associated with one or more locations of a medical device within the patient's anatomical region; An image capturing device configured to capture first image data of patient anatomical structures within the anatomical region when the medical device is located within the anatomical region; A processor communicatively coupled to the position sensor and the image capture device; as well as A memory for storing instructions that, when executed by the processor, cause the system to perform operations including: A point cloud of coordinate points is generated, at least in part, based on the location sensor data. Receive second image data of the anatomical region, wherein the second image data is generated at least in part based on imaging of the anatomical region. One or more correspondences are generated by matching the patient's anatomical structures in one or more images of the first image data with the patient's anatomical structures of the anatomical region in the second image data. When the patient's anatomical structure in the first image data matches the patient's anatomical structure in the second image data based on the one or more correspondences, one or more coordinate points are added to the point cloud at one or more points corresponding to one or more known locations of the image capture device. Registration is generated between at least a portion of the point cloud and at least a portion of the second image data, and The registration is updated at least in part based on the first image data.

2. The system of claim 1, wherein the patient anatomical structure in one or more images of the first image data and the patient anatomical structure in the anatomical region of the second image data are one or more branch points of anatomical channels in the anatomical region.

3. The system of claim 1, wherein generating the registration includes weighting the one or more added coordinate points differently from other coordinate points of the point cloud generated from the position sensor data.

4. The system of claim 1, wherein the portion of the point cloud comprises only the one or more added coordinate points.

5. The system of claim 1, wherein the operation further comprises determining a transformation for aligning an image in one or more images of the first image data with a corresponding patient anatomical structure of the anatomical region in the second image data, and wherein generating the registration comprises generating the registration at least in part based on the transformation.

6. The system of claim 1, wherein the operation further comprises determining at least a portion of a pathway taken by the medical device throughout the anatomical region based at least in part on the first image data, and wherein generating the registration comprises generating the registration between at least the portion of the point cloud and a segment of the anatomical region corresponding to the portion of the pathway.

7. The system of claim 1, wherein the operation further includes estimating the registration error between the correspondence in the one or more correspondences and the generated registration.

8. The system of claim 7, wherein the operation further includes coloring the display of the generated registration based at least in part on the magnitude of the estimated registration error.

9. The system of claim 8, wherein the operation further comprises: Real-time estimation of the registration error at the current position of the medical device within the anatomical region; as well as The corresponding part of the display is colored.

10. The system according to any one of claims 1-9, wherein the operation further comprises: Based at least in part on the generated registration, a virtual image of the patient's anatomical structure in the anatomical region is calculated from the perspective of the image capture device at the current position of the image capture device within the anatomical region; as well as Determine a transformation for aligning the virtual image with the first image data corresponding to the current position of the image capture device.

11. The system of claim 10, wherein updating the registration includes updating the registration at least in part based on the determined transformation.

12. The system of claim 10, wherein determining the transformation comprises: The transformation is determined only for a portion of the generated registration within a threshold distance from the current point of the image capture device; or The transformation is determined for the specific respiratory and / or cardiac phase of the patient.

13. The system according to any one of claims 1-9, wherein the operation further comprises: Based at least in part on the generated registration, a virtual image of the patient's anatomical structure within the anatomical region is calculated from the viewpoint of the image capturing device at the current or previous location of the image capturing device within the anatomical region, wherein the virtual image is associated with a first timestamp; and The first image data that best matches the virtual image is determined, wherein the image of the first image data is included in a group of two or more images of the first image data, and wherein each of the two or more images is associated with a timestamp occurring before, during, and / or after the first timestamp.

14. The system of claim 13, wherein the operation further comprises determining a difference between (i) a timestamp associated with the image of the first image data that best matches the virtual image and (ii) the first timestamp, and wherein updating the registration comprises updating the registration at least in part based on the determined difference.

15. The system according to any one of claims 1-9, wherein the operation further comprises: Determine when the current position or orientation of the medical device changed the threshold amount; as well as In response to the determination, a correspondence is generated by matching the patient's anatomical structures in the image of the first image data with the patient's anatomical structures in the portion of the anatomical region in the second image data.

16. The system according to any one of claims 1-9, wherein the operation further comprises: The timing of the medical device's location at the first patient anatomy within the anatomical region is determined at least in part based on the generated registration. as well as In response to the determination, a correspondence is generated by matching the first patient anatomical structure in the first image data with the first patient anatomical structure in the portion of the anatomical region in the second image data.

17. The system according to any one of claims 1-9, wherein the operation further comprises: Determine when the medical device is subjected to a commanded movement through an anatomical passageway across the anatomical region; as well as In response to the determination, the registration is generated and / or updated.

18. A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the computing system to perform operations including the following when executed by one or more processors of a computing system: A point cloud of coordinate points is generated, at least in part, based on position sensor data captured using a position sensor, wherein the position sensor data is associated with one or more locations of a medical device within the patient's anatomical region; Receive first image data of patient anatomical structures captured using an image capture device located within the anatomical region; Receive second image data of the anatomical region, wherein the second image data is generated at least in part based on preoperative or intraoperative imaging of the anatomical region; One or more correspondences are generated by matching the patient's anatomical structures in one or more images of the first image data with the patient's anatomical structures of the anatomical region in the second image data; When the patient's anatomical structure in the first image data matches the patient's anatomical structure in the second image data based on the one or more correspondences, one or more coordinate points are added to the point cloud at one or more points corresponding to one or more known locations of the image capture device; Registration is generated between at least a portion of the point cloud and at least a portion of the second image data; as well as The registration is updated at least in part based on the first image data.

19. The non-transient computer-readable medium of claim 18, wherein generating the registration includes weighting the one or more added coordinate points differently from other coordinate points of the point cloud generated from the position sensor data.

20. The non-transient computer-readable medium of claim 18, wherein the operation further comprises determining a transformation for aligning an image in one or more images of the first image data with a corresponding patient anatomical structure of the anatomical region in the second image data, and wherein generating the registration comprises generating the registration at least in part based on the transformation.

21. The non-transient computer-readable medium of claim 18, wherein the operation further comprises determining at least a portion of a pathway taken by the medical device throughout the anatomical region based at least in part on the first image data, and wherein generating the registration comprises generating the registration between at least said portion of the point cloud and a segment of the anatomical region corresponding to said portion of the pathway.

22. The non-transient computer-readable medium of claim 18, wherein the operation further comprises: Estimate the registration error between the correspondence in the one or more correspondences and the generated registration; as well as The generated registration is colored based at least in part on the magnitude of the estimated registration error.

23. The non-transient computer-readable medium of claim 18, wherein the operation further comprises: Real-time estimation of (i) the current location of the medical device within the anatomical region and (ii) the registration error between the correspondence in one or more correspondences and the generated registration; as well as The corresponding portion of the generated registration is colored based at least in part on the magnitude of the estimated registration error.

24. The non-transient computer-readable medium according to any one of claims 18-23, wherein the operation further comprises: Based at least in part on the generated registration, a virtual image of the patient's anatomical structure in the anatomical region is calculated from the perspective of the image capture device at the current position of the image capture device within the anatomical region; as well as Determine a transformation for aligning the virtual image with the first image data corresponding to the current position of the image capture device.

25. The non-transient computer-readable medium according to any one of claims 18-23, wherein the operation further comprises: Based at least in part on the generated registration, a virtual image of the patient's anatomical structure within the anatomical region is calculated from the viewpoint of the image capturing device at the current or previous location of the image capturing device within the anatomical region, wherein the virtual image is associated with a first timestamp; and The first image data that best matches the virtual image is determined, wherein the image of the first image data is included in a group of two or more images of the first image data, and wherein each of the two or more images is associated with a timestamp occurring before, during, and / or after the first timestamp.

26. The non-transient computer-readable medium of claim 25, wherein the operation further comprises determining a difference between (i) a timestamp associated with the image of the first image data that best matches the virtual image and (ii) the first timestamp, and wherein updating the registration comprises updating the registration at least in part based on the determined difference.

27. A method comprising: A point cloud of coordinate points is generated, at least in part, based on position sensor data captured using position sensors of a robotic system, wherein the position sensor data is associated with one or more locations of medical devices within the anatomical region of the patient; When the image capture device of the robotic system is located within the anatomical region, it receives first image data of the patient's anatomical structures captured using the image capture device; Receive second image data of the anatomical region, wherein the second image data is at least partially based on preoperative or intraoperative imaging of the anatomical region; One or more correspondences are generated by matching the patient's anatomical structures in one or more images of the first image data with the patient's anatomical structures of the anatomical region in the second image data; When the patient's anatomical structure in the first image data matches the patient's anatomical structure in the second image data based on the one or more correspondences, one or more coordinate points are added to the point cloud at one or more points corresponding to one or more known locations of the image capture device; Registration is generated between at least a portion of the point cloud and at least a portion of the second image data; as well as The registration is updated at least in part based on the first image data.

Images