Robot-assisted navigation intraoperative imaging

Abstract

Systems and methods for navigating during a medical procedure are described. The processing unit receives, from the sensing system, an indication of a position in a first coordinate system of a first sensor and a second sensor disposed at the flexible elongate device. The processing unit also receives imaging data from the imaging unit and determines an indication of a position of the reference of the flexible elongate device in the second coordinate system based on the imaging data. The reference is disposed at the flexible elongate device at a separation distance from the first sensor. The processing unit then registers the first coordinate system to the second coordinate system based on the received imaging data and the received sensor system data.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit on the filing date of U.S. Provisional Patent Application No. 63 / 514,191, filed July 18, 2023, entitled “INTRA-OPERATIVE IMAGING TO ROBOTIC-ASSISTED NAVIGATION”. The entire contents of that U.S. Provisional Patent Application are expressly incorporated herein by reference. Technical Field

[0003] The disclosed examples relate to planning and / or navigating minimally invasive medical procedures, and more specifically to the registration between the coordinates of a robot-assisted system that actuates an elongation device and the coordinates of intraoperative images. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, physicians can insert minimally invasive medical devices (including surgical, diagnostic, therapeutic, and / or biopsy instruments) to reach target tissue locations. One such technique utilizes flexible and / or steerable elongated devices (e.g., flexible catheters) that can be inserted into an anatomical channel and navigated toward regions of interest within the patient's anatomy.

[0005] The combination of positioning sensors located at flexible elongation devices and intraoperative imaging can greatly aid in the planning and navigation of minimally invasive surgeries. Specifically, combining sensor data with intraoperative images enables accurate determination of the position, orientation, and / or pose of the flexible elongation device within the patient's anatomy. However, accurate and rapid registration between the sensor coordinate system and the intraoperative image coordinate system remains a challenge using current technology. Summary of the Invention

[0006] The following presents a brief overview of the various examples described herein, and is not intended to identify key or important elements or define the scope of the claims.

[0007] In some examples, a tangible, non-transitory computer-readable medium (CRM) stores instructions that, when executed by one or more processors, cause one or more processors to receive indications of the positions of a first and second sensor disposed on a flexible elongation medical device in a first coordinate system. The instructions may also cause one or more processors to determine, based on intraoperative imaging data, an indication of the position of a reference point of the flexible elongation device in a second coordinate system, wherein the reference point is disposed along the flexible elongation device at a separation distance from the first sensor. Furthermore, the instructions may cause one or more processors to register the first coordinate system to the second coordinate system, at least in part, based on the indications of the positions of the first and second sensors in the first coordinate system, the indication of the position of the reference point in the second coordinate system, and the separation distance.

[0008] In other examples, the medical device includes: a flexible elongated body having an axis; and a position sensor disposed at the flexible elongated body of the medical device, the position sensor being configured to generate an indication of position in a first coordinate system. The medical device also includes a reference disposed at the flexible elongated body of the device and having a rotationally asymmetric shape relative to the axis of the flexible elongated body of the device.

[0009] In other examples, the medical system includes: a flexible elongation device; a first sensor and a second sensor disposed at the flexible elongation device; and a reference disposed at the flexible elongation device at a separation distance from the first sensor. The system also includes one or more processors configured to receive indications of the positions of the first and second sensors in a first coordinate system. The processors are further configured to determine the position of the reference in a second coordinate system based on intraoperative imaging data. Furthermore, the processors are configured to register the first coordinate system to the second coordinate system based on the received indications of the positions of the first and second sensors in the first coordinate system, the determined position of the reference in the second coordinate system, and the separation distance.

[0010] In other examples, a method includes receiving, by one or more processors, indications of the positions of a first sensor and a second sensor disposed at a flexible elongating medical device in a first coordinate system. The method further includes determining, by one or more processors, an indication of the position of a reference point of the flexible elongating device in a second coordinate system based on intraoperative imaging data, wherein the reference point is disposed at a separation distance from the first sensor at the flexible elongating device. Furthermore, the method includes registering the first coordinate system to the second coordinate system by one or more processors, at least in part, based on the indications of the positions of the first and second sensors in the first coordinate system, the indication of the position of the reference point in the second coordinate system, and the separation distance.

[0011] In other examples, a tangible, non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause one or more processors to: receive an indication of the position of a position sensor disposed at a flexible elongator of the medical device in a first coordinate system. The instructions may also cause one or more processors to: determine, based on intraoperative imaging data, an indication of the position of a reference disposed at the flexible elongator and having a rotationally asymmetric shape relative to the axis of the flexible elongator in a second coordinate system. Furthermore, the instructions may cause one or more processors to: register the first coordinate system to the second coordinate system, at least in part, based on the indication of the position sensor's position and the indication of the reference's position in the second coordinate system.

[0012] In other examples, a method includes receiving, by one or more processors, an indication of the position of a position sensor disposed at a flexible elongator of a medical device in a first coordinate system. The method further includes determining, based on intraoperative imaging data and by one or more processors, an indication of the position of a reference disposed at the flexible elongator and having a rotationally asymmetric shape relative to the axis of the flexible elongator in a second coordinate system. Furthermore, the method includes registering the first coordinate system to the second coordinate system by one or more processors, at least in part, based on the indication of the position of the position sensor and the indication of the position of the reference in the second coordinate system. Attached Figure Description

[0013] Figure 1A An example system for navigation during medical procedures within an operating environment is described.

[0014] Figure 1B This is a simplified diagram of a flexible elongation device installed within an anatomical structure.

[0015] Figure 2A The rigid bodies in two example coordinate systems are depicted.

[0016] Figure 2B An example coordinate registration process is illustrated schematically.

[0017] Figures 3A to 3E An example configuration of sensors and references set at the flexible elongation device is illustrated schematically.

[0018] Figures 4A to 4D An example configuration of a reference with a rotationally asymmetric shape relative to the axis of the flexible elongated body of the device is schematically shown.

[0019] Figure 5A and Figure 5B An example configuration of a baseline that is detachably attached to a flexible elongation device is illustrated schematically.

[0020] Figure 6A and Figure 6BThis is a block diagram of a method for implementing the technology disclosed herein.

[0021] Figure 7 It is a simplified diagram based on some examples of medical systems.

[0022] Figure 8A These are simplified diagrams of medical device systems based on some examples.

[0023] Figure 8B This is a simplified diagram of a medical device that includes medical tools within an elongating device, based on some examples.

[0024] Figure 9A and Figure 9B It is a simplified diagram of a patient coordinate space side view based on some examples, including a medical device mounted on an insert component.

[0025] Examples of this disclosure and its advantages can be better understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more of the accompanying drawings, which are shown for illustrative purposes and not for limiting the scope of this disclosure. Detailed Implementation

[0026] In the following description, specific details are set forth in relation to some examples conforming to this disclosure. Numerous specific details are set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that some examples can be practiced without some or all of these specific details. The specific examples disclosed herein are intended to be illustrative and not restrictive. Other elements within the scope and spirit of this disclosure can be implemented by those skilled in the art, although not specifically described herein. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless otherwise specifically described or if one or more features would render the example inoperable. In some cases, well-known methods, processes, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the examples.

[0027] This disclosure describes various instruments and parts thereof based on their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or part of an object (e.g., one or more rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of that object or part of an object in at least one rotational degree of freedom (e.g., up to six total degrees of freedom). As used herein, the term “shape” refers to the set of poses, positions, and / or orientations measured along an object. As used herein, the term “distal” refers to a location closer to the process site, and the term “proximal” refers to a location further away from the process site. Thus, when an instrument is designed to perform a process, the distal portion or distal end of the instrument is closer to the process site than the proximal portion or proximal end of the instrument.

[0028] This disclosure generally relates to systems and methods for facilitating user (e.g., physician) planning and / or user navigation during medical procedures (e.g., intracavitary procedures). These systems and methods can provide precise and accurate registration between a first coordinate system (e.g., the coordinate system of a robot-assisted system for actuating medical devices) and a second coordinate system (e.g., the coordinate system of intraoperative images) to improve the accuracy and efficiency of the procedure.

[0029] The methods and systems of this disclosure use data from one or more sensors (e.g., point sensors) positioned on a flexible elongation device (e.g., a catheter) to identify the position and / or orientation of the sensors in the coordinate system of a robotic system (i.e., using robotic system coordinates) and use data from intraoperative images in an image coordinate system (i.e., using imaging system coordinates) to register the two coordinate systems to each other. The registration process may include determining a one-to-one correspondence between the two coordinate systems and / or mathematical transformations to calculate the coordinates of a point in one coordinate system based on the coordinates of the same point in the other coordinate system. Registration can be updated as needed during the process, for example, in response to a change in one of the coordinate systems. Changes in the physical environment can accelerate the change in coordinate systems, for example, by altering the magnetic field sensed by one or more sensors.

[0030] Some previous registration methods focused on obtaining a multi-point (e.g., dozens of points) description of the shape of the flexible elongation device in robotic system coordinates and a multi-point description of its shape in image coordinates (e.g., by segmenting the flexible elongation device from intraoperative images). To determine the shape in robotic system coordinates, fiber optic shape sensors can be positioned along the flexible elongation device. On the other hand, to determine the shape in image coordinates, a computational system can perform segmentation of the flexible elongation device based on image data. Shape-based coordinate transformations may have certain limitations. For example, the flexible elongation device may not have an accompanying fiber optic shape sensor. Furthermore, segmenting the flexible elongation device from image data may require user intervention to identify pixels or voxels associated with the device.

[0031] The systems and methods described in this disclosure can use point sensors, such as electromagnetic (EM) sensors, as an alternative to using fiber optic shape sensors. In some examples, the system can segment an image portion corresponding to a flexible elongation device (e.g., showing at least a portion of the flexible elongation device) from the remainder of an intraoperative image and identify the position and / or orientation of at least one EM sensor positioned at the flexible elongation device within the segmented portion. A single EM sensor having all six degrees of freedom (DOF) in both robot system coordinates and image coordinates may be sufficient for registration, at least in the vicinity of the sensor (if not the entire imaging field). When an EM sensor cannot provide all six DOFs, the system can use multiple (at least two) EM sensors to obtain the missing DOFs in robot system coordinates. If the EM sensor package does not have certain symmetries, a well-resolution image of one of the EM sensors may be sufficient to obtain the six DOFs in image coordinates. However, sometimes the EM sensor package may not be sufficiently visible in the image or may have poor resolution or indistinguishable symmetries that obscure the orientation information. That is, there are cases where the presence of an EM sensor package is an unsatisfactory reference for determining the position and / or orientation of the sensor in the image coordinate system. Additional references can be set at the flexible elongation device to more accurately and precisely identify the position and / or orientation in the image coordinates.

[0032] In methods that rely on point positions and / or orientation sensors in robot coordinates and references in image coordinates, the reference does not need to coincide with the position of one of the sensors. The reference can be set at a known distance from one of the sensors along the body of the flexible elongator. If the separation between the sensors along the body of the flexible elongator is also known, the position of the reference relative to one sensor indicates its position relative to the other sensor (along the body of the flexible elongator).

[0033] It should be noted that the references can be distributed along the flexible elongation device. Therefore, an image of the references can indicate the position of multiple points along the flexible elongation device. That is, multiple reference elements are positioned relative to the device at known relative positions on the flexible elongation device, and whether connected or disconnected, they can be considered a single reference. Such references can include elements of various materials such as metals, plastics, confined fluids (e.g., bubbles), etc. In one implementation, the references can include multiple circular or elliptical rings surrounding a flexible elongation device having a generally circular or elliptical cross-section. The elements of the references (e.g., the rings surrounding the flexible elongation device) can be arranged to create a pattern that, in some sense, encodes a specific position along the device. Furthermore, the references can include axial elements parallel to the axis or long-dimensional orientation of the elongation device to, for example, help identify orientation in image coordinates. Additionally, the references can be detachably positioned at the flexible elongation device, for example, by being positioned at a detachable sleeve or probe that can be detachably attached to the flexible elongation device.

[0034] Registration between coordinate systems does not need to rely solely on the position of the point sensor and the position of the imaging reference. Various constraints can facilitate registration. For example, the disclosed registration method can use constraints based on the mechanics of a flexible elongating device, such as curvature assumptions between successive points of the device, orientation constraints between the robotic system, the imaging system, and the patient's anatomy. Additionally or alternatively, the operating environment (e.g., a horizontal patient bed or operating table) and position or orientation can provide additional constraints to facilitate coordinate registration.

[0035] These methods can include various specific combinations of the number and type of point sensors and reference shapes. For example, L-shaped reference elements, non-uniformly spaced ring elements, etc., can be included in the reference. The system can use one, two, three, or any other suitable number of sensors to obtain six-DOF information and / or redundant position information, thereby improving registration accuracy.

[0036] After determining the registration between coordinate systems, the system can generate a graphical representation of the flexible elongation device in a graphical user interface (GUI). The system can overlay the graphical representation of the flexible elongation device onto a model of the anatomical structure in which the flexible elongation device is located, or onto the model of the anatomical structure. Furthermore, the system can display sensor locations and / or references in a joint coordinate system. The system can display one or more sensor locations along with portions of the flexible elongation device where one or more sensors and / or references are located.

[0037] Figure 1AAn example system 100 for navigation within an operating environment 101 during a medical procedure is depicted. System 100 can acquire images of a portion of the operating environment 101 positioned within the field of view F (generally demarcated by dashed lines) of an imaging unit 110. For this purpose, system 100 can be communicatively connected to imaging unit 110. Imaging unit 110 can use color imaging, infrared imaging, ultrasound imaging, X-ray imaging, fluorescence imaging, computed tomography (CT) imaging, magnetic resonance imaging (MRI) imaging, or some other type of imaging device to generate two-dimensional or volumetric images of at least a portion of the operating environment. Additionally, system 100 can acquire data from one or more sensors positioned within the operating environment. For this purpose, system 100 can be communicatively connected to one or more sensors via a communication connection with sensing unit 115.

[0038] System 100 includes a processing unit 120 and a display unit 130 that are communicatively connected to each other. Although in Figure 1A In this embodiment, the imaging unit 110 and sensing unit 115 are depicted differently from system 100; however, in other examples, system 100 may include imaging unit 110 and / or sensing unit 115. In any case, one or more processors of processing unit 120 of system 100 may be configured to receive images and / or processed image information from imaging unit 110 and receive data from one or more sensors via sensing unit 115.

[0039] Throughout this disclosure, the description of the example operations performed by processing unit 120 below will be understood as being performed by one or more processors of processing unit 120. In some examples, one or more processors may include hardware specifically configured (e.g., hardwired or programmable) to perform at least a portion of the example operations described in this disclosure. Additionally or alternatively, one or more processors may be configured to perform at least a portion of the example operations described in this disclosure by executing a set of software instructions. For this purpose, system 100 may include a tangible, non-transitory computer-readable medium or communicatively connected to a tangible, non-transitory computer-readable medium. The medium may store instructions that, when executed by processing unit 120, perform any one or more of the example operations described below. For example, the instructions may cause processing unit 120 to perform image processing operations on an image received from imaging unit 110 and / or perform calculations (e.g., for coordinate registration) based on data received from sensing unit 115. Furthermore, instructions can cause the processing unit 120 to cause the display unit 130 to display information via a graphical user interface based on the processing of the image received from the imaging unit 110 and the processing of the data received by the sensing unit 115. For example, the processing unit 120 can send information to the display unit 130, or send data representing the entire graphical user interface including that information.

[0040] Operators of the medical system (e.g., physicians, other medical practitioners, or fully automated robotic surgical systems) can use information displayed at display unit 130 to perform medical procedures (e.g., endoscopy, biopsy, drug therapy, and / or treatments such as ablation). During the medical procedure, the operator can control a flexible elongation device 140 inserted through orifice O into the anatomical structure A of patient P positioned at table T. For example, the medical procedure may include navigating the flexible elongation device 140 (indicated by solid lines outside patient P and dashed lines inside patient P) toward a region of interest R within anatomical structure A using information displayed at display unit 130. Region R may, for example, be a designated surgical site for examination, biopsy, treatment, or any other medical procedure.

[0041] Reference 142 may be disposed at the flexible elongation device 140 (e.g., physically contacting, integrated therein, fixedly attached to, or detachably attached to the flexible elongation device 140 in a manner that forms a rigid relationship with the flexible elongation device 140 during operation / use). Reference 142 is configured to be visible in the image acquired by imaging unit 110. For this purpose, reference 142 may include elements of various materials and / or structures such as metals, plastics, etched glass, dyes, radioactive or fluorescent markings, confined fluids (e.g., bubbles), etc. Reference 142 does not need to be positioned at a specific point along the flexible elongation device (e.g., within 1 mm, 2 mm, 5 mm, 10 mm), but may have elements distributed along the length of the flexible elongation device 140. For example, distributed reference 142 may span any length from a few millimeters to tens of centimeters. Despite spanning a certain length, reference 142 may allow a large portion (e.g., 20% to 99%) of the span along the flexible elongation device 140 to be unobstructed. At least some of the elements of reference 142 may be integrated (e.g., etched, deposited, painted, or otherwise securely attached) to the flexible elongation device 140. Alternatively or additionally, reference 142 may include elements detachably disposed at the flexible elongation device 140. For example, reference 142 may be integrated into a detachable structure such as a sleeve or probe, which may then be detachably attached to the flexible elongation device 140. The detachable structure may include elements that allow the detachable structure to be registered to the flexible elongation device 140 (e.g., at a fixed distance from the distal end of the flexible elongation device 140).

[0042] Sensors 144a and 144b may also be disposed at the flexible elongation device. Sensors 144a and 144b may be mechanical sensors, optical sensors, electromagnetic (EM) sensors, or any other suitable sensors. Sensors 144a and 144b may be disposed at the flexible elongation device with a known geometric relationship relative to reference 142. Sensors 144a and 144b may be integrated into the flexible elongation device 140 or detachably attached to the flexible elongation device 140. Sensors 144a and 144b may be configured to communicate with sensing unit 115.

[0043] In some examples, sensors 144a and 144b are active sensors configured to transmit electromagnetic (e.g., optical, radio, low-frequency magnetic) or radioactive radiation. Sensing unit 115 may include components for receiving radiation emitted by sensors 144a and 144b and performing triangulation or trilateration of the positions of sensors 144a and 144b in sensor system coordinates. Sensor system coordinates may be those of a robot-assisted system configured to manipulate, control, or guide the flexible elongation device 140.

[0044] In other examples, sensors 144a and 144b are passive and do not emit radiation. Passive sensors 144a and 144b can sense radiation emitted by components of sensing unit 115 disposed within operating environment 101. For example, in an EM sensing system, one or more charged coils may be disposed within operating environment 101 to generate static or dynamic magnetic fields. Sensors 144a and 144b can be configured to pick up changes in dynamic magnetic fields, or, as sensors 144a and 144b move, pick up sensed changes in static or quasi-static magnetic fields and convert the sensed changes into current received by sensing unit 115. In turn, sensing unit 115 can calculate an indication of the position of sensors 144a and 144b based on the received current.

[0045] In some examples, the point sensor may include an accelerometer, a gyroscope, and / or a magnetometer. Furthermore, the point sensor may include an inertial measurement unit (IMU) combining multiple sensors (e.g., accelerometers, gyroscopes) and / or an inertial and magnetic measurement unit (IMMU) combining multiple sensors (e.g., accelerometers, gyroscopes, magnetometers). The IMU and / or IMMU may generate signals indicating the orientation of the flexible elongation device at a given point relative to (e.g., the Earth's) gravitational and / or magnetic fields. In some examples, an additional magnetic field may be introduced into the sensor environment. Additionally or alternatively, the IMU and / or IMMU may generate signals indicating the motion of the flexible elongation device (e.g., caused by motion of anatomical structures due to breathing and / or other factors and / or independent motion of the flexible elongation device within anatomical structures). In some examples, the sensing unit 115 may combine the orientation indication (e.g., up to three degrees of freedom) from the IMU with the position indication from other (e.g., EM) sensors to generate more complete data indicating the pose of the flexible elongation device. Furthermore, in some examples, sensing unit 115 can combine data from sensors in multiple sensor coordinate systems. Sensing unit 115 and / or processing unit 120 can align multiple sensor coordinate systems with each other.

[0046] Figure 1B This is a simplified diagram of a flexible elongation device installed within anatomical structure A. (Includes...) Figure 1BThis is to provide a magnified and more detailed view of a portion of the operating environment 101 set within the field of view F. Anatomical structure A may be the lung of patient P. For the purpose of investigating or treating pathology in region R, the flexible elongation device 140 may be inserted and navigated, for example, by an operator toward region R. The techniques described in this disclosure can facilitate the navigation process by generating and displaying timely and accurate sensing (e.g., imaging) and detection (e.g., identification) of the flexible elongation device 140. These techniques combine position data in one coordinate system obtained using sensors 144a, 144b with position data in another coordinate system obtained using reference 142. Compared to using data solely from imaging unit 110 or sensing unit 115, this technique uses combined data to more accurately and rapidly sense changes in position and / or orientation of at least a portion (e.g., distal or another suitable portion) of the flexible elongation device 140. The combined data can improve the speed, accuracy, reliability, and / or safety of medical procedures. For example, the processing unit 120 can combine data from the imaging unit 110 and / or the sensing unit 115 to quickly and accurately determine the position, orientation and / or pose of at least a portion of the flexible elongation device 140 relative to the anatomical structure A of the patient P.

[0047] Furthermore, processing unit 120 can generate a graphical user interface (GUI) or update GUI data for display on display device 130 to assist the operator in the medical procedure. In some examples, processing unit 120 can generate data and / or control signals for the control unit of a robotic system configured to manipulate and / or navigate the flexible elongation device 140. Additionally or alternatively, processing unit 120 can be configured to generate one or more alarms based on the combined imaging and sensing data. Alarms may include, for example, alarms indicating proximity to region R, alarms indicating potential navigation errors, and alarms indicating that the confidence level of the position of the tip of the flexible elongation device 140 has fallen below a threshold level. It should be noted that combining data from imaging unit 110 and sensing unit 115 requires aligning the two coordinate systems relative to each other, as shown below. Figure 2A and Figure 2B Described.

[0048] Figure 2AA rigid body 202 is depicted in two example coordinate systems: a first coordinate system 204 and a second coordinate system 206. The first coordinate system 204 may correspond to the coordinates of sensors 144a, 144b acquired or generated by sensing unit 115. Sensing unit 115 may be part of a robotic unit configured to actuate and / or manipulate flexible elongation device 140. Therefore, the first coordinate system 204 may represent the coordinate system of the robotic unit. The second coordinate system 206 may correspond to the coordinates of an image acquired or generated by imaging system 120. Generally, in this disclosure, the terms "robot coordinate system" and "sensor (or sensing) coordinate system" are used interchangeably and in contrast to the term "imaging coordinate system".

[0049] A rigid body 202 may represent a portion (e.g., a segment of length) of the flexible elongating device 140. While the flexible elongating device 140 is flexible, short portions (e.g., infinitesimally small segments) may be considered rigid for all practical purposes. The depicted rigid body 202 possesses neither translational nor rotational symmetry. On the other hand, segments of the flexible elongating device 140 may possess rotational symmetry (continuous or discrete) along an axis representing a local centerline along the length of the flexible elongating device 140, at least under the approximate constraints of imaging. Furthermore, segments of the flexible elongating device 140 may be substantially indistinguishable from adjacent segments (e.g., within an image). That is, the flexible elongating device 140 practically possesses a degree of translational symmetry along its axis. In view of the aforementioned symmetries, the techniques of this disclosure are at least partially aimed at resolving the state (e.g., pose) of the flexible elongating device 140.

[0050] The state (e.g., position and orientation) of the rigid body 202 in three-dimensional space can be described using a first coordinate system 204 and / or a second coordinate system 206. A rigid body without symmetry (e.g., body 202) has six degrees of freedom (DOF), and its position and orientation can be described using six coordinates. In the first coordinate system 204, the rigid body 202 can have coordinates (x, y, z, ...). (x, y, and z) where x, y, and z are the position coordinates of the center 208 of the rigid body 202 relative to the origin O along the axes of the first coordinate system 204. In other examples, position coordinates can be specified for any point within the rigid body 202 or for any point that has a geometrically defined rigid relationship with the body 202. and The orientation of the body 202 can be described using orientation vector 209, where orientation vector 209 is... Figure 2A In the example, it originates from center 208 and passes through the middle of one of the small planes of rigid body 202. For example, It can be the elevation angle relative to the z-axis. It can be an azimuth angle parallel to the xy plane, and These can be the rotation angles of the rigid body 202 about the orientation vector 209. Alternatively, the three orientation coordinates can be the roll, pitch, and yaw of the rigid body 202 relative to any suitable reference direction. Similar to the coordinates (x, y, z, ...) of the first coordinate system 204. The coordinates of the second coordinate system 206 are (x', y', z'). ) describes the position (x', y', z') relative to the origin O' and the orientation relative to, for example, the z' axis of a rigid body 202. Registration of the first coordinate system 204 with the second coordinate system 206 at least near the rigid body 202 includes at least the coordinates (x, y, z, ...) ) and coordinates (x',y',z', Find a mapping (e.g., transformation, mathematical relation, etc.) between the two. Another way to view registration is as follows: the mapping is that any point (u',v',w') near the position (x',y',z') defines the corresponding point (u,v,w) near the position (x,y,z).

[0051] In the above discussion, it is assumed that the first coordinate system 204 and the second coordinate system 206 have the same scale. That is, the units of length along the corresponding axes of coordinate systems 204 and 206 are the same. In other words, the transformation from the first coordinate system 204 to the second coordinate system 206 is rigid. However, in some examples, the mapping from the first coordinate system 204 to the second coordinate system 206 may include one or more scaling factors for the axes. Therefore, the mapping may include three translation variables, three rotation variables, and / or three scaling variables. Furthermore, each of these variables may depend on the position.

[0052] In some examples, each of coordinate systems 204 and 206 is independently calibrated to achieve accurate and consistent scaling within the shared operating volume. The coordinate registration process can then be defined based on three translation constants and three rotation constants of the shared operating volume. In other examples, gradual changes in scaling within at least one of coordinate systems 204 and / or 206 may require the use of up to three translation variables and up to three rotation variables, each a function of the position within the shared operating volume.

[0053] Figure 2B An example coordinate registration process is illustrated schematically. In one example, a processing unit (e.g., processing unit 120) can obtain the position and orientation of a rigid body (e.g., rigid body 202) within a first coordinate system 204, or determine the position and orientation of the rigid body (e.g., rigid body 202) within the first coordinate system 204 as (x, y, z, ...). And obtain the position and orientation of the rigid body (e.g., rigid body 202) in the second coordinate system 206, or determine the position and orientation of the rigid body (e.g., rigid body 202) in the second coordinate system 206 as (x', y', z', The processing unit can then generate a mapping M between the two coordinate systems 204 and 206. The processing unit can be configured to map a new position (u', v', w') within the second coordinate system 206 to the corresponding position (u, v, w) within the first coordinate system 204. Alternatively or concurrently, the processing unit can be configured to map coordinates from the first coordinate system 204 to the second coordinate system 206. The mapping can be valid only in the region around (x, y, z). By collecting rigid volume coordinates in both coordinate systems 204 and 206, the processing unit can extend the validity of the mapping over any portion of the shared operational volume of the two coordinate systems 204 and 206. For example, the processing unit can implement coordinate registration as a linear mapping M of Equation 1:

[0054] (1)

[0055] in, and These are rotational parameters (e.g., roll, pitch, and yaw), S 11 S 22 and S 33 d1, d2, and d3 are scaling parameters (which, as discussed above, can be units), and d1, d2, and d3 are displacement factors. Therefore, there can be nine mapping parameters for scaling factors, and six mapping parameters can exist when scaling can be ignored. As discussed above, the linear mapping can be a function of the input position coordinates (u', v', w'). For example, the processing unit can store and / or access a lookup table to find entries for mapping parameters corresponding to the input position coordinates. Because the lookup table can only have a finite number of recorded input coordinates (recorded coordinates in this case), the system can use the entry corresponding to the recorded coordinates closest to the input coordinates. Alternatively, the system can interpolate the mapping parameters corresponding to a set of recorded coordinates near the input coordinates. In other examples, the processing unit can store and / or access a polynomial, spline curve, or another suitable fitting function that relates the input coordinates to the mapping parameters.

[0056] It should be noted that the transformation M can vary as a function of time. For example, vibration of any structure within the operating environment (e.g., operating environment 101) may cause the coordinate system of the imaging system (e.g., second coordinate system 206) to shift relative to the coordinate system of the sensing or robotic system (e.g., first coordinate system 204). Furthermore, some sensing systems may have associated instabilities. For example, the baseline field sensed by a passive EM sensor may be distorted by a moving metallic object within the measurement range of the EM sensing system. The processing unit can update at least a portion of the mapping M at appropriate time intervals (e.g., every 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 50 seconds, 200 seconds, 500 seconds, or any other suitable number of seconds). For this purpose, the processing unit can obtain a new image from the imaging unit 120 including an image of a reference (e.g., reference 142) and / or obtain data from the sensing unit 15 indicating the coordinates of the sensors (e.g., sensors 144a, 144b) in the sensing coordinate system. (Refer to below...) Figure 4A To Figure 4E and Figure 5A Figure 5D discusses the registration of the imaging coordinate system to the sensing coordinate system in more detail.

[0057] Figures 3A to 3E An example configuration of a sensor and one or more reference elements disposed at the flexible elongation device is schematically shown. It should be noted that... Figures 3A to 3E In each of these, the reference element shown can be all or part of a single reference. Figure 3A Sections 340a to 340c of the flexible elongation device are shown. Reference elements 342a and 342b (each of which can be reference 142) are respectively disposed at sections 340c and 340b. Reference elements 342a and 342b each have a rotationally asymmetric shape relative to the axis of the corresponding section of the flexible elongation device (i.e., sections 340b and 340c). An example of the asymmetric shape of the reference element is shown below. Figures 4A to 4D A more detailed description follows. Sensors 344a and 344b are respectively disposed at sections 340a and 340b. At section 340b, reference element 342b and sensor 344b are disposed in substantially the same location.

[0058] A processing unit (e.g., processing unit 120) may receive, for example, an indication of the position of sensor 344b in a first coordinate system (e.g., a sensor or robot coordinate system) from sensing unit 115. The processing unit may acquire one or more intraoperative images (e.g., from imaging unit 110) and determine an indication of the position of reference element 342b in a second coordinate system based on the intraoperative imaging data. The processing unit may register the first coordinate system to the second coordinate system at least in part based on the indications of the position of sensor 344b and the position of reference element 342b. In some examples, the processing unit may receive indications of the position and orientation of sensor 344b including 6DOF and determine, (based on one or more intraoperative images) an indication of the position and orientation of reference element 342b, also including 6DOF. In such examples, the processing unit may register the first coordinate system to the second coordinate system based solely on the received indications of the position and orientation of reference element 342b and sensor 344b, at least near the overlap of reference element 342b and sensor 344b. In other examples, the processing unit may require additional constraints to perform registration. Such constraints can be obtained, for example, from components of additional sensors and / or distributed references.

[0059] In some examples, the overlapping positions of the reference element 342b and the sensor 344b may lead to errors in the indication of the position or orientation of the reference element 342a or the sensor 344b. For example, the sensor 344b may interfere with the accurate segmentation of the reference element 342b from intraoperative images. On the other hand, the reference element 342b may interfere with the sensing unit accurately obtaining the position or orientation of the sensor 344b. For example, the reference element 342b may include a metallic element that distorts the magnetic field sensed by the sensor 344b (which may be an EM sensor). Therefore, it may be advantageous to geometrically / spatially separate the reference element from the sensor located at the flexible elongation device.

[0060] Figure 3B One configuration is shown in which a reference element 342c and two sensors 344c and 344d are positioned at different locations along a segment 340d of the flexible elongation device, with the reference element 342c positioned between the sensors 344c and 344d along the segment 340d. Sensor 344c is positioned at a separation distance L1 from the reference element 342c, and sensor 344d is positioned at a separation distance L2 from the reference element 342c. The separation distances L1 and L2 do not need to be the same, and each can be 2 mm, 5 mm, 10 mm, 20 mm, 50 mm, or any other suitable length. Furthermore, in some examples, one of the separation distances L1 and L2 can be zero, resulting in an overlap between the reference element 342c and one of the sensors 344c and 344d.

[0061] In some examples, sensors 344c and 344d are not individually configured to provide six DOFs, but when combined, they can provide six DOFs. For this purpose, one of sensors 344c and 344d can provide at least three DOFs while the other can provide at least four DOFs, one being the rotation angle relative to segment 340d. To determine the six DOFs in the sensing coordinate system, the processing unit can assume that segment 340d remains rigid between the two sensors 344c and 344d.

[0062] In examples where both L1 and L2 are non-zero, the position in the first coordinate system corresponding to the position of the reference element 342c can be interpolated (by the processing unit) based on the positions of sensors 344c and 344d. In some examples, the interpolation may assume the rigidity of the segment 340d between sensors 344c and 344d. In other examples, see the following reference... Figure 3C The processing unit does not need to assume that there is rigidity between sensors 344c and 344d.

[0063] Figure 3C A configuration is shown in which a reference element 342d and two sensors 344e and 344f are positioned at different locations along a segment 340d of the flexible elongation device, wherein the reference element 342d is positioned along segment 340e between sensors 344e and 344f. This configuration is similar to Figure 3B The configuration is as described in the text, but it has a bend in segment 340e. The processing unit can determine the separation distance r based on the indication of the positions of the sensors 344e and 344f. Based on the separation distance r and the distance between sensors 344e and 344f (e.g., L1+L2, if...), Figure 3C The configuration in is Figure 3B Given the configuration (where there is curvature in segment 340d) and the separation distance r, the processing unit can estimate the curvature angle. For this purpose, the processing unit can assume that the curvature follows an arc. Therefore, the processing unit can interpolate the position of the reference element 342d in the sensing coordinate system along the arc, thereby aiding in the registration between the sensing coordinate system and the image coordinate system.

[0064] In some examples, the processing unit can obtain the orientation coordinates of sensors 344e and 344f (which do not necessarily include rotation about the axis of segment 340e), as shown by lines 349a and 349b. The processing unit can calculate in the sensing coordinate system whether lines 349a and 349b intersect or at least nearly intersect to assess the validity of the assumption of smooth curvature in segment 340e. When the processing unit determines that the assumption of smooth curvature may be violated, the processing unit can generate an alert indicating the level of uncertainty in the registration between the sensing and the image in the coordinate system.

[0065] Figure 3D A configuration is shown in which a multi-element portion of a reference 342e and two sensors 344g and 344h are positioned at different locations along a segment 340f of a flexible elongation device. The reference 342e is distributed along the length of segment 340f. That is, the reference 342e includes elements at two different locations along the length of segment 340f. A processing unit can determine the indications of the two locations of the reference 342e based on intraoperative imaging data. The processing unit can align the sensing and imaging coordinate systems to each other, at least in part, based on the indications of the two locations of the reference 342e. For example, sensor 344h is positioned between the two elements of the reference 342e, while sensor 344g is not positioned between the two elements of the reference 342e. Therefore, the processing unit can uniquely identify sensor 344h, distinguishing it from sensor 344g. More generally, in a similar manner, the reference elements can encode the positions of the sensors along the flexible elongation device. For this purpose, the reference elements can include different geometric features, such as those provided by… Figure 3D The reference 342e is shown. Furthermore, the distributed elements of the reference can improve the accuracy of registration between two coordinate systems by providing additional position and / or orientation information.

[0066] Figure 3E A configuration is shown in which a multi-element portion of a reference 342f and two sensors 344i and 344j are positioned at different locations along a segment 340g of a flexible elongation device. Due to having three non-uniformly spaced elements, the distributed reference 342f indicates two distinct distances L3 and L4 along the segment 340g. Any number of non-uniformly spaced elements of the reference can encode the longitudinal position along the flexible elongation device, which can help identify uniformly spaced sensors along the flexible elongation device and generally improve the accuracy of registration between the sensing coordinate system and the imaging coordinate system. For example, the processing unit can use data indicating the position of the elements separating L3 of the reference 342f to interpolate the position of sensor 344i in the imaging system coordinates. The interpolation method can be similar to that of the reference. Figure 3B and Figure 3C The interpolation method discussed involves interchange of the coordinate systems and functions of the reference element and the sensor.

[0067] Figures 4A to 4D An example configuration of a reference is schematically shown, which has a rotationally asymmetric shape relative to the axis of the flexible elongation of the device and is located in a section of the flexible elongation of the device. Figure 4AA section 440a of a device with a flexible elongated body and a reference element 442a is shown. The flexible elongated body has an axis 445, and the reference element 442a has a rotationally asymmetric elliptical shape relative to the axis 445. It should be noted that for a bent flexible elongated device, the orientation of the axis varies along the length of the device. At any point along the length of the device, the axis can be considered as a straight line tangent to the bending centerline of the device. Figures 4B to 4D In the text, the axis is omitted to avoid visual clutter.

[0068] exist Figure 4B In this configuration, reference element 442b includes a ring and an attachment segment parallel to the axis of segment 440b. When viewed perpendicular to the axis of segment 440b, reference element 442b has an L-shaped or T-shaped projection. Figure 4C In this configuration, reference element 442c includes an open ring disposed at segment 440c. The opening can be aligned with the ring within any suitable angular range. Figure 4D In the middle, the reference element 442d set at section 440d has a ring element and a point or ball element separate from the ring element.

[0069] Figures 4A to 4D Reference elements 442a to 442d are examples of reference elements, similar to the rigid body 202 in Figure 2, having six DOFs with uniquely defined positions and orientations in space. It should be noted that reference elements having asymmetrical shapes relative to an axis on the flexible elongated body and at specific locations along the length of the flexible elongated body can include many other configurations of reference elements. For example, portions of reference elements 442a to 442d (e.g., loops, open loops, ellipses, points, and / or lines) can be reconfigured into many alternative reference elements without introducing rotational symmetry.

[0070] Figure 5A and Figure 5B An example configuration of a baseline device that can be detachably attached to a flexible elongation device is schematically shown. Figure 5A In this configuration, reference element 542a can be fixedly attached to sleeve 545, and sleeve 545 can then be detachably attached to flexible elongation device 540a. Figure 5B In this configuration, a portion of the reference element 542b is fixedly attached to probes 546a and 546b, which can then be detachably attached (e.g., inserted or passed through) to the flexible elongation device. In some examples, the sleeve 545 or probe 546a can be mechanically or optically registered to the corresponding flexible elongation device 540a or 540b. The registration of the detachable sleeve 545 and / or the detachable probes 546a and 546b can be relative to the distal end of the corresponding flexible elongation device 540a and 540b or any other suitable point.

[0071] Figure 6AThis is a block diagram of a method 600a for implementing the technology of this disclosure. At block 610, method 600a includes receiving, by one or more processors (e.g., processing unit 120), indications of the positions of a first sensor and a second sensor (e.g., sensors 144a and 144b, 344c and 344d, 344e and 344f, 344g and 344h, or 344i and 344j) in a first coordinate system.

[0072] At frame 620a, method 600a includes an indication of the location of a reference for the flexible elongation device in a second coordinate system, determined by one or more processors based on intraoperative imaging data. The reference may be positioned at a separation distance (e.g., L1) from the first sensor (e.g., sensor 344c) at the flexible elongation device. Method 600a may more generally include the use of the above reference. Figure 2A and Figure 2B , Figures 3A to 3E and Figures 4A to 4D Any technique discussed, or any appropriate number of DOFs for determining any reference element set at the flexible elongation device based on intraoperative images.

[0073] An appropriately configured imaging unit (e.g., imaging unit 110) can generate intraoperative imaging data. Intraoperative imaging data may include, for example, cone-beam computed tomography (CBCT) data. For this purpose, the imaging unit may include a C-arm CBCT imaging system. In some examples, intraoperative imaging data may include fluorescence fluoroscopy X-ray data (e.g., generated by a C-arm X-ray device), tomographic synthesis data (reconstructed from 2D X-ray images (e.g., fluorescence fluoroscopy images) into 3D volume), and / or MRI data. Additionally or alternatively, intraoperative imaging data may be obtained using thermal imaging techniques, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, or any other suitable imaging technique.

[0074] At block 630a, method 600a may include registering the first coordinate system to the second coordinate system by one or more processors, at least in part, based on indications of the positions of the first and second sensors in the first coordinate system, indications of the position of the reference in the second coordinate system, and a separation distance. For this purpose, the method may use, for example, the reference described above. Figure 2A and Figure 2B , Figures 3A to 3E and Figures 4A to 4D Any of the technologies discussed.

[0075] Registration can be based on any suitable number of sensors (e.g., 2, 3, 4, 5, etc.). As mentioned above, the sensors can have designated positions along the flexible elongation device. For example, a designated position of the sensor along the flexible elongation device relative to the distal tip of the device can indicate the separation distance between any pair of sensors. The separation distance along the flexible elongation device can effectively encode or identify the sensors. Furthermore, the reference can be at least partially integrated into the sensors. That is, at least a portion of one or more sensors can be identifiable and locatable within the intraoperative imaging data.

[0076] In the example, the system can use three sensors (each with 3 DOFs) and match the position of each sensor in the sensor coordinate system with its corresponding position in the intraoperative imaging coordinate system. To do this, a portion of the flexible elongation device with the three sensors can be segmented from the intraoperative image, and the sensor positions can be identified using reference markers. Registration can be performed using Iterative Closest Point (ICP) or another suitable algorithm.

[0077] In another example, a 5 DOF sensor and a 3 DOF sensor at a known location relative to the tip of the flexible elongation device can be matched with the pose of the corresponding segment along the flexible elongation device extracted from intraoperative images.

[0078] In some examples, a portion of the reference may have poor resolution in the intraoperative imaging data. The reference may include redundant structures at different points along the flexible elongation device, thus fully defining the image coordinates around these points. In some examples, the system may compute several coordinate mappings between a first and a second coordinate system. The system may then generate coordinate registration based on a weighted mapping function (e.g., based on the quality or confidence of each coordinate mapping), or generate spatially varied (e.g., non-rigid) registration between the first and second coordinate systems. Typically, additional sensors and reference elements can improve the robustness of the registration method, taking into account noisy or poorly resolved intraoperative imaging and / or sensor data.

[0079] At block 640a, method 600a may optionally include at least in part generating a graphical user interface (GUI) based on registering a first coordinate system to a second coordinate system on a display unit and by one or more processors, the GUI displaying the positions of the first sensor, the second sensor, and the reference in the joint coordinate system. Generating the GUI may include initially generating the GUI and / or updating the GUI based on a previously generated GUI. Method 600a may also include generating a measure of registration uncertainty, comparing the measure to a threshold, and generating an alarm when the measure of uncertainty exceeds the threshold. Additionally or alternatively, method 600a may generate one or more alarms indicating the position, orientation, and / or pose of any segment of the flexible elongation device.

[0080] Method 600a can be implemented by one or more processors (e.g., processing unit 120). Instructions for implementing method 600a by one or more processors can be stored on a tangible, non-transitory computer-readable medium (CRM).

[0081] Figure 6B This is a block diagram of an alternative method 600b for implementing the technology of this disclosure. At block 610b, method 600b includes receiving, by one or more processors (e.g., of processing unit 120), an indication of the position of a position sensor (e.g., any one of sensors 144a and 144b, 344c and 344d, 344e and 344f, 344g and 344h, or 344i and 344j) located at a flexible elongation of a medical device (e.g., device 140 or any device represented by segments 340a to 340g, 440a to 440d, or 540a, 540b) in a first coordinate system. Optionally, indications of the positions of one or more additional position sensors may be received at block 610b.

[0082] At frame 620b, method 600b includes determining, based on intraoperative imaging data and by one or more processors, the position in a second coordinate system of a reference positioned at the flexible elongator and having a rotationally asymmetric shape relative to the axis of the flexible elongation device. Furthermore, considering the possibility that portions of the reference may be poorly resolved in the imaging data, the reference may have multiple rotationally asymmetric elements to improve the robustness of coordinate registration. The reference may be positioned at the flexible elongator at a separation distance from the sensors (e.g., any one of sensors 144a and 144b, 344c and 344d, 344e and 344f, 344g and 344h, or 344i and 344j). Method 600b may more generally include using the above-described reference... Figure 2A and Figure 2B , Figures 3A to 3E and Figures 4A to 4DAny technique discussed here refers to determining any suitable number of DOFs for any reference element positioned at the flexible elongation device based on intraoperative images. Intraoperative imaging data can be obtained, for example, as shown in Figure 1 and... Figure 6A Obtained as described.

[0083] At block 630b, method 600b may include registering a first coordinate system to a second coordinate system by one or more processors, at least in part based on indications of the position of a position sensor and indications of the position of a reference in a second coordinate system. The reference may include two or more portions physically separated from each other and arranged along the length of the flexible elongating device. For this purpose, the method may use, for example, the references described above. Figure 2A and Figure 2B , Figures 3A to 3E and Figures 4A to 4D Any of the technologies discussed.

[0084] In the example, the 6 DOF sensor can be registered with a reference having a rotationally asymmetric shape relative to the flexible elongation device. The reference can thus provide 6 DOFs at points along the flexible elongation device. In some examples, the reference location is close enough to the flexible elongation device that (e.g., assuming the flexible elongation device is rigid between the reference and the 6 DOF sensor) the position and orientation of the sensor in a first coordinate system and the position and orientation of the reference in a second coordinate system can be registered in both coordinate systems, at least near the sensor.

[0085] At block 640b, method 600b may optionally include: generating a graphical user interface on a display unit and by one or more processors, at least in part based on registering a first coordinate system to a second coordinate system, the graphical user interface displaying the position of the sensor in the joint coordinate system and the position of the reference in the joint coordinate system. Generating the graphical user interface may include initially generating the graphical user interface and / or updating the graphical user interface based on a previously generated graphical user interface. Method 600b may also include generating a measure of registration uncertainty, comparing the measure to a threshold, and generating an alarm when the measure of uncertainty exceeds the threshold. Additionally or alternatively, method 600b may generate one or more alarms indicating the position, orientation, and / or pose of any segment of the flexible elongation device.

[0086] Method 600b can be implemented by one or more processors (e.g., processing unit 120). Instructions for implementing method 600b by one or more processors can be stored on a tangible, non-transitory computer-readable medium (CRM).

[0087] Figures 7 to 9BA diagram is depicting a medical system, in some examples, that can be used to manipulate a medical device including a flexible elongation device according to any of the methods and systems described above. For example, each reference above to "system" may refer to the system discussed below (e.g., system 700) or its subsystem.

[0088] Figure 7 This is a simplified diagram of a medical system 700 based on some examples. Medical system 700 may include at least a portion of system 100 described with reference to FIG1. ​​Medical system 700 may be applicable to procedures such as surgery, diagnosis (e.g., biopsy), or treatment (e.g., ablation, electroporation, etc.). While some examples of these procedures are provided herein, any references to medical or surgical instruments and methods are non-limiting. The systems, instruments, and methods described herein can be used with animal or human cadavers, animal carcasses, portions of human or animal anatomy, non-surgical diagnostics, and in industrial systems, general-purpose or special-purpose robotic systems, general-purpose or special-purpose remote operating systems, or robotic medical systems.

[0089] like Figure 7 As shown, the medical system 700 may include a manipulator assembly 702 that controls the operation of a medical device 704 when performing various procedures on a patient (e.g., patient P on table T in FIG. 1). The medical device 704 may include the flexible extension device 140 of FIG. 1. The medical device 704 may extend into an internal portion of the patient P's body via an opening within the patient P's body. The manipulator assembly 702 may be a remotely operated, non-remotely operated, or hybrid remotely and non-remotely operated assembly having one or more degrees of freedom of motion that can be electrically operated and / or one or more degrees of freedom of motion that can be non-electrically operated (e.g., manually operated). The manipulator assembly 702 may be mounted to and / or positioned near the patient table T. A master assembly 706 enables an operator O (e.g., a surgeon, clinician, physician, or other user as described above) to control the manipulator assembly 702. In some examples, the master assembly 706 enables the operator O to view the procedure site or other graphical or information displays. In some examples, the manipulator assembly 702 may be excluded from the medical system 700, and the device 704 may be directly controlled by operator O. In some examples, the manipulator assembly 702 may be manually controlled by operator O. Direct operator control may include various handles and operator interfaces for handheld operation of the device 704.

[0090] The main component 706 may be located at a surgeon's console near the patient table T where the patient P is located (e.g., in the same room as the patient table T), such as on the side of the patient table T. In some examples, the main component 706 is located away from the patient table T, such as in a different room or a different building. The main component 706 may include one or more control devices for controlling the manipulator component 702. The control devices may include any number of various input devices, such as joysticks, trackballs, rollers, steering pads, buttons, data gloves, trigger guns, manual controllers, voice recognition devices, motion or presence sensors, etc.

[0091] Manipulator assembly 702 supports medical device 704 and may include a kinematic structure of links providing a setting structure. Links may include one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place) and / or one or more servo-controlled links (e.g., one or more links that can be controlled in response to commands, for example, from control system 712). Manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on medical device 704 in response to commands, for example, from control system 712. Actuators may include a drive system that moves medical device 704 in various ways when coupled to it. For example, one or more actuators may advance medical device 704 into a natural or surgically generated anatomical opening. Actuators may control engagement of medical device 704, for example, by moving the distal end (or any other part) of medical device 704 in multiple degrees of freedom. These degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, and Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, and Z Cartesian axes). One or more actuators may control the rotation of the medical device about its longitudinal axis. Actuators may also be used to move the engageable end effector of the medical device 704 (e.g., for grasping tissue in the jaws of a biopsy device, etc.), or may be used to move or otherwise control tools inserted within the medical device 704 (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.).

[0092] The control system 704 may include at least a portion of the processing unit 120. Additionally or alternatively, the control system 704 may be communicatively connected to the processing unit 120. In some examples, the output of the processing unit 120 according to the above-described technology may enable the control system 704 to autonomously (without input from operator O) control certain movements of the device 104.

[0093] The medical system 700 may include a sensor system 708 (which may include at least a portion of a sensor unit 115), the sensor system 708 having one or more subsystems for receiving information about the manipulator assembly 702 and / or the medical device 704. Such subsystems may include: a position sensor system (e.g., using an electromagnetic (EM) sensor or other type of sensor to detect position or location); a shape sensor system for determining the position, orientation, velocity, rate, pose, and / or shape of one or more segments and / or distal ends of the flexible body of the medical device 704; a visualization system (e.g., using a color imaging device, infrared imaging device, ultrasound imaging device, X-ray imaging device, fluorescence imaging device, computed tomography (CT) imaging device, magnetic resonance imaging (MRI) imaging device, or some other type of imaging device or imaging unit 110) for capturing images, for example, from the distal end of the medical device 704 or from some other location; and / or an actuator position sensor (e.g., a resolver, encoder, potentiometer, etc.) describing the rotation and / or orientation of actuators controlling the medical device 704.

[0094] It should be noted that the position and orientation of the sensors in sensor system 708 can be determined in the sensor coordinate system. In some examples, the sensor coordinate system is integrated with or the same as the coordinate system of the manipulator assembly 702.

[0095] The medical system 700 may include a display system 710 for displaying images or representations of the procedure site and the medical device 704. The display system 710 and the main component 706 may be oriented such that a physician O can control the medical device 704 and the main component 706 using telepresent perception. The display system 710 may include at least a portion of a display unit 130.

[0096] In some examples, medical device 704 may include a visualization system that includes an image capture component that records simultaneous or real-time images of the procedure site and provides the images to the operator O via one or more displays of display system 710. The image capture component may include various types of imaging devices. The simultaneous images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the anatomical procedure site. The visualization system may acquire intraoperative images in image system coordinates, different from those in sensor system coordinates. In some examples, the visualization system may include an endoscope component that may be integrally or detachably coupled to medical device 704. Additionally or alternatively, a separate endoscope attached to a separate manipulator assembly may be used with medical device 704 to image the procedure site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, such as control system 712.

[0097] Display system 710 can also display images of the process site and medical device, which can be captured by a visualization system. In some examples, medical system 700 provides operator O with a telepresented perception. For example, an image captured by an imaging device at the distal portion of medical device 704 can be presented by display system 710 to provide operator O with a perception of the distal portion of medical device 704. Inputs provided by operator O to master component 706 can move the distal portion of medical device 704 in a manner corresponding to the nature of the input (e.g., the distal end turns to the right when the trackball is rolled to the right), and cause a corresponding change in the viewing angle of the image captured by the imaging device at the distal portion of medical device 704. Thus, operator O's telepresent perception is maintained when medical device 704 is moved using master component 706. Operator O can manipulate the hand controls of master component 706 and medical device 704 as if viewing a workspace in a substantially real-world setting, simulating the experience of physically manipulating medical device 704 from within the patient's anatomy.

[0098] In some examples, the display system 710 can present virtual images of the procedure site, created using image data recorded preoperatively (e.g., before the procedure performed by the medical device system 200) or intraoperatively (e.g., simultaneously with the procedure performed by the medical device system 200), such as image data created in the manner of: computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The virtual images can include two-dimensional, three-dimensional, or higher-dimensional images (e.g., including information based on time or rate). In some examples, one or more models are created based on preoperative or intraoperative image datasets, and virtual images are generated using one or more models.

[0099] In some examples, for the purpose of image-guided medical procedures, the display system 710 can display a virtual image generated based on the position of the tracking medical device 704. For example, the tracking position of the medical device 704 can be registered with a model generated using preoperative or intraoperative images (e.g., a dynamic reference), where different parts of the model correspond to different locations of the patient's anatomy. As the medical device 704 moves through the patient's anatomy, the registration is used to determine the parts of the model corresponding to the position and / or viewpoint of the medical device 704, and a virtual image is generated using the determined parts of the model. This presents the operator O with a virtual image of the internal process site corresponding to the tracking position of the medical device 704, based on the viewpoint of the medical device 704.

[0100] The display system 710 may include a display unit 130, and can be referenced above. Figure 2A The technique described in Figure 6 is based on registering the sensor coordinate system with the imaging coordinate system to display an image in the joint coordinate system, including the position, orientation, and / or pose of the medical device 704.

[0101] The medical system 700 may also include a control system 712, which may include a processing circuitry (e.g., processing unit 120) that implements some or all of the methods or functions discussed herein. The control system 712 may include at least one memory and at least one processor for controlling the operation of the manipulator assembly 702, medical device 704, main assembly 706, sensor system 708, and / or display system 710. The control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing instructions) that, when executed by at least one processor, configure one or more processors to implement some or all of the methods or functions discussed herein. Although the control system 712... Figure 7While shown as a single block, control system 712 may include two or more separate data processing circuits, with some processing performed at manipulator component 702, others at main component 706, and so on. In some examples, control system 712 may include other types of processing circuitry systems, such as application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs). Control system 712 may be implemented using hardware, firmware, software, or a combination thereof.

[0102] In some examples, the control system 712 may receive feedback from the medical device 704, such as force and / or torque feedback. In response to this feedback, the control system 712 may transmit a signal to the main component 706. In some examples, the control system 712 may transmit a signal instructing one or more actuators of the manipulator component 702 to move the medical device 704. In some examples, the control system 712 may transmit information about the feedback to a display system 710 for presentation or to perform other types of actions based on the feedback.

[0103] Control system 712 may include a virtual visualization system to provide navigational assistance to operator O when controlling medical device 704 during image-guided medical procedures. Virtual navigation using the virtual visualization system may be based on a preoperative or intraoperative dataset of the acquired anatomical pathways of patient P. Control system 712 or a separate computing device may, alone or in combination with operator input, use programmed instructions to convert recorded images into a model of the patient's anatomy. This model may include a segmented two-dimensional or three-dimensional synthetic representation of parts or entire anatomical organs or regions. The image dataset may be associated with the synthetic representation. The virtual visualization system may obtain sensor data from sensor system 708, which is used to calculate (e.g., approximate) position of medical device 704 relative to the anatomical structures of patient P. Sensor system 708 may be used to register and display medical device 704 and images recorded preoperatively or intraoperatively. For example, PCT disclosure WO 2016 / 191298 (published December 1, 2016, entitled "Systems and Methods of Registration for Image Guided Surgery") discloses an example system, which is incorporated herein by reference in its entirety. Additional or alternative land registration can be based on the above references. Figure 2A The techniques discussed in Figure 6.

[0104] During the virtual navigation process, sensor system 708 can be used to calculate the (e.g., approximate) position of medical device 704 relative to the anatomical structure of patient P. This position can be used to generate both a macroscopic (e.g., external) tracking image of the anatomical structure of patient P and a virtual internal image of the anatomical structure of patient P. The system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display the medical device and preoperatively recorded medical images. For example, U.S. Patent No. 8,900,131 (filed May 13, 2011, entitled “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated herein by reference in its entirety.

[0105] The medical system 700 may also include operating and support systems (not shown), such as lighting systems, steering and maneuvering control systems, flushing systems, and / or suction systems. In some examples, the medical system 700 may include more than one manipulator assembly and / or more than one master assembly. The exact number of manipulator assemblies may depend on factors such as the medical procedure and space constraints within the operating room. Multiple master assemblies may be located in the same location or may be positioned in separate locations. Multiple master assemblies allow more than one operator to control one or more manipulator assemblies in various combinations.

[0106] Figure 8A This is a simplified diagram of a medical device system 800 based on some examples. The medical device system 800 includes a flexible elongation device 802 (e.g., device 140) (also referred to as elongation device 802), a drive unit 804, and a medical tool 826, which together serve as an example of a medical device 704 of the medical system 700. The medical system 700 can be a remote operating system, a non-remote operating system, or a hybrid of remote and non-remote operating systems, as shown in reference... Figure 7 As described above. The visualization system 831, tracking system 830, and navigation system 832 are also included. Figure 8A The diagram shows an example component of the control system 712 of the medical system 700. In some examples, the medical device system 800 can be used for non-remotely operated exploration procedures or for procedures involving routine manual operation of medical devices (e.g., endoscopy). The medical device system 800 can be used to collect (e.g., measure) a set of data points corresponding to positions within the anatomical passage of a patient (e.g., patient P).

[0107] The elongation device 802 is coupled to the drive unit 804. The elongation device 802 includes a channel 821 through which a medical instrument 826 can be inserted. The elongation device 802 navigates within the patient's anatomy to deliver the medical instrument 826 to the procedure site. The elongation device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818. In some examples, the flexible body 816 may have an outer diameter of approximately 3 mm. Other flexible bodies may have larger or smaller outer diameters.

[0108] Medical device system 800 may include a tracking system 830 for determining the position, orientation, velocity, rate, pose, and / or shape of a flexible body 816 at its distal end 818 and / or along one or more segments 824 of the flexible body 816, as will be described in further detail below. Tracking system 830 may include one or more sensors and / or imaging devices. The flexible body 816 (e.g., the length between the distal end 818 and the proximal end 817) may include multiple segments 824. Tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, tracking system 830 is... Figure 7 This is part of the control system 712 shown. The tracking system 830 can realize a reference... Figure 1A At least some of the techniques described in Figure 6, and for this purpose, may include Figure 1A At least a portion of the processing unit 120 or related to Figure 1A The processing unit 120 is connected to the communication network.

[0109] The tracking system 830 can use a shape sensor 822 to track the distal end 818 and / or one or more segments 824 of the flexible body 816. It should be noted that the shape sensor 822 can be omitted when utilizing the techniques of this disclosure. The shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., disposed within an internal channel of the flexible body 816 or mounted externally along the flexible body 816). In some examples, the optical fiber may have a diameter of approximately 800 μm. In other examples, the diameter may be larger or smaller. The optical fiber of the shape sensor 822 can form an optical fiber bending sensor for determining the shape of the flexible body 816. Optical fibers including fiber Bragg gratings (FBGs) can be used to provide strain measurements of the structure in one or more dimensions. Various systems and methods applicable to monitoring the shape and relative position of optical fibers in three dimensions are described in U.S. Patent Application Publication No. 2006 / 0013523 (filed July 13, 2005, entitled "Fiber optic position and shape sensing device and method relating thereto"), U.S. Patent No. 7,772,541 (filed March 12, 2008, entitled "Fiber Optic Position and / or Shape Sensing Based on Rayleigh Scatter"), and U.S. Patent No. 8,773,650 (filed September 2, 2010, entitled "Optical Position and / or Shape Sensing"), all of which are incorporated herein by reference in their entirety. In some examples, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering.

[0110] In some examples, other techniques may be used to determine the shape of the flexible body 816. For example, the history of the position and / or pose of the distal end 818 of the flexible body 816 may be used to reconstruct the shape of the flexible body 816 over time intervals, such as when the flexible body 816 advances or retracts within a patient's anatomy. In some examples, the tracking system 830 may alternatively and / or additionally use a position sensor system 820 to track the distal end 818 of the flexible body 816. The position sensor system 820 may be a component of an EM sensor system, wherein the position sensor system 820 includes one or more position sensors. Although the position sensor system 820 is shown proximate to the distal end 818 of the flexible body 816 to track the distal end 818, the number and position of the position sensors in the position sensor system 820 may vary to track different regions along the flexible body 816. In one example, the position sensors include conductive coils that can withstand externally generated electromagnetic fields. Each coil of the position sensor system 820 may generate an induced electrical signal having characteristics that depend on the position and orientation of the coil in relation to the externally generated electromagnetic field. The position sensor system 820 can measure one or more position coordinates and / or one or more orientation angles associated with one or more portions of the flexible body 816. In some examples, the position sensor system 820 can be configured and positioned to measure six degrees of freedom, such as three position coordinates X, Y, and Z, and three orientation angles indicating pitch, yaw, and roll of a reference point. In some examples, the position sensor system 820 can be configured and positioned to measure five degrees of freedom, such as three position coordinates X, Y, and Z, and two orientation angles indicating pitch and yaw of a reference point. Further description of the position sensor system applicable to some examples is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999, entitled "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0111] Based on the above references Figure 2A According to the technology of this disclosure described in FIG6, the processing unit (e.g., processing unit 120) can improve the accuracy of the position obtained by the sensor system 820 by combining the data obtained by the sensor system 820 with the data obtained by the external imaging system (e.g., through the imaging unit 110).

[0112] In some examples, the tracking system 830 may alternatively and / or additionally rely on a set of pose, position, and / or orientation data stored at points for the elongation device 802 and / or medical instrument 826, captured during one or more cycles of alternating movement (e.g., breathing). This stored data can be used to develop shape information about the flexible body 816. In some examples, a series of position sensors (not shown)—such as EM sensors like those in position sensor 820 or some other type of position sensor—can be positioned along the flexible body 816 and used for shape sensing. In some examples, the history of data acquired during the process from one or more of these position sensors can be used to represent the shape of the elongation device 802, particularly where the anatomical passage is typically static.

[0113] Figure 8B This is a simplified diagram of a medical tool 826 within an elongation device 802, based on some examples. The flexible body 816 of the elongation device 802 may include a channel 821 sized and shaped to accommodate the medical tool 826. In some examples, the medical tool 826 may be used for procedures such as diagnostics, imaging, surgery, biopsy, ablation, illumination, irrigation, aspiration, electroporation, etc. The medical tool 826 can be deployed through the channel 821 of the flexible body 816 and operate at a procedure site within an anatomical structure. The medical tool 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, gripper, brush, etc.), an ablation tool (e.g., a laser ablation tool, a radiofrequency (RF) ablation tool, a cryoablation tool, a thermal ablation tool, a heated liquid ablation tool, etc.), an electroporation tool, and / or another surgical, diagnostic, or therapeutic tool. In some examples, the medical tool 826 may include an end effector with a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, etc. Other end effector types can include, for example, forceps, grippers, scissors, sutures, clamps, etc. Other end effectors can also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, etc.

[0114] Medical tool 826 may be a biopsy tool for removing sample tissue or cell samples from a target anatomical location. In some examples, the biopsy tool is a flexible needle. The biopsy tool may also include a sheath that can surround the flexible needle to protect the needle and the inner surface of the channel 821 when the biopsy tool is within the channel 821. Medical tool 826 may be an image capture probe that includes a distal portion having a stereo or single-field-of-view camera that can be positioned at or near the distal end 818 of the flexible body 816 for capturing images (e.g., still or video images). The captured images may be processed by visualization system 831 for display and / or provided to tracking system 830 to support tracking of the distal end 818 of the flexible body 816 and / or one or more segments 824 of the flexible body 816. The image capture probe may include a cable for transmitting the captured image data, the cable being coupled to an imaging device at the distal portion of the image capture probe. In some examples, the image capture probe may include a bundle of optical fibers, such as a fiber optic endoscope, coupled to a closer imaging device, such as the visualization system 831. The image capture probe may be monospectral or multispectral, capturing image data in one or more of the visible, near-infrared, infrared, and / or ultraviolet spectra. The image capture probe may also include one or more light emitters that provide illumination to facilitate image capture. In some examples, the image capture probe may use ultrasound, X-ray, fluoroscopy, CT, MRI, or other types of imaging techniques.

[0115] In some examples, an image capture probe is inserted within the flexible body 816 of the elongation device 802 to facilitate visual navigation of the elongation device 802 to the procedure site, and then the image capture probe is replaced within the flexible body 816 with another type of medical instrument 826 for performing the procedure. In some examples, the image capture probe may be located together with another type of medical instrument 826 within the flexible body 816 of the elongation device 802 to facilitate simultaneous image capture and tissue intervention, for example, within the same channel 821 or in different channels. The medical instrument 826 may advance from an opening in the channel 821 to perform the procedure (or some other function) and then retract into the channel 821 upon completion of the procedure. The medical instrument 826 may be removed from the proximal end 817 of the flexible body 816 or along the flexible body 816 from another optional instrument port (not shown).

[0116] In some examples, the extension device 802 may include integrated imaging capabilities instead of utilizing a removable image capture probe. For example, the imaging device (or fiber bundle) and light emitter may be located at the distal end 818 of the extension device 802. The flexible body 815 may include one or more dedicated channels carrying cables and / or optical fibers between the distal end 818 and the visualization system 831. Here, the medical device system 800 can perform imaging and tooling operations simultaneously.

[0117] In some examples, the medical tool 826 is capable of controlled engagement. The medical tool 826 may house a cable (also referred to as a traction cable), linkage, or other actuation controls (not shown), extending between its proximal and distal ends to controllably bend the distal end of the medical tool 826, such as those discussed herein with respect to the flexible elongation device 802. The medical tool 826 may be coupled to a drive unit 804 and a manipulator assembly 702. In these examples, the elongation device 802 may be excluded from the medical device system 800, or may be a flexible device without controlled engagement. The steerable maneuvering apparatus or tool applicable to some examples is further described in detail in U.S. Patent No. 7,316,681 (filed October 4, 2005, entitled "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent No. 9,259,274 (filed September 30, 2008, entitled "Passive Preload and Capstan Drive for Surgical Instruments"), which are incorporated herein by reference in their entirety.

[0118] The flexible body 816 of the elongation device 802 may also, or alternatively, accommodate a cable, linkage, or other steering control (not shown) extending between the drive unit 804 and the distal end 818 to controllably bend the distal end 818, such as, for example, by Figure 2A The distal end 818 is depicted by a dashed line as shown in Figure 819. In some examples, at least four cables are used to provide independent up-and-down steering maneuvers to control the pitch of the distal end 818 and left-and-right steering maneuvers to control the yaw of the distal end 881. In these examples, the flexible elongation device 802 may be a steerable maneuverable conduit. Examples of steerable maneuverable conduits suitable for some examples are described in detail in PCT Publication WO 2019 / 018736 (published January 24, 2019, entitled "Flexible Elongate Device Systems and Methods"), which is incorporated herein by reference in its entirety.

[0119] In examples where the elongation device 802 and / or medical tool 826 is actuated by a remotely operated component (e.g., manipulator component 702), the drive unit 804 may include a drive input detachably coupled to and receiving power from a drive element (e.g., actuator) of the remotely operated component. In some examples, the elongation device 802 and / or medical tool 826 may include a grasping feature, a manual actuator, or other components for manually controlling the movement of the elongation device 802 and / or medical tool 826. The elongation device 802 may be steerable, or alternatively, it may be non-steerable, without an integrated mechanism for operator control of bending of the distal end 818. In some examples, one or more channels 821 (which may also be referred to as lumens) may be defined by the inner wall of the flexible body 816 of the elongation device 802, through which the medical tool 826 may be deployed and used at a target anatomical location.

[0120] In some examples, medical device system 800 (e.g., extension device 802 or medical tool 826) may include flexible bronchial instruments, such as bronchoscopes or bronchial tubes, for the examination, diagnosis, biopsy, and / or treatment of the lungs. Medical device system 800 may also be adapted to navigate and treat other tissues in any anatomical system of a variety of anatomical systems via naturally or surgically generated access channels, including the colon, intestines, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, etc.

[0121] Information from tracking system 830 can be sent to navigation system 832, where it can be combined with information from visualization system 831 and / or preoperatively acquired models to provide real-time location information to physicians, clinicians, surgeons, or other operators. Tracking system 830, navigation system 832, and visualization system 831 can be used to implement reference... Figure 1A The techniques described in Figure 6 at least partially collaborate to achieve the functionality of system 100. In some examples, real-time location information can be displayed on display system 710 for controlling medical device system 800. In some examples, navigation system 832 can utilize the location information as feedback for locating medical device system 800.

[0122] Figure 9A and Figure 9B This is a simplified diagram based on some examples, including a side view of a medical device mounted on an insertion assembly in patient coordinate space. (e.g.) Figure 9A and Figure 9BAs shown, the surgical environment 900 may include a patient P positioned on a patient table T. Patient P may be stationary within the surgical environment 900 because overall patient movement is restricted by sedation, restraint, and / or other means. Periodic anatomical movements of patient P (including respiratory and cardiac movements) may continue. Within the surgical environment 900, a medical device 904 is used to perform medical procedures, which may include, for example, surgery, biopsy, ablation, illumination, irrigation, aspiration, or electroporation. The medical device 904 may also be used to perform other types of procedures, such as a registration process that associates position, orientation, and / or pose data captured by a sensor system 708 with a desired (e.g., anatomical or systemic) reference frame. The medical device 904 may be, for example, a medical device 704. In some examples, the medical device 904 may include an elongation device 910 (e.g., a catheter) coupled to an instrument body 912. The elongation device 910 may be the elongation device 140 of FIG. 1. The elongation device 910 includes one or more channels sized and shaped to accommodate medical instruments.

[0123] The elongation device 910 may also include one or more sensors (e.g., components of sensor system 708). In some examples, a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912. The proximal point 916 of the shape sensor 914 may move with the instrument body 912, and the position of the proximal point 916 relative to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 914 may measure the shape from the proximal point 916 to another point (e.g., the distal end 918 of the elongation device 910). The shape sensor 914 may be aligned with the elongation device 910 (e.g., disposed within an internal channel or mounted externally). In some examples, the shape sensor 914 may be an optical fiber used to generate shape information of the elongation device 910.

[0124] In some examples, position sensors (e.g., EM sensors) may be incorporated into medical device 904. A series of position sensors may be positioned along the flexible elongation device 910 and used for shape sensing. Position sensors may be used in place of or in conjunction with shape sensor 914, for example, to improve the accuracy of shape sensing or to verify shape information.

[0125] The extension device 910 may accommodate cables, linkages, or other steering control mechanisms that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918. In some examples, at least four cables are used to provide independent up-and-down steering control to control the pitch of the distal end 918 and to provide left-and-right steering control to control the yaw of the distal end 918. The instrument body 912 may include a drive input that is removably coupled to and receives power from a drive element (e.g., an actuator) of the manipulator assembly.

[0126] The instrument body 912 may be coupled to an instrument holder 906. The instrument holder 906 may be mounted to an insertion stage 908 fixed within the surgical environment 900. Alternatively, the insertion stage 908 may be movable but has a known position within the surgical environment 900 (e.g., via a tracking sensor or other tracking device). The instrument holder 906 may be part of a manipulator assembly (e.g., manipulator assembly 702) coupled to the medical device 904 to control movement (e.g., yaw, pitch, and / or roll) and / or insertion movement (e.g., movement along insertion axis A) of the distal end 918 of the extension device 910 in multiple directions. The instrument holder 906 or the insertion stage 908 may include actuators, such as servo motors, for controlling movement of the instrument holder 906 along the insertion stage 908.

[0127] Sensor device 920 (which may be a component of sensor system 708) can provide information about the position of instrument body 912 as it moves along insertion axis A relative to insertion stage 908. Sensor device 920 may include one or more rotary transformers, encoders, potentiometers, and / or other sensors that measure the rotation and / or orientation of actuators controlling the movement of instrument carriage 906, thereby indicating the movement of instrument body 912. In some examples, insertion stage 908 has, for example, […]. Figure 9A and Figure 9B The linear track is shown. In some examples, the insertion stage 908 may have a curved track or a combination of curved track segments and linear track segments.

[0128] Figure 9A The instrument body 912 and instrument holder 906 are shown in the retracted position along the insertion stage 908. In this retracted position, the proximal point 916 is located at position L0 on the insertion axis A. The position of the proximal point 916 can be set to zero and / or other reference values ​​to provide a basic reference (e.g., corresponding to the origin of the desired reference system) to describe the position of the instrument holder 906 along the insertion stage 908. In the retracted position, the distal end 918 of the extension device 910 can be positioned precisely within the inlet orifice of the patient P. Also in the retracted position, data captured by the sensor device 920 can be set to zero and / or other reference values ​​(e.g., I=0). Figure 9BIn this configuration, the instrument body 912 and instrument holder 906 have advanced along the linear track of the insertion stage 908, and the distal end 918 of the extension device 910 has advanced into the patient P. At this advanced position, the proximal point 916 is at position L1 on the insertion axis A. In some examples, rotation and / or orientation of the actuator measured by a sensor device 920 indicating the movement of the instrument holder 906 along the insertion stage 908 and / or by one or more position sensors associated with the instrument holder 906 and / or the insertion stage 908 can be used to determine the position L1 of the proximal point 916 relative to position L0. In some examples, position L1 can also serve as an indicator of the distance or insertion depth of the distal end 918 of the extension device 910 into the channel of the anatomical structure of the patient P.

[0129] One or more components of the examples discussed in this disclosure (e.g., control system 712) can be implemented in software to execute on one or more processors of a computer system. The software may include code that, when executed by one or more processors, configures the processors to perform the various functions discussed herein. The code may be stored in a non-transitory computer-readable storage medium (e.g., memory, magnetic storage device, optical storage device, solid-state storage device, etc.). The computer-readable storage medium may be part of a computer-readable storage device, such as electronic circuitry, a semiconductor device, a semiconductor memory device, a read-only memory (ROM), flash memory, an erasable programmable read-only memory (EPROM), a floppy disk, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded to the computer-readable storage medium for storage via a computer network such as the Internet, an intranet, etc. The code may be executed by any of a variety of centralized or distributed data processing architectures. The programming instructions of the code may be implemented as multiple separate programs or subroutines, or they may be integrated into multiple other aspects of the system described herein. Components of the computing system discussed herein may be connected using wired and / or wireless connections. In some examples, wireless connectivity can use wireless communication protocols such as Bluetooth, Near Field Communication (NFC), Infrared Data Association (IrDA), Home RF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), and Wireless Medical Telemetry Service (WMTS).

[0130] Various general-purpose computer systems can be used to perform one or more processes, methods, or functions described herein. Additionally or alternatively, various special-purpose computer systems can be used to perform one or more processes, methods, or functions described herein. Furthermore, various programming languages ​​can be used to implement one or more processes, methods, or functions described herein.

[0131] While certain examples and illustrations have been described above and shown in the accompanying drawings, it should be understood that these examples and illustrations are merely exemplary and not limited to the specific constructions and arrangements shown and described, as various other alternatives, modifications and equivalents will be understood by those skilled in the art.

Claims

1. A tangible, non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Receives indications of the positions of the first and second sensors, which are located at the flexible elongation device, in the first coordinate system; The indication of the reference position of the flexible elongation device in the second coordinate system is determined based on intraoperative imaging data, wherein... The reference is positioned along the flexible elongation device at a distance from the separation distance of the first sensor; as well as The first coordinate system is registered to the second coordinate system based at least in part on the indications of the positions of the first and second sensors in the first coordinate system, the indications of the position of the reference in the second coordinate system, and the separation distance.

2. The tangible, non-transitory computer-readable medium according to claim 1, wherein, The reference includes distributed references configured to indicate at least two locations along the flexible elongation device, and wherein the instructions, when executed by the one or more processors, also cause the one or more processors to: Based on the intraoperative imaging data, the indications of at least two locations of the distributed reference are determined, and The first coordinate system is registered to the second coordinate system, at least in part based on the determined indications of the at least two locations of the distributed reference.

3. The tangible, non-transitory computer-readable medium according to claim 2, wherein, The reference is configured to indicate at least two distinct distances along the flexible elongation device, and wherein the instruction, when executed by the one or more processors, also causes the one or more processors to: Based on the intraoperative imaging data, indications are determined along the at least two different distances of the flexible elongation device, and The first coordinate system is registered to the second coordinate system, at least in part based on determined indications along the at least two different distances of the flexible elongation device.

4. The tangible, non-transitory computer-readable medium according to any one of claims 1 to 3, wherein, The reference is configured to indicate orientation, and wherein the instructions, when executed by the one or more processors, also cause the one or more processors to: Indication of the reference orientation determined based on the intraoperative imaging data, and The first coordinate system is registered to the second coordinate system, at least in part based on the determined indication of the orientation of the reference.

5. The tangible, non-transitory computer-readable medium according to claim 4, wherein, The reference is configured to have a rotationally asymmetric shape about the axis of the flexible elongation device, and the indication of the orientation of the reference is based at least in part on the rotationally asymmetric shape.

6. The tangible, non-transitory computer-readable medium according to any one of claims 1 to 5, wherein, When executed by the one or more processors, the instructions also cause the one or more processors to generate an alarm at least in part based on registering the first coordinate system to the second coordinate system, the alarm indicating one or more of the positions or orientations of at least a portion of the flexible elongation device.

7. The tangible, non-transitory computer-readable medium according to any one of claims 1 to 6, wherein, The instructions, when executed by the one or more processors, also cause the one or more processors to: Calculate a measure of the uncertainty in registering the first coordinate system to the second coordinate system; The calculated metric is compared with the threshold; as well as An alert is generated, at least in part, based on comparing the calculated metric with the threshold, the alert indicating that the calculated metric exceeds the threshold.

8. The tangible, non-transitory computer-readable medium according to any one of claims 1 to 7, wherein, When executed by the one or more processors, the instructions also cause the one or more processors to generate a graphical user interface on a display unit and at least in part based on registering the first coordinate system to the second coordinate system, the graphical user interface displaying at least a portion of the flexible elongation device including the positions of the first sensor, the second sensor, and the reference in the joint coordinate system.

9. A medical device comprising: A flexible elongated body having an axis; A position sensor is disposed at the flexible elongated body of the medical device, and the position sensor is configured to generate an indication of position in a first coordinate system; as well as A reference is provided at the flexible elongated body of the medical device and has a rotationally asymmetrical shape relative to the axis of the flexible elongated body of the medical device.

10. The medical device according to claim 9, wherein, The reference includes: a ring element that is rotationally symmetrical with respect to the axis of the flexible elongated body of the medical device and a rotationally asymmetrical element with respect to the axis of the flexible elongated body of the medical device.

11. The medical device according to claim 10, wherein, The rotationally asymmetric element comprises one or more points or lines.

12. The medical device according to claim 10 or 11, wherein, The rotationally asymmetric element is physically separated from the ring element.

13. The medical device according to any one of claims 9 to 12, wherein, The reference includes an open ring in a plane substantially perpendicular to the axis of the flexible elongated body of the medical device.

14. The medical device according to any one of claims 9 to 13, wherein, The reference includes metal components.

15. The medical device according to any one of claims 9 to 14, wherein, At least a portion of the reference is disposed at the sleeve of the flexible elongated body that is detachably attached to the medical device.

16. The medical device according to any one of claims 9 to 15, wherein, At least a portion of the reference is located at the probe that is detachably inserted through the flexible elongator of the medical device.

17. The medical device according to any one of claims 9 to 16, wherein, The reference includes at least two elements disposed at the flexible elongated body of the medical device, wherein the position sensor is disposed at the flexible elongated body of the medical device between two of the at least two elements of the reference.

18. A method comprising: One or more processors receive indications of the positions of a first sensor and a second sensor located at the flexible elongation device in a first coordinate system. An indication of the position of a reference for the flexible elongation device in a second coordinate system, determined by the one or more processors based on intraoperative imaging data, wherein the reference is located at the flexible elongation device at a distance from the separation distance of the first sensor; as well as The first coordinate system is registered to the second coordinate system by the one or more processors based at least in part on the indications of the positions of the first and second sensors in the first coordinate system, the indication of the position of the reference in the second coordinate system, and the separation distance.

19. The method according to claim 18, wherein, The reference includes distributed references configured to indicate at least two locations along the flexible elongation device, and the method further includes: Based on the intraoperative imaging data, the indications of at least two locations of the distributed reference are determined, and The first coordinate system is registered to the second coordinate system, at least in part based on the determined indications of the at least two locations of the distributed reference.

20. The method according to claim 19, wherein, The reference is configured to indicate at least two distinct distances along the flexible elongation device, and the method further includes: Based on the intraoperative imaging data, indications are determined along the at least two different distances of the flexible elongation device, and The first coordinate system is registered to the second coordinate system, at least in part based on determined indications along the at least two different distances of the flexible elongation device.

21. The method according to any one of claims 18 to 20, wherein, The reference is configured to indicate orientation, and the method further includes: Indication of the reference orientation determined based on the intraoperative imaging data, and The first coordinate system is registered to the second coordinate system, at least in part based on the determined indication of the orientation of the reference.

22. The method according to claim 21, wherein, The reference is configured to have a rotationally asymmetric shape about the axis of the flexible elongation device, and the indication of the orientation of the reference is based at least in part on the rotationally asymmetric shape.

23. The method according to any one of claims 18 to 22, wherein, When executed by the one or more processors, the instructions also cause the one or more processors to generate an alarm at least in part based on registering the first coordinate system to the second coordinate system, the alarm indicating one or more of the positions or orientations of at least a portion of the flexible elongation device.

24. The method according to any one of claims 18 to 23, further comprising: Calculate a measure of the uncertainty in registering the first coordinate system to the second coordinate system; The calculated metric is compared with the threshold; as well as An alert is generated, at least in part, based on comparing the calculated metric with the threshold, the alert indicating that the calculated metric exceeds the threshold.

25. The method of claim 18, further comprising: A graphical user interface is generated on the display unit and by the one or more processors, at least in part, based on registering the first coordinate system to the second coordinate system. The graphical user interface displays at least a portion of the flexible elongation device, including the positions of the first sensor, the second sensor, and the reference in the joint coordinate system.

26. A medical system comprising: Flexible elongation device; The first and second sensors are installed at the flexible elongation device; A reference is provided at the flexible elongation device at a distance from the separation distance of the first sensor; as well as One or more processors, said one or more processors being configured to: Receive indications of the positions of the first sensor and the second sensor in the first coordinate system; The position of the reference in the second coordinate system is determined based on intraoperative imaging data; as well as Based on the received indications of the positions of the first and second sensors in the first coordinate system, the determined position of the reference in the second coordinate system, and the separation distance, the first coordinate system is registered to the second coordinate system.

27. The medical system according to claim 26, wherein, The reference includes a distributed reference, which is configured to indicate at least two locations along the flexible elongation device, and wherein the one or more processors are configured to: Based on the intraoperative imaging data, the indications of at least two locations of the distributed reference are determined, and The first coordinate system is registered to the second coordinate system, at least in part based on the determined indications of the at least two locations of the distributed reference.

28. The medical system according to claim 27, wherein, The reference is configured to indicate at least two distinct distances along the flexible elongation device, and wherein the one or more processors are configured to: Based on the intraoperative imaging data, indications are determined along the at least two different distances of the flexible elongation device, and The first coordinate system is registered to the second coordinate system, at least in part based on determined indications along the at least two different distances of the flexible elongation device.

29. The medical system according to any one of claims 26 to 28, wherein, The reference is configured to indicate orientation, and wherein the one or more processors are configured to: Indication of the reference orientation determined based on the intraoperative imaging data, and The first coordinate system is registered to the second coordinate system, at least in part based on the determined indication of the orientation of the reference.

30. The medical system according to claim 29, wherein, The reference is configured to have a rotationally asymmetric shape about the axis of the flexible elongation device, and the indication of the orientation of the reference is based at least in part on the rotationally asymmetric shape.

31. The medical system according to any one of claims 26 to 30, wherein, The one or more processors are configured to generate alarms at least in part based on registering the first coordinate system to the second coordinate system, the alarms indicating one or more of the positions or orientations of at least a portion of the flexible elongation device.

32. The medical system according to any one of claims 26 to 31, wherein, The one or more processors are configured to: Calculate a measure of the uncertainty in registering the first coordinate system to the second coordinate system; The calculated metric is compared with the threshold; as well as An alert is generated, at least in part, based on comparing the calculated metric with the threshold, the alert indicating that the calculated metric exceeds the threshold.

33. The medical system according to any one of claims 26 to 32, wherein, When executed by the one or more processors, the instructions also cause the one or more processors to generate a graphical user interface on a display unit and at least in part based on registering the first coordinate system to the second coordinate system, the graphical user interface displaying at least a portion of the flexible elongation device including the positions of the first sensor, the second sensor, and the reference in the joint coordinate system.

34. A tangible, non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: The position sensor located at the flexible elongated body of the medical device is received as an indication of its position in the first coordinate system. Based on intraoperative imaging data, an indication of the position in a second coordinate system of a reference set at the flexible elongator and having a rotationally asymmetrical shape relative to the axis of the flexible elongator; as well as The first coordinate system is registered to the second coordinate system, based at least in part on the position indication of the position sensor and the position indication of the reference in the second coordinate system.

35. The tangible, non-transitory computer-readable medium according to claim 34, wherein, When executed by the one or more processors, the instructions also cause the one or more processors to generate a graphical user interface on a display unit and at least in part based on registering the first coordinate system to the second coordinate system, the graphical user interface displaying at least a portion of the flexible elongation device including the positions of the sensor and the reference in the joint coordinate system.

36. A method comprising: The position of a position sensor located at the flexible elongated body of the medical device in a first coordinate system is received by one or more processors. Based on intraoperative imaging data and determined by one or more processors, an indication of the position in a second coordinate system of a reference located at the flexible elongator and having a rotationally asymmetric shape relative to the axis of the flexible elongator. as well as The first coordinate system is registered to the second coordinate system by the one or more processors, based at least in part on the indication of the position of the position sensor and the indication of the position of the reference in the second coordinate system.

37. The method of claim 36, further comprising: A graphical user interface is generated on the display unit and at least in part based on registering the first coordinate system to the second coordinate system, the graphical user interface displaying at least a portion of the flexible elongation device including the positions of the sensor and the reference in the joint coordinate system.

Images