System and method for registering an instrument to an image using changes in instrument orientation data

Abstract

A system can include a processor and a memory having computer readable instructions stored thereon. When executed by the processor, the computer readable instructions cause the system to record position data of an instrument during an image capture session and determine an instrument position change from the recorded position data. When executed by the processor, the computer readable instructions can also cause the system to compare the instrument position change to a position change threshold and, based on the comparison, determine whether to use image data captured by an imaging system during the image capture session in a registration procedure.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. provisional application 62 / 932,858, filed November 8, 2019, which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to systems and methods for registering an instrument and an image frame of reference. BACKGROUND

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue that is damaged during a medical procedure, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in a patient’s anatomy or through one or more surgical incisions. Through these natural orifices or incisions, an operator can insert a minimally invasive medical tool to reach a target tissue location. The minimally invasive medical tool includes instruments such as therapeutic, diagnostic, biopsy, and surgical instruments. The medical tool can be inserted into an anatomical passageway and navigated toward a site of interest within a patient’s anatomy. Image assisted navigation of the anatomical passageway can be used. Improved systems and methods are needed to accurately perform registration between the medical tool and an image of the anatomical passageway. SUMMARY

[0005] Consistent with some embodiments, a system can include a processor and a memory having computer readable instructions stored thereon. When executed by the processor, the computer readable instructions cause the system to record position data of an instrument during an image capture period and determine an instrument position change from the recorded position data. When executed by the processor, the computer readable instructions can also cause the system to compare the instrument position change to a position change threshold and, based on the comparison, determine whether to use image data captured by an imaging system during the image capture period in a registration procedure.

[0006] Consistent with some embodiments, a non-transitory machine readable medium can include a plurality of machine readable instructions that, when executed by one or more processors associated with a computer-assisted medical system device, are adapted to cause the one or more processors to perform a method that can include recording position data of an instrument during an image capture period and determining an instrument position change from the recorded position data. The performed method can also include comparing the instrument position change to a position change threshold and, based on the comparison, determining whether to use image data captured by an imaging system during the image capture period in a registration procedure.

[0007] Consistent with some embodiments, a system can include a processor and a memory having computer-readable instructions stored thereon. When executed by the processor, the computer-readable instructions can cause the system to record shape data of an instrument positioned in a patient anatomy and compare an instrument shape determined from the shape data to a registration shape threshold. Based on the comparison, the system can determine whether to initiate an image capture procedure to capture an image of the patient anatomy and the instrument.

[0008] According to some embodiments, a clinical system can include a medical system including a shape sensor and an imaging system in communication with the medical system. The clinical system can also include a processor and a memory having computer-readable instructions stored thereon. When executed by the processor, the computer-readable instructions can cause the clinical system to generate a communication signal including synchronization information and transmit the communication signal between the medical system and the imaging system. Based on the synchronization information, the clinical system can synchronize recording of shape data from the shape sensor with recording of image data from the imaging system during an image capture period.

[0009] Other embodiments include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0010] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are intended to provide a further explanation of the present disclosure as claimed. In this regard, additional aspects, features, and advantages of the present disclosure will be apparent from the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1A A simplified diagram of a robotic or teleoperated medical system is illustrated in accordance with some embodiments.

[0012] Figure 1B Communication between a control system and an intraoperative imaging system is illustrated.

[0013] Figure 2 A simplified diagram of a medical instrument system and an intraoperative imaging system is illustrated in accordance with some embodiments.

[0014] Figure 3 A display system displaying an image of a medical instrument registered to an anatomical image is illustrated.

[0015] Figure 4 A method for evaluating a shape of a medical instrument to determine whether an intraoperative imaging procedure should be performed is illustrated.

[0016] Figure 5A method for synchronizing a teleoperated medical system with an imaging system during an image capture session of the imaging system is illustrated.

[0017] Figure 6A A plurality of points forming a shape of a medical instrument are illustrated.

[0018] Figure 6B A detailed cross-sectional view of a plurality of points of Figure 6A

[0019] Figure 7 A method for evaluating use of image data in a registration procedure is illustrated.

[0020] Figure 8 A method for evaluating anatomical motion is illustrated.

[0021] Embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description along with the accompanying drawings. It should be understood that like reference numerals are used to identify like elements throughout the several views of the drawings in which the same or similar elements are identified throughout. The purpose of the drawings is to illustrate embodiments of the present disclosure and not to limit the same. DETAILED DESCRIPTION

[0022] The technology disclosed in this document can be used to register a medical instrument frame of reference to an image frame of reference of an intraoperative image that includes a medical instrument image. Often, anatomical motion can cause the intraoperative image to be too distorted to isolate and segment a catheter. Changes in shape and position of the medical instrument can be evaluated during an image capture session before attempting to register the intraoperative image to the medical instrument. If the shape of the medical instrument changes more than a threshold amount, the captured image can be deemed unsuitable for registration and instructions can be provided to initiate a new image capture procedure.

[0023] ​In some embodiments, the registration techniques of the present disclosure can be used in image-guided medical procedures performed with a teleoperated medical system, as described in further detail below. As shown in FIG. la, a clinical system 10 includes a teleoperated medical system 100 and an intraoperative imaging system 118. The teleoperated medical system 100 generally includes a manipulator assembly 102 for operating a medical instrument system 104 to perform various procedures on a patient P positioned on a surgical table T in a surgical environment 101. The manipulator assembly 102 can be a teleoperated, non-teleoperated, or hybrid teleoperated and non-teleoperated assembly having selected degrees of freedom of motion that can be motorized and / or teleoperated and selected degrees of freedom of motion that can be non-motorized and / or non-teleoperated. A master assembly 106, which can be internal or external to the surgical environment 101, generally includes one or more control devices for controlling the manipulator assembly 102. The manipulator assembly 102 supports the medical instrument system 104 and can optionally include a plurality of actuators or motors that drive inputs on the medical instrument system 104 in response to commands from a control system 112. The actuators can optionally include a drive system that, when coupled to the medical instrument system 104, can advance the medical instrument system 104 into a naturally or surgically created anatomical orifice. Other drive systems can move a distal end of the medical instrument system 104 in multiple degrees of freedom, which can include three degrees of linear motion freedom (e.g., linear motion along X, Y, Z Cartesian axes) and three degrees of rotational motion freedom (e.g., rotation about X, Y, Z Cartesian axes). In addition, the actuators can be used to actuate an articulated end effector of the medical instrument system 104 for grasping tissue in the jaws of a biopsy device and / or the like.

[0024] The teleoperated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and the medical instrument system 104 generated by the sensor system 108 and / or endoscopic imaging system 109. The display system 110 and the master assembly 106 can be oriented such that an operator O can control the medical instrument system 104 and the master assembly 106 with a telepresent perception.

[0025] In some embodiments, the medical instrument system 104 can include components for surgery, biopsy, resection, illumination, irrigation, or aspiration. Optionally, the medical instrument system 104, together with the sensor system 108, can be used to collect (i.e., measure) a set of data points corresponding to a location within an anatomical passageway of a patient, such as the patient P. In some embodiments, the medical instrument system 104 can include components of an imaging system 109, which can include an imaging range assembly or an imaging instrument that records an instant or real-time image of a surgical site and provides the image to an operator, or operator O, through a display system 110. The instant image can be, for example, a two-dimensional or three-dimensional image captured by an imaging instrument positioned within the surgical site. In some embodiments, the imaging system components can be integrally or removably coupled to the medical instrument system 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly can be used with the medical instrument system 104 to image the surgical site. The imaging system 109 can be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which can include the processors of the control system 112.

[0026] The sensor system 108 can include a position / location sensor system (e.g., an electromagnetic (EM) sensor system) and / or a shape sensor system for determining the position, orientation, velocity, speed, pose, and / or shape of the medical instrument system 104.

[0027] The teleoperated medical system 100 can also include a control system 112. The control system 112 includes at least one memory 116 and at least one computer processor 114 for effecting control between the medical instrument system 104, the master assembly 106, the sensor system 108, the endoscopic imaging system 109, and the display system 110. The control system 112 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing instructions) to implement some or all of the methods described in accordance with aspects disclosed herein, including instructions for providing information to the display system 110.

[0028] The control system 112 can optionally further include a virtual visualization system to provide navigation assistance to the operator O in controlling the medical instrument system 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based on a reference to a preoperative or intraoperative data set of the acquired anatomical passageway. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like.

[0029] The intraoperative imaging system 118 can be disposed in the surgical environment 101 proximate to the patient P to obtain images of the patient P during a medical procedure. The intraoperative imaging system 118 can provide real-time or near real-time images of the patient P. In some embodiments, the system 118 can be a mobile C-arm cone beam CT imaging system for generating three-dimensional images. For example, the system 118 can be a Dyna CT imaging system from Siemens Corporation of

[0030] FIG. lb illustrates communication between the control system 112 and the intraoperative imaging system 118. In some embodiments, the control system 112 includes a communication device 120 and a clock 122. While the control system 112 is shown as a single block in the simplified schematic of FIGS. la and lb, the control system 112 can include multiple processors, memories, communication devices, and clocks. Further, components of the control system 112 can be distributed throughout the medical system 100, including at the manipulator assembly 102, the instrument system 104, and the master assembly 106. In some embodiments, the intraoperative imaging system includes a processor 124, a memory 126, a communication device 128, and a clock 130. The processor 124 is configured to execute, for example, programmed instructions stored on the memory 126 to implement some or all of the methods described in accordance with aspects disclosed herein. The clocks 122, 130 can include any type of digital clock, analog clock, software-based clock, or other timing device. The communication devices 120, 128 can include an information transmitter, an information receiver, an information transceiver, or a combination of transmitting or receiving devices that enable wired or wireless communication between the imaging system 118 and the control system 112 and / or between the clocks 122, 130. The communication devices 120, 128 can be used to exchange information between the two systems, including, for example, clock signals, start and stop signals, image data signals, patient data signals, and sensor data signals.

[0031] Figure 2 A surgical environment 200 is illustrated having a surgical reference frame (X S , Y S , Z S ) with a patient P positioned on an operating table T. The patient P can be stationary within the surgical environment as the patient’s overall movement is limited by sedation, restraint, and / or other means. Periodic anatomical motion including respiration and cardiac motion of the patient P can continue unless the patient is asked to hold their breath to temporarily suspend respiratory motion. Within the surgical environment 200, a medical instrument reference frame (X M , Y M , Z MA medical instrument 204 (e.g., medical instrument system 104) of the medical instrument 204 is coupled to an instrument carriage 206. In this embodiment, the medical instrument 204 includes an elongated device 210, such as a flexible catheter, coupled to an instrument body 212. The instrument carriage 206 is mounted to an insertion table 208 that is fixed within the surgical environment 200. Alternatively, the insertion table 208 can be movable but have a known position within the surgical environment 200 (e.g., via tracking sensors or other tracking devices). In these alternatives, the medical instrument reference frame is fixed or otherwise known relative to the surgical reference frame. The instrument carriage 206 can be a component of a teleoperational manipulator assembly (e.g., teleoperational manipulator assembly 102) that is coupled to the medical instrument 204 to control insertion motion (i.e., motion along the axis A) and, optionally, motion of a distal end 218 of the elongated device 210 in multiple directions (including yaw, pitch, and roll). The instrument carriage 206 or the insertion table 208 can include actuators, such as servo motors (not shown), that control motion of the instrument carriage 206 along the insertion table 208.

[0032] In this embodiment, the sensor system (e.g., sensor system 108) includes a shape sensor 214. The shape sensor 214 may include an optical fiber extending within and aligned with the elongated device 210. In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor 214 forms an optical fiber bending sensor for determining the shape of the elongated device 210. Alternatively, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in one-dimensional or multi-dimensional structures. Various systems and methods for monitoring the shape and relative orientation of optical fibers in three dimensions are described in U.S. Patent Application No. 11 / 180,389 (filed July 13, 2005) (disclosing "Fiber optic position and shape sensing device and method relating thereto"), U.S. Patent Application No. 12 / 047,056 (filed July 16, 2004) (disclosing "Fiber-optic shape and relative position sensing"), and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing "Optical Fibre Bend Sensor"), all of which are incorporated herein by reference in their entirety. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the conduit may be determined using other techniques. For example, the history of the distal orientation of the elongated device 210 may be used to reconstruct the shape of the elongated device 210 over time intervals.

[0033] like Figure 2 As shown, the device body 212 is coupled and fixed relative to the device holder 206. In some embodiments, a fiber optic shape sensor 214 is fixed at a proximal point 216 on the device body 212. In some embodiments, the proximal point 216 of the fiber optic shape sensor 214 may move with the device body 212, but the position of the proximal point 216 may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 214 is positioned in a medical device reference frame (X). M Y M Z M The shape is measured from the proximal point 216 to another point (e.g., the distal end 18 of the elongated device 210).

[0034] The elongate device 210 includes a passageway (not shown) sized and shaped to receive a medical instrument 222. In some embodiments, the medical instrument 222 can be used for procedures such as surgery, biopsy, resection, illumination, irrigation, or aspiration. The medical instrument 222 can be deployed through the elongate device 210 and used at a target location within the anatomy. The medical instrument 222 can include, for example, an image capture probe, a biopsy instrument, a laser resection fiber, and / or other surgical, diagnostic, or therapeutic tools. The medical instrument 222 can be advanced from the distal end 218 of the elongate device 210 to perform a procedure and then retracted into the passageway when the procedure is complete. The medical instrument 222 can be removed from the proximal end of the elongate device 210 or from another optional instrument port (not shown) along the elongate device 210.

[0035] The elongate device 210 can also house cables, links, or other steering controls (not shown) to controllably bend the distal end 218. In some examples, at least four cables are used to provide independent “up-down” steering to control the pitch of the distal end 218 and “left-right” steering to control the yaw of the distal end 218.

[0036] The position measurement device 220 provides information about the position of the instrument body 212 as the instrument body 212 is moved along the insertion axis A on the insertion stage 208. The position measurement device 220 can include resolvers, encoders, potentiometers, and / or other sensors that determine the rotation and / or orientation of actuators that control the motion of the instrument carriage 206 and thus the instrument body 212. In some embodiments, the insertion stage 208 is linear, while in other embodiments, the insertion stage 208 can be curved or have a combination of curved and linear sections.

[0037] The intraoperative imaging system 230 (e.g., the imaging system 118) is arranged near the patient P to obtain three-dimensional images of the patient as the elongate device 210 extends within the patient. The intraoperative imaging system 230 can provide real-time or near real-time images of the patient P.

[0038] In some embodiments, another component of medical device 204 or a remotely operated medical system registered to medical device 204 may include device clock 224 (e.g., clock 122 of distributed control system 112). Imaging system 230 may include imaging clock 226 (e.g., clock 130). Clocks 224 and 226 may be time-synchronized according to a predetermined schedule or in response to a synchronization initiation event generated by a user, control system, or synchronization system. In some embodiments, clocks 224 and 226 may be components of a synchronization system, which may be a centralized or distributed system, further including a server, wired or wireless communication network, communication equipment, or other components for performing synchronization algorithms and protocols. In some embodiments, another component of medical device 204 or a remotely operated medical system registered to medical device 204 may include communication device 228 (e.g., communication device 120 of distributed control system 112). Imaging system 230 may include communication device 232 (e.g., communication device 128 of imaging system 118).

[0039] In some embodiments and referring to Figure 3 This allows for image-guided surgical procedures, where display system 300 (e.g., display system 110) can display a virtual navigation image 302, which has an image reference frame (X). I Y I Z I In the image reference frame (X) I Y I Z I The image 304 of the medical device 204 is registered (i.e., dynamically referenced) with an anatomical model 306 of the patient P derived from preoperative and / or intraoperative image data. In some embodiments, the virtual navigation image can present a virtual image of the internal surgical site to the physician O from the viewpoint of the medical device 204. In some examples, the viewpoint may be from the distal end of the medical device 204. In some examples, the medical device 204 may not be visible in the virtual image.

[0040] Generating a synthetic virtual navigation image 302 involves using an image reference frame (X). I Y I Z I Registration to the surgical reference system (X) S Y S Z S ) and / or medical device reference system (X M Y M Z MThis registration can be rotated, translated, or otherwise manipulated using rigid or non-rigid transformation points associated with the segmented instrument shape from the image data and points associated with the shape data from the instrument shape sensor 214. Such registration between the image and instrument reference frames can be achieved, for example, by using a point-based Iterative Closest Point (ICP) technique or another point cloud registration technique, which is described in U.S. Provisional Patent Applications Nos. 62 / 205,440 and 62 / 205,433, incorporated herein by reference.

[0041] Figure 4 The illustration depicts a method 400 for evaluating the shape of a medical device to determine whether an intraoperative imaging procedure should be performed (particularly when the implementation of the imaging procedure would expose the patient P to radiation). Shape data from a medical device configured with curvature in each of the three dimensions may be particularly useful in three-dimensional registration with anatomical image data compared to a medical device configured in a straight line.

[0042] At process 402, when the device is located within the anatomical structure of the patient P, shape data of the device (e.g., medical device systems 104, 204) is recorded. For example, shape data collected from shape sensor 214 can provide shape information for device 204, including orientation and orientation information along multiple points of device 204.

[0043] At process 404, the recorded instrument shape data is compared with a registration shape threshold. In some embodiments, the registration shape threshold may be a curvature threshold, which corresponds to the shape of a bend included in at least one or two of the three dimensions in a three-dimensional environment. In some embodiments, the registration shape threshold may be the shape of a bend included in all three dimensions of the three-dimensional environment. In this example, any viewing plane in the three-dimensional environment will include a bend shape that allows for full three-dimensional registration.

[0044] At process 406, based on comparison, it is determined whether the instrument shape determined from the shape data exceeds the registration shape threshold.

[0045] At process 408, if the recorded instrument shape does indeed exceed the registration shape threshold, the shape of instrument 204 can be considered sufficiently curved to allow image capture. An image capture program can be initiated to capture images of the patient's anatomy and the instrument. For example, a start signal can be sent from the communication device 120 of the control system 112 to the communication device 128 of the imaging system 118 to initiate an image capture program utilizing the imaging system.

[0046] At process 410, if the recorded instrument shape does not exceed the registration shape threshold, the shape of instrument 204 can be considered insufficiently curved or too straight to allow image capture. The image capture procedure is not initiated, and no start signal is sent from control system 112 to imaging system 118. In some embodiments, instructions may be provided to alert the user that the image capture procedure has not been initiated and why. These instructions may be provided, for example, by text or images displayed on a user interface (e.g., display system 110), by audio messages sent to the user, or by other types of communication perceptible to the user. In some embodiments, this may be a text message such as “Image capture not initiated. Instrument curvature does not meet the bending threshold.” In some embodiments, corrective instructions may be further provided. For example, text instructions may be provided to bend the instrument or otherwise reconfigure the instrument until a bending sufficient to exceed the threshold is achieved. Additionally or alternatively, a guide image may be provided to guide the user to bend the instrument to produce a shape as shown in the guide image.

[0047] At process 412, which may be an alternative to process 410, an image capture program can be initiated to capture images of the patient's anatomy and instruments. For example, a start signal can be sent from communication device 120 of control system 112 to communication device 128 of imaging system 118 to initiate an image capture program utilizing the imaging system. The captured images can then be discarded, suppressed, or otherwise not used in the registration process.

[0048] In some embodiments, the location of curvature exceeding a registration shape threshold can be determined. Based on the location of the curvature, it can be determined whether an image of the patient's anatomy will include the curvature. If the imaging system's field of view will not include the curvature, the image capture procedure may not be initiated. The medical device can be rearranged until it forms a curvature exceeding the curvature threshold within the imaging system's field of view. Additionally or alternatively, instructions can be sent to the user to move the imaging system to different imaging orientations or orientations to capture images of the patient's anatomy that will include curvature exceeding the registration shape threshold.

[0049] Figure 5 The illustration depicts a method 500 for synchronizing remote operation of a medical system (e.g., system 100) and an imaging system (e.g., imaging system 230) during an image capture period of an imaging system. During the synchronized image capture period, two different representations of a physical entity (e.g., medical device 204) can be recorded. One representation can be a three-dimensional image of the device generated by the imaging system, and the other representation can be a three-dimensional shape generated by a shape sensor.

[0050] At process 502, a communication signal is generated. The communication signal includes synchronization information, which may include start and / or stop signals, clock signals, synchronization protocol information, image capture period duration information, or other information used to synchronize the medical system and the imaging system. For example, the communication signal may be generated by the communication device 120 of the control system 112 or the communication device 128 of the imaging system.

[0051] At process 504, a communication signal is transmitted between the medical system and the imaging system. For example, the communication signal can be transmitted from communication device 120 to communication device 128 or from communication device 128 to communication device 120.

[0052] At process 506, based on synchronization information from the communication signal, the recording of shape data from the device shape sensor of the medical system and the recording of image data from the imaging system are synchronized during the image acquisition period. Because the medical system and the imaging system are synchronized, image data and shape data are recorded within the same time period. When a medical device including a device shape sensor is located within the patient's imaging anatomy, both shape data and image data can be recorded. As the data acquisition periods of the shape sensor and the imaging system are synchronized, two different representations of the medical device are recorded. Image data including the device shape can be matched with shape sensor data describing the shape of the same device for the exact same time period. As further described below, the matched dataset can be used to establish an image reference frame (X). I Y I Z I ) and Medical Device Reference System (X) M Y M Z M Registration.

[0053] In one embodiment, the clock 122 of the medical system is synchronized with the clock 130 of the imaging system during the image capture period. Communication signals including synchronization information for synchronizing clocks 122 and 130 can be transmitted between the medical system 100 and the imaging system 118. In some examples, the synchronization information includes a signal from clock 122 sent from communication device 120 to communication device 128 for synchronization with clock 130. In other examples, the synchronization information includes a signal from clock 130 sent from communication device 128 to communication device 120 for synchronization with clock 122. Clock synchronization may occur precisely before or simultaneously with the start of the image capture period. During the image capture period, synchronization may occur once or may be repeated once or multiple times. In some embodiments, clock synchronization can be achieved via wired or wireless communication between communication devices, either directly or via a network connection.

[0054] In another embodiment, the medical system 100 synchronizes with the imaging system 118 via a start signal and a stop signal. A communication signal including synchronization information in the form of a start signal can be transmitted between the medical system 100 and the imaging system 118. The start signal can be initiated by, occur simultaneously with, or be triggered by the start of an image capture period during which the imaging system records image data. Receiving or transmitting the start signal can also initiate the recording or marking of a start point in a longer record of shape data of the instrument system 104 of the medical system 100. At the end of the image capture period, another communication signal including synchronization information in the form of a stop signal can be transmitted between the medical system 100 and the imaging system 118. The stop signal can be initiated by, occur simultaneously with, or be triggered by the end of an image capture period during which the imaging system records image data. Receiving or transmitting the stop signal can also terminate the recording or marking of an end point in a longer record of shape data of the instrument system 104 of the medical system 100. In some examples, start and stop signals are sent from communication device 128 to communication device 120 to enable and disable synchronization of imaging system 118. In some examples, start and stop signals are sent from communication device 120 to communication device 128 to enable and disable synchronization of medical system 100.

[0055] In some embodiments, after determining to initiate image capture (such as...) Figure 4 As described in [the document], and after the medical system and imaging system are synchronized (as described in [the document]). Figure 5 As described in [the document], it can be determined whether image data is suitable for use in registration. Figure 7 The illustration depicts a method 600 for evaluating anatomical image data to be used in a registration procedure. Typically, anatomical motion can cause intraoperative anatomical image data to be too distorted to isolate and segment medical devices. Before attempting to register the intraoperative images to the medical device, changes in the shape and orientation of the medical device can be evaluated during the image capture period. If the shape change of the medical device exceeds a threshold amount, the captured images can be considered unsuitable for registration, and instructions can be provided to initiate a new image capture procedure.

[0056] At process 602, orientation data of the instrument (e.g., medical device systems 104, 204) is recorded during the image capture period of the imaging system. In some embodiments, the image capture period corresponds to the time period during which the intraoperative imaging system 230 is activated to acquire and record image data of the patient P. During this period, orientation data of the instrument 204 located in the patient P can be recorded. For example, shape data collected from shape sensor 214 can provide orientation information of the instrument 204 and in the medical device reference frame (X). M YM Z M Multiple points along device 204 in the medical device reference system (X) M Y M Z M ) relative to the surgical reference frame (X S Y S Z S It is known that during this time period, instrument 204 may be unaffected by commanded movements (e.g., advance or bend as commanded by the operator), but may be affected by anatomical movements from breathing, cardiac activity, or other active or inactive patient movements. For example, when instrument 204 is positioned within the anatomical structure of patient P, an image scan can be performed using intraoperative imaging system 230 during the image capture period, unaffected by commanded movements.

[0057] At step 604, the change in instrument orientation during the image capture period is determined from the recorded orientation data. (Reference) Figure 6A and Figure 6B In some embodiments, images corresponding to the medical device reference frame (X) are collected during the image capture period. M Y M Z M The orientation and orientation data of a set of points / point sets S representing the shape of instrument 204 in the image. Due to anatomical motion, point S may have a configuration S at time T1 during the image capture period. T1 And it can have configuration S at time T2 during the image capture period. T2 Therefore, the shape and orientation of instrument 204 can change due to anatomical motion during the image capture period. The magnitude of the change can be quantified in any of several ways. For example, the change can be quantified as the maximum change in orientation data. Figure 6B The diagram illustrates segments of a point set S. The distance D1 between point A in set S at times T1 and T2 can be the maximum change at point A. The distance D2 between point B in set S at times T1 and T2 can be the maximum change at point B. In some embodiments, the orientation change of points in the segment closest to the distal end of instrument 204 may be of primary interest in determining the maximum change. In some embodiments, the instrument orientation change can be determined from the average orientation change of points in point set S. In some embodiments, the instrument orientation change can be determined from the standard deviation of the orientation data of points in point set S. The determined orientation change can be based on the change of the entire set of points S or the change of segments within the set of points S. The orientation change can be determined in one dimension, two dimensions, three dimensions, or a combination thereof.

[0058] At process 406, the determined azimuth change is compared to an azimuth change threshold. For example, if the determined azimuth change is 1 cm and the azimuth change threshold is approximately 2 cm, the azimuth change has not reached the threshold. If the determined azimuth change is 3 cm and the azimuth change threshold is 2 cm, the azimuth change has exceeded the threshold. In some embodiments, the azimuth change threshold may be established in one, two, or three dimensions, or a combination thereof, having the same or different thresholds in different dimensions. In some embodiments, the comparison may also include a comparison of orientation and shape. In various embodiments, the azimuth change threshold may be greater than or less than 2 cm. In various embodiments, the azimuth change threshold may be a threshold based on, for example, the maximum azimuth change, the average azimuth change, or the standard deviation of the azimuth change.

[0059] At step 608, it is determined whether the image data captured by the imaging system during the image capture period can be used in the registration procedure to establish an image reference frame (X). I Y I Z I Registration to the medical device reference system (X) M Y M Z M ) and / or surgical reference system (X S Y S Z S If it is determined that instrument 204 has moved too much (e.g., exceeding the orientation change threshold), the difference between image data from intraoperative imaging system 230 and shape data from instrument 204 may prevent the image reference frame (X) from being positioned. I Y I Z I Registration to the medical device reference system (X) M Y M Z M ) and / or surgical reference system (X S Y S Z SIn such cases, it will be determined that the image data will not be used for the registration procedure. For example, if the instrument orientation change is 3 cm, exceeding the 2 cm orientation change threshold, this may indicate that the instrument 204 has moved too much during the image capture period, and the image data generated by the intraoperative imaging system 230 may be too distorted to provide accurate registration with the instrument shape data S collected during the image capture period. However, if the instrument orientation change is 1 cm, below the 2 cm orientation change threshold, this may indicate that the movement of the instrument 204 is within acceptable limits, and the image data generated by the intraoperative imaging system 230 can be used for registration. In some embodiments, control signals sent to the instrument may be evaluated to determine whether a commanded movement, such as an operator-commanded advance or bend, has occurred during the image capture period. If it is determined that a commanded movement has indeed occurred, the image data may be discarded, suppressed, or otherwise not used for registration.

[0060] At optional step 610, if the image data is determined to be acceptable for registration, registration can be performed. In some embodiments, as part of the registration process, image units, such as pixels or voxels, in the image data from imaging system 230 corresponding to medical device 204 are identified. In some embodiments, manually entered computer software, alone or in combination, is used to convert the image data into a two-dimensional or three-dimensional composite representation or model of a segmented part or whole of an anatomical organ or anatomical site. This model can describe the different locations and shapes of anatomical passages and the connectivity of anatomical passages. More specifically, during the segmentation process, pixels or voxels can be divided into segments or elements, or labeled to indicate that they share certain characteristics or calculated properties, such as color, density, intensity, and texture. Image data corresponding to the image of the medical device can be segmented or filtered from the image data, and a model of the device shape can be generated. For example, medical device 204 can be identified as a medical device in the image data by segmentation or filtering based on the CT number or Hounsfield value associated with medical device 204. This data associated with medical device 204 can be isolated from other portions of the image data associated with patient P or a specific tissue type. A three-dimensional mesh model can be formed around the isolated data and / or a centerline representing the centerline of the medical device can be determined. The segmented image data of device 204 can be used in an image reference frame (X). I Y I Z I This is indicated in ().

[0061] The segmented shape of medical device 204 can be registered with shape data acquired from medical device 204 during the image capture period. The shape data from the medical device can be used in the medical device reference frame (X). M Y M Z M ) and / or surgical reference system (XS Y S Z S This registration can be represented in ( ). The registration can be rotated, translated, or otherwise manipulated through rigid or non-rigid transformation points associated with the segmented shape and points associated with the shape data. Such registration between the model and the device reference frame can be achieved, for example, by using ICP or another point cloud registration technique. In some embodiments, the segmented shape of the medical device is registered to shape data, and then the associated transformation (a vector applied to each point in the segmented shape to be aligned with the shape data in the shape sensor reference frame) can be applied to the entire 3D image and / or a 3D image subsequently acquired during a medical procedure. This transformation can be a six-degree-of-freedom (6DOF) transformation, allowing the shape data to be translated or rotated in the X, Y, and Z axes, as well as in any one or all of pitch, roll, and yaw.

[0062] By using the image reference frame (X) I Y I Z I Registration to the medical device reference system (X) M Y M Z M The image displayed to the operator O on the display system 110 can allow the operator to more accurately steer the medical device, visualize the target lesion relative to the medical device, view the view from the distal end of the medical device, and / or improve the efficiency and effectiveness of the target medical procedure.

[0063] In some embodiments, intraoperative image data may be registered with preoperative image data obtained from the same or different imaging systems. Therefore, by registering shape data to intraoperative image data, the registration from shape data to preoperative image data can also be determined. In some embodiments, anatomical images generated from intraoperative and / or preoperative image data may be displayed together with images of the instrument 204 derived from instrument shape sensor data. For example, a model of the instrument 204 generated from instrument shape data may be overlaid on an image of the patient's anatomy generated from the image data.

[0064] At optional step 612, if it is determined that the image data is unacceptable for registration, the registration procedure can be aborted. Instructions can be given to the user to start a new image capture procedure, or the control system can start a new image capture procedure.

[0065] In some embodiments, before initiating the image capture procedure, it can be determined whether the instrument or a portion of the instrument adjacent to an anatomical target (e.g., a lesion or nodule) is in an area where anatomical movement (e.g., due to respiratory or cardiac processes) exceeds a threshold for appropriate image capture. Figure 8The illustration depicts a method 700 for assessing anatomical motion. Typically, anatomical motion can cause intraoperative anatomical image data to be too distorted to isolate and segment medical devices. The magnitude of anatomical motion can be assessed before capturing intraoperative images onto the medical device. If the magnitude of anatomical motion exceeds a threshold amount, any captured images may be distorted and potentially deemed unsuitable for registration, and instructions may be provided to pause image capture and / or move the device to a different anatomical region.

[0066] At process 702, when the instrument is positioned within the anatomical structure of patient P, the magnitude of movement of the instrument or a portion thereof (e.g., medical device systems 104, 204) is determined. For example, shape data collected from shape sensor 214 may be recorded for the portion of the instrument that will be in the field of view of the imaging system during the image capture procedure. When the instrument portion is not affected by commanded movement, shape data may be recorded during a time period prior to the initiation of the image capture procedure. The magnitude of movement can be determined by evaluating changes in the shape data during this time period.

[0067] At process 704, it is determined whether the movement amplitude exceeds a threshold movement amplitude. The threshold movement amplitude can be predetermined based on the movement amplitude that would result in an unusable image. The movement threshold can be a measure of displacement, rotation, rate, and / or other motion components.

[0068] At process 706, if the movement amplitude does not exceed the threshold movement amplitude, the image capture procedure can be initiated to capture images of the patient's anatomy and instrument parts.

[0069] At process 708, if the movement amplitude does exceed a threshold movement amplitude, the image capture program can be paused. In some embodiments, the user may be instructed to pause the image capture program or move the instrument to a different anatomical region.

[0070] In the description, specific details describing some embodiments have been set forth. Numerous specific details have been set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Those skilled in the art will recognize other elements within the scope and spirit of this disclosure, although not specifically described herein.

[0071] In other embodiments, implementations, or applications not specifically shown or described, elements described in detail with reference to one embodiment, implementation, or application may optionally be included, as long as practicable. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, that element may still be required to be included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless otherwise specifically described, unless the one or more elements would render the embodiment or implementation inoperable, or unless two or more of the elements provide conflicting functionality.

[0072] Any changes and modifications to the described devices, apparatus, methods, and principles of this disclosure are fully contemplated, as would normally occur to those skilled in the art to which this disclosure pertains. In particular, it is fully contemplated that features, components, and / or steps described with respect to one embodiment may be combined with features, components, and / or steps described with respect to other embodiments of this disclosure. Furthermore, the dimensions provided herein are for specific examples, and it is contemplated that the concepts of this disclosure may be implemented using different sizes, dimensions, and / or ratios. To avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative embodiment may be used or omitted where applicable to other illustrative embodiments. For brevity, numerous iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used throughout the drawings to refer to the same or similar parts.

[0073] While this document provides some embodiments of medical procedures, any references to medical or surgical instruments and methods are non-limiting. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial use, general robotic use, and sensing or manipulating non-tissue artifacts. Other example applications relate to cosmetic improvements, imaging of human or animal anatomy, collecting data from human or animal anatomy, and training medical or non-medical personnel. Additional example applications include procedures for handling tissue removed from human or animal anatomy (without returning the anatomy) and performing procedures on human or animal cadavers. Furthermore, these techniques can also be used in surgical and non-surgical medical treatments or diagnostic procedures.

[0074] The methods described herein are illustrated as a set of operations or processes. Not all illustrated processes may be performed in all embodiments of the methods. Additionally, one or more processes not explicitly described may be included before, after, between, or as part of the illustrated processes. In some embodiments, one or more of the processes may be performed by the control system 112 or may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., processor 114 of the control system 112), may cause one or more processors to perform one or more of the processes.

[0075] One or more elements of the embodiments of this disclosure can be implemented in software to execute on a processor of a computer system (e.g., a control processing system). When implemented in software, the elements of the embodiments of the invention are essentially code segments that perform necessary tasks. Programs or code segments can be stored in a processor-readable storage medium or device, which may have been downloaded via a computer data signal embodied on a carrier wave over a transmission medium or communication link. Processor-readable storage devices can include any medium capable of storing information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits, semiconductor devices, semiconductor storage devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments can be downloaded via a computer network (e.g., the Internet, intranet, etc.). Any of a variety of centralized or distributed data processing architectures can be employed. Programming instructions can be implemented as multiple separate programs or subroutines, or programming instructions can be integrated into multiple other aspects of the system described herein. In one embodiment, the control system supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Radio Frequency), IEEE 802.11, DECT (Digital Enhanced Wireless Communication), and wireless telemetry.

[0076] Please note that the presented processes and displays may not inherently relate to any particular computer or other device. Various general-purpose systems may be used with the programs taught herein, or it may prove convenient to construct more specialized devices to perform the described operations. The structures required for various such systems will appear as elements in the claims. Furthermore, embodiments of the invention are not described with reference to any particular programming language. It should be understood that the teachings of the invention as described herein can be implemented using a variety of programming languages.

[0077] In some cases, well-known methods, procedures, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the embodiments. This disclosure describes various instruments, instrument parts, and anatomical structures in three-dimensional space according to their states. As used herein, the term "orientation" refers to the position of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "attitude" refers to the orientation of an object or part of an object in at least one translational degree of freedom and the orientation of the object or part of an object in at least one rotational degree of freedom (up to six total degrees of freedom). As used herein, the term "shape" refers to a set of attitudes, orientations, or orientations measured along an object.

[0078] While certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that such embodiments are merely illustrative of the invention in a broad sense and not limiting, and that embodiments of the invention are not limited to the specific constructions and arrangements shown and described, as various other modifications will be apparent to those skilled in the art.

[0079] The following numbered examples illustrate various aspects of the subject matter described in this article.

[0080] Example 1: A non-transitory machine-readable medium comprising a plurality of machine-readable instructions, which, when executed by one or more processors associated with a computer-assisted medical system device, are adapted to cause the one or more processors to perform a method comprising: recording orientation data of an instrument during an image capture period; determining an orientation change of the instrument from the recorded orientation data; comparing the orientation change of the instrument with an orientation change threshold; and, based on the comparison, determining whether to use image data captured by an imaging system during the image capture period in a registration procedure.

[0081] Example 2: The non-transitory machine-readable medium of Example 1, wherein the orientation data includes orientation information of multiple points used to form the shape of the instrument.

[0082] Example 3: The non-transitory machine-readable medium of Example 2, wherein determining the orientation change of an instrument includes determining the maximum change in orientation information of multiple points.

[0083] Example 4: The non-transitory machine-readable medium of Example 2, wherein determining the orientation change of an instrument includes determining the average change of orientation information of multiple points.

[0084] Example 5: The non-transitory machine-readable medium of Example 2, wherein determining the orientation change of an instrument includes determining the standard deviation of orientation information for multiple points.

[0085] Example 6: The non-transitory machine-readable medium of Example 1, wherein comparing instrument orientation change with an orientation change threshold includes comparing instrument orientation change of the distal portion of the instrument with an orientation change threshold.

[0086] Example 7: The non-transitory machine-readable medium of Example 1, wherein determining whether to use image data captured by the imaging system includes determining not to use the image if the instrument orientation change exceeds an orientation change threshold.

[0087] Example 8: The non-transitory machine-readable medium of Example 1, wherein determining whether to use image data captured by the imaging system includes initiating a registration procedure if an orientation change threshold exceeds the instrument orientation change threshold.

Claims

1. A system comprising: a processor; and a memory having computer readable instructions stored thereon that, when executed by the processor, cause the system to: record position data of an instrument in a patient anatomy during an image capture session of an imaging system external to the patient anatomy; determine an instrument position change from the recorded position data; compare the instrument position change to a position change threshold; and based on the comparison, determine whether to use image data captured by the imaging system during the image capture session in a registration procedure.

2. The system of claim 1, wherein the position data comprises position information of a plurality of points forming a shape of the instrument.

3. The system of claim 2, wherein determining the instrument position change comprises determining a maximum change in the position information of the plurality of points.

4. The system of claim 2, wherein determining the instrument position change comprises determining an average change in the position information of the plurality of points.

5. The system of claim 2, wherein determining the instrument position change comprises determining a standard deviation in the position information of the plurality of points.

6. The system of claim 1, wherein the instrument moves during the image capture session.

7. The system of claim 1, wherein the position change threshold is less than 2 cm.

8. The system of claim 1, wherein comparing the instrument position change to a position change threshold comprises comparing the instrument position change of a distal portion of the instrument to the position change threshold.

9. The system of claim 1, wherein determining whether to use the image data captured by the imaging system comprises determining not to use the image if the instrument position change exceeds the position change threshold.

10. The system of claim 9, further comprising providing instructions to an operator of the imaging system.

11. The system of claim 1, wherein determining whether to use the image data captured by the imaging system comprises initiating the registration procedure if the position change threshold exceeds the instrument position change.

12. The system of claim 1, wherein the position data of the instrument is recorded from a fiber optic shape sensor extending within the instrument.

13. The system of claim 1, further comprising the imaging system.

14. The system of claim 1, further comprising the instrument. ​

Images