Medical imaging systems, devices, and methods for visualizing the deployment of intracorporeal therapeutic devices

Abstract

An end effector state estimation system for use in minimally invasive surgery (MIS) applications executes sensing and prediction algorithms to determine and visualize the state of an implantable treatment device during deployment. The end effector state estimation system includes a flexible elongate member, a treatment device coupled to a distal portion of the flexible elongate member, a sensor that measures at least one parameter related to the deployed state of the treatment device, and a processor. The processor receives image data from an imaging system representing the treatment device positioned within a patient's body cavity and the at least one parameter from the sensor, determines a deployed state of the treatment device based on the image data and the at least one parameter, and outputs a graphical representation of the deployed state of the treatment device.

Description

[Technical Field]

[0001] The subject matter described herein relates to medical imaging systems, devices, and methods for estimating and visualizing the deployment of an in-vivo therapeutic device. The use of medical imaging to estimate and visualize the deployment of a therapeutic device has utility particular to, but not exclusive to, intracardiac transcatheter mitral valve repair. [Background technology]

[0002] Mitral valve (MV) disease is one of the most common valvular pathologies in the human heart, second only to aortic stenosis. The MV is a bileaflet valve located on the left side of the heart and maintains the unidirectional flow of blood from the left atrium (LA) to the left ventricle (LV). Pathologies such as functional wear or calcification of the valve tissue can cause deterioration of the valve leaflets, resulting in improper coaptation. This results in backflow of blood from the LV back into the LA during LV compression. This condition, known as mitral regurgitation (MR), results in reduced output from the heart's pumping mechanism and, if left untreated, can lead to severe complications, including heart failure. The fundamental goal of treatment for MR patients is to reduce MR to a level that restores cardiac function and improves the patient's overall quality of life.

[0003]

[0003] With the advancement of minimally invasive treatments (e.g., minimally invasive surgery or MIS) to replace open-thoracotomy, transcatheter mitral valve repair (TMVR) has emerged as a powerful alternative for patients at high risk for open-thoracotomy. Many devices exist for treating structural heart disease (SHD). One exemplary TMVR device is the mitral valve clip (MC) device. Clinical studies have shown that MC is highly effective in improving patient outcomes by reducing MR to tolerable levels. In addition, transcatheter approaches reduce associated hospital stays compared to open-thoracotomy. This further reduces postoperative complications, thus not only improving patient outcomes but also reducing the burden on hospital systems. Summary of the Invention [Problem to be solved by the invention]

[0004] The use of procedures such as MC implantation is expected to increase over the next decade. However, due to the current complexity of MC devices and their delivery systems, the procedure requires highly skilled practitioners to both operate the device and interpret the X-ray and ultrasound images critical to navigation and deployment tasks. The procedure requires experienced echocardiographers who can operate the ultrasound probe and imaging system to obtain high-quality images and views of the device relative to the surrounding anatomical structures and quickly identify the status of the device and the procedure. The procedure also requires clear communication between team members, especially for less experienced operators. Communication requires accurate representation of the device's end effector (e.g., clip), which is difficult to achieve given the many combinations of device position, orientation, open / closed state, and other factors.

[0005]

[0005] It should therefore be appreciated that such commonly used TMVR procedures suffer from many drawbacks, including difficulties with image capture, image interpretation, understanding the pose and state of the device's end effector, etc. Accordingly, there exists a long-felt need for improved TMVR systems and procedures that address the aforementioned and other concerns.

[0006]

[0006] The information contained in this "Background Art" section of this specification, including any references cited herein and any description or discussion of the references, is included for technical reference purposes only and should not be considered as subject matter that should limit the scope of the disclosure. [Means for solving the problem]

[0007] An end effector state estimation system for use in transcatheter minimally invasive surgery (MIS) applications is disclosed that includes sensing and predictive algorithms for determining the state of an implantable treatment device based on control inputs and / or imaging data during deployment-related operations, and annotation of medical images that include a graphical representation of the treatment device. It will be understood that in this disclosure, a deployable treatment device, or a deployment assembly for deploying a treatment device, is referred to as an end effector.

[0008] Structural heart procedures performed using transcatheter techniques are highly image-intensive. Navigation, target selection, and deployment are more easily performed when real-time 3D imaging is available. However, ascertaining the exact state of the end-effector is challenging. An end-effector state estimation system would enable procedures to be performed more reliably in 3D and reduce the cognitive burden on the clinician to find the optimal 2D slice from the 3D data.

[0009]

[0009] High quality ultrasound images depend on optimal sonication of the object of interest, but suffer from image artifacts that interfere with visualization of tissues and devices. An end-effector state estimation system can help complement traditional imaging with accurate device state, potentially increasing the reliability of the procedure and therefore improving treatment efficacy and shortening the learning curve. The end-effector state estimation system disclosed herein has utility specific to, but not exclusive to, intracardiac transcatheter mitral valve repair.

[0010] According to an embodiment of the present disclosure, there is provided a system for determining a deployment state of a treatment device coupled to a distal portion of a flexible elongate member disposed within a body cavity of a patient, the system comprising: at least one sensor configured to measure at least one parameter related to the deployment state of the treatment device; and a processor configured to receive image data acquired by an imaging system representing the treatment device disposed within the body cavity of the patient and the at least one parameter, determine a deployment state of the treatment device based on the image data and the at least one parameter, and output a graphical representation of the deployment state of the treatment device on a display in communication with the processor.

[0011] In some embodiments, the processor is configured to output, on the display, a screen display including an image of the patient's body cavity generated based on the image data and a graphical representation of the deployed state of the treatment device superimposed on the image of the body cavity. In some embodiments, the graphical representation of the deployed state of the treatment device includes a visualization of a three-dimensional model of the treatment device. In some embodiments, the at least one parameter includes at least one of a position or an angle of a control mechanism of the treatment device. In some embodiments, the system includes a transcatheter delivery device, the transcatheter delivery device including a flexible elongate member, and the at least one parameter includes at least one of a position or an angle of the transcatheter delivery device. In some embodiments, the at least one sensor includes an encoder coupled to a mechanical control mechanism of the transcatheter delivery device. In some embodiments, the at least one sensor includes a magnetic sensor configured to measure the at least one parameter by obtaining position measurements of magnetic seeds disposed on the treatment device.

[0012] In some embodiments, the imaging system includes an ultrasound imaging system. In some embodiments, the imaging system includes an X-ray imaging system. In some embodiments, the processor is configured to determine the deployment state of the treatment device by using image recognition to match an image generated based on the image data to a model of the treatment device. In some embodiments, the at least one parameter includes a measure of deployment, and the processor is configured to determine a confidence level of the matched model and determine the deployment state based on the measure of deployment when the confidence level is below a threshold.

[0013]

[0013] According to another embodiment of the present disclosure, a method for determining a deployment state of a treatment device includes measuring at least one parameter related to the deployment state of the treatment device by a sensor, the sensor being coupled to a distal portion of a flexible elongate member; acquiring by an imaging system an image including the treatment device; determining by a processor a deployment state of the treatment device based on the image and the at least one parameter; and outputting by the processor a graphical representation of the deployment state of the treatment device on a display in communication with the processor.

[0014] In some embodiments, the method includes outputting, on a display, a screen display including at least one of the images and a graphical representation of the deployed state of the treatment device superimposed on the at least one image. In some embodiments, the at least one parameter includes a position or angle of a control mechanism of the flexible elongate member. In some embodiments, the at least one parameter includes a position or angle of the treatment device. In some embodiments, the imaging system includes an ultrasound imaging system. In some embodiments, the imaging system includes an X-ray imaging system. In some embodiments, the sensor includes a magnetic sensor, and measuring the at least one parameter includes obtaining position measurements of magnetic seeds disposed on the treatment device. In some embodiments, combining information from the sensor and the imaging system to determine the deployed state of the treatment device includes using image recognition to match images from the imaging system with a model of the treatment device.

[0015]

[0015] According to another embodiment of the present disclosure, a system for determining a deployment state of a therapeutic device comprises a mitral valve clip placement device comprising a catheter and a mitral valve clip coupled to a distal portion of the catheter; an ultrasound imaging device configured to acquire ultrasound images of the mitral valve clip positioned within a patient's body cavity; an x-ray imaging device configured to acquire x-ray images of the mitral valve clip positioned within the patient's body cavity; a sensor coupled to the mitral valve clip placement device and configured to provide a sensor signal indicative of a parameter related to the deployment state of the mitral valve clip; and a processor configured to determine the deployment state of the mitral valve clip based on the parameter and the x-ray image, and output at least one of the ultrasound images and a graphical representation of the deployment state of the therapeutic device to a display in communication with the processor.

[0016]

[0016] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of the features, details, usefulness, and advantages of the end effector state estimation system, as defined in the claims, is provided in the written description below and illustrated in the accompanying drawings of various embodiments of the present disclosure.

[0017]

[0017] Exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram of an exemplary end effector state estimation system, in accordance with at least one embodiment of the present disclosure. [Figure 2]

[0019] FIG. 1 is a schematic diagram of an exemplary transcatheter mitral valve repair procedure, in accordance with at least one embodiment of the present disclosure. [Figure 3]

[0020] 1 is an ultrasound image of an exemplary transcatheter mitral valve repair procedure in accordance with at least one embodiment of the present disclosure. [Figure 4]

[0021] FIG. 1 is a schematic diagram of an exemplary transcatheter clip delivery system in accordance with at least one embodiment of the present disclosure. [Figure 5]

[0022] FIG. 1 is a schematic diagram of an example end effector state estimation system 100, in accordance with at least one embodiment of the present disclosure. [Figure 6]

[0023] FIG. 1 is a perspective view of at least a portion of an exemplary transcatheter clip delivery system 110, in accordance with at least one embodiment of the present disclosure. [Figure 7]

[0024] 1 is an X-ray image during an exemplary transcatheter mitral valve repair procedure, in accordance with at least one embodiment of the present disclosure. [Figure 8]

[0025] FIG. 10 is a perspective view of a 3D model of an end effector, such as a mitral valve clip, in accordance with at least one embodiment of the present disclosure. [Figure 9]

[0026] FIG. 10 is a perspective view showing a detailed model of an end effector (e.g., a mitral valve clip) in six different open / closed states, in accordance with at least one embodiment of the present disclosure. [Figure 10]

[0027] FIG. 10 is a schematic diagram of the inputs and outputs of an algorithm that matches an ultrasound image with a model of an end effector, in accordance with at least one embodiment of the present disclosure. [Figure 11]

[0028] FIG. 1 is a schematic diagram of an example convolutional neural network used to determine the state and pose of an end effector of a transcatheter device, in accordance with at least one embodiment of the present disclosure. [Figure 12]

[0029] FIG. 2 is a schematic diagram of a processor circuit according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0019]

[0030] An end-effector state estimation system is disclosed for use in transcatheter minimally invasive surgery (MIS) applications. Systems and methods are disclosed that utilize sensing within a device manipulator (e.g., a manual control mechanism) in combination with machine learning-based prediction to accurately capture the state of an implantable treatment device during deployment-related manipulations. Systems and methods, as well as combinations of systems and methods, that use image-based information to derive device state are also disclosed. In various embodiments, a treatment device may be or comprise an end-effector. Aspects of the present disclosure apply to end-effectors and / or treatment devices that are / comprise end-effectors, including any suitable end-effector and / or treatment device that expands, contracts, bends, flexes, articulates, or otherwise changes shape at a distal end, a proximal end, or any one or more points between the proximal and distal ends.

[0020]

[0031] Structural heart procedures performed using transcatheter techniques are highly image-intensive. Mitral valve leaflet repair, in particular, is driven by the need for high-quality dynamic (e.g., real-time) imaging. X-ray images clearly visualize the device but not the tissue of interest. Ultrasound images, on the other hand, display the tissue but make it difficult to clearly visualize the device. 2D ultrasound and X-ray imaging guidance is used during the procedure, which essentially involves solving a 3D problem in two dimensions. There is growing evidence that navigation, target selection, and deployment are more easily performed when real-time 3D imaging is available. However, even with 3D imaging, it is difficult to confirm the precise orientation and open / closed state of the end effector relative to the target tissue. The disclosed end effector state estimation system enables procedures to be performed more reliably in 3D and also allows the use of improved 2D images through analysis of 3D information. The end effector state estimation system also reduces the cognitive burden on the ultrasound operator to find the optimal 2D slice that describes the 3D state of the end effector of the transcatheter device.

[0021]

[0032] High-quality ultrasound images depend on optimal sonication of objects of interest (e.g., mitral valve leaflets and clips or other end effectors for transcatheter devices). Ultrasound imaging can suffer from artifacts, such as acoustic dropouts, shadowing, and bloom artifacts caused by inadequate gain, which can make clear visualization of tissue and devices in the same image difficult. End-effector state estimation systems can help complement traditional imaging with accurate device state, even when such artifacts clutter the image.

[0022]

[0033] A key task in valve leaflet repair using clip-type devices is to properly grasp each individual leaflet within the clip's grasping mechanism. Gain is often increased to aid in ultrasound imaging of the mitral valve leaflets, which can hinder visualization of the clip's thin grasping arms. However, proper understanding of the clip's open / closed state is crucial to performing the procedure. An end-effector state estimation system can provide this information, potentially increasing procedural reliability, improving treatment efficacy, and shortening procedure time.

[0023]

[0034] As noted above, MIS procedures, such as transcatheter intracardiac procedures, are driven by the quality of imaging and effective communication among the team of clinicians performing the procedure. This results in an associated learning curve, requiring many procedures to be performed by team members before a high level of proficiency is achieved. An end-effector state estimation system can provide feedback to the team, including device state and pose, potentially improving the learning curve and increasing confidence during the procedure.

[0024]

[0035] The present disclosure significantly aids in the performance of minimally invasive surgery (MIS) by improving visualization of device end effectors. The end effector state estimation system disclosed herein, when implemented on a processor in communication with a transcatheter device control system and one or more external imaging systems, provides practical enhancement or annotation of raw ultrasound images during MIS procedures. This enhanced visualization enables the use of other imaging modalities offering varying degrees of clarity and accuracy, allowing less skilled operators to perform transcatheter therapeutic procedures. This unconventional approach enhances the capabilities of transcatheter device control systems by allowing the system to more easily perform complex tasks, such as intracardiac repair procedures.

[0025]

[0036] The end effector state estimation system operates via a control process executing on a processor that generates real-time image enhancements or annotations viewable on a display, receives user input from a keyboard, mouse, or touchscreen interface, and communicates with one or more medical devices. In that regard, the control process performs certain specific operations in response to various inputs or selections made at various times and in response to various triggers. While some of the structure, functionality, and operation of the processor, display, sensors, and user input system are known in the art, others are described herein to specifically enable novel features or aspects of the present disclosure.

[0026]

[0037] These descriptions are provided for illustrative purposes only and should not be considered to limit the scope of the end effector state estimation system. Some features may be added, deleted, or modified without departing from the spirit of the claimed subject matter.

[0027]

[0038] To promote an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the embodiments. It will nevertheless be understood that no limitations on the scope of the present disclosure are intended. Any changes and further modifications to the described devices, systems, and methods, and any other applications of the principles of the present disclosure, as would normally occur to one skilled in the art to which the present disclosure pertains, are fully contemplated and encompassed herein. In particular, it is fully contemplated that features, components, and / or steps described with respect to one embodiment may be combined with features, components, and / or steps described with respect to other embodiments of the present disclosure. However, for the sake of brevity, multiple iterations of such combinations will not be separately described.

[0028]

[0039] 1 is a schematic diagram of an exemplary therapeutic device state estimation system 100, also referred to as an end-effector state estimation system, in accordance with at least one embodiment of the present disclosure. The exemplary end-effector state estimation system 100 is configured to assist a clinician in operating a transcatheter clip delivery system 110 (e.g., Abbott's MitraClip or TriClip, or Edwards Life Sciences' Pascal system) configured to manipulate a clip or other related end-effector device within a heart 120. The end-effector state estimation system 100 receives data from an external reference-based position sensor 125 (e.g., a magnetic position sensor, an optical position sensor), a position or angle encoder 130 or related sensors, an ultrasound imaging system 140, and / or an X-ray imaging system 150.

[0029]

[0040] The end effector state estimation system 100 also includes a processing circuit 160. In the example shown in FIG. 1 , the processing circuit 160 can interface with and / or act as a control unit that reads encoder information from sensors 130 disposed on the mechanical manipulator or from a spatial localization system. The sensors 130 can measure one or more parameters related to the deployment state of a therapeutic clip device delivered using the transcatheter clip delivery system 110. The processing circuit 160 can further perform traditional data (image, signal) processing algorithms and / or machine learning and deep learning algorithms on image data from the ultrasound imaging system 140 and / or the X-ray imaging system 150. The processing circuit 160 can further generate complex, annotated 2D or 3D renderings 170 and display the renderings on a display 180, e.g., corresponding to multi-channel renderings. In some embodiments, the processing circuit comprises a single processing component disposed within the housing. In other embodiments, the processing component includes multiple hardware components and / or hardware components such as a central processing unit (CPU), a graphics processing unit (GPU), or any other suitable components located within a single housing or distributed among multiple housings.

[0030]

[0041] It will be understood that the examples described above are provided for purposes of illustration and are not intended to be limiting, and that other devices and / or single device configurations may be utilized to perform the operations described herein.

[0031]

[0042] 2 is a schematic illustration of an exemplary transcatheter mitral valve repair (TMVR) procedure 200 in accordance with at least one embodiment of the present disclosure. The procedure 200 uses a clip device 205 (e.g., Abbott's MitraClip) with two arms 210 and two grippers 220. The procedure also includes using a catheter-based delivery system 230 that uses a flexible, elongated member to enter the heart 120 through the right atrium 240, then puncture the right atrium 240 into the left atrium 250, and advance through the mitral valve 260 into the left ventricle 270 (step "a"). The clip 205 is then opened (step "b"), and subsequently closed to grasp two leaflets 280 of the mitral valve 260 (step "c"). The clip 205 is then detached from the delivery system 230 (step "d"), and the delivery system 230 is withdrawn from the heart 120. Clip device 205 may be referred to as a treatment device and / or an end effector of delivery system 230. Those skilled in the art will understand that other devices and procedures may be used to repair structures within heart 120, such as for valve replacement, tissue ablation, installation or removal of pacemaker components, and may be used with the end effector state estimation system, so long as the device has an end effector that is large enough to be visualized on ultrasound images and is acoustically reflective.

[0032]

[0043] Placing and deploying a therapeutic device, such as a mitral valve clip, within the mitral valve is a challenging procedure. In this regard, the mitral valve is typically still in motion while the procedure is being performed. While x-ray imaging data provides clear images of the mitral valve clip and delivery system, the position of the device relative to the mitral valve is not visible in the x-ray image. While ultrasound provides more anatomical information and detail, such as the mitral valve, the echogenicity of the mitral valve clip and delivery system makes it difficult to achieve images with adequate gain, contrast, and other image settings to clearly visualize the location and deployment of the mitral valve clip with sufficient precision to perform the procedure. Furthermore, when two-dimensional ultrasound imaging is used, it is difficult to determine the orientation of the mitral valve clip relative to the field of view of the ultrasound probe, regardless of the image settings.

[0033]

[0044] In this regard, Figure 3 is an ultrasound image 300 of an exemplary transcatheter mitral valve repair (TMVR) procedure in accordance with at least one embodiment of the present disclosure. Such an image is captured using, for example, a transesophageal echocardiogram (TEE) probe. Thresholding and / or edge gradient filtering is used to remove noise in the image and to increase contrast between tissue, blood, and solid components of the treatment device.

[0034]

[0045] Visible in image 300 are mitral valve clip 205, left atrium 250, left ventricle 270, mitral valve leaflets 280, as well as arms 210 and gripper 220 of mitral valve clip 205. During a TMVR procedure, for example, it is desirable to position clip 205 so that arms 210 are positioned in left ventricle 270 and gripper 220 are positioned in left atrium 250, with leaflets 280 positioned between arms 210 and gripper 220, respectively. In this configuration, clip 205 can be closed to permanently hold leaflets 280 together, which can limit mitral regurgitation (MR) and thus improve the health of MR patients.

[0035]

[0046] As seen in the exemplary image 300 of FIG. 3 , it is very difficult for many clinicians to capture a real-time ultrasound image that includes both leaflets 280, both arms 210, and both grippers 220. Capturing such an image of a beating heart requires proper position and orientation of the ultrasound probe, proper gain settings to display the leaflets 280, and the absence of noise, shadows, and bloom artifacts to show the structure of the clip 205. The imaging clinician must also select, in real time, a 2D slice of the 3D volume data in which these features are located. It is a feature of the present disclosure that reduces the burden on the imaging clinician by allowing the position, orientation, and open / closed state of the clip (or other end effector) to be determined and visually reported without the need for manual optimization of the ultrasound image in real time.

[0036]

[0047] FIG. 4 is a schematic diagram of an exemplary transcatheter clip delivery system 110 in accordance with at least one embodiment of the present disclosure. A clip 205 is visible, including an arm 210 and a gripper 220. The arm 210 is actuated by an actuator 410, while the gripper 220 is actuated by a tensioning member (tendon) 420. Both the tensioning member 420 and the actuator 410 are threaded through a catheter or flexible elongate member 430. The clip 205 may be coupled to a distal portion 431 of the flexible elongate member 430 and is considered an end effector. FIG. 4A illustrates the clip 205 in a partially open configuration. FIG. 4B illustrates the clip 205 in a closed configuration. In some aspects, the clip 205 is advanced through the mitral valve in the closed configuration and then opened to engage the leaflets of the mitral valve, as shown in FIG. 4A. The gripper 220 can then be opened and brought closer to the arm 210 to clamp opposing leaflets of the mitral valve between the arm 210 and the gripper 220. FIG. 4C shows the clip 205 covered with a fabric cover 440 that promotes tissue growth over the clip 205 once the clip 205 is implanted into the patient's mitral valve. FIG. 4D shows a clip control mechanism 450 and a delivery system control mechanism 480 of an exemplary delivery system 110. The clip control mechanism 450 includes an arm articulation knob 460 and a pair of individual gripper articulators 470. The open / closed states of the arm 210 and the gripper 220 are referred to as parameters related to the deployed state of the clip device 205.

[0037]

[0048] 5 is a schematic diagram of an exemplary end effector state estimation system 100 in accordance with at least one embodiment of the present disclosure. A mechanical manipulator 450 of a clip delivery system 110 is used to control articulation of a valve repair device (e.g., a clip 205 that is part of a removable end effector 405, such as shown in FIG. 4). Sensors 510 and / or 520 can be used to sense or encode articulation based on mechanical manipulation of a handle of the clip delivery system, detecting the state of controls such as an arm articulation knob 460 and a gripper articulator 470. Sensor 510 or 520 generates a digitized, machine-readable signal and includes, but is not limited to, an electrical encoder (e.g., a rotary encoder), an optical encoder, an embedded sensing potentiometer, a magnetic spatial localization system (e.g., tracking a control surface, actuator, or pulling member with one or more magnetic seeds), an IR-based active or passive optical spatial localization system (e.g., tracking the position of one or more IR-reflective or IR-emitting fiducial markers), or a vision-based (e.g., RGB, monochrome, etc.) optical tracking system that estimates the position of one or more visible reference features.

[0038]

[0049] The digitized signal (e.g., signal 515 or 525) is received by a manipulation detection unit 530 of a computing unit corresponding to processing circuit 160 (FIG. 1). Computing unit 160 also includes an image processing (e.g., machine learning algorithm) unit 540, which receives imaging data 550 from imaging system 140 or 150. Computing unit 160 has an interface (e.g., software and / or hardware) for processing the relevant information and providing it to one or more algorithms. For example, variable α is assigned to the position of the knob controlling the arm of the "clip" device, and β is assigned to the position of the manipulator controlling the gripper. The one or more algorithms receive two input sources: (1) a signal representing the current manipulation state (α, β) of the device handle, and (2) imaging information I from ultrasound, X-ray, or two integrated real-time interventional imaging modalities during the procedure. us , I x-ray , I us+x-ray The algorithm is designed to generate information that most accurately represents the physical articulation of the device within the body using a combination of algorithms. The algorithm may be composed of basic modules of the following types, including, but not limited to: In some embodiments, the signals are received, digitized, and / or processed by an interface device or control unit in communication with processor 160, and the interface device provides processor 160 with position information based on the received signals.

[0039]

[0050] In therapeutic devices such as mitral valve clips, the physical device state, which varies from fully closed to fully open, is controlled by a screw and tension member drive system. The control mechanism for such devices has only one degree of freedom (DOF) (e.g., the manipulator state α maps to the clip's open state, and similarly, β maps to the gripper state), and the open / close response maps unpredictably and irreproducibly to the 1-DOF joint motion. Therefore, a lookup table relating specific control surface settings to individual device states is not a reliable predictor of device state in all cases. This difficulty in mathematically modeling the relationship is caused by the physical characteristics of the device delivery system and the effects of the device delivery system's operating environment on those physical characteristics. For example, the hysteresis of this system can be modeled in a data-driven manner. U.S. Provisional Patent Application No. 62 / 825,914, filed March 29, 2019, U.S. Provisional Patent Application No. 62 / 825,905, filed March 29, 2019, and U.S. Provisional Patent Application No. 62 / 811,705, filed February 28, 2019, which are incorporated by reference in their entireties, describe methods for estimating the motion of electromechanical devices. These techniques can be applied to derive the state of an implantable end effector prior to deployment.

[0040]

[0051] One way to account for the amount of deviation between control state information (e.g., encoder feedback) and the actual position or deployment state of the therapeutic device is to use a neural network to regress the device's arm state θ (e.g., angle in degrees) against the knob rotation α. ​​Data collection for this can be done by placing a magnetic tracking seed (e.g., 5 DOF, approximately 5 mm) on the arm and collecting data on (a) the digital signal from the knob manipulation, quantized to a desired range that is sensitive enough for the application, and (b) the position of the arm location converted to an angle value. This can then be solved as a regression problem. Data acquisition can also be achieved in a controlled robotic manner (e.g., moving a knob or other control surface back and forth in various combinations and recording the effect on the device state). The above approach can also be applied to other articulations (e.g., gripper articulation), such as those shown in Figure 4.

[0041]

[0052] 6 is a perspective view of at least a portion of an exemplary trans-catheter clip delivery system 110 in accordance with at least one embodiment of the present disclosure. The trans-catheter clip delivery system 110 includes a clip control mechanism 450 and a delivery system control mechanism 480 housed in a guide handle 610, a steerable sleeve handle 620, and a clip delivery handle 630. As discussed above, one or more sensors or encoders may be incorporated into any or all of the control mechanisms, such as the clip control mechanism 450 and / or the delivery system control mechanism 480, to provide position and / or deployment state information regarding the end effector.

[0042]

[0053] 7 illustrates an X-ray image 700 during an exemplary transcatheter mitral valve repair (TMVR) procedure in accordance with at least one embodiment of the present disclosure. X-ray images can be advantageous during MIS procedures because they clearly display the position, orientation, and open / closed state of the end effector 405 (e.g., clip 205) of the transcatheter delivery system 110 from certain angles. This is represented as an angle θ that is measured or estimated, for example, manually by a clinician or automatically using an image recognition algorithm based on the X-ray image 700. The angle may be determined, for example, by segmentation, force measurement, shape sensing, or by a user drawing an angle on a touchscreen. However, X-ray images do not display soft tissue, such as the leaflets of the mitral valve, which limits the usefulness of X-ray imaging during the procedure for steering and deploying the end effector device 405 relative to the tissue of interest. Nevertheless, the device state (e.g., angle θ) obtained from the X-ray image 700 during the procedure is used to update, supplement, confirm, or adjust the device state obtained by sensing the control surfaces within the delivery system 110. The device state (e.g., angle θ) is then used to assist in image recognition of the 3D position, 3D orientation, and open / closed state of the end effector 405, as described below. In some embodiments, separate angle measurements are taken for various components, such as the arms and grippers, of the clip device 205.

[0043]

[0054] 8 is a perspective view of a 3D model 800 of an end effector 405, such as a mitral valve clip, in accordance with at least one embodiment of the present disclosure. The model includes a detailed version 810 that resembles the appearance of the end effector in, for example, an x-ray image, a simplified version 830 that resembles the appearance of the end effector in, for example, an ultrasound image, and an intermediate version 820 that serves as a computational intermediate step for deriving the simplified model 830 from the detailed model 810.

[0044]

[0055] The deployment of the leaflet repair implant is performed under live (e.g., real-time, during the procedure) x-ray and ultrasound guidance. The device exhibits a distinct morphology and predictable pattern when optimally visualized in each imaging modality. These imaging features can be used to drive image-based algorithms (traditional deterministic image processing, machine learning, and / or a combination thereof) to not only capture the device state but also predict its behavior. Because the physical and geometric relationships of the device and its articulation are known a priori, this information can be incorporated into a model-fitting algorithm. In this case, the algorithm fits a discrete, rigid-body model of the device using conventional methods, including but not limited to, level set methods, in-service geometry, and in-service statistical shape and appearance models. These device models can be either a family of discretized models or continuous parametric models with single degrees of freedom (1 DOF) or multiple degrees of freedom (e.g., 2 DOF, 3 DOF, etc.) of articulation. A version of the algorithm is described below.

[0045]

[0056] The end effector state estimation system 100 is configured to fit a model of an interventional device (e.g., mitral valve clip 205) to an ultrasound (US) image. This fitting involves aligning a 3D model 800 to the device shown in the ultrasound image, hereafter referred to as the "target." The model 800 is represented, for example, as a triangular mesh or by a binary mask, point cloud, or the like, which can be converted to a triangular mesh using methods known in the art. Because the model 800 of the device's end effector 405 is typically very detailed, while the representation of the end effector in the ultrasound image is not, the model representation is simplified using mesh simplification methods known in the art. An example of model generation and simplification is shown in FIG. 8.

[0046]

[0057] In instances where the model is rigid, it is desirable to find a rigid transformation that allows the best alignment of the model to the target, such as an iterative closest point (ICP) algorithm. To define a correspondence between each mesh (M) point and the image (I), a target (T) point is defined by thresholding the image. The threshold (th) can be fixed or adaptive. For each mesh point, the algorithm then determines the corresponding target, for example, along the surface normal (N) of the mesh point.

number

[0047]

[0058] where T(p) is the target point, M(p) is the mesh point, and N(p) is the surface normal vector at point p. The algorithm then defines the signed magnitude of the displacement d as follows:

number

[0048]

[0059] Once the correspondence between the mesh points and the target points is determined, the algorithm finds a rigid body transformation using, for example, singular value decomposition (SVD). The above steps can then be repeated iteratively until the process converges (i.e., until the average distance between the mesh points and the target points becomes negligible). In this manner, the end effector 405 (e.g., a mitral valve clip) can be positioned in an image (e.g., an x-ray or ultrasound image) in real time or near real time, and the orientation of the end effector can be determined. Those skilled in the art will appreciate that while this process is described for only one model, the algorithm can also determine the state of the end effector 405 by matching the image against multiple models, each representing a different open / closed state, as described below.

[0049]

[0060] FIG. 9 is a perspective view illustrating a model of an end effector 405 (e.g., a mitral valve clip) in six different open / closed states, in accordance with at least one embodiment of the present disclosure. For mitral valve clips and many other devices, their shapes can change during their deployment. For example, the mitral valve clip includes arms 210 that can open and close in various orientations, as shown in FIG. 9. The model depicted here includes arms 210 but does not include grippers 220 (e.g., as shown in FIGS. 2, 3, and 4). Those skilled in the art will understand that various models may include various permutations of arm and gripper positions or other features of various end effectors 405. In the example shown in FIG. 9, model 810a represents the end effector 405 with the arm 210 fully closed, while model 810f represents the end effector 405 with the arm 210 fully open, and models 810b, 810c, 810d, and 810e represent various intermediate states between these extremes.

[0050]

[0061] FIG. 10 is a schematic diagram of the inputs and outputs of an algorithm for matching ultrasound images with models of an end effector, according to at least one embodiment of the present disclosure. These device models can be either discretized model families or continuous parametric models with single or multiple degrees of freedom (1 DOF) of articulation (e.g., 2 DOF, 3 DOF, etc.). In the example shown in FIG. 10, models 810a through 810f represent various open / closed states of the end effector. For a given ultrasound image (e.g., an instantaneous 3D snapshot of a volume of interest), each model is matched against the image data, and a confidence score is calculated that reflects the probability that the given model accurately represents the state of the device shown in the image. The ordered group of models is called a model family (MF). Because it is unknown in the initial image frame how far the device has opened, one possible approach consists of fitting all models in the MF using the algorithm described above and then selecting the best-fitting model. Generally, the model with the highest confidence score may be selected as the "true" or most likely current state of the end effector. This confidence score may be derived in several ways, primarily based on image features.

[0051]

[0062] This selection can be made by defining a fit confidence measure (CM). This CM can be defined as a weighted sum of two criteria: the Dice coefficient and the edge gradient measure. The Dice coefficient (Dice) is based on the model and the target volume. The target volume is defined by thresholding the image within a region of interest (ROI) that precisely surrounds the fitted model. This measure is maximized when the mesh maximally overlaps the target. The edge gradient measure involves calculating the sum of normalized image gradients at all points of the fitted mesh. This measure is maximized when the mesh boundary is optimally aligned with the boundary of the object in the image. Alternatively or additionally, an appropriate model can be selected based on the input position of the control mechanism and / or the X-ray image, as described above, or the user can manually select the model and device position / or orientation as a starting point or at intervals during the procedure.

[0052]

[0063] Once a mechanism for fitting an MF to a frame is defined, the image data can also be used for model tracking. Because object motion is continuous between frames of a real-time ultrasound image sequence, the model transformation from the previous frame can be used as the initial model pose for the next frame. The algorithm can also incorporate the knowledge that model opening / closing is also continuous between frames, and therefore does not need to test the entire MF, but only the best model found in the previous frame and the "adjacent" models in the MF sequence. This approach reduces the computational burden and makes it more feasible to run the algorithm in real time.

[0053]

[0064] 10 , ultrasound image 1010a corresponds to model 810a, and therefore an outline of model 810a is superimposed on the ultrasound image as graphical indicator 1012a. Image 1010c corresponds to and has superimposed graphical indicator 1012c representing model 810c, while images 1010d and 1010e have superimposed corresponding graphical indicators 1012d and 1012e representing models 810d and 810e, respectively. Because ultrasound images can be grainy, noisy, low-resolution, and prone to imaging artifacts, this model superimposition allows a clinician or other user to at a glance interpret the position, orientation, and open / closed state of the end effector in ultrasound images across a range of image qualities. In some embodiments, the ultrasound image 1010 and the corresponding graphical indicator 1012 for the ultrasound image 1010 are output as a graphical user interface or screen display and are updated in real time to indicate the deployment status of the treatment device. In some embodiments, other types of graphical indicators or diagrams may be included in the screen display to indicate the deployment status, such as wireframe diagrams, two-dimensional diagrams, numerical indicators, text indicators, and / or any other suitable type of graphical indicator. In some embodiments, the amount or percentage of deployment (e.g., angle, percent of maximum articulation) is shown on the screen display. In some embodiments, the screen display includes a representation of the reliability of the position measurement, such as a numerical indicator, or a graphical representation similar or identical to graph 1020. In some embodiments, the graphical indicator and / or another indicator on the screen display indicates the orientation of the treatment device relative to the field of view of the image being used.

[0054]

[0065] Control surface measurements (Fig. 5) or X-ray images (see Fig. 7) can greatly aid in the selection and superposition of the most appropriate model, and if the open / closed state of the end effector is known, the computational load can be reduced to a certain extent within a certain confidence range.

[0055]

[0066] 11 is a schematic diagram of an exemplary convolutional neural network (CNN) 1140 used to determine the state and pose of an end effector of a transcatheter device, in accordance with at least one embodiment of the present disclosure. In some embodiments, the device state can be derived based on the convolutional neural network's ability to model complex patterns observed in natural and medical images. This data-driven method can be used to derive the device state directly from 3D image data 1130 and / or by transforming image features into a set of precursor parameters that are input to traditional image processing methods (such as those described above). An exemplary approach utilizes a supervised learning paradigm, in which the network 1140 is trained with a large amount of labeled image data 1110 to generate a network layer 1120 for regressing the end effector state and pose. The labels can be hard prior information representing (a.) the state of the end effector, (b.) the pose of the end effector, and (c.) the location of the device within the ultrasound coordinate system of the 3D volume data 1130. Using this approach, a model can be directly fitted to determine an output vector 1150, which can include, for example, the pose, state, and position of the end effector. This method runs in an "on-demand" manner, depending on available computational power, as a complement to more traditional algorithms such as those described above. The neural network 1140 can take various forms in terms of architecture (e.g., recurrent networks), as well as paradigms known in the art (e.g., unsupervised learning, reinforcement learning, etc.).

[0056]

[0067] In some embodiments, a deep learning (DL) algorithm starts with a 3D grid to find a rough estimate of where the device is within the 3D volume 1130. Including an X-ray image as input for DL ​​improves robustness. In other embodiments, a 2D ultrasound probe is swept across the volume of interest to generate the 3D image volume 1130.

[0057]

[0068] As explained above, providing redundancy in the user feedback system is beneficial to (a.) repeatedly understand the state of the device, especially when interacting with tissue, (b.) lower the barrier to entry and shorten the learning curve for the procedure, and (c.) increase the reliability and therapeutic effectiveness of transcatheter MIS procedures.

[0058]

[0069] In some embodiments, the end effector state estimation system 100 can cross-reference the device state based on machine operation with the image-derived state to achieve a high degree of redundancy when either input falls below a certain reliability criterion for the input. Real-world scenarios can include poor-looking end effector images in ultrasound images and the corresponding difficulty in determining the end effector's pose and state due to shadowing, dropout, or excessive gain used to visualize the tissue, as well as possible shadowing or foreshortening in X-ray images. In particular, gripper visualization is beneficial during the steps of inserting and grasping the valve leaflet. This is often problematic, and the end effector state estimation system 100 can provide this missing information, for example, based on the measured state of the control surfaces. The end effector state estimation system 100 can also improve the performance of image-based methods by reducing the optimization search space when image features are scarce.

[0059]

[0070] The end effector state estimation system 100 may be implemented as a weighted sum of the outputs of physical and image-based modules using traditional techniques, or as a machine learning (ML) regression problem. Visualization and feedback may be presented to the user using currently existing display modules, for example, as described in U.S. Patent Application Publication No. 2019 / 0371012, filed January 15, 2018, and U.S. Provisional Patent Application No. 63 / 042,801, filed June 23, 2020, both of which are incorporated by reference in their entireties. In some embodiments, the end effector state estimation system 100 provides recommendations (e.g., textual or graphical instructions) to the user regarding appropriate control surface movements to achieve a desired end effector state at various times during the procedure.

[0060]

[0071] 12 is a schematic diagram of a processor circuit 1250 according to an embodiment of the present disclosure. The processor circuit 1250 may be implemented in the ultrasound imaging system 140 or other device, a workstation (e.g., a third-party workstation, a network router, etc.), or a cloud processor or other remote processing unit, as needed to perform the method. The processor circuit 1250 includes a processor 1260, a memory 1264, and a communication module 1268, as shown. These elements communicate with each other directly or indirectly, for example, via one or more buses.

[0061]

[0072] Processor 1260 includes a central processing unit (CPU), digital signal processor (DSP), ASIC, controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other related logic devices, including mechanical and quantum computers. Processor 1260 also includes another hardware device, firmware device, or any combination thereof, configured to perform the operations described herein. Processor 1260 may also be implemented with a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration.

[0062]

[0073] Memory 1264 includes cache memory (e.g., processor 1260 cache memory), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, other forms of volatile and non-volatile memory, or a combination of various types of memory. In one embodiment, memory 1264 includes a non-transitory computer-readable medium. Memory 1264 stores instructions 1266. Instructions 1266 include instructions that, when executed by processor 1260, cause processor 1260 to perform the operations described herein. Instructions 1266 are also referred to as code. The terms “instructions” and “code” should be interpreted broadly to include any type of computer-readable description. For example, the terms "instructions" and "code" refer to one or more programs, routines, subroutines, functions, procedures, etc. "Instructions" and "code" include a single computer-readable statement or multiple computer-readable statements. The output of processor circuitry 1250 is sometimes visible on display 1270.

[0063]

[0074] The communications module 1268 may comprise any electronic and / or logic circuitry for facilitating direct or indirect communication of data between the processor circuit 1250 and other processors or devices. In that regard, the communications module 1268 may be an input / output (I / O) device. The communications module 1268 may facilitate direct or indirect communication between various elements of the processor circuit 1250 and / or ultrasound imaging system 140, as the case may be. The communications module 1268 communicates within the processor circuit 1250 by a number of methods or protocols. Serial communications protocols include US SPI, I / O, and the like. 2Serial protocols include, but are not limited to, I / O, RS-232, RS-485, CAN, Ethernet, ARINC429, MODBUS, MIL-STD-1553, or any other suitable method or protocol. Parallel protocols include, but are not limited to, ISA, ATA, SCSI, PCI, IEEE-488, IEEE-1284, and other suitable protocols. Serial and parallel communications may be bridged, as needed, by UART, USART, or any other suitable subsystem.

[0064]

[0075] External communications (including, but not limited to, software updates, firmware updates, sharing pre-configurations between the processor and a central server, or readings from the ultrasound device) are accomplished using any suitable wireless or wired communications technology, such as a cable interface such as a USB, micro-USB, Lightning, or FireWire interface, Bluetooth, Wi-Fi, ZigBee, Li-Fi, or a cellular data connection such as 2G / GSM, 3G / UMTS, 4G / LTE / WiMax, or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish a connection with a cloud service for data transmission and receiving software patches. The controller is configured to communicate with a remote server or a local device such as a laptop, tablet, or handheld device, or includes a display capable of displaying status variables and other information. Information may also be transferred on a physical medium, such as a USB flash drive or memory stick.

[0065]

[0076] As will be readily understood by those skilled in the art after becoming familiar with the teachings herein, an end effector state estimation system can incorporate information from surgical device control surfaces, ultrasound imaging volumes, and 2D x-ray images and generate image enhancements or annotations that provide clear, real-time guidance regarding the location, pose, and state of a transcatheter surgical device's end effector. It can thus be seen that the end effector state estimation system fulfills a long-standing need in the art by reducing the amount of skill and training required to interpret images used to perform image-intensive, transcatheter, minimally invasive surgical procedures. Output can take the form of guidance superimposition and associated visualization on ultrasound and x-ray images during MIS procedures, such as mitral valve leaflet repair. In some embodiments, the system includes physical modifications of the device with electromechanical sensors feeding a control unit and / or the presence of corresponding fiducials on a spatial tracking system and mechanical manipulator. The system, as described above, can be used to reliably and repeatedly derive the precise state of an implanted device or end effector.

[0066]

[0077] Several variations on the above examples and embodiments are possible. For example, the system may use advanced visualization based on on-screen rendering and render a 3D virtual space using augmented reality (AR) and virtual reality (VR) techniques. The system may be used to detect and / or trigger tissue-specific imaging presets required for ultrasound or X-ray imaging systems based on the state of the end effector. The system may also be used to create new device-specific imaging presets on ultrasound and X-ray imaging platforms that enhance the navigation experience. The end effector state estimation system may perform calculations based on 2D ultrasound images rather than 3D ultrasound volumes, or on 3D rather than 2D X-ray image data. Other imaging modalities may be used instead of or in addition to those listed above, including, but not limited to, intracardiac echocardiography (ICE), transthoracic ultrasound, intravascular ultrasound (IVUS), etc.

[0067]

[0078] Accordingly, the logical operations making up the embodiments of the technology described herein are referred to variously as operations, steps, objects, elements, components, or modules. Furthermore, it should be understood that logical operations may be performed, arranged, or executed in any order unless expressly recited otherwise in a claim or unless a particular order is inherently required by the claim language.

[0068]

[0079] It should be further understood that the described techniques may be used in other types of medical procedures, including veterinary procedures, and may also be used in non-medical procedures where a transcatheter end effector is used and detailed real-time knowledge of the state of the end effector is required to perform the procedure.

[0069]

[0080] All directional references, such as upper, lower, inner, outer, upward, downward, left, right, lateral, front, rear, above, below, above, below, vertical, horizontal, clockwise, counterclockwise, proximal, and distal, are used for identification purposes only to aid the reader's understanding of the claimed subject matter and do not create limitations with respect to the location, orientation, or use of the end effector state estimation system in particular. References to connections, such as attached, coupled, connected, and joined, should be interpreted broadly and include intermediate members between a collection of elements and relative movement between the elements, unless otherwise indicated. Thus, references to connections do not necessarily imply that two elements are directly connected and in a fixed relationship to each other. The term "or" should be interpreted to mean "and / or" rather than "exclusive or." The word "comprising" does not exclude other elements or steps, and the singular form "a," "an," or "an" does not exclude a plurality. Unless otherwise noted in the claims, the values ​​recited are to be construed as merely illustrative and not limiting.

[0070]

[0081] The above specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the end effector state estimation system defined in the claims. Although various embodiments of the claimed subject matter have been described above with a certain degree of particularity, or with reference to one or more individual embodiments, those skilled in the art could make numerous modifications to the disclosed embodiments without departing from the spirit or scope of the claimed subject matter.

[0071]

[0082] Still other embodiments are contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as merely illustrative of particular embodiments and not limiting. Changes in detail or structure may be made without departing from the basic elements of the subject matter defined in the following claims.

Claims

1. 1. A system for determining a deployment state of a treatment device coupled to a distal portion of a flexible elongate member disposed within a body cavity of a patient, the treatment device changing shape upon deployment, the system comprising: at least one sensor that measures at least one parameter related to the deployment state of the treatment device; Processor and wherein the processor: image data acquired by an imaging system, the image data representing the treatment device positioned within the body cavity of the patient; and The at least one parameter comprises a measure of expansion. Receive, using image recognition to determine the deployment state of the treatment device by matching an image generated based on the image data with a model of the treatment device; determining a confidence level for the matched model; determining the deployment state based on the deployment measurements when the confidence level falls below a threshold; and outputting a graphical representation of the deployed state of the treatment device on a display in communication with the processor; system.

2. the processor: an image of the body cavity of the patient generated based on the image data; the graphical representation of the deployed state of the treatment device superimposed on the image of the body cavity; The system of claim 1 , wherein the system outputs a screen display to the display, the screen display including:

3. The system of claim 2 , wherein the graphical representation of the deployed state of the treatment device includes a visualization of a three-dimensional model of the treatment device.

4. The system of claim 1 , wherein the at least one parameter includes at least one of a position or an angle of a control mechanism of the treatment device.

5. 10. The system of claim 1, wherein the system comprises a transcatheter delivery device, the transcatheter delivery device comprising a flexible elongate member, and the at least one parameter comprises at least one of a position or an angle of the transcatheter delivery device.

6. The system of claim 5 , wherein the at least one sensor includes an encoder coupled to a mechanical control mechanism of the transcatheter delivery device.

7. The system of claim 5 , wherein the at least one sensor includes a magnetic sensor that measures the at least one parameter by obtaining position measurements of magnetic seeds disposed on the treatment device.

8. The system of claim 1 , wherein the imaging system comprises an ultrasound imaging system.

9. The system of claim 1 , wherein the imaging system comprises an X-ray imaging system.

10. 1. A method of operating a system for determining a deployment state of a treatment device coupled to a distal portion of a flexible elongate member disposed within a body cavity of a patient, the treatment device changing shape upon deployment, the system comprising a sensor and a processor, the method comprising: the sensor measuring at least one parameter related to the deployment state of the treatment device, the parameter including a measure of deployment; receiving, by the processor, an image acquired by an imaging system, the image including the treatment device; the processor using image recognition to determine the deployment state of the treatment device by matching an image generated based on the image with a model of the treatment device; the processor determining a confidence level of the matched model; the processor determining the deployment state based on the deployment measurements when the confidence level falls below a threshold; outputting, by the processor, a graphical representation of the deployed state of the treatment device on a display in communication with the processor; The method of operation comprises:

11. The method of operation includes the processor displaying on the display: At least one of the images; the graphical representation of the deployed state of the treatment device superimposed on the at least one image; 11. The method of claim 10, further comprising the step of outputting a screen display including:

12. The method of claim 10 , wherein the at least one parameter comprises a position or angle of a control mechanism of the flexible elongate member.

13. The method of claim 10 , wherein the at least one parameter includes a position or angle of the treatment device.

14. The method of claim 10 , wherein the imaging system comprises an ultrasound imaging system.

15. The method of claim 10 , wherein the imaging system comprises an X-ray imaging system.

16. 11. The method of claim 10, wherein the sensor includes a magnetic sensor, and the step of measuring the at least one parameter by the sensor comprises the step of the sensor obtaining position measurements of magnetic seeds positioned on the treatment device.

17. 1. A system for determining a deployment state of a mitral valve clip, the system comprising: catheters, and a mitral valve clip coupled to a distal portion of the catheter; a mitral valve clip deployment device comprising: an ultrasound imaging device that acquires ultrasound images of the mitral valve clip positioned within a body cavity of a patient; an x-ray imaging device that acquires x-ray images of the mitral valve clip positioned within the body cavity of the patient; a sensor coupled to the mitral valve clip deployment device to provide a sensor signal indicative of a parameter related to the deployment state of the mitral valve clip, the parameter including a measure of deployment; determining the deployment state of the mitral valve clip by matching an image generated based on the x-ray image with a model of the mitral valve clip using image recognition; determining a confidence level for the matched model; and determining the deployment state based on the deployment measurements when the confidence level falls below a threshold; and to a display that communicates with the processor, at least one of the ultrasound images; and a graphical representation of the deployed state of the mitral valve clip; Output The processor and A system comprising:

Images