Augumented ultrasound imaging for minimally-invasive procedures

Abstract

Systems and methods are provided for guiding a minimally-invasive procedure. A system includes an imaging system that provides an image of a region of interest and instructions are executable by the processor to provide a tracking sensor interface that receives a position of a first medical device within the region of interest. An overlay generator generates a model of one or both of the first medical device and a second medical device, having a known position and orientation relative to the first medical device, based on the position of the first medical device and a state associated with the device. An image merger registers the overlay to the image and combines the overlay with the image to generate the augmented representation of the region of interest. A display provides the augmented representation of the region of interest to a user in real-time.

Description

AUGUMENTED ULTRASOUND IMAGING FOR MINIMALLY-INVASIVE PROCEDURESRELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 664,551 , filed June 26, 2024, the entirety of which is hereby incorporated by reference for all purposes.TECHNICAL FIELD

[0002] This invention relates to medical imaging systems, and more particularly, to augmented ultrasound imaging for minimally-invasive procedures.BACKGROUND

[0003] Current clinical methods of imaging anatomy during minimally-invasive procedures using ultrasound, for example, transesophageal echocardiography, have been shown to be feasible. Unfortunately, further clinical experience has identified challenges with these imaging modalities which require significant and constant monitoring. In general, these issues add significant cognitive burden and stress during the procedure because of the high level of vigilance required to perfectly position a medical device relative to an anatomical landmark, while avoiding contact between the delivery system and anatomical features that are inherently difficult to see with typical echogram views, cropped out of view, or shadowed by the delivery system itself.SUMMARY

[0004] In accordance with one example, a perioperative imaging system is provided for guiding a minimally-invasive procedure. The system includes an echograph system that provides an image of a region of interest, a processor, and a non-transitory computer readable medium that stores instructions executable by the processor for providing an augmented representation of the image of the region of interest. The instructions are executable by the processor to provide a tracking sensor interface configured to receive a position of a first medical device within the region of interest. An overlay generator generates a model of one of the first medicaldevice and a second medical device, having a known position and orientation relative to the first medical device, based on the position of the first medical device and a state associated with the one of the first medical device and the second medical device. An image merger registers the overlay to the image of the region of interest and combines the overlay with the image of the region of interest to generate the augmented representation of the region of interest. A display provides the augmented representation of the region of interest to a user in real-time.

[0005] In accordance with another example, a method is provided for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium. The method includes acquiring an image of a region of interest from an echograph system and detecting a position of a deployment device associated with the medical implant within the region of interest. The medical implant has a known position and orientation relative to the deployment device. A model of one or both of the deployment device and the medical implant is generated based on the position of the deployment device and a state associated with the either or both of the deployment device and medical implant. The overlay is registered to the image of the region of interest and combined with the image of the region of interest to generate the augmented representation of the region of interest. The augmented representation of the region of interest is provided to a user in real-time.

[0006] In accordance with a further example, a perioperative imaging system is provided for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium. The perioperative imaging system includes an echograph system that provides an image of a region of interest, a processor, and a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest. The instructions are executable by the processor to provide a tracking sensor interface that receives a position of a deployment device within the region of interest. An overlay generator generates a model of one or both of the deployment device and the medical implant based on the position of the deployment device and a state associated with one or both of the deployment device and the medical implant. The medical implant has a known position and orientation relative to the deployment device. The overlay generator determines a current stage of the insertion of themedical implant at a machine learning system from the position of the deployment device and the image of the region of interest and selects the state associated with the one of the deployment device and the medical implant from a plurality of states according to the current stage. An image merger registers the overlay to the image of the region of interest and combines the overlay with the image of the region of interest to generate the augmented representation of the region of interest.

[0007] In accordance with a still further example, a method is provided for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium. The method includes acquiring an image of a region of interest from an echograph system and detecting a position of a deployment device associated with the medical implant within the region of interest. The medical implant has a known position and orientation relative to the deployment device. A model of one or both of the deployment device and the medical implant is generated based on the position of the deployment device and a state associated with either or both of the deployment device and the medical implant. The overlay is registered to the image of the region of interest and combined with the image of the region of interest to generate the augmented representation of the region of interest. The augmented representation of the region of interest is provided to a user in real-time. The user is allowed to select a position associated with an anatomical landmark at a user interface, and an indicator is displayed at the selected position.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 illustrates an example of system for guiding a minimally-invasive procedure;

[0009] FIG. 2 illustrates another example of a system for guiding a minimally- invasive procedure;

[0010] FIG. 3 illustrates an example of a system for guiding an automated system in a minimally-invasive procedure;

[0011] FIG. 4 illustrates one example of a method for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium;

[0012] FIG. 5 illustrates another example of a method for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium; and

[0013] FIG. 6 is a schematic block diagram illustrating an exemplary system of hardware components capable of implementing examples of the systems and methods disclosed herein.DEFINITIONS

[0014] A “machine learning model,” as used herein, is a mathematical or machine learning model that generates a parameter representing an unknown quantity from provided parameters, images, audio, or other input data. Examples of machine learning models include artificial neural networks, convolutional neural networks, convolutional autoencoders, linear regression models, logistic regression models, Bayesian networks, such as naive-bayes, random forest models, boosting and bagging methods, decision trees, hidden Markov models, support vector machines, K-means clustering, and K-nearest neighbor classifiers.

[0015] A “minimally-invasive procedure,” as used herein, is a procedure that is performed using small incisions with an imaging device to guide the procedure.

[0016] A “medical device” as used herein, is a medical instrument or medical device used within a minimally-invasive procedure. It will be appreciated that the term “medical device” explicitly includes implantable medical devices, capsules used for deploying medical implants, and any device, such as a catheter, used to navigate an implantable device to an appropriate location with the body.

[0017] A “deployment device,” as used herein, is a medical device that is used to navigate a medical implant through the cardiovascular system to a target location.

[0018] A “state” of a medical device refers to a configuration of a medical device that differs in size, position, orientation / angle (e.g., degree of steering, etc.), and / or shape from other configurations of a medical device. Examples of changes to a state of a device include the linear or annular extension or retraction of a portion of the device, the movement or rotation of a device, etc.

[0019] A “stage” of a minimally-invasive procedure is a discrete step in the procedure, usually associated with a specific action taken to advance the procedure. In the example of a transcatheter mitral valve repair, the procedure can be divided into eight steps:

[0020] 1 . Advancement of the catheter sheath into the left atrium;

[0021] 2. Advancement of the catheter into the left atrium;

[0022] 3. Advancement of the catheter across the mitral valve annulus;

[0023] 4. Partial expansion of the replacement valve;

[0024] 5. Advancement of a deployment capsule into the mitral valve to a target location;

[0025] 6. Expansion of a fixation ring associated with the replacement valve;

[0026] 7. Full deployment of the replacement valve in the mitral valve annulus; and

[0027] 8. Retraction of the delivery system.DETAILED DESCRIPTION

[0028] Systems and methods are provided herein for guiding a minimally- invasive procedure using an ultrasound image and / or image(s) from one or more other imaging modalities (e.g., fluoroscopy, computed tomography (CT), magnetic resonance imaging (MRI), etc.). Specifically, one or more medical devices can be displayed as an avatar including alerts and projections of components of the system for which the location is not directly tracked. This provides a real-time position for the devices in a three-dimensional workspace for reference throughout the procedure, allowing the position of the avatar relative to anatomy to be visualized on ultrasound. This avatar can also be used for automated collision detection and automated alignment, with positions of anatomical landmarks and other key points for a given procedure either indicated by the user or determined via automated segmentation of the ultrasound (etc.) image(s).

[0029] FIG. 1 illustrates one example of a system 100 for guiding a minimally- invasive procedure. In one example, the surgical procedure is implantation of a medical device into an intracardiac space, such as a heart valve or cardiac ostium. The system 100 includes an echograph (and / or other imaging modality) system 102 (e.g., for imaging via ultrasound and / or one or more additional imaging modalities, etc.), a processor 104, a display 106, and a non-transitory computer readable medium 110 storing executable instructions, executed by the processor 104. It will be appreciated that the executable instructions can be spread across multiple non-transitory computer readable media that are operatively connected via an appropriate data connection, such that the executable instructions can be executed by multiple processors.

[0030] The executable instructions stored on the non-transitory computer readable medium 110 include a tracking sensor interface 112 configured to receive a position of a first medical device within the region of interest. It will be appreciated that the location of the first medical device can be determined using a surgical tracking system and one or more sensors implemented on the first medical device and provided to the system 100 via the tracking sensor interface 112. Using the example of a medical implant, a tracking sensor can be implemented on either a catheter used to guide an implant capsule or the capsule itself, and the state of either or both of the capsule and the implant can be tracked.

[0031] An overlay generator 114 that generates a model of either or both of a first medical device and a second medical device, having a known position and orientation relative to the first medical device, based on the position of the first medical device and a state associated with either the first medical device or the second medical device. The overlay generator 114 can store one or more models associated with various medical devices as three-dimensional models, with at least a subset of the models representing devices having multiple states. In one implementation, a medical device can be recognized by the overlay generator 114 when it is plugged in to a delivery system, and an appropriate model can be selected. In another implementation, a user can manually enter an identity of the device.

[0032] A device having multiple states can be represented by multiple models, with each model representing a different state, or a configurable model in which each state is associated with a different configuration of the model. The state of a given device can be selected directly by a user or it can be responsive to a given stage of the surgery, as indicated by the user or determined by the overlay generator 1 14. In various examples, image(s) from various imaging systems are stored on the non- transitory computer readable medium 110 and registered three-dimensionally into the overlay generator 114 (e.g., in examples involving two or more imaging modalities, the imaging is co-registered to each other, etc.). In one implementation,the overlay generator 114 determines the current stage of the minimally-invasive procedure from the position of the first medical device and the image(s) of the region of interest. For example, when positioning a medical implant, each stage of the procedure can be associated with a position and orientation of the medical implant, delivery capsule, or catheter relative to an anatomical landmark and a state of the implant or capsule, allowing the current stage to be determined without user input.

[0033] An image merger 116 registers the overlay generated at the overlay generator to the image(s) of the region of interest and combines the overlay with the image(s) of the region of interest to generate an augmented representation of the region of interest. Essentially, the position and orientation of the first medical device and a known field of view of the ultrasound (etc.) imager can be used to project the three-dimensional models of either or both of the first and second medical devices to a plane associated with the ultrasound (etc.) image(s), allowing an avatar representation of the model to be correctly shaped and positioned on the image(s) relative to the surrounding anatomy. As a result, the user can locate all portions of the device and understand the position and orientation of the device (including positions that would result from steering, etc.) even when obscured by anatomy or shadowed by other components. The output of the imager merger 116 can be provided to a user at the display 106.

[0034] FIG. 2 illustrates another example of a system 200 for guiding a minimally-invasive procedure. In the illustrated implementation, the minimally- invasive procedure is the insertion of a medical implant into either a heart valve or a cardiac ostium. In one example, the surgical procedure is a transcatheter procedure, such as a transcatheter mitral valve replacement or repair, a transcatheter tricuspid valve replacement or repair, etc. The system 200 includes an echograph (etc.) system 202 (e.g., for imaging via ultrasound and / or one or more additional imaging modalities, etc.), a processor 204, a display 206, and a non-transitory computer readable medium 210 storing executable instructions, executed by the processor 204. It will be appreciated that the executable instructions can be spread across multiple non-transitory computer readable media that are operatively connected via an appropriate data connection, such that the executable instructions can be executed by multiple processors.

[0035] The executable instructions stored on the non-transitory computer readable medium 210 include an imager interface 212 that receives image(s) of the heart of the patient from the echograph (etc.) system 202 (e.g., via ultrasound and / or one or more other imaging modalities, such as fluoroscopy, CT, MRI, etc.). It will be appreciated that the imager interface 212 can provide conditioning or formatting for the received image(s) for further processing. A tracking sensor interface 214 is configured to receive a position of a deployment device within the region of interest. It will be appreciated that the location of the deployment can be determined using a surgical tracking system and one or more sensors implemented on the deployment device and provided to the system 200 via the tracking sensor interface 214. In one example, the tracking sensor can be implemented as a device containing at least one conductive coil in which an electrical signal is induced based on a magnetic flux change of the EM field. The location of the medical instrument in a coordinate system of the EM field is identified based on the induced electrical signal in the sensor. One example of such a system can be found in U.S. Published Patent Application No. US 2016 / 0174873, filed October 22, 2015 and entitled “Medical instrument with sensor for use in a system and method for electromagnetic navigation.” The entire content of this application is hereby incorporated by reference.

[0036] The executable instructions further include an overlay generator 216 that generates a model of either or both of the deployment device and the implant, based on the detected position of the deployment device and a state associated with either the deployment device and the implant. The overlay generator 216 can store one or more models associated with various medical devices as three-dimensional models, with at least a subset of the models representing devices having multiple states. In one implementation, a medical device can be recognized by the overlay generator 216 when it is plugged in to a delivery system, and an appropriate model can be selected. In another implementation, a user can manually enter an identity of the device.

[0037] A device having multiple states can be represented by multiple models, with each model representing a different state, or a configurable model in which each state is associated with a different configuration of the model. For example, thestates of a model representing a medical device, such as a medical implant or deployment device, can include a first state in which an annular member associated with the model is deployed and a second state in which the annular member is not deployed. Similarly, the states of the model can include a first state in which an axial extension associated with the model is deployed and a second state in which the axial extension is not deployed. The state of a given device can be selected directly by a user or it can be responsive to a given stage of the surgery, as indicated by the user via a user interface 218 or determined by the overlay generator 216. For example, when positioning a medical implant, each stage of the procedure can be associated with a position and orientation of the medical implant, delivery capsule, or catheter relative to an anatomical landmark and a state of the implant or capsule, allowing the current stage to be determined without user input.

[0038] In one implementation, the overlay generator 216 can include a machine learning model 219 that determines the current stage of the minimally-invasive procedure from at least the image(s) of the region of interest. The machine learning model 219 can operate with parameters extracted from the image(s) and the known position of the deployment device, for example, calculated distances between the deployment device or medical implant and various anatomical landmarks, or operate directly on the ultrasound (etc.) image(s). The machine learning model 219 can be implemented as any of a plurality of machine learning algorithms or combinations of multiple machine learning algorithms, including artificial neural networks, rule-based classifiers, logistic regression models, Bayesian networks, boosting and bagging models, random forest models, hidden Markov models, and support vector machines. In the one example, the machine learning model 219 uses one or more supervised learning algorithms, each trained on a set of training samples, with a given training sample containing one or both of an ultrasound (etc.) image and a set of parameters extracted from an ultrasound (etc.) image and a known stage of the procedure associated with the ultrasound (etc.) image. Where multiple classification or regression models are used, an arbitration element can be utilized to provide a coherent result from the plurality of models. The training process of a given classifier will vary with its implementation, but training generally involves a statistical aggregation of training data into one or more parameters associated with the outputclass. For rule-based models, such as decision trees, domain knowledge, for example, as provided by one or more human experts, can be used in place of or to supplement training data in selecting rules for determining a current stage of the procedure.

[0039] For example, a support vector machine (SVM) classifier can utilize a plurality of functions, referred to as hyperplanes, to conceptually divide boundaries in the N-dimensional feature space, where each of the N dimensions represents one associated feature of the feature vector. The boundaries define a range of feature values associated with each class. Accordingly, an output class, representing the current stage, and an associated confidence value can be determined for a given input feature vector according to its position in feature space relative to the boundaries. In one implementation, the SVM can be implemented via a kernel method using a linear or non-linear kernel.

[0040] An artificial neural network includes a plurality of nodes having a plurality of interconnections, referred to as links. Input values are provided to a plurality of input nodes, which provide these input values to layers of one or more intermediate nodes, referred to as hidden nodes. A given intermediate node receives one or more output values from previous nodes, which are weighted according to a series of weights established during the training of the classifier. An intermediate node translates its received values into a single output according to a transfer function at the node. For example, the intermediate node can sum the received values and subject the sum to a binary step function, a sigmoid function, a hyperbolic tangent function, or a rectified linear unit function. A final layer of nodes provides the confidence values for the output classes of the artificial neural network, with each node having an associated value representing a confidence for one of the stages of the procedure.

[0041] A rule-based classifier applies a set of logical rules to the extracted features to select an output class. Generally, the rules are applied in order, with the logical result at each step influencing the analysis at later steps. The specific rules and their sequence can be determined from any or all of training data, analogical reasoning from previous cases, or existing domain knowledge. One example of a rule-based classifier is a decision tree algorithm, in which the values of features in afeature set are compared to corresponding threshold in a hierarchical tree structure to select a class for the feature vector. A random forest classifier is a modification of the decision tree algorithm using a bootstrap aggregating, or “bagging” approach. In this approach, multiple decision trees are trained on random samples of the training set, and an average (e.g., mean, median, or mode) result across the plurality of decision trees is returned. For a classification task, the result from each tree would be categorical, and thus a modal outcome can be used to determine the current stage of the procedure.

[0042] The output of the overlay generator 216 can be provided to an alignment component 220 that determines if a current position of the deployment device would result in an acceptable deployment of the medical implant from the image(s) of the region of interest. In one implementation, the alignment component 220 can determine if the alignment is appropriate when the current position of the deployment device is within a first range of positions relative to an anatomical landmark and a current orientation of the deployment device is within a first range of angles relative to the anatomical landmark. Additionally or alternatively, the alignment component 220 can determine that the deployment will be acceptable when a projected cross- sectional area of a ventricular outflow tract after deployment of the device is above a threshold value. Additionally or alternatively, in various examples, the output of the overlay generator 216 is used to visually display data such as measurements, distances, and / or angles from one or more defined locations, independent of any indication of whether the current position of the deployment device would result in an acceptable deployment of the medical implant, allowing a user to determine acceptability, etc. based on the visually displayed data.

[0043] A collision detector 222 also receives the image(s) and model from the overlay generator 216 determines if either of the deployment device and the medical implant is within a threshold distance of one or more of tissue generally, a user selected position, a machine-detected position, or an anatomical landmark. In one implementation, the collision detector 222 can segment the ultrasound (etc.) image(s), for example, using a convolutional neural network trained on labelled examples, such as the U-Net architecture, to establish the tissue boundaries. The user selected positions can be determined by a user at an appropriate user interface218 (machine-detected positions can be detected via techniques discussed herein, such as via artificial intelligence models, etc.)- It will be appreciated the collision detector 222 can also be performed prospectively by altering the dimensions, position, orientation, etc. of the model representing the deployment device or the medical implant to simulate a change in the state of the device and determining if this would result in an unwanted collision with tissue. In one example, the change in the state of the device can be disallowed based on this determination.

[0044] Further, either or both of the display of user-selected (or machine- detected) positions and collision alerts associated with these positions can depend on the state of either of the deployment device and the medical implant. For example, a given user-selected (or machine-detected, etc.) position may be displayed when the device or implant is in a given state, but not displayed when the device or implant is in another state. In various examples, alerts (e.g., visual, auditory, etc.) are generated based on detection of actual or potential collisions, such as based on an ultrasound or on an additional co-registered scan (e.g., CT, MRI, etc.). For example, an area of collision or potential collision can be indicated by a change in color or otherwise annotating the image. Additionally or alternatively, in response to detected collision(s), cross-section(s) or portions of the anatomy associated with the collision are displayed to indicate the region at risk of collision. Similarly, collision detection associated with the user-selected (or machine-detected) position may be active when the device or implant is in a given state, but not active when the device or implant is in another state. This allows for both computing resources and visible screen area to be conserved when the procedure is in a stage in which display and monitoring of the user-selected (or machine-detected) position is unlikely to be helpful.

[0045] An image merger 224 registers the overlay to the image(s) of the region of interest and combines the overlay with the image(s) of the region of interest to generate the augmented representation of the region of interest. Specifically, the overlay generated from the 3-D model at the overlay generator 216 is placed in the appropriate position and orientation within the ultrasound (etc.) image(s) to allow the user to clearly see the position and orientation of either or both of the deployment device and the medical implant. It will be appreciated that this operation isperformed in real-time to allow the indicated position to be responsive to movement of the devices during the procedure. The output of the imager merger 220 can be provided to a user at the display 206.

[0046] FIG. 3 illustrates an example of a system 300 for guiding an automated system (not shown) in a minimally-invasive procedure. In one example, the surgical procedure is implantation of a medical device into an intracardiac space, such as a heart valve or cardiac ostium. The system 300 includes an echograph (etc.) system 302 (e.g., an echograph system and / or one or more other imaging systems, such as fluoroscopy, CT, MRI, etc.), a processor 304, and a non-transitory computer readable medium 310 storing executable instructions, executed by the processor 304. It will be appreciated that the executable instructions can be spread across multiple non-transitory computer readable media that are operatively connected via an appropriate data connection, such that the executable instructions can be executed by multiple processors.

[0047] The executable instructions stored on the non-transitory computer readable medium 310 include a tracking sensor interface 312 configured to receive a position of a deployment device within the region of interest. It will be appreciated that the location of the deployment device can be determined using a surgical tracking system and one or more sensors implemented on the deployment and provided to the system 300 via the tracking sensor interface 312. In another implementation, the location can be provided directly as feedback from the robotic surgery system using a combination of one or more tracking sensors and the known movements of the robotic system relative to a coordinate system associated with the procedure. In one implementation, one or more imaging modalities, such as computed tomography or magnetic resonance imaging, can be used to provide a three-dimensional model that includes the region of interest. In examples including two or more imaging modalities (e.g., ultrasound and CT / MRI, etc.), imaging from the two or more imaging modalities is co-registered. In various examples, imaging data from a first imaging modality is used to generate one or more of measurements, collision detection, predictions, etc. based on imaging data from the same imaging modality or another imaging modality co-registered to the first imaging modality. For example, imaging data (e.g., including pre-operative imaging data, etc.) from CTand / or MRI can be co-registered to imaging data (e.g., peri-operative imaging data, etc.) from ultrasound and / or fluoroscopy, and tracked (e.g., in a 3D model, etc.) together. Based on live ultrasound data, measurements, collision detection, predictions, etc. can be generated based on any tracked data, allowing for generation of alerts based on features visible on live ultrasound, or features not visible on live ultrasound but tracked based on co-registered imaging data (e.g., CT, MRI, etc.).

[0048] An overlay generator 314 that generates a model of either or both of the deployment device and the medical implant based on the position of the first medical device and a state associated with either the deployment device or the medical implant. The overlay generator 314 can store one or more models associated with various medical devices as three-dimensional models, with at least a subset of the models representing devices having multiple states. In one implementation, a medical device can be recognized by the overlay generator 314 when it is plugged in to a delivery system, and an appropriate model can be selected. In another implementation, a user can manually enter an identity of the device.

[0049] A device having multiple states can be represented by multiple models, with each model representing a different state, or a configurable model in which each state is associated with a different configuration of the model. In the illustrated implementation, the state of a given device is responsive to a given stage of the surgery. In one implementation, the overlay generator 314 determines the current stage of the minimally-invasive procedure from the position of the first medical device and the image(s) of the region of interest. For example, when positioning a medical implant, each stage of the procedure can be associated with a position and orientation of the medical implant, delivery capsule, or catheter relative to an anatomical landmark and a state of the implant or capsule, allowing the current stage to be determined without user input. Alternatively, a machine learning model 315, such as the machine learning model 219 described with respect to the system of FIG 2, can be employed to determine a current stage of procedure and control the change in state of the device.

[0050] An image merger 316 registers the overlay generated at the overlay generator to the image(s) of the region of interest and combines the overlay with theimage(s) of the region of interest to generate an augmented representation of the region of interest. Essentially, the position and orientation of the first medical device and a known field of view of the ultrasound (etc.) imager(s) can be used to project the three-dimensional models of either or both of the first and second medical devices to a plane associated with the ultrasound (etc.) image(s), allowing an avatar representation of the model to be correctly shaped and positioned on the image(s) relative to the surround anatomy. Additionally, in various examples, the position and orientation of the device are used to inform the direction and position that portion(s) of the device will move in response to steering. This augmented representation can be used to facilitate alignment of the device and collision avoidance by the robotic surgical system. The output of the imager merger 316 can be provided to the robotic surgical system via a surgical system interface 318.

[0051] In view of the foregoing structural and functional features described above, methods in accordance with various aspects of the present invention will be better appreciated with reference to FIGS. 4 and 5. While, for purposes of simplicity of explanation, the methods of FIGS. 4 and 5 are shown and described as executing serially, it is to be understood and appreciated that the present invention is not limited by the illustrated order, as some aspects could, in accordance with the present invention, occur in different orders and / or concurrently with other aspects from that shown and described herein. Moreover, not all illustrated features may be required to implement a method in accordance with an aspect of the present invention.

[0052] FIG. 4 illustrates one example of a method 400 for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium. At 402, at least one image of a region of interest is acquired from an echograph (etc.) system (e.g., via ultrasound and / or one or more other imaging modalities, such as fluoroscopy, CT, MRI, etc.) and a position of a deployment device associated with the medical implant is detected within the region of interest at 404. It will be appreciated that the medical implant is carried through the cardiovascular system by the deployment device and will have a known position and orientation relative to the deployment device throughout the procedure.

[0053] A model of one or both of the deployment device and the medical implant is generated at 406 based on the position of the deployment device and a state associated with the either or both of the deployment device and medical implant. In one example, the state is selected from a plurality of states that include at least a first state in which an annular member associated with the device is deployed, and at least a second state in which the annular member is not deployed. Additionally or alternatively, the state can be selected from a plurality of states that include at least a first state in which an axial extension associated with the device is deployed, and at least a second state in which the axial extension is not deployed. In one implementation, a current stage of the insertion of the medical implant can be determined and the state can be selected from the plurality of states according to the current stage. The current stage can be provided by the user via a user interface or determined at a machine learning system from at least the position of the deployment device and the image(s) of the region of interest. The overlay is registered to the image(s) of the region of interest at 408 and combined with the image(s) of the region of interest to generate the augmented representation of the region of interest at 410. The augmented representation of the region of interest is provided to a user in real-time at 412.

[0054] FIG. 5 illustrates a further example of a method 500 for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium. The method includes acquiring at least one image of a region of interest from an echograph (etc.) system at 502 (e.g., via ultrasound and / or one or more other imaging modalities, such as fluoroscopy, CT, MRI, etc.) and detecting a position of a deployment device associated with the medical implant within the region of interest at 504. It will be appreciated that the medical implant is carried through the cardiovascular system by the deployment device and will have a known position and orientation relative to the deployment device throughout the procedure.

[0055] A model of one or both of the deployment device and the medical implant is generated at 506 based on the position of the deployment device and a state associated with either or both of the deployment device and the medical implant. In one example, the state can be selected from a plurality of states that include at least a first state in which an extension associated with the device is deployed, and atleast a second state in which the extension is not deployed. In one implementation, a current stage of the insertion of the medical implant can be determined and the state can be selected from the plurality of states according to the current stage. The current stage can be provided by the user via a user interface or determined at a machine learning system from at least the position of the deployment device and the image(s) of the region of interest.

[0056] The overlay is registered to the image(s) of the region of interest at 508 and combined with the image(s) of the region of interest at 510 to generate the augmented representation of the region of interest. The augmented representation of the region of interest is provided to a user in real-time at 512. The augmented representation can also be used to determine if a current position of the deployment device would result in an acceptable deployment of the medical implant from the image(s) of the region of interest, for example, based on the proximity of the device to tissue boundaries and anatomical landmarks.

[0057] The user is allowed to select a position associated with an anatomical landmark at a user interface at 514, and an indicator is displayed at the selected position at 516. In various examples, the user can additionally or alternatively select or define a path to navigate to the anatomical landmark. It will be appreciated that display of user-selected (or machine-detected) positions (and / or paths) can depend on the state of either of the deployment device and the medical implant. For example, a user-selected (or machine-detected) position / path may be displayed when the device or implant is in a given state, but not displayed when the device or implant is in another state. At 518, the user is alerted if the model of the one of the deployment device and the medical implant is within a threshold distance of the selected position. This alert, like the display of the user-selected position, can be conditioned on the state of the deployment device or medical implant, such that an alert is given when the model is within the threshold distance and the device or implant is in a given state, but not given when the device or implant is in another state even if the model is within the threshold distance.

[0058] FIG. 6 is a schematic block diagram illustrating an exemplary system 600 of hardware components capable of implementing examples of the systems and methods disclosed herein. The system 600 can include various systems andsubsystems. The system 600 can be a personal computer, a laptop computer, a workstation, a computer system, an appliance, an application-specific integrated circuit (ASIC), a server, a server BladeCenter, a server farm, etc.

[0059] The system 600 can include a system bus 602, a processing unit 604, a system memory 606, memory devices 608 and 610, a communication interface 612 (e.g., a network interface), a communication link 614, a display 616 (e.g., a video screen), and an input device 618 (e.g., a keyboard, touch screen, and / or a mouse). The system bus 602 can be in communication with the processing unit 604 and the system memory 606. The additional memory devices 608 and 610, such as a hard disk drive, server, standalone database, or other non-volatile memory, can also be in communication with the system bus 602. The system bus 602 interconnects the processing unit 604, the memory devices 606-610, the communication interface 612, the display 616, and the input device 618. In some examples, the system bus 602 also interconnects an additional port (not shown), such as a universal serial bus (USB) port.

[0060] The processing unit 604 can be a computing device and can include an application-specific integrated circuit (ASIC). The processing unit 604 executes a set of instructions to implement the operations of examples disclosed herein. The processing unit can include a processing core.

[0061] The additional memory devices 606, 608, and 610 can store data, programs, instructions, database queries in text or compiled form, and any other information that may be needed to operate a computer. The memories 606, 608 and 610 can be implemented as computer-readable media (integrated or removable), such as a memory card, disk drive, compact disk (CD), or server accessible over a network. In certain examples, the memories 606, 608 and 610 can include text, images, video, and / or audio, portions of which can be available in formats comprehensible to human beings.

[0062] Additionally or alternatively, the system 600 can access an external data source or query source through the communication interface 612, which can communicate with the system bus 602 and the communication link 614.

[0063] In operation, the system 600 can be used to implement one or more parts of a system for generating augmented representations of ultrasound imaging forminimally-invasive procedures in accordance with the present invention. Computer executable logic for implementing the diagnostic system resides on one or more of the system memory 606, and the memory devices 608 and 610 in accordance with certain examples. The processing unit 604 executes one or more computer executable instructions originating from the system memory 606 and the memory devices 608 and 610. The term "computer readable medium" as used herein refers to a medium that participates in providing instructions to the processing unit 604 for execution. This medium may be distributed across multiple discrete assemblies all operatively connected to a common processor or set of related processors.

[0064] Implementation of the techniques, blocks, steps, and means described above can be done in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions described above, and / or a combination thereof.

[0065] Also, it is noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in the figure. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0066] Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting language, and / or microcode, the program code orcode segments to perform the necessary tasks can be stored in a machine readable medium such as a storage medium. A code segment or machine-executable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and / or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, ticket passing, network transmission, etc.

[0067] For a firmware and / or software implementation, the methodologies can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions can be used in implementing the methodologies described herein. For example, software codes can be stored in a memory. Memory can be implemented within the processor or external to the processor. As used herein the term "memory" refers to any type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.

[0068] Moreover, as disclosed herein, the term "storage medium" can represent one or more memories for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other machine readable mediums for storing information. The term "machine-readable medium" includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage mediums capable of storing that contain or carry instruction(s) and / or data.

[0069] In the preceding description, specific details have been set forth in order to provide a thorough understanding of example implementations of the invention described in the disclosure. However, it will be apparent that various implementations may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown ascomponents in block diagram form in order not to obscure the example implementations in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the examples. The description of the example implementations will provide those skilled in the art with an enabling description for implementing an example of the invention, but it should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention. Accordingly, the present invention is intended to embrace all such alterations, modifications, and variations that fall within the scope of the appended claims.

[0070] The following additional examples are provided in connection with various aspects:1 . A perioperative imaging system for guiding a minimally-invasive procedure, the perioperative imaging system comprising: an imaging system that provides an image of a region of interest; a processor; a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest, the executable instructions comprising: a tracking sensor interface that receives a position of a first medical device within the region of interest; an overlay generator that generates a model of one of the first medical device and a second medical device, having a known position and orientation relative to the first medical device, based on the position of the first medical device and a state associated with the one of the first medical device and the second medical device; and an image merger that registers the overlay to the image of the region of interest and combines the overlay with the image of the region of interest to generate the augmented representation of the region of interest; anda display that provides the augmented representation of the region of interest to a user in real-time.2. The perioperative imaging system of example 1 , wherein the executable instructions further comprise a user interface, the state associated with the one of the first medical device and the second medical device being selectable by the user via the user interface.3. The perioperative imaging system of example 2, wherein the user interface allows the user to indicate a current stage of the minimally-invasive procedure, the state associated with the one of the first medical device and the second medical device being responsive to the indicated stage of the minimally-invasive procedure.4. The perioperative imaging system of any of examples 2-3, wherein the user interface allows the user to select a position associated with an anatomical landmark, the overlay generator displaying an indicator at the selected position.5. The perioperative imaging system of example 4, wherein the overlay generator displays the indicator at the selected position when the state associated with the one of the first medical device and the second medical device is a first state and does not display the indicator at the selected position when the state associated with the one of the first medical device and the second medical device is a second state.6. The perioperative imaging system of any of examples 4-5, wherein the executable instructions further comprise a collision detector that determines if the model of the one of the first medical device and the second medical device is within a threshold distance of the selected position.7. The perioperative imaging system of any of examples 4-6, wherein the executable instructions further comprise a collision detector that determines if the model of the one of the first medical device and the second medical device is within a threshold distance of a tissue boundary.8. The perioperative imaging system of example 7, wherein the collision detector determines if a change in the state of the one of the first medical device and the second medical device will result in contact with a tissue boundary.9. The perioperative imaging system of example 8, wherein the collision detector does not allow a change in the state of the one of the first medical device and the second medical device if the change in the one of the first medical device and the second medical device will result in contact with a tissue boundary.10. The perioperative imaging system of any of examples 8-9, wherein the collision detector does not allow a change in the state of the one of the first medical device and the second medical device if the change in the one of the first medical device and the second medical device will result in deployment outside of an acceptable range of positions for the first medical device or the second medical device.11 . The perioperative imaging system of any of examples 1 -10, wherein the overlay generator selects the model of one of the first medical device and the second medical device according to an identifier associated with the one of the first medical device and the second medical device, the overlay generator receiving the identifier automatically when the one of the first medical device and the second medical device is connected to the perioperative imaging system.12. The perioperative imaging system of any of examples 1 -11 , wherein the executable instructions further comprise an alignment component that determines if a current position of the first medical device would result in an acceptable deployment of the second medical device from the image of the region of interest.13. The perioperative imaging system of example 12, wherein the current position of the first medical device is determined to result in the acceptable deployment of the second medical device when the current position of the first medical device is within a first range of positions relative to an anatomical landmark and a current orientation of the first medical device is within a first range of angles relative to the anatomical landmark.14. The perioperative imaging system of any of examples 12-13, wherein the current position of the first medical device is determined to result in the acceptable deployment of the second medical device when a projected cross-sectional area of a ventricular outflow tract is above a threshold value.15. The perioperative imaging system of any of examples 1 -14, wherein the minimally-invasive procedure is insertion of the second medical device into a one of a heart valve and a cardiac ostium and the first medical device comprises a catheter.16. The perioperative imaging system of any of examples 1 -15, wherein the overlay generator determines a current stage of the minimally-invasive procedure from at least the position of the first medical device and the image of the region of interest, the state associated with one of the first medical device and the second medical device being responsive to the indicated stage of the minimally-invasive procedure.17. The perioperative imaging system of any of examples 1 -16, wherein the imaging system is one of an echograph system, a fluoroscopy system, a computed tomography system, or a magnetic resonance imaging system.18. The perioperative imaging system of any of examples 1 -17, wherein the overlay generator generates the model based at least in part on additional imaging data, wherein the additional imaging data is associated with a different imaging modality than the imaging system.19. A method for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium, the method comprising: acquiring an image of a region of interest from an imaging system; detecting a position of a deployment device associated with the medical implant within the region of interest, the medical implant having a known position and orientation relative to the deployment device; generating a model of one of the deployment device and the medical implant based on the position of the deployment device and a state associated with the one of the deployment device and medical implant; registering the overlay to the image of the region of interest; combining the overlay with the image of the region of interest to generate the augmented representation of the region of interest; andproviding the augmented representation of the region of interest to a user in real-time.20. The method of example 19, wherein the state associated with the one of the deployment device and the medical implant is selected from a plurality of states comprising at least a first state, in which an annular member associated with the one of the deployment device and the medical implant is deployed, and at least a second state, in which the annular member is not deployed.21 . The method of any of examples 19-20, wherein the state associated with the one of the deployment device and the medical implant is selected from a plurality of states comprising at least a first state, in which an axial extension associated with the one of the deployment device and the medical implant is deployed, and at least a second state, in which the axial extension is not deployed.22. The method of any of examples 19-21 , wherein the state associated with the one of the deployment device and the medical implant is selected from a plurality of states comprising at least a first state, in which a radial extension associated with the one of the deployment device and the medical implant is deployed, and at least a second state, in which the radial extension is not deployed.23. The method of any of examples 19-22, further comprising: determining a current stage of the insertion of the medical implant; and selecting the state associated with the one of the deployment device and the medical implant from a plurality of states according to the current stage.24. The method of example 23, wherein determining the current stage of the insertion of the medical implant comprises receiving an input indicating the current stage from the user via a user interface.25. The method of any of examples 23-24, wherein determining the current stage of the insertion of the medical implant comprises determining the current stage at a machine learning system from at least the position of the deployment device and the image of the region of interest.26. The method of any of examples 19-25, further comprising:allowing the user to select a position associated with an anatomical landmark via a user interface; displaying an indicator at the selected position; and alerting the user when the model of the one of the deployment device and the medical implant is within a threshold distance of the selected position.27. The method of example 26, wherein alerting the user when the model of the one of the deployment device and the medical implant is within a threshold distance of the selected position comprises alerting the user when the model of the one of the deployment device and the medical implant is within a threshold distance of the selected position and the state associated with the one of the deployment device and the medical implant is a first state.28. A perioperative imaging system for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium, the perioperative imaging system comprising: an imaging system that provides an image of a region of interest; a processor; and a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest, the executable instructions comprising: a tracking sensor interface that receives a position of a deployment device within the region of interest; an overlay generator that generates a model of one of the deployment device and the medical implant, having a known position and orientation relative to the deployment device, based on the position of the deployment device and a state associated with the one of the deployment device and the medical implant, the overlay generator determining a current stage of the insertion of the medical implant at a machine learning system from the position of the deployment device and the image of the region of interest and selecting the state associated with the one of thedeployment device and the medical implant from a plurality of states according to the current stage; and an image merger that registers the overlay to the image of the region of interest and combines the overlay with the image of the region of interest to generate the augmented representation of the region of interest.29. The perioperative imaging system of example 28, further comprising a surgical system interface that provides the augmented representation of the region of interest to an associated robotic surgery system.30. The perioperative imaging system of any of examples 28-29, wherein the executable instructions further comprise an alignment component that determines if a current position of the deployment device would result in an acceptable deployment of the second medical device from the image of the region of interest.31 . The perioperative imaging system of any of examples 28-30, wherein the executable instructions further comprise a collision detector that determines if the model of the one of the deployment device and the medical implant is within a threshold distance of an anatomical landmark.32. A method for guiding insertion of a medical implant into one of a heart valve and a cardiac ostium, the method comprising: acquiring an image of a region of interest from an imaging system; detecting a position of a deployment device associated with the medical implant within the region of interest, the medical implant having a known position and orientation relative to the deployment device; generating a model of one of the deployment device and the medical implant based on the position of the deployment device and a state associated with the one of the deployment device and the medical implant; registering the overlay to the image of the region of interest; combining the overlay with the image of the region of interest to generate the augmented representation of the region of interest;providing the augmented representation of the region of interest to a user in real-time; allowing the user to select a position associated with an anatomical landmark at a user interface; and displaying an indicator at the selected position.33. The method of example 32, further comprises determining if a current position of the deployment device would result in an acceptable deployment of the medical implant from the image of the region of interest.34. The method of any of examples 32-33, further comprising: determining a current stage of the insertion of the medical implant; and selecting the state associated with the one of the deployment device and the medical implant from a plurality of states according to the current stage.35. The method of any of examples 32-34, wherein the state associated with the one of the deployment device and the medical implant is selected from a plurality of states comprising at least a first state, in which an extension associated with the one of the deployment device and the medical implant is deployed, and at least a second state, in which the extension is not deployed.36. The method of any of examples 32-35, the method further comprising alerting the user if the model of the one of the deployment device and the medical implant is within a threshold distance of the selected position.

Claims

What is claimed is:1 . A perioperative imaging system for guiding a minimally-invasive procedure, the perioperative imaging system comprising: an imaging system that provides an image of a region of interest; a processor; a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest, the executable instructions comprising: a tracking sensor interface that receives a position of a first medical device within the region of interest; an overlay generator that generates a model of one of the first medical device and a second medical device, having a known position and orientation relative to the first medical device, based on the position of the first medical device and a state associated with the one of the first medical device and the second medical device; and an image merger that registers the overlay to the image of the region of interest and combines the overlay with the image of the region of interest to generate the augmented representation of the region of interest; and a display that provides the augmented representation of the region of interest to a user in real-time.

2. The perioperative imaging system of claim 1 , wherein the executable instructions further comprise a user interface, the state associated with the one of the first medical device and the second medical device being selectable by the user via the user interface.

3. The perioperative imaging system of claim 2, wherein the user interface allows the user to indicate a current stage of the minimally-invasive procedure, thestate associated with the one of the first medical device and the second medical device being responsive to the indicated stage of the minimally-invasive procedure.

4. The perioperative imaging system of claim 2, wherein the user interface allows the user to select a position associated with an anatomical landmark, the overlay generator displaying an indicator at the selected position.

5. The perioperative imaging system of claim 4, wherein the overlay generator displays the indicator at the selected position when the state associated with the one of the first medical device and the second medical device is a first state and does not display the indicator at the selected position when the state associated with the one of the first medical device and the second medical device is a second state.

6. The perioperative imaging system of claim 4, wherein the executable instructions further comprise a collision detector that determines if the model of the one of the first medical device and the second medical device is within a threshold distance of the selected position.

7. The perioperative imaging system of claim 4, wherein the executable instructions further comprise a collision detector that determines if the model of the one of the first medical device and the second medical device is within a threshold distance of a tissue boundary.

8. The perioperative imaging system of claim 7, wherein the collision detector determines if a change in the state of the one of the first medical device and the second medical device will result in contact with a tissue boundary.

9. The perioperative imaging system of claim 8, wherein the collision detector does not allow a change in the state of the one of the first medical device and the second medical device if the change in the one of the first medical device and the second medical device will result in contact with a tissue boundary.

10. The perioperative imaging system of claim 8, wherein the collision detector does not allow a change in the state of the one of the first medical device and the second medical device if the change in the one of the first medical device and the second medical device will result in deployment outside of an acceptable range of positions for the first medical device or the second medical device.11 . The perioperative imaging system of claim 1 , wherein the overlay generator selects the model of one of the first medical device and the second medical device according to an identifier associated with the one of the first medical device and the second medical device, the overlay generator receiving the identifier automatically when the one of the first medical device and the second medical device is connected to the perioperative imaging system.

12. The perioperative imaging system of claim 1 , wherein the executable instructions further comprise an alignment component that determines if a current position of the first medical device would result in an acceptable deployment of the second medical device from the image of the region of interest.

13. The perioperative imaging system of claim 12, wherein the current position of the first medical device is determined to result in the acceptable deployment of the second medical device when the current position of the first medical device is within a first range of positions relative to an anatomical landmark and a current orientation of the first medical device is within a first range of angles relative to the anatomical landmark.

14. The perioperative imaging system of claim 12, wherein the current position of the first medical device is determined to result in the acceptable deployment of the second medical device when a projected cross-sectional area of a ventricular outflow tract is above a threshold value.

15. The perioperative imaging system of claim 1 , wherein the minimally-invasive procedure is insertion of the second medical device into a one of a heart valve and a cardiac ostium and the first medical device comprises a catheter.

Images