Cardiac catheter path planning system

Abstract

Techniques for catheter path planning for a cardiac ablation procedure to treat an arrhythmia of a patient are provided. One technique receives data that includes a three-dimensional (3D) image of the heart of the patient and an electrocardiogram collected from the patient and that includes a catheter specification. The technique applies a path planning system to the 3D image, a location (e.g., source location or target location), and the catheter specification to generate a candidate path from a starting point to a target point that is based on the location. The technique outputs an indication of the candidate path to help inform the ablation procedure to treat the arrhythmia of the patient.

Description

CARDIAC CATHETER PATH PLANNING SYSTEMCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to U.S. Provisional Application No. 63 / 550,020, filed February 5, 2024, entitled “CARDIAC CATHETER PATH PLANNING SYSTEM,” which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] Cardiac catheter ablation is a medical procedure widely used to treat various types of cardiac arrhythmias such as atrial fibrillation and ventricular tachycardia. A cardiac catheter ablation is a minimally invasive procedure that involves navigating a catheter through the vascular system to the heart, where targeted tissue causing abnormal electrical signals is ablated by applying, for example, radiofrequency energy or cryotherapy. The success of this procedure critically depends on precise navigation and accurate placement of the catheter tip at the specific locations within the heart’s intricate anatomy, making path planning an essential aspect of the procedure.

[0003] Path planning for cardiac catheter ablation typically relies on the skill and experience of a physician, supported by real-time imaging techniques such as fluoroscopy, ultrasound, or intracardiac echocardiography. Although such path planning is somewhat effective, it presents challenges such as exposure to radiation (in the case of fluoroscopy) and the need for extensive physician training. Furthermore, the variability in heart anatomy among patients adds complexity to the path planning, increasing the risk of complications like cardiac perforation or damage to adjacent structures. There is, therefore, a significant need for advancements in path planning techniques to enhance safety, accuracy, and efficiency.

[0004] To satisfy this need, a path planning technology should seamlessly integrate with existing medical systems, offer real-time adaptability based on the patient’s cardiac anatomy and physiology, and reduce reliance on physician experience. Such path planning technology could not only improve the safety and efficacy of catheter ablation procedures but also make these life-saving treatments more accessible and consistent across various healthcare settings.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram that illustrates components of a path planning system in some embodiments.

[0006] Figure 2 is a flow diagram that illustrates processing of the path planning system in some embodiments.

[0007] Figure 3 is a flow diagram that illustrates the processing of a path planning algorithm component of the path planning system in some embodiments.

[0008] Figure 4 is a flow diagram that illustrates the processing of an adjust feasibility score component of the path planning system in some embodiments.

[0009] Figure 5 is a flow diagram that illustrates the process of a generate training dataset component based on electronic health records of the path planning system.

[0010] Figure 6 is a flow diagram that illustrates processing of a generate simulated training dataset component of the path planning system in some embodiments.DETAILED DESCRIPTION

[0011] In some embodiments, a cardiac catheter path planning (PP) system employs machine learning (ML) techniques and / or algorithmic techniques to identify candidate cardiac catheter paths (candidate paths) to help inform treatment (e.g., an ablation procedure) of a patient. The PP system identifies candidate paths based on PP input data that includes patient data, planning data, and / or a catheter specification. When employing an ML technique, the PP system generates one or more feature vectors based on the PP input data and applies a PP ML model to the feature vectors to generate a candidate path. The PP ML model may be trained using PP input data extracted from electronic health records (EHRs) of patients who have had successful cardiac procedures. When employing an algorithmic technique, the PP system may employ a PP algorithm that searches a space of possible paths to identify one or more candidate paths factoring in the PP input data. The PP algorithm may employ various techniques such as a breadth-first search or a depth-first search of the space of possible paths. The candidate paths identified via the PP algorithm may be employed to train a PP ML model instead of or in addition to the EHR data or employed to bootstrap the training of the PP ML model. A candidate path may be provided to a catheter navigationdevice that may employ robotic magnetic navigation to help inform navigation of a catheter. A candidate path may also be displayed superimposed on a graphic of a heart. The PP system may also be employed during a procedure to generate a new candidate path given the current location of the catheter, for example, factoring new patient data. The PP system provides real-time, data-driven recommendations for path planning. As a result, the PP system tends to generate improved candidate paths that may lead to reduced procedure times and improved patient outcomes. Although described primarily in the context of an endocardial ablation, the PP system may also be employed in path planning of an epicardial ablation.

[0012] The patient data may include a three-dimensional (3D) model, tissue data, electrical activity data, hemodynamic data, epicardial-related structures, and so on associated with a patient’s heart. The 3D model may be generated from 3D images (e.g., CT or MRI images) of the patient’s heart and may be represented as a 3D mesh or a point cloud. Tissue data may include specifications of tissue type (e.g., scar, borderzone, and healthy), muscle fiber orientation, and other characteristics of tissue such as density, elasticity, and the presence of pathological conditions may affect the maneuverability of a catheter. For example, pliable tissue may affect movement of the catheter because of its deformability, and dense tissue may inhibit smooth catheter movement. Muscle fiber orientation may also affect catheter movement. For example, movement of the catheter in the direction that is parallel to the muscle fiber orientation would encounter less resistance than movement in the direction that is perpendicular to the muscle fiber orientation. Because of the less resistance when moving parallel, it is easier to control movement of the catheter than when moving perpendicular. Also, because of the greater resistance when moving perpendicular, the risk of damage to heart tissues is greater than when moving parallel. Electrical activity data may include action potential propagation data, a cardiogram such as an electrocardiogram (ECG) or vectorcardiogram (VCG), an activation map, and so on. The electrical activity data may be used to identify source locations of arrhythmias. Hemodynamic data may include preload and afterload of muscle fiber, blood pressure, blood velocity, blood flow patterns, and so on. The epicardial-related structures may include phrenic nerve, esophagus, and coronary arteries. During an ablation procedure, the PP system may collect patient data in real time and provide revised paths based on that patient data.

[0013] The planning data may include starting point, target point, ablation pattern, catheter tip angle, avoidance structures, ablation type (endocardial or epicardial), and so on. The target point (also referred to as target location) is the location at which catheter tip is to be positioned at the start of the ablation which may be the source location of an arrhythmia or a location near the source location. The avoidance structures for an endocardial ablation may include the heart wall, the ventricular septum, chordae tendineae, certain heart valves, heart valve leaflets, papillary muscles, and so on. The avoidance structures for an epicardial ablation may include coronary arteries, epicardial vessels, phrenic nerves, coronary artery blockages, and so on.

[0014] The catheter specification may specify catheter length and diameter, steering mechanism (e.g., electromagnetic, pull wires), maneuverability characteristics, and tip characteristics. The maneuverability characteristics may include flexibility, pushability, torque response, trackability, crossability, and so on. The flexibility may specify bend radius (e.g., 2 cm). The torque response may be specified by efficiency ratio (e.g., 95%), lag angle (e.g., 2s), maximum torque transmission (e.g., 10 N mm), and so on. The catheter tip characteristics may include shape, diameter, length, electrode arrangement data, torsional stability, stiffness, and so on.

[0015] The PP system may be employed to identify a path for performing an epicardial ablation. The path may have a starting point within the coronary sinus. When identifying a path, the PP system factors in any blockages within the coronary vessels and adjusts the path to minimize adverse effects of passing through a blockage. The blockage may be determined based on angiography in which a contrast agent is inserted into the blood stream and fluoroscopy is employed to image the blood flow.

[0016] The blockage state of a blockage of a coronary vessel indicates the blockage location of the blockage and its blockage amount which may be expressed as a blockage percentage. The blockage location may indicate the start and end of a blockage. The blockage percentage indicates the amount of narrowing with 0 percent representing no blockage and 100 percent representing complete blockage. The blockage percentage may be derived from a flow percentage that indicates the percentage of the maximum blood flow (assuming no blockage) that is flowing with 0 percent indicating no blood flow and 100 percent indicating the maximum blood flow. The PP system may employ a configurable avoidance blockage percentage to indicatethat a path should avoid blockages with blockage percentages at or above the avoidance blockage percentage.

[0017] Figure 1 is a block diagram that illustrates components of a path planning system in some embodiments. The PP system includes segmentation system 101 , patient classification system 102, arrhythmia classification system 103, mapping system 104, ablation pattern system 105, and candidate path system 106. The PP system inputs patient data, planning data, and a catheter specification and generates candidate paths.

[0018] The segmentation system inputs a patient 3D image (of the patient data) and generates a segmented 3D mesh or a segmented point cloud. The 3D image may be preprocessed to remove noise, adjust contrast, and remove artifacts. The segmentation system may be a segmentation ML model that is based on a convolutional neural network (CNN). The segmentation ML model may include a 3D mesh ML submodel and a segmentation ML sub-model. The 3D mesh ML sub-model inputs the 3D image and outputs a 3D mesh and is trained with 3D images labeled with 3D meshes. The training dataset may be generated by generating 3D meshes representing different cardiac geometries. For each 3D mesh, 2D slices of a 3D image are generated based on what cardiac structures (e.g., endocardium of left ventricle or papillary muscle) would be visible on 2D slice through the 3D mesh. The segmentation ML sub-model inputs a 3D mesh and outputs a segmented 3D mesh. The segmentation ML sub-model is trained with training dataset that includes 3D meshes with cardiac structures demarcated such as left atrium, right ventricle, septa, tricuspid valve, pulmonary veins, septum, coronary arteries, phrenic nerves, and so on. The 3D mesh ML sub-model and the segmentation ML sub-model may be CNNs. Alternatively, rather than input a 3D mesh, a segmentation ML sub-model may input a 3D image and generate a segmented 3D image. A 3D mesh sub-model may input the segmented 3D image and output a segmented 3D mesh. Techniques for generating a 3D mesh are described in PCT Pub. No. WO2023 / 168017 entitled “Overall Ablation Workflow System” and published on September 7, 2023, which is hereby incorporated by reference.

[0019] The patient classification system inputs a segmented 3D mesh and patient data and generates a patient classification. Patients with similar segmented 3D meshes and patient data tend to have the same patient classification. The patient classificationsystem may be implemented as a patient classification ML model (supervised or unsupervised). The training dataset for a supervised ML model may contain examples (also referred to instances or rows) with a feature vector derived from a segmented 3D mesh and patient data that is labeled (e.g., manually) with a patient classification. The training dataset for a unsupervised ML model may contain examples with a feature vector derived from a segmented 3D mesh and patient data but without a label. A patient classification ML model may employ a variety of ML techniques such as Naive Bayes, k-Nearest Neighbor (kNN), neural network (e.g., CNN), gradient boosting machines, k-means clustering, and so on. The patient classification ML model may employ a combination of ML models such as a CNN that inputs a segmented 3D mesh, a first neural network that inputs patient data, and a second neural network that inputs the outputs of the CNN and the first neural network and outputs a classification. The parameters (e.g., weights and biases) of this combination of ML model may be learning in parallel (e.g., a multimodal ML architecture). The patient classification ML model may alternatively employ a k-means clustering technique or kNN model. Alternatively, rather than employing a patient classification ML model, the patient data (rather than the patient classification) may be directly input to the ablation pattern system and / or the candidate path system.

[0020] The arrhythmia classification system may be an arrhythmia classification ML model that inputs an ECG (or other cardiogram) of a patient and generates a classification of the arrhythmia (e.g., AF or PVC) that the ECG (or VCG) represents. The arrhythmia classification ML model inputs a feature vector that may be an image of the ECG, a voltage-time series representation of the ECG, a collection of features derived from the ECG such as a QRS integral, an RR interval, a wavelet transform, mean and variance, and so on. The arrhythmia classification ML model may employ ML techniques such as a support vector machine, a neural network (e.g., CNN), kNN, long short-term memory networks (LSTM), k-means clustering, and so on. The training dataset for the arrhythmia classification ML model may be derived from EHRs that specify an ECG and an arrhythmia type. Each example of the training dataset specifies a feature vector derived from an ECG labeled with an arrhythmia type. Alternatively, the training dataset may be generated based on simulating electrical activity of hearts having different cardiac characteristics including a specification of a source (e.g., reentrant circuit) of a certain arrhythmia type. If the simulation stabilizes, then asimulated ECG is generated from the simulated electrical activity. The feature vector of an example is derived from that simulated ECG and the label of that example is the arrhythmia type. Techniques for simulating electrical activity of a heart are described in U.S. Pat. Pub. No. 2021 / 0065906 titled “Calibration of Simulated Cardiograms” and published on March 4, 2021 , which is hereby incorporated by reference. Techniques for determining stability are described Krummen, D., et al., Rotor Stability Separates Sustained Ventricular Fibrillation from Self-Terminating Episodes in Humans, Journal of the American College of Cardiology, Vol. 63, No. 24, 2014, which is hereby incorporated by reference.

[0021] The mapping system inputs an arrhythmia classification and an ECG (and optionally other patient data) and outputs a source location of an arrhythmia represented by the ECG. Alternatively, a separate mapping system may be employed for each arrhythmia classification. With this alternative, the mapping system may only input a feature vector derived from the ECG (and not an arrhythmia classification) and output a source location. In such an alternative, the arrhythmia classification system would be employed to identify which mapping system to apply to the ECG. The mapping system may employ a mapping ML model or a mapping algorithm. Various mapping systems are described in the ‘906 publication. The mapping systems described in the ‘906 publication include a mapping ML model and a mapping algorithm (based on a library that maps ECGs or VCGs to source locations). The mapping systems are described primarily in the context of simulations of electrical activity of a heart. Each simulation assumes a different set of cardiac characteristics that include anatomical, electrophysiological, source location, and so on. The mapping ML model is trained with a training dataset that includes simulated ECGs labeled with simulated source locations. The mapping algorithm accesses a mapping that associates simulated ECGs with simulated source locations. To identify the source location, the mapping algorithm inputs an ECG, finds a similar simulated ECG in the mapping, and outputs the associated simulated source location.

[0022] The ablation pattern system may be implemented as an ablation pattern ML model that inputs a patient classification and a source location and outputs an ablation pattern for the ablation procedure. Techniques for implementing an ablation pattern system are described in the ‘906 publication. Such techniques may be adapted to input both a source location and a patient classification. The ablation patterns mayinclude a linear ablation pattern, a circumferential ablation pattern, a point-by-point ablation pattern, a focal ablation pattern, a complex fractionated atrial electrogram ablation pattern, and so on.

[0023] The candidate path system may be implemented as a candidate path ML model that inputs a patient classification, an ablation pattern, a source location, planning data, a 3D mesh, avoidance structures (e.g., identified based on a segmentation of the segmentation system), a catheter specification, and so on and outputs one or more candidate paths. The candidate path system may also employ a separate candidate path sub-system for each catheter specification (e.g., catheter type.) The candidate path ML model may be a neural network, a transformer, and so on.

[0024] In some embodiments, the path planning ML model may employ an attention ML model. The attention ML model employs a neural network that inputs a segmented 3D mesh, patient data, and a catheter specification. The attention ML model includes an attention mechanism that allows the neural network to focus on patient data that is particularly relevant to path planning. For example, the attention mechanism assigns different levels of importance to different types of patient data such as cardiac geometry represented by the 3D mesh and target location. The attention ML model may be trained on clinical data and / or simulated data. After the attention ML model is trained, to generate a candidate path for a patient, the path planning ML system applies the attention ML model to a 3D mesh generated from a 3D image of the patient, patient data of the patient, and a catheter specification to generate a candidate path.

[0025] The ML models may be continually retrained based on data collected and physician feedback based on ablation procedures supported by the PP system. This continual retraining helps to account for unique anatomical variations and pathological conditions of patients.

[0026] The PP system may employ variations of the architecture illustrated in Figure 1 . For example, the PP system may not employ an ablation pattern system or an arrhythmia classification system. Also, rather than generating a patient classification, the PP system may input a 3D mesh, patient data, and / or ECG directly into the candidate path system. The PP system may be trained using simulated data as described below.

[0027] The PP system may also employ dynamic PP techniques in which a candidate path is dynamically generated during an ablation procedure and / or is dynamically updated based on updated information. For example, the candidate path ML model may dynamically identify a next path location given the actual catheter location. Also, if an actual path deviates from the candidate path originally identified, the candidate path ML model may be employed to generate an updated candidate path assuming the current catheter location as the starting point. As another example, based on electrical activity data identified during the ablation procedure, a new target location may be identified. In such a case, the candidate patent ML model may be used to generate an updated candidate path from the actual path to the new target location. The actual path may be represented by the current locations of points along the catheter. As another example, if the anatomical characteristics are determined to be different from those employed to generate the original candidate path, the PP system may be employed to generate an updated patient classification and candidate path. As another example, the candidate path ML model may be employed to generate a candidate path from an ablation location to a next target location.

[0028] To provide multiple candidate paths for a procedure, the PP system may train multiple instances of the PP ML model using different sets of training data. Each PP ML model may be employed to determine different candidate paths to allow a medical provider to evaluate different candidate paths and select one for a procedure.

[0029] In some embodiments, the PP system provides a user interface through which a person can manually indicate a proposed path. For example, the person can draw the proposed path on a 3D graphic of a heart. The PP system can then be used to validate that the path is feasible. To validate, the PP system determines if a feasibility criterion is satisfied for various portions of the path. If the feasibility criterion is not satisfied, the PP system may highlight the non-feasible portions of the path and allow the person to adjust the path. Once adjusted, the PP system may again determine if the feasibility criterion is satisfied. Alternatively, the PP system may generate a proposed adjustment to the proposed path. For example, the PP system may try to identify a proposed adjustment from the beginning to the end of the non-feasible portion. If none can be identified, the PP system may try to identify a proposed adjustment that starts before the beginning and / or after the end of the non-feasible portion. The PP system may employ a feasibility ML model to determine whether a portion is feasible.The feasibility ML model may be trained with a training dataset with feature vectors that include that include a cardiac geometry and a path (or a portion of a path) of successful ablations. Alternatively, the PP system may determine feasibility of a path based on a catheter specification and cardiac geometry. For each path, the PP system determines whether the catheter would support such a path. For example, if the bend radius is exceeded or the catheter length is shorter than the path length, the catheter would not support such a path.

[0030] Figure 2 is a flow diagram that illustrates processing of the path planning system in some embodiments. The PP system 200 inputs patient data, planning data, and a catheter specification and generates and displays one or more candidate paths for a patient’s ablation procedure. In block 201 , the PP system applies a segmentation ML model to a 3D image of the patient data to generate a segmented 3D mesh. In block 202, the PP system generates and displays a 3D graphic based on the 3D mesh. In block 203, the PP system applies the patient classification ML model to the 3D mesh and other patient data to generate a patient classification. In block 204, the PP system applies the arrhythmia classification ML model to a cardiogram of the patient data to generate an arrhythmia classification. In block 205, the PP system applies a mapping system to the arrhythmia classification and the cardiogram to generate a source location of the arrhythmia. In block 206, the PP system applies an ablation pattern ML model to the patient classification and the source location to generate an ablation pattern. In block 207, the PP system applies the candidate path ML model to the ablation pattern, the source location, and the patient classification to generate a candidate path. In block 208, the PP system displays the candidate path on the 3D graphic of the patient’s heart. The PP system may allow a user to manually alter the candidate path to generate an updated candidate path. The PP system may also output the candidate path to a catheter navigation system.

[0031] The PP system may be employed as part of a method for treating a patient. The method applies the PP system to identify and display a candidate path. As part of the method, the patient is treated by performing an ablation. The ablation may be performed by an electrophysiologist manually guiding the catheter along the path. As the catheter is moved, the PP system may display updates to the candidate path. When the catheter tip is at the target location with, for example, the desired electrode placement, the electrophysiologies activates the delivery of energy to the cardiac tissue.The candidate path may be provided to an ablation guidance system (e.g., a robotic system) that controls movement of the catheter. In such a case, the ablation guidance system may activate the delivery of energy preferably after confirmation by the electrophysiologist.

[0032] In some embodiments, the PP algorithm employs a breadth-first search of paths to identify a candidate path from the starting point to the target location. The paths to be searched may be represented as a tree structure of vertices (representing points along a path) and edges from parent vertices to child vertices. The starting point is represented as a parent vertex (aka starting vertex) with edges leading to child vertices and with each child vertex being a parent vertex with edges leading to its child vertices. The direction of the edges from a parent vertex to its child vertices are at different angles. For example, if there are 11 child nodes of a parent vertex the angles may range from -10 degrees to +10 degrees in angular increments of 2 degrees with 0 degrees being based on an average direction of some number of ancestor vertices. The angular increments may be based on the flexibility of the catheter with a more flexible catheter having a larger angular increment or more child vertices. The child vertices are located at a path incremental distance (e.g., 1 mm) from their parent vertex.

[0033] To perform the breath-first search, the PP algorithm starting with the starting vertex selects each of its child vertices in sequence. For each selected child node, the PP algorithm performs a feasibility analysis that impacts a feasibility score that may range from 0.0 to 1 .0. If the path from the starting vertex to a selected child vertex satisfies a feasibility criterion, then the path is extended by that child vertex, which becomes a parent vertex. However, if it does not satisfy a feasibility criterion, then the feasibility score of the path is set to 0.0 and path is terminated and not further processed. Paths that lead to the target location (within proximity criterion) are considered candidate paths. The feasibility score for each path may be initially set to 1 .0 and may be reduced based on path characteristics. If a feasibility score of a path drops below a discard threshold (e.g., 0.1 ), then that path is discarded.

[0034] The feasibility analysis may be based on a catheter specification, tissue data, hemodynamic data, a cardiac geometry, avoidance structures, and so on. If the path were extended to a child vertex that would be inconsistent with the flexibility of the catheter, the path is not feasible and thus discarded. For example, the path isinconsistent if it has a portion with a curvature that is not feasible based on the catheter flexibility. The curvature may be represented as the radius of a circle that would include the curve on its perimeter. If the path is extended to a child vertex that is located within or near an avoidance structure, it is discarded. For example, an avoidance structure is the heart wall and ventricular septum because a path cannot pass through those structures. If a path is longer than the catheter, it is discarded. If a vertex is near a heart wall but not within but the catheter tip characteristics indicate that the tip would be within the heart wall, the path is discarded.

[0035] The shape of a path is dynamic in the sense that vertices of the path may move, for example, based on the flexibility of the catheter. For example, a path that has a curvature may result in some vertices moving to new locations because of the flexibility. So, the feasibility of a path to a child vertex may depend on the feasibility of the entire path. For example, if a vertex moves so that it is in contact with the heart wall may have its feasibility score reduced based on the size of the angle between the path direction at the vertex and the orientation of the muscle fiber. The hemodynamic data may also affect the path curvature based on the speed and volume of blood that flows through the path.

[0036] If a path intersects another path in the same general direction as the intersected path, one of the paths may be discarded such as the path with the lower feasibility score.

[0037] After a path terminates because it is near the target location, the feasibility score of the paths may be reduced based on its path length when shorter paths are considered more desirable that longer paths. The PP system may display a graphic of a heart with the path superimposed on the graphic and may indicate the feasibility score.

[0038] Figure 3 is a flow diagram that illustrates the processing of a path planning algorithm component of the path planning system in some embodiments. The PP algorithm component 300 inputs patient data, planning data, and a catheter specification and generates one or more candidate paths from the starting point to the target point. In block 301 , the component appends (enqueues) the starting vertex to a path queue. The path queue maintains a list of partial paths that have been identified and have not been discarded as being infeasible paths. In decision block 302, if the path queue is empty, then the component completes returning an indication of acandidate path list, else the component continues at block 303. The path queue contains a list of candidate paths from the starting point to an end point of the candidate path. The path queue may be empty if no feasible candidate paths have been identified. In block 303, the component removes (dequeues) a path from the path queue. In block 304, the component selects a next vertex to be added to the retrieved path. For example, the component may perform the processing for 1 1 possible next vertices for the retrieved path. In decision block 305, if all the child vertices have already been selected, then the component loops to block 302 to determine if the path queue is empty, else the component continues at block 306. In block 306, the component invokes an adjust feasibility score component to adjust the feasibility score of the extended path. In decision block 307, if the path extended by the child vertex is feasible, then the component continues at block 308, else the component loops to block 304 to select the next child vertex that may extend the path. In decision block 308, if the next child vertex is at the target point (e.g., within a specified distance), then the component continues at block 310, else the component continues at block 309. In block 309, the component adds the path extended by the child vertex to the path queue and loops to block 304 to select the next vertex. In block 310, the component adds the path extended by the next vertex to the candidate path list and loops to block 301 to select the next point. The component may also adjust the feasibility score based on the difference between the angle of the catheter tip and the heart wall at the target location and a desired angle as indicated by the planning data and based in path length. In some embodiments, rather than employing a path queue, the PP system may implement a recursive algorithm that invokes an extend component passing the starting point as the last point on the path and a feasibility score of 1 .0 and returns a path and a feasibility score. The extend component adjusts the feasibility score. If the feasibility score is not below a threshold score and the catheter tip is not at the target point, for each child vertex, the extend component recursively invokes the extend component passing the path with the child vertex and the updated feasibility score. If the feasibility score is below a threshold feasibility score, the extend component returns indicating that the path is not feasible.

[0039] Figure 4 is a flow diagram that illustrates the processing of an adjust feasibility score component of the path planning system in some embodiments. The adjust feasibility score component 400 is invoked to adjust the feasibility score of a pathto be extended by a child vertex. The feasibility score may be set to 0.0 if the extended path is not feasible. In block 401 , the component adjusts the feasibility score based on the catheter specification. For example, the feasibility score may be reduced by an amount of the bend with the child vertex added. If there is no bend, the feasibility score is not reduced. The reduction in feasibility score may increase exponentially as the bend approaches the maximum bend of the catheter. In block 402, the component adjusts the feasibility score based on the avoidance structures. For example, if the catheter tip is not within an avoidance distance from an avoidance structure, the feasibility score is not reduced. (The avoidance distance may be specific to the type of avoidance structure). If the catheter tip is within the avoidance distance, the reduction in feasibility score may increase exponentially as the distance to the avoidance structure decreases. In block 403, the component adjusts the feasibility score based on closeness to another path. For example, if a path is very close to another path, the paths may be considered redundant and the feasibility score of the path may be set to zero so that it is not added back to the path queue. In block 404, the component adjusts the feasibility score based on the planning data. For example, the feasibility score may be reduced if the distance to the target point and the length of the path is very close to the length of the catheter. In block 405, the component adjusts the feasibility score based on tissue data. For example, if tissue data indicates that a cardiac wall is very thin, the feasibility score may be reduced based on distance to the cardiac wall (e.g., like an avoidance structure) and thinness of the cardiac wall. In block 406, the component adjusts the feasibility score based on hemodynamic data. For example, if the child vertex is within an area of increased blood flow, the feasibility score is reduced.

[0040] Figure 5 is a flow diagram that illustrates the process of a generate training dataset component based on electronic health records of the path planning system. In some embodiments, the generate training dataset component 500 retrieves EHRs of patients who have had successful ablation procedures and generates training dataset based on each EHR. In block 501 , the component selects the next EHR. In decision block 502, if all the EHRs have already been selected, then the component completes, else the component continues at block 503. In block 503, the component retrieves a 3D image from the EHR associated with the ablation procedure. In block 504, the component derives features from the 3D image (e.g., the 3D image itself). In block 505, the component retrieves other patient data (e.g., ablation location) as featuresassociated with the ablation procedure. In block 506, the component retrieves an ECG associated with the ablation procedure. In block 507, the component derives features from the ECG. In block 508, the component retrieves a catheter specification, tissue data, and hemodynamic data associated with the ablation procedure and generates features. In block 509, the component retrieves the catheter path of the ablation procedure. The catheter path may be derived from 3D images collected during the ablation procedure. In block 510, the component labels a feature vector of the features with the path. In block 511 , the component stores the feature vector with the label as an example of the training dataset and loops to block 501 to select the next EHR.

[0041] Figure 6 is a flow diagram that illustrates processing of a generate simulated training dataset component of the path planning system in some embodiments. The generate simulated training dataset component 600 generates a training dataset for the PP system that includes feature vectors based on simulated patient data including simulated anatomies, simulated target points, catheter characteristics, and so on. Each feature vector is labeled with a candidate path. In block 601 , the component selects a next simulated catheter specification. In decision block 602, if all the simulated catheter specification have already been selected, then the component completes, else the component continues at block 603. In block 603, the component selects a next simulated EHR. A simulated EHR many have a 3D image (e.g., generated from a simulated cardiac geometry), a simulated ECG (e.g., derived from a simulation of electrical activity of a heart, simulated tissue state, and so on. Alternatively (or additionally), the EHRs may be derived from patient EHRs of patients). For example, a cardiac geometry may be derived from a 3D image of a patient EHR. In decision block 604, if all the EHRs have already been selected, then the component loops to block 601 to select the next simulated catheter specification, else the component continues at block 605. The ellipsis represent that a feature vector is generated based on simulated EHRs and / or actual EHRs such as avoidance structures, cardiac geometry, planning data, tissue data, hemodynamic data, and so on. In block 605, the component invokes the path planning algorithm to identify candidate paths. In block 606, the component generates an stores examples of the training dataset. Each example includes the feature vector labeled with a candidate path. The component the loops to block 603 to selected the next EHR.

[0042] The computing systems (e.g., network nodes or collections of network nodes) on which the PP system and the other described systems may be implemented may include a central processing unit, input devices, output devices (e.g., display devices and speakers), storage devices (e.g., memory and disk drives), network interfaces, graphics processing units, communications links (e.g., Ethernet, Wi-Fi, cellular, and Bluetooth), global positioning system devices, and so on. The input devices may include keyboards, pointing devices, touch screens, gesture recognition devices (e.g., for air gestures), head and eye tracking devices, microphones for voice recognition, and so on. The computing systems may include high-performance computing systems, distributed systems, cloud-based computing systems, client computing systems that interact with cloud-based computing system, desktop computers, laptops, tablets, e-readers, personal digital assistants, smartphones, gaming devices, servers, and so on. The computing systems may access computer- readable media that include computer-readable storage mediums and data transmission mediums. The computer-readable storage mediums are tangible storage means that do not include a transitory, propagating signal. Examples of computer- readable storage mediums include memory such as primary memory, cache memory, and secondary memory (e.g., DVD), and other storage. The computer-readable storage media may have recorded on them or may be encoded with computer-executable instructions or logic that implements the PP system and the other described systems. The data transmission media are used for transmitting data via transitory, propagating signals or carrier waves (e.g., electromagnetism) via a wired or wireless connection. The computing systems may include a secure crypto processor as part of a central processing unit (e.g., Intel Secure Guard Extension (SGX)) for generating and securely storing keys and for encrypting and decrypting data using the keys and for securely executing all or some of the computer-executable instructions of the PP system. Some of the data sent by and received by the PP system may be encrypted, for example, to preserve patient privacy (e.g., to comply with government regulations such the European General Data Protection Regulation (GDPR) or the Health Insurance Portability and Accountability Act (HIPAA) of the United States). The PP system may employ asymmetric encryption (e.g., using private and public keys of the Rivest-Shamir- Adleman (RSA) standard) or symmetric encryption (e.g., using a symmetric key of the Advanced Encryption Standard (AES)).

[0043] The one or more computing systems may include client-side computing systems and cloud-based computing systems (e.g., public or private) that each executes computer-executable instructions of the PP system. A client-side computing system may send data to and receive data from one or more servers of the cloud-based computing systems of one or more cloud data centers. For example, a client-side computing system may send a request to a cloud-based computing system to perform tasks such as run a simulation (e.g., patient specific) of electrical activity of a heart or train an ML model. A cloud-based computing system may respond to the request by sending to the client-side computing system data derived from performing the task such as an ECG derived from a simulation or a simulated 3D image. The servers may perform computationally expensive tasks in advance of processing by a client-side computing system such as training an ML model or in response to data received from a client-side computing system. A client-side computing system may provide a user experience (e.g., user interface) to a user of the PP system. The user experience may originate from a client computing device or a server computing device. For example, a client computing device may generate a patient-specific graphic of a heart and display the graphic. Alternatively, a cloud-based computing system may generate the graphic (e.g., in a Hyper-Text Markup Language (HTML) format or an extensible Markup Language (XML) format) and provide it to the client-side computing system for display. A client-side computing system may also send data to and receive data from various medical devices such as an ECG monitor, an ablation therapy device, an ablation planning device, and so on. The data received from the medical devices may include an ECG, actual ablation characteristics (e.g., ablation location and ablation pattern), and so on. The data sent to a medical device may include data, for example, data in a Digital Imaging and Communications in Medicine (DICOM) format. A client-side computing device may also send data to and receive data from medical computing systems that store patient medical history data, descriptions of medical devices (e.g., type, manufacturer, and model number) of a medical facility, that store, medical facility device descriptions, that store results of procedures, and so on. The term cloud-based computing system may encompass computing systems of a public cloud data center provided by a cloud provider (e.g., Azure provided by Microsoft Corporation) or computing systems of a private server farm (e.g., operated by the provider of the PP system).

[0044] The PP system and the other described systems may be described in the general context of computer-executable instructions, such as program modules and components, executed by one or more computers, processors, or other devices. Generally, program modules or components include routines, programs, objects, data structures, and so on that perform tasks or implement data types of the PP system and the other described systems. Typically, the functionality of the program modules may be combined or distributed as desired in various examples. Aspects of the PP system and the other described systems may be implemented in hardware using, for example, an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0045] An ML model may be any of a variety or combination of supervised, semisupervised, self-supervised, unsupervised, or reinforcement learning ML models including a neural network such as fully connected, convolutional, recurrent, or autoencoder neural network, or restricted Boltzmann machine, a support vector machine, a Bayesian classifier, k-means clustering, decision tree, generative adversarial networks, transformer, and so on. When the ML model is a deep neural network, the model is trained using a training dataset that includes examples. Each example of a training dataset for a path planning ML model includes a feature vector with features from data (e.g., EHRs and catheter specifications) and a label that is a candidate path. The training results in a set of weights for the activation functions of the layers of the deep neural network. The trained deep neural network can then be applied to new data to generate a label for that new data. When the ML model is a support vector machine, a hyper-surface is found to divide the space of possible inputs. For example, for a patient classification ML model of the patient classification system (e.g., for classifying ECGs as AF or not), the hyper-surface attempts to split the examples of different types of arrhythmias by maximizing the distance between the nearest of the positive examples (e.g., AF ECGs) and negative examples (e.g., VT ECGs) to the hyper-surface. The trained support vector machine can then be applied to an ECG to generate a classification (e.g., AF or not) for the ECG. An ML model may generate values of discrete domain (e.g., classification), probabilities, and / or values of a continuous domain (e.g., regression value, classification probability).

[0046] Various techniques can be used to train a support vector machine such as adaptive boosting, which is an iterative process that runs multiple tests on a collectionof training data. Adaptive boosting transforms a weak learning algorithm (an algorithm that performs at a level only slightly better than chance) into a strong learning algorithm (an algorithm that displays a low error rate). The weak learning algorithm is run on different subsets of the training data. The algorithm concentrates increasingly on those examples in which its predecessors tended to show mistakes. The algorithm corrects the errors made by earlier weak learners. The algorithm is adaptive because it adjusts to the error rates of its predecessors. Adaptive boosting combines rough and moderately inaccurate rules of thumb to create a high-performance algorithm. Adaptive boosting combines the results of each separately run test into a single, very accurate classifier. Adaptive boosting may use weak classifiers that are single-split trees with only two leaf nodes.

[0047] A neural network model has three major components: architecture, loss function, and search algorithm. The architecture defines the functional form relating the inputs to the outputs (in terms of network topology, unit connectivity, and activation functions). The search in weight space for a set of weights that minimizes the loss function is the training process. A neural network model may use a radial basis function (RBF) network and a standard or stochastic gradient descent as the search technique with backpropagation.

[0048] A convolutional neural network (CNN) has multiple layers such as a convolutional layer, a rectified linear unit (ReLU) layer, a pooling layer, a fully connected (FC) layer, and so on. Some more complex CNNs may have multiple convolutional layers, pooling layers, and FC layers. Each layer includes a neuron for each output of the layer. A neuron inputs outputs of prior layers (or original input) and applies an activation function to the inputs to generate an output.

[0049] A convolutional layer may include multiple filters (also referred to as kernels or activation functions). A filter inputs a convolutional window, for example, of an image, applies weights to each pixel of the convolutional window, and outputs value for that convolutional window. For example, if a 2D slice of an image is 4096 by 4096 pixels, the convolutional window may be 32 by 32 pixels. The filter may apply a different weight to each of the 1024 pixels in a convolutional window to generate the value.

[0050] An activation function has a weight for each input and generates an output by combining the inputs based on the weights. The activation function may be a rectifiedlinear unit (ReLU) that sums the values of each input times its weight to generate a weighted value and outputs max(0, weighted value) to ensure that the output is not negative. The weights of the activation functions are learned when training an ML model. The ReLU function of max(0, weighted value) may be represented as a separate ReLU layer with a neuron for each output of the prior layer that inputs that output and applies the ReLU function to generate a corresponding “rectified output.”

[0051] A pooling layer may be used to reduce the size of the outputs of the prior layer by downsampling the outputs. For example, each neuron of a pooling layer may input 16 outputs of the prior layer and generate one output resulting in a 16-to-1 reduction in outputs.

[0052] An FC layer includes neurons that each input all the outputs of the prior layer and generate a weighted combination of those inputs. For example, if the penultimate layer generates 256 outputs and the FC layer inputs a neuron for each of three classifications (e.g., AF, VF, AFL), each neuron inputs the 256 outputs and applies weights to generate value for its classification.

[0053] Multimodal ML combines different modalities of input data to make a prediction. The modalities may be, for example, 3D images and ECGs (e.g., ECG voltage-time series or ECG images.)

[0054] In one multimodal ML approach, referred to as “early fusion,” data of the different modalities is combined at the input stage and is then trained on the multimodal data. The training dataset for these modalities may include examples that each has feature vector with features derived from a 3D image and an ECG and a label with a patient classification. An ML model trained with such a training dataset may correspond to a combination of the segmentation system and the patient classification system. The 3D image and the ECG may be used in its original form or preprocessed. For example, the dimensionality of the feature vector may be reduced by compressing the data into byte arrays or applying a principal component analysis. As another example, the resolution of a 3D image may be reduced. A byte array may be processed by a crossattention mechanism to condense the bytes into a vector of a fixed size. The vectors are then used to train an ML model primarily using supervised approaches.

[0055] In a second multimodal ML approach, features from different modalities may be kept separate at the input stage and used as inputs to different, modality-specific ML models (e.g., a CNN for a 3D image and a recurrent neural network (RNN) for an ECG). The modality-specific ML models may be trained jointly such that information from across different modalities is combined to make predictions, and the combined (cross-modality) loss is used to adjust model weights. Alternatively, the modality-specific ML models may also be trained separately using a separate loss function for each modality. A combined ML model is then trained based on the outputs of the modality specific models and labels.

[0056] Transformer machine learning was introduced as an alternative to a recurrent neural network that is both more effective and more parallelizable. (See, Vaswani, Ashish, et aL, “Attention is all you need,” Advances in Neural Information Processing Systems 30 (2017), which is hereby incorporated by reference.) Transformer machine learning was originally described in the context of natural language processing (NLP) but has been adapted to other applications such as image processing to augment or replace a CNN. In the following, transformer machine learning is described in the context of NLP as introduced by Vaswani.

[0057] A transformer includes an encoder whose output is input to a decoder. The encoder includes an input embedding layer followed by one or more encoder attention layers. The input embedding layer generates an embedding of the inputs. For example, if a transformer ML model is used to process a sentence as described by Vaswani, each word may be represented as a token that includes an embedding of a word and its positional information. Such an embedding is a vector representation of a word such that words with similar meanings are closer in the vector space. The positional information is based on the position of the word in the sentence.

[0058] The first encoder attention layer inputs the embeddings and the other encoder attention layers input the output from the prior encoder attention layer. An encoder attention layer includes a multi-head attention mechanism followed by a normalization sublayer whose output is input to a feedforward neural network followed by a normalization sublayer. A multi-head attention mechanism includes multiple selfattention mechanisms that each inputs the encodings of the previous layer and weighs the relevance encodings to other encodings. For example, the relevance may be determined by the following attention function:where Q represents a query, K represents a key, V represents a value, and dk represents the dimensionality of K. This attention function is referred to as scaled dotproduct attention. In Vaswani, the query, key, and value of an encoder multi-head attention mechanism is set to the input of the encoder attention layer. The multi-head attention mechanism determines the multi-head attention as represented by the following:MultiHeadfQ, K, V) = concat heacf , . . . , head8) W°where W represents weights that are learned during training. The weights for the feedforward networks are also learned during training. The weights may be initialized to random values. A normalization layer normalizes its input to a vector having a dimension as expected by the next layer or sub-layer.

[0059] The decoder includes an output embedding layer, decoder attention layers, a linear layer, and a softmax layer. The output embedding layer inputs the output of the decoder shifted right. Each decoder attention layer inputs the output of the prior decoder attention layer (or the output embedding layer) and the output of the encoder. The embedding layer is input to the decoder attention layer, the output of the decoder attention layer is input the linear layer, and the output of the linear layer is input to the softmax layer which outputs probabilities. A decoder attention layer includes a decoder masked multi-head attention mechanism followed by a normalization sublayer, a decoder multi-head attention mechanism followed by a normalization sublayer, and a feedforward neural network followed by a normalization sublayer. The decoder masked multi-head attention mechanism masks the input so that predictions for a position are only based on outputs for prior positions. A decoder multi-head attention mechanism inputs the normalized output of the decoder masked multi-head attention mechanism as a query and the output of the encoder as a key and a value. The feedforward neural network inputs the normalized output of the decoder multi-head attention mechanism. The normalized output of the feedforward neural network is the output of that multi-head attention layer. The weights of the linear layer are also learned during training.

[0060] After being trained, a sentence may be input to encoder to generate an encoding of the sentence that is input to the decoder. Initially, the output of the decoder that is input to the decoder is set to null. The decoder then generates an output based on the encoding and the null input. The output of the decoder is appended to the decoder’s current input, and the decoder generates a new output. This decoding process is repeated until the encoder generates a termination symbol. If the transformer is trained with English sentences labeled with French sentences, then a termination symbol is added to the end of the French sentences. When translating a sentence, the transformer terminates its translation when the termination symbol is generated indicating the end of the French sentence that is completion of the translation.

[0061] Although initially developed to process sentences, transformers have been adapted for image recognition. The input a decoder of a transformer may be a representation of fixed-size patches of the image. (See, Dosovitskiy, et aL, “An Image is worth 16X16 Words: Transformers for image Recognition at Scale,” arXiv:2020- 1 1929, Jun. 3, 2021 , which is hereby incorporated by reference.) The representation of a patch may be, for each pixel of the patch, an encoding of its row, column, and color. The output of the encoder is fed into neural network to generate a classification of the image.

[0062] An unsupervised ML technique trains an ML model using unlabeled feature vectors. K-means clustering is an example ML technique. Given the unlabeled feature vectors, k-means clustering clusters the unlabeled feature vectors into clusters based on similarity of their features. With k-means clustering, the number of clusters may be predefined. For example, the classification system may employ 5 clusters (k=5) to represent clusters of ECGs. An example training technique initially randomly places a feature vector in each cluster. The training then repeatedly calculates a mean feature vector of each cluster, selects a feature vector not in a cluster, identifies the cluster whose mean is most similar, adds the feature vector to that cluster, and moves the feature vectors already in the clusters to the cluster with the most similar mean. Similarity may be determined, for example, based on a Pearson similarity, a cosine similarity, and so on. The training ends when all the feature vectors have been added to a cluster. The clusters may be manually labeled to represent the classification (e.g., arrhythmia type) that the ECGs in the clusters represent. To classify a 3D mesh and patient data, a patient feature vector is generated based on the 3D mesh and the patientdata. The cluster with a mean feature vector that is most similar to the patient feature vector is identified, and the patient data is assigned the classification of that cluster.

[0063] An ML decision tree defines information (e.g., arrhythmia classification) that is associated with entities (e.g., patients) that have certain characteristics. An ML decision may have the same form as a manually generated decision tree. However, the feature associated with each decision node of an ML decision tree may be selected automatically based on analysis of the training data. Each decision node (i.e., non-leaf node) of a decision tree corresponds to a feature and each branch from node may correspond to a value or range of values for the feature of that decision node. For example, a decision node corresponding to a QRS integral may have branches for low, normal, high, and very high, and a decision node corresponding to heart rate may have branches for bradycardia, normal heart rate, and tachycardia. The leaf nodes of the decision tree may indicate a classification. The assessment of a leaf node is intended for a person with features that match the values of the features along the path from the root node to that leaf node.

[0064] For an ML decision tree, an entropy score may be used by a ML decision tree generator to select the feature to be associated with each node. The entropy score for a possible feature for a node is based on the distribution of its values in node feature vectors for that node. A node feature vector for a node has the values of the branches along the path from the root node to that node. If a first possible feature for a node has an equal number of node feature vectors for each value, the entropy for the first possible feature is considered to be high. In contrast, if a second possible feature has node feature vectors for which 75% have the same value, then the entropy for the second possible feature is considered to be low. In such a case, the ML decision tree generator would select the second possible feature for that node. The ML decision tree generator may also analyze features with continuous values to identify cut points that tend to minimize entropy. For example, as mentioned above, heart rate may be categorized as bradycardia (<60), normal (60-100), or tachycardia (>100). However, since heart rate is a continuous value, the PP system may employ techniques to identify cut points (other than 60 and 100) that would reduce the entropy. For example, a single cut point of 80 or cut points of 50 and 90 may tend to minimize entropy. Techniques for identifying cut points are described in Fayyad, U.M. and Irani, K.B., “On the Handling in DecisionTree of Continuous- Valued Attributes Generation,” Machine Learning, 8, pp.87-102 (1992), which is hereby incorporated by reference.

[0065] A kNN model provides information relating to a patient. The training dataset for a kNN model includes examples that each has a training feature vector (e.g., ECG images) and a training label indicating information (e.g., arrhythmia classification such as AF or VF) associated with such a feature vector. A kNN model may be used without a training phase, that is without learning weights or other parameters to represent the training dataset. In such a case, the patient feature vector is compared to the training feature vectors to identify a number (e.g., represented by the “k” in kNN) of similar training feature vectors. Once the number of similar training feature vectors are identified, the labels associated with the similar training feature vectors are analyzed to provide information for the patient (e.g., arrhythmia type). The labels of the training feature vectors that are more similar to the patient feature vector may be given a higher weight than those that are less similar. For example, if k is 10 and four training feature vectors are very similar and six are less similar, similarity weights of 0.9 may be assigned to the very similar training feature vectors and 0.2 to the less similar. If three of the four and one of the six have the same information, then the information for the entity is primarily based on that information even though most of the 10 have different information. Conceptually, training feature vectors that are very similar are closer to the patient feature vector in a multi-dimensional space of features and a similarity weight is based on distance between the training feature vectors and the patient feature vector. Various techniques may be employed to calculate a similarity metric indicating similarity between a candidate feature vector and a training feature vector such as a dot product, cosine similarity, a Pearson’s correlation, and so on.

[0066] If the number of training feature vectors is large, various techniques may be employed to effectively “compress” the training dataset during a training phase. For example, a clustering technique may be employed to identify clusters of training feature vectors that are similar and have the same label. A training feature vector may be generated for each cluster (e.g., one from the cluster or one based on mean values for the features) as a cluster feature vector and assign a cluster weight to it based on number of training feature vectors in the cluster.

[0067] The ML models that input a cardiogram input a feature vector of one or more features derived from the cardiogram. The features may include an image of a cardiogram, a voltage-time series specifying voltages and time increments of the cardiogram, images and voltage-time series of portions of the cardiogram (e.g., QRS complex), length in seconds of various intervals (e.g., R-R interval, QRS complex, T wave, T-Q interval, and Q-R interval), QRS integral, maximum, minimum, mean, and variance of voltages of portions of the cardiogram, a maximal vector of QRS loop and angle of the vector derived from VCG, location of a peak (Q peak) or zero crossing relative to a maximum peak (T peak) in an interval, and so on. The features used by an ML model may be manually or automatically selected. An assessment of which features may be useful in providing an accurate output for an ML model are referred to as informative feature. The assessment of which features are informative may be based on various feature selection techniques such as a predictive power score, a lasso regression, a mutual information analysis, and so on.

[0068] The features may also be latent vectors generated using an ML model such as an autoencoder. For example, an autoencoder may be trained using ECG images. In such a case, when an ECG image is input into the trained autoencoder, the latent vector that is generated is a feature vector that represents the ECG image. That feature vector can be input into another trained ML model such as a neural network or support vector machine to generate an output. When training the other ML model, for example, to classify an ECG as representing an atrial fibrillation or a ventricular fibrillation, the training ECG images are input to the autoencoder to generate training feature vectors that are labeled as being atrial fibrillation or ventricular fibrillation. The other ML model is then trained using the labeled feature vectors. The autoencoder may be trained using the training ECG images or may have been previously trained using a collection of ECG images. Rather pre-training an autoencoder, only the portion of the autoencoder that generates the latent vector may be trained in parallel with the other ML model using a combined loss function. In such a case, no autoencoding is performed. Rather the latent vector represents features of an ECG image that are particularly relevant to generating the output of the other ML model. Such an ML architecture may be used, for example, when the other ML model (e.g., transformer) is not designed to process ECG images directly.

[0069] The following paragraphs describe various aspects of the PP system. An implementation of the PP system may employ any combination or sub-combination of the aspects and may employ additional aspects. The processing of the aspects may be performed by one or more computing systems with one or more processors that execute computer-executable instructions that implement the aspects and that are stored on one or more computer-readable storage mediums.

[0070] In some aspects, the techniques described herein relate to a method for treating a patient, the method including: under control of one or more computing systems, generating a candidate path for a catheter to be used in a cardiac ablation procedure by accessing patient data and a 3D image of a patient; applying a patient classification system to features derived from the patient data and a 3D mesh generated based on the 3D image to generate a patient classification for the patient; accessing a cardiogram of the patient data; identifying, based on the cardiogram, a source location of an arrhythmia of the patient; applying a candidate path machine learning (ML) model to the 3D mesh, the patient classification, and the source location to generate a candidate path; and outputting an indication of the candidate path; and treating the patient with a cardiac ablation using a catheter whose catheter path is based on the candidate path. In some aspects, the techniques described herein relate to a method wherein the generating further includes applying an ablation pattern system to the patient classification and the source location to generate an ablation pattern and wherein the candidate path ML model is further applied to the ablation pattern. In some aspects, the techniques described herein relate to a method wherein the generating further includes segmenting the 3D image wherein the 3D mesh is generated based on the segmented 3D image. In some aspects, the techniques described herein relate to a method wherein the candidate path ML model is further applied to planning data and a catheter specification. In some aspects, the techniques described herein relate to a method wherein the patient data includes an electrical activation map of the heart of the patient. In some aspects, the techniques described herein relate to a method wherein the patient classification system is based on a k-nearest neighbor classifier, a k-means clustering system, or a convolutional neural network. In some aspects, the techniques described herein relate to a method wherein the candidate path ML model is a neural network. In some aspects, the techniques described herein relate to a method wherein the candidate path ML model is trained using a training dataset with examples derivedfrom simulated data. In some aspects, the techniques described herein relate to a method wherein the candidate path is updated during a procedure based on an actual path of the catheter.

[0071] In some aspects, the techniques described herein relate to a method performed by one or more computing systems for generating one or more candidate paths for a catheter during a cardiac ablation procedure, the method including: accessing a starting point, a target point, a cardiac geometry, and a catheter specification; initializing a path to the starting point; repeating until the path satisfies a termination criterion based on a feasibility criterion and target criterion, adding a next point to the path to extend the path by the next point, assessing whether the path satisfies the feasibility criterion based on the cardiac geometry and catheter specification, and assessing whether the path satisfies the target criterion; and when the path satisfies the target criterion and the feasibility criterion, indicating that the path is a candidate path. In some aspects, the techniques described herein relate to a method wherein the adding is performed for a plurality of next points wherein each next point extends the path to form a new path and assessing of whether a path satisfies a feasibility criterion is performed for each new path. In some aspects, the techniques described herein relate to a method wherein the cardiac geometry is derived from a 3D image of a patient. In some aspects, the techniques described herein relate to a method wherein the cardiac geometry is represented by a 3D mesh. In some aspects, the techniques described herein relate to a method wherein the 3D mesh is derived from a segmentation of a 3D image of a patient. In some aspects, the techniques described herein relate to a method wherein the assessing whether the path satisfies a feasibility criterion factors in avoid structures.

[0072] In some aspects, the techniques described herein relate to a method performed by one or more computing systems for catheter path planning for a cardiac ablation procedure for a patient, the method including: receiving patient data that includes a three-dimensional (3D) representation, tissue data, and electrical activation data of a heart of a patient, receiving planning data and avoidance structure data; receiving a catheter specification; applying a path planning system to the 3D representation, patient data, planning data, avoidance structure data, and catheter specification to generate one or more candidate paths; and outputting an indication of one or more of the candidate paths. In some aspects, the techniques described hereinrelate to a method further including treating the patient with an ablation based on a candidate path. In some aspects, the techniques described herein relate to a method wherein the outputting includes generating a 3D graphic based on the 3D representation, superimposing an indication of a candidate path on the 3D graphic, and displaying the 3D graphic. In some aspects, the techniques described herein relate to a method wherein the planning data includes a starting point and target location of a path. In some aspects, the techniques described herein relate to a method wherein the path planning system includes a patient classification ML model that inputs a feature vector derived from the 3D representation and the patient data and outputs a patient classification wherein patients with similar patient data have the same classification. In some aspects, the techniques described herein relate to a method wherein the patient classification ML system is based on a k-nearest neighbor classifier, a k-means clustering system, or a convolutional neural network. In some aspects, the techniques described herein relate to a method wherein the path planning system inputs a source location identified by a mapping system based on a cardiogram of the patient data. In some aspects, the techniques described herein relate to a method wherein the path planning system inputs an ablation pattern identified by an ablation pattern system based on a source location of an arrhythmia and the patient data. In some aspects, the techniques described herein relate to a method wherein the 3D representation is a 3D mesh. In some aspects, the techniques described herein relate to a method wherein the path planning system includes a candidate path system. In some aspects, the techniques described herein relate to a method wherein the candidate path system is a candidate path ML model. In some aspects, the techniques described herein relate to a method wherein the candidate path system is an algorithm.

[0073] In some aspects, the techniques described herein relate to one or more computing systems for catheter path planning for a cardiac ablation procedure to treat an arrhythmia of a patient, the one or more computing systems including: one or more computer-readable storage mediums that store a three-dimensional (3D) image of the patient’s heart and an electrocardiogram collected from the patient; and computerexecutable instructions for controlling the one or more computing systems to: identify a source location of the arrhythmia based on the electrocardiogram; generate a 3D mesh based on the 3D image; apply a path planning system to the 3D mesh, a starting point, and a target point to generate a candidate path from the starting point to the target point,the target point derived from the source location; and output an indication of the candidate path as a path for the catheter; and one or more processors for controlling the one or more computing systems to execute one or more of the computer-executable instructions. In some aspects, the techniques described herein relate to one or more computing systems wherein the indication of the candidate path includes a 3D graphic derived from the 3D mesh wherein the 3D graphic with the indication of a candidate path included in the 3D graphic. In some aspects, the techniques described herein relate to one or more computing systems wherein the path planning system includes a patient classification machine learning (ML) model that inputs a feature vector derived from the 3D mesh and the electrocardiogram and outputs a patient classification wherein patients with similar 3D meshes and electrocardiograms have similar classifications. In some aspects, the techniques described herein relate to one or more computing systems wherein the computer-executable instructions further control the one or more computing systems to generate feasibility scores for partial candidate paths based on a cardiac geometry represented by the 3D image, avoidance structures, and a catheter specification. In some aspects, the techniques described herein relate to one or more computing systems wherein the computer-executable instructions further control the one or more computing systems to output an indication of the candidate path to an ablation device for use during the ablation procedure.

[0074] In some aspects, the techniques described herein relate to one or more computing systems for training a candidate path machine learning (ML) model to identify a candidate path for a catheter to be used in an ablation on a heart, the one or more computing systems including: one or more computer-readable storage mediums that store computer-executable instructions for controlling the one or more computing systems to: access a training dataset with examples that each have a feature vector with features and a label indicating a candidate path, the features including a representation of a cardiac geometry of a heart, a source location, and an indication of avoidance structures; train the candidate path ML model using the training dataset to learn weights for the candidate path ML model; and store the weights so that the weights can be used when applying the candidate path ML model to a feature vector derived from a patient to identify the candidate path; and one or more processors for controlling the one or more computing systems to execute one or more of the computer-executable instructions. In some aspects, the techniques described herein relate to one or morecomputing systems wherein the computer-executable instructions further control the one or more computing systems to apply the candidate path ML model to a feature vector with features derived from the cardiac geometry, a source location, and avoidance structures associated with a target patient to identify a target candidate path. In some aspects, the techniques described herein relate to one or more computing systems wherein the target patient is treated with an ablation procedure wherein a procedure path of the catheter used in the ablation procedure is based on the target candidate path. In some aspects, the techniques described herein relate to one or more computing systems wherein at least some of the examples of the training dataset are derived from simulated cardiac geometries, simulated source locations, and simulated avoidance structures. In some aspects, the techniques described herein relate to one or more computing systems wherein at least some of the examples of the training dataset are derived from electronic health records of patients.

[0075] Although the subject matter has been described in language specific to structural features and / or acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMS1 . A method for treating a patient, the method comprising: under control of one or more computing systems, generating a candidate path for a catheter to be used in a cardiac ablation procedure by accessing patient data and a 3D image of a patient; applying a patient classification system to features derived from the patient data and a 3D mesh generated based on the 3D image to generate a patient classification for the patient; accessing a cardiogram of the patient data; identifying, based on the cardiogram, a source location of an arrhythmia of the patient; applying a candidate path machine learning (ML) model to the 3D mesh, the patient classification, and the source location to generate a candidate path; and outputting an indication of the candidate path; and treating the patient with a cardiac ablation using a catheter whose catheter path is based on the candidate path.

2. The method of claim 1 wherein the generating further includes applying an ablation pattern system to the patient classification and the source location to generate an ablation pattern and wherein the candidate path ML model is further applied to the ablation pattern.

3. The method of claim 1 wherein the generating further includes segmenting the 3D image wherein the 3D mesh is generated based on the segmented 3D image.

4. The method of claim 1 wherein the candidate path ML model is further applied to planning data and a catheter specification.

5. The method of claim 4 wherein the patient data includes an electrical activation map of the heart of the patient.

6. The method of claim 1 wherein the patient classification system is based on a k-nearest neighbor classifier, a k-means clustering system, or a convolutional neural network.

7. The method of claim 1 wherein the candidate path ML model is a neural network.

8. The method of claim 1 wherein the candidate path ML model is trained using a training dataset with examples derived from simulated data.

9. The method of claim 1 wherein the candidate path is updated during a procedure based on an actual path of the catheter.

10. A method performed by one or more computing systems for generating one or more candidate paths for a catheter during a cardiac ablation procedure, the method comprising: accessing a starting point, a target point, a cardiac geometry, and a catheter specification; initializing a path to the starting point; repeating until the path satisfies a termination criterion based on a feasibility criterion and target criterion, adding a next point to the path to extend the path by the next point, assessing whether the path satisfies the feasibility criterion based on the cardiac geometry and catheter specification, and assessing whether the path satisfies the target criterion; and when the path satisfies the target criterion and the feasibility criterion, indicating that the path is a candidate path.1 1 . The method of claim 10 wherein the adding is performed for a plurality of next points wherein each next point extends the path to form a new path and assessing of whether a path satisfies a feasibility criterion is performed for each new path.

12. The method of claim 10 wherein the cardiac geometry is derived from a 3D image of a patient.

13. The method of claim 10 wherein the cardiac geometry is represented by a 3D mesh.

14. The method of claim 13 wherein the 3D mesh is derived from a segmentation of a 3D image of a patient.

15. The method of claim 10 wherein the assessing whether the path satisfies a feasibility criterion factors in avoid structures.

16. A method performed by one or more computing systems for catheter path planning for a cardiac ablation procedure for a patient, the method comprising: receiving patient data that includes a three-dimensional (3D) representation, tissue data, and electrical activation data of a heart of a patient, receiving planning data and avoidance structure data; receiving a catheter specification; applying a path planning system to the 3D representation, patient data, planning data, avoidance structure data, and catheter specification to generate one or more candidate paths; and outputting an indication of one or more of the candidate paths.

17. The method of claim 16 further including treating the patient with an ablation based on a candidate path.

18. The method of claim 16 wherein the outputting includes generating a 3D graphic based on the 3D representation, superimposing an indication of a candidate path on the 3D graphic, and displaying the 3D graphic.

19. The method of claim 16 wherein the planning data includes a starting point and target location of a path.

20. The method of claim 16 wherein the path planning system includes a patient classification ML model that inputs a feature vector derived from the 3D representation and the patient data and outputs a patient classification wherein patients with similar patient data have the same classification.21 . The method of claim 20 wherein the patient classification ML system is based on a k-nearest neighbor classifier, a k-means clustering system, or a convolutional neural network.

22. The method of claim 16 wherein the path planning system inputs a source location identified by a mapping system based on a cardiogram of the patient data.

23. The method of claim 16 wherein the path planning system inputs an ablation pattern identified by an ablation pattern system based on a source location of an arrhythmia and the patient data.

24. The method of claim 16 wherein the 3D representation is a 3D mesh.

25. The method of claim 16 wherein the path planning system includes a candidate path system.

26. The method of claim 16 wherein the candidate path system is a candidate path ML model.

27. The method of claim 16 wherein the candidate path system is an algorithm.

28. One or more computing systems for catheter path planning for a cardiac ablation procedure to treat an arrhythmia of a patient, the one or more computing systems comprising: one or more computer-readable storage mediums that store a three-dimensional (3D) image of the patient’s heart and an electrocardiogram collected from the patient; andcomputer-executable instructions for controlling the one or more computing systems to: identify a source location of the arrhythmia based on the electrocardiogram; generate a 3D mesh based on the 3D image; apply a path planning system to the 3D mesh, a starting point, and a target point to generate a candidate path from the starting point to the target point, the target point derived from the source location; and output an indication of the candidate path as a path for the catheter; and one or more processors for controlling the one or more computing systems to execute one or more of the computer-executable instructions.

29. The one or more computing systems of claim 28 wherein the indication of the candidate path includes a 3D graphic derived from the 3D mesh wherein the 3D graphic with the indication of a candidate path included in the 3D graphic.

30. The one or more computing systems of claim 28 wherein the path planning system includes a patient classification machine learning (ML) model that inputs a feature vector derived from the 3D mesh and the electrocardiogram and outputs a patient classification wherein patients with similar 3D meshes and electrocardiograms have similar classifications.31 . The one or more computing systems of claim 28 wherein the computerexecutable instructions further control the one or more computing systems to generate feasibility scores for partial candidate paths based on a cardiac geometry represented by the 3D image, avoidance structures, and a catheter specification.

32. The one or more computing systems of claim 28 wherein the computerexecutable instructions further control the one or more computing systems to output an indication of the candidate path to an ablation device for use during the ablation procedure.

33. One or more computing systems for training a candidate path machine learning (ML) model to identify a candidate path for a catheter to be used in an ablation on a heart, the one or more computing systems comprising: one or more computer-readable storage mediums that store computerexecutable instructions for controlling the one or more computing systems to: access a training dataset with examples that each have a feature vector with features and a label indicating a candidate path, the features including a representation of a cardiac geometry of a heart, a source location, and an indication of avoidance structures; train the candidate path ML model using the training dataset to learn weights for the candidate path ML model; and store the weights so that the weights can be used when applying the candidate path ML model to a feature vector derived from a patient to identify the candidate path; and one or more processors for controlling the one or more computing systems to execute one or more of the computer-executable instructions.

34. The one or more computing systems of claim 33 wherein the computerexecutable instructions further control the one or more computing systems to apply the candidate path ML model to a feature vector with features derived from the cardiac geometry, a source location, and avoidance structures associated with a target patient to identify a target candidate path.

35. The one or more computing systems of claim 34 wherein the target patient is treated with an ablation procedure wherein a procedure path of the catheter used in the ablation procedure is based on the target candidate path.

36. The one or more computing systems of claim 33 wherein at least some of the examples of the training dataset are derived from simulated cardiac geometries, simulated source locations, and simulated avoidance structures.

37. The one or more computing systems of claim 33 wherein at least some of the examples of the training dataset are derived from electronic health records of patients.

Images