Full Ablation Workflow System

Abstract

Techniques are provided for supporting cardiac stereotactic ablation radiotherapy (SABR) procedures. The technique collects an arrhythmic electrocardiogram (ECG) and a CT scan of a patient. The technique employs a mapping system to generate a bounded generic three-dimensional (3D) mesh based on the ECG. The bounded generic 3D mesh has bounded ablation targets. The technique employs a 3D machine learning (ML) model to generate a patient-specific 3D mesh based on the CT scan. The technique uses the bounded ML model to generate a bounded patient-specific 3D mesh based on the patient-specific 3D mesh and the bounded generic 3D mesh. The bounded patient-specific 3D mesh has bounded ablation targets, accounting for differences in cardiac geometry between the patient-specific 3D mesh and the bounded generic 3D mesh.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. patent application Ser. No. 63 / 316,960, entitled "OVERALL ABLATION WORKFLOW SYSTEM," filed Mar. 4, 2022, which is incorporated herein by reference. [Background technology]

[0002] Many cardiac diseases can cause symptoms, morbidity (e.g., syncope or stroke) and mortality. Common cardiac diseases caused by arrhythmias include inappropriate sinus tachycardia (IST), ectopic atrial rhythm, junctional rhythm, ventricular escape rhythm, atrial fibrillation (AF), ventricular fibrillation (VF), focal atrial tachycardia (focal AT), atrial micro-reentry, ventricular tachycardia (VT), atrial flutter (AFL), premature ventricular contractions (PVC), premature atrial contractions (PAC), atrioventricular nodal reentrant tachycardia (AVNRT), atrioventricular reentrant tachycardia (AVRT), permanent junctional reciprocating tachycardia (PJRT), and junctional tachycardia (JT). Arrhythmia sources include electrical rotors (e.g., ventricular fibrillation), recurrent electrical focal sources (e.g., atrial tachycardia), anatomical-based reentry (e.g., ventricular tachycardia), etc. These sources are important contributors to the development of persistent or clinically significant arrhythmias. Arrhythmias can be treated by ablation by targeting the source of cardiac disease using a variety of techniques, including radiofrequency energy ablation, other electromagnetic energy ablation, cryoablation, ultrasound ablation, laser ablation, external radiation sources, directed gene therapy, etc. Because the source and site of cardiac disease vary from patient to patient, targeted therapy, even for common cardiac diseases, requires individual identification of the source of arrhythmia for each patient.

[0003] One technique that supports treating patients with ablation is stereotactic ablative radiation therapy (SABR), also known as stereotactic body radiation therapy (SBRT). SABR is a highly focused radiation therapy that delivers a concentrated high-intensity dose of radiation to a target site in an organ (e.g., the heart or lungs) while limiting the dose to surrounding organs. The high-intensity dose is delivered in a series of lower-intensity doses delivered from different angles to the target site. The delivery angle can be specified by the orientation of the delivery arm. The purpose of applying the lower-intensity doses at various angles is to reduce the chance of damaging healthy tissue, while the cumulative application of the lower-intensity doses focused on the target site will deliver a high-intensity dose to the target site that is sufficient to stop the electrical activity of the defective tissue at the target site.

[0004] The SABR device includes radiation delivery components mounted on a robotic manipulator, a scanning device (e.g., a CT scanner), and a guidance system. The guidance system guides the delivery of radiation during a procedure based on a delivery plan created in part based on images of the patient collected during the patient's treatment. The delivery plan includes 3D images (e.g., 2D slices) of the organ (e.g., the heart) and surrounding structures (e.g., the lungs), with the organ target site for ablation and structures to be avoided (avoidance structures) demarcated on the 3D images. The delivery plan also specifies the high-intensity doses to be applied. The SABR device determines the delivery angle of each low-intensity dose. The guidance system guides the robotic manipulator to a predetermined position at a desired angle based on CT scans collected during the procedure and factors of organ and structure movement (e.g., due to breathing). Once positioned at an angle, the SABR device delivers the low-intensity dose specified for that angle.

[0005] Unfortunately, it currently takes a significant amount of time for a healthcare provider, such as an electrophysiologist (EP), to create a delivery plan for a SABR procedure for a patient. To perform an ablation on the heart to treat an arrhythmia, the healthcare provider must first identify the source of the ablation. The source location has traditionally been identified by first acquiring an arrhythmic electrocardiogram (ECG) of the patient during the arrhythmia episode. The healthcare provider then invasively paces the heart at various locations while acquiring images and a pacing ECG. If the pacing ECG matches the arrhythmic ECG, the healthcare provider can assume that the pacing location is the source location of the arrhythmia. The pacing location is typically determined manually from images acquired during pacing at the pacing location. The healthcare provider can alternatively identify the source based on analysis of a CT scan acquired during the arrhythmia episode.

[0006] The healthcare provider also collects a CT scan of the patient using a standalone CT scanner. The healthcare provider then manually demarcates the location of the source on the CT scan as a target site for ablation, generating a demarcated CT scan. The healthcare provider also demarcates avoidance structures on the CT scan. Because this is a manual process, the demarcated CT scan may not have the desired accuracy. For example, if the avoidance structures are not precisely demarcated, the ablation procedure may not have the desired results.

[0007] At the start of the ablation procedure, a SABR CT scan is acquired from the patient using a CT scanner that is part of the SABR device. A healthcare provider, which may be different from the healthcare provider that generated the demarcated CT scan, then transfers the demarcations from the demarcated CT scan to the SABR CT scan, generating the demarcated SABR CT scan. Typically, the healthcare facility's standalone CT scanner has a higher resolution than the SABR CT scanner, so the SABR CT scan has a lower resolution than the demarcated CT scan. As a result, the transfer of the demarcated CT scan must take into account the difference in resolution. Again, because this is a manual process, the demarcated SABR CT scan may not have the desired accuracy.

[0008] The healthcare provider creates a delivery plan of the delivery angles and the low intensity dose to deliver at each angle based on the demarcated SABR CT scan, and then relies on the SABR device to execute the delivery plan. However, the SABR device does not allow the healthcare provider to create the delivery plan. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a flow diagram illustrating an overall process for treating a patient using an OAW system in some embodiments. [Diagram 2] FIG. 2 is a block diagram illustrating components of an OAW system in some embodiments. [Diagram 3] FIG. 1 is a flow diagram illustrating the processing of delimited patient-specific 3D mesh components of an OAW system in some embodiments. [Figure 4] FIG. 2 is a flow diagram illustrating the processing of the mapping system in some embodiments. [Diagram 5] FIG. 1 is a flow diagram illustrating the processing of the train bounding machine learning model component of an OAW system in some embodiments. [Figure 6]FIG. 11 is a flow diagram illustrating the processing of a delivery plan creation component of the OAW system in some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Methods and systems are provided that support a therapeutic procedure for treating a patient with arrhythmia or other medical procedure to ensure that the treatment is successful and that the treatment plan is performed efficiently and accurately in less time than required by current therapeutic procedures. In some embodiments, an ocular ablation workflow (OAW) system provides information to an electrophysiologist (EP) as he or she performs an ablation on a patient. The OAW system can employ a 3D machine learning (ML) model to generate a patient-specific 3D representation of all or a portion of the patient's heart, as well as labels for segments within the patient's heart and segments that are non-cardiac structures within the patient's thorax. The patient-specific 3D representation can be implemented as a patient-specific 3D mesh having vertices, edges, and faces based on the patient's heart geometry. After the patient-specific 3D mesh is generated, the OAW system accesses a bounded generic 3D mesh having bounded cardiac features (e.g., ablation target locations). The bounded generic 3D mesh can be derived, for example, from a simulated 3D mesh based on a simulated cardiac geometry used in simulating cardiac electrical activity based on simulated cardiac characteristics (e.g., simulated cardiac geometry and simulated source locations). The OAW system can employ a bounding ML model that inputs a patient-specific 3D mesh and a bounded generic 3D mesh and outputs a patient-specific 3D mesh in which one or more cardiac features are bounded. The bounded patient-specific 3D mesh can then be provided to an ablation therapy device, displayed to a healthcare provider, employed to create a treatment delivery plan, etc., to aid in the treatment of the patient.

[0011] The OAW system is described primarily in the context of a 3D image that is a CT scan with CT slices. However, the OAW system can be used for any type of representation of a 3D image, such as an MRI scan, a 3D image specified with a 3D mesh, a collection of 2D projections of the heart (or thorax), etc. The OAW system is also described primarily in the context of ablation performed on the heart. The OAW system can also be used to assist in ablation of various organs (e.g., heart, lungs, gastrointestinal tract, and brain) of various patients (e.g., humans and animals) or other treatments based on electrograms that represent electromagnetic energy corresponding to abnormal electrical activity of these organs.

[0012] In some embodiments, the 3D ML model is trained using training data with a training set including CT scans of the heart, each labeled with a 3D mesh representing the heart. The training data can be generated, for example, by generating simulated 3D meshes based on various heart geometries and generating simulated CT scans for each simulated 3D mesh. The simulated CT slices of the simulated CT scans can be generated by simulating the amount of radiation emitted on the simulated CT slices from a simulated radiation source passing through a heart in the shape of the simulated 3D mesh. The amount of radiation impinging on the simulated CT slices can be derived from attenuation coefficients based on the material type of the cardiac and non-cardiac segments of the simulated thorax. Multiple CT scans can be generated based on the simulated 3D mesh, each based on a different segmentation of the simulated thorax, such as different heart orientations, lung sizes, etc. To generate the simulated CT slices, the OAW system may employ techniques for generating simulated 2D images from the simulated 3D representations, as described in U.S. Pat. No. 10,713,791 (the '791 patent) and 10,713,774, issued July 15, 2020, entitled "Computational Simulations of Anatomical Structures and Body Surface Electrode Positioning," which is incorporated herein by reference. The three-dimensional ML model may comprise ML sub-models, such as a segmentation ML sub-model, a labeling ML sub-model, and a three-dimensional ML sub-model. In such cases, each ML sub-model may have its own training data and may be trained with separate or combined loss functions.

[0013] The segmentation ML sub-model inputs a CT slice and outputs a segmentation of the CT slice. The segments may be defined by the boundaries of the segmentation sites of the CT slice and, optionally, their material types. Since the CT slice may include non-cardiac structures within the thorax (e.g., lungs, esophagus), the segmentation ML sub-model may also identify segments for such non-cardiac structures. However, a separate ML model may be used for segmenting the non-cardiac structures. The material type of the segments may be air (e.g., in the lungs), blood (e.g., in the ventricles or arteries), cardiac tissue, bone tissue, etc. The segments and material types may be specified using pixel coloring or shading. Alternatively, the segments and material types may be specified using metadata that specifies the boundaries of the segments and their material types. The segmentation ML sub-model may be trained using clinical data or simulated data. The clinical data may be CT scans and corresponding segmented CT scans collected from an electronic health record (EHR), and the simulated data may be simulated CT scans and corresponding simulated segmented CT scans. The segmentation ML sub-model can be implemented using various ML architectures, such as a convolutional neural network (e.g., a U-Net architecture or other ML architectures described below) that inputs a CT slice and outputs a segmented CT slice with its constituent material types identified.

[0014] The labeling ML sub-model inputs segmented CT slices of a CT scan and outputs labeling of segments of the CT scan. The segment labeling identifies chest structures (e.g., heart, esophagus, and liver). Furthermore, for the heart, the labels identify cardiac segments such as the right atrium, left ventricle, interventricular septum, tricuspid valve, pulmonary artery and pulmonary veins, other cardiac vessels, cardiac nerves, etc. The labeling ML sub-model can be implemented using various architectures such as a convolutional neural network that inputs segmented CT slices and outputs labeled CT slices. The labeling ML sub-model can be trained using clinical data or simulated data. The clinical data can be segmented CT scans and corresponding labeling CT scans collected from an EHR, and the simulated data can be simulated segmented CT scans and corresponding simulated labeling CT scans. In some embodiments, the segmentation ML sub-model and the labeling ML sub-model can be combined into a single ML sub-model that inputs a CT scan and outputs labeling of segments of the CT scan.

[0015] The 3D ML submodel inputs the labeled CT scans and outputs a patient-specific 3D mesh. The 3D ML submodel can be a recurrent convolutional neural network or a graph neural network that outputs the coordinates of the vertices of the patient-specific 3D mesh once the labeled CT slices are processed by the 3D ML submodel.

[0016] Rather than employing a 3D ML model to generate a patient-specific 3D mesh from either the CT scan or its ML submodel, the OAW system can employ other systems to generate the patient-specific 3D mesh. Such systems include 3D Slicer, OsiriX, and ITK-SNAP. For example, the OAW system can employ 3D Slicer to convert the output of the labeling ML submodel into a patient-specific 3D mesh. Various segmentation techniques are described in Wirjdai, O., "Survey of 3D Image Segmentation Methods," Model and Algorithms in Image Processing, Fraunhofer ITWM, Kaiserslautern, Germany, 2007, which is incorporated herein by reference.

[0017] The OAW system can identify cardiac characteristics using a patient arrhythmic electrocardiogram collected from the patient during an arrhythmia episode. The electrocardiogram can be, for example, an electrocardiogram (ECG) or a vectorcardiogram (VCG). The OAW system then provides the patient's arrhythmic electrocardiogram to a mapping system, which provides the patient's arrhythmic electrocardiogram to a mapping system that identifies a simulation having simulated cardiac characteristics that will generate a simulated arrhythmic electrocardiogram similar to the patient's arrhythmic electrocardiogram. The mapping system can generate a bounded generic 3D mesh based on a simulation of the cardiac electrical activity assuming different cardiac characteristics, such as the location of the arrhythmia source, the location of scar tissue, the shape and orientation of the heart, electrical properties, etc. Techniques for simulating cardiac electrical activity are described in U.S. Pat. No. 10,860,754 (the '754 patent), entitled "Calibration of Simulated Cardiograms," issued Dec. 8, 2020, which is incorporated herein by reference. The mapping system can generate a bounded generic 3D mesh in advance or on-demand based on a cardiac geometry associated with the identified simulation. Furthermore, since many simulations can be based on the same simulated cardiac geometry, the mapping system can have a pre-generated generic 3D mesh for each simulated cardiac geometry, rather than one for each simulation. In such cases, the mapping system generates a bounded generic 3D mesh on demand based on the generic 3D mesh and simulated cardiac characteristics of the identified simulation. The generic 3D mesh is "generic" in the sense that it is not generated based on the patient's cardiac geometry. However, the OAW system can employ the calibration techniques of the '754 patent. In such cases, the generic 3D mesh can be considered patient-specific. Instead of or in addition to using a simulated 3D mesh, the mapping system can be based on clinical data collected from an EHR with cardiac geometry data and other cardiac characteristics from which a bounded generic 3D mesh can be generated.

[0018] After identifying a bounded generic 3D mesh having one or more bounded regions of interest (ROIs), the OAW system inputs the patient-specific 3D mesh and the bounded generic 3D mesh into a bounding ML model to generate a bounded patient-specific 3D mesh. The ROIs can be, for example, simulated origin locations of arrhythmias and sites of simulated scar tissue. As another example, the ROIs can be sites that define an ablation pattern for an ablation to be performed. Techniques for identifying an ablation pattern are described in U.S. Pat. No. 11,259,871, entitled "Identify Ablation Pattern for Use in an Ablation" (the '871 patent), issued March 1, 2022, which is incorporated herein by reference. The OAW system can specify the ROI of the bounded patient-specific 3D mesh using pixel coloring or shading, or using metadata that defines the ROI. The bounding ML model can employ a neural network that inputs the patient-specific 3D mesh and the bounded generic 3D mesh and outputs a bounded patient-specific 3D mesh. The demarcated patient-specific 3D mesh represents a heart based on the patient's cardiac geometry with ROIs (e.g., locations of arrhythmia generators) placed to account for differences between the patient's cardiac geometry and a generic cardiac geometry. Rather than employing machine learning, the OAW system can generate a transformation matrix that transforms vertices of the demarcated generic 3D mesh to corresponding vertices of the patient-specific 3D mesh, and then apply the transformation matrix to the ROIs to convert them into the patient-specific 3D mesh. The demarcated patient-specific 3D mesh is converted to a Digital Imaging and Communications in Medicine (DICOM) format and then provided to a treatment device to aid in treatment.

[0019] In some embodiments, the OAW system can provide the demarcated patient-specific 3D mesh or its information content in another format (e.g., CT scan) to a SABR device or other treatment device for use in patient treatment planning. For example, the OAW system can provide a demarcated patient-specific CT scan with simulated CT slices derived from slices of the demarcated patient-specific 3D mesh to an ablation treatment device, as described in the '791 patent. The OAW system generates the demarcated patient-specific CT scan at the resolution of a scanning device integrated with the SABR device, which may be below the resolution of typical scanning devices in a medical facility. The demarcated patient-specific CT scan also includes the demarcation of non-cardiac structures that can be identified by a segmentation ML submodel.

[0020] The ablation therapy device can generate a treatment delivery plan based on the demarcated patient-specific CT scan or the demarcated patient-specific 3D mesh. Alternatively, the OAW system can generate a delivery plan and provide it to an ablation therapy device adapted to receive such a delivery plan. To generate a delivery plan, the OAW system identifies avoidance segments (e.g., nerves and esophagus) based on segment labeling. The OAW system then applies the planning ML model to the demarcated patient-specific CT scan (or 3D mesh) and the indicated avoidance segments and radiation dose information to generate a delivery plan. The delivery plan specifies the motion of the robot manipulator, the fraction of the dose to be delivered in each direction of the delivery arm, the shape of the delivery beam, etc. The treatment can then be performed by the ablation therapy device according to the delivery plan.

[0021] In some embodiments, the mapping system compares the patient's arrhythmic electrocardiogram to a library of library electrocardiograms, each associated with a bounded generic 3D mesh based on simulated cardiac characteristics. The library cardiograms can be based on simulated electrical activity of hearts having different simulated cardiac characteristics (or cardiac data), such as different cardiac geometries, electrical characteristics, scar locations, source locations, prior ablation locations, prior or suggested ablation patterns, etc. Each library myocardial infarction is associated with the simulated myocardial characteristics used in the simulation from which the library myocardial infarction was generated. A source site associated with a library cardiogram that is similar to the patient's arrhythmic cardiogram (e.g., based on a similarity or matching metric generated using Pearson correlation or other similarity criteria) can represent the source site of the patient's arrhythmia and can represent a target site for ablation. The ablation pattern (see '781 patent) used in the simulation to terminate the arrhythmia can also be the ablation pattern used in the ablation procedure on the patient. To identify similar library cardiograms, the mapping system may first filter the library (e.g., calibrate the library) to identify library cardiograms having simulated cardiac characteristics (e.g., cardiac geometry or conduction velocity) similar to the patient's cardiac characteristics. (The mapping system then identifies library echocardiograms from the filtered library that are similar to the patient's echocardiogram. A bounded generic 3D mesh is generated based on the cardiac characteristics (e.g., shape and orientation) associated with the similar library phonocardiograms. The bounded generic 3D mesh may be generated before the simulation, after the simulation, or on-demand when planning a procedure.

[0022] The OAW system may alternatively employ a mapping system that applies a mapping ML model to the arrhythmic ECG (and possibly some patient cardiac characteristics) to identify the source location of the arrhythmia, with the mapping ML model outputting the source location and possibly other cardiac characteristics or ROIs. The mapping ML model may be trained using library ECGs that have been labeled with the associated source location and possibly labeled with simulated characteristics such as simulated cardiac geometry. (See the '754 patent for an example of a mapping ML model). The mapping ML model may also be trained using a bounded generic 3D mesh as a label, whereby the mapping ML model also outputs a bounded generic 3D mesh. Prior to training, the library ECGs may be calibrated to the patient cardiac characteristics by identifying (filtering) a subset of the library ECGs that have similar cardiac characteristics and training the mapping ML model based on that subset. In such cases, the mapping ML model may be trained during procedure planning using techniques such as those described in the '754 patent, such as model weight transfer. In some embodiments, the OAW system can also identify an ablation pattern for ablation using an Ablation Pattern Identification (API) system such as that described in the '871 patent.

[0023] In some embodiments, the OAW system generates a patient-specific 3D graphic based on the bounded patient-specific 3D mesh and displays the patient-specific 3D graphic by superimposing a representation of one or more source locations (i.e., as a volume) on the heart. A system for generating such a graphic, referred to as a source location (SL) graphic, is described in U.S. Pat. No. 10,709,347, entitled "Heart Graphic Display System," issued July 14, 2020 (the '347 patent), which is incorporated herein by reference. The '347 patent describes identifying one or more source locations using a mapping system such as described above. The source locations can be represented by various indicia, such as an X to mark the source location, a color change, such as a change in intensity, to distinguish between likely and unlikely source locations.

[0024] In some embodiments, the OAW system can generate a projected patient-specific CT scan by projecting the projected patient-specific 3D mesh onto a patient CT scan acquired from a patient in which the ROI indicated by the mapping system was projected. The OAW system can employ a projection algorithm that identifies, for each 2D slice of the patient's CT scan, a corresponding 2D slice of the bounded patient-specific 3D mesh. The OAW system then bounds the 2D slice of the patient's CT scan based on the ROI of the corresponding 2D slice of the bounded patient-specific 3D mesh.

[0025] The OAW system may alternatively employ a projection ML model to generate a bounded patient-specific CT scan. To train the projection ML model, the OAW system accesses training data with training sets each including a CT scan of a heart and a bounded 3D mesh of the heart labeled with a ROI bounded CT scan. The OAW system may generate a feature vector and a label for each training set. The feature vector includes one or more features from the CT scan and the bounded 3D mesh, and the label may be the bounded CT scan itself. For example, the CT scan features may be the CT scan itself, segments of the CT scan, features based on principal component analysis, etc. The OAW system then trains the projection ML with the feature vector and labeling. The OAW system uses the projector ML model to generate a bounded patient-specific CT scan given a patient's CT scan and a bounded patient-specific 3D mesh. The training data may be from CT scans and bounded CT scans created for procedures that do not employ an OAW system.

[0026] The projection ML model can be based on various ML architectures. For example, the projection ML model can include a convolutional neural network for the patient-specific 3D mesh and a convolutional neural network for the CT scan. The output of the convolutional neural network is input to a fully connected layer that outputs the demarcated patient-specific CT scan. The loss function can be a combined loss function of the convolutional neural network and the fully connected layer trained jointly. If the demarcation is represented as metadata, a different ML architecture can be employed. In that case, in addition to the two convolutional neural networks described above, another neural network can be employed that inputs the metadata and generates an output that is provided to the fully connected layer. Also, rather than employing a convolutional neural network, a neural network can input the demarcated patient-specific 3D mesh and features from the CT scan and output the demarcated patient-specific CT scan. As another example, an autoencoder can be trained with the demarcated 3D mesh and another autoencoder can be trained with the CT scan. A neural network can then be trained using the latent vectors of the autoencoder labeled with the demarcated CT scan. Alternatively, the autoencoder and the neural network can be trained jointly. As another option, the autoencoder can be trained with the bounded CT scan and the labels of the neural network can be the corresponding latent vectors. To generate the bounded CT scan, the autoencoder is employed to generate a 3D mesh and latent vectors of the CT scan, the latent vectors are input to a neural network to generate a latent vector representing the bounded CT scan, and the latent vector representing the bounded CT mesh is input to a corresponding autoencoder that outputs a bounded 3D image.A convolutional neural network that inputs a CT scan or other 3D image can apply a 3D convolution window so that the activation function represents pixels not only in the same CT slice but also in adjacent CT slices. These ML architectures can be employed in the various ML models and ML sub-models described above.

[0027] In some embodiments, the demarcated patient-specific CT scan and the ablation therapy planning CT scan may have different resolutions, such as different pixel resolutions (e.g., 4K×4K vs. 2K×2K) and different number of slices (e.g., 1024 vs. 512). A transfer algorithm may be employed to set a pixel value of a pixel (e.g., representing a voxel) for a target pixel location in the 3D space of the demarcated planning CT scan based on pixel values ​​of pixels of the demarcated patient-specific CT scan that have pixel locations in the vicinity of the target pixel location. For example, the pixel value of the specified location may be set to the average value of the pixel values ​​of the neighboring pixels.

[0028] In some embodiments, the OAW system can train an avoidance ML model to generate an avoidance-defined (planning) CT scan with bounded target and avoidance regions. (The avoidance regions can alternatively be identified by coordinates in 3D space relative to the location of the heart). The OAW system accesses training data using training sets including the CT scans, the target regions, and the avoidance-defined CT scans. The OAW system generates a feature vector for each training set. The feature vector includes one or more features from the CT scan (e.g., the CT scan itself or a latent vector) and one or more features from the target regions, and the labeling is from the avoidance-defined CT scan (e.g., the bounded CT scan itself or a latent vector). The OAW system trains the avoidance ML model with the feature vector and the labels. The avoidance ML model can employ segmentation ML sub-models and labeling ML sub-models as described above. The OAW system uses the avoidance ML model to generate an avoidance-defined CT scan given the CT scan and the target regions. In some embodiments, the avoidance ML model can be trained to output an indication of the avoidance regions for the CT scan. Thus, an avoidance boundary-defined CT scan can be considered to be a combination of the CT scan and metadata (e.g., annotations or tags) that indicate the avoidance region. Training data can be obtained from CT scans created in procedures that do not employ an OAW system, the target region, and the avoidance boundary-defined CT scan. The avoidance machine learning model can use various ML architectures, such as those described above.

[0029] In some embodiments, the OAW system can train a planning ML model to generate a delivery plan for an ablation procedure. The OAW system accesses training data having training sets including avoidance-bounded CT scans with bounded target sites (and possibly ablation patterns) and other non-cardiac structures and avoidance regions, target doses to be delivered to the target sites, maximum doses to the non-target sites, and a delivery plan (e.g., including angles of delivery, low intensity doses for each angle, arc of motion of the robotic manipulator, shape of the delivery beam). The OAW system generates feature vectors and labels for each training set. The feature vectors include features obtained from the avoidance-bounded CT scans (e.g., indicating shapes and positions of the avoidance and target sites), target doses, maximum doses to the non-target sites, etc., and the labeling is obtained from the delivery plan. The OAW system trains a planning ML model with the feature vectors and labels. The OAW system uses the planning ML model to generate a delivery plan given an avoidance-bounded CT scan with an indication of the target sites and dose information. The training data can be obtained from avoidance boundary defined CT scans, dose, and delivery plans created in procedures that do not employ an OAW system, In some embodiments, the avoidance boundary defined CT scans used for training can be generated from CT scans acquired from patients or from simulated CT scans used to simulate the mapping system.

[0030] In some embodiments, the OAW system can identify possible delivery angles based on the avoidance structure. The OAW system can employ a planning algorithm to identify delivery angles (i.e., supported by the ablation therapy device) that intersect the target region and do not intersect the avoidance structure. Such angles are considered candidate angles. The planning algorithm can generate a delivery plan by selecting delivery angles from the candidate angles that do not intersect other candidate angles outside the target region and specifying a target dose for each delivery angle. The target dose is divided by the number of delivery angles to obtain the angle dose for the delivery angle. The dose applied to a target sub-region of the target region is the sum of the target doses of the delivery angles that intersect with that target sub-region. The OAW system can employ a greedy algorithm that processes the target sub-regions in sequence by selecting a delivery angle for each target sub-region based on the number of delivery angles selected for other target sub-regions and the maximum dose for the target sub-region given the selected delivery angle. If the maximum dose is exceeded for any target sub-region, another set of delivery angles is selected for one or more sub-regions. If the angle dose is variable, the planning algorithm also varies the angle dose as part of the greedy algorithm. The greedy algorithm may not find an optimal (e.g., least number of angles) delivery plan, but the delivery plan found will deliver the target dose to each subregion.

[0031] In some embodiments, the OAW system allows the healthcare provider to modify the demarcations, delivery plans, etc., as they see fit. For example, a patient may have a medical or physical condition that results in a segment being designated that is not an accurate representation of the corresponding avoidance site (e.g., too small or malformed) or whose avoidance status is incorrect. Even with such revisions, the use of an OAW system has significant advantages over current technology, including speed, accuracy in most cases, and the potential to be more comprehensive since it may identify and factor aspects that the healthcare provider cannot identify.

[0032] The machine learning (ML) models of the OAW system can be any of a variety or combination of supervised, semi-supervised, self-supervised, or unsupervised ML models, including fully connected, convolutional, recurrent, or autoencoder neural networks, or neural networks such as restricted Boltzmann machines, support vector machines, Bayesian classifiers, k-means clustering, decision trees, generative adversarial networks, and the like. If the ML model is a deep neural network, the model is trained with training data that includes features derived from the data and labeling corresponding to the data. For example, the data can be images of animals and the labels can be the names of the animals. The result of the training is a set of weights for the operating functions of the layers of the deep neural network. The trained deep neural network can be applied to new data to generate labels for the new data. If the ML model is a support vector machine, a hypersurface is found that partitions the space of possible inputs. For example, the hypersurface attempts to split positive examples (e.g., real images represented by the features) from negative examples (e.g., fake images represented by the features) by maximizing the distance between the positive and negative examples closest to the hypersurface. The trained support vector machine can then be applied to new data to generate a classification (e.g., real or fake) for the new data. The ML model can generate discrete domain values ​​(e.g., classifications), probabilities, and / or continuous domain values ​​(e.g., regression values, classification probabilities).

[0033] Training a support vector machine can use various techniques, such as adaptive boosting, which is an iterative process of running multiple tests on a set of training data. Adaptive boosting transforms a weak learning algorithm (one that performs at a level only slightly better than chance) into a strong learning algorithm (one that exhibits a low error rate). The weak learning algorithm is run on different subsets of the training data. The algorithm focuses increasingly on examples where the predecessor is prone to making mistakes. The algorithm corrects for mistakes made by the previous weak learner. The algorithm is adaptive because it adapts to the error rate of its predecessor. Adaptive boosting combines crude and moderately inaccurate heuristics to produce a high-performance algorithm. Adaptive boosting combines the results of the tests run separately into one highly accurate classifier. Adaptive boosting can use weak classifiers, which are unipartition trees with only two leaf nodes.

[0034] Neural network models have three main components: architecture, loss function, and search algorithm. The architecture defines the functional form that relates inputs and outputs (in terms of the network's topology, unit connectivity, and activation functions). The training process is to search in the weight space for a set of weights that minimizes the loss function. Neural network models can use standard or stochastic gradient descent as search methods with radial basis function (RBF) networks and backpropagation.

[0035] A convolutional neural network (CNN) has multiple layers, such as convolutional layers, pooling layers, and fully connected (FC) layers. Some more complex CNNs can have multiple convolutional layers, pooling layers, and FC layers. Each layer contains a neuron that corresponds to each output of that layer. A neuron inputs the output of the previous layer (or the original input) and applies an activation function to the input to generate an output.

[0036] A convolutional layer can include multiple filters (also called kernels or activation functions). A filter inputs, for example, a convolutional window of an image, applies a weight to each pixel in the convolutional window, and outputs a value for that convolutional window. For example, if a still image is 256x256 pixels, the convolutional window can be 8x8 pixels. The filter can apply a different weight to each of the 64 pixels in the convolutional window to generate a value.

[0037] The activation function has a weight for each input and generates an output by combining the inputs based on the weight. The activation function can be a rectified linear unit (ReLU) that sums the values ​​of each input multiplied by the weight to generate a weighted value and outputs max(0, weight) so that the output is non-negative. The weights of the activation function are learned during training of the ML model. The max(0, weight) ReLU function can be represented as another ReLU layer with a neuron for each output of the preceding layer that inputs that output and applies the ReLU function to generate a corresponding "rectified output".

[0038] A pooling layer can be used to reduce the size of the output of a previous layer by downsampling the output: for example, each neuron in a pooling layer inputs 16 outputs of the previous layer and produces one output.

[0039] In an FC layer, each neuron inputs all the outputs of the previous layer and can generate a weighted combination of those inputs. For example, if the previous layer generates 256 outputs and the FC layer inputs a neuron for each of three categories (e.g., dogs, cats, and birds), each neuron inputs the 256 outputs and applies a weight to generate a value for that category.

[0040] Graph Neural Networks (GNNs) are designed to operate on graph data based on convolutions over the neighborhoods of the nodes in the graph data. (See Gori, M. et al., "A New Model for Learning in Graph Domains," Proc. 2005 IEEE Int. Joint Conf. on Neural Networks, 2005 (vol. 2, pp. 729-734) IEEE.) The training data can include a graph where each node has features and labeling. When training a GNN, a node sends information to its neighbors based on a loss function to learn the weights of the node's features. After the GNN is trained, the graph data is input to the GNN, which can generate labels for each node based on the features of the nodes in the graph data.

[0041] Generative Adversarial Networks (GANs) or attributes (attGANs) can also be used. attGANs use GANs to train generative models. (See Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen, "AttGAN: Facial Attribute Editing by Only Changing What You Want," IEEE Transactions on Image Processing, 2019, and Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, "Generative Adversarial Nets," Advances in Neural Information Processing Systems, pp. 2672-2680, 2014, which are incorporated herein by reference.) attGANs include a generator specifier and an attGAN classifier, which are trained using training data that includes input images of objects and input attribute values ​​for each object. The generator includes a generator encoder and a generator decoder. The generator encoder receives an input image and is trained to generate a latent vector of latent variables that represent the input image. The generator decoder receives the latent vector of the input image and the input attribute values. The attGAN classifier receives an image and generates a prediction for the attribute values. The attGAN is trained to generate modified images that represent the input image modified based on the attribute values. The generator encoder and generator decoder form a generative model.

[0042] Multimodal machine learning combines different modalities of input data to make predictions. The modalities can be, for example, images, text, and audio.

[0043] In one multimodal machine learning approach, called "early fusion," data from different modalities are combined at the input stage and then trained on the multimodal data. The training data for these modalities includes a dataset of images, associated text, and associated audio and labels. The images, text, and audio can be used in their raw form or preprocessed to reduce dimensionality, for example by compressing the data into a byte array or applying principal component analysis. The resolution of the images and audio can also be reduced. The concatenated bytes can then be processed with a cross-attention mechanism to condense the concatenated bytes into vectors of a fixed size. The vectors are then used to train the ML model, primarily using supervised techniques, but self-supervised or unsupervised techniques can also be used.

[0044] In the second multimodal machine learning approach, data from different modalities can be kept separate at the input stage and used as input for different modality-specific ML models (e.g., CNN for image data and Recurrent Neural Network (RNN) for sequential data). The modality-specific ML models can be jointly trained to combine information from different modalities to make predictions, and a combination loss (across modalities) is used to adjust the weights of the models. Alternatively, the modality-specific ML models can be trained separately with separate loss functions for each modality. A combined ML model is then trained based on the output of the modality-specific models. Continuing with the example, the training data for each modality-specific ML model can be based on its data along with the labeling. The combined ML model is then trained with the output of the modality-specific ML models along with the final labeling.

[0045] Transformer machine learning uses an attention mechanism to process tokens representing the input in parallel. For example, when processing a sentence using a Transformer ML model, each word is represented as a token that includes the word's embedding and its position. The embedding is a vector representation of the word such that words with similar meanings are closer in vector space. The position is based on the position of the word in the sentence.

[0046] The Transformer ML model includes an encoder with a neural network and a self-attention mechanism, and a decoder with an attention mechanism, a self-attention mechanism, and a neural network.

[0047] The first layer of the encoder inputs the tokens, and each of the other layers inputs the encodings of the previous layer. The self-attention mechanism inputs the encodings of the previous layer (or the tokens for the first encoder) and weights the relevance of each encoding to each other to generate inputs for a neural network that generates encodings.

[0048] The first layer of the decoder inputs data based on tokens, and each other layer inputs the decoding of the previous layer. Each layer of the decoder also inputs the encoding of the encoder. The attention and self-attention mechanisms input the decoding (or the tokens of the first layer) and weight the relevance of each decoding to generate inputs for the neural network.

[0049] Although originally developed for processing text, Transformers have also been applied to image recognition. The input to a Transformer is a representation of a fixed-size patch of the image. The patch representation can be an encoding of each pixel in the patch, its row, column and color. The output of the Transformer is, for example, a classification of the image.

[0050] Fine-tuning learning is an ML technique that leverages previously trained ML models to train an ML model. For example, if a cat / tiger ML model was trained to recognize whether an image has a cat, a tiger, or either of them, fine-tuning can be used to train a cat / tiger breed ML model to recognize whether an image has a specific cat breed (e.g., Toyger), a specific tiger breed (e.g., Bengal), or either of them. The inner layer of the cat / tiger breed ML model has the same weights as the inner layer of the cat / tiger ML model. The input and output layers of the cat / tiger breed ML model are trained using images labeled as the cat breed, the tiger breed, or neither. Thus, training only requires learning the weights of the input and output layers. In the case of cats and tigers, the weights of the inner layers of the separately trained cat / tiger ML model and the cat / tiger breed ML model are similar because cats and tigers have similar characteristics. However, the input and output layers of the cat / tiger breed ML model need to be trained to take into account the differences between a cat breed and a tiger breed. Furthermore, fine-tuning may be performed based on a smaller training dataset than that used to train the Cat-Tiger ML model.

[0051] Self-supervised learning is an ML technique based on unlabeled training data. First, self-supervised learning augments the training data to generate additional training data to generate a set of similar training data. For example, if the training data are images of cats and tigers, self-supervised learning generates images with different sizes, shading, and orientations for each image data. The ML model can have an encoder layer, a pre-text task layer, and contrastive learning. The encoder layer encodes the images into latent vectors. The pre-text task layer contains weights to group the images into different clusters based on their differences using contrastive learning. Contrastive learning uses images as contrasts and a loss function to adjust the weights of the encoder and pre-text task layers. This technique is similar to k-means clustering, but is based on contrast instead of similarity. The weights of the pre-text layer can be used as initial weights for a primary task, such as a cat / tiger breed ML model that can be trained using labeled training data. Self-supervised learning can be performed on multimodal data as described above.

[0052] A computer system on which the OAW system may be implemented may include a central processing unit, input devices, output devices (e.g., display devices and speakers), storage devices (e.g., memory and disk drives), network interfaces, graphics processing units, cellular radio link interfaces, global positioning system devices, and the like. Input devices may include keyboards, pointing devices, touch screens, gesture recognition devices (e.g., for air gestures), head and eye tracking devices, microphones for voice recognition, and the like. Computer systems may include desktop computers, laptops, tablets, e-readers, personal digital assistants, smartphones, gaming devices, servers, and the like. Computer systems may access computer-readable media, including computer-readable storage media (or media) and data transmission media. A computer-readable storage medium is a tangible storage means that does not include a transitory propagating signal. Examples of computer-readable storage media include memory and other storage, such as primary memory, cache memory, secondary memory (such as DVDs), and the like. A computer-readable storage medium may be recorded or encoded with computer-executable instructions or logic that implements the OAW system. The data transmission medium is used to transmit data via ephemeral propagating signals or carrier waves (e.g., electromagnetic) over wired or wireless connections. The computer system may include a secure crypto-processor as part of the central processing unit to generate and securely store keys and to encrypt and decrypt data using the keys. The computer system may be a server housed in a data center, such as a cloud-based data center.

[0053] The OAW system may be described in the general context of computer-executable instructions, such as program modules and components, executed by one or more computers, processors, or other devices. Generally, program modules or components include routines, programs, objects, data structures, etc. that perform particular tasks or implement particular data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. Aspects of the OAW system may be implemented in hardware using, for example, application specific integrated circuits ("ASICs") or field programmable gate arrays ("FPGAs").

[0054] FIG. 1 is a flow diagram illustrating an overall process of treating a patient using an OAW system in some embodiments. The treatment process collects data from a patient, including 3D images that are CT scans of the patient's thorax, patient characteristics, and arrhythmic ECG, employs a mapping system to generate a bounded generic 3D mesh with bounded target sites, employs a 3D ML model and a bounding ML model to generate the bounded patient-specific 3D mesh and labels of the structures (e.g., labeling), and employs a planning ML model to generate a delivery plan based on the bounded patient-specific 3D mesh and labeling of the structures along with other information such as dosage information. In block 101, the process obtains an ECG from the patient. In block 102, the process obtains a CT scan of the patient's heart and other parts of the thorax. In block 103, the process obtains patient characteristics such as heart geometry, scar location, electrical properties, etc. In block 104, the process applies the mapping system to the ECG to generate a bounded (DM) generic (G) 3D mesh with bounded arrhythmia source locations. If the mapping system employs an electrocardiogram that is a VCG, the ECG can be converted to a VCG. In block 105, the process applies the 3D ML model to the CT scan to generate a patient-specific (PS) 3D mesh and a segmentation of the CT scan. In block 106, the process applies the boundary definition ML model to the patient-specific 3D mesh and the boundary definition generic 3D mesh to generate a boundary defined patient-specific 3D mesh. In block 107, the process determines the dose of radiation to apply to the cardiac target region. In block 108, the process applies the planning ML model to the boundary defined patient-specific 3D mesh, segmentation and dose information to generate a delivery plan. In block 109, the process completes by instructing execution of the delivery plan.

[0055] FIG. 2 is a block diagram illustrating components of an OAW system in some embodiments. The OAW system 200 is connected to a CT device 210, an ECG device 220, and a SABR device 230. The OAW system includes a control system 201, a mapping system 202, a 3D ML model component 203, a boundary definition ML model component 204, a planning ML model component 205, an execution delivery plan system 206, and an ML training system 207. The OAW system also includes a training dataset store 208 and a model parameter store 209. A controller controls the overall operation of the OAW system. The controller collects the patient's ECG from the ECG device and the patient's CT scan from the CT device. The controller invokes the mapping system to generate a boundary-defined generic 3D mesh from the ECG. The controller uses the 3D ML model component to generate a patient-specific 3D mesh and segmentation based on the CT scan. The controller uses the boundary definition ML model component to generate a boundary-defined patient-specific 3D mesh based on the boundary-defined generic 3D mesh and the patient-specific 3D mesh. Although not shown, the controller can generate a bounded patient-specific CT scan based on the bounded patient-specific 3D mesh. The controller uses the planning ML model component to generate a delivery plan based on the bounded patient-specific 3D mesh, segmentation (e.g., showing avoidance sites), radiation dose information, characteristics of the SABR device (e.g., available delivery angles), etc. The OAW system can employ a planning ML model specific to the type (e.g., manufacturer) of the SABR device. For example, different types of SABR devices may require different types of information in the delivery plan and have different capabilities (e.g., delivery angles). One SABR device may expect an input delivery plan to include only a target dose and calculate the dose for each delivery angle, or receive dose guidance from a technician during the procedure; and another SABR device may expect an input delivery plan that includes the dose for each angle.The ML training system trains various ML models (described above) using the training data in the training data store and creates and stores model parameters (e.g., weights and biases) for each of the ML models in the model parameter store.

[0056] FIG. 3 is a flow diagram illustrating a process for generating a bounded patient-specific 3D mesh component of an OAW system in some embodiments. The generate bounded patient-specific 3D mesh component 300 collects patient data and generates a bounded patient-specific 3D mesh using a mapping system. In block 301, the component retrieves the patient's ECG and 3D images (e.g., CT scan). In block 302, the component retrieves patient characteristics such as cardiac geometry, scar location, past ablation locations, etc. In block 303, the component applies the mapping system to the ECG and optionally the patient characteristics to generate a bounded generic 3D mesh. In block 304, the component generates a feature vector based on features derived from the 3D images and the patient characteristics. In block 305, the component applies a 3D ML model to the feature vector to generate the patient-specific 3D mesh. In block 306, the component generates a feature vector based on features derived from the patient-specific 3D mesh and the bounded generic 3D mesh. In block 307, the component applies the delimited ML model to the feature vector to generate a delimited patient-specific 3D mesh and completes.

[0057] FIG. 4 is a flow diagram illustrating the processing of a mapping system in some embodiments. The mapping system 400 is provided with a patient ECG and patient characteristics and returns a bounded generic 3D mesh. In block 401, the component accesses a library that maps each library electrocardiogram (e.g., VCG) to simulation data used in the simulation to generate the library electrocardiogram. The simulation data may include, for example, a simulated cardiac geometry of the heart used in the simulation and a simulation mesh having vertices corresponding to locations in the heart where values ​​of electrical characteristics are calculated for each simulation interval. In block 402, the component filters library electrocardiogram waveforms associated with simulation characteristics that are not similar to the patient characteristics based on a similarity criterion. For example, the filtering may exclude library electrocardiograms associated with simulation characteristics that are significantly different from the patient characteristics (e.g., scar location). In block 403, the component identifies library electrocardiograms that match the patient's ECG. If the library electrocardiogram is a library VCG, the component may convert the patient ECG to the patient VCG and compare the library VCG to the patient VCG to determine whether the similarity criterion is met. In block 404, the component searches for simulation data associated with a matching library electrocardiogram. In block 405, the component generates a bounded generic 3D mesh with a bounded ROI based on the simulation data. For example, the bounded generic 3D mesh can be generated from the simulated heart geometry or the simulation mesh. Alternatively, the bounded generic 3D mesh can be generated in advance for each simulation and stored in a library mapped to the library myocardial images for that simulation. The component is then complete.

[0058] FIG. 5 is a flow diagram illustrating the processing of the train bounding machine learning model component of the OAW system in some embodiments. The train bounding ML model component 500 trains a bounding ML model to generate a bounded patient-specific 3D mesh from the patient-specific 3D mesh and the bounded generic 3D mesh. The bounding ML model is trained using a training set that includes a patient-specific 3D mesh labeled with the bounded generic 3D mesh and the bounded patient-specific 3D mesh, respectively. In block 501, the component selects the next training set. In decision block 502, if all training sets have already been selected, the component proceeds to block 510, otherwise the component proceeds to block 503. In block 503, the component retrieves an electrocardiogram (ECG or VCG) in the training set. In block 504, the component retrieves a 3D image in the training set. In block 505, the component provides the electrocardiogram to a mapping system, which returns a bounded generic 3D mesh with the source location. In block 506, the component applies the 3DML model to the 3D image to generate a patient-specific 3D mesh with a segmentation of the 3D image. As described above, the 3D ML model can include a segmentation ML sub-model, a labeling ML sub-model, and a 3D sub-model. The segmentation ML sub-model can generate a segmentation of the cardiac and non-cardiac structures in the 3D image. Alternatively, a separate segmentation ML model can be employed to segment the non-cardiac structures in the 3D image. In block 507, the component creates a feature vector that includes the patient-specific 3D mesh and the bounded generic 3D mesh of the training set. In block 508, the component generates labels based on the bounded patient-specific 3D mesh of the training set. In block 509, the component stores the feature vector and labels in a training data store as training data and loops to block 501 to select the next training set.In block 510, the component trains a bounded ML model using the training data in the training data store and stores model parameters (e.g., weights and biases of an activation function) of the bounded ML model in the model parameter store. Other ML models are trained in a similar manner, with the training sets as described above.

[0059] FIG. 6 is a flow diagram illustrating the process of a delivery plan generation component of an OAW system in some embodiments. The delivery plan generation component 600 employs a planning ML model to generate a delivery plan for treating a patient. In block 601, the component receives dose information for radiation for an ablation procedure, such as a maximum dose to a target. In block 602, the component receives a bounded, patient-specific 3D mesh generated for the patient. In block 603, the component receives a segmentation of the patient's heart and non-cardiac structures within the patient's thorax. In block 604, the component applies the planning ML model to the bounded, patient-specific 3D mesh, segmentation, and dose information to generate a delivery plan (DP) for the patient. In block 605, the component provides the delivery plan to a SABR device and completes.

[0060] The following paragraphs describe various embodiments of the OAW system and other system aspects. A system implementation may employ any combination of the embodiments. The processes described below may be performed by a computer system having a processor executing computer-executable instructions stored in a computer-readable storage medium that implements the system.

[0061] In some aspects, the techniques described herein relate to a method executed by one or more computer systems for planning a cardiac stereotactic ablative radiation therapy procedure for a patient, the method including the steps of accessing an arrhythmic electrocardiogram (ECG) of the patient, a 3D image of the thorax collected from the patient, and patient characteristics; applying a mapping system to the arrhythmic ECG and the patient characteristics to generate a bounded generic 3D mesh of the heart having a bounded region of interest (ROI); applying a 3D machine learning (ML) model to the 3D image to generate a patient-specific 3D mesh and labels for the thorax segments; applying the bounded ML model to the patient-specific 3D mesh and the bounded generic 3D mesh to generate a bounded patient-specific 3D mesh; and generating a delivery plan for the patient by: identifying chest segments representing avoidance structures, and applying the planning ML model to the bounded patient-specific 3D mesh, the avoidance structures, and radiation dosage information to generate a stereotactic ablative radiation therapy delivery plan. In some aspects, the techniques described herein relate to a method, where the delivery plan includes a movement of a delivery arm of a stereotactic ablative radiation therapy device, a dose of radiation relative to the direction of the delivery arm, and a shape of the delivery beam relative to the dose. In some aspects, the techniques described herein relate to a method, further including transmitting the delivery plan to a stereotactic ablative radiation therapy device and directing the performance of a cardiac stereotactic ablative radiation therapy procedure according to the delivery plan. In some aspects, the techniques described herein relate to a method, where the first ROI is a source location and the second ROI is scar tissue. In some aspects, the techniques described herein relate to a method, where the mapping system includes a mapping ML model that inputs an arrhythmic ECG and patient characteristics and outputs a bounded generic 3D mesh. In some aspects, the techniques described herein relate to a method, where the avoidance structure is bounded by a specification of a volume and a location within the patient's thorax. In some aspects, the techniques described herein relate to a method, where the ROI is specified by a volume at a location within the bounded patient-specific 3D mesh of a target of the stereotactic ablative radiation therapy procedure.In some embodiments, the technology described herein relates to a method, wherein the 3D image is a computed tomography scan, hi some embodiments, the technology described herein relates to a method, wherein the 3D image is a magnetic resonance imaging scan.

[0062] In some aspects, the technology described herein relates to a method of planning a procedure on a patient's organ, the method including the steps of: applying a mapping system that inputs a patient electrogram representing electrical activity of the patient's organ and patient characteristics of the patient, and outputs a bounded generic three-dimensional (3D) mesh representing the organ based on a generic organ geometry, the one or more ROIs within the bounded generic 3D mesh, the one or more ROIs including a target site of the procedure; acquiring a 3D image of at least a portion of the patient's thorax including the patient's organ; generating a patient-specific 3D mesh representing the patient's organ based on the 3D image, the patient-specific 3D mesh representing the patient's organ geometry of the patient's organ; identifying segments within the 3D image and generating labels indicating segment type for the segments; and applying a bounded machine learning (ML) model to the patient-specific 3D mesh and the bounded generic 3D mesh to generate a bounded patient-specific 3D mesh with one or more ROIs bounded taking into account differences between the generic organ geometry and the patient organ geometry. In some aspects, the techniques described herein relate to a method further comprising generating a patient delivery plan based on the bounded patient-specific 3D mesh, segment labels, and radiation dosage information. In some aspects, the techniques described herein relate to a method in which a mapping system applies a mapping ML model that inputs a patient electrogram and patient characteristics and outputs a bounded generic 3D mesh. In some aspects, the techniques described herein relate to a method in which a mapping system identifies a library electrogram matching the patient electrogram based on a matching criterion from a library of associations between library electrograms and library bounded generic 3D meshes, and outputs a library bounded generic 3D mesh associated with the matching library electrogram. In some aspects, the techniques described herein relate to a method in which a mapping system generates a bounded generic 3D mesh based on a simulated organ geometry for use in simulating electrical activity of the organ based on simulated organ characteristics including the simulated organ geometry.In some aspects, the techniques described herein relate to a method further comprising applying a 3D ML model to the 3D image to generate a patient-specific 3D mesh, identify segments, and generate labels, In some aspects, the techniques described herein relate to a method, wherein the organ is selected from the group consisting of the brain, the digestive system, the heart, and the lungs.

[0063] In some aspects, the techniques described herein relate to a method, executed by one or more computer systems, for generating a three-dimensional (3D) machine learning (ML) model, the method comprising: accessing training data comprising training sets based on 3D images including an organ having an organ geometry and each including features based on features associated with the organ, each including a training set indicating a labeling of segments of the 3D images and indicating a 3D mesh based on the organ geometry, and training a 3D ML model based on the training set, the 3D ML model for inputting features derived from a patient 3D image of an organ of the patient and characteristics associated with the patient's organ, and outputting a patient-specific 3D mesh representing the patient's organ geometry with the labeling of segments. In some aspects, the techniques described herein relate to a method, the 3D ML model includes a segmentation ML sub-model that inputs the 3D images and outputs a segmentation of the 3D images, a labeling ML sub-model that inputs the segmentation of the 3D images and outputs a labeling of the segments, and a 3D ML sub-model that inputs the labeling of the segments and outputs a patient-specific 3D mesh. In some aspects, the technology described herein relates to methods, wherein the organ is the heart. In some aspects, the techniques described herein relate to a method, where at least a portion of the 3D images are collected using a scanning device and include a labeling of segments of the 3D images. In some aspects, the techniques described herein relate to a method, where at least a portion of the 3D images are simulated 3D images, where each simulated 3D image represents a different combination of segment geometry and segment location within a body. In some aspects, the techniques described herein relate to a method, further comprising: accessing a patient 3D image of an organ of the patient and patient characteristics; and applying a 3D ML model to the patient 3D image and patient characteristics to generate a patient-specific 3D mesh of the organ of the patient and a labeling of segments in the patient 3D image.

[0064] In some aspects, the techniques described herein include a method executed by one or more computer systems for generating a planning machine learning (ML) model, the method comprising: accessing training data including training sets, each training set including features derived from a bounded three-dimensional (3D) mesh representing an organ having a bounded target site for a medical procedure and features derived from other structures within the body, the features being labeled with a delivery plan for the medical procedure; and training a planning ML model based on the training sets, the planning ML model for inputting features derived from a bounded patient-specific 3D mesh representing an organ of a patient having a bounded target site and features derived from other structures within the patient's body, and outputting a delivery plan for a medical procedure to treat the target site of the organ of the patient; In some aspects, the techniques described herein relate to a method, wherein the target site is bounded using metadata associated with a bounded 3D mesh. In some aspects, the techniques described herein relate to a method, wherein the features derived from the bounded 3D mesh are 3D images including an organ. In some aspects, the techniques described herein relate to a method, wherein the features are based on dosage information. In some aspects, the techniques described herein relate to a method, wherein the organ is a heart, further comprising acquiring 3D images of a portion of a patient's body including the heart and non-cardiac structures, generating features based on a bounded patient-specific 3D mesh derived from the acquired 3D images bounded with labeling of the target site and segments in the acquired 3D images, and applying a planning ML model to the features to generate a delivery plan for treating the patient.

[0065] In some aspects, the techniques described herein relate to a method performed by one or more computing systems for treating a patient by performing a cardiac stereotactic ablative radiation therapy procedure on the patient's heart, the method including: acquiring an arrhythmic electrocardiogram (ECG) and patient characteristics of the patient, receiving a bounded generic three-dimensional (3D) mesh representing a generic cardiac geometry by inputting the arrhythmic ECG and the patient characteristics into a mapping system that outputs a bounded generic 3D mesh having a bounded region of interest (ROI), acquiring a 3D image of the patient's heart, generating a patient-specific 3D mesh representing the patient's heart based on the 3D image, and generating a bounded patient-specific 3D mesh based on the patient-specific 3D mesh and the bounded generic 3D mesh, where the ROI is bounded to reflect a difference between the patient's cardiac geometry and the generic cardiac geometry. In some aspects, the techniques described herein relate to a method further including transmitting the bounded patient-specific 3D mesh to a stereotactic ablative radiation therapy device. In some aspects, the techniques described herein relate to a method further comprising generating a 3D image corresponding to the demarcated patient-specific 3D mesh and delivering the 3D image to a stereotactic ablative radiation therapy device. In some aspects, the techniques described herein relate to a method further comprising generating a labeling of segments in the 3D image and generating a delivery plan based on the demarcated patient-specific 3D mesh, the labeled segments, and the target dose. In some aspects, the techniques described herein relate to a method, wherein the delivery plan is generated using a planning machine learning model.

[0066] In some aspects, the technology described herein includes one or more computer systems for supporting treatment of an arrhythmia patient, comprising one or more computer readable storage media storing an arrhythmic electrocardiogram acquired from the patient, a 3D image of the patient's heart, and computer executable instructions, the computer executable instructions including: generating a bounded generic 3D mesh representing a generic cardiac geometry by inputting the arrhythmic electrocardiogram into a mapping system that outputs a bounded generic 3D mesh having a bounded region of interest (ROI); generating a patient-specific 3D mesh representing the patient's heart based on the 3D image; and one or more processors for controlling the one or more computer systems to execute one or more of the computer-executable instructions, the one or more computer-readable storage media being for controlling the one or more computer systems to generate a patient-specific 3D mesh based on the patient-specific 3D mesh and the bounded generic 3D mesh, the bounded patient-specific 3D mesh having a ROI bounded based on a difference between the patient's heart geometry and the generic heart geometry. In some aspects, the technology described herein relates to one or more computer systems, wherein the instructions for generating the patient-specific 3D mesh apply a 3D machine learning (ML) model including a segmentation ML sub-model, a labeling ML sub-model, and a 3D ML sub-model. In some aspects, the technology described herein relates to one or more computer systems, wherein the computer-executable instructions include instructions for generating a delivery plan based on the bounded patient-specific 3D mesh, the labeling of segments within the patient's thorax, and the dosage information. In some aspects, the technology described herein relates to one or more computer systems, where the computer-executable instructions include instructions for displaying a 3D graphic of the heart based on a patient-specific 3D mesh.In some aspects, the technology described herein relates to one or more computer systems, where the computer executable instructions include instructions for generating a bounded, patient-specific 3D image based on a bounded, patient-specific 3D mesh.In some aspects, the technology described herein relates to one or more computer systems, where the computer executable instructions include instructions for generating a bounded, patient-specific 3D image based on a bounded, patient-specific 3D mesh and a 3D image.

[0067] Although the subject matter of the present invention has been described in terms of specific structural features and / or acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. [Explanation of symbols]

[0068] Treat 100 patients 101 Obtaining an ECG 102 Acquire 3D images 103 Acquire a trait 104 DM-G 3D Mesh Map (ECG) 105 PS 3D Mesh, Segment = 3D ML (3D Image) 106 DM-PS 3D mesh = DM ML (PS 3D mesh, DM-G 3D mesh) 107 Determine dose information 108 Delivery Plan = PlanML (DM-PS 3D Mesh, Segment, Dose) 109 Execute the delivery plan done Completed

Claims

1. 1. A method executed by one or more computer systems for planning a cardiac stereotactic ablation radiation therapy procedure on a patient, comprising: accessing an arrhythmia electrocardiogram (ECG) of a patient, a 3D image of the thorax acquired from the patient, and patient characteristics; applying a mapping system to the arrhythmic electrocardiogram and patient characteristics to generate a bounded generic 3D mesh of the heart having a bounded region of interest (ROI); applying a 3D machine learning (ML) model to the 3D images to generate a patient-specific 3D mesh and labels for thorax segments; applying a bounded ML model to the patient-specific 3D mesh and the bounded generic 3D mesh to generate a bounded patient-specific 3D mesh; below: identifying thoracic segments representing avoidance structures; and applying a planning ML model to the delimited patient-specific 3D mesh, avoidance structure, and radiation dosage information to generate a stereotactic ablative radiation therapy delivery plan. generating a delivery plan for the patient by A method comprising:

2. 10. The method of claim 1, wherein the delivery plan includes a delivery arm movement of a stereotactic ablative radiation therapy device, a dose of radiation relative to a direction of the delivery arm, and a shape of the delivery beam relative to the dose.

3. 3. The method of claim 2, further comprising transmitting the delivery plan to the stereotactic ablative radiation therapy device and instructing performance of the cardiac stereotactic ablative radiation therapy procedure in accordance with the delivery plan.

4. The method of claim 1 , wherein the first ROI is a source location and the second ROI is scar tissue.

5. The method of claim 1 , wherein the mapping system includes a mapping ML model that inputs the arrhythmic ECG and the patient characteristics and outputs the bounded generic 3D mesh.

6. The method of claim 1 , wherein the avoidance structure is defined by a specification of a volume and a location within the patient's thorax.

7. The method of claim 1 , wherein the ROI is specified by a volume at a location within the demarcated patient-specific 3D mesh of a target of the stereotactic ablative radiation therapy procedure.

8. The method of claim 1 , wherein the 3D image is a computed tomography scan.

9. The method of claim 1 , wherein the 3D image is a magnetic resonance imaging scan.

10. 1. A method for planning a procedure on an organ of a patient, comprising: applying a mapping system to input a patient electrogram representing electrical activity of an organ of the patient and patient characteristics of the patient, and output a bounded generic three-dimensional (3D) mesh representing the organ based on a generic organ geometry, the bounded generic 3D mesh having one or more regions of interest (ROIs) within the bounded generic 3D mesh, the one or more ROIs including a target site of the procedure; acquiring a 3D image of at least a portion of the patient's thorax including the patient's organs; generating a patient-specific 3D mesh representing an organ of the patient based on the 3D image, the patient-specific 3D mesh representing a patient organ geometry of the organ of the patient; identifying segments within the 3D image and generating labels for the segments indicative of their segment types; applying a boundary-delimited machine learning (ML) model to the patient-specific 3D mesh and the boundary-delimited generic 3D mesh to generate a boundary-delimited patient-specific 3D mesh with the one or more ROIs delimited taking into account differences between the generic organ geometry and the patient organ geometry; A method comprising:

11. The method of claim 10, further comprising generating a delivery plan for the patient based on the demarcated patient-specific 3D mesh, the segment labels, and radiation dosage information.

12. The method of claim 10 , wherein the mapping system applies a mapping ML model that inputs the patient electrogram and the patient characteristics and outputs the bounded generic 3D mesh.

13. The method of claim 10, wherein the mapping system identifies a library electrogram that matches the patient electrogram based on a matching criterion from a library of associations between the library electrograms and library bounded generic 3D meshes, and outputs the library bounded generic 3D mesh associated with the matching library electrogram.

14. 11. The method of claim 10, wherein the mapping system generates the bounded generic 3D mesh based on simulated organ properties including a simulated organ geometry for use in simulating electrical activity of the organ.

15. The method of claim 10, further comprising applying a 3D ML model to the 3D images to generate the patient-specific 3D mesh, identify the segments, and generate the labels.

16. 11. The method of claim 10, wherein the organ is selected from the group consisting of the brain, the digestive organs, the heart, and the lungs.

17. 1. A method executed by one or more computer systems for generating a three-dimensional (3D) machine learning (ML) model, comprising: accessing training data including training sets based on 3D images including an organ having an organ geometry and each including a training set including a training set including labels indicative of labeling of segments of the 3D images and indicative of a 3D mesh based on the organ geometry, the training set including a ... training the 3D ML model based on the training set, the 3D ML model for inputting features derived from patient 3D images of the patient's organ and properties related to the patient's organ and for outputting a patient-specific 3D mesh representing the organ geometry of the patient and a labeling of the segments comprising the 3D images; A method comprising:

18. 18. The method of claim 17, wherein the 3D ML model includes a segmentation ML sub-model that inputs the 3D image and outputs a segmentation of the 3D image, a labeling ML sub-model that inputs the segmentation of the 3D image and outputs a labeling of the segments, and a 3D ML sub-model that inputs the labeling of the segments and outputs the patient-specific 3D mesh.

19. 18. The method of claim 17, wherein the organ is the heart.

20. The method of claim 17 , wherein at least a portion of the 3D image is acquired using a scanning device and includes labeling of segments of the 3D image.

21. 20. The method of claim 17, wherein at least some of the 3D images are simulated 3D images, each of the simulated 3D images representing a different combination of segment geometry and segment location within the body.

22. 18. The method of claim 17, further comprising: accessing a patient 3D image and patient characteristics of the organ of a patient; and applying the 3D ML model to the patient 3D image and the patient characteristics to generate a patient-specific 3D mesh of the organ of the patient and a labeling of segments in the patient 3D image.

23. 1. A method executed by one or more computer systems for generating a planned machine learning (ML) model, comprising: accessing training data including training sets, each of the training sets including features derived from a defined three-dimensional (3D) mesh representing an organ having a defined medical procedure target site and features derived from other structures within a body, the features being labeled with a delivery plan for the medical procedure; training the planning ML model based on the training set, the planning ML model for inputting features derived from a demarcated patient-specific 3D mesh representing a patient organ having a demarcated target site and features derived from other structures within the patient's body, and for outputting a delivery plan of a medical procedure for treating the target site of the patient organ; A method comprising:

24. 24. The method of claim 23, wherein the target site is delimited using metadata associated with the delimited 3D mesh.

25. 25. The method of claim 24, wherein the features derived from the bounded 3D mesh are a 3D image including the organ.

26. 24. The method of claim 23, wherein the characteristics are based on dosage information.

27. 25. The method of claim 24, wherein the organ is a heart, further comprising the steps of acquiring 3D images of a portion of a patient's body including the heart and non-cardiac structures, generating features based on a bounded patient-specific 3D mesh derived from the acquired 3D images bounded by labeling of target sites and segments within the acquired 3D images, and applying the planning ML model to the features to generate a delivery plan for treating the patient.

28. 1. A method executed by one or more computing systems for treating a patient by performing a cardiac stereotactic ablation radiation therapy procedure on the patient's heart, comprising the steps of: acquiring an arrhythmic electrocardiogram (ECG) and patient characteristics of the patient; receiving a bounded generic three-dimensional (3D) mesh representing a generic cardiac geometry by inputting the arrhythmic ECG and the patient characteristics into a mapping system that outputs the bounded generic 3D mesh having a bounded region of interest (ROI); acquiring a 3D image of the patient's heart; generating a patient-specific 3D mesh representing the patient's heart based on the 3D images; generating a bounded patient-specific 3D mesh based on the patient-specific 3D mesh and the bounded generic 3D mesh, the bounded patient-specific 3D mesh having the ROI bounded to reflect differences between the patient's cardiac geometry and the generic cardiac geometry; A method comprising:

29. 30. The method of claim 28, further comprising delivering the demarcated patient-specific 3D mesh to a stereotactic ablative radiation therapy device.

30. 30. The method of claim 28, further comprising generating a 3D image corresponding to the demarcated patient-specific 3D mesh, and transmitting the 3D image to a stereotactic ablative radiation therapy device.

31. 30. The method of claim 28, further comprising: generating a labeling of segments in the 3D image; and generating a delivery plan based on the demarcated patient-specific 3D mesh, the labeled segments, and a target dose.

32. 32. The method of claim 31, wherein the delivery plan is generated using a planning machine learning model.

33. 1. One or more computer systems for supporting treatment of an arrhythmia patient, comprising: one or more computer readable storage media storing an arrhythmic electrocardiogram acquired from the patient, a 3D image of the patient's heart, and computer executable instructions, the computer executable instructions comprising: generating a bounded generic 3D mesh representing a generic cardiac geometry by inputting an arrhythmic electrocardiogram into a mapping system that outputs a bounded generic 3D mesh having a bounded region of interest (ROI); generating a patient-specific 3D mesh representing the patient's heart based on the 3D images; generating a bounded patient-specific 3D mesh based on the patient-specific 3D mesh and the bounded generic 3D mesh, the bounded patient-specific 3D mesh having the ROI bounded based on a difference between the patient's cardiac geometry and the generic cardiac geometry; one or more computer readable storage media for controlling the one or more computer systems to one or more processors for controlling the one or more computer systems to execute one or more of the computer-executable instructions; One or more computer systems comprising:

34. 34. The one or more computer systems of claim 33, wherein the instructions to generate the patient-specific 3D mesh apply a 3D machine learning (ML) model including a segmentation ML sub-model, a labeling ML sub-model, and a 3D ML sub-model.

35. 34. The one or more computer systems of claim 33, wherein the computer executable instructions include instructions for generating a delivery plan based on the defined patient-specific 3D mesh, labeling of segments within the patient's thorax, and dosage information.

36. 34. The one or more computer systems of claim 33, wherein the computer executable instructions include instructions for displaying a 3D graphic of the heart based on the patient-specific 3D mesh.

37. 34. The one or more computer systems of claim 33, wherein the computer executable instructions include instructions for generating a delimited patient-specific 3D image based on the delimited patient-specific 3D mesh.

38. 34. The one or more computer systems of claim 33, wherein the computer executable instructions include instructions for generating a demarcated, patient-specific 3D image based on the demarcated, patient-specific 3D mesh and the 3D image.

Images