Method for estimating real-time patient status and real-time tracking of a target using a magnetic resonance linear accelerator

Abstract

Methods of estimating real-time patient states and real-time tracking of a target using a magnetic resonance linear accelerator are disclosed. For example, a method can include generating a dictionary of extended potential patient measurements and corresponding potential patient states with a preliminary motion model. The method can include training a correspondence motion model that associates an input patient measurement with an output patient state using the dictionary with a machine learning technique. The method can include estimating a patient state corresponding to a 2D MR image with a processor using the correspondence motion model. The method can include directing radiotherapy to the target according to the patient state using the MR linear accelerator.

Description

[0001] CLAIM OF PRIORITY

[0002] This application claims the benefit of priority to U.S. Patent Application Serial No. 16 / 170,818, filed October 25, 2018, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] Embodiments of the present disclosure generally relate to medical image and artificial intelligence processing techniques. In particular, the present disclosure relates to utilizing machine learning for real-time patient state estimation. BACKGROUND

[0004] In radiotherapy or radiosurgery, a treatment plan is typically performed based on a patient’s medical images, and the treatment plan requires delineation of target regions and normal critical organs in the medical images. It is a challenge to accurately track various objects (e.g., tumors, healthy tissues, or other aspects of a patient’s anatomy) when the patient is moving (e.g., breathing).

[0005] Current techniques are not able to directly measure a changing patient state in real time. For example, some techniques use 2D imaging, such as 2D kV projections or 2D MRI slices, neither of which can fully track various objects.

[0006] Other techniques can rely on detecting surface information directly or through tracking markers on clothing or boxes fixed to the patient. These techniques assume that surface information is related to the patient’s internal state, which is often inaccurate.

[0007] Other techniques can also rely on implanted markers (e.g., magnetic tracking markers) or X-ray detection using radio-opaque markers. These techniques are invasive and only correspond to limited points in the patient’s body. BRIEF DESCRIPTION OF DRAWINGS

[0008] In the drawings, which are not necessarily drawn to scale, like numerals describe substantially similar components throughout the several views. Like numerals having different letter suffixes

[0009] Figure 1 An exemplary radiotherapy system suitable for performing image patient state estimation processing is shown.

[0010] Figure 2 An exemplary image-guided radiotherapy device is shown.

[0011] Figure 3A partial cutaway view of an exemplary system including a combined radiation therapy system and imaging system, such as a magnetic resonance (MR) imaging system, is shown.

[0012] Figure 4 An exemplary flowchart for estimating patient state using local measurements and a preliminary patient model is shown.

[0013] Figure 5 An exemplary flowchart showing patient state dictionary generation techniques is shown.

[0014] Figure 6 An exemplary regression model machine learning engine for use in estimating patient state is shown.

[0015] Figure 7 A flowchart of exemplary operations for estimating patient state is shown.

[0016] Figure 8 A flowchart of exemplary operations for performing radiation therapy techniques is shown. DETAILED DESCRIPTION

[0017] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration illustrative embodiments in which the application can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the application, and it is to be understood that other embodiments can be utilized and that structural, logical, and electrical changes can be made without departing from the scope of the present application. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present application is defined only by the appended claims and equivalents thereof.

[0018] Image-guided radiation therapy (IGRT) is a technique that uses imaging of the patient in the treatment position immediately prior to radiation. This enables more accurate targeting of dissecting structures, such as organs, tumors, or organs at risk. If the patient is expected to move during treatment, such as motion caused by respiration, which causes quasi-periodic motion of lung tumors, or bladder filling, which causes prostate position drift, additional margins can be placed around the target to contain the expected patient motion. These larger margins come at the cost of high doses to surrounding normal tissue, which can lead to increased side effects.

[0019] IGRT can use computed tomography (CT) imaging, cone beam CT (CBCT), magnetic resonance (MR) imaging, positron emission tomography (PET) imaging, etc. to obtain 3D or 4D images of the patient prior to radiation. For example, a CBCT-enabled linear accelerator (linac) can consist of a kV source / detector fixed to the gantry at a 90 degree angle to the radiation beam, or a MR linac device can consist of a linac integrated directly with a MR scanner.

[0020] Localization of motion during actual radiation therapy delivery (intrafraction motion) can allow for reduction of additional treatment margins that would otherwise be used to account for motion, thus allowing for delivery of higher doses, reduction of side effects, or both. Many IGRT imaging techniques are generally not fast enough to image intrafraction motion. For example, CBCT requires multiple kV images from different angles to reconstruct a full 3D patient image, and 3D MR requires multiple 2D slices or filling of a full 3D k-space, each process potentially taking several minutes to generate a full 3D image.

[0021] In some cases, real-time or quasi-real-time data that would normally be acquired completely prior to generating a 3D IGRT image (as it is collected) can be used to estimate an instantaneous 3D image from an incomplete but fast incoming stream of information, as it is collected. For example, 2D kV projections or 2D MR slices can be used to estimate a full 3D CBCT-like or 3D MR-like image that evolves during treatment as the actual patient motion develops. Although fast, these 2D images themselves provide only a particular perspective of the patient, not a full 3D picture.

[0022] A patient state generator can receive as input local measurements (e.g., 2D images) and generate (e.g., estimate) as output a patient state (e.g., a 3D image). To generate the patient state, the generator can use a single current local measurement, a future (predicted) or past local measurement, or multiple local measurements (e.g., the last 10 measurements). These local measurements can be from a single modality, e.g., an X-ray projection or an MRI slice, or from multiple modalities, e.g., the position of reflective surface markers on the patient surface synchronized with X-ray projections. The patient state can be a 3D image, or a "multi-modal" 3D image, e.g., the patient state can include two or more 3D images that provide different information about the patient state, e.g., a "MR-like" for enhanced tissue contrast, a "CT-like" for high geometric accuracy and density related voxels for dose calculation, or a "functional MR-like" for providing functional information about the patient. The patient state can also include non-imaging information. The patient state includes one or more points of interest (e.g., target positions), contours, surfaces, deformation vector fields, or any information relevant to optimizing patient treatment.

[0023] The above local measurements can be received in a real-time stream of images (e.g., 2D images) acquired from, e.g., a kV imager or an MR imager. The kV imager can generate stereoscopic 2D images (e.g., two X-ray images acquired orthogonally and substantially simultaneously) for the real-time stream. The kV imager can be fixed in the room or coupled to the treatment device (e.g., attached to the gantry). The MR imager can produce 2D MR slices, which can be orthogonal or parallel. The patient state can be generated from the received images or image pairs. For example, at any given time, a patient state can be generated for the last received image from the real-time stream.

[0024] In an example, the patient model can be based on data currently collected in a given segment at a pre-treatment stage (after the patient is setup and before the beam is turned on), data collected from another segment, or data collected using a generic patient anatomy model using a mechanical model with other patients during simulation / planning, or any other information that can help define the patient state from the local measurements. In an example, the patient model is a 4D data set, a pre-treatment acquired, which represents the change in the patient state over a finite time period (e.g., one representative breathing cycle). The patient model can be trained (e.g., using machine learning techniques) to associate input patient measurements (e.g., images or image pairs from the real-time stream) with output patient states, e.g., using a dictionary that will define reconstructed patient measurements with corresponding patient states. The patient model can be warped by a deformation vector field (DVF) as a function of one or more parameters to generate the patient state.

[0025] The patient model in the 4D data set can include a patient state that varies with a single parameter (e.g., a phase in a respiratory cycle). The patient model can be used to establish a time-varying patient state over a representative respiratory cycle that can treat each breath as more or less the same. This simplifies modeling by allowing large chunks of local imaging data to be acquired from different respiratory cycles and assigned to a single representative respiratory cycle. A 3D image can then be reconstructed for each phase “bin.”

[0026] In an example, the patient state can be represented, for example, as a 3D image or a 3D DVF plus a 3D reference image. These can be equivalent in that the 3D DVF and elements of the 3D reference image can be used to acquire (e.g., deform the 3D reference image using the 3D DVF) the 3D image.

[0027] Figure 1 An example radiation therapy system adapted to perform a patient state estimation process is shown. The patient state estimation process is performed to enable the radiation therapy system to provide radiation therapy to a patient based on particular aspects of captured medical imaging data. The radiation therapy system includes an image processing computing system 110 that hosts patient state processing logic 120. The image processing computing system 110 can be connected to a network (not shown), and such a network can be connected to the Internet. For example, the network can connect the image processing computing system 110 with one or more medical information sources (e.g., a radiation information system (RIS), a medical records system (e.g., an electronic medical record (EMR) / electronic health record (EHR) system), an oncology information system (OIS)), one or more image data sources 150, image acquisition devices 170, and treatment devices 180 (e.g., radiation therapy devices). As an example, the image processing computing system 110 can be configured to perform image patient state operations as part of operations to generate and customize a radiation therapy treatment plan to be used by a treatment device 180 by executing instructions or data from the patient state processing logic 120.

[0028] The image processing computing system 110 can include processing circuitry 112, memory 114, storage 116, and other hardware and software operable features such as a user interface 140, a communication interface, and the like. The storage 116 can store computer executable instructions such as an operating system, radiation therapy treatment plans (e.g., original treatment plans, modified treatment plans, and the like), software programs (e.g., radiation therapy treatment planning software and artificial intelligence implementations such as deep learning models, machine learning models, and neural networks), and any other computer executable instructions to be executed by the processing circuitry 112.

[0029] In the example, processing circuitry 112 may include processing devices, such as one or more general-purpose processing devices like a microprocessor, central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc. More specifically, processing circuitry 112 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing circuitry 112 may also be implemented by one or more special-purpose processing devices such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-a-chip (SoCs), etc. As those skilled in the art will understand, in some examples, processing circuitry 112 may be a special-purpose processor rather than a general-purpose processor. Processing circuitry 112 may include one or more known processing devices, such as those from Intel. TM Manufactured Pentium TM Core TM Xeon TM or This series of microprocessors comes from AMD. TM Turion manufactured TM Athlon TM Sempron TM Opteron TM FX TM Phenom TM This refers to any microprocessor from the Sun Microsystems series or any processor from various processors manufactured by Sun Microsystems. The processing circuitry 112 may also include processors from sources such as Nvidia. TM Manufactured series and by Intel TM GMA and Iris manufactured TM series or by AMD TM Radeon manufactured TM The series of GPUs' graphics processing units. The processing circuitry 112 may also include components such as those from Intel... TM Xeon Phi manufactured TMa series of accelerated processing units. The disclosed implementations are not limited to any particular type of processor configured to otherwise meet the computational demands of identifying, analyzing, maintaining, generating, and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. Moreover, the term “processor” can include more than one processor, such as a multi-core design or multiple processors each having a multi-core design. The processing circuitry 112 can execute sequences of computer program instructions stored in memory 114 and accessed from storage 116 to perform various operations, processes, methods that will be explained in more detail below.

[0030] The memory 114 can include read-only memory (ROM), phase-change random access memory (PRAM), static random access memory (SRAM), flash, random access memory (RAM), dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), electrically erasable programmable read-only memory (EEPROM), static storage (e.g., flash memory, static random access memory), and other types of random access memory, cache memory, registers, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs), or other optical storage, magnetic cassettes, other magnetic storage devices, or any other non-transitory medium that can be used to store information including program, data, or computer executable instructions that can be accessed by processing circuitry 112 or by any other type of computer apparatus (e.g., stored in any format).

[0031] The storage 116 can constitute a drive unit including a machine -readable medium on which is stored one or more sets of instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein (including, in various examples, patient state processing logic 120 and user interface 140). The instructions can also reside, completely or at least partially, within the memory 114 and / or processing circuitry 112 during execution thereof by image processing computing system 110, with the memory 114 and the processing circuitry 112 also constituting machine -readable media.

[0032] The memory device 114 and the storage device 116 can constitute non-transitory computer-readable media. For example, the memory device 114 or the storage device 116 can store or load instructions for one or more software applications on a computer-readable medium. The software applications stored or loaded with the memory device 114 or the storage device 116 can include, for example, an operating system for a general-purpose computer system and for software-controlled devices. The image processing computing system 110 can also operate various software programs including software code for implementing the patient state processing logic 120 and the user interface 140. Further, the memory device 114 and the storage device 116 can store or load entire software applications, portions of software applications, or code or data associated with software applications that are executable by the processing circuit 112. In another example, the memory device 114 or the storage device 116 can store, load, or manipulate one or more radiation therapy treatment plans, imaging data, patient state data, dictionary entries, artificial intelligence model data, labels and mapping data, and the like. It can be desirable that software programs can be stored not only on the storage device 116 and the memory 114, but also on removable computer media such as hard drives, computer disks, CD-ROMs, DVDs, HDs, Blu-ray DVDs, USB flash drives, SD cards, memory sticks, or any other suitable media; such software programs can also be delivered or received over a network.

[0033] Although not depicted, the image processing computing system 110 can include communication interfaces, network interface cards, and communication circuitry. Example communication interfaces can include, for example, network adapters, cable connectors, serial connectors, USB connectors, parallel connectors, high-speed data transfer adapters (e.g., fiber optic, USB 3.0, thunderbolt, etc.), wireless network adapters (e.g., IEEE 802.11 / Wi-Fi adapters), telecommunication adapters (e.g., communicating with 3G, 4G / LTE, and 5G networks, etc.), and the like. Such communication interfaces can include one or more digital and / or analog communication devices that allow the machine to communicate with other machines and devices, such as remotely located components, via a network. The network can provide the functionality of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), and the like. For example, the network can be a LAN or a WAN that can include other systems, including additional image processing computing systems or image-based components associated with medical imaging or radiation therapy operations.

[0034] In an example, the image processing computing system 110 can obtain image data 160 from the image data source 150 to be hosted on the storage 116 and the memory 114. In an example, a software program running on the image processing computing system 110 can convert medical images of one format (e.g., MRI) to another format (e.g., CT), for example, by producing a synthetic image such as a pseudo-CT image. In another example, the software program can register or associate a patient medical image (e.g., a CT image or MR image) with a dose distribution (e.g., also represented as an image) of a radiation therapy treatment for that patient, so that the respective image voxels and dose voxels are appropriately associated. In yet another example, the software program can replace a function of a patient image, such as a signed distance function or a processed version of an image that emphasizes some aspect of the image information. Such a function can emphasize edges or differences in voxel texture or other structural aspects. In another example, the software program can visualize, hide, emphasize, or not emphasize certain aspects of anatomical features within a medical image, patient measurements, patient status information, or dose or treatment information. The storage 116 and the memory 114 can store and host data for performing these purposes, including the image data 160, patient data, and other data needed to create and implement radiation therapy treatment plans and associated patient status estimation operations.

[0035] The processing circuit 112 can be communicatively coupled to the memory 114 and the storage 116, and the processing circuit 112 can be configured to execute computer-executable instructions from the memory 114 or the storage 116 stored thereon. The processing circuit 112 can execute the instructions to cause medical images from the image data 160 to be received or fetched in the memory 114 and processed using the patient status processing logic 120. For example, the image processing computing system 110 can receive image data 160 from the image acquisition device 170 or the image data source 150 via a communication interface and a network to be stored or buffered in the storage 116. The processing circuit 112 can also send or update medical images stored in the memory 114 or the storage 116 to another database or data store (e.g., a medical device database) via a communication interface. In some examples, one or more systems can form a distributed computing / simulation environment that cooperatively performs the implementations described herein using a network. Additionally, such a network can be connected to the Internet to communicate with servers and clients remotely residing on the Internet.

[0036] In further examples, processing circuitry 112 can utilize a software program (e.g., a treatment planning software) along with image data 160 and other patient data to create a radiation therapy treatment plan. In examples, image data 160 can include 2D or 3D images such as from CT or MR. Further, processing circuitry 112 can utilize the software program to generate an estimated patient state from measurements and a dictionary of corresponding patient states, for example, utilizing a correspondence motion model and machine learning algorithms (e.g., regression algorithms).

[0037] Further, such a software program can utilize patient state processing logic 120 to implement patient state estimation workflow 130 using techniques further discussed herein. However, processing circuitry 112 can then transmit an executable radiation therapy treatment plan via a communication interface and network to a treatment device 180 where the radiation therapy plan will be used to treat a patient with radiation via the treatment device, consistent with the results of patient state estimation workflow 130. Other outputs and uses of software programs and patient state estimation workflow 130 can result using image processing computing system 110.

[0038] As discussed herein (e.g., with reference to patient state estimation discussed herein), processing circuitry 112 can execute a software program that invokes patient state processing logic 120 to implement functions including generating preliminary motion models, creating dictionaries, training patient state generators using machine learning, patient state estimation, and other aspects of automated processing and artificial intelligence. For example, processing circuitry 112 can execute a software program that uses a system trained using machine learning to estimate patient states.

[0039] In examples, the image data 160 can include one or more MRI images (e.g., 2D MRI, 3D MRI, 2D flow MRI, 4D MRI, 4D volume MRI, 4D cine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), computed tomography (CT) images (e.g., 2D CT, cone beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), positron emission tomography (PET) images, X-ray images, fluoroscopy images, radiotherapy portal images, single photon emission computed tomography (SPECT) images, computer-generated synthetic images (e.g., pseudo-CT images), etc. Additionally, the image data 160 can also include or be associated with medical image processing data such as training images, ground truth images, contour images, and dose images. In examples, the image data 160 can be received from image acquisition devices 170 and stored in one or more of the image data sources 150 (e.g., a picture archiving and communication system (PACS), a vendor neutral archive (VNA), a medical records or information system, a data warehouse, etc.). Thus, the image acquisition devices 170 can include MRI imaging devices, CT imaging devices, PET imaging devices, ultrasound imaging devices, fluoroscopy devices, SPECT imaging devices, integrated linear accelerator and MRI imaging devices, or other medical imaging devices for acquiring medical images of patients. The image data 160 can be received and stored in any data type or any type of format (e.g., in a digital imaging and communications in medicine (DICOM) format) that the image acquisition devices 170 and the image processing computing system 110 can use to perform operations consistent with the disclosed implementations.

[0040] In examples, the image acquisition devices 170 can be integrated with the therapy devices 180 as a single device (e.g., an MRI device combined with a linear accelerator, also referred to as an “MR linear accelerator,” as shown and described below in Figure 3 Such MRI linear accelerators can be used, for example, to accurately determine the location of a target organ or target tumor within a patient’s body to accurately direct radiation therapy to a predetermined target in accordance with a radiation therapy treatment plan. For example, a radiation therapy treatment plan can provide information about a specific radiation dose to be applied to each patient. The radiation therapy treatment plan can also include other radiation therapy information, such as beam angles, dose-histogram-volume information, the number of radiation beams to be used during treatment, the dose of each beam, etc.

[0041] The image processing computing system 110 can communicate with external databases over a network to send / receive a plurality of various types of data related to image processing and radiotherapy operations. For example, the external databases can include machine data, which is information associated with the treatment device 180, the image acquisition device 170, or other machines related to radiotherapy or medical procedures. Machine data information can include beam size, arc placement, beam on and off duration, machine parameters, segments, multi-leaf collimator (MLC) configuration, gantry speed, MRI pulse sequence, etc. The external databases can be storage devices and can be equipped with appropriate database management software programs. Further, such databases or data sources can include multiple devices or systems located in a centralized or distributed manner.

[0042] The image processing computing system 110 can collect and acquire data via a network and communicate with other systems using one or more communication interfaces that can be communicatively coupled to the processing circuitry 112 and the memory 114. For example, the communication interfaces can provide a communication connection between the image processing computing system 110 and radiotherapy system components (e.g., allowing data to be exchanged with external devices). For example, in some examples, the communication interfaces can have appropriate interface circuitry with the output device 142 or the input device 144 to connect to the user interface 140, which can be a hardware keyboard, keypad, or touchscreen through which a user can input information into the radiotherapy system.

[0043] As an example, the output device 142 can include a display device that outputs a representation of the user interface 140 and one or more aspects, visualizations, or representations of medical images. The output device 142 can include one or more display screens that display medical images, interface information, treatment plan parameters (e.g., contours, dose, beam angles, labels, maps, etc.), treatment plans, targets, locate or track targets, patient state estimates (e.g., 3D images), or any information relevant to a user. The input device 144 connected to the user interface 140 can be a keyboard, keypad, touchscreen, or any type of device through which a user can input information to the radiotherapy system. Alternatively, the features of the output device 142, the input device 144, and the user interface 140 can be integrated into a single device such as a smartphone or tablet computer (e.g., Apple iPhone®, Apple iPad®, Lenovo ThinkPad®, Samsung Galaxy®, etc.). Lenovo Samsung etc.

[0044] Further, any and all components of the radiation therapy system can be implemented as virtual machines (e.g., via a VMWare, Hyper-V, or the like virtualization platform). For example, a virtual machine can be software that acts as a hardware. Thus, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that collectively act as hardware. For example, the image processing computing system 110, the image data source 150, or the like can be implemented as a virtual machine or within a cloud-based virtualization environment.

[0045] The patient state processing logic 120 or other software programs can cause the computing system to communicate with the image data source 150 to read images into the memory 114 and storage 116 or to store images or associated data from the memory 114 or storage 116 to the image data source 150 and from the image data source 150 to the memory 114 or storage 116. For example, the image data source 150 can be configured to store and provide a plurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D fluoroscopy images, X-ray images, raw data from MR scans or CT scans, Digital Imaging and Communications in Medicine (DICOM) metadata, etc.) of a set of images from the image data 160 obtained from one or more patients via the image acquisition device 170 hosted by the image data source 150. The image data source 150 or other database can also store data to be used by the patient state processing logic 120 when executing a software program that performs a patient state estimation operation or when creating a radiation therapy treatment plan. Further, various databases can store data produced by the preliminary motion model (e.g., dictionary), the correspondence motion model, or the machine learning model, including network parameters that make up a model learned through a network and resulting prediction data. In connection with performing image patient state estimation as part of a treatment or diagnostic operation, the image processing computing system 110 can thus obtain and / or receive image data 160 (e.g., 2D MRI slice images, CT images, 2D fluoroscopy images, X-ray images, 3D MRI images, 4D MRI images, etc.) from the image data source 150, the image acquisition device 170, the treatment device 180 (e.g., MRI linear accelerator), or other information systems.

[0046] The image acquisition device 170 can be configured to acquire one or more images of a patient's anatomy for a region of interest (e.g., a target organ, a target tumor, or both). Each image, typically a 2D image or slice, can include one or more parameters (e.g., 2D slice thickness, orientation, and location, etc.). In an example, the image acquisition device 170 can acquire 2D slices in any orientation. For example, the orientation of the 2D slices can include a sagittal orientation, a coronal orientation, or an axial orientation. The processing circuitry 112 can adjust one or more parameters, such as the thickness and / or orientation of the 2D slices, to include the target organ and / or the target tumor. In an example, the 2D slices can be determined from information such as a 3D MRI volume. Such 2D slices can be acquired "in real-time" by the image acquisition device 170 while the patient is being treated with radiation therapy, for example, when using the treatment device 180 (where "in real-time" means acquiring data within 10 milliseconds or less). In another example for some applications, real-time can include a time range of (e.g., up to) 200 milliseconds or 300 milliseconds. In an example, real-time can include a time period fast enough to address a clinical problem with the techniques described herein. In this example, real-time can vary depending on target speed, radiation therapy margins, time lag, response time of the treatment device, etc.

[0047] The patient state processing logic 120 in the image processing computing system 110 is depicted as implementing various aspects of a patient state estimation workflow 130 that utilizes model generation and estimation processing operations. In an example, the patient state estimation workflow 130 operated by the patient state processing logic 120 generates and utilizes a preliminary motion model 132 generated from patient data (e.g., from a patient being treated, from a plurality of previous patients, etc.). The preliminary motion model 132 can include a model of a patient in motion (e.g., breathing) generated based on patient measurements and corresponding patient states. The patient state estimation workflow 130 includes creating a dictionary 134 by generating sample (potential) patient measurements and corresponding patient states using the preliminary motion model. The patient state estimation workflow 130 includes training a correspondence motion model based on the dictionary 134 utilizing machine learning 136 (e.g., using regression-based machine learning techniques). The patient state estimation workflow 130 includes estimating a patient state 138 using the correspondence motion model and a current patient measurement (e.g., a 2D image).

[0048] The software program, such as treatment planning software (e.g., Monaco® manufactured by Elekta AB of Stockholm, Sweden) can be utilized to generate the preliminary motion model 132. In an example, the preliminary motion model 132 can be generated from a plurality of patient measurements and corresponding patient states. In an example, the preliminary motion model 132 can be generated from a plurality of 2D images and corresponding patient states. In an example, the preliminary motion model 132 can be generated from a plurality of 3D volumes and corresponding patient states. The image processing computing system 110 uses patient state processing logic 120 and patient state estimation workflow 130 to generate a radiotherapy treatment plan. To generate the radiotherapy treatment plan, the image processing computing system 110 can communicate with an image acquisition device 170 (e.g., a CT device, MRI device, PET device, X-ray device, ultrasound device, etc.) to capture and access images of the patient and delineate the target (e.g., a tumor). In some examples, it may be necessary to delineate one or more organs at risk (OARs), such as healthy tissue surrounding or adjacent to the tumor.

[0049] To depict a target organ or tumor relative to the OAR, medical images of a patient undergoing radiotherapy, such as MRI, CT, PET, fMRI, X-ray, ultrasound, radiotherapy field images, SPECT images, etc., can be non-invasively acquired via image acquisition device 170 to reveal the internal structure of the body part. Based on information from the medical images, a 3D structure of the anatomical site can be obtained. Furthermore, during treatment planning, numerous parameters can be considered to balance, for example, determining where the OAR might be at a given time while the patient is moving (e.g., breathing) to achieve effective treatment of the target tumor (e.g., ensuring the target tumor receives a sufficient radiation dose for effective therapy) with low radiation from the OAR (e.g., the OAR receiving the lowest possible radiation dose). Other parameters that can be considered include: the location of the target organ and tumor, the location of the OAR, and the movement of the target relative to the OAR. For example, a 3D structure can be obtained by outlining the target or the OAR within each 2D layer or slice of the MRI or CT image and combining the outlines in each 2D layer or slice. The contour can be generated manually (e.g., by a physician, dosimeter, or healthcare professional using a program, such as one manufactured by Elekta AB in Stockholm, Sweden). (or automatically generated (e.g., using a program, such as the Atlas-based automatic segmentation software manufactured by Elekta AB in Stockholm, Sweden)). ).

[0050] After the target tumor and OAR have been located and mapped, a dosimeter, physician, or healthcare professional can determine the radiation dose to be applied to the target tumor, as well as any maximum dose that adjacent OARs (e.g., left and right parotid glands, optic nerve, eye, lens, inner ear, spinal cord, brainstem, etc.) can tolerate. After the radiation dose has been determined for each anatomically defined structure (e.g., target tumor, OAR), a process known as inverse planning can be performed to determine one or more treatment planning parameters that will achieve the desired radiation dose distribution. Examples of treatment planning parameters include volume mapping parameters (e.g., those defining the target volume, contour-sensitive structures, etc.), margins around the target tumor and OAR, beam angle selection, collimator settings, and beam-on count. During inverse planning, the physician can define dose constraint parameters that limit how much radiation the OAR can tolerate (e.g., limiting the full dose to the tumor target and zero dose to any OAR; limiting 95% of the dose to the target tumor; limiting the spinal cord, brainstem, and optic nerve structures to ≤45 Gy, ≤55 Gy, and <54 Gy, respectively). The results of the reverse planning can form a radiotherapy treatment plan that can be stored. Some of these treatment parameters may be interrelated. For example, adjusting one parameter (e.g., weighting for different targets, such as increasing the dose to the target tumor) in an attempt to change the treatment plan may affect at least one other parameter, which in turn may lead to the development of different treatment plans. Thus, the image processing computing system 110 can generate a customized radiotherapy treatment plan with these parameters so that the treatment device 180 can provide appropriate radiotherapy treatment to the patient.

[0051] Figure 2 An exemplary image-guided radiotherapy apparatus 202 is shown, which includes a radiation source such as an X-ray source or a linear accelerator, a bed 216, an imaging detector 214, and a radiotherapy output 204. The radiotherapy apparatus 202 can be configured to emit a radiation beam 208 to provide treatment to a patient. The radiotherapy output 204 may include one or more attenuators or collimators, such as multi-leaf collimators (MLCs).

[0052] As an example, a patient can be positioned in area 212, supported by treatment bed 216, to receive treatment according to a radiotherapy plan (e.g., by...). Figure 1The radiotherapy output 204 can be mounted or attached to the gantry 206 or other mechanical support. One or more gantry motors (not shown) can rotate the gantry 206 and the radiotherapy output 204 around the couch 216 as the couch 216 is inserted into the treatment region. In an example, the gantry 206 can continuously rotate around the couch 216 as the couch 216 is inserted into the treatment region. In another example, the gantry 206 can rotate to a predetermined position as the couch 216 is inserted into the treatment region. For example, the gantry 206 can be configured to rotate the therapy output 204 around an axis (“A”). Both the couch 216 and the radiotherapy output 204 can be independently movable to other positions around the patient, for example, movable along a transverse direction (“T”), movable along a lateral direction (“L”), or rotatable around one or more other axes, for example, a transverse axis (denoted as “R”). A controller communicatively connected to one or more actuators (not shown) can control the movement or rotation of the couch 216 to properly position the patient in or out of the radiation beam 208 according to a radiotherapy treatment plan. Because both the couch 216 and the gantry 206 can move independently of each other in multiple degrees of freedom, this allows the patient to be positioned so that the radiation beam 208 can be accurately targeted at a tumor.

[0053] Figure 2 The coordinate system shown in FIG. 1, including axes A, T, and L, can have an origin at the isocenter 210. The isocenter can be defined as the location at which a central axis of the radiotherapy beam 208 intersects the origin of the coordinate axes, for example, to deliver a prescribed radiation dose to a location on or in the patient. Alternatively, the isocenter 210 can be defined as the location at which a central axis of the radiotherapy beam 208 intersects the patient for various rotational positions of the radiotherapy output 204 around axis A as positioned by the gantry 206.

[0054] The gantry 206 can also have an imaging detector 214 attached. The imaging detector 214 is preferably located opposite the radiotherapy source (output 204) and, in an example, can be located within the field of the therapy beam 208.

[0055] Preferably, the imaging detector 214 can be mounted on the gantry 206 opposite the radiotherapy output 204 so as to remain aligned with the therapy beam 208. As the gantry 206 rotates, the imaging detector 214 rotates about the rotation axis. In an example, the imaging detector 214 can be a flat panel detector (e.g., a direct detector or a scintillator detector). In this manner, the imaging detector 214 can be used to monitor the therapy beam 208, or the imaging detector 214 can be used to image the patient's anatomy, such as portal imaging. Control circuitry of the radiotherapy apparatus 202 can be integrated within the radiotherapy system or remote from the radiotherapy system.

[0056] In an illustrative example, one or more of the couch 216, the therapy output 204, or the gantry 206 can be automatically positioned, and the therapy output 204 can establish the therapy beam 208 according to a specified dose for a particular treatment delivery instance. A therapy delivery sequence can be specified according to one or more different orientations or positions of the gantry 206, the couch 216, or the therapy output 204, for example, in accordance with a radiotherapy treatment plan. Therapy delivery can occur sequentially, but can intersect in a desired therapy site on or in the patient, for example, at the isocenter 210. Thereby a prescribed cumulative dose of radiotherapy can be delivered to the therapy site while reducing or avoiding damage to tissue near the therapy site.

[0057] Thus, Figure 2 An example of a radiotherapy apparatus 202 is shown in particular, which is operable to provide radiotherapy treatment to a patient, the radiotherapy apparatus 202 having a configuration in which a radiotherapy output can rotate about a central axis (e.g., axis "A"). Other radiotherapy output configurations can be used. For example, the radiotherapy output can be mounted to a robotic arm or manipulator having multiple degrees of freedom. In another example, the therapy output can be fixed, for example, in a region laterally separated from the patient, and a platform supporting the patient can be used to align a radiotherapy isocenter with a specified target site within the patient. In another example, the radiotherapy apparatus can be a combination of a linear accelerator and an image acquisition apparatus. As will be appreciated by those of ordinary skill in the art, in some examples, the image acquisition apparatus can be an MRI, X-ray, CT, CBCT, helical CT, PET, SPECT, optical tomography, fluorescence imaging, ultrasound imaging, MR linear accelerator, or radiotherapy portal imaging apparatus, among others.

[0058] Figure 3An exemplary radiation therapy system 300 (e.g., referred to in the art as an MR linear accelerator) is depicted, which can include a combination of a radiation therapy device 202 and an imaging system (e.g., a magnetic resonance (MR) imaging system consistent with the disclosed embodiments). As shown, the system 300 can include a bed 310, an image acquisition device 320, and a radiation delivery device 330. The system 300 delivers radiation therapy to a patient in accordance with a radiation therapy treatment plan. In some embodiments, the image acquisition device 320 can correspond to the image acquisition device 170 in FIG. 1. Figure 1

[0059] The bed 310 can support a patient (not shown) during a treatment session. In some implementations, the bed 310 can move along a horizontal translation axis (labeled "I") so that the bed 310 can move a patient lying on the bed 310 into or out of the system 300. The bed 310 can also rotate about a central vertical rotation axis that is transverse to the translation axis. To allow such movement or rotation, the bed 310 can have motors (not shown) that enable the bed to move in various directions and rotate about various axes. A controller (not shown) can control these movements or rotations in order to properly position the patient in accordance with a treatment plan.

[0060] In some embodiments, the image acquisition device 320 can include an MRI machine for acquiring 2D or 3D MRI images of the patient before, during, or after a treatment session. The image acquisition device 320 can include a magnet 321 for generating a main magnetic field for magnetic resonance imaging. Magnetic field lines generated by the operation of the magnet 321 can extend substantially parallel to the central translation axis I. The magnet 321 can include one or more coils whose axes extend parallel to the translation axis I. In some embodiments, one or more of the coils in the magnet 321 can be spaced apart so that a central window 323 of the magnet 321 is free of coils. In other embodiments, the coils in the magnet 321 can be thin enough or have a reduced density so that they are substantially transmissive to radiation of the wavelengths generated by the radiation therapy device 330. The image acquisition device 320 can also include one or more shield coils that can generate magnetic fields of approximately equal magnitude and opposite polarity outside the magnet 321 to cancel or reduce any magnetic field outside the magnet 321. As described below, the radiation source 331 of the radiation therapy device 330 can be positioned in a region where the magnetic field is canceled (at least to first order) or reduced.

[0061] ​The image acquisition apparatus 320 can also include two gradient coils 325 and 326 that can generate a gradient magnetic field superimposed on the main magnetic field. The coils 325 and 326 can generate a gradient in the resulting magnetic field that allows spatial encoding of the protons so that the location of the protons can be determined. The gradient coils 325 and 326 can be positioned with the magnet 321 around a common central axis and can be displaced along the central axis. The displacement can create a gap or window between the coil 325 and the coil 326. In embodiments where the magnet 321 also includes a central window 323 between the coils, the two windows can be aligned with each other.

[0062] Image acquisition is used to track the motion of the tumor. Sometimes, internal surrogates or external surrogates can be used. However, during radiotherapy treatment, implanted seeds can move or be displaced from their initial position. In addition, using surrogates assumes a correlation between the tumor motion and the displacement of the external surrogates. However, there can be a phase shift between the external surrogates and the tumor motion and their positions can often lose correlation over time. It is well known that there can be a mismatch of more than 9 mm between the tumor and the surrogates. In addition, any deformation of the shape of the tumor during tracking is unknown.

[0063] Magnetic resonance imaging (MRI) has the advantage of providing superior soft tissue contrast to visualize the tumor in more detail. Using multiple intra-fraction MR images enables determination of both the shape and the position (e.g., centroid) of the tumor. In addition, MRI images improve any manual contouring performed by a radiation oncologist even when using automatic contouring software (e.g., ) as the high contrast between the tumor target and the background region is provided by the MR images.

[0064] Another advantage of using an MR linear accelerator system is that the treatment beam can be continuously turned on and intra-fraction tracking of the target tumor can be performed thereby. For example, an optical tracking device or a stereoscopic X-ray fluoroscopy system can detect the tumor position at 30 Hz by using a tumor surrogate. With MRI, the imaging acquisition rate is faster (e.g., 3 fps to 6 fps). Thus, the centroid position of the target can be determined and artificial intelligence (e.g., neural network) software can predict the future target position. Another advantage of intra-fraction tracking using an MR linear accelerator is that by being able to predict the future target position, the leaves of a multi-leaf collimator (MLC) will be able to conform to the target contour and its predicted future position. Thus, the frequency of predicting the future tumor position with MRI is the same as the imaging frequency during tracking. Being able to clearly track the motion of the target tumor by using detailed MRI imaging enables delivery of a highly conformal radiation dose to the moving target.

[0065] In some embodiments, the image acquisition device 320 can be an imaging device other than an MRI, such as an X-ray, CT, CBCT, helical CT, PET, SPECT, optical tomography, fluorescence imaging, ultrasound imaging, or radiotherapy portal imaging device, among others. As will be appreciated by one of ordinary skill in the art, the above description of the image acquisition device 320 relates to certain embodiments and is not intended to be limiting.

[0066] The radiotherapy device 330 can include a radiation source 331, such as an X-ray source or a linear accelerator, and a multi-leaf collimator (MLC) 333. The radiotherapy device 330 can be mounted on a gantry 335. One or more gantry motors (not shown) can rotate the gantry 335 around the couch 310 when the couch 310 is inserted into the treatment region. In embodiments, the gantry 335 is capable of continuous rotation around the couch 310 when the couch 310 is inserted into the treatment region. The gantry 335 can also have an attached radiation detector (not shown) that is preferably positioned opposite the radiation source 331, and where the axis of rotation of the gantry 335 is positioned between the radiation source 331 and the detector. In addition, the device 330 can include a control circuit (not shown) for controlling, for example, one or more of the couch 310, the image acquisition device 320, and the radiotherapy device 330. The control circuit of the radiotherapy device 330 can be integrated within the system 300 or remote from it.

[0067] During a radiotherapy treatment session, a patient can be positioned on the couch 310. The system 300 can then move the couch 310 into the treatment region defined by the magnetic coils 321, 325, 326, and the gantry 335. The control circuit can then control the radiation source 331, the MLC 333, and the one or more gantry motors to deliver radiation to the patient through the window between the coil 325 and the coil 326 according to a radiotherapy treatment plan.

[0068] Figure 4 An exemplary flowchart for estimating a patient state is shown. Figure 4 A patient state generator 408 is included for estimating a patient state with a correspondence motion model. The patient state generator 408 utilizes the instantaneous local measurements 402 of the patient and a preliminary motion model 406 to estimate a patient state that is output at block 410. The preliminary motion model 406 is generated with previous measurements 404, which include previous patient states corresponding to the previous measurements 404.

[0069] In practical radiotherapy applications, local measurements (e.g., 2D images or image slices) provide incomplete information about the patient state (e.g., 3D image). For example, a 2D MRI slice is a single section through a 3D representation of the patient, and an X-ray projection is an integration of voxels through a ray line of the 3D representation. Fair information is produced with either image (e.g., 2D images of patient anatomy rather than 3D representation). The patient state generator 408 can utilize the local information and a patient model 406 generated from past measurements and / or offline (pre-treatment) acquisition to estimate a patient state 410.

[0070] The patient model generator 408 can include creation of a low-dimensional patient state representation. In an example, previous measurements are first reconstructed into 4D images. Examples can include 4D CT acquired during a planning phase for generating a treatment plan; 4D CBCT acquired (e.g., by rotating a kV imager around the patient to generate) on a regular linear accelerator immediately before each treatment session in which the patient is in a treatment position; 4D MR acquired on a MR linear accelerator or the like before each treatment session.

[0071] A 4D image can include a series of 3D images of a representative respiratory cycle. For example, for 4D CBCT, multiple x-ray projections are acquired and sorted into a plurality of bins. The sorting can be performed, for example, by directly detecting the diaphragm position in each projection in the image, or using a separate respiratory signal acquired simultaneously with the kV projections, and binning the projections according to the phase or amplitude of the signal. Each bin is then reconstructed using the kV projections assigned to that bin to form a 3D image for each bin. Similar techniques can be used to generate 4D MR images. The 4D images can then be used as an intermediate step to reconstruct a model.

[0072] In an example, a reference phase of the 4D images is selected (e.g., a reference phase used for a treatment plan), and deformable image registration (DIR) is performed between the 3D images of each phase and the 3D image of the reference phase. The reference phase can include advanced treatment information (e.g., GTV, organs at risk, etc.). The output of the DIR process can include a displacement vector field (DVF) that links each phase to the reference phase.

[0073] Such a DVF-based motion model provides a mechanism for deforming a reference patient state (e.g., treatment information as defined on a 3D reference image) to a particular anatomy displayed in each of the other phases of a representative respiratory cycle represented in a 4D dataset.

[0074] To interpolate or extrapolate the preliminary motion model 406 to generate new DVF, one or more principal degrees of freedom of the respiratory motion can be identified using unsupervised dimensionality reduction techniques such as principal component analysis (PCA), independent component analysis (ICA), canonical correlation analysis (CCA), etc. In an example, 2 or 3 degrees of freedom can be sufficient to accurately estimate the patient state. In this example, other degrees of freedom can be ignored or discarded (e.g., when they provide little useful information and are mostly noise). For example, PCA of the DVF motion model can yield a low-dimensional patient motion model that corresponds to a mean DVF and 2 or 3 DVF "eigenmodes" (e.g., weighted inputs representing degrees of freedom). A DVF at any point in the motion cycle can be expressed as a weighted sum of the mean and eigenmodes. For example, the mean DVF can be denoted by DVF0and the eigenmodes can be DVF1and DVF2, which are two full 3D vector fields, and then a DVF at any time during the cycle can be written as DVF = DVF0+ a1*DVF1+ a2*DVF2, where a1and a2are scalar numbers and represent the time variation. In this example, the motion model is simplified to identify a1and a2at a particular time rather than the entire DVF. Once computed, the DVF can be used to warp a reference 3D image to obtain a current 3D image (and can be extended to multiple patient images) representing the patient state 310.

[0075] In some cases, the transition step of reconstructing the 4D image can not be necessary and a low-dimensional state representation can be created directly from the measurements.

[0076] In an example, the advantage of using pre-treatment images is that since the data is acquired shortly before the treatment, it is likely to represent the patient's respiratory degrees of freedom well. In some cases, it can be advantageous to use data from previous 4D images, for example, higher quality images can be available (e.g., using MRI when it is not available during the treatment session, or using CT when only CBCT is available before the treatment), and it can take more time to generate and validate the patient model. In yet another example, data from multiple patients can be used to generate a more robust model, for example, to avoid over-constraining the model.

[0077] Figure 5 An example flowchart showing patient state dictionary generation techniques is shown. Generating a dictionary for use with a machine learning algorithm to output a patient state estimate can use pairs of latent patient states and measurements.

[0078] Techniques for generating the dictionary include operation 502 to receive a measurement or a set of measurements and a corresponding patient state or a corresponding set of patient states. For example, the measurements can include 2D images or other local patient state information or data. The patient states can include 3D or 4D representations of the patient corresponding to the measurements. Thus, the dictionary can include labeled data for training or testing a machine learning algorithm. In an example, the received measurements can include digitally reconstructed radiograph (DRR) images for a CT-based patient model or 2D MRI slices for an MRI-based patient model.

[0079] In an example, generating the dictionary can include not using the 2D images directly as measurements, but instead computing 2D DVFs on the 2D images. For example, a PCA analysis of the 2D DVFs yields several parameters. In this example, the input is 2D PCA of the 2D DVF parameters. In another example, a real-time 2D image is registered to a reference 2D image (e.g., with the same contrast) taken during the same session. This allows using fast, highly parallelizable, deformable image registration techniques such as the demon algorithm to generate the 2D DVF. The demon algorithm can be adapted for parallel implementation in GPUs with real-time performance. In another example, a convolutional neural network (CNN) can be used to estimate the 2D optical flow between two images in real-time to generate the 2D DVF.

[0080] Techniques include operation 504 to generate a set of extended potential measurements and corresponding potential patient states. The potential measurements and potential patient states can be generated by taking the initial actual measurements (e.g., received in operation 502) and corresponding actual patient states and adding noise, interference, or otherwise extrapolating other measurement-patient state pairs that can occur for a particular patient or a plurality of patients. Operation 504 allows a set of labeled data to be generated from even a single actual measurement and patient state pair.

[0081] Techniques include operation 506 to save the set of extended potential measurements and corresponding potential patient states in the dictionary for use with machine learning techniques. In an example, the received measurement or set of measurements and the received corresponding patient state or set of patient states can also be saved in the dictionary for use with machine learning techniques. In an example, the measurements (actual or potential) can be used as input data for the machine learning techniques, where the corresponding patient states (actual or potential) are output from the machine learning techniques, and the correspondence is used as a label for the data.

[0082] In operation 504, an expanded latent measurement can be generated using a low-dimensional patient state representation (e.g., using PCA, ICA, CCA, etc.). The low-dimensional patient state representation can be used to generate possible patient states that can potentially occur during treatment. For example, the reasonable range of coefficients (e.g., a1 and a2 as described in the DVF equation described above in Figure 3

[0083] In an example, to generate a set of expanded latent measurements and corresponding latent patient states, small rigid transformations can be applied to the 3D image for data augmentation for latent patient translations. In an example, the dictionary of latent patient states can be generated from multiple patients, not the specific patient being treated, using acquired 3D images, biomechanical models, etc. In another example, for practical considerations, utilizing the specific patient can be used to limit the data required and ensure the data is relevant to the patient being treated.

[0084] Generating expanded latent measurements and latent patient states using the PCA method can include generating PCA coefficients. To generate realistic training patient states, the coefficients can be randomly drawn from a normal distribution, for example centered around the mean trajectory (e.g., within the 4D dataset of received patient states), where the standard distribution is equal to a percentage (e.g., 10%) of the dynamic range of each coefficient. Next, PCA to DVF reconstruction is performed. The complete DVF can be reconstructed with the randomly generated PCA coefficients (e.g., 2 to 3 coefficients representing the degrees of freedom of the moving patient). The DVF is converted to the original patient state volume by warping the reference volume using the complete DVF. Local measurements are created from the original patient state volume. For CT-based motion models, 2D digitally reconstructed radiographs are computed from the original patient state volume using, for example, the Siddon-Jacobs algorithm. For MRI-based motion models, the 3D volume is resampled to extract 2D MRI slices. Small rigid transformations are applied to the 3D volume for data augmentation to account for small inter-fraction patient position differences.

[0085] ​At the same time, the original patient state volume output and local measurements are used together as training samples to be saved into a dictionary (e.g., the input is the measurement and the output is the patient state). This workflow can be repeated for a large number of training samples, e.g., 1000 samples, or an optimized number of training samples.

[0086] For each generated latent patient state, one or more latent patient measurements are simulated. These are measurements that could potentially have led to the corresponding state. For example, for a state represented by a 3D MRI image, 2D slices from a particular orientation and position (e.g., sagittal) that are expected to be used during treatment can be extracted. For a 3D CBCT image, kV x-ray projections can be simulated for a particular gantry angle, e.g., by ray tracing through the 3D image and integrating voxels along the ray spectrum using the Siddon-Jacobs algorithm. More complex algorithms can be considered, e.g., utilizing Monte Carlo algorithms to simulate realistic 2D kV x-ray images that could result from the effects such as scatter or beam hardening. The imaging characteristics (e.g., slice or gantry angle) can be randomly sampled, or uniformly sampled, or fixed to known values. In some examples, a separate AI algorithm (e.g., a generative adversarial network (GAN)) can be utilized to estimate measurements from patient states, especially when patient measurements cannot be easily computed from the state (e.g., 2D MR slice measurements from 3D density patient state information). In some examples, the dimensionality of the dictionary can be further reduced by performing unsupervised dimensionality reduction (e.g., PCA, ICA, or CCA) on the latent measurements or patient states. In other examples, a demon algorithm (utilizing image registration to a reference image) or a CNN can be used to generate DVF for the latent measurements.

[0087] Figure 6 An example regression model machine learning engine 600 for use in estimating patient states is shown. The machine learning engine 600 utilizes a training engine 602 and an estimation engine 604. The training engine 602 inputs historical transaction information 606 (e.g., patient measurements and corresponding patient states) into a feature determination engine 608. The historical action information 606 can be labeled to indicate the correspondence between the measurements and the patient states.

[0088] The feature determination engine 608 determines one or more features 610 from this historical information 606. Generally, a feature 610 is a set of information inputs and includes information determined to be predictive of a particular outcome. In examples, the features 610 can be determined by a hidden layer. The machine learning algorithm 612 produces a correspondence motion model 620 based on the features 610 and the labels.

[0089] In the estimation engine 604, the current action information 614 (e.g., current patient measurements) can be input into a feature determination engine 616. The feature determination engine 616 can determine features of the current information 614 to estimate a corresponding patient state. In some examples, the feature determination engine 616 and 608 are the same engine. The feature determination engine 616 produces a feature vector 618 that is input into a model 620 to generate one or more standard weights 622. The training engine 602 can operate in an offline manner to train the model 620. However, the estimation engine 604 can be designed to operate in an online manner. It should be noted that the model 620 can be periodically updated via additional training or user feedback (e.g., additional, changed, or deleted measurements or patient states).

[0090] The machine learning algorithm 612 can be selected from many different potential supervised or unsupervised machine learning algorithms. Examples of supervised learning algorithms include artificial neural networks, Bayesian networks, instance-based learning, support vector machines, decision trees (e.g., Iterative Dichotomiser 3, C4.5, Classification and Regression Trees (CART), Chi-squared Automatic Interaction Detector (CHAID), etc.), random forests, linear classifiers, quadratic classifiers, k-nearest neighbors, linear regression, logistic regression, and hidden Markov models. Examples of unsupervised learning algorithms include expectation maximization algorithms, vector quantization, and information bottleneck methods. Unsupervised models can be free of the training engine 602.

[0091] In an example, a regression model is used, and the model 620 is a vector of coefficients corresponding to the learned importance of each feature in the vector of features 610, 618. A regression model is shown in block 624, showing an example linear regression. The machine learning algorithm 612 is trained using the dictionary generated as described herein. The machine learning algorithm 612 is trained as to how the patient measurements correspond to the patient state. In an example, the machine learning algorithm 612 implements a regression problem (e.g., linear, polynomial, regression tree, kernel density estimation, support vector regression, random forest implementation, etc.). The resulting trained parameters limit the patient state generator to a corresponding relationship motion model for the selected machine learning algorithm.

[0092] In the case of a conventional linear accelerator, such training can be performed separately for each possible gantry angle (e.g., in one angular increment) since the X-ray acquisition orientation can be restricted to an orthogonal angle with respect to the treatment beam. In the case of an MR linear accelerator, the physician can be given control over the position or orientation on a 2D acquisition plane. Repeating the cross-validation on the training data with different selections of 2D planes can reveal which 2D planes provide the best surrogate information for a given patient / tumor site.

[0093] In some cases, the model 620 can be updated using patient measurements. In some cases, a computation can be performed to determine if the patient measurements are consistent with the model 620, and if not, to suspend treatment (e.g., for a KDE algorithm, a threshold on variance is used, or to determine if there is sufficient data in the dictionary in the neighborhood of the measurement). When treatment is suspended, a new model 620 can be generated, or the old model 620 can be reused if the measurement (e.g., motion) is like the aberration.

[0094] In some applications, the entire real-time patient image can not be needed, and only its features can be useful. For example, the target centroid can be used to make a geometric correction to a multi-leaf collimator (MLC) or to turn the beam on or off. In this case, a single DVF vector connecting the center of the target in the reference image with the current target can be used, rather than computing the entire 3D DVF and deforming the entire reference image each time, making real-time processing more efficient.

[0095] After the patient state generator has been successfully trained and the patient model 620 is aligned with the patient, the treatment beam is turned on and instantaneous local measurements are acquired at a given frequency. For each received measurement, the process can include normalizing the 2D image of the received measurement to match the contrast of the training images. The patient state generator can use the normalized measurement to infer the model coefficients, and can use the model to reconstruct the DVF. The reference volume and treatment information can be warped using the reconstructed DVF into the current patient state, which can be output or saved.

[0096] In some cases, during treatment, the model can not align well with the patient. This can occur if the patient moves between the 4D image and treatment, if a model from the previous day is used, or if data from other patients is used. The patient model (pre-treatment computation) can then be aligned with the actual patient position by rigidly registering with new patient measurements while the patient is in the treatment position. During this time, a CBCT or MRI is acquired to achieve a coarse model-to-patient alignment. After the CBCT or MRI acquisition, a precise alignment of the patient model can be applied using multiple sample images (e.g., X-ray or 2D MRI slices) to account for bed shifts.

[0097] The contrast difference between the synthetic generated training measurements and actual 2D imaging acquisition can hinder the generator’s ability to infer the 3D patient state. Certain intensity normalization procedures can be used to correct for this issue. For example, local or global linear normalization methods can be used. Other examples can include using a generative adversarial network (GAN) to map the intensities of real images with synthetic images.

[0098] Figure 7A flowchart 700 showing example operations for estimating a patient state is shown. The flowchart 700 includes an optional operation 702 for receiving, e.g., using a processor, patient data including a set of patient measurements and a corresponding patient state. The corresponding patient state can include a 3D or 4D patient image, such as a 3D CT, 3D CBCT, 3D MRI, 3D PET, 3D ultrasound, 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound image. The patient measurements can include 2D MRI slices, MRI k-space data, ID MRI navigator, 2D MRI projections, x-ray 2D projection data, PET data, 2D ultrasound slices, etc. In some cases, the detectors can be arranged to obtain patient measurements simultaneously from multiple views, e.g., with stereo kV imaging, or from multiple coexisting modalities (e.g., kV imaging combined with a surface camera). In one example, the patient data can be generated from a single patient. In another example, the patient data can include data from multiple patients.

[0099] The flowchart 700 includes an operation 704 for identifying, e.g., based on the set of patient measurements and the corresponding patient state, a preliminary motion model of the patient in motion. In an example, the preliminary motion model can be generated based on a 4D data set acquired prior to radiotherapy treatment. The preliminary motion model can be generated from a 4D MR or 4D CBCT acquired during treatment. DVFs between each of the phases of the 4D image and a reference phase can be computed. PCA analysis can be performed on the DVFs. The preliminary motion model can be a 3D DVF parameterized by 2 to 3 scalars plus a reference 3D image. Potential 3D images that can occur during treatment can be generated from the 2 to 3 scalars from which the DVFs can be computed. The DVFs can be used to deform the reference 3D image to compute new 3D images.

[0100] The flowchart 700 includes an operation 706 to generate an expanded dictionary of potential patient measurements and corresponding potential patient states using a motion model. The expanded potential patient measurements can include deformations of 3D or 4D patient images. In an example, the deformations include deformation vector fields (DVF) computed using a deformable registration algorithm. In an example, the expanded potential patient measurements include 2D projection images. The expanded potential patient measurements can be generated using one or more of the following methods: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, using Monte Carlo techniques to simulate X-ray interactions with 3D images, using folded cone convolution techniques, using superposition and convolution techniques, using generative adversarial networks, using convolutional neural networks, using recurrent neural networks, etc. The dictionary can include possible 3D images that can occur during treatment by randomly sampling 2 to 3 scalars, generating 3D DVF from the scalars, and deforming a reference image, resulting in a corresponding potential patient state.

[0101] The expanded potential patient measurements can be generated by computing 2D DVF on 2D input images. In an example, the 2D DVF can be computed by performing PCA analysis of the 2D input images. In another example, the 2D DVF can be computed by registering the 2D input images with a reference 2D image (e.g., taken at or just before the start of radiotherapy treatment) and using a deformable image registration technique. The 2D input images and the reference 2D image can have the same contrast to allow registration. The deformable image registration technique can be a fast, highly parallelizable technique such as the demon algorithm (e.g., implemented in parallel on a GPU with real-time performance). In yet another example, a CNN can be used to generate the 2D DVF to estimate the 2D optical flow between the 2D input images and the 2D reference images. The CNN can run in real-time.

[0102] The corresponding potential patient states can be associated with patient measurements that would result in the corresponding patient states. For example, 2D slices are extracted through 3D images at specific locations or angles, or 2D projections are extracted through images. In an example, the original 2D images can not be used as measurements. Instead, the 2D DVF between the 2D images and the reference 3D images formatted in the corresponding image format can be used with PCA analysis of the resulting 2D DVF. For example, the measurements can be processed versions of the measurements, rather than direct measurements of the patient data. The measurements can be PCA components of the 2D DVF, which can include the expanded potential patient measurements. The pairs of expanded potential patient measurements (PCA of 2D DVF) and corresponding patient states (PCA of 3D DVF) can form the dictionary.

[0103] The flowchart 700 includes an operation 708 to train a correspondence motion model that relates input patient measurements to output patient states using a dictionary with a machine learning technique. The correspondence motion model can include a deformation vector field (DVF) as a function of one or more parameters. In an example, the one or more parameters can be determined by reducing the dimensionality of a preliminary DVF computed between two or more phases of a 4D image and a reference phase. For example, reducing the dimensionality can include using principal component analysis (PCA), independent component analysis (ICA), or canonical correlation analysis (CCA). In an example, the correspondence motion model can be generated with a random forest regression, linear regression, polynomial regression, regression tree, kernel density estimation, support vector regression algorithm, CNN, RNN, etc. The machine learning algorithm can be used to correlate pairs of entries in the dictionary. The algorithm can be used with the measurement inputs to provide the patient states. The measurement inputs can include PCA components of 2D DVF of 2D images and reference images, and the patient states can include 3D DVF.

[0104] In an example, the previous operations occur during a pre-treatment period, and the subsequent operations occur during a treatment period. The flowchart 700 includes an operation 710 to estimate a patient state corresponding to a patient measurement of a patient with the correspondence motion model. The patient state can be saved or output. For example, the patient state can be output for display on a user interface of a display device. In an example, estimating the patient state can include receiving the patient measurement as input to the correspondence motion model, which includes a real-time stream of 2D images. The real-time stream of 2D images can include stereoscopic kV images (e.g., from a kV imager rotating around the patient in the case of a conventional linear accelerator) or pairs of 2D MR slice images (e.g., from an MR linear accelerator). In an example, the stereoscopic kV images can include two x-ray images that are acquired simultaneously or substantially simultaneously (e.g., within a few milliseconds or a few hundred milliseconds) and orthogonal or substantially orthogonal (e.g., within 10 degrees). The kV imager can be fixed in the room, or the kV imager can be fixed to a gantry (e.g., including a linear accelerator). The pair of 2D MR slice images can be orthogonal to each other or parallel to each other. In another example, two kV imagers can be used, e.g., with each at 45 degrees to the treatment beam (and 90 degrees to each other). In this example, the two kV imagers can be used simultaneously, or the two kV imagers can be used in an alternating fashion.

[0105] In an example, images can be acquired from a kV imager or two kV imagers and simultaneously internal ultrasound data is acquired. The ultrasound data can be used to reduce the kV dose by having, for example, less dose or pulses, or to have a lower kV imaging frame rate. These auxiliary data can be included directly in the measurements to compute the patient state, or a separate correspondence model between the kV and auxiliary data streams can be generated and this separate model can be used to relate the auxiliary data to the patient state. For example, a correlation model relating the kV PCA components to parameters extracted from the auxiliary measurement stream can be established and continuously updated, and this correlation model can be utilized to determine the kV PCA components when the auxiliary stream data is acquired.

[0106] During treatment, 2D images can be received. A 2D DVF between the incoming 2D images and the reference 2D images can be computed. A PCA analysis can be performed on the DVF. The result is a real-time "measurement" as used herein. The trained machine learning algorithm can take the measurements as input and compute the PCA components of the 3D DVF from the input measurements. The PCA components are used to generate a 3D DVF which is used to warp the 3D reference images with the 3D DVF to form a current real-time 3D patient image representing the patient at the current time. The patient state can be the 3D image itself, the reference image plus the 3D DVF, etc. (in an example, one can be computed from the other).

[0107] In an example, the operations can include outputting the patient state, such as outputting two or more MR-like 3D images displaying tissue contrast, outputting non-imaging information, outputting a CT-like 3D image, etc.

[0108] Figure 8 A flowchart showing example operations for performing radiation therapy techniques is shown.

[0109] Flowchart 800 includes operation 802 to generate, e.g., as described above with respect to operation 706, a dictionary of extended potential patient measurements and corresponding potential patient states using a motion model. The extended potential patient measurements can be generated by computing 2D DVFs on 2D input images. In an example, the 2D DVFs can be computed by performing a PCA analysis of the 2D input images. In another example, the 2D DVFs can be computed by registering the 2D input images with a reference 2D image (e.g., taken at the beginning of radiotherapy administration or immediately before the beginning of radiotherapy treatment) and utilizing a deformable image registration technique. The 2D input images and the reference 2D image can have the same contrast to allow for registration. The deformable image registration technique can be a fast, highly parallelizable technique such as the demon algorithm (e.g., implemented in parallel on a GPU with real-time performance). In yet another example, a CNN can be used to generate the 2D DVFs to estimate 2D optical flow between the 2D input images and the 2D reference image. The CNN can run in real-time.

[0110] In an example, the extended potential patient measurements can be generated from 4D images including 4D CT, 4D CBCT, 4D MRI, 4D PET, 4D ultrasound images, etc. In an example, the extended potential patient measurements include 2D projection images and are generated by utilizing at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, utilizing Monte Carlo techniques to simulate x-ray interactions with 3D images, utilizing fold cone convolution techniques, utilizing superposition and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, recurrent neural networks, etc.

[0111] Flowchart 800 includes operation 804 to train, e.g., as described above with respect to operation 708, a correspondence motion model that associates input patient measurements with output patient states using the dictionary using a machine learning technique. The correspondence motion model can include a deformation vector field (DVF) as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of a 4D image and a reference phase. The correspondence motion model can be generated using random forest regression, linear regression, polynomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, recurrent neural networks, etc.

[0112] Flowchart 800 includes operation 806 to receive a real-time stream of 2D images from an image acquisition device (e.g., a kV x-ray, MR device, CT device, or other image acquisition device). The real-time stream of 2D images can include stereoscopic kV images (e.g., from a kV imager rotating around a patient in the case of a conventional linear accelerator) or pairs of 2D MR slice images (e.g., from a MR linear accelerator). In another example, the real-time stream of 2D images can include k-space data, low resolution 3D MR images, ID navigators, or other MR information.

[0113] In an example, the stereoscopic kV images can include two x-ray images that are acquired simultaneously or substantially simultaneously (e.g., within a few milliseconds or a few hundred milliseconds). The kV imager can be fixed in the room or the kV imager can be fixed to a gantry (e.g., including a linear accelerator). The pair of 2D MR slice images can be orthogonal to each other or parallel to each other.

[0114] Flowchart 800 includes operation 808 to estimate a patient state corresponding to an image of the real-time stream of 2D images using a correspondence motion model. For example, the patient state can be output as an image (e.g., 3D MR or CT), as non-image information, or as both an image and non-image information. The patient state can include information (e.g., an image or text) describing a patient anatomy (e.g., a tumor or organ of interest), or the patient state can be used to establish a target (e.g., a radiotherapy target) (e.g., on a portion of a tumor).

[0115] Flowchart 800 includes operation 810 to localize a radiotherapy target in a patient using the patient state.

[0116] Flowchart 800 includes operation 812 to track a radiotherapy target of a patient in real-time using the patient state. For example, successive patient states can be output using consecutive images from the real-time stream of 2D images, with the target tracked from one patient state to the next.

[0117] Flowchart 800 includes operation 814 to direct radiotherapy to the target according to the patient state using a treatment device (e.g., a stand-alone treatment device, a device coupled to the image acquisition device (e.g., a MR linear accelerator), etc.). For example, the target can be localized in operation 810 or tracked in operation 812, and radiotherapy can be applied according to the location or tracking. In an example, the location or tracking information can be displayed on a display device, e.g., using a user interface present on the display device.

[0118] Supplemental Description

[0119] The detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the application can be practiced. These embodiments are also referred to as "examples." Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples shown or described herein (or one or more aspects thereof).

[0120] All publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as if each were individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document prevails.

[0121] In this document, the terms "a," "an," "the," and "said" are used to include one or more than one, independent of any other instances or usages of "at least one" or "one or more." In this document, the term "or" is used to mean, and is used in the same sense as "and / or" to indicate a disjunction, unless otherwise indicated.

[0122] In the appended claims, the terms "including," "containing," and "having" are used as the plain-English equivalents of the respective terms "comprising," and "wherein." Also, in the following claims, the terms "first," "second," and "third," etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0123] The application also relates to a computing system adapted, configured or operated to perform the operations herein. This system can be specially reconstituted for the desired purpose, or it can comprise a general-purpose computer selectively activated or reconfigured by a computer program (e.g., instructions, code, etc.) stored in the computer. The order of execution or performance of the operations in the embodiments of the application illustrated and described herein is not essential, unless otherwise specified. That is, it is contemplated that executing or performing the operations in any order is feasible, and embodiments of the application can include additional or fewer operations than those disclosed herein. For example, it is contemplated that a particular operation will be performed or executed prior to, contemporaneously with, or subsequent to one or more other operations, in accordance with aspects of the application.

[0124] In light of the above, it will be seen that the several objects of the application are achieved and other advantageous results attained. Having described aspects of the application in detail, it will be apparent to those skilled in the art that modifications and variations are possible without departing from the scope of aspects of the application defined by the appended claims. Since various changes could be made in the above-described constructions, products, and methods without departing from the scope of aspects of the application, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

[0125] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) can be used in combination with each other. In addition, many modifications could be made to adapt a particular situation or material to the teachings of the application without departing from its scope. The dimensions, types, and examples parameters, functions, and implementations of materials described herein are intended to define parameters of the application, but are by no means limiting implementations, but are exemplary implementations. Many other implementations will be apparent to those skilled in the art upon reviewing the above description. Therefore, the scope of the application should be determined, not by the description of the examples, but by the appended claims, and their full scope of equivalents.

[0126] Further, in the detailed description of embodiments, various features are grouped together in one or more embodiments for the purpose of streamlining the disclosure. This should not be interpreted as intending that the claimed application is necessarily limited to these grouped features. Rather, inventive subject matter can reside in less than all features of a given public disclosure. In addition, the description below assumes a certain amount of knowledge in the art. Therefore, the scope of the application should be determined not with reference to the above description, but with reference to the appended claims, along with the full range of equivalents to which such claims are entitled.

[0127] Each of these non-limiting examples can exist independently or can be combined in various permutations or combinations with one or more of the other examples.

[0128] Example 1 is a method for estimating real-time patient state during radiotherapy treatment, the method comprising: receiving, using a processor, patient data comprising a set of reconstructed patient measurements; identifying, based on the set of reconstructed patient measurements, a preliminary motion model of a patient in motion; generating, with the preliminary motion model, a dictionary of extended potential patient measurements and corresponding potential patient states; training, using a machine learning technique, a correspondence motion model that associates input patient measurements with output patient states using the dictionary; and estimating, using the processor, a patient state corresponding to a patient measurement of the patient using the correspondence motion model.

[0129] In Example 2, the subject matter of Example 1 includes, wherein the corresponding patient state comprises a 3D patient image.

[0130] In Example 3, the subject matter of Example 2 includes, wherein the corresponding potential patient state comprises a deformation of the 3D patient image.

[0131] In Example 4, the subject matter of Example 3 includes, wherein the deformation comprises a deformation vector field (DVF) computed using a deformable registration algorithm.

[0132] In Example 5, the subject matter of Examples 1-4 includes, wherein the patient measurement comprises a 2D MRI slice, MRI k-space data, a ID MRI navigator, a 2D MRI projection, x-ray 2D projection data, PET data, or a 2D ultrasound slice.

[0133] In Example 6, the subject matter of Examples 1-5 includes, wherein the patient data comprises a 4D image.

[0134] In Example 7, the subject matter of Example 6 includes, wherein the 4D image is a 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound image.

[0135] In Example 8, the subject matter of Examples 1-7 includes, wherein the correspondence motion model comprises a deformation vector field (DVF) as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of the 4D image and a reference phase.

[0136] In Example 9, the subject matter of Examples 1-8 includes, wherein the extended potential patient measurements include 2D projection images and are generated by utilizing at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, utilizing Monte Carlo techniques to simulate x-ray interactions with 3D images, utilizing fold cone convolution techniques, utilizing superposition and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

[0137] In Example 10, the subject matter of Examples 1-9 includes, wherein the correspondence motion model is generated utilizing random forest regression, linear regression, polynomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

[0138] In Example 11, the subject matter of Examples 1-10 includes, wherein estimating the patient state corresponding to the patient measurements includes receiving the patient measurements as input to the correspondence motion model, the input including a live stream of 2D images.

[0139] In Example 12, the subject matter of Example 11 includes, wherein the live stream of 2D images includes stereoscopic kV images or paired 2D MR slice images.

[0140] In Example 13, the subject matter of Examples 1-12 includes, outputting the patient state as two or more MR-like 3D images that display tissue contrast.

[0141] In Example 14, the subject matter of Examples 1-13 includes, wherein the patient state includes non-imaging information.

[0142] In Example 15, the subject matter of Examples 1-14 includes, generating a preliminary motion model based on a 4D data set acquired prior to radiation therapy treatment.

[0143] In Example 16, the subject matter of Examples 1-15 includes, generating the reconstructed patient measurements by computing 2D deformation vector fields (DVF) on the 2D input images.

[0144] In Example 17, the subject matter of Examples 1-16 includes, wherein generating the reconstructed patient measurements includes performing an analysis of principal component analysis (PCA) of the 2D input images.

[0145] In Example 18, the subject matter of Examples 1-17 includes, wherein generating the reconstructed patient measurements includes registering the 2D input images with reference 2D images and computing 2D DVF utilizing deformable image registration techniques.

[0146] In Example 19, the subject matter of Examples 1-18 includes, wherein generating the reconstructed patient measurements includes utilizing a convolutional neural network (CNN) to estimate 2D optical flow between 2D input images and 2D reference images to compute 2D DVF.

[0147] Example 20 is a system for estimating a patient state during radiotherapy treatment, the system comprising: a processor coupled to a memory, the memory comprising instructions that, when executed by the processor, cause the processor to perform the following operations: receiving patient data comprising a set of reconstructed patient measurements; identifying a preliminary motion model of a patient in motion based on the set of reconstructed patient measurements; generating, with the preliminary motion model, a dictionary of extended latent patient measurements and corresponding latent patient states; training, with a machine learning technique, a correspondence motion model that associates input patient measurements with output patient states using the dictionary; and estimating, with the correspondence motion model, a patient state corresponding to a patient measurement of the patient.

[0148] Example 21 is a method for estimating real-time patient states during radiotherapy treatment using a magnetic resonance linear accelerator (MR linear accelerator), the method comprising: generating, with a preliminary motion model, a dictionary of extended latent patient measurements and corresponding latent patient states; training, with a machine learning technique, a correspondence motion model that associates input patient measurements with output patient states using the dictionary; receiving, from an image acquisition device, a real-time stream of 2D MR images; estimating, with the correspondence motion model, a patient state corresponding to an image of the real-time stream of 2D MR images using a processor; and directing, using a treatment device coupled to the image acquisition device, radiotherapy to a target according to the patient state.

[0149] In Example 22, the subject matter of Example 21 includes, wherein the extended latent patient measurements comprise deformations of 3D patient images, and wherein the deformations comprise deformation vector fields (DVF) computed with a deformable registration algorithm.

[0150] In Example 23, the subject matter of Examples 21-22 includes, wherein the extended latent patient measurements are generated from 4D images, the 4D images comprising 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images.

[0151] In Example 24, the subject matter of Examples 21-23 includes, wherein the correspondence motion model comprises deformation vector fields (DVF) as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of the 4D images and a reference phase.

[0152] In Example 25, the subject matter of Examples 21-24 includes, wherein the extended potential patient measurements include 2D projection images and are generated by utilizing at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, utilizing Monte Carlo techniques to simulate x-ray interactions with 3D images, utilizing fold cone convolution techniques, utilizing superposition and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

[0153] In Example 26, the subject matter of Examples 21-25 includes, wherein the correspondence motion model is generated utilizing random forest regression, linear regression, polynomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

[0154] In Example 27, the subject matter of Examples 21-26 includes, outputting the patient state as two or more MR-like 3D images that display tissue contrast.

[0155] In Example 28, the subject matter of Examples 21-27 includes, generating the extended potential patient measurements by computing 2D deformation vector fields (DVF) on the 2D input images.

[0156] In Example 29, the subject matter of Examples 21-28 includes, wherein generating the extended potential patient measurements includes performing an analysis of a principal component analysis (PCA) of the 2D input images.

[0157] In Example 30, the subject matter of Examples 21-29 includes, wherein generating the extended potential patient measurements includes registering the 2D input images with reference 2D images and computing 2D DVF utilizing deformable image registration techniques.

[0158] In Example 31, the subject matter of Examples 21-30 includes, wherein generating the extended potential patient measurements includes utilizing a convolutional neural network (CNN) to estimate 2D optical flow between the 2D input images and 2D reference images to compute 2D DVF.

[0159] Example 32 is a method for generating real-time target localization data, the method comprising: generating, utilizing a preliminary motion model, a dictionary of extended potential patient measurements and corresponding potential patient states; training, utilizing a machine learning technique, a correspondence motion model that associates input patient measurements with output patient states using the dictionary; receiving, from an image acquisition device, a real-time stream of 2D images; estimating, utilizing a processor, patient states corresponding to images of the real-time stream of 2D images utilizing the correspondence motion model; localizing, utilizing the patient states, a radiotherapy target within a patient; and outputting, on a display device, a location of the radiotherapy target.

[0160] In Example 33, the subject matter of Example 32 includes, wherein the real-time stream of 2D images comprises stereoscopic kV images or pairs of 2D MR slice images.

[0161] In Example 34, the subject matter of Examples 32-33 includes, wherein the extended potential patient measurements comprise a deformation of a 3D patient image, and wherein the deformation comprises a deformation vector field (DVF) computed with a deformable registration algorithm.

[0162] In Example 35, the subject matter of Examples 32-34 includes, wherein the extended potential patient measurements are generated from a 4D image, the 4D image comprising a 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound image.

[0163] In Example 36, the subject matter of Examples 32-35 includes, wherein the correspondence motion model comprises a deformation vector field (DVF) as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of the 4D image and a reference phase.

[0164] In Example 37, the subject matter of Examples 32-36 includes, wherein the extended potential patient measurements comprise 2D projection images and are generated by utilizing at least one of: extracting 2D slices from a 3D image, ray tracing through a 3D image to generate 2D projection images, simulating x-ray interactions with a 3D image utilizing a Monte Carlo technique, utilizing a collapsed cone convolution technique, utilizing a superposition and convolution technique, utilizing a generative adversarial network, a convolutional neural network, or a recurrent neural network.

[0165] In Example 38, the subject matter of Examples 32-37 includes, wherein the correspondence motion model is generated utilizing a random forest regression, a linear regression, a polynomial regression, a regression tree, a kernel density estimation, a support vector regression algorithm, a convolutional neural network, or a recurrent neural network.

[0166] In Example 39, the subject matter of Examples 32-38 includes, outputting the patient state as two or more MR-like 3D images displaying tissue contrast.

[0167] In Example 40, the subject matter of Examples 32-39 includes, generating the extended potential patient measurements by computing a 2D deformation vector field (DVF) on the 2D input images.

[0168] In Example 41, the subject matter of Examples 32-40 includes, wherein generating the extended potential patient measurements comprises performing an analysis of a principal component analysis (PCA) of the 2D input images.

[0169] In Example 42, the subject matter of Examples 32-41 includes, wherein generating the expanded potential patient measurements comprises registering the 2D input images with reference 2D images and computing 2D DVF with a deformable image registration technique.

[0170] In Example 43, the subject matter of Examples 32-42 includes, wherein generating the expanded potential patient measurements comprises estimating 2D optical flow between 2D input images and 2D reference images with a convolutional neural network (CNN) to compute 2D DVF.

[0171] Example 44 is a method for real-time tracking of a target, the method comprising: generating, with a preliminary motion model, a dictionary of expanded potential patient measurements and corresponding potential patient states; training, with a machine learning technique, a correspondence motion model that associates input patient measurements with output patient states using the dictionary; receiving a real-time stream of 2D images from an image acquisition device; estimating, with a processor, patient states corresponding to images in the real-time stream of 2D images with the correspondence motion model; tracking, in real-time, a radiotherapy target of a patient with the patient states; and outputting tracking information for the radiotherapy target for display on a display device.

[0172] In Example 45, the subject matter of Example 44 includes, wherein the real-time stream of 2D images comprises stereoscopic kV images or paired 2D MR slice images.

[0173] In Example 46, the subject matter of Examples 44-45 includes, wherein the expanded potential patient measurements comprise deformations of 3D patient images, and wherein the deformations comprise deformation vector fields (DVF) computed with a deformable registration algorithm.

[0174] In Example 47, the subject matter of Examples 44-46 includes, wherein the expanded potential patient measurements are generated from 4D images, the 4D images comprising 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images.

[0175] In Example 48, the subject matter of Examples 44-47 includes, wherein the correspondence motion model comprises a deformation vector field (DVF) as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of the 4D images and a reference phase.

[0176] In Example 49, the subject matter of Examples 44-48 includes, wherein the extended potential patient measurements include 2D projection images and are generated by utilizing at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, utilizing Monte Carlo techniques to simulate x-ray interactions with 3D images, utilizing fold cone convolution techniques, utilizing superposition and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

[0177] In Example 50, the subject matter of Examples 44-49 includes, wherein the correspondence motion model is generated utilizing random forest regression, linear regression, polynomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

[0178] In Example 51, the subject matter of Examples 44-50 includes, outputting the patient state as two or more MR-like 3D images that display tissue contrast.

[0179] In Example 52, the subject matter of Examples 44-51 includes, generating the extended potential patient measurements by computing 2D deformation vector fields (DVF) on the 2D input images.

[0180] In Example 53, the subject matter of Examples 44-52 includes, wherein generating the extended potential patient measurements includes performing a principal component analysis (PCA) analysis of the 2D input images.

[0181] In Example 54, the subject matter of Examples 44-53 includes, wherein generating the extended potential patient measurements includes registering the 2D input images with reference 2D images and computing 2D DVF utilizing deformable image registration techniques.

[0182] In Example 55, the subject matter of Examples 44-54 includes, wherein generating the extended potential patient measurements includes utilizing convolutional neural networks (CNNs) to estimate 2D optical flow between the 2D input images and 2D reference images to compute 2D DVF.

[0183] Example 56 is at least one machine readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations for implementing any of Examples 1-55.

[0184] Example 57 is an apparatus comprising means for implementing any of Examples 1-55.

[0185] Example 58 is a system for implementing any of Examples 1-55.

[0186] Example 59 is a method for implementing any of Examples 1-55.

[0187] Method examples described herein can be machine or computer- implemented at least in part. Some examples can include a computer- readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in above examples. An implementation of such methods can include code, such as microcode, assembly language code, a higher-level languages code, or the like. Such code can include computer readable instructions for performing various methods. The code can form portions of computer program products. Further, in an example, the code can be tangibly embodied on one or more volatile or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media can include, but are not limited to, volatile memory, non-volatile memory, magnetic storage, optical storage, and the like.

Claims

1. A method for estimating real-time patient state using a magnetic resonance linear accelerator, the method comprising: generating an extended dictionary of potential patient measurements and corresponding potential patient states using a preliminary motion model, wherein the extended potential patient measurements are simulated and comprise 2D images, 3D images, or 4D images, or deformations of 2D images, 3D images, or 4D images, and wherein the potential patient states comprise 3D or 4D representations corresponding to the extended potential patient measurements; training a correspondence motion model that associates an input patient measurement with an output patient state using the dictionary with a machine learning technique, wherein the correspondence motion model comprises a deformation vector field (DVF) as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of a 4D image and a reference phase; receiving a real-time stream of images from an image acquisition device; and estimating, using a processor, the real-time patient state corresponding to an image of the real-time stream of images using the correspondence motion model.

2. The method of claim 1, wherein, the extended potential patient measurements comprise deformations of 3D patient images, and wherein the deformations comprise deformation vector fields (DVF) computed using a deformable registration algorithm.

3. The method of claim 1, wherein, the extended potential patient measurements are generated from 4D images comprising 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images.

4. The method of claim 1, wherein, the extended potential patient measurements comprise 2D projection images and are generated by using at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, simulating x-ray interactions with 3D images using Monte Carlo techniques, using folded cone convolution techniques, using superposition and convolution techniques, using generative adversarial networks, using convolutional neural networks, or using recurrent neural networks.

5. The method of claim 1, wherein, the correspondence motion model is generated using random forest regression, linear regression, polynomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

6. The method according to claim 1, further comprising: the real-time patient state is output as two or more MR-like 3D images that display tissue contrast.

7. The method of claim 1, further comprising: the extended potential patient measurements are generated by computing 2D deformation vector fields (DVF) on 2D input images.

8. The method according to claim 7, wherein, generating the extended potential patient measurements comprises performing a principal component analysis (PCA) of the 2D input images.

9. The method of claim 8, wherein, generating the extended potential patient measurements comprises i) registering the 2D input images with reference 2D images, ii) computing 2D DFVs using deformable image registration techniques, and iii) estimating 2D optical flow between the 2D input images and reference 2D images using convolutional neural networks (CNNs) to compute the 2D DFVs.

10. The method of any one of claims 1 to 9, wherein, the real-time stream of images comprises 2D MR images, low resolution 3D MR images, or 1D navigators.

11. A system for real-time tracking of a target using a magnetic resonance linear accelerator, the system comprising: a non-transitory computer readable medium configured to store computer executable instructions; and processing circuitry configured to execute the computer executable instructions stored in the non-transitory computer readable medium to implement: generating, with a preliminary motion model, an extended dictionary of potential patient measurements and corresponding potential patient states, wherein the extended potential patient measurements are simulated and comprise 2D images, 3D images or 4D images, or deformations of 2D images, 3D images or 4D images, and wherein the potential patient states comprise 3D or 4D representations corresponding to the extended potential patient measurements; training, with a machine learning technique, a correspondence motion model that associates input patient measurements with output patient states using the dictionary, wherein the correspondence motion model comprises a deformation vector field, DVF, as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of a 4D image and a reference phase; receiving a real-time stream of images from an image acquisition device; estimating, with the correspondence motion model, patient states corresponding to images of the real-time stream of images; generating, with the estimated patient states, tracking information for a target; and outputting the tracking information for the target; and a display device configured to display the tracking information for the target output by the processing circuitry.

12. The system of claim 11, wherein, the extended potential patient measurements comprise deformations of 3D patient images, and wherein the deformations comprise deformation vector fields, DVF, computed with a deformable registration algorithm.

13. The system of claim 11, wherein, the extended potential patient measurements are generated from 4D images comprising 4D CT, 4D CBCT, 4D MRI, 4D PET or 4D ultrasound images.

14. The system of claim 11, wherein, the extended potential patient measurements comprise 2D projection images, and the processing circuitry is configured to generate the extended potential patient measurements by utilizing at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, simulating x-ray interactions with 3D images with a Monte Carlo technique, with a folded cone convolution technique, with a superposition and convolution technique, with a generative adversarial network, with a convolutional neural network or with a recurrent neural network.

15. The system of claim 11, wherein, the processing circuitry is configured to generate the correspondence motion model by utilizing a random forest regression, a linear regression, a polynomial regression, a regression tree, a kernel density estimation, a support vector regression algorithm, a convolutional neural network or a recurrent neural network.

16. The system of claim 11, wherein, the processing circuitry is configured to output the estimated patient states as two or more MR-like 3D images displaying tissue contrast.

17. The system of claim 11, wherein, the processing circuitry is configured to generate the extended potential patient measurements by: i) computing 2D deformation vector fields, DVF, on 2D input images, and ii) performing a principal component analysis, PCA, analysis of the 2D input images.

18. The system of any one of claims 11 to 17, wherein, the real-time stream of images comprises 2D MR images, low resolution 3D MR images or 1D navigators.

Images