Machine learning method for real-time patient motion monitoring

Abstract

Systems and techniques can be used to estimate patient state during radiotherapy treatment. For example, a method can include generating, with a preliminary motion model, an extended dictionary of potential patient measurements and corresponding potential patient states. The method can include training, with a machine learning technique, a correspondence motion model that associates input patient measurements with output patient states using the dictionary. The method can include estimating, with a processor, a patient state corresponding to an input image using the correspondence motion model.

Description

[0001] Priority requirements

[0002] This application claims the benefit of priority to U.S. Patent Application Serial No. 16 / 170,807, filed October 25, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The implementation of this disclosure generally relates to medical image and artificial intelligence processing techniques. In particular, this disclosure relates to utilizing machine learning for real-time patient status estimation. Background Technology

[0004] In radiotherapy or radiosurgery, treatment planning is typically based on medical images of the patient, and this planning requires depictions of the target area and normal key organs within those images. Accurately tracking various objects (e.g., tumors, healthy tissue, or other aspects of the patient's anatomical structures) while the patient is moving (e.g., breathing) is a challenge.

[0005] Current technologies cannot directly measure changing patient conditions in real time. For example, some technologies use 2D imaging such as 2D kV projection or 2D MRI slices, neither of which can fully track a wide range of objects.

[0006] Other technologies may rely on detecting surface information directly or by tracking markers on clothing or boxes attached to the patient. These technologies assume that surface information is correlated with the patient's internal state, which is often inaccurate.

[0007] Other techniques may rely on implanted markers (e.g., magnetic tracking markers) or X-ray detection using radiopaque markers. These techniques are invasive and only correspond to a limited number of points within the patient's body. Attached Figure Description

[0008] In the accompanying drawings, which are not necessarily drawn to scale, the same reference numerals describe substantially similar parts throughout several views. The same reference numerals with different letter suffixes indicate different instances of substantially similar parts. The drawings generally illustrate the various embodiments discussed in this document by way of example rather than limitation.

[0009] Figure 1 An exemplary radiotherapy system suitable for performing image-based patient status estimation processing is shown.

[0010] Figure 2 An exemplary image-guided radiotherapy device is shown.

[0011] Figure 3A partial cross-sectional view of an exemplary system including a combined radiotherapy system and an imaging system such as a magnetic resonance (MR) imaging system is shown.

[0012] Figure 4 An exemplary flowchart is shown for estimating patient status using local measurements and a preliminary patient model.

[0013] Figure 5 An exemplary flowchart illustrating a patient status dictionary generation technique is shown.

[0014] Figure 6 An exemplary regression model machine learning engine is shown for use in estimating patient status.

[0015] Figure 7 A flowchart of an exemplary operation for estimating a patient's condition is shown.

[0016] Figure 8 A flowchart illustrating an exemplary operation for performing radiotherapy techniques is shown. Detailed Implementation

[0017] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and illustrate illustrative embodiments through which the invention can be practiced. These embodiments, also referred to herein as “examples,” are described in sufficient detail to enable those skilled in the art to practice the invention, and it should be understood that embodiments may be combined or other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. Therefore, the following detailed description is not restrictive, and the scope of the invention is defined by the appended claims and their equivalents.

[0018] Image-guided radiotherapy (IGRT) is a technique that uses an image of the patient in the treatment position immediately preceding radiation. This allows for more precise targeting of dissected structures, such as organs, tumors, or organs at risk. If patient movement is anticipated during treatment, such as movement caused by breathing (which causes quasi-periodic movement in lung tumors) or bladder distension causing prostate displacement, additional margin can be placed around the target to contain the desired patient movement. These larger margins come at the cost of a higher dose to surrounding normal tissue, which can lead to increased side effects.

[0019] IGRT can obtain 3D or 4D images of the patient before radiation using computed tomography (CT), cone-beam computed tomography (CBCT), magnetic resonance (MR) imaging, positron emission tomography (PET) imaging, etc. For example, a CBCT-enabled linear accelerator (linear accelerator) can consist of a kV source / detector fixed to the gantry at a 90-degree angle to the radiation beam, or an MR linear accelerator device can consist of a linear accelerator directly integrated with an MR scanner.

[0020] Localizing motion during actual radiotherapy delivery (infrafractional motion) allows for a reduction in additional treatment margin that would otherwise be used for motion, thus allowing for the delivery of higher doses, reduced side effects, or both. Many IGRT imaging techniques are typically not fast enough to image intrafractional motion. For example, CBCT requires multiple kV images from different angles to reconstruct a complete 3D patient image, and 3D MR requires multiple 2D slices or filling of the complete 3D k-space, each process potentially taking several minutes to generate a complete 3D image.

[0021] In some cases, transient 3D images can be estimated at a faster refresh rate using real-time or near-real-time data (as it was collected) that is typically acquired fully before generating 3D IGRT images, based on an incomplete but rapid inbound stream of information. For example, 2D kV projections or 2D MR slices can be used to estimate complete 3D BCT-like or 3D MR-like images that develop with actual patient movement during treatment. Although fast, these 2D images themselves only provide a specific viewpoint of the patient, rather than a complete 3D picture.

[0022] A patient state generator can receive local measurements (e.g., 2D images) as input and generate (e.g., estimate) a patient state (e.g., a 3D image) as output. To generate a patient state, the generator can use a single current local measurement, a future (predicted) or past local measurement, or multiple local measurements (e.g., the last 10 measurements). These local measurements can come from a single modality, such as an X-ray projection or MRI slice, or from multiple modalities, such as the location of reflective surface markers on the patient surface synchronized with an X-ray projection. The patient state can be a 3D image, or a “multimodal” 3D image. For example, a patient state can include two or more 3D images providing different information about the patient state, such as an “MR-like” image for enhancing tissue contrast, a “CT-like” image for high geometric accuracy and dose calculation using density-related voxels, or a “functional MR-like” image for providing functional information about the patient. The patient state can also include non-imaging information. The patient state includes one or more points of interest (e.g., target locations), contours, surfaces, deformation vector fields, or any information relevant to optimizing patient treatment.

[0023] The aforementioned local measurements can be received in a real-time image stream (e.g., 2D images) acquired from, for example, a kV imager or an MR imager. The kV imager can generate stereoscopic 2D images of the real-time stream (e.g., two orthogonal and substantially simultaneous X-ray images). The kV imager can be fixed in a room or coupled to a treatment device (e.g., attached to a gantry). The MR imager can produce 2D MR slices, which can be orthogonal or parallel. Patient states can be generated based on the received images or image pairs. For example, at any given moment, a patient state can be generated for the last received image from the real-time stream.

[0024] In the example, the patient model can be based on data currently collected in a given segment during the pre-treatment phase (after patient placement and before bundle opening), data collected from another segment, data collected during simulation / planning using a mechanical model with other patients utilizing a common patient profiling structure, or any other information collected that can help define the patient state based on local measurements. In the example, the patient model is a 4D dataset, the acquired pre-treatment, representing changes in the patient state over a finite time period (e.g., a representative respiratory cycle). The patient model can be trained (e.g., using machine learning techniques) to associate input patient measurements (e.g., images or image pairs from a real-time stream) with output patient states, for example, using a dictionary that defines the constructed patient measurements with corresponding patient states. The patient model can be warped using a deformation vector field (DVF) as a function of one or more parameters to generate patient states.

[0025] Patient models in a 4D dataset can include patient states that vary with a single parameter (e.g., a phase in the respiratory cycle). These patient models can be used to establish time-varying patient states within a representative respiratory cycle in which each breath can be considered more or less the same. This simplifies modeling by allowing large chunks of localized imaging data from different respiratory cycles to be assigned to a single representative respiratory cycle. 3D images can then be reconstructed for each phase “bin.”

[0026] In the example, the patient's state can be represented, for example, as a 3D image or a 3D DVF plus a 3D reference image. These may be equivalent because elements of both the 3D DVF and the 3D reference image can be used to obtain (e.g., by deforming the 3D reference image using the 3D DVF).

[0027] Figure 1 An exemplary radiotherapy system suitable for performing patient state estimation processing is illustrated. This patient state estimation processing is performed to enable the radiotherapy system to deliver radiotherapy to a patient based on specific aspects of captured medical imaging data. The radiotherapy system includes an image processing computing system 110 that hosts patient state processing logic 120. The image processing computing system 110 can be connected to a network (not shown), and such a network can be connected to the Internet. For example, the network can connect the image processing computing system 110 to one or more medical information sources (e.g., radiation information systems (RIS), medical record systems (e.g., electronic medical record (EMR) / electronic health record (EHR) systems), oncology information systems (OIS)), one or more image data sources 150, image acquisition devices 170, and treatment devices 180 (e.g., radiotherapy devices). As an example, the image processing computing system 110 can be configured to perform image patient state operations as part of generating and customizing a radiotherapy treatment plan to be used by the treatment device 180 by executing instructions or data from the patient state processing logic 120.

[0028] The image processing computing system 110 may include a processing circuit system 112, a memory 114, a storage device 116, and other hardware and software operable features such as a user interface 140 and a communication interface. The storage device 116 may store computer-executable instructions such as an operating system, radiotherapy treatment plans (e.g., original treatment plans, modified treatment plans, etc.), software programs (e.g., radiotherapy treatment planning software and artificial intelligence implementations such as deep learning models, machine learning models, and neural networks), and any other computer-executable instructions to be executed by the processing circuitry 112.

[0029] In the example, processing circuitry 112 may include processing devices, such as one or more general-purpose processing devices like a microprocessor, central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc. More specifically, processing circuitry 112 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing circuitry 112 may also be implemented by one or more special-purpose processing devices such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-a-chip (SoCs), etc. As those skilled in the art will understand, in some examples, processing circuitry 112 may be a special-purpose processor rather than a general-purpose processor. Processing circuitry 112 may include one or more known processing devices, such as those from Intel. TM Manufactured Pentium TM Core TM Xeon TM or This series of microprocessors comes from AMD. TM Turion manufactured TM Athlon TM Sempron TM Opteron TM FX TM Phenom TM This refers to any microprocessor from the Sun Microsystems series or any processor from various processors manufactured by Sun Microsystems. The processing circuitry 112 may also include processors from sources such as Nvidia. TM Manufactured series and by Intel TM GMA and Iris manufactured TM series or by AMD TM Radeon manufactured TM The series of GPUs' graphics processing units. The processing circuitry 112 may also include components such as those from Intel... TM XeonPhi manufactured TMThis is a series of accelerated processing units. The disclosed embodiments are not limited to any type of processor otherwise configured to meet computational needs for identifying, analyzing, maintaining, generating, and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. Furthermore, the term "processor" can include more than one processor, for example, a multi-core design or multiple processors each having a multi-core design. Processing circuitry 112 can execute a sequence of computer program instructions stored in memory 114 and accessed from storage device 116 to perform various operations, processes, and methods, which will be described in more detail below.

[0030] Memory 114 may include read-only memory (ROM), phase-change random access memory (PRAM), static random access memory (SRAM), flash memory, random access memory (RAM), dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), electrically erasable programmable read-only memory (EEPROM), static memory (e.g., flash memory, flash disk, static random access memory) and other types of random access memory, cache memory, registers, compressed optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage devices, magnetic tape, other magnetic storage devices, or any other non-transitory medium that can be used to store information including images, data, or computer-executable instructions (e.g., stored in any format) accessible by processing circuitry 112 or any other type of computer device. For example, computer program instructions may be accessed by processing circuitry 112, may be read from ROM or any other suitable memory location, and may be loaded into RAM for execution by processing circuitry 112.

[0031] Storage device 116 may constitute a drive unit including a machine-readable medium on which one or more sets of instructions and data structures (e.g., software) implemented or utilized by any one or more of the methods or functions described herein are stored (in various examples, including patient status processing logic 120 and user interface 140). During the execution of instructions by image processing computing system 110, the instructions may also reside wholly or at least partially in memory 114 and / or processing circuitry 112, wherein memory 114 and processing circuitry 112 also constitute machine-readable media.

[0032] Memory device 114 and storage device 116 can constitute a non-transitory computer-readable medium. For example, memory device 114 or storage device 116 can store or load instructions for one or more software applications on a computer-readable medium. Software applications stored or loaded using memory device 114 or storage device 116 can include, for example, operating systems for general-purpose computer systems and for software control devices. Image processing computing system 110 can also operate various software programs including software code for implementing patient status processing logic 120 and user interface 140. Furthermore, memory device 114 and storage device 116 can store or load the entire software application, a portion of the software application, or code or data associated with the software application that can be executed by processing circuitry 112. In another example, memory device 114 or storage device 116 can store, load, or manipulate one or more radiotherapy treatment plans, imaging data, patient status data, dictionary entries, artificial intelligence model data, labeling and mapping data, etc. It is conceivable that not only can the software program be stored on storage device 116 and memory 114, but also on removable computer media such as hard disk drive, computer disk, CD-ROM, DVD, HD, Blu-ray DVD, USB flash drive, SD card, memory stick or any other suitable media; such software programs can also be transmitted or received via a network.

[0033] Although not depicted, the image processing computing system 110 may include communication interfaces, network interface cards, and communication circuitry. Example communication interfaces may include, for example, network adapters, cable connectors, serial connectors, USB connectors, parallel connectors, high-speed data transmission adapters (e.g., fiber optic, USB 3.0, Thunderbolt, etc.), wireless network adapters (e.g., IEEE 802.11 / Wi-Fi adapters), telecommunications adapters (e.g., for communicating with 3G, 4G / LTE, and 5G networks, etc.). Such communication interfaces may include one or more digital and / or analog communication devices that allow the machine to communicate with other machines and devices, such as remote components, via a network. The network may provide the functionality of a local area network (LAN), wireless network, cloud computing environment (e.g., Software as a Service, Platform as a Service, Infrastructure as a Service, etc.), client-server, wide area network (WAN), etc. For example, the network may be a LAN or WAN that may include other systems, including additional image processing computing systems or image-based components associated with medical imaging or radiotherapy operations.

[0034] In one example, the image processing computing system 110 may obtain image data 160 from image data source 150 for hosting on storage device 116 and memory 114. In another example, a software program running on the image processing computing system 110 may, for example, convert a medical image of one format (e.g., MRI) into another format (e.g., CT) by generating a synthetic image such as a pseudo-CT image. In yet another example, the software program may register or correlate a patient's medical image (e.g., CT or MR image) with the dose distribution of the patient's radiotherapy treatment (e.g., also represented as an image), thereby appropriately associating the corresponding image voxels and dose voxels. In yet another example, the software program may substitute for functions of the patient image, such as a signed distance function or processed version of the image that emphasizes certain aspects of the image information. Such a function might emphasize edges or differences in voxel texture or other structural aspects. In yet another example, the software program may visualize, hide, emphasize, or de-emphasize certain aspects of anatomical features, patient measurements, patient status information, or dose or treatment information within a medical image. Storage device 116 and memory 114 can store and host data used to perform these purposes, including image data 160, patient data, and other data required to create and implement radiotherapy treatment plans and associated patient status estimation operations.

[0035] Processing circuitry 112 may be communicatively coupled to memory 114 and storage device 116, and processing circuitry 112 may be configured to execute computer-executable instructions stored thereon from memory 114 or storage device 116. Processing circuitry 112 may execute instructions to cause medical images from image data 160 to be received or acquired in memory 114 and processed using patient state processing logic 120. For example, image processing computing system 110 may receive image data 160 from image acquisition device 170 or image data source 150 via a communication interface and network for storage or caching in storage device 116. Processing circuitry 112 may also send or update medical images stored in memory 114 or storage device 116 to another database or data store (e.g., a medical device database) via a communication interface. In some examples, one or more systems may form a distributed computing / simulation environment that uses a network to collaboratively perform the implementations described herein. Additionally, such a network may be connected to the Internet to communicate with remote servers and clients residing on the Internet.

[0036] In another example, processing circuitry 112 may utilize software programs (e.g., treatment planning software) along with image data 160 and other patient data to create a radiotherapy treatment plan. In this example, image data 160 may include 2D or 3D images such as those from CT or MR scans. Furthermore, processing circuitry 112 may utilize software programs to generate estimated patient states based on a dictionary of measurements and corresponding patient states, for example, using a correspondence motion model and machine learning algorithms (e.g., regression algorithms).

[0037] Furthermore, this software program can utilize patient status processing logic 120 to implement patient status estimation workflow 130 using techniques further discussed herein. However, processing circuitry 112 can then send an executable radiotherapy treatment plan to treatment device 180 via a communication interface and network, where the radiotherapy plan will be used to treat the patient with radiation via the treatment device, consistent with the results of patient status estimation workflow 130. Further outputs and uses of the software program and patient status estimation workflow 130 can be generated using image processing computing system 110.

[0038] As discussed herein (e.g., referring to patient state estimation discussed herein), processing circuitry 112 can execute software programs that invoke patient state processing logic 120 to implement functions including generating a preliminary motion model, creating a dictionary, using machine learning to train a patient state generator, patient state estimation, and other aspects of automated processing and artificial intelligence. For example, processing circuitry 112 can execute software programs that use a system trained with machine learning to estimate patient states.

[0039] In this example, image data 160 may include one or more MRI images (e.g., 2D MRI, 3D MRI, 2D flow cytometry MRI, 4D MRI, 4D volumetric MRI, 4D imaging MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), computed tomography (CT) images (e.g., 2D CT, cone-beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), positron emission tomography (PET) images, X-ray images, fluorescence microscopy images, radiotherapy portal images, single-photon emission computed tomography (SPECT) images, computer-generated composite images (e.g., pseudo-CT images), etc. Furthermore, image data 160 may also include medical image processing data such as training images, ground truth images, contour images, and dose images, or be associated with medical image processing data such as training images, ground truth images, contour images, and dose images. In this example, image data 160 can be received from image acquisition device 170 and stored in one or more image data sources 150 (e.g., Picture Archiving and Communication System (PACS), Vendor Neutral Archive (VNA), medical record or information system, data warehouse, etc.). Therefore, image acquisition device 170 may include an MRI imaging device, CT imaging device, PET imaging device, ultrasound imaging device, fluorescence microscope device, SPECT imaging device, integrated linear accelerator and MRI imaging device, or other medical imaging device for acquiring medical images of a patient. Image data 160 can be received and stored in any data type or any type of format (e.g., Medical Digital Imaging and Communication (DICOM) format) that can be used by image acquisition device 170 and image processing computing system 110 to perform operations consistent with the disclosed embodiments.

[0040] In the example, the image acquisition device 170 can be integrated with the treatment device 180 into a single device (e.g., as shown below). Figure 3 The MRI apparatus shown and described in combination with a linear accelerator is also referred to as an "MR linear accelerator". Such an MRI linear accelerator can be used, for example, to accurately determine the location of a target organ or target tumor within a patient's body, so as to precisely guide radiotherapy to the predetermined target according to a radiotherapy treatment plan. For example, a radiotherapy treatment plan can provide information about the specific radiation dose to be applied to each patient. A radiotherapy treatment plan may also include other radiotherapy information, such as beam angle, dose-histogram-volume information, the number of radiation beams to be used during treatment, the dose per beam, etc.

[0041] The image processing computing system 110 can communicate with an external database via a network to send / receive various types of data related to image processing and radiotherapy operations. For example, the external database may include machine data, which is information associated with the treatment device 180, the image acquisition device 170, or other machines related to radiotherapy or medical procedures. Machine data information may include beam size, arc placement, beam-on and beam-off durations, machine parameters, segments, multi-leaf collimator (MLC) configuration, gantry speed, MRI pulse sequences, etc. The external database may be a storage device and may be equipped with appropriate database management software programs. Furthermore, such a database or data source may include multiple devices or systems located in a centralized or distributed manner.

[0042] The image processing computing system 110 can use one or more communication interfaces to collect and acquire data via a network and communicate with other systems. These communication interfaces can be communicatively coupled to the processing circuitry 112 and the memory 114. For example, the communication interface can provide a communication connection between the image processing computing system 110 and components of a radiotherapy system (e.g., allowing data exchange with external devices). For example, in some examples, the communication interface may have appropriate interface circuitry with output device 142 or input device 144 to connect to a user interface 140, which may be a hardware keyboard, keypad, or touchscreen through which a user can input information into the radiotherapy system.

[0043] As an example, output device 142 may include a display device that outputs a representation of user interface 140 and one or more aspects, visualizations, or representations of medical images. Output device 142 may include one or more displays showing medical images, interface information, treatment planning parameters (e.g., contours, doses, beam angles, labels, maps, etc.), treatment plans, targets, target localization or tracking, patient status estimates (e.g., 3D images), or any user-related information. Input device 144 connected to user interface 140 may be a keyboard, keypad, touchscreen, or any type of device that allows the user to input information into the radiotherapy system. Alternatively, features of output device 142, input device 144, and user interface 140 may be integrated into a device such as a smartphone or tablet (e.g., Apple). Lenovo Samsung In a single device (etc.).

[0044] Furthermore, any and all components of a radiotherapy system can be implemented as virtual machines (e.g., via virtualization platforms such as VMware, Hyper-V, etc.). For example, a virtual machine can be software used as hardware. Therefore, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that collectively serve as hardware. For example, an image processing computing system 110, an image data source 150, or similar components can be implemented as virtual machines or implemented within a cloud-based virtualization environment.

[0045] Patient status processing logic 120 or other software programs can enable the computing system to communicate with image data source 150 to read images into memory 114 and storage device 116, or to store images or associated data from memory 114 or storage device 116 to image data source 150 and vice versa. For example, image data source 150 can be configured to store and provide multiple images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, raw data from MR or CT scans, Medical Digital Imaging and Communications (DICOM) metadata, etc.) hosted by image data source 150 from image acquisition device 170 acquired from one or more patients. Image data source 150 or other databases can also store data to be used by patient status processing logic 120 when executing software programs performing patient status estimation operations or when creating radiotherapy treatment plans. In addition, various databases can store data generated by preliminary motion models (e.g., dictionaries), correspondence motion models, or machine learning models, including network parameters constituting the model learned through the network and the resulting prediction data. In relation to the estimation of patient state through performing images as part of a treatment or diagnostic procedure, the image processing computing system 110 can therefore acquire and / or receive image data 160 (e.g., 2D MRI slice images, CT images, 2D fluorescence fluoroscopy images, X-ray images, 3D MRI images, 4D MRI images, etc.) from image data sources 150, image acquisition devices 170, treatment devices 180 (e.g., MRI linear accelerators), or other information systems.

[0046] Image acquisition device 170 can be configured to acquire one or more images of a patient's anatomical structure for a region of interest (e.g., a target organ, a target tumor, or both). Each image—typically a 2D image or slice—can include one or more parameters (e.g., 2D slice thickness, orientation, and location, etc.). In the example, image acquisition device 170 can acquire 2D slices of any orientation. For example, the orientation of a 2D slice can include sagittal, coronal, or axial orientation. Processing circuitry 112 can adjust one or more parameters, such as the thickness and / or orientation of the 2D slice, to include the target organ and / or target tumor. In the example, the 2D slice can be determined based on information such as 3D MRI volume. When a patient is receiving radiation therapy, such as when using treatment device 180, such 2D slices can be acquired by image acquisition device 170 "in real time" (where "in real time" means acquiring data in 10 milliseconds or less). In another example for some applications, real time can include a time range of (e.g., up to) 200 or 300 milliseconds. In the example, real time can include a time period fast enough to address a clinical problem using the techniques described herein. In this example, real-time may vary depending on target velocity, radiotherapy margin, time delay, response time of the treatment device, etc.

[0047] The patient state processing logic 120 in the image processing computing system 110 is described as implementing a patient state estimation workflow 130 that utilizes various aspects of model generation and estimation processing operations. In an example, the patient state estimation workflow 130, operated by the patient state processing logic 120, generates and utilizes a preliminary motion model 132 generated based on patient data (e.g., from a patient currently being treated, from multiple previous patients, etc.). The preliminary motion model 132 may include a model of a patient in motion (e.g., breathing) generated based on patient measurements and corresponding patient states. The patient state estimation workflow 130 includes creating a dictionary 134 by using the preliminary motion model to generate sample (potential) patient measurements and corresponding patient states. The patient state estimation workflow 130 includes training a correspondence motion model based on the dictionary 134 using machine learning 136 (e.g., using regression-based machine learning techniques). The patient state estimation workflow 130 includes estimating the patient state 138 using the correspondence motion model and current patient measurements (e.g., a 2D image).

[0048] Software programs such as treatment planning software (e.g., those manufactured by Elekta AB in Stockholm, Sweden) can be used to achieve this. The image processing computing system 110 uses patient state processing logic 120 and patient state estimation workflow 130 to generate a radiotherapy treatment plan. To generate the radiotherapy treatment plan, the image processing computing system 110 can communicate with an image acquisition device 170 (e.g., a CT device, MRI device, PET device, X-ray device, ultrasound device, etc.) to capture and access images of the patient and delineate the target (e.g., a tumor). In some examples, it may be necessary to delineate one or more organs at risk (OARs), such as healthy tissue surrounding or adjacent to the tumor.

[0049] To depict a target organ or tumor relative to the OAR, medical images of a patient undergoing radiotherapy, such as MRI, CT, PET, fMRI, X-ray, ultrasound, radiotherapy field images, SPECT images, etc., can be non-invasively acquired via image acquisition device 170 to reveal the internal structure of the body part. Based on information from the medical images, a 3D structure of the anatomical site can be obtained. Furthermore, during treatment planning, numerous parameters can be considered to balance, for example, determining where the OAR might be at a given time while the patient is moving (e.g., breathing) to achieve effective treatment of the target tumor (e.g., ensuring the target tumor receives a sufficient radiation dose for effective therapy) with low radiation from the OAR (e.g., the OAR receiving the lowest possible radiation dose). Other parameters that can be considered include: the location of the target organ and tumor, the location of the OAR, and the movement of the target relative to the OAR. For example, a 3D structure can be obtained by outlining the target or the OAR within each 2D layer or slice of the MRI or CT image and combining the outlines in each 2D layer or slice. The contour can be generated manually (e.g., by a physician, dosimeter, or healthcare professional using a program, such as one manufactured by Elekta AB in Stockholm, Sweden). (or automatically generated (e.g., using a program, such as the Atlas-based automatic segmentation software manufactured by Elekta AB in Stockholm, Sweden)). ).

[0050] After the target tumor and OAR have been located and mapped, a dosimeter, physician, or healthcare professional can determine the radiation dose to be applied to the target tumor, as well as any maximum dose that adjacent OARs (e.g., left and right parotid glands, optic nerve, eye, lens, inner ear, spinal cord, brainstem, etc.) can tolerate. After the radiation dose has been determined for each anatomically defined structure (e.g., target tumor, OAR), a process known as inverse planning can be performed to determine one or more treatment planning parameters that will achieve the desired radiation dose distribution. Examples of treatment planning parameters include volume mapping parameters (e.g., those defining the target volume, contour-sensitive structures, etc.), margins around the target tumor and OAR, beam angle selection, collimator settings, and beam-on count. During inverse planning, the physician can define dose constraint parameters that limit how much radiation the OAR can tolerate (e.g., limiting the full dose to the tumor target and zero dose to any OAR; limiting 95% of the dose to the target tumor; limiting the spinal cord, brainstem, and optic nerve structures to ≤45 Gy, ≤55 Gy, and <54 Gy, respectively). The results of the reverse planning can form a radiotherapy treatment plan that can be stored. Some of these treatment parameters may be interrelated. For example, adjusting one parameter (e.g., weighting for different targets, such as increasing the dose to the target tumor) in an attempt to change the treatment plan may affect at least one other parameter, which in turn may lead to the development of different treatment plans. Thus, the image processing computing system 110 can generate a customized radiotherapy treatment plan with these parameters so that the treatment device 180 can provide appropriate radiotherapy treatment to the patient.

[0051] Figure 2 An exemplary image-guided radiotherapy apparatus 202 is shown, which includes a radiation source such as an X-ray source or a linear accelerator, a bed 216, an imaging detector 214, and a radiotherapy output 204. The radiotherapy apparatus 202 can be configured to emit a radiation beam 208 to provide treatment to a patient. The radiotherapy output 204 may include one or more attenuators or collimators, such as multi-leaf collimators (MLCs).

[0052] As an example, a patient can be positioned in area 212, supported by treatment bed 216, to receive treatment according to a radiotherapy plan (e.g., by...). Figure 1The radiotherapy system (which generates a treatment plan) receives the radiotherapy dose. The radiotherapy output 204 can be mounted or attached to the frame 206 or other mechanical support. When the bed 216 is inserted into the treatment area, one or more chassis motors (not shown) can rotate the frame 206 and the radiotherapy output 204 about the bed 216. In one example, the frame 206 can rotate continuously about the bed 216 when it is inserted into the treatment area. In another example, the frame 206 can rotate to a predetermined position when the bed 216 is inserted into the treatment area. For example, the frame 206 can be configured to rotate the radiotherapy output 204 about an axis (“A”). Both the bed 216 and the radiotherapy output 204 can be moved independently to other positions around the patient, for example, along a lateral direction (“T”), along a side direction (“L”), or rotated about one or more other axes, such as about a lateral axis (denoted as “R”). A controller communicatively connected to one or more actuators (not shown) can control the movement or rotation of the bed 216 to properly position the patient within or outside the radiation beam 208 according to the radiotherapy treatment plan. Because the bed 216 and the gantry 206 can move independently of each other in multiple degrees of freedom, this allows the patient to be positioned such that the radiation beam 208 can precisely target the tumor.

[0053] Figure 2 The coordinate system shown (including axes A, T, and L) may have an origin located at isocenter 210. The isocenter may be defined as a position where the central axis of the radiotherapy beam 208 intersects the origin of the coordinate axes, for example, at a position where a predetermined radiation dose is delivered to or within the patient. Alternatively, isocenter 210 may be defined as a position where, for various rotational positions of the radiotherapy output 204, such as those positioned by the gantry 206, about axis A, the central axis of the radiotherapy beam 208 intersects the patient.

[0054] The gantry 206 may also have an attached imaging detector 214. The imaging detector 214 is preferably located opposite the radiation source (output section 204), and in this example, the imaging detector 214 may be located within the field of the therapy beam 208.

[0055] Preferably, the imaging detector 214 can be mounted on the gantry 206 opposite to the radiotherapy output section 204 to maintain alignment with the therapy beam 208. As the gantry 206 rotates, the imaging detector 214 rotates about a rotation axis. In an example, the imaging detector 214 can be a flat panel detector (e.g., a direct detector or a scintillator detector). In this way, the imaging detector 214 can be used to monitor the therapy beam 208, or it can be used to image the patient's anatomical structures, such as field imaging. The control circuitry of the radiotherapy device 202 can be integrated within or remote from the radiotherapy system.

[0056] In the illustrative example, one or more of the bed 216, therapy output 204, or gantry 206 can be automatically positioned, and the therapy output 204 can establish a treatment beam 208 according to a specified dose for a particular treatment delivery instance. Therapy delivery sequences can be specified according to a radiotherapy treatment plan, for example, using one or more different orientations or positions of the gantry 206, bed 216, or therapy output 204. Therapy deliveries can occur sequentially, but can cross at desired therapy sites on the patient or within the patient, for example, at isocenter 210. This allows a prescribed cumulative dose of radiotherapy to be delivered to the therapy site while minimizing or avoiding damage to tissues near the therapy site.

[0057] therefore, Figure 2 Specifically, an example of a radiotherapy device 202 is shown, which is operable to provide radiotherapy treatment to a patient. The radiotherapy device 202 has a configuration in which a radiotherapy output section can rotate about a central axis (e.g., axis "A"). Other radiotherapy output section configurations can be used. For example, the radiotherapy output section can be mounted to a robotic arm or manipulator with multiple degrees of freedom. In another example, the therapy output section can be fixed, for example, located in a region laterally separated from the patient, and a platform supporting the patient can be used to align the center of the radiotherapy with a designated target site within the patient's body. In another example, the radiotherapy device can be a combination of a linear accelerator and an image acquisition device. As those skilled in the art will recognize, in some examples, the image acquisition device can be MRI, X-ray, CT, CBCT, spiral CT, PET, SPECT, optical tomography, fluorescence imaging, ultrasound imaging, MR linear accelerator, or radiotherapy field imaging device, etc.

[0058] Figure 3An exemplary radiotherapy system 300 (e.g., referred to in the art as an MR linear accelerator) is depicted, which may include a combination of a radiotherapy device 202 and an imaging system (e.g., a magnetic resonance (MR) imaging system consistent with the disclosed embodiments). As shown, system 300 may include a bed 310, an image acquisition device 320, and a radiation delivery device 330. System 300 delivers radiotherapy to a patient according to a radiotherapy treatment plan. In some embodiments, the image acquisition device 320 may correspond to a device capable of acquiring images. Figure 1 Image acquisition device 170 in the middle.

[0059] Bed 310 can support a patient (not shown) during treatment. In some implementations, bed 310 can move along a horizontal translation axis (labeled "I"), allowing bed 310 to move a patient lying on bed 310 into or out of system 300. Bed 310 can also rotate about a vertical rotation axis transverse to the center of the translation axis. To allow such movement or rotation, bed 310 may have motors (not shown) that enable the bed to move in various directions and rotate along various axes. A controller (not shown) can control these movements or rotations to properly position the patient according to the treatment plan.

[0060] In some embodiments, the image acquisition device 320 may include an MRI machine for acquiring 2D or 3D MRI images of a patient before, during, or after a treatment procedure. The image acquisition device 320 may include a magnet 321 for generating a main magnetic field for magnetic resonance imaging. The magnetic field lines generated by the operation of the magnet 321 may extend substantially parallel to the central translation axis I. The magnet 321 may include one or more coils whose axes extend parallel to the translation axis I. In some embodiments, one or more coils of the magnet 321 may be spaced apart such that the central window 323 of the magnet 321 is free of coils. In other embodiments, the coils in the magnet 321 may be thin enough or have a reduced density such that they are substantially transmissive to radiation of wavelengths generated by the radiotherapy device 330. The image acquisition device 320 may also include one or more shielding coils that can generate magnetic fields with substantially equal amplitudes and opposite polarities outside the magnet 321 to eliminate or reduce any magnetic field outside the magnet 321. As described below, the radiation source 331 of the radiotherapy device 330 can be positioned in an area where the magnetic field is eliminated (at least to the first order) or reduced.

[0061] The image acquisition device 320 may further include two gradient coils 325 and 326, which can generate a gradient magnetic field superimposed on the main magnetic field. The coils 325 and 326 can generate gradients in the resulting magnetic field, which allow for spatial encoding of protons to determine their positions. The gradient coils 325 and 326 can be positioned together with the magnet 321 about a common central axis and can be displaced along this axis. Displacement can create gaps or windows between the coils 325 and 326. In embodiments where the magnet 321 also includes a central window 323 between the coils, the two windows can be aligned with each other.

[0062] Image acquisition is used to track tumor movement. Sometimes, internal or external substitutes can be used. However, during radiotherapy, the implanted seed may move or be displaced from its initial location. Furthermore, the use of substitutes assumes a correlation between tumor movement and the displacement of the external substitute. However, phase shifts may exist between external substitutes and tumor movement, and their positions may often lose correlation over time. It is known that mismatches exceeding 9 mm can exist between the tumor and the substitute. Moreover, any deformation of the tumor shape during tracking is unknown.

[0063] The advantage of magnetic resonance imaging (MRI) lies in providing superior soft tissue contrast for more detailed visualization of tumors. Using multiple intra-fraction MR images makes it possible to determine both the shape and location of the tumor (e.g., centroid). Furthermore, MRI images improve upon what radiation oncologists can achieve even when using automated contouring software (e.g., [missing information]). This is due to the high contrast between the tumor target and the background area provided by the MR image, and any manual contouring performed during this process.

[0064] Another advantage of using an MR linear accelerator system is the ability to continuously switch the treatment beam and thus perform intra-fractional tracking of the target tumor. For example, optical tracking devices or stereoscopic X-ray fluorescence fluoroscopy systems can detect tumor location at 30 Hz using a tumor substitute. With MRI, the imaging acquisition rate is even faster (e.g., 3 fps to 6 fps). Therefore, the centroid position of the target can be determined, and artificial intelligence (e.g., neural network) software can predict the future target location. Another advantage of intra-fractional tracking using an MR linear accelerator is that, by being able to predict the future target location, the blades of the multi-leaf collimator (MLC) can be aligned with the target profile and its predicted future location. Therefore, the frequency of predicting the future tumor location using MRI is the same as the imaging frequency during tracking. The ability to clearly track the movement of the target tumor using detailed MRI imaging enables the delivery of highly conformal radiation doses to a moving target.

[0065] In some embodiments, the image acquisition device 320 may be an imaging device other than MRI, such as X-ray, CT, CBCT, spiral CT, PET, SPECT, optical computed tomography, fluorescence imaging, ultrasound imaging, or radiotherapy field imaging devices. As those skilled in the art will recognize, the above description of the image acquisition device 320 relates to certain embodiments and is not intended to be limiting.

[0066] The radiotherapy device 330 may include a radiation source 331, such as an X-ray source or a linear accelerator, and a multi-leaf collimator (MLC) 333. The radiotherapy device 330 may be mounted on a chassis 335. When the bed 310 is inserted into the treatment area, one or more chassis motors (not shown) may rotate the chassis 335 around the bed 310. In an embodiment, the chassis 335 is capable of continuous rotation around the bed 310 when it is inserted into the treatment area. The chassis 335 may also have an attached radiation detector (not shown), preferably positioned relative to the radiation source 331, wherein the axis of rotation of the chassis 335 is positioned between the radiation source 331 and the detector. Furthermore, the device 330 may include control circuitry (not shown) for controlling one or more of, for example, the bed 310, the image acquisition device 320, and the radiotherapy device 330. The control circuitry of the radiotherapy device 330 may be integrated within or remotely from the system 300.

[0067] During the radiotherapy treatment, the patient can be positioned on bed 310. System 300 can then move bed 310 to the treatment area defined by magnetic coils 321, 325, 326 and chassis 335. Control circuitry can then control radiation source 331, MLC 333, and one or more chassis motors to deliver radiation to the patient through a window between coils 325 and 326 according to the radiotherapy treatment plan.

[0068] Figure 4 An exemplary flowchart for estimating patient status is shown. Figure 4 Includes a patient state generator 408 for estimating patient state using a correspondence motion model. The patient state generator 408 uses an instantaneous local measurement 402 of the patient and a preliminary motion model 406 to estimate the patient state output at box 410. The preliminary motion model 406 is generated using a previous measurement 404, which includes a previous patient state corresponding to the previous measurement 404.

[0069] In practical radiotherapy applications, local measurements (e.g., 2D images or image slices) provide incomplete information about the patient's condition (e.g., 3D images). For example, a 2D MRI slice is a single section through a 3D representation of the patient, and an X-ray projection is an integration of voxels along the ray lines of the 3D representation. Unbiased information is generated using either image (e.g., a 2D image of the patient's anatomical structure rather than a 3D representation). The patient condition generator 408 can estimate the patient condition 410 using local information and a generated patient model 406 obtained based on past measurements and / or offline (pre-treatment) data.

[0070] The patient model generator 408 may include the creation of a low-dimensional representation of the patient's state. In the example, previous measurements are first reconstructed into 4D images. Examples may include a 4D CT acquired during the planning phase for generating a treatment plan; a 4D CBCT acquired on a conventional linear accelerator (e.g., generated by rotating a kV imager around the patient) immediately preceding each treatment step in which the patient is in the treatment position; and a 4D MR acquired on an MR linear accelerator, etc., preceding each treatment step.

[0071] 4D images can comprise a series of 3D images representing a representative respiratory cycle. For example, in 4DCBCT, multiple X-ray projections are acquired and classified into multiple bins. This can be done, for example, by directly detecting the aperture position in each projection in the image, or by using a separate respiratory signal acquired simultaneously with the kV projection and binning the projections based on the phase or amplitude of the signal. Each bin is then reconstructed individually using the kV projection assigned to that bin to form a 3D image for each bin. Similar techniques can be used to generate 4D MR images. These 4D images can then be used as an interim step in model reconstruction.

[0072] In the example, a reference phase of the 4D image is selected (e.g., a reference phase for treatment planning), and deformable image registration (DIR) is performed between the 3D image of each phase and the 3D image of the reference phase. The reference phase may include high-level treatment information (e.g., GTV, organs at risk, etc.). The output of the DIR processing may include a displacement vector field (DVF) linking each phase to the reference phase.

[0073] Such a DVF-based motion model provides a mechanism for transforming a reference patient state (e.g., treatment information defined on a 3D reference image) into a specific anatomical structure displayed in each of the other phases of a representative respiratory cycle represented in a 4D dataset.

[0074] To interpolate or extrapolate the initial motion model 406 to generate a new DVF, unsupervised dimensionality reduction techniques such as principal component analysis (PCA), independent component analysis (ICA), and canonical correlation analysis (CCA) can be used to identify one or more principal degrees of freedom for respiratory motion. In this example, two or three degrees of freedom may be sufficient to accurately estimate the patient's state. In this example, other degrees of freedom can be ignored or discarded (e.g., when they provide little useful information and are mostly noise). For example, PCA of the DVF motion model can produce a low-dimensional patient motion model corresponding to the mean DVF and two or three DVF "eigenmodes" (e.g., weighted inputs representing degrees of freedom). The DVF at any point in time during the motion cycle can be represented as a weighted sum of the mean and the eigenmodes. For example, the mean DVF can be represented by DVF0 and the intrinsic modes can be DVF1 and DVF2, which are two complete 3D vector fields. The DVF at any time during the period can then be written as DVF = DVF0 + a1*DVF1 + a2*DVF2, where a1 and a2 are scalar numbers representing time variations. In this example, the motion model is simplified to identifying a1 and a2 at a specific time rather than the entire DVF. Once calculated, the DVF can then be used to warp a reference 3D image to obtain a current 3D image representing patient state 310 (and can be extended to multiple patient images).

[0075] In some cases, the transitional steps for reconstructing 4D images may not be necessary, and low-dimensional state representations can be created directly from measurements.

[0076] In the example, the advantage of using pre-treatment images is that, since the data is acquired immediately before treatment, it is likely to represent the patient's respiratory degrees of freedom quite well. In some cases, using previous 4D images may be beneficial; for example, higher quality images may be feasible (e.g., using MRI when it is unavailable during treatment, or using CT when only CBCT is available before treatment), and it may take more time to generate and validate the patient model. In yet another example, data from multiple patients could be used to generate a more robust model, for example, to avoid over-constrained models.

[0077] Figure 5 An exemplary flowchart illustrating a patient state dictionary generation technique is shown. Generating a dictionary to be used with machine learning algorithms to output patient state estimates can utilize pairs of potential patient states and measurements.

[0078] The technique for generating the dictionary includes operation 502, which receives a measurement or a set of measurements and a corresponding patient state or a corresponding set of patient states. For example, the measurement may include a 2D image or other local patient state information or data. The patient state may include a 3D or 4D representation of the patient corresponding to the measurement. Therefore, the dictionary may include labeled data for training or testing machine learning algorithms. In the example, the received measurement may include a digitally reconstructed radiographic (DRR) image for a CT-based patient model or a 2D MRI slice for an MRI-based patient model.

[0079] In the examples, generating the dictionary could involve computing the 2D DVF on the 2D image instead of directly using it as a measurement. For instance, PCA analysis of the 2D DVF yields several parameters. In this example, the input is a 2D PCA of the 2D DVF parameters. In another example, a real-time 2D image is registered with a reference 2D image (e.g., with the same contrast) acquired during the same stage. This allows for the generation of the 2D DVF using fast, highly parallelizable, deformable image registration techniques such as the demon algorithm. The demon algorithm can be adapted for parallel implementations in GPUs with real-time performance. In yet another example, a convolutional neural network (CNN) can be used to estimate the 2D optical flow between two images in real time to generate the 2D DVF.

[0080] The technique includes operation 504, which generates an extended set of potential measurements and corresponding potential patient states. Potential measurements and potential patient states can be generated by taking an initial actual measurement (e.g., the measurement received in operation 502) and its corresponding actual patient state and adding noise, interference, or otherwise extrapolating other measurement-patient state pairs that might occur for a particular patient or multiple patients. Operation 504 allows for the generation of a set of labeled data from even a single actual measurement and patient state pair.

[0081] The technique includes operation 506, which stores a set of extended potential measurements and corresponding potential patient states in a dictionary for use with machine learning techniques. In the example, received measurements or a set of measurements and received corresponding patient states or a set of patient states can also be stored in a dictionary for use with machine learning techniques. In the example, measurements (actual or potential) can be used as input data for machine learning techniques, where the machine learning techniques output corresponding patient states (actual or potential), and the correspondence serves as a label for the data.

[0082] In operation 504, low-dimensional patient state representations (e.g., using PCA, ICA, CCA, etc.) can be used to generate extended latent measurements. Low-dimensional patient state representations can be used to generate potential patient states that may arise during treatment. For example, coefficients (e.g., as described above) can be used... Figure 3 The reasonable ranges of a1 and a2 described in the DVF equation are subdivided into multiple step sizes (e.g., equal step sizes), and the resulting patient state can be computed for each step size. The result can form a dictionary of latent states corresponding to the latent measurements determined for each step size. In the example, the coefficients are randomly sampled such that they represent the most probable movements occurring in an actual patient. For example, a Gaussian distribution centered on a curve representing the average respiratory cycle can be used to sample the coefficients. The patient state is computed by distorting a reference image (e.g., a 3D patient image) according to the DVF. In the example, the dictionary can be used, for example, to infer a 3D image from a 2D input using regression analysis in a supervised machine learning algorithm.

[0083] In the example, to generate an expanded set of potential measurements and corresponding potential patient states, a small rigid transformation can be applied to the 3D image for data augmentation to facilitate potential patient translation. In this example, the dictionary of potential patient states can be generated from multiple patients rather than a specific patient being treated, using the acquired 3D image, biomechanical model, etc. In another example, for practical reasons, utilizing a specific patient can be used to limit the required data and ensure that the data is relevant to the patient being treated.

[0084] Using PCA methods to generate extended latent measurements and latent patient states may include generating PCA coefficients. To generate realistic training patient states, the coefficients can be randomly plotted according to a normal distribution, for example, centered on the mean trajectory (e.g., within a 4D dataset of received patient states), where the standard distribution is equal to a percentage of the dynamic range of each coefficient (e.g., 10%). Next, PCA to DVF reconstruction is performed. The complete DVF can be reconstructed using randomly generated PCA coefficients (e.g., 2 to 3 coefficients representing the degrees of freedom of the moving patient). The DVF is converted to the original patient state volume by distorting the reference volume using the complete DVF. Local measurements are created based on the original patient state volume. For CT-based motion models, 2D digitally reconstructed radiographs are computed based on the original patient state volume using, for example, the Siddon-Jacobs algorithm. For MRI-based motion models, the 3D volume is resampled to extract 2D MRI slices. Small stiffness transformations are applied to the 3D volume for data augmentation to address minor differences in patient position fractions.

[0085] Simultaneously, the original patient state volume output and local measurements are used together as training samples to be stored in a dictionary (e.g., the input is a measurement and the output is the patient state). This workflow can be repeated for a large number of training samples, such as 1000 samples or an optimized number of training samples.

[0086] For each generated potential patient state, one or more potential patient measurements are simulated. These are measurements that may potentially lead to the corresponding state. For example, for a state represented by a 3D MRI image, 2D slices from a specific orientation and location (e.g., sagittal) that are expected to be used during treatment can be extracted. For a 3D CBCT image, for a specific gantry angle, kV x-ray projection can be simulated, for example, by using the Siddon-Jacobs algorithm to ray-trace the 3D image and integrate voxels along the ray spectrum. More sophisticated algorithms can be considered, such as those utilizing Monte Carlo algorithms, to simulate the results that realistic 2D kV x-ray images might produce, including effects such as scattering or beam hardening. Imaging characteristics (e.g., slices or gantry angles) can be randomly sampled, uniformly sampled, or fixed to known values. In some examples, a standalone AI algorithm (e.g., a Generative Adversarial Network (GAN)) can be used to estimate measurements based on patient state, especially when patient measurements cannot be easily computed based on state (e.g., 2D MR slice measurements based on 3D density patient state information). In some examples, the dimensionality of the dictionary can be further reduced by performing unsupervised dimensionality reduction on the latent measurement or patient state (e.g., PCA, ICA, or CCA). In other examples, a demon algorithm (utilizing image registration with a reference image) or a CNN can be used to generate a DVF for the latent measurement.

[0087] Figure 6 An exemplary regression model machine learning engine 600 for use in estimating patient status is shown. The machine learning engine 600 utilizes a training engine 602 and an estimation engine 604. The training engine 602 inputs historical transaction information 606 (e.g., patient measurements and corresponding patient statuses) into a feature determination engine 608. The historical transaction information 606 can be labeled to indicate the correspondence between measurements and patient statuses.

[0088] Feature determination engine 608 determines one or more features 610 based on the historical information 606. Generally, feature 610 is a set of information inputs and includes information determined to predict a specific outcome. In the example, feature 610 can be determined by a hidden layer. Machine learning algorithm 612 generates a correspondence motion model 620 based on feature 610 and labels.

[0089] In estimation engine 604, current action information 614 (e.g., current patient measurement) can be input into feature determination engine 616. Feature determination engine 616 can determine features of the current information 614 to estimate the corresponding patient state. In some examples, feature determination engine 616 and 608 are the same engine. Feature determination engine 616 produces feature vector 618, which is input into model 620 to generate one or more standard weights 622. Training engine 602 can operate offline to train model 620. However, estimation engine 604 can be designed to operate online. It should be noted that model 620 can be periodically updated via additional training or user feedback (e.g., additional, changed, or deleted measurements or patient states).

[0090] Machine learning algorithms can be chosen from many different potential supervised or unsupervised machine learning algorithms. Examples of supervised learning algorithms include artificial neural networks, Bayesian networks, instance-based learning, support vector machines, decision trees (e.g., Iterative Bisoresistive 3, C4.5, Classification and Regression Trees (CART), Chi-Square Automatic Interaction Detector (CHAID), etc.), random forests, linear classifiers, quadratic classifiers, k-nearest neighbors, linear regression, logistic regression, and hidden Markov models. Examples of unsupervised learning algorithms include expectation-maximization algorithms, vector quantization, and information bottleneck methods. Unsupervised models may not require a training engine.

[0091] In the example, a regression model is used, and model 620 is a vector of coefficients corresponding to the importance of the learned features in the vectors of features 610 and 618. The regression model is shown in box 624, illustrating an example linear regression. Machine learning algorithm 612 is trained using a dictionary generated as described herein. Machine learning algorithm 612 is trained on how patient measurements correspond to patient states. In the example, machine learning algorithm 612 implements a regression problem (e.g., linear, multinomial, regression tree, kernel density estimation, support vector regression, random forest implementation, etc.). The resulting training parameters constrain the patient state generator to a correspondence motion model for the selected machine learning algorithm.

[0092] In the case of a conventional linear accelerator, since the X-ray acquisition orientation can be restricted to an orthogonal angle relative to the treatment beam, such training can be performed individually for each possible gantry angle (e.g., in one angular increment). In the case of an MR linear accelerator, the physician can be given control over the position or orientation on the 2D acquisition plane. Repeated cross-validation of the training data using different choices of the 2D plane can reveal which 2D planes provide the best alternative information for a given patient / tumor site.

[0093] In some cases, patient measurements can be used to update model 620. In other cases, calculations can be performed to determine whether the patient measurements are consistent with model 620, and if not, treatment is paused (e.g., for the KDE algorithm, using a threshold for variance, or determining whether there is sufficient data in the dictionary of the measurement's neighborhood). When treatment is paused, a new model 620 can be generated, or if the measurement (e.g., motion) is an aberration, the old model 620 can be reused.

[0094] In some applications, the entire real-time patient image may not be necessary, and only its features may be useful. For example, the target centroid can be used for geometric correction of a multi-leaf collimator (MLC) or to turn the bundle on or off. In this case, a single DVF vector connecting the center of the target in the reference image to the current target can be used instead of calculating the entire 3DDVF and deforming the entire reference image each time, making real-time processing more efficient.

[0095] After successfully training the patient state generator and aligning the patient model 620 with the patient, the treatment beam is opened and instantaneous local measurements are acquired at a given frequency. For each received measurement, the process may include normalizing a 2D image of the received measurement to match the contrast of the training image. The patient state generator can use the normalized measurements to infer model coefficients, and the model can be used to reconstruct the DVF. The reconstructed DVF can be used to distort the reference volume and treatment information into a current patient state that can be output or saved.

[0096] In some cases, the model may not align well with the patient during treatment. This can happen if the patient moves between 4D images and treatment, if a model from the previous day is used, or if data from another patient is used. The patient model (pre-treatment calculation) can then be aligned with the actual patient position by rigid registration with the new patient while the patient is in the treatment position. During this time, a CBCT or MRI scan is acquired to achieve a coarse model-to-patient alignment. After the CBCT or MRI acquisition, a precise alignment of the patient model using multiple sample images (e.g., X-rays or 2D MRI slices) can be applied to address bed displacement.

[0097] The contrast discrepancy between synthetically generated training measurements and those obtained from actual 2D imaging can hinder the generator's ability to infer the 3D patient state. This problem can be corrected using certain intensity normalization processes. For example, local or global linear normalization methods can be used. Other examples could include using generative adversarial networks (GANs) to map the intensity of real and synthetic images.

[0098] Figure 7A flowchart 700 illustrating exemplary operations for estimating patient status is shown. Flowchart 700 includes an optional operation 702 for receiving patient data, such as using a processor, including a set of patient measurements and corresponding patient statuses. The corresponding patient statuses may include 3D or 4D patient images, such as 3D CT, 3D CBCT, 3D MRI, 3D PET, 3D ultrasound, 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images. Patient measurements may include 2D MRI slices, MRI k-space data, 1D MRI navigators, 2D MRI projections, X-ray 2D projection data, PET data, 2D ultrasound slices, etc. In some cases, detectors may be arranged to acquire patient measurements simultaneously from multiple fields of view, for example, using stereo kV imaging, or from multiple coexisting modalities (e.g., kV imaging combined with a surface camera). In one example, patient data may be generated from a single patient. In another example, patient data may include data from multiple patients.

[0099] Flowchart 700 includes operation 704, which is used to identify a preliminary motion model of a moving patient, for example, based on measurements of the group of patients and the corresponding patient state. In the example, a preliminary motion model can be generated based on a 4D dataset acquired prior to radiotherapy treatment. A preliminary motion model can be generated based on 4D MR or 4D CBCT acquired during treatment. The DVF between each phase of the 4D image and a reference phase can be calculated. PCA analysis can be performed on these DVFs. The preliminary motion model can be a 3D DVF parameterized by two to three scalars plus a reference 3D image. Potential 3D images that may occur during treatment can be generated based on the two to three scalars from which the DVF can be calculated. This DVF can be used to deform the reference 3D image to calculate a new 3D image.

[0100] Flowchart 700 includes operation 706, which is used to generate a dictionary of extended potential patient measurements and corresponding potential patient states using a motion model. The extended potential patient measurements may include deformations of 3D or 4D patient images. In the example, the deformation includes a deformable vector field (DVF) calculated using a deformable registration algorithm. In the example, the extended potential patient measurements include 2D projected images. The extended potential patient measurements can be generated using one or more of the following methods: extracting 2D slices from a 3D image, ray tracing from a 3D image to generate a 2D projected image, using Monte Carlo techniques to simulate X-ray interaction with a 3D image, using folded cone convolution techniques, using stacking and convolution techniques, using generative adversarial networks, using convolutional neural networks, using recurrent neural networks, etc. The dictionary may include corresponding potential patient states derived by randomly sampling 2 to 3 scalars, generating 3D DVFs from the scalars, and deforming a reference image into possible 3D images that may appear during treatment.

[0101] Extended potential patient measurements can be generated by computing the 2D DVF on a 2D input image. In one example, the 2D DVF can be computed by performing PCA analysis on the 2D input image. In another example, the 2D DVF can be computed by registering the 2D input image with a reference 2D image (e.g., taken at the start of radiotherapy or shortly before the start of radiotherapy) and utilizing deformable image registration techniques. The 2D input image and the reference 2D image can have the same contrast to allow for registration. Deformable image registration techniques can be fast, highly parallelizable techniques such as demon algorithms (e.g., implemented in parallel on a GPU with real-time performance). In yet another example, a CNN can be used to generate the 2D DVF to estimate the 2D optical flow between the 2D input image and the 2D reference image. The CNN can run in real time.

[0102] A corresponding potential patient state can be associated with a patient measurement that would produce the corresponding patient state. For example, a 2D slice can be extracted from a 3D image at a specific location or angle, or a 2D projection can be extracted from an image. In this example, the original 2D image may not be used as a measurement. Instead, a 2D DVF between the 2D image and a corresponding formatted reference 3D image can be used with PCA analysis of the resulting 2D DVF. For example, the measurement could be a processed version of the measurement, rather than directly measured patient data. The measurement could be a PCA component of the 2D DVF, which could include extended potential patient measurements. Pairs of extended potential patient measurements (PCA of 2D DVF) and corresponding patient states (PCA of 3D DVF) can form a dictionary.

[0103] Flowchart 700 includes operation 708, which uses machine learning techniques and a dictionary to train a correspondence motion model that associates input patient measurements with output patient states. The correspondence motion model may include a deformable vector field (DVF) as a function of one or more parameters. In the example, one or more parameters may be determined by reducing the dimensionality of an initial DVF calculated between two or more phases of a 4D image and a reference phase. For example, dimensionality reduction may include using principal component analysis (PCA), independent component analysis (ICA), or canonical correlation analysis (CCA). In the example, the correspondence motion model may be generated using random forest regression, linear regression, multinomial regression, regression trees, kernel density estimation, support vector regression algorithms, CNNs, RNNs, etc. Machine learning algorithms may be used to associate paired terms in a dictionary. The algorithm may be used with measurement inputs to provide patient states. Measurement inputs may include PCA components of 2D DVFs of a 2D image and a reference image, and patient states may include 3D DVFs.

[0104] In the example, the preceding operation occurs during pre-treatment, while the subsequent operation occurs during treatment. Flowchart 700 includes operation 710 for estimating a patient state corresponding to a patient measurement using a correspondence motion model. The patient state can be saved or output. For example, the patient state can be output to be displayed on a user interface of a display device. In the example, estimating the patient state can include receiving patient measurements as input to the correspondence motion model, which includes a real-time stream of 2D images. The real-time stream of 2D images can include stereo kV images (e.g., from a kV imager rotating around the patient in the case of a conventional linear accelerator) or paired 2D MR slice images (e.g., from an MR linear accelerator). In the example, the stereo kV images can include two orthogonal or substantially orthogonal (e.g., within 10 degrees) X-ray images acquired simultaneously or substantially simultaneously (e.g., within milliseconds or hundreds of milliseconds). The kV imager can be fixed in a room or fixed to a gantry (e.g., including a linear accelerator). A pair of 2D MR slice images can be orthogonal to each other or parallel to each other. In another example, two kV imagers can be used, for example, each at a 45-degree angle to the treatment beam (and 90 degrees to each other). In this example, two kV imagers can be used simultaneously, or they can be used alternately.

[0105] In the example, images can be acquired from one or two kV imagers, along with internal ultrasound data simultaneously. The ultrasound data can be used to reduce the kV dose by, for example, using fewer doses or pulses, or to achieve a lower kV imaging frame rate. This ancillary data can be directly included in the measurements to calculate patient status, or a separate correspondence model between kV and the ancillary data stream can be generated and used to correlate the ancillary data with patient status. For example, a correlation model can be established and continuously updated to associate the kV PCA component with parameters extracted from the ancillary measurement stream, and this correlation model can be used to determine the kV PCA component when acquiring the ancillary stream data.

[0106] During treatment, 2D images can be received. A 2D DVF (2DDVF) between the input 2D image and a reference 2D image can be calculated. PCA analysis can be performed on the DVF. The result is a real-time “measurement” as used herein. A trained machine learning algorithm can take the measurement as input and calculate the PCA component of the 3D DVF based on the input measurement. The PCA component is used to generate a 3D DVF, which is used to deform the 3D reference image to form a current real-time 3D patient image representing the patient at the current time. The patient state can be the 3D image itself, the reference image plus the 3D DVF, etc. (in this example, one can be calculated based on the other).

[0107] In the example, the operation may include outputting patient status, such as outputting two or more MR-like 3D images showing tissue contrast, outputting non-imaging information, outputting CT-like 3D images, etc.

[0108] Figure 8 A flowchart illustrating an exemplary operation for performing radiotherapy techniques is shown.

[0109] Flowchart 800 includes operation 802 for (e.g., as described above with respect to operation 706) utilizing a motion model to generate a dictionary of extended potential patient measurements and corresponding potential patient states. Extended potential patient measurements can be generated by computing a 2D DVF on a 2D input image. In one example, the 2D DVF can be computed by performing PCA analysis on the 2D input image. In another example, the 2D DVF can be computed by registering the 2D input image with a reference 2D image (e.g., taken at the start of radiotherapy treatment or shortly before the start of radiotherapy treatment) and utilizing a deformable image registration technique. The 2D input image and the reference 2D image can have the same contrast to allow for registration. The deformable image registration technique can be a fast, highly parallelizable technique such as the demon algorithm (e.g., implemented in parallel on a GPU with real-time performance). In yet another example, a CNN can be used to generate the 2D DVF to estimate the 2D optical flow between the 2D input image and the 2D reference image. The CNN can run in real time.

[0110] In the example, extended potential patient measurements can be generated from 4D images, including 4D CT, 4DCBCT, 4D MRI, 4D PET, 4D ultrasound images, etc. In the example, the extended potential patient measurements include 2D projection images and are generated using at least one of the following: extracting 2D slices from a 3D image; performing ray tracing on a 3D image to generate a 2D projection image; utilizing Monte Carlo techniques to simulate X-ray interaction with a 3D image; utilizing folded cone convolution techniques; utilizing stacking and convolution techniques; utilizing generative adversarial networks, convolutional neural networks, recurrent neural networks, etc.

[0111] Flowchart 800 includes operation 804, which is used (e.g., as described above with respect to operation 708) to train a correspondence motion model using a dictionary to associate input patient measurements with output patient states using machine learning techniques. The correspondence motion model may include a deformable vector field (DVF) as a function of one or more parameters, with one or more parameters determined by reducing the dimensionality of an initial DVF calculated between two or more phases of the 4D image and a reference phase. The correspondence motion model can be generated using random forest regression, linear regression, multinomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, recurrent neural networks, etc.

[0112] Flowchart 800 includes operation 806 for receiving a real-time stream of 2D images from an image acquisition device (e.g., a kV x-ray, MR, CT, or other image acquisition device). The real-time stream of 2D images may include stereo kV images (e.g., from a kV imager rotating around the patient in the case of a conventional linear accelerator) or paired 2D MR slice images (e.g., from an MR linear accelerator). In another example, the real-time stream of 2D images may include k-space data, low-resolution 3D MR images, 1D navigator data, or other MR information.

[0113] In the example, a stereo kV image can include two orthogonal or substantially orthogonal (e.g., within 10 degrees) X-ray images acquired simultaneously or substantially simultaneously (e.g., within milliseconds or hundreds of milliseconds). The kV imager can be mounted in a room or mounted to a gantry (e.g., including a linear accelerator). A pair of 2D MR slice images can be orthogonal to each other or parallel to each other.

[0114] Flowchart 800 includes operation 808 for estimating a patient state corresponding to an image in a real-time stream of 2D images using a correspondence motion model. For example, the patient state may be output as an image (e.g., 3D MR or CT), as non-image information, or both. The patient state may include information (e.g., images or text) describing the patient's anatomical structures (e.g., a tumor or organ of interest), or the patient state may be used (e.g., on a portion of a tumor) to establish a target (e.g., a radiotherapy target).

[0115] Flowchart 800 includes operations 810 for locating a radiotherapy target within the patient's body using the patient's condition.

[0116] Flowchart 800 includes operations 812 for tracking a patient's radiotherapy target in real time using the patient's status. For example, successive images from a real-time stream of 2D images can be used to output corresponding patient statuses, where the target is tracked from one patient status to the next.

[0117] Flowchart 800 includes operation 814 for guiding radiotherapy to a target using a treatment device (e.g., a stand-alone treatment device, a device coupled to an image acquisition device (e.g., an MR linear accelerator), etc.) based on the patient's condition. For example, the target may be located in operation 810 or tracked in operation 812, and radiotherapy may be applied based on the location or tracking. In the example, the location or tracking information may be displayed on the display device, for example, using a user interface present on the display device.

[0118] Supplementary Explanation

[0119] The above detailed description includes reference to the accompanying drawings, which form part of the detailed description. The drawings illustrate, by way of illustration and not limitation, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Furthermore, the inventors contemplate examples using any combination or substitution of those elements (or one or more aspects of those elements) shown or described with respect to a particular example (or one or more aspects of that particular example) or with respect to other examples shown or described herein (or one or more aspects of those other examples).

[0120] All publications, patents, and patent documents mentioned in this document are incorporated herein by reference in their entirety, as if they were incorporated individually by reference. In the event of any inconsistency between this document and those incorporated by reference, the usage in the incorporated (one or more) references shall be considered supplementary to the usage in this document; in the event of any inconsistency, the usage in this document shall prevail.

[0121] In this document, when introducing elements of various aspects of the invention or embodiments thereof, the terms “a,” “an,” “the,” and “the” are used, as is common in patent literature, to include one or more elements, independent of any other instance or usage of “at least one” or “one or more.” In this document, the term “or” is used to indicate non-exclusivity, or such that, unless otherwise stated, “A or B” includes “A but not B,” “B but not A,” and “A and B.”

[0122] In the appended claims, the terms "including" and "in which" are used as common English equivalents to the corresponding terms "comprising" and "wherein". Furthermore, in the appended claims, the terms "comprising", "including", and "having" are intended to be open-ended, meaning that other elements may be present in addition to those listed, such that anything following such terms in the claims (e.g., comprising, including, having) is still considered to fall within the scope of the claims. Additionally, in the appended claims, the terms "first", "second", and "third", etc., are used merely as designations and are not intended to impose numerical requirements on their objects.

[0123] The present invention also relates to a computing system adapted, configured, or operated to perform the operations described herein. This system may be specifically rebuilt for a desired purpose, or it may comprise a general-purpose computer selectively started or reconfigured by computer programs (e.g., instructions, code, etc.) stored in the computer. Unless otherwise stated, the order in which operations are performed or executed in embodiments of the invention shown and described herein is not necessarily fixed. That is, operations may be performed in any order unless otherwise stated, and embodiments of the invention may include additional or fewer operations compared to those disclosed herein. For example, it is contemplated that running or performing a particular operation before, simultaneously with, or after another operation is within the scope of this invention.

[0124] In view of the foregoing, it will be seen that several objectives of the present invention have been achieved and other advantageous results have been obtained. Having described aspects of the invention in detail, it will be apparent that modifications and variations are possible without departing from the scope of the invention as defined in the appended claims. Since various changes can be made to the above-described constructions, products, and methods without departing from the scope of the invention, everything contained in the foregoing description and shown in the accompanying drawings is intended to be illustrative rather than restrictive.

[0125] The above description is intended to be illustrative and not restrictive. For example, the examples (or one or more aspects of the examples) described above may be used in combination with each other. Furthermore, many modifications may be made to adapt particular situations or materials to the teachings of the invention without departing from the scope of the invention. Although the dimensions, types, and example parameters of the materials, functions, and implementations described herein are intended to define the parameters of the invention, they are by no means limiting embodiments, but rather exemplary embodiments. Many other embodiments will become apparent to those skilled in the art upon review of the above description. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

[0126] Furthermore, in the above-described embodiments, various features may be combined to simplify this disclosure. This should not be construed as meaning that all unclaimed disclosed features are necessary for any claim. Rather, the subject matter of the invention may lie in fewer than all features of a particular disclosed embodiment. Therefore, the appended claims are hereby incorporated into the embodiments, wherein each claim exists as an independent embodiment. The scope of the invention should be determined with reference to the appended claims and the full scope of the equivalents conferred by such claims.

[0127] Each of these non-restrictive examples can exist independently, or can be combined with one or more of the other examples in various permutations or combinations.

[0128] Example 1 is a method for estimating real-time patient status during radiotherapy treatment, the method comprising: receiving patient data including a set of constructed patient measurements using a processor; identifying a preliminary motion model of the patient in motion based on the set of constructed patient measurements; generating a dictionary of extended potential patient measurements and corresponding potential patient states using the preliminary motion model; training a correspondence motion model that associates input patient measurements with output patient states using machine learning techniques with the dictionary; and estimating the patient state corresponding to the patient's patient measurements using the correspondence motion model using the processor.

[0129] In Example 2, the subject of Example 1 includes: where the corresponding patient state includes a 3D patient image.

[0130] In Example 3, the subject of Example 2 includes: where the corresponding potential patient state includes a deformation of a 3D patient image.

[0131] In Example 4, the subject of Example 3 includes: wherein the deformation includes a deformation vector field (DVF) calculated using a deformable registration algorithm.

[0132] In Example 5, the subjects of Examples 1 to 4 include: wherein patient measurements include 2D MRI slices, MRI k-space data, 1D MRI navigators, 2D MRI projections, X-ray 2D projection data, PET data, or 2D ultrasound slices.

[0133] In Example 6, the subject matter of Examples 1 through 5 includes: where patient data includes 4D images.

[0134] In Example 7, the subject of Example 6 includes: wherein the 4D image is a 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound image.

[0135] In Example 8, the subject matter of Examples 1 to 7 includes: wherein the correspondence motion model includes a deformation vector field (DVF) as a function of one or more parameters, said one or more parameters being determined by reducing the dimension of an initial DVF calculated between two or more phases of a 4D image and a reference phase.

[0136] In Example 9, the subject matter of Examples 1 to 8 includes: wherein the extended potential patient measurement includes 2D projected images and is generated by utilizing at least one of the following: extracting 2D slices from a 3D image, performing ray tracing on a 3D image to generate a 2D projected image, utilizing Monte Carlo techniques to simulate X-ray interaction with a 3D image, utilizing folded cone convolution techniques, utilizing stacking and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

[0137] In Example 10, the topics of Examples 1 through 9 include: generating a corresponding motion model using random forest regression, linear regression, multinomial regression, regression tree, kernel density estimation, support vector regression algorithm, convolutional neural network or recurrent neural network.

[0138] In Example 11, the subject matter of Examples 1 to 10 includes: wherein estimating the patient state corresponding to a patient measurement includes receiving the patient measurement as input to a correspondence motion model, the input including a real-time stream of 2D images.

[0139] In Example 12, the subject of Example 11 includes: wherein the real-time stream of 2D images includes stereo kV images or pairs of 2D MR slice images.

[0140] In Example 13, the subject matter of Examples 1 to 12 includes: outputting patient status as two or more MR-like 3D images displaying tissue contrast.

[0141] In Example 14, the subject matter of Examples 1 to 13 includes: where the patient status includes non-imaging information.

[0142] In Example 15, the topics of Examples 1 through 14 include: generating a preliminary motion model based on a 4D dataset acquired prior to radiotherapy treatment.

[0143] In Example 16, the topics of Examples 1 through 15 include: generating constructed patient measurements by computing a 2D deformation vector field (DVF) on a 2D input image.

[0144] In Example 17, the topics of Examples 1 through 16 include: wherein generating the constructed patient measurements includes performing principal component analysis (PCA) of the 2D input images.

[0145] In Example 18, the subject matter of Examples 1 to 17 includes: wherein generating the constructed patient measurement involves registering a 2D input image with a reference 2D image and using deformable image registration techniques to compute a 2D DVF.

[0146] In Example 19, the subject matter of Examples 1 through 18 includes: wherein generating the constructed patient measurement involves using a convolutional neural network (CNN) to estimate the 2D optical flow between a 2D input image and a 2D reference image to compute a 2D DVF.

[0147] Example 20 is a system for estimating patient status during radiotherapy treatment. The system includes: a processor coupled to a memory including instructions that, when executed by the processor, cause the processor to: receive patient data comprising a set of constructed patient measurements; identify a preliminary motion model of a moving patient based on the set of constructed patient measurements; generate a dictionary of extended potential patient measurements and corresponding potential patient states using the preliminary motion model; train a correspondence motion model using the dictionary to associate input patient measurements with output patient states using machine learning techniques; and estimate the patient state corresponding to the patient's patient measurements using the correspondence motion model.

[0148] Example 21 is a method for estimating real-time patient status during radiotherapy treatment using a magnetic resonance linear accelerator (MR linear accelerator), the method comprising: generating an extended dictionary of potential patient measurements and corresponding potential patient statuses using a preliminary motion model; using machine learning techniques to train a correspondence motion model that associates input patient measurements with output patient statuses using the dictionary; receiving a real-time stream of 2D MR images from an image acquisition device; using a processor to estimate the patient status corresponding to an image in the real-time stream of 2D MR images using the correspondence motion model; and using a treatment device coupled to the image acquisition device to guide radiotherapy toward a target based on the patient status.

[0149] In Example 22, the subject of Example 21 includes: wherein the extended potential patient measurement includes deformation of a 3D patient image, and wherein the deformation includes a deformation vector field (DVF) calculated using a deformable registration algorithm.

[0150] In Example 23, the subject matter of Examples 21 to 22 includes: wherein extended potential patient measurements are generated based on 4D images, including 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images.

[0151] In Example 24, the subject matter of Examples 21 to 23 includes: wherein the correspondence motion model includes a deformation vector field (DVF) as a function of one or more parameters, said one or more parameters being determined by reducing the dimension of an initial DVF calculated between two or more phases of a 4D image and a reference phase.

[0152] In Example 25, the subject matter of Examples 21 to 24 includes: wherein the extended potential patient measurement comprises a 2D projected image and is generated by utilizing at least one of the following: extracting 2D slices from a 3D image, performing ray tracing on a 3D image to generate a 2D projected image, utilizing Monte Carlo techniques to simulate X-ray interaction with a 3D image, utilizing folded cone convolution techniques, utilizing stacking and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

[0153] In Example 26, the topics of Examples 21 through 25 include: generating corresponding motion models using random forest regression, linear regression, multinomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

[0154] In Example 27, the subject of Examples 21 through 26 includes: outputting patient status as two or more MR-like 3D images displaying tissue contrast.

[0155] In Example 28, the topics of Examples 21 through 27 include: generating extended potential patient measurements by computing a 2D deformation vector field (DVF) on a 2D input image.

[0156] In Example 29, the subject matter of Examples 21 to 28 includes: wherein generating extended potential patient measurements includes performing principal component analysis (PCA) of the 2D input images.

[0157] In Example 30, the subject matter of Examples 21 to 29 includes: wherein generating extended potential patient measurements includes registering a 2D input image with a reference 2D image and using deformable image registration techniques to compute a 2D DVF.

[0158] In Example 31, the subject matter of Examples 21 to 30 includes: wherein generating extended potential patient measurements includes using a convolutional neural network (CNN) to estimate the 2D optical flow between a 2D input image and a 2D reference image to compute a 2D DVF.

[0159] Example 32 is a method for generating real-time target localization data, comprising: generating an extended dictionary of potential patient measurements and corresponding potential patient states using a preliminary motion model; using machine learning techniques to train a correspondence motion model that associates input patient measurements with output patient states using the dictionary; receiving a real-time stream of 2D images from an image acquisition device; using a processor to estimate the patient states corresponding to the images in the real-time stream of 2D images using the correspondence motion model; localizing a radiotherapy target within the patient using the patient states; and outputting the position of the radiotherapy target on a display device.

[0160] In Example 33, the subject of Example 32 includes: wherein the real-time stream of 2D images includes stereo kV images or pairs of 2D MR slice images.

[0161] In Example 34, the subject matter of Examples 32 to 33 includes: wherein the extended potential patient measurement includes deformation of a 3D patient image, and wherein the deformation includes a deformation vector field (DVF) calculated using a deformable registration algorithm.

[0162] In Example 35, the subject matter of Examples 32 to 34 includes: wherein extended potential patient measurements are generated based on 4D images, including 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images.

[0163] In Example 36, the subject matter of Examples 32 to 35 includes: wherein the correspondence motion model includes a deformation vector field (DVF) as a function of one or more parameters, said one or more parameters being determined by reducing the dimension of an initial DVF calculated between two or more phases of a 4D image and a reference phase.

[0164] In Example 37, the subject matter of Examples 32 to 36 includes: wherein the extended potential patient measurement comprises a 2D projected image and is generated by utilizing at least one of the following: extracting 2D slices from a 3D image, performing ray tracing on a 3D image to generate a 2D projected image, utilizing Monte Carlo techniques to simulate X-ray interaction with a 3D image, utilizing folded cone convolution techniques, utilizing stacking and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

[0165] In Example 38, the topics of Examples 32 through 37 include: generating corresponding motion models using random forest regression, linear regression, multinomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

[0166] In Example 39, the subject of Examples 32 through 38 includes: outputting patient status as two or more MR-like 3D images displaying tissue contrast.

[0167] In Example 40, the topics of Examples 32 through 39 include: generating extended potential patient measurements by computing a 2D deformation vector field (DVF) on a 2D input image.

[0168] In Example 41, the subject matter of Examples 32 to 40 includes: wherein generating extended potential patient measurements includes performing principal component analysis (PCA) of the 2D input images.

[0169] In Example 42, the subject matter of Examples 32 to 41 includes: wherein, generating extended potential patient measurements includes registering a 2D input image with a reference 2D image and using deformable image registration techniques to compute a 2D DVF.

[0170] In Example 43, the subject matter of Examples 32 to 42 includes: wherein generating extended potential patient measurements includes using a convolutional neural network (CNN) to estimate the 2D optical flow between a 2D input image and a 2D reference image to compute a 2D DVF.

[0171] Example 44 is a method for real-time tracking of a target, comprising: generating an extended dictionary of potential patient measurements and corresponding potential patient states using a preliminary motion model; using machine learning techniques to train a correspondence motion model that associates input patient measurements with output patient states using the dictionary; receiving a real-time stream of 2D images from an image acquisition device; using a processor to estimate the patient states corresponding to images in the real-time stream of 2D images using the correspondence motion model; tracking the patient's radiotherapy target in real time using the patient states; and outputting tracking information for the radiotherapy target for display on a display device.

[0172] In Example 45, the subject of Example 44 includes: wherein the real-time stream of 2D images includes stereo kV images or pairs of 2D MR slice images.

[0173] In Example 46, the subject matter of Examples 44 to 45 includes: wherein the extended potential patient measurements include deformations of 3D patient images, and wherein the deformations include a deformation vector field (DVF) calculated using a deformable registration algorithm.

[0174] In Example 47, the subject matter of Examples 44 to 46 includes: wherein extended potential patient measurements are generated based on 4D images, including 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images.

[0175] In Example 48, the subject matter of Examples 44 to 47 includes: wherein the correspondence motion model includes a deformation vector field (DVF) as a function of one or more parameters, said one or more parameters being determined by reducing the dimension of an initial DVF calculated between two or more phases of a 4D image and a reference phase.

[0176] In Example 49, the subject matter of Examples 44 to 48 includes: wherein the extended potential patient measurement comprises a 2D projected image and is generated by utilizing at least one of the following: extracting 2D slices from a 3D image, performing ray tracing on a 3D image to generate a 2D projected image, utilizing Monte Carlo techniques to simulate X-ray interaction with a 3D image, utilizing folded cone convolution techniques, utilizing stacking and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

[0177] In Example 50, the topics of Examples 44 to 49 include: generating corresponding motion models using random forest regression, linear regression, multinomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

[0178] In Example 51, the subject of Examples 44 to 50 includes: outputting patient status as two or more MR-like 3D images displaying tissue contrast.

[0179] In Example 52, the topics of Examples 44 through 51 include: generating extended potential patient measurements by computing a 2D deformation vector field (DVF) on a 2D input image.

[0180] In Example 53, the subject matter of Examples 44 to 52 includes: wherein generating extended potential patient measurements includes performing principal component analysis (PCA) of the 2D input images.

[0181] In Example 54, the subject matter of Examples 44 to 53 includes: wherein generating extended potential patient measurements includes registering a 2D input image with a reference 2D image and using deformable image registration techniques to compute a 2D DVF.

[0182] In Example 55, the subject of Examples 44 to 54 includes: wherein generating extended potential patient measurements includes using a convolutional neural network (CNN) to estimate the 2D optical flow between a 2D input image and a 2D reference image to compute a 2D DVF.

[0183] Example 56 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform an operation for implementing any one of Examples 1 to 55.

[0184] Example 57 is a device that includes means for implementing any one of Examples 1 through 55.

[0185] Example 58 is a system for implementing any one of Examples 1 through 55.

[0186] Example 59 is a method for implementing any one of Examples 1 through 55.

[0187] The methods described herein may be implemented, at least in part, by a machine or computer. Some examples may include computer-readable or machine-readable media encoded with instructions operable to configure electronic devices to perform the methods described in the examples above. Implementations of these methods may include code such as microcode, assembly language code, higher-level language code, etc. Such code may include computer-readable instructions for performing various methods. The code may form part of a computer program product. Furthermore, in the examples, the code may be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, for example, during execution or at other times. Examples of such tangible computer-readable media may include, but are not limited to, hard disks, removable disks, removable optical discs (e.g., compact discs and digital video disks), magnetic tape cartridges, memory cards or memory sticks, random access memory (RAM), read-only memory (ROM), etc.

Claims

1. A method for estimating real-time patient state during radiotherapy treatment, the method comprising: identifying, using a processor, a preliminary motion model of a patient in motion, wherein the preliminary motion model is based on a set of constructed measurements generated from 2D input images of a particular patient acquired prior to the radiotherapy treatment, a 3D representation of the particular patient, or a 4D representation of the particular patient; generating, using the preliminary motion model, simulated extended potential patient measurements and corresponding potential patient states from an initial actual measurement and a corresponding actual patient state, wherein the simulated extended potential patient measurements and the corresponding potential patient states are generated by extrapolating measurement-patient state pairs from the initial actual measurement and the corresponding actual patient state; generating a dictionary of the simulated extended potential patient measurements and the corresponding potential patient states, wherein the initial actual measurement and the simulated extended potential patient measurements comprise 2D image data, and wherein the corresponding actual patient state and the corresponding potential patient state comprise 3D image data; training, prior to the radiotherapy treatment, a correspondence motion model that associates an input patient measurement with an output patient state using the dictionary with a machine learning technique; and estimating, using the processor, a real-time patient state corresponding to a patient measurement of the patient during radiotherapy treatment using the correspondence motion model, wherein the patient measurement comprises 2D image data of the particular patient acquired during the radiotherapy treatment.

2. The method of claim 1, wherein, the corresponding potential patient state comprises a 3D patient image.

3. The method of claim 2, wherein, the corresponding potential patient state comprises a deformation of the 3D patient image.

4. The method of claim 3, wherein, the deformation comprises a deformation vector field (DVF) computed with a deformable registration algorithm.

5. The method of claim 4, wherein, the DVF is a 3D DVF applied to the correspondence motion model to generate the corresponding potential patient state with deformation.

6. The method of claim 3, wherein, the deformation comprises a parameterization of a 3D DVF, and the preliminary motion model comprises a reference image.

7. The method of claim 1, wherein, the initial actual measurement and the patient measurement during treatment comprise 2D MRI slices, MRI k-space data, 2D MRI projections, x-ray 2D projection data, 2D PET data, or 2D ultrasound slices.

8. The method of claim 1, further comprising: receiving patient data comprising patient measurements and corresponding patient states, wherein the preliminary motion model is identified based on the patient measurements and the corresponding patient states, and wherein the patient data comprises 4D images.

9. The method of claim 8, wherein, the 4D images are 4D CT, 4D CBCT, 4D MRI, 4D PET, or 4D ultrasound images.

10. The method of claim 1, wherein, the correspondence motion model comprises a deformation vector field (DVF) as a function of one or more parameters determined by reducing a dimensionality of a preliminary DVF computed between two or more phases of a 4D image and a reference phase.

11. The method of claim 1, wherein, The simulated extended potential patient measurements include 2D projection images and are generated by utilizing at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, utilizing Monte Carlo techniques to simulate x-ray interactions with 3D images, utilizing collapsed cone convolution techniques, utilizing superposition and convolution techniques, utilizing generative adversarial networks, convolutional neural networks, or recurrent neural networks.

12. The method of claim 1, wherein, The correspondence motion model is generated utilizing random forest regression, linear regression, polynomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks, or recurrent neural networks.

13. The method of claim 1, wherein, Estimating real-time patient states corresponding to the patient measurements during treatment includes receiving the patient measurements as inputs to the correspondence motion model, the inputs including a real-time stream of 2D images.

14. The method of claim 13, wherein, The real-time stream of 2D images includes stereoscopic kV images or paired 2D MR slice images.

15. The method of claim 1, further comprising outputting the real-time patient states as two or more MR-like 3D images displaying tissue contrast.

16. The method of claim 1, wherein, The real-time patient states include non-imaging information.

17. The method of claim 1, further comprising: The preliminary motion model is generated based on a 4D data set acquired prior to the radiation therapy treatment.

18. The method of any one of claims 1 to 10, further comprising: The simulated extended potential patient measurements are generated by computing 2D deformation vector fields (DVF) on 2D input images.

19. The method of any one of claims 1 to 10, wherein, Generating the simulated extended potential patient measurements includes performing a principal component analysis (PCA) analysis of 2D input images.

20. The method of any one of claims 1 to 10, wherein, Generating the simulated extended potential patient measurements includes registering 2D input images with reference 2D images and computing 2D DVF utilizing deformable image registration techniques.

21. The method of any one of claims 1 to 10, wherein, Generating the simulated extended potential patient measurements includes estimating 2D optical flow between 2D input images and 2D reference images utilizing convolutional neural networks (CNN) to compute 2D DVF.

22. A machine-readable medium storing instructions for generating real-time target localization data, the instructions, when executed by processing circuitry of an image processing computing system, cause the processing circuitry to perform operations comprising: generating simulated extended potential patient measurements and corresponding potential patient states from initial actual measurements and corresponding actual patient states using a preliminary motion model, wherein the preliminary motion model is generated based on a set of constructed measurements from 2D input images of a particular patient, a 3D representation of the particular patient, or a 4D representation of the particular patient acquired prior to a radiation therapy treatment; and wherein the simulated extended potential patient measurements and the corresponding potential patient states are generated by extrapolating measurement-patient state pairs from the initial actual measurements and the corresponding actual patient states; generating a dictionary of the simulated extended potential patient measurements and the corresponding potential patient states, wherein the initial actual measurements and the simulated extended potential patient measurements include 2D image data, and wherein the corresponding actual patient states and the corresponding potential patient states include 3D image data; training, prior to the radiotherapy treatment, a correspondence motion model that associates input patient measurements with output patient states using the dictionary with a machine learning technique; receiving a real-time stream of 2D images from an image acquisition device; estimating, during the radiotherapy treatment, a real-time patient state corresponding to an image of the real-time stream of 2D images using the correspondence motion model; localizing a radiotherapy target in the patient using the real-time patient state; and outputting the location of the radiotherapy target on a display device.

23. The machine readable medium of claim 22, wherein, The real-time stream of 2D images comprises stereoscopic kV images or paired 2D MR slice images.

24. The machine readable medium of claim 22, wherein, The simulated extended potential patient measurements comprise deformations of 3D patient images, and wherein the deformations comprise deformation vector fields DVF computed using a deformable registration algorithm.

25. The machine readable medium of claim 22, wherein, The simulated extended potential patient measurements are generated from 4D images comprising 4D CT, 4D CBCT, 4D MRI, 4D PET or 4D ultrasound images.

26. The machine readable medium of claim 22, wherein, The correspondence motion model comprises deformation vector fields DVF as a function of one or more parameters determined by reducing the dimensionality of preliminary DVF computed between two or more phases of 4D images and a reference phase.

27. The machine readable medium of claim 22, wherein, The simulated extended potential patient measurements comprise 2D projection images and are generated by using at least one of: extracting 2D slices from 3D images, ray tracing through 3D images to generate 2D projection images, simulating x-ray interactions with 3D images using Monte Carlo techniques, using folded cone convolution techniques, using superposition and convolution techniques, using generative adversarial networks, convolutional neural networks or recurrent neural networks.

28. The machine readable medium of any one of claims 22 to 27, wherein, The correspondence motion model is generated using random forest regression, linear regression, polynomial regression, regression trees, kernel density estimation, support vector regression algorithms, convolutional neural networks or recurrent neural networks.

29. The machine readable medium of any one of claims 22 to 27, wherein, The instructions cause the processing circuitry to perform further operations comprising outputting the real-time patient state as two or more MR-like 3D images displaying tissue contrast.

30. The machine readable medium of claim 22, wherein, Generating the simulated extended potential patient measurements comprises performing an analysis of principal component analysis PCA of 2D input images.

31. The machine readable medium of claim 22, wherein, Generating the simulated extended potential patient measurements comprises registering 2D input images with reference 2D images and computing 2D DVF using deformable image registration techniques.

32. The machine readable medium of claim 22, wherein, Generating the simulated extended potential patient measurements comprises estimating 2D optical flow between 2D input images and 2D reference images using convolutional neural networks CNN to compute 2D DVF.

33. A machine-readable medium storing instructions for real-time tracking of a target, the instructions, when executed by processing circuitry of an image processing computing system, cause the processing circuitry to perform operations comprising: generating simulated extended potential patient measurements and corresponding potential patient states from an initial actual measurement and a corresponding actual patient state using a preliminary motion model, wherein the simulated extended potential patient measurements and the corresponding potential patient states are generated by extrapolating measurement-patient state pairs from the initial actual measurement and the corresponding actual patient state; generating a dictionary of the simulated extended potential patient measurements and the corresponding potential patient states, wherein the initial actual measurement and the simulated extended potential patient measurements comprise 2D image data, and wherein the corresponding actual patient state and the corresponding potential patient states comprise 3D image data; training, prior to a radiation therapy treatment, a correspondence motion model that associates an input patient measurement with an output patient state using the dictionary with a machine learning technique; receiving a real-time stream of 2D images from an image acquisition device; estimating, during the radiation therapy treatment, a real-time patient state corresponding to an image in the real-time stream of 2D images using the correspondence motion model; tracking, in real-time, a radiation therapy target of the patient using the real-time patient state; and outputting tracking information for the radiation therapy target for display on a display device.

34. The machine readable medium of claim 33, wherein, The real-time stream of 2D images comprises stereoscopic kV images or paired 2D MR slice images.

35. The machine readable medium of claim 33, wherein, The instructions cause the processing circuit to perform further operations comprising generating the simulated extended potential patient measurements by computing 2D deformation vector fields (DVF) on 2D input images.

36. The machine readable medium of claim 33, wherein, Generating the simulated extended potential patient measurements comprises performing a principal component analysis (PCA) analysis of 2D input images.

37. The machine readable medium of claim 33, wherein, Generating the simulated extended potential patient measurements comprises registering 2D input images with reference 2D images and computing 2D DVF using a deformable image registration technique.

38. The machine readable medium of claim 33, wherein, Generating the simulated extended potential patient measurements comprises estimating 2D optical flow between 2D input images and 2D reference images using a convolutional neural network (CNN) to compute 2D DVF.

39. A machine-readable medium storing instructions for generating real-time target localization data, the instructions, when executed by processing circuitry of an image processing computing system, cause the processing circuitry to perform operations comprising: generating a 3D deformation vector field (DVF) parameterized by two or more scalars from a 4D image by: computing a DVF between each phase of the 4D image and a reference image; and performing a principal component analysis (PCA) analysis on the DVF; generating potential 3D images by: randomly sampling the two or more scalars; generating new 3D DVF from the randomly sampled two or more scalars; and deforming the reference image to generate a corresponding potential patient state; associating the corresponding potential patient state with a corresponding potential patient measurement corresponding to the randomly sampled two or more scalars; computing a corresponding 2D DVF between the corresponding potential patient measurement and a portion of the reference image; performing PCA on the corresponding 2D DVF to obtain a simulated expanded latent patient measurement; generating a dictionary of the simulated expanded latent patient measurements and the corresponding latent patient states, wherein the simulated expanded latent patient measurements comprise 2D image data and the corresponding latent patient states comprise 3D image data; training, prior to a radiotherapy treatment, a correspondence motion model that relates the simulated expanded latent patient measurements to the corresponding latent patient states using the dictionary using a machine learning technique; receiving a 2D image of a patient during the radiotherapy treatment; calculating, in real-time, a measurement by: calculating a 2D DVF between the 2D image and a portion of the reference image; and performing PCA on the 2D DVF; inputting the measurement into the correspondence motion model to generate a PCA component of a reconstructed 3D DVF; generating the reconstructed 3D DVF from the PCA component; and deforming the reference image with the reconstructed 3D DVF to generate a current real-time 3D patient image.

40. The machine readable medium of claim 39, further comprising: outputting a current real-time patient state comprising the current real-time 3D patient image or the reference image and the reconstructed 3D DVF.

41. The machine readable medium of claim 39, wherein, The corresponding latent patient measurements comprise 2D images extracted from 3D images or 2D projections from 3D images.

42. The machine readable medium of claim 39, wherein, The 4D images are 4D MR images or 4D CBCT images. The 4D images are 4D MR images or 4D CBCT images.

Images