Adversity prediction for radiotherapy treatment planning

Through deep learning technology, GAN and CycleGAN models are used to automatically predict radiotherapy machine parameters, solving the complex and time-consuming problems of the creation process of existing radiotherapy plans, improving the accuracy and efficiency of treatment, and achieving efficient automation of radiotherapy plans .

CN115361998BActive Publication Date: 2024-12-03ELEKTA AB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180026513.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-02-07
Filing Date
2021-02-03
Publication Date
2024-12-03
Estimated Expiration
2041-02-03

AI Technical Summary

Technical Problem

The creation process of existing radiation therapy plans is complex and time-consuming, especially when multiple organs are considered, and traditional methods rely on manual techniques and numerical optimization, making it difficult to ensure the quality and efficiency of the treatment plan.

Method used

Using deep learning technologies such as generative adversarial network (GAN) and cyclic consistent generative adversarial network (CycleGAN), the model is trained based on patient anatomical structure image data to automatically predict radiotherapy machine parameters, including multi-leaf collimator blade position and Beam intensity to generate an optimized treatment plan.

Benefits of technology

Reduces the time and complexity of the creation of treatment plans, improves the accuracy and efficiency of treatments, reduces the risk of unanticipated radiation exposure, and can be automated for various medical settings and radiation therapy devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115361998B_ABST
    Figure CN115361998B_ABST
Patent Text Reader

Abstract

A system and method for generating radiation therapy treatment machine parameters based on projection images of target anatomical structures is disclosed. The system and method include operations including: receiving a set of image data pairs for each gantry angle of the radiation therapy treatment machine, wherein each pair in the set of image data pairs includes a given projection image representing a view of the anatomical structure of the object from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at a given gantry angle based on the given projection image; training a generative adversarial network (GAN) model based on the set of image data pairs for each gantry angle; and using the trained GAN model, predicting the aperture image of the MLC leaf positions for a desired gantry angle based on the projection images representing the view of the anatomical region of interest.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Priority declaration

[0002] This application claims the benefit of priority to U.S. Application Serial No. 16 / 784,919, filed on February 7, 2020, which is hereby incorporated by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure generally relate to determining machine parameters that guide radiation therapy performed by a radiation therapy treatment system. Specifically, the present disclosure relates to using deep learning techniques to determine machine parameters that define a treatment plan in a radiation therapy system. Background Art

[0004] Radiation therapy (or "radiotherapy") can be used to treat cancer or other diseases in mammalian (e.g., human and animal) tissue. One such radiation therapy technique is the Gamma Knife, by which a patient is irradiated with a large number of low-intensity gamma rays focused at a target (e.g., a tumor) with high intensity and precision. In another embodiment, radiation therapy is provided using a linear accelerator, whereby the tumor is irradiated with high-energy particles (e.g., electrons, protons, ions, high-energy photons, etc.). The placement and dosage of the radiation beam must be precisely controlled to ensure that the tumor receives the prescribed radiation, and the placement of the beam should minimize damage to surrounding healthy tissue, which is generally referred to as organs at risk (OARs). The radiation is called "prescribed" because the physician sends a predefined amount of radiation to the tumor and surrounding organs, similar to the prescription of a drug. Typically, ionizing radiation in the form of a collimated beam is directed to the patient from an external radiation source.

[0005] Specified or selectable beam energies may be used, for example, to deliver a diagnostic energy level range or a therapeutic energy level range. Modulation of the radiation beam may be provided by one or more attenuators or collimators (e.g., a multi-leaf collimator (MLC)). The intensity and shape of the radiation beam may be adjusted by collimation to avoid damaging healthy tissue adjacent to the target tissue (e.g., OAR) by conforming the projected beam to the contours of the target tissue.

[0006] The treatment planning process can include using a three-dimensional (3D) image of the patient to identify a target area (e.g., a tumor) and identify critical organs near the tumor. The creation of a treatment plan can be a time-consuming process in which the planner attempts to comply with various treatment goals or constraints (e.g., dose volume histogram (DVH), overlap volume histogram (OVH)), taking into account their respective importance (e.g., weights) in order to produce a clinically acceptable treatment plan. This task can be a time-consuming trial-and-error process that is complicated by the various OARs because the complexity of the process increases as the number of OARs increases (e.g., a dozen or more for head and neck treatments). OARs far from the tumor can be easily spared from radiation, while OARs close to the target tumor or overlapping with the target tumor may be difficult to spare.

[0007] Traditionally, for each patient, an initial treatment plan can be generated in an "offline" manner. The treatment plan can be formulated well before delivering radiotherapy, for example, using one or more medical imaging techniques. Imaging information can include images from, for example, X-rays, computed tomography (CT), magnetic resonance (MR), positron emission tomography (PET), single photon emission computed tomography (SPECT), or ultrasound. A health care provider (e.g., a physician) can use 3D imaging information indicating the patient's anatomical structure to identify one or more target tumors and OARs near the tumor. The health care provider can use manual techniques to depict the target tumor to receive a prescribed radiation dose, and the health care provider can similarly depict nearby tissues, such as organs, that are at risk of damage from radiotherapy. Alternatively or additionally, an automatic tool (e.g., ABAS provided by Elekta of Sweden) can be used to help identify or depict target tumors and organs at risk. A radiation therapy treatment plan ("treatment plan") can then be created using numerical optimization techniques that minimize an objective function consisting of clinical and dosimetric goals and constraints, such as maximum radiation dose, minimum radiation dose, and fractional radiation dose ("95% of the target should receive no less than 100% of the prescribed dose") to a fraction of the tumor volume, and similar measures for critical organs. The optimized plan includes numerical parameters that specify the direction, cross-sectional shape, and intensity of each radiation beam.

[0008] The treatment plan can then be subsequently executed by positioning the patient in the treatment machine and delivering the prescribed radiation therapy guided by the optimized planning parameters. Radiation therapy treatment plans may include dose "fractionation" to provide a sequence of radiation treatments over a predetermined period of time (e.g., 30 to 45 fractions per day), with each treatment comprising a specified fraction of the total prescribed dose. However, during treatment, the position of the patient and the position of the target tumor relative to the treatment machine (e.g., a linear accelerator - "linac") is very important to ensure that the target tumor is irradiated and healthy tissue is not irradiated. Summary of the invention

[0009] In some embodiments, a method, system, and computer-readable medium for generating one or more radiation therapy treatment plans are provided. The method, system, and computer-readable medium perform operations, the operations including: receiving a set of image data pairs for each gantry angle of a radiation therapy treatment machine, wherein each pair in the set of image data pairs includes a given projection image representing a view of an anatomical structure of an object from a given gantry angle and a given graphical aperture image of a multi-leaf collimator (MLC) leaf position at a given gantry angle based on the given projection image; training a generative adversarial network (GAN) model based on the set of image data pairs for each gantry angle; and using the trained GAN model, predicting an aperture image of an MLC leaf position for a desired gantry angle based on the projection image representing a view of an anatomical region of interest.

[0010] In some implementations, the GAN includes a conditional adversarial network (cGAN).

[0011] In some implementations, the GAN includes a cycle-consistent generative adversarial network (CycleGAN).

[0012] In some implementations, a given gantry angle of a first pair in a set of image data pairs differs by a predetermined amount from a given gantry angle of a second pair in the set of image data pairs.

[0013] In some implementations, the GAN is configured to train a generative model using a discriminative model; use adversarial training between the discriminative model and the generative model to establish values ​​applied by the generative model and the discriminative model; and the generative model and the discriminative model include respective convolutional neural networks.

[0014] In some implementations, adversarial training includes: training a generative model to generate a first synthetic graphic aperture image representation of an MLC blade position at a first gantry angle based on a projection image, wherein the projection image represents a view of the anatomical structure of the training object from the first gantry angle; and training a discriminative model to classify the first synthetic graphic aperture image as a synthetic training example graphic aperture image or a real training example graphic aperture image; and the output of the generative model is used to train the discriminative model, and the output of the discriminative model is used to train the generative model.

[0015] In some implementations, a cycle-consistent generative adversarial network (CycleGAN) including a generative model and a discriminative model is used to train the GAN, the generative model is a first generative model, and the discriminative model is a first discriminative model, and the CycleGAN also includes: a second generative model, which is trained to: process a given graphic aperture image representation of an MLC blade position at a given gantry angle as input based on a given pair in a set of image data pairs; and provide a synthetic projection image as an output, wherein the synthetic projection image represents a view of the anatomical structure of the training object from the given gantry angle; and a second discriminative model, which is trained to classify the synthetic projection image as a synthetic projection image or a real projection image.

[0016] In some implementations, CycleGAN includes a first part for training a first generative model, the first part being trained to: obtain a set of training projection images representing different views of a patient's anatomical structure from a previous treatment, the set of training projection images being paired with a training graphic aperture image corresponding to each of the different views, each of the training graphic aperture images being aligned with a corresponding training projection image in the training projection images; transmitting the set of training projection images to an input of the first generative model to output a first set of graphic aperture images; receiving the first set of graphic aperture images at an input of a first discriminative model to classify the first set of graphic aperture images as a set of synthetic training graphic aperture images or a set of real training graphic aperture images; and receiving the first set of graphic aperture images at an input of a second generative model to generate a first set of cycle projection images for calculating a cycle consistency loss.

[0017] In some implementations, CycleGAN includes a second part that is trained to: transmit a training graphic aperture image corresponding to each of the different views to an input of a second generation model to output a first set of synthetic projection images; receive the first set of synthetic projection images at an input of a second discriminative model to classify the first set of synthetic projection images as synthetic training projection images or real training projection images; and receive the first set of synthetic projection images at an input of the first generation model to generate a first set of cycle graphic aperture images for calculating a cycle consistency loss.

[0018] In some implementations, a cycle consistency loss is generated based on a comparison of a first set of cyclic projection images with a set of training projection images and a comparison of a first set of cyclic graphic aperture images with the training graphic aperture images; a first generative model is trained to minimize or reduce a first loss term, the first loss term representing an expectation of the difference between a plurality of synthetic graphic aperture images and their respective paired training graphic aperture images; and a second generative model is trained to minimize or reduce a second loss term, the second loss term representing an expectation of the difference between a plurality of synthetic projection images and their respective paired training projection images.

[0019] In some implementations, the operations include: obtaining radiotherapy treatment machine parameter information representing MLC leaf positions at gantry angles corresponding to each of views represented by projection images of a set of image data pairs and radiotherapy beam intensities corresponding to each of the projection images; generating training graphic aperture image representations based on the obtained radiotherapy treatment machine parameter information; and aligning each of the generated training graphic aperture image representations with the corresponding projection image.

[0020] In some implementations, the training graphic aperture image representations and corresponding projection images are two-dimensional images or three-dimensional images that include stacks of two-dimensional projection image and graphic aperture image pairs corresponding to an entire treatment fraction.

[0021] In some implementations, the operations include generating one or more radiation therapy treatment machine parameters from the aperture image predicted by the trained GAN model.

[0022] In some implementations, the operations include calculating a dose volume histogram or a three-dimensional dose distribution based on the generated one or more radiation therapy treatment machine parameters.

[0023] In some implementations, the one or more parameters include at least one of a gantry angle, an MLC jaw position, an MLC leaf position, or a radiotherapy beam intensity.

[0024] In some implementations, the operations include calculating dosimetry parameters based on the generated one or more radiation therapy treatment machine parameters.

[0025] In some implementations, a given projection image in a set of image data pairs is generated by ray tracing or Fourier reconstruction.

[0026] In some implementations, the operations include generating a plurality of two-dimensional projection images as projection images in a set of image data pairs, the plurality of two-dimensional projection images representing a plurality of two-dimensional views of the anatomical structure from a plurality of different gantry angles.

[0027] In some implementations, the plurality of two-dimensional projection images represent views of the anatomical structure at multiple angles across an arc of motion of a gantry of a radiation therapy treatment machine during treatment.

[0028] The above summary is intended to provide an overview of the subject matter of this patent application. It is not intended to provide an exclusive or exhaustive explanation of the subject matter of the present invention. Specific embodiments are included to provide further information about this patent application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In the accompanying drawings, which are not necessarily drawn to scale, similar reference numerals describe substantially similar components throughout the several views. Similar reference numerals with different letter suffixes represent different instances of substantially similar components. The accompanying drawings illustrate generally various embodiments discussed in this document by way of example and not by way of limitation.

[0030] Figure 1 An exemplary radiation therapy system according to some embodiments of the present disclosure is shown.

[0031] Figure 2A An exemplary radiation therapy system that may include a radiation therapy output configured to provide a therapeutic beam in accordance with some embodiments of the present disclosure is shown.

[0032] Figure 2B An exemplary system including a combined radiation therapy system and an imaging system, such as a cone beam computed tomography (CBCT) imaging system, according to some embodiments of the present disclosure is shown.

[0033] Figure 3 A partial cross-sectional view of an exemplary system including a combined radiation therapy system and an imaging system, such as a magnetic resonance (MR) imaging (MRI) system, is shown according to some embodiments of the present disclosure.

[0034] Figure 4A and Figure 4B Depicted are the differences between an exemplary MRI image and a corresponding CT image, respectively, according to some embodiments of the present disclosure.

[0035] Figure 5 Exemplary collimator configurations for shaping, directing or modulating the intensity of a radiation therapy beam are shown in accordance with some embodiments of the present disclosure.

[0036] Figure 6An exemplary Gamma Knife radiotherapy system is shown in accordance with some embodiments of the present disclosure.

[0037] FIG. 7A to FIG. 7D An exemplary flow chart for deep learning according to some embodiments of the present disclosure is shown.

[0038] Fig. 8A and Figure 8B Projection views from different directions are shown according to some embodiments of the present disclosure.

[0039] Fig. 9 An exemplary method for generating training data for deep learning according to some embodiments of the present disclosure is shown.

[0040] Fig.10 An exemplary method for training deep learning using projection images and graphical representations of blade positions according to some embodiments of the present disclosure is shown.

[0041] Fig.11A and Fig. 11B An exemplary data flow for training a machine learning model and using the machine learning model to generate radiation therapy device parameters according to some embodiments of the present disclosure is shown.

[0042] Fig.12 A method for generating radiation therapy device parameters from projection images of a view of a given anatomy from a given gantry angle using trained deep learning in accordance with some embodiments of the present disclosure is shown.

[0043] Fig.13 An exemplary block diagram of a machine that can implement one or more of the methods discussed herein is shown. DETAILED DESCRIPTION

[0044] In the following specific embodiments, reference is made to the accompanying drawings that form a part of the specific embodiments, and the specific embodiments are shown by way of illustrated specific embodiments in which the present disclosure can be practiced. These embodiments, also referred to herein as "examples", are described in sufficient detail to enable those skilled in the art to practice the present disclosure, and it should be understood that the embodiments may be combined or other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the present disclosure. Therefore, the following specific embodiments should not be considered in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0045] Intensity modulated radiation therapy (IMRT) and volumetric modulated arc therapy (VMAT) have become the standard of care in modern cancer radiotherapy. Creating an individual patient IMRT or VMAT treatment plan is a process of trial and error, which weighs the compromise between target dose and OAR protection from harm and adjusts plan constraints, the impact of which on plan quality metrics and dose distribution may be difficult to predict. In fact, the order in which plan constraints are adjusted may itself lead to dose differences. The quality of treatment plans usually depends on the subjective judgment of the planner, which depends on his / her experience and skills. Even the most skilled planners still cannot guarantee that their plans are close to the best possible, or whether a little or a lot of effort will produce a significantly better plan.

[0046] The present disclosure includes various techniques for improving and enhancing radiation therapy treatments by predicting radiation therapy treatment machine parameters (also referred to as control points) for delivering an expected dose distribution based on a patient's delineated anatomy. Technical benefits include reduced radiation therapy treatment plan creation time, and may yield a number of significant medical benefits (including improved accuracy of radiation therapy treatments, reduced exposure to unintended radiation, etc.). The disclosed techniques may be applicable to a variety of medical and diagnostic settings or radiation therapy treatment equipment and devices.

[0047] The present disclosure predicts or estimates control points by training a machine learning model, such as a generative adversarial network (GAN), based on training data consisting of pairs of control points (represented by graphic aperture images) and corresponding images of patients (represented by projection images). These data pairs are constructed and aligned so that each element of the image data corresponds to an element of the control point data. The disclosed machine learning method then determines a mapping or relationship between the two data domains. The learned model relates the image data and the control point data without explicit reference to the physics of radiation transmission or many of the details involved in the creation of the treatment plan from which the training data was extracted.

[0048] According to some embodiments, the present disclosure directly learns the parameters of a population of exemplary treatment plans for a common diagnosis (based on which a machine learning model is trained), and then uses the model to predict the treatment machine parameter settings required to deliver the expected dose distribution based solely on the delineated anatomy of a new patient. Treatment parameters include linear accelerator (linac) gantry angles, beam aperture shape through which the therapeutic radiation beam is projected at the target and the intensity of the beam, any parameters for delivering electron or particle therapy, etc. The present disclosure estimates the machine parameters that drive the treatment machine given the patient's images and the target and OAR delineation.

[0049] According to some embodiments, a deep neural network-based statistical learning (deep learning) is used to obtain a more detailed model of the connection between the patient's anatomy and treatment constraints and the resulting treatment machine parameter settings that will deliver the expected 3D dose distribution. Such a result is impossible for any analysis in the early plan quality analysis. By adopting a machine learning model, the present disclosure produces a model of the treatment planning process that encapsulates many (subjective) decisions made during plan creation and enables: generating plan templates to start plan creation, evaluating the possible quality of existing plans, helping treatment clinics that lack deep local IMRT / VMAT expertise, and fully automated treatment planning. By using deep learning to generate control points and implicit treatment plans for radiotherapy devices, the computational complexity of the overall plan optimization process is reduced, and the time required to create a treatment plan for a given patient is reduced.

[0050] Specifically, according to some embodiments, a machine learning model is trained to establish a relationship between a projection of a given image of a patient's anatomical structure corresponding to a given angle of a gantry of a radiotherapy device and a graphical aperture image representation of an MLC leaf position of the radiotherapy device. The machine learning model is trained by obtaining projection images representing different views of the patient's anatomical structure from previous treatments; and for each previous treatment in the previous treatments: obtaining control point information representing the MLC leaf position at the gantry angle corresponding to each of the different views and the radiotherapy beam intensity corresponding to each of the projection images, so as to generate a ground truth graphical aperture image representation based on the obtained control point information paired with the projection image. Subsequently, one or more parameters of the machine learning model are adjusted based on the paired generated training graphical aperture image representation and the corresponding projection image. Once trained, a new patient projection image is obtained, and the trained machine learning model is used to estimate the corresponding graphical aperture image representation of the MLC leaf position of the radiotherapy device. In addition to the MLC leaf position, other embodiments may also include a graphical representation of the intensity of the beam (e.g., a bar graph).

[0051] In some embodiments, control points for the radiotherapy device are then calculated based on the estimated graphic aperture image representation of the MLC leaf positions. That is, control points (e.g., beam intensity, gantry angle, and / or MLC leaf positions of individual leaves of the MLC) are generated using an inverse mapping function to achieve a beam corresponding to the estimated graphic aperture image representation of the MLC leaf positions. Specifically, the graphic aperture image representation of the MLC leaf positions identifies the resulting beam shape output by the radiotherapy device, and control points are generated using an inverse mapping function to provide the resulting beam shape represented by the graphic aperture image representation of the MLC leaf positions.

[0052] Figure 1 An exemplary radiation therapy system 100 for providing radiation therapy to a patient is shown. The radiation therapy system 100 includes an image processing device 112. The image processing device 112 may be connected to a network 120. The network 120 may be connected to the Internet 122. The network 120 may connect the image processing device 112 with one or more of the following: a database 124, a hospital database 126, an oncology information system (OIS) 128, a radiation therapy device 130, an image acquisition device 132, a display device 134, and a user interface 136. The image processing device 112 may be configured to generate a radiation therapy treatment plan 142 to be used by the radiation therapy device 130.

[0053] The image processing device 112 may include a memory device 116, an image processor 114, and a communication interface 118. The memory device 116 may store computer executable instructions, such as an operating system 143, a radiation therapy treatment plan 142 (e.g., an original treatment plan, an adjusted treatment plan, etc.), a software program 144 (e.g., artificial intelligence, deep learning, a neural network, radiation therapy treatment planning software), and any other computer executable instructions to be executed by the processor 114. In one embodiment, the software program 144 may convert a medical image in one format (e.g., MRI) into another format (e.g., CT) by generating a synthetic image such as a pseudo-CT image. For example, the software program 144 may include an image processing program that trains a prediction model for converting a medical image 146 of one modality (e.g., an MRI image) into a synthetic image of a different modality (e.g., a pseudo-CT image); alternatively, the trained prediction model may convert a CT image into an MRI image. In another embodiment, the software program 144 can align a patient image (e.g., a CT image or an MR image) with a dose distribution (also represented as an image) for the patient so that corresponding image voxels and dose voxels are appropriately associated by the network. In yet another embodiment, the software program 144 can replace a function of the patient image, such as a signed distance function or a processed version of the image that emphasizes some aspect of the image information. Such a function might emphasize edges or differences in voxel texture, or any other structural aspect useful for neural network learning. In another embodiment, the software program 144 can replace a function of the dose distribution that emphasizes some aspect of the dose information. Such a function might emphasize steep gradients around a target or any other structural aspect useful for neural network learning. The memory device 116 can store data, including medical images 146, patient data 145, and other data required to create and implement the radiation therapy treatment plan 142.

[0054] In yet another embodiment, the software program 144 can generate projection images for a collection of two-dimensional (2D) CT images or MR images and / or 3D CT images or MR images depicting anatomical structures (e.g., one or more targets and one or more OARs), the projection images representing different views of the anatomical structures from one or more gantry angles of a radiotherapy device. For example, the software program 144 can process a collection of CT images or MR images and create a stack of projection images depicting different views of the anatomical structures depicted in the CT images or MR images from various viewing angles of a gantry of the radiotherapy device. In particular, one projection image can represent a view of the anatomical structure at 0 degrees from the gantry, a second projection image can represent a view of the anatomical structure at 45 degrees from the gantry, and a third projection image can represent a view of the anatomical structure at 90 degrees from the gantry. The degree can be the position of the MLC relative to a particular axis of the anatomical structure depicted in the CT image or MR image. The axis can remain the same for each of the different degrees measured.

[0055] exist Fig. 8A A projection view of a simple ellipse 802 is schematically shown in FIG. Fig. 8A 802, the views are oriented relative to the center of the ellipse and capture the shape and extent of the ellipse 802 as seen from each angle (e.g., 0 degrees represented by view 803, 45 degrees represented by view 804, and 90 degrees represented by view 805). For example, when viewed from an angle of 0 degrees relative to the y-axis 807 of the ellipse 802, the view of the ellipse 802 is projected as view 803. For example, when viewed from an angle of 45 degrees relative to the y-axis 807 of the ellipse 802, the view of the ellipse 802 is projected as view 804. For example, when viewed from an angle of 90 degrees relative to the y-axis 807 of the ellipse 802, the view of the ellipse 802 is projected as view 805.

[0056] exist Figure 8B 8 shows a projection of the male pelvic anatomy relative to a set of original 3D CT images 801. Selected organs at risk and target organs are contoured in the 3D CT images 801 and their voxels are assigned constant values. Projection images 850 at selected angles (0, 45, and 90 degrees) around the central axis of the 3D CT image 801 can be obtained using the forward projection capability of a cone beam CT reconstruction program. The projection images can also be calculated by directly recreating the projection view geometry by ray tracing or by Fourier reconstruction as in computed tomography.

[0057] For example, the projection image may be computed by tracing the path of light as pixels in the image plane and simulating the effect of its encounter with virtual objects. In some implementations, the projection image is generated by tracing the path from an imaginary eye (either the MLC view or the eye view of the beam) to each pixel in the virtual screen and computing the color of the objects visible through it. Any other tomographic reconstruction technique may be utilized to generate the projection image from the eye view of the MLC beam of the anatomical structure depicted in the 3D CT image 801.

[0058] For example, a group (or set) of 3D CT images 801 may be used to generate one or more views of anatomical structures (e.g., bladder, prostate, seminal vesicle, rectum, first target, and second target) depicted in 3D CT images 801. These views may be from the perspective of the MLC including the gantry of the radiotherapy device, and for simplicity, reference is made to FIG. Figure 8B , these views are measured in degrees relative to the y-axis of the 3D CT image 801 and based on the distance between the anatomical structure depicted in the image and the MLC. Specifically, the first view 810 represents the projection of the 3D CT image 801 when the gantry is at 0 degrees relative to the y-axis and at a given distance from the anatomical structure depicted in the 3D CT image 801, the second view 820 represents the projection of the 3D CT image 801 when the gantry is at 45 degrees relative to the y-axis and at a given distance from the anatomical structure depicted in the 3D CT image 801, and the third view 830 represents the projection of the 3D CT image 801 when the gantry is at 90 degrees relative to the y-axis. Any other views may be provided, such as different views at each of the 360 ​​degrees around the anatomical structure depicted in the 3D CT image 801.

[0059] Return to reference Figure 1 In yet another embodiment, the software program 144 can generate graphic aperture image representations (sometimes referred to as graphic aperture images) of the MLC leaf positions at various gantry angles. These graphic aperture images are also referred to as aperture images or training graphic aperture images. In particular, the software program 144 can receive a set of control points for controlling a radiotherapy device to generate a radiotherapy beam. The control points can represent machine parameters such as beam intensity, gantry angle relative to the patient position, and leaf position of the MLC. Graphic images can be generated based on these control points to graphically represent the beam shape and intensity output by the MLC at each specific gantry angle. The software program 144 can align and pair each graphic image of the aperture at a specific gantry angle with the corresponding projection image generated at that angle. The image is aligned with the projection and scaled so that each projection image pixel is aligned with the corresponding aperture image pixel.

[0060] Fig. 9 An exemplary method for generating training data for deep learning according to some embodiments of the present disclosure is shown. Fig. 9 In , the image and control point parameters are converted into a 3D image volume. Fig. 9 At the top, the 3DCT image 801 is combined with Fig. 8A and Figure 8B The discussed approach is similarly converted into a stack of projection images 850 at selected angles around the central axis of the 3D CT image 801. Fig. 9 , the control point angle 910, aperture, and intensity corresponding to each angle of the gantry represented by the projection image 850 are recreated as a graphic aperture image 920. The shaded portion 922 shows the opening between the MLC left blade edge and the MLC right blade edge that allows radiation to pass through each graphic aperture image 920. The graphic aperture image 920 is aligned with the projection image 850 and scaled so that each projection pixel is aligned with the corresponding aperture pixel that radiates it. In some embodiments, a bar object can be presented for each graphic aperture image (e.g., at the bottom left of the graphic aperture image 920). The length of the bar object encodes the radiation intensity at the control point and is estimated together with the blade position.

[0061] Control point parameters 910 Represents the gantry angle, the MLC aperture at each gantry angle, and the radiation intensity at that angle. The aperture is depicted as a graphic aperture image 920. Each graphic aperture image 920 is assigned and paired with a corresponding anatomical projection image in the anatomical projection image 850 at the same gantry angle. Each shaded portion 922 represents the opening between pairs of opposing tungsten MLC blades (e.g., 532A to 532J), and it is these apertures that control the shape of the X-ray beam to cover the target to the prescribed radiation dose. The projection image 850 and the graphic aperture image 920 are scaled and aligned to ensure that each anatomical pixel in the projection image 850 is aligned with the corresponding aperture pixel that radiates the anatomical element. The construction of the control point data and the anatomical structure is represented as an aligned 3D image volume with a common size, pixel spacing, and origin.

[0062] Return to reference Figure 1In yet another embodiment, the software program 144 stores treatment planning software that includes a trained machine learning model, such as a trained GAN or a trained CycleGAN, to generate or estimate a synthetic graphic aperture image representation of MLC leaf positions at a given gantry angle for projection images representing views of anatomy from a given gantry angle. The software program 144 may also store functions that convert or calculate machine parameters or control points for a given type of machine to output a beam from the MLC that achieves the same or similar estimated graphic aperture image representation of MLC leaf positions. That is, the treatment planning software may output an image representing an estimated image of the beam shape and intensity for a given gantry angle and for a given projection image of the gantry at that angle, and the functions may calculate control points for a given radiotherapy device to achieve that beam shape and intensity.

[0063] In addition to the memory device 116 storing the software program 144, it is contemplated that the software program 144 may be stored on a removable computer medium, such as a hard drive, a computer disk, a CD-ROM, a DVD, an HD, a Blu-ray DVD, a USB flash drive, an SD card, a memory stick, or any other suitable medium; and that the software program 144 may be executed by the image processor 114 when downloaded to the image processing device 112.

[0064] The processor 114 may be communicatively coupled to the memory device 116, and the processor 114 may be configured to execute computer executable instructions stored on the memory device 116. The processor 114 may send the medical image 146 to the memory device 116 or receive the medical image 146. For example, the processor 114 may receive the medical image 146 from the image acquisition device 132 via the communication interface 118 and the network 120 to store in the memory device 116. The processor 114 may also send the medical image 146 stored in the memory device 116 to the network 120 via the communication interface 118 to be stored in the database 124 or the hospital database 126.

[0065] In addition, the processor 114 can create a radiation therapy treatment plan 142 using a software program 144 (e.g., treatment planning software) along with medical images 146 and patient data 145. The medical images 146 can include information such as imaging data associated with a patient's anatomical region, organ, or segmented data volume of interest. The patient data 145 can include information such as: (1) functional organ modeling data (e.g., serial organs versus parallel organs, appropriate dose response models, etc.); (2) radiation dose data (e.g., DVH information); or (3) other clinical information about the patient and treatment process (e.g., other surgeries, chemotherapy, previous radiation therapy, etc.).

[0066] In addition, the processor 114 may utilize the software program to generate intermediate data, such as, for example, updated parameters to be used by a machine learning model, such as a neural network model; or to generate intermediate 2D images or 3D images, which may then be subsequently stored in the memory device 116. The processor 114 may then subsequently transmit the executable radiation therapy treatment plan 142 to the radiation therapy device 130 via the communication interface 118 to the network 120, where the radiation therapy plan will be used to treat the patient with radiation. In addition, the processor 114 may execute the software program 144 to implement functions such as image conversion, image segmentation, deep learning, neural networks, and artificial intelligence. For example, the processor 114 may execute a software program 144 that trains or contours medical images; such a software program 144, when executed, may train a boundary detector or utilize a shape dictionary.

[0067] The processor 114 may be a processing device including one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc. More specifically, the processor 114 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processor 114 may also be implemented by one or more special-purpose processing devices such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc. As will be appreciated by those skilled in the art, in some embodiments, the processor 114 may be a special-purpose processor rather than a general-purpose processor. The processor 114 may include one or more known processing devices, such as those from Intel. TM Pentium manufactured TM 、Core TM , Xeon TM or series of microprocessors from AMD TM Made by Turion TM , Athlon TM 、Sempron TM 、Opteron TM FX TM 、Phenom TM series of microprocessors or any of a variety of processors manufactured by Sun Microsystems. Processor 114 may also include a graphics processing unit, such as a graphics processing unit from NvidiaTM Manufactured series, by Intel TM GMA, Iris TM series or by AMD TM Made by Radeon TM Processor 114 may also include an accelerated processing unit, such as an Intel TM Xeon Phi manufactured TM The disclosed embodiments are not limited to any type of processor that is otherwise configured to meet the computational requirements of identifying, analyzing, maintaining, generating and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. In addition, the term "processor" may include more than one processor (e.g., a multi-core design or multiple processors each having a multi-core design). The processor 114 may execute sequences of computer program instructions stored in the memory device 116 to perform various operations, processes, and methods that will be described in more detail below.

[0068] The memory device 116 can store medical images 146. In some embodiments, the medical images 146 can include one or more MRI images (e.g., 2D MRI, 3D MRI, 2D flow MRI, four-dimensional (4D) MRI, 4D volume MRI, 4D cine MRI, projection images, graphic aperture images, and pairing information between projection images and graphic aperture images, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), CT images (e.g., 2D CT, cone beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), one or more projection images representing views of anatomical structures depicted in MRI, synthetic CT (pseudo-CT) and / or CT images at different angles of the gantry relative to the patient axis, PET images, X-ray images, fluoroscopic images, radiotherapy portal images, SPECT images, computer-generated synthetic images (e.g., pseudo-CT images), aperture images, graphic aperture image representations of MLC leaf positions at different gantry angles, etc. In addition, the medical image 146 may also include medical image data, such as training images and training images, contour images, and dose images. In an embodiment, the medical image 146 may be received from the image acquisition device 132. Thus, the image acquisition device 132 may include an MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluoroscopy device, a SPECT imaging device, an integrated linear accelerator and MRI imaging device, or other medical imaging devices for obtaining medical images of a patient. The medical image 146 may be received and stored in any data type or any format type that the image processing device 112 may use to perform operations consistent with the disclosed embodiments.

[0069] The memory device 116 may be a non-transitory computer-readable medium such as a read-only memory (ROM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a flash memory, a random access memory (RAM), a dynamic random access memory (DRAM) such as a synchronous DRAM (SDRAM), an electrically erasable programmable read-only memory (EEPROM), a static memory (e.g., flash memory, a flash disk, a static random access memory), and other types of random access memory, a cache memory, a register, a CD-ROM, a DVD or other optical storage device, a cassette tape, other magnetic storage device, or any other non-transitory medium that may be used to store information including images, data, or computer-executable instructions that can be accessed by the processor 114 or any other type of computer device (e.g., stored in any format). The computer program instructions may be accessed by the processor 114, read from the ROM or any other suitable memory location, and loaded into the RAM for execution by the processor 114. For example, the memory device 116 may store one or more software applications. The software applications stored in the memory device 116 may include, for example, an operating system 143 for common computer systems and for software-controlled devices. In addition, the memory device 116 may store an entire software application or only a portion of a software application that may be executed by the processor 114. For example, the memory device 116 may store one or more radiation therapy treatment plans 142.

[0070] The image processing device 112 can communicate with the network 120 via a communication interface 118, which can be communicatively coupled to the processor 114 and the memory device 116. The communication interface 118 can provide a communication connection between the image processing device 112 and components of the radiation therapy system 100 (e.g., allowing data to be exchanged with external devices). For example, in some embodiments, the communication interface 118 can have appropriate interface circuitry to connect to the user interface 136, which can be a hardware keyboard, keypad, or touch screen through which a user can enter information into the radiation therapy system 100.

[0071] The communication interface 118 may include, for example, a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter (e.g., such as optical fiber, USB 3.0, thunderbolt, etc.), a wireless network adapter (e.g., such as a WiFi adapter), a telecommunication adapter (e.g., 3G, 4G / LTE, etc.), etc. The communication interface 118 may include one or more digital communication devices and / or analog communication devices that allow the image processing device 112 to communicate with other machines and devices (e.g., remotely located components) via the network 120.

[0072] The network 120 may provide the functionality of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), etc. For example, the network 120 may be a LAN or a WAN, which may include other systems S1 (138), S2 (140), and S3 (141). The systems S1, S2, and S3 may be the same as the image processing device 112, or may be different systems. In some embodiments, one or more of the systems in the network 120 may form a distributed computing / simulation environment that collaboratively performs the embodiments described herein. In some embodiments, one or more of the systems S1, S2, and S3 may include a CT scanner that obtains a CT image (e.g., a medical image 146). In addition, the network 120 may be connected to the Internet 122 to communicate with servers and clients remotely residing on the Internet.

[0073] Thus, the network 120 may allow for data transfer between the image processing device 112 and a number of different other systems and devices, such as the OIS 128, the radiation therapy device 130, and the image acquisition device 132. In addition, data generated by the OIS 128 and / or the image acquisition device 132 may be stored in the memory device 116, the database 124, and / or the hospital database 126. Data may be sent / received via the network 120 through the communication interface 118 for access by the processor 114, as desired.

[0074] The image processing device 112 can communicate with the database 124 through the network 120 to send / receive a variety of different types of data stored on the database 124. For example, the database 124 may include machine data (control points) including information associated with the radiation therapy device 130, the image acquisition device 132, or other machines related to radiation therapy. The machine data information may include control points such as radiation beam size, arc placement, duration of beam on and off, machine parameters, segmentation, MLC configuration, gantry speed, MRI pulse sequence, etc. The database 124 may be a storage device and may be equipped with an appropriate database management software program. Those skilled in the art will appreciate that the database 124 may include multiple devices located in a central manner or in a distributed manner.

[0075] In some embodiments, the database 124 may include a processor-readable storage medium (not shown). Although the processor-readable storage medium in the embodiments may be a single medium, the term "processor-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized database or a distributed database, and / or associated caches and servers) storing one or more sets of computer-executable instructions or data. The term "processor-readable storage medium" should also be considered to include any medium capable of storing or encoding an instruction set that is executed by a processor and causes the processor to perform any one or more of the methods of the present disclosure. Therefore, the term "processor-readable storage medium" should be considered to include, but is not limited to, solid-state memory, optical media, and magnetic media. For example, a processor-readable storage medium may be one or more volatile, non-transient, or non-volatile tangible computer-readable media.

[0076] The image processor 114 may communicate with the database 124 to read images into the memory device 116 or store images from the memory device 116 to the database 124. For example, the database 124 may be configured to store a plurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, raw data from MR scans or CT scans, Digital Imaging and Communications in Medicine (DIMCOM) data, projection images, graphic aperture images, etc.) received by the database 124 from the image acquisition device 132. The database 124 may store data to be used by the image processor 114 when executing the software program 144 or when creating the radiation therapy treatment plan 142. The database 124 may store data generated by a trained machine learning model, such as a neural network, including network parameters constituting a model learned by the network and resulting prediction data. The image processing device 112 can receive imaging data such as medical images 146 (e.g., 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, 3D MRI images, 4D MRI images, projection images, graphic aperture images, etc.) from a database 124, a radiotherapy device 130 (e.g., MRI-linac) and / or an image acquisition device 132 to generate a treatment plan 142.

[0077] In one embodiment, the radiation therapy system 100 may include an image acquisition device 132 that can acquire medical images of the patient (e.g., MRI images, 3D MRI, 2D flow MRI, 4D volume MRI, CT images, cone beam CT, PET images, functional MRI images (e.g., fMRI, DCE-MRI, and diffusion MRI), X-ray images, fluoroscopic images, ultrasound images, radiation therapy portal images, SPECT images, etc.). The image acquisition device 132 may be, for example, an MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound device, a fluoroscopic device, a SPECT imaging device, or any other suitable medical imaging device for obtaining one or more medical images of the patient. The images acquired by the image acquisition device 132 may be stored in the database 124 as imaging data and / or test data. By way of example, the images acquired by the image acquisition device 132 may also be stored in the memory device 116 by the image processing device 112 as medical images 146.

[0078] In one embodiment, for example, the image acquisition device 132 can be integrated with the radiation therapy device 130 into a single device (e.g., MRI-linac). Such an MRI-linac can be used, for example, to determine the location of a target organ or a target tumor in a patient's body to accurately guide the radiation therapy to a predetermined target according to the radiation therapy treatment plan 142.

[0079] The image acquisition device 132 can be configured to acquire one or more images of the patient's anatomical structure for a region of interest (e.g., a target organ, a target tumor, or both). Each image, typically a 2D image or slice, can include one or more parameters (e.g., 2D slice thickness, orientation, and position, etc.). In one embodiment, the image acquisition device 132 can acquire 2D slices at any orientation. For example, the orientation of the 2D slice can include a sagittal orientation, a coronal orientation, or an axial orientation. The processor 114 can adjust one or more parameters such as the thickness and / or orientation of the 2D slice to include the target organ and / or the target tumor. In one embodiment, the 2D slice can be determined based on information such as a 3D MRI volume. For example, in the case of using a radiotherapy device 130, such a 2D slice can be acquired by the image acquisition device 132 in "real time" while the patient is undergoing radiotherapy treatment, where "real time" means acquiring data at least in milliseconds or less.

[0080] The image processing device 112 can generate and store a radiation therapy treatment plan 142 for one or more patients. The radiation therapy treatment plan 142 can provide information about the specific radiation dose to be applied to each patient. The radiation therapy treatment plan 142 can also include other radiation treatment information, such as control points, including beam angles, gantry angles, beam intensities, dose histogram volume information, the number of radiation beams to be used during treatment, the dose per beam, etc.

[0081] The image processor 114 may be configured to process the treatment plan using a software program 144 such as treatment planning software (e.g., ELekta® manufactured by Elekta AB of Stockholm, Sweden). ) to generate a radiation therapy treatment plan 142. To generate the radiation therapy treatment plan 142, the image processor 114 can communicate with an image acquisition device 132 (e.g., a CT device, an MRI device, a PET device, an X-ray device, an ultrasound device, etc.) to access images of the patient and depict a target such as a tumor. In some embodiments, depiction of one or more OARs such as healthy tissue surrounding a tumor or close to a tumor may be required. Therefore, when the OAR is close to the target tumor, segmentation of the OAR can be performed. In addition, if the target tumor is close to the OAR (e.g., the prostate close to the bladder and rectum), by segmenting the OAR from the tumor, the radiation therapy system 100 can study not only the dose distribution in the target, but also the dose distribution in the OAR.

[0082] In order to depict the target organ or target tumor from the OAR, medical images such as MRI images, CT images, PET images, fMRI images, X-ray images, ultrasound images, radiotherapy portal images, SPECT images, etc. of patients undergoing radiotherapy can be obtained non-invasively by the image acquisition device 132 to reveal the internal structure of the body part. Based on the information from the medical image, the 3D structure of the relevant anatomical part can be obtained. In addition, during the treatment planning process, many parameters can be considered to achieve a balance between the effective treatment of the target tumor (e.g., so that the target tumor receives enough radiation dose for effective treatment) and the low radiation of the OAR (e.g., the OAR receives the lowest possible radiation dose). Other parameters that can be considered include the position of the target organ and the target tumor, the position of the OAR, and the movement of the target relative to the OAR. For example, the 3D structure can be obtained by contouring the target or contouring the OAR in each 2D layer or slice of the MRI image or CT image and combining the contours of each 2D layer or slice. The measurement may be performed manually (eg, by a physician, dosimetrist, or medical staff using a program such as MONACO manufactured by Elekta AB of Stockholm, Sweden). TM ) or automatically (e.g., using a program such as the Atlas-based automatic segmentation software ABAS manufactured by Elekta of Stockholm, Sweden TM and subsequent automatic segmentation software product ADMIRE TM ) Generate a contour. In some embodiments, the 3D structure of the target tumor or OAR can be automatically generated by the treatment planning software.

[0083] After the target tumor and (one or more) OARs have been located and delineated, a dosimetrist, physician, or medical staff member can determine the radiation dose to be applied to the target tumor and any maximum amount of dose that can be received by OARs close to the tumor (e.g., left and right parotid glands, optic nerves, eyes, lenses, inner ears, spinal cord, brainstem, etc.). After the radiation dose is determined for each anatomical structure (e.g., target tumor, OAR), a process known as inverse planning can be performed to determine one or more treatment planning parameters that will achieve the desired radiation dose distribution. Examples of treatment planning parameters include volume delineation parameters (e.g., which define target volumes, contour-sensitive structures, etc.), margins around target tumors and OARs, beam angle selection, collimator settings, and beam on time. During the inverse planning process, the physician may define dose constraint parameters that set boundaries on how much radiation an OAR may receive (e.g., defining a full dose to a tumor target and zero dose to any OAR; defining a dose to 95% of a target tumor; defining spinal cord, brain stem, and optic nerve structures to receive ≤45 Gy, ≤55 Gy, and <54 Gy, respectively). The results of the inverse planning may constitute a radiation therapy treatment plan 142 that may be stored in the memory device 116 or the database 124. Some of these treatment parameters may be related. For example, adjusting one parameter (e.g., weights for different objects, such as increasing the dose to a target tumor) in an attempt to change the treatment plan may affect at least one other parameter, which in turn may result in the development of a different treatment plan. Therefore, the image processing device 112 may generate a customized radiation therapy treatment plan 142 with these parameters so that the radiation therapy device 130 provides radiation therapy treatment to the patient.

[0084] In addition, the radiation therapy system 100 may include a display device 134 and a user interface 136. The display device 134 may include one or more display screens that display medical images, interface information, treatment plan parameters (e.g., projection images, graphic aperture images, contours, doses, beam angles, etc.), treatment plans, targets, positioning targets, and / or tracking targets, or any related information to a user. The user interface 136 may be a keyboard, keypad, touch screen, or any type of device through which a user can input information to the radiation therapy system 100. Alternatively, the display device 134 and the user interface 136 may be integrated into a computer such as a tablet computer (e.g., an Apple Lenovo Samsung etc.) device.

[0085] In addition, any and all components of the radiation therapy system 100 can be implemented as virtual machines (e.g., VMWare, Hyper-V, etc.). For example, a virtual machine can be software that acts as hardware. Therefore, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that together act as hardware. For example, the image processing device 112, the OIS 128, and the image acquisition device 132 can be implemented as virtual machines. Given the available processing power, memory, and computing power, the entire radiation therapy system 100 can be implemented as a virtual machine.

[0086] Figure 2A An exemplary radiation therapy device 202 is shown, which may include a radiation source such as an x-ray source or a linear accelerator, a bed 216, an imaging detector 214, and a radiation therapy output 204. The radiation therapy device 202 may be configured to emit a radiation beam 208 to provide treatment to a patient. The radiation therapy output 204 may include one or more attenuators or collimators, such as the following: Figure 5 The MLC is described in the illustrative embodiment of the present invention.

[0087] Return to reference Figure 2A , a patient can be positioned in region 212 and supported by treatment couch 216 to receive a radiation therapy dose according to a radiation therapy treatment plan. The radiation therapy output 204 can be mounted or attached to a gantry 206 or other mechanical support. When the couch 216 is inserted into the treatment area, one or more chassis motors (not shown) can rotate the gantry 206 and the radiation therapy output 204 around the couch 216. In one embodiment, the gantry 206 can continuously rotate around the couch 216 when the couch 216 is inserted into the treatment area. In another embodiment, the gantry 206 can rotate to a predetermined position when the couch 216 is inserted into the treatment area. For example, the gantry 206 can be configured to rotate the treatment output 204 around an axis ("A"). Both the couch 216 and the radiation therapy output 204 can be independently moved to other positions around the patient, for example, in a transverse direction ("T"), in a lateral direction ("L"), or as a rotation about one or more other axes, such as a rotation about a transverse axis (shown as "R"). A controller communicatively connected to one or more actuators (not shown) can control the movement or rotation of the couch 216 so as to appropriately position the patient within or outside of the radiation beam 208 according to the radiation therapy treatment plan. Both the couch 216 and the gantry 204 can be moved independently of each other in multiple degrees of freedom, which enables the patient to be positioned so that the radiation beam 208 can be precisely aimed at the tumor. The MLC can be integrated and included within the gantry 206 to deliver a specifically shaped radiation beam 208.

[0088] Figure 2A The coordinate system shown in , including axes A, T, and L, can have an origin located at an isocenter 210. The isocenter can be defined as the location where the central axis of the radiation beam 208 intersects the origin of the coordinate axes so as to deliver a prescribed radiation dose to a location on or in the patient. Alternatively, the isocenter 210 can be defined as the location where the central axis of the radiation beam 208 intersects the patient for various rotational positions of the radiation therapy output 204 positioned by the gantry 206 about axis A. As discussed herein, the gantry angle corresponds to the position of the gantry 206 relative to axis A, although any other axis or combination of axes can be referenced and used to determine the gantry angle.

[0089] The gantry 206 may also have an attached imaging detector 214. The imaging detector 214 is preferably located opposite the radiation source and, in one embodiment, may be located within the field of the radiation beam 208.

[0090] The imaging detector 214 can be mounted on the gantry 206 (preferably opposite the radiotherapy output 204) to maintain alignment with the treatment beam 208. The imaging detector 214 rotates about the rotation axis as the gantry 206 rotates. In one embodiment, the imaging detector 214 can be a flat panel detector (e.g., a direct detector or a scintillator detector). In this manner, the imaging detector 214 can be used to monitor the radiation beam 208, or the imaging detector 214 can be used to image the patient's anatomical structure, such as portal imaging. The control circuit system of the radiotherapy device 202 can be integrated into the system 100 or remote from the system 100.

[0091] In an illustrative embodiment, one or more of the couch 216, the treatment output 204, or the gantry 206 can be automatically positioned, and the treatment output 204 can establish the radiation beam 208 according to the specified dose for a particular treatment delivery instance. A treatment delivery sequence can be specified according to a radiation therapy treatment plan (e.g., using one or more different orientations or positions of the gantry 206, the couch 216, or the treatment output 204). The treatment deliveries can occur sequentially, but can be interleaved in the desired treatment site on or within the patient (e.g., at the isocenter 210). Thus, a prescribed cumulative dose of radiation therapy can be delivered to the treatment site while reducing or avoiding damage to tissue near the treatment site.

[0092] Figure 2BAn exemplary radiation therapy apparatus 202 is shown, which may include a combined linear accelerator and imaging system, such as a CT imaging system. The radiation therapy apparatus 202 may include an MLC (not shown). The CT imaging system may include an imaging X-ray source 218 that provides X-ray energy, such as in the kiloelectronvolt (keV) energy range. The imaging X-ray source 218 may provide a fan-shaped and / or cone-shaped radiation beam 208 directed toward an imaging detector 222, such as a flat panel detector. The radiation therapy apparatus 202 may be similar to that described with respect to Figure 2A The depicted system, for example, includes a radiation therapy output 204, a gantry 206, a couch 216, and another imaging detector 214 (eg, a flat panel detector). An X-ray source 218 may provide a relatively low energy diagnostic X-ray beam for imaging.

[0093] exist Figure 2B In an illustrative embodiment, the radiation therapy output 204 and the x-ray source 218 can be mounted on the same rotating gantry 206, rotated 90 degrees apart from each other. In another embodiment, two or more x-ray sources can be mounted along the periphery of the gantry 206, for example, each x-ray source has its own detector arrangement to provide diagnostic imaging at multiple angles simultaneously. Similarly, multiple radiation therapy outputs 204 can be provided.

[0094] Figure 3 An exemplary radiation therapy system 300 is depicted, which may include a combined radiation therapy device 202 and an imaging system such as a magnetic resonance (MR) imaging system (e.g., known in the art as an MR-linac) consistent with disclosed embodiments. As shown, the system 300 may include a couch 216, an image acquisition device 320, and a radiation delivery device 330. The system 300 delivers radiation therapy to a patient according to a radiation therapy treatment plan. In some embodiments, the image acquisition device 320 may correspond to Figure 1 The original image of the first modality can be obtained (for example, Figure 4A ) or a target image of a second modality (e.g., Figure 4B An image acquisition device 132 (such as the CT image shown in FIG. 1 ) is provided.

[0095] The bed 216 can support a patient (not shown) during a treatment phase. In some implementations, the bed 216 can move along a horizontal translation axis (labeled "I") so that the bed 216 can move a patient lying on the bed 216 into and / or out of the system 300. The bed 216 can also rotate around a central vertical rotation axis that is transverse to the translation axis. To allow such movement or rotation, the bed 216 can have motors (not shown) that enable the bed 216 to move in various directions and rotate along various axes. A controller (not shown) can control these movements or rotations to appropriately position the patient according to the treatment plan.

[0096] In some embodiments, the image acquisition device 320 may include an MRI machine for acquiring 2D MRI images or 3D MRI images of the patient before, during, and / or after a treatment session. The image acquisition device 320 may include a magnet 321 for generating a main magnetic field for magnetic resonance imaging. The magnetic field lines generated by the operation of the magnet 321 may extend substantially parallel to the central translation axis I. The magnet 321 may include one or more coils having axes extending parallel to the translation axis I. In some embodiments, one or more coils in the magnet 321 may be spaced apart so that a central window 323 of the magnet 321 is free of coils. In other embodiments, the coils in the magnet 321 may be sufficiently thin or have a reduced density so that the coils are substantially transmissive to radiation of a wavelength generated by the radiotherapy device 330. The image acquisition device 320 may also include one or more shielding coils that may generate magnetic fields of approximately equal amplitude and opposite polarity outside the magnet 321 so as to cancel or reduce any magnetic field outside the magnet 321. As described below, the radiation source 331 of the radiation delivery device 330 may be positioned in a region where the magnetic field is at least cancelled to first order or reduced.

[0097] The image acquisition device 320 may also include two gradient coils 325 and 326, which may generate a gradient magnetic field superimposed on the main magnetic field. The coils 325 and 326 may generate a gradient in the resulting magnetic field, which enables spatial encoding of protons so that the position of the protons may be determined. The gradient coils 325 and 326 may be positioned around a common central axis with the magnet 321 and may be displaced along the central axis. The displacement may create a gap or window between the coils 325 and 326. In an embodiment where the magnet 321 may also include a central window 323 between the coils, the two windows may be aligned with each other.

[0098] In some embodiments, the image acquisition device 320 may be an imaging device other than MRI, such as X-ray, CT, CBCT, spiral CT, PET, SPECT, optical tomography, fluorescence imaging, ultrasound imaging, radiation therapy portal imaging device, etc. As will be appreciated by one of ordinary skill in the art, the above description of the image acquisition device 320 relates to certain embodiments and is not intended to be limiting.

[0099] The radiation delivery device 330 may include a radiation source 331 such as an X-ray source or a linear accelerator and an MLC 332 (described below). Figure 5330). The radiation delivery device 330 may be mounted on a chassis 335. When the couch 216 is inserted into the treatment area, one or more chassis motors (not shown) may cause the chassis 335 to rotate about the couch 216. In one embodiment, the chassis 335 may continuously rotate about the couch 216 when the couch 216 is inserted into the treatment area. The chassis 335 may also have an attached radiation detector (not shown) that is preferably located opposite the radiation source 331 and wherein the rotation axis of the chassis 335 is positioned between the radiation source 331 and the detector. In addition, the device 330 may include control circuitry (not shown) for controlling, for example, one or more of the couch 216, the image acquisition device 320, and the radiation treatment device 330. The control circuitry of the radiation delivery device 330 may be integrated within the system 300 or remote from the system 300.

[0100] During a radiation therapy treatment session, a patient may be positioned on the couch 216. The system 300 may then move the couch 216 into a treatment area defined by the magnets 321, coils 325, 326, and chassis 335. The control circuitry may then control the radiation source 331, MLC 332, and chassis motors to deliver radiation to the patient through the window between the coils 325 and 326 according to the radiation therapy treatment plan.

[0101] Figure 2A , Figure 2B and Figure 3 An embodiment of a radiotherapy device configured to provide radiotherapy treatment to a patient is generally shown, including a configuration in which a radiotherapy output can rotate about a central axis (e.g., axis "A"). Other radiotherapy output configurations can be used. For example, the radiotherapy output can be mounted to a robotic arm or manipulator having multiple degrees of freedom. In yet another embodiment, the treatment output can be fixed (e.g., located in an area laterally separated from the patient), and a platform supporting the patient can be used to align the radiotherapy isocenter with a designated target site in the patient's body.

[0102] As mentioned above, by Figure 2A , Figure 2B and Figure 3 The described radiation therapy apparatus includes an MLC for shaping, directing or modulating the intensity of a radiation therapy beam to a designated target site within a patient. Figure 5An exemplary MLC 332 is shown that includes blades 532A-532J that can be automatically positioned to define an aperture that approximates a cross-section or projection of a tumor 540. Blades 532A-532J allow for modulation of the radiation therapy beam. Blades 532A-532J can be made of a material that is designated to attenuate or block the radiation beam in areas other than the aperture, depending on the radiation therapy plan. For example, blades 532A-532J can include a metal plate (e.g., comprising tungsten) with the long axis of the plate oriented parallel to the beam direction and having ends oriented orthogonal to the beam direction (e.g., Figure 2A 540 or other target site. This is compared to using a static collimator configuration, or compared to using an MLC 332 configuration determined specifically using "offline" treatment planning techniques. Radiotherapy techniques that use MLC 332 to produce a specified radiation dose distribution to a tumor or a specific area within a tumor may be referred to as IMRT. The resulting beam shape output using MLC 332 is represented as a graphic aperture image. That is, a given graphic aperture image is generated to represent the appearance (beam shape) of the beam and its intensity after passing through and being output by MLC 332.

[0103] Figure 6 630 (e.g., a Leksell Gamma Knife) according to some embodiments of the present disclosure. Figure 6 As shown in , during a radiotherapy treatment phase, a patient 602 can wear a coordinate frame 620 to keep a body part (e.g., head) of the patient undergoing surgery or radiotherapy stable. The coordinate frame 620 and the patient positioning system 622 can establish a spatial coordinate system that can be used when imaging the patient or during radiosurgery. The radiotherapy device 630 may include a protective shell 614 to enclose a plurality of radiation sources 612. The radiation source 612 may generate a plurality of radiation beams (e.g., beamlets) through a beam channel 616. The plurality of radiation beams may be configured to focus on an isocenter 210 from different directions. Although each individual radiation beam may have a relatively low intensity, when multiple doses from different radiation beams are accumulated at the isocenter 210, the isocenter 210 may receive a relatively high level of radiation. In some embodiments, the isocenter 210 may correspond to a target in surgery or treatment, such as a tumor.

[0104] Fig. 7AAn exemplary data flow for training and using a GAN suitable for generating a synthetic graphical aperture image representation of multi-leaf collimator (MLC) leaf positions from a received projection image representing a view of an anatomical structure of an object image is shown. For example, a generator model 760 trained to generate Fig. 7A The generator model 732 can be trained to achieve Figure 1 The processing functionality is provided as part of the image processor 114 in the radiation therapy system 100. Thus, the GAN model uses the data stream 750 (predicted) in Fig. 7A 7 is depicted as providing new data 770 (e.g., projection images representing views of the anatomy of a subject input image from a new patient) to a trained generator model 760, and using the trained generator model 760 to produce predictions or estimates of generator outputs (images) 734 (e.g., synthetic graphic aperture images corresponding to the input projection images representing views of the anatomy of the subject image). The projection images may be generated from one or more CT or MR images of the patient's anatomy representing views of the anatomy from a given angle of the gantry.

[0105] GAN includes two networks: a generating network (e.g., generator model 732), which is trained to perform classification or regression; and a discriminator network (e.g., discriminator model 740), which samples the output distribution of the generating network (e.g., generator output (image) 734) or the training graphic aperture image from the training image 723, and decides whether the sample is the same as or different from the true test distribution. The goal of this network system is to drive the generator network to learn the true model as accurately as possible, so that the discriminator network can only determine the correct origin of the generator sample with a probability of 50%, which is balanced with the generator network. The discriminator has access to the real situation, but the generator only accesses the training data through the detector's response to the generator output.

[0106] Fig. 7A The data flow of 710 shows the receipt of training input 710, which includes various values ​​of model parameters 712 and training data 720 (where such training images 723 include a set of projection images representing different views of the anatomy of the subject patient imaging data paired with a true graphical aperture image representation of multi-leaf collimator (MLC) leaf positions corresponding to the patient imaging data at different views, and conditions or constraints 726). The training input 710 is provided to GAN model training 730 to produce a trained generator model 760 used in GAN model use 750.

[0107] As part of GAN model training 730, the generator model 732 is used to represent the object image pair 722 (in Figure 7B723) and corresponding training projection images of views of anatomical structures of the MLC blade positions (also depicted as 723 in FIG. 7 ) are trained to generate and map segment pairs in the CNN. In this way, the generator model 732 is trained to generate a simulated or synthetic graphic aperture image of the MLC blade positions based on the input map as a generator output (image) 734. The discriminator model 740 decides whether the simulated graphic aperture image comes from the training data (e.g., a training graphic aperture image representation or a real graphic aperture image representation of the MLC blade positions) or from the generator (e.g., a synthetic graphic aperture image representation, as passed between the generator model 732 and the discriminator model 740). The discriminator output 736 is a decision of the discriminator model 740 indicating whether the received image is a simulated image or a real image, and is used to train the generator model 732. In some cases, the generator model 732 is trained using the discriminator on the generated images, and the generator model 732 is further trained based on the cycle consistency loss information. The training process results in back-propagation of weight adjustments 738 , 742 to improve the generator model 732 and the discriminator model 740 .

[0108] A batch of training data may be selected from the patient images and expected results during training of the generator model 732. The selected training data may include at least one projection image of the patient's anatomy representing a view of the patient's anatomy from a given gantry angle and a corresponding training graphic aperture image or a true graphic aperture image and / or machine parameter data at the given gantry angle. The selected training data may include multiple projection images of the patient anatomy representing views of the same patient anatomy from multiple isometric or non-isometric gantry angles (e.g., from 0 degrees, from 15 degrees, from 45 degrees, from 60 degrees, from 75 degrees, from 90 degrees, from 105 degrees, from 120 degrees, from 135 degrees, from 150 degrees, from 165 degrees, from 180 degrees, from 195 degrees, from 210 degrees, from 225 degrees, from 240 degrees, from 255 degrees, from 270 degrees, from 285 degrees, from 300 degrees, from 315 degrees, from 330 degrees, from 345 degrees, and / or from 360 degrees), and corresponding training graphic aperture images and / or machine parameter data at those different isometric or non-isometric gantry angles.

[0109] Thus, in this example, data preparation for GAN model training 730 requires a graphic aperture image representation of the MLC blade position, paired with a projection image representing a view of the anatomical structure of the object image (these may be referred to as training projection images representing a view of the anatomical structure of the object image at various gantry angles). That is, the training data includes a paired set of graphic aperture images at the same gantry angle as the corresponding projection images. In the example, the raw data includes pairs of projection images representing a view of the anatomical structure of the object at various gantry angles, and corresponding graphic aperture image representations of the MLC blade position at the corresponding gantry angle, which may be registered and resampled to a common coordinate system to produce paired anatomical structure derived images. The training data may include a plurality of these pairs of images for a plurality of patients at any number of different gantry angles. In some cases, the training data may include 360 ​​pairs of projection images and graphic aperture images, one pair of projection images and graphic aperture images for each angle of the gantry for each training patient. In some cases, 8 pairs may be included, each pair representing a different 45-degree view of the anatomical structure relative to the gantry.

[0110] The expected results may include an estimate or synthesized graphic aperture image representation of the leaves of the MLC, which may be used to calculate control points for generating beam shapes at corresponding gantry angles to define the delivery of radiation therapy to the patient. The control points or machine parameters may include at least one gantry angle, at least one multi-leaf collimator leaf position, and at least one aperture weight or strength.

[0111] In detail, in the GAN model, the generator (e.g., generator model 732) learns the distribution p on the data x G (x), from a distribution p Z (z) because the generator learns the mapping G(z; θ G ): p Z (z)→p G (x), where G is the layer with weights and bias parameters θ G The discriminator D(x; θ D ) (e.g., the discriminator model 740) maps the generator output to a binary scalar {true, false} if the generator output comes from the actual data distribution p 数据 (x) is true, and if the generator output comes from the generator distribution p G (x) is judged to be false. In other words, D(x) is x from p 数据 (x) rather than from p GIn another embodiment, paired training data can be used, where, for example, Y depends on (is dependent on) X. In this case, the GAN generator maps G(y|x; θ) from data domains X and Y. G ): X→Y represents that in the data domain X, the data x∈X represents the anatomical structure projection image, and in the domain Y, the data y∈Y represents the control point aperture corresponding to x. Here, the estimation of the aperture depends on its projection. Another difference from the straight GAN is that the projection image x is the generator input, rather than the random noise z input. For this embodiment, the discriminator is set up the same as above. In general, the generator model 732 and the discriminator model 740 are in a circular data flow, with the results of one fed into the other. The discriminator obtains training or generated images, and its output is used both to adjust the discriminator weights and to guide the training of the generator network.

[0112] Figure 7B An exemplary CNN model suitable for generating a synthetic graphic aperture image representation (or graphic aperture image) of an MLC blade position at one or more gantry angles according to the present disclosure is shown. Specifically, Figure 7B The model shown in depicts an arrangement of a "U-Net" deep CNN designed to generate an output dataset (output graphic aperture image 736) based on an input training set (e.g., paired projection images and graphic aperture image 723). The name is derived from the "U" configuration, and it is well known that this form of CNN model can produce pixel-level classification or regression results. In some cases, the first path to the CNN model includes one or more deformable offset layers and one or more convolutional layers, one or more convolutional layers including convolutions, batch normalization, and activations such as rectified linear units (ReLU) or one of their variants. The model generates a graphic aperture image 736 as an output dataset.

[0113] The left side of the model operation (the "encoding" operation 792) learns a set of features that the right side (the "decoding" operation 794) uses to reconstruct the output result. The U-Net has n stages consisting of conv / BN / ReLU (convolution / batch normalization / rectified linear unit) blocks 790, and each block has skip connections to enable residual learning. The block size is Figure 7B denoted by the “S” and “F” numbers in the image; the input image is SxS in size, and the number of feature layers is equal to F. The output of each block is the pattern of feature responses in an array of the same size as the image.

[0114] Along the encoding path, the block size is reduced by 1 / 2 or 2 at each stage -1 , while the feature size is increased by a factor of 2 as usual. The decoding side of the network is increased from S / 2 nStart scaling back up while adding on the feature content from the left at the same level; this is copy / connect data communication. The difference between the output image and the training version of the image drives the generator network weight adjustment via back propagation. For inference or testing, using the model, the input will be a single image or a collection of images of the projection image 723 (e.g., at different gantry angles), and the output will be the pattern aperture image 736 (e.g., pattern images corresponding to different gantry angles).

[0115] therefore, Figure 7B The representation of the model shows the training and prediction of a generative model suitable for performing regression rather than classification. Figure 7C An exemplary CNN model suitable for discriminating synthetic graphic aperture images according to the present disclosure is shown. Figure 7C The discriminator network shown in may include several blocks of convolutional layers, batch normalization layers, and ReLU layers configured with stride 2, as well as separate pooling layers. At the end of the network, there will be one or several fully connected layers to form 2D patches for discriminative purposes. Figure 7C The discriminator shown in FIG. 1 may be a patch-based discriminator configured to receive an input graphic aperture image (e.g., from Figure 7B ), classifies the image as real or fake, and provides the classification as an output detection result 744.

[0116] Consistent with embodiments of the present disclosure, therapeutic modeling methods, systems, devices, and / or processes based on such models include two stages: training a generative model using a discriminator / generator pair in a GAN; and making predictions using the generative model using a generator trained by the GAN. Various examples involving GANs and CycleGANs for graphic aperture image generation are discussed in detail in the following examples. It should be understood that other variations and combinations of types of deep learning models and other neural network processing methods can also be implemented using the present technology. In addition, although the following examples are discussed with reference to images and image data, it should be understood that the following networks and GANs can operate using other non-image data representations and formats.

[0117] In deep CNN training, the learned model is the values ​​of the layer node parameters θ (node ​​weights and layer biases) determined during training. Training uses the maximum likelihood or cross entropy between the training data and the model distribution. The cost function that expresses this relationship is:

[0118]

[0119] The exact form of the cost function for a particular problem depends on the nature of the model used. 模型 (y|x)=N(y:f(x;θ)) means a cost function such as:

[0120]

[0121] It includes a constant term that does not depend on θ. Therefore, Minimization produces a mapping f(x;θ) that approximates the distribution of the training data.

[0122] A useful extension of GANs is conditional GANs. Conditional adversarial networks learn a mapping G:{x,z}→y from an observed image x and random noise Z. Both adversarial networks consist of two networks: a discriminator (D) and a generator (G). The generator G is trained to produce outputs that cannot be distinguished from "real" or actual training images by an adversarially trained discriminator D, which is trained to be maximally accurate at detecting "fake" or outputs of G. Conditional GANs differ from unconditional GANs in that both the discriminator and generator reasoning depend on example images of type X.

[0123] The conditional GAN ​​loss function is:

[0124]

[0125] where G tries to minimize this loss, as opposed to adversarial D which tries to maximize this loss, or In addition, we want the generator G to make the difference between the training estimate and the actual training real image, Minimize, therefore, the full loss is the λ-weighted sum of the two losses The generator in conditional GAN ​​can be U-Net.

[0126] According to some embodiments, the generator of the conditional GAN ​​is trained to receive a projection image and generate a synthetic graphic aperture image corresponding to the projection image. The discriminator receives the synthetic graphic aperture image from the generator and is trained to distinguish the received synthetic graphic aperture image from a real image or a fake or synthetic image. The generator is trained to minimize the difference between the synthetic graphic aperture image and the corresponding training graphic aperture image. To this end, after the generator generates the synthetic graphic aperture image, the training graphic aperture image corresponding to the projection image is retrieved. For example, the projection image can be an image of a 2D or 3D projection of a CT image from a gantry angle of 15 degrees during a specific time interval of a treatment fractionation of a given patient previously treated. A graphic aperture image can be generated that represents the radiotherapy treatment machine parameters for the exact same patient, the same specific time interval, and the same gantry angle of 15 degrees. The graphic aperture image is paired with the projection image and retrieved as a training graphic aperture image. A comparison is made between the synthetic graphic aperture image generated for the projection image and this retrieved training graphic aperture image. The parameters of the generator are then updated based on this difference in an attempt to minimize this difference. The discriminator is similarly trained based on whether it correctly classifies the generated image as real or fake. The parameters of the generator are updated based on a total loss function that takes into account the discriminator error and the generator difference.

[0127] Subsequently, a second projection image for the same or different patient and at the same or different gantry angle is retrieved along with the paired training graphic aperture image. The generator receives the second projection image and generates a second synthetic graphic aperture image. A difference between the paired training graphic aperture image and the synthetically generated graphic aperture image is generated, and the parameters of the generator are again updated based on the difference. Once all or most of the training data has been processed and used to update the parameters of the generator, and / or once a specified number of epochs, or when the error (between the synthetic graphic aperture image and the training graphic aperture image) is within a threshold, the training ends and the parameters of the generator are output.

[0128] Another useful extension of GAN is to combine Fig.7D CycleGAN described. Fig.7D The training and use of CycleGAN 731 for generating a set of graphic aperture images (e.g., a set of synthesized or estimated graphic aperture images) from a received set of projection images according to some examples of the present disclosure are shown. CycleGAN 731 includes a first generator model 735, a second generator model 737, a first discriminator model 739A, and a second discriminator model 739B. The first generator model 735 includes a deformable offset layer and a convolution block, and the second generator model 737 includes a deformable offset layer and a convolution block. The two models 735 and 737 can each be a generator model 732 ( Fig. 7A ), and the first discriminator model 739A and the second discriminator model 739B can each be an implementation of the discriminator model 740 (e.g., as a classification type DCNN model). CycleGAN 731 can be divided into two parts, a first part 733A and a second part 733B.

[0129] The convolution blocks of each generator model 735 and 737 can be trained together or separately from the training of other generator and discriminator models. Specifically, the convolution blocks of the generator models 735 and 737 are trained to obtain the correct weights to perform their functions. The deformable offset layers can each be trained to coordinate offsets, resamples, and perform interpolation. The deformable offset layers can be trained together or separately from the training of the generator and discriminator models. The role of these offset layers is to change the original regular sampling grid from the upper convolution block, introduce coordinate offsets, and resample the image using interpolation. Alternatively or in addition, the deformable offset layer can be implemented using a spatial transformer, other types of convolution layers, and / or any other module that can store deformable structure information of an image. The number of offset layers in the deformable offset layer can vary based on the image size, the number of downsampling convolution layers, and other factors.

[0130] In an example, in the first part 733A, the first generator model 735 may be trained to receive a set of training projection images 723A (which may include one of the image pairs 722) and generate a set of corresponding first synthetic graphic aperture images as first generation results 736A. The first generator model 735 is referred to as G proj2aperture .

[0131] The first generation result 736A may be provided to the first discriminator model 739A. The first discriminator model 739A may classify the set of synthetic graphic aperture images into a set of real graphic aperture training images or a set of simulated graphic aperture training images, and provide the classification as a detection result 744A. The first generation result 736A and the detection result 744A may be fed back to the first generator model 735 and the first discriminator model 739A to adjust the weights implemented by the first generator model 735 and the first discriminator model 739A. For example, the first generation result 736A (e.g., a set of graphic aperture images generated by the first generator model 735) and the detection result 744A may be used to calculate the adversarial loss.

[0132] The first generated result 736A (e.g., a set of synthetic graphic aperture images) can also be provided to the second generator model 737 at the same time. The second generator model 737 can receive the first generated result 736A and generate a corresponding set of simulated projection images as output. The set of simulated projection images can be referred to as a set of loop projection images 741 and can be used to calculate the loop loss to adjust the weights of the first generator model 735 / the second generator model 737. The second generator model 737 that generates the set of first loop projection images 741 is referred to as G aperture2proj .

[0133] In the example, in the second part 733B, the second generator model 737 can be trained to receive a set of real training pattern aperture images 723B (which may include one of the image pairs 722) and generate a set of corresponding synthetic projection images (a set of synthetic or simulated projection images) as a first generation result 736B. The second generator model 737 that generates the first generation result 736B is the same generator as the generator used in the first part 733A.

[0134] The first generation result 736B may be provided to the second discriminator model 739B. The second discriminator model 739B may classify the set of synthetic projection images as a set of real projection training images or a set of simulated projection training images, and provide the classification as a detection result 744B. The first generation result 736B and the detection result 744B may be fed back to the second generator model 737 and the second discriminator model 739B to adjust the weights implemented by the second generator model 737 and the second discriminator model 739B. For example, the first generation result 736B (e.g., a set of synthetic projection images generated by the second generator model 737) and the detection result 744B may be used to calculate the adversarial loss.

[0135] The first generation result 736B (e.g., a set of synthetic projection images) may also be provided to the first generator model 735 at the same time. The first generator model 735 may receive the first generation result 736B and generate a corresponding cyclic graphic aperture image 743 as an output. The cyclic graphic aperture image 743 may be used to calculate the cyclic loss to adjust the weights of the first generator model 735 / the second generator model 737. The first generator model 735 that generates the cyclic graphic aperture image 743 is the same generator as the generator used in the first part 733A, and the second generator model 737 that generates the cyclic graphic aperture image 743 is the same generator as the generator used in the first part 733A.

[0136] In some examples, the “adversarial loss” may explain the classification loss of the first discriminator model 739A and the second discriminator model 739B. The first discriminator model 739A and the second discriminator model 739B may classify whether the synthesized image has a distribution similar to that of the real image. For the cycle consistency loss, the loss is calculated between each pair of sets of real projection images and sets of cyclic projection images, and between each pair of real graphic aperture images and cyclic graphic aperture images, respectively. For example, a first loss may be calculated between a set of projected training images 723A and a set of cyclic projection images 741, and a second loss may be calculated between a set of real training graphic aperture images 723B and a set of cyclic graphic aperture images 743. Both the set of cyclic projection images 741 and the cyclic graphic aperture images 743 may be obtained by performing forward and backward cycles. Each pair of sets of real projection images 723A and sets of cyclic projection images 741 may be in the same projection image set domain, and each pair of real training graphic aperture images 723B and cyclic graphic aperture images 743 may be in the same graphic aperture image domain. CycleGAN 731 may accordingly rely on the entire pool (or multiple) of real or true projection training images 723A and the entire pool (or multiple) of real training graphic aperture images 723B to generate synthetic graphic aperture images (a set of graphic aperture images), a set of synthetic projection images, a set of cyclic projection images 741, and cyclic graphic aperture images 743. Based on the “adversarial loss” and the “cycle consistency loss”, CycleGAN 731 may generate a clear synthetic graphic aperture image having an image resolution similar to that of a real graphic aperture image.

[0137] In some examples, a processor (e.g., of system 100) can apply image registration to register the real graphic aperture training image to the set of training projection images. This can create a one-to-one correspondence between the projection images at different gantry angles in the training data and the graphic aperture images at each of the different gantry angles. This relationship can be referred to as a paired or paired projection image and graphic aperture image.

[0138] In some implementations, CycleGAN 731 can be implemented to generate a set of graph aperture images according to an objective function including an adversarial loss term and a cycle consistency loss term. The CycleGAN network has two independent adversarial losses. Similar to the conditional GAN, the mapping G: X→Y and its associated discriminator D Y Given by Indicated loss.

[0139] The adversarial loss may be determined using the first generator model 735 / the second generator model 737 and the first discriminator model 739A / the second discriminator model 739B, and may be expressed as:

[0140] Fighting Loss:

[0141] L GANs =L GAN (G proj2aperture , D aperture ,x,y)+L GAN (G aperture2proj , D proj , y, x),

[0142] in:

[0143] L GAN (G proj2aperture , D aperture ,x,y)=E x~p(proj) [D ct (G proj2aperture (x))] 2 +E y~p(aperture) [D aperture (y)-1] 2 ,

[0144] L GAN (G aperture2proj , D proj , y, x) = E y~p(aperture) [D proj (G aperture2proj (y))] 2 +E x~p(proj) [D proj (x)-1] 2 .

[0145] D aperture It is the first discriminator model that determines whether an image is a set of real pattern aperture images or a set of synthetic pattern aperture images. proj It is a second discriminator model that determines whether an image is a set of real projection images or a set of synthetic projection images.

[0146] Cycle consistency loss is applied to both generators G proj2aperture and G aperture2proj This cycle consistency loss term can be determined using images 741 and 743 and can be expressed as:

[0147] L CYC (G proj2aperture , G aperture2proj , x, y)

[0148] =E x~p(proj) ||Gaperture2proj (G proj2aperture (x))-x||1

[0149] +E y~p(aperture) ||G proj2aperture (G aperture2proj (y))-y||1.

[0150] The role of minimizing the “cycle consistency” loss term is to maintain the original structure and avoid unnecessary structural deformation, and the role of minimizing the “adversarial” loss term is to learn a mapping or distribution transformation from one domain to its adversary domain. When the real sample of the target domain Y is input, the identity loss can be used to regularize the generator to be close to the identity mapping. The identity loss can be given by To express.

[0151] Therefore, the overall objective function can be expressed as: Total loss: L 总 =L GANs +λ 循环 ·L 循环 +λ 身份 ·L 身份 , where λ 循环 and λ 身份 The relative strength of the control loss, or it can be expressed as:

[0152] CycleGAN 731 may train the first generator 735 and the second generator 737 according to:

[0153]

[0154] This can be done using some common optimization algorithms used in the field of deep learning, such as stochastic gradient descent, Adam method, or other popular methods.

[0155] The previous examples provide examples of how a GAN, conditional GAN, or CycleGAN can be trained based on a set of image aperture images and a set of projection image pairs, specifically from image data in 2D or 3D image slices in multiple parallel or sequential paths. It should be understood that the GAN, conditional GAN, or CycleGAN can process other forms of image data (e.g., 3D or other multi-dimensional images). In addition, although only grayscale (including black and white) images are depicted by the accompanying drawings, it should be understood that color images can be generated and / or processed by the GAN, as discussed below in the examples.

[0156] As an example, a batch of training data can be used to train CycleGAN, the batch of training data including paired projection images and graphic aperture images of various patients. Specifically, the first batch of training data can include a set of projection images of a first patient, which is used to generate a treatment plan for treating the first patient in a first treatment fraction. Based on the treatment plan, a set of corresponding graphic aperture images can be generated. Each graphic aperture image is associated with the first patient and is associated with the first treatment fraction. Each graphic aperture image is paired with a projection image, such that a projection image for a given gantry angle at a given time point in a treatment fraction is associated with a corresponding graphic aperture image.

[0157] In some implementations, a first batch of training data is processed by CycleGAN to train a first generator and a second generator and a first discriminator and a second discriminator. Specifically, a first generator in a first part of CycleGAN receives a first projection image corresponding to a first gantry angle and a first portion of a treatment fraction for a first patient. The first generator generates a first synthetic graphic aperture image corresponding to the first projection image. The first synthetic graphic aperture image is provided to a first discriminator, which attempts to distinguish whether the first synthetic graphic aperture image is a real or training graphic aperture image or a synthetic graphic aperture image. A first training graphic aperture image corresponding to the first projection image for the first gantry angle and the first portion of a treatment fraction for the first patient is retrieved and used in a loss function to train the first generator and the first discriminator.

[0158] The first synthetic graphic aperture image is provided to the second generator simultaneously or sequentially. The second generator generates a cyclic projection image corresponding to the first synthetic graphic aperture. The cyclic projection image is then compared with the first projection image to generate a cyclic loss. The first generator, the second generator, and the first discriminator are trained based on a loss function including the cyclic loss.

[0159] In parallel with receiving the first portion of the first projection image, a second generator in a second portion of the CycleGAN receives a first training pattern aperture image corresponding to the first projection image. The second generator generates a first synthetic projection image corresponding to the first training pattern aperture image. The first synthetic projection image is provided to a second discriminator that attempts to distinguish whether the first synthetic projection image is a real or training projection image or a synthetic projection image. First projection images corresponding to the first training pattern aperture image for a first portion of a treatment fraction for a first gantry angle and a first patient are retrieved and used in a loss function to train the second generator and the second discriminator.

[0160] The first synthetic projection image is provided to the first generator simultaneously or sequentially. The first generator generates a cyclic graph aperture image corresponding to the first synthetic projection image. The cyclic graph aperture image is then compared with the first training graph aperture image to generate a cyclic loss. The first generator, the second generator, and the second discriminator are trained based on a loss function including the cyclic loss.

[0161] The first and second parts then retrieve another batch of training data and perform similar operations on another paired set of projection images and graphic aperture images of another patient for whom a treatment plan was previously generated. Once a specified number of epochs or training data has been processed, and / or when the loss function reaches a specified threshold, training is complete and the first generator is output and used to generate a graphic aperture image for a new set of projection images corresponding to a new patient or subject.

[0162] Fig.10 An embodiment of a method 1000 for training a GAN, such as a CycleGAN, for determining a set of machine parameters or a graphic aperture image for a particular gantry angle based on at least one medical image projection depicting a view of a patient's anatomy from a gantry angle. The GAN may receive a collection of medical images representing different image projections depicting views of the patient's anatomy from different gantry angles. The projection images may be generated from CT images, MRI images, synthetic CT, and / or PET images. The GAN may also receive a function of a corresponding anatomical voxel labeling map and a labeled object signed distance map. Machine parameters or graphic aperture images may then be determined for each collection of medical images for the various gantry angles depicted in the projection images. In one embodiment, the machine parameters or graphic aperture image may be received along with the collection of medical projection images.

[0163] The machine parameters may include at least one gantry angle, at least one multi-leaf collimator leaf position, and at least one aperture weight or strength. The machine parameters may be expressed by In other words, Y can represent a set of K machine parameter data objects, φ can represent the gantry angle, The nth left multi-leaf collimator blade position and the right multi-leaf collimator blade position under the angle φ can be represented, and y φThe aperture weight or strength for the gantry angle φ may be represented. To start network training, the iteration index may be set to an initial value of zero. A batch of training data may be formed by a subset of the received set of medical image projections and the corresponding graphic aperture images for the gantry angles represented by the medical image projections. In particular, a batch of medical image projections may represent the anatomy of a first training subject from a plurality of gantry angles, and the training data includes graphic aperture images generated based on machine parameters used at each of the plurality of gantry angles in treating the first training subject. Another batch of medical image projections may represent the anatomy of a second training subject from a plurality of gantry angles, and the training data includes graphic aperture images generated based on machine parameters used at each of the plurality of gantry angles in treating the second training subject. [Replace old Word math graphics with new Word math graphics.]

[0164] The batch of training data can be provided to the GAN, and the GAN parameters can be updated based on the batch of training data. For a given set of received media image projections, the GAN can provide a set of output machine parameters or graphic aperture images based on the current parameters of the GAN. A comparison can be made between the sets of output machine parameters or graphic aperture images that correspond to the set of received medical image projections (e.g., training machine parameters or graphic aperture images) in the batch of training data. A corresponding set of errors is determined based on the comparison, where each error value can be the difference between an estimated machine parameter or graphic aperture image and a corresponding training machine parameter or graphic aperture image. The parameters of the GAN can then be updated based on the corresponding errors, for example by using back propagation.

[0165] In one embodiment, the parameters θ of the GAN may be updated to minimize or reduce a cost function, such as the cost function where Y can represent the machine parameters or the graphical aperture image determined by GAN, where Y * can represent the known machine parameters or graphical aperture image corresponding to the batch of training data, and where Θ * It can be expressed that GAN corresponds to Y and Y * The least squared difference between the parameters (e.g., layer node weights and biases as described above).

[0166] In one embodiment, the cost function may include a probability function, where the cost function may be expressed as 训练 = argmax Θ P(Y|X; Θ) or Θ 训练 =argnax Θ ∑ t∈r logP(Y t |Xt ; Θ) to determine the parameters of GAN, where Θ 训练 may denote the parameters of a fully trained GAN, and X may denote a set of medical image projections at different gantry angles obtained as a function of an anatomical structure voxel labeling map or a labeled object signed distance map.

[0167] After updating the parameters of the GAN, the iteration index may be increased by a value of 1. The iteration index may correspond to the number of times the parameters of the GAN have been updated. A stopping criterion may be calculated, and if the stopping criterion is met, the GAN model may be saved in a memory such as the memory device 116 of the image processing device 112, and training may be stopped. If the stopping criterion is not met, training may be continued by obtaining another batch of training images from the same training object or another training object. In one embodiment, the stopping criterion may include the value of the iteration index (e.g., the stopping criterion may include whether the iteration index is greater than or equal to a determined maximum number of iterations). In one embodiment, the stopping criterion may include the accuracy of the set of output machine parameters or graphic aperture images (e.g., the stopping criterion may include whether the difference between the set of output machine parameters or graphic aperture images and the machine parameters or graphic aperture images corresponding to the set of received medical image projections in the batch of training data is less than a threshold). In one embodiment, the threshold may correspond to the asymptotic minimum of all errors determined. In one embodiment, the machine parameters or graphic aperture images may be presented to the GAN in the form of an image with a fixed format that specifies, for example, aperture, angle, and intensity values. In one embodiment, the patient images may be pooled with the machine parameters and may be presented as an array of real numbers.

[0168] The motivation for using machine learning models to predict machine parameters or control points is to accelerate treatment plan calculations. Currently, conventional treatment plans are planned from CT images or MR images through anatomical structures and target depictions. Modern IMRT and VMAT plans use images and depicted structures to define 3D dose distributions and machine parameters for delivering the dose distributions. The resulting clinical plans based on CT images or MR images are usually generated after two stages of numerical optimization. In the case of MR images, additional steps are provided to convert MR images into synthetic CT images. CT image pixel intensity is a function of the X-ray absorption of tissues and provides necessary physical information for treatment planning procedures. MR-based plans create synthetic CT images to provide the X-ray absorption data for the plan. The first stage (e.g., Fluence Map Optimization or "FMO") produces an idealized dose distribution that meets the requirements of the planner. The second stage (e.g., segment shape, weight optimization) creates machine parameters for a given linear accelerator and MLC to produce a 3D dose distribution in the patient's body that conforms to the fluence map distribution as much as possible.

[0169] The disclosed embodiments improve the quality and speed of creating treatment plans using control point predictions (e.g., DL plan estimates). The GAN workflow contemplates GAN segmentation of anatomical structures and targets and the generation of synthetic CT for MR images. The GAN trains a model that generates control points based on very good treatment plans that include two optimization stages. The control points estimated by the trained GAN in a sense contain information from both planning stages. One way in which the planning process can be shortened includes eliminating the fluence map stage all together and using the control point estimates as input to the second stage (segmented shape, weight optimization). This speeds up the second stage calculations, thereby speeding up the planning process.

[0170] The challenge with anatomy-based control point prediction is that the anatomy and the control points have fundamentally different common representations. The anatomy is described by rectilinear medical images of various modalities, while the control points are vectors of real parameters. Furthermore, even though the apertures of the control points are represented by graphical representations in the images, the orientations of the apertures do not correspond to any of the standard 2D views or 3D views of the anatomy. As the linear accelerator travels in an arc around the patient, the view of the anatomy at any moment is a projection image of the anatomy, equivalent to a planar radiograph of the anatomy at that angle. According to some embodiments, the control point aperture data is reformatted, converted into an aperture image, and aligned with the projection of the anatomy at the corresponding angle.

[0171] Once trained, the GAN can compute estimates of the graphic aperture image 1010 for one or more gantry angles for a new patient (e.g., a patient that does not contribute a known image or image function used in the training process) given only the projection image 850 of the new patient at one or more gantry angles. In one embodiment, the GAN may include a GAN for regression. The GAN may be stored in a memory such as the memory device 116 of the image processing device 112. The GAN may be composed of (possibly many) convolutional layers organized in blocks of layers, where each block generally operates on a successively lower resolution version of the image and control point data. The projection image 850 data and the corresponding graphic aperture image 920 data may be introduced into the GAN through a convolutional layer with a kernel of width one. Each convolutional layer block may consist of at least one convolutional layer and a nonlinear layer (e.g., a rectified linear unit or ReLU layer) that may apply a nonlinear function to the output of the convolutional layer. Each block may also contain a batch normalization layer, a scaling layer, or other layers that may be determined to provide the most accurate estimate of the desired graphic aperture image value. The last layer in the last block of each set can be a pooling layer that can downsample the convolution output data by a certain amount (e.g., half) and can take the maximum of the relevant layer node values ​​in the output convolution layer during a process called max pooling. The changing shape of each set of convolution blocks can indicate that the spatial resolution of the image data can be reduced (block height) while the number of parameters per layer can be increased (block width). The arrangement of layers and the composition of layer combinations can preserve the information content of the DCNN.

[0172] During network training, the graph aperture image output using the projection image 850 can be compared to the correct graph aperture image 920 (e.g., training graph aperture image), and the difference (e.g., error) can be used to correct network parameters. The correction process is called back propagation. Network parameters can be associated with layer nodes whose values ​​can be reset during back propagation. Fig.10 The layer node weight coefficients and bias terms are summarized as Θ in . Network training produces Fig.10 The network model or function f(X:Θ) shown in . The model depends on both the input training images X and the network parameters Θ. Once trained, the network model can produce a graph aperture image 1010 output at the output layer for a patient that was not used to train the network model. Graph aperture image 1010 is a graph aperture image estimate for a new patient.

[0173] Fig.11AA flow chart of a process 1100A of exemplary operations for training a first generative model adapted to output a set of synthetic graphic aperture images from a set of input projection images is shown. Process 1100A is shown from the perspective of a radiation therapy system 100 that uses a GAN or CycleGAN to train and utilize the first generative model, as discussed in the previous examples. However, the corresponding operations may be performed by other devices or systems (including in an offline training or validation setting separate from a particular image processing workflow or medical treatment).

[0174] As shown, the first stage of the flowchart workflow begins with operations (1110, 1120) to establish parameters for training and model operations. Process 1100A begins with an operation (operation 1110) of receiving (e.g., obtaining, extracting, identifying) training image data and an operation (operation 1120) of receiving (e.g., obtaining, extracting, identifying) constraints or conditions for training. In an example, the training image data may include image data from multiple human subjects, involving specific conditions, anatomical features, or anatomical regions, such as paired projection images and true graphic aperture images of target regions of the subject and various gantry angles. In addition, in an example, the constraints may involve imaging devices, treatment devices, patients, or medical treatment considerations. In an example, these constraints may include adversarial losses and cycle consistency-based losses.

[0175] The second stage of process 1100A continues with training operations, including adversarial training of a generative model and a discriminative model in a generative adversarial network (operation 1130). In an example, the adversarial training includes training a first generative model to generate a simulated graphic aperture image by processing a set of input projection images (operation 1142). The set of first simulated graphic aperture images is provided to the discriminative model to train the discriminative model to classify the generated set of simulated graphic aperture images as simulated training data or real training data (operation 1144). In addition, in this adversarial training, the output of the first generative model is used to train the discriminative model, and the output of the discriminative model is used to train the first generative model. As combined Fig. 11B As described in process 1100B, the set of first simulated graphic aperture images are respectively passed to the second generative model. The second generative model generates a circular projection image from the set of first synthetic graphic aperture images. The circular projection image is used in the loss term to train the first generative model. The training of the second generative model is shown as Fig. 11B Operations 1146 and 1148 in .

[0176] In various examples, the first generative model and the discriminative model include respective convolutional neural networks. In other examples, the generative adversarial network is a cyclic generative adversarial network in which multiple generative models and adversarial models are employed, and the output from one generative model is provided as input to a second generative model.

[0177] Process 1100A continues by outputting the first generative model for use in generating a set of synthetic graphic aperture images (operation 1150), because the first generative model is suitable for generating a set of graphic aperture images based on a set of input projection images of the object. In some cases, the output of the second generative model is used to generate a set of synthetic projection images ( Fig. 11B Operations 1152 and 1162 in FIG.

[0178] Process 1100A continues by utilizing the trained first generative model to generate a set of synthetic graphic aperture images based on the set of input projection images of the object (operation 1160).

[0179] Process 1100A ends with a final stage of implementing an update to the first generative model, including updating the generative model based on the additional training data (operation 1170) and outputting the updated trained first generative model (operation 1180). In some cases, the final stage is implementing an update to the second generative model, including updating the second generative model based on the additional training data (operation 1172, Fig. 11B ). In various examples, the update can be generated in conjunction with the receipt of additional training image data and constraints (e.g., in a manner similar to operations 1110, 1120), or the performance of additional adversarial training (e.g., in a manner similar to operations 1130, 1142, 1144). In other examples, the generative model can be specifically updated based on approval, change, or use of the set of graphic aperture images (e.g., caused by modification, verification, or change of image data by a medical professional). The flowchart ends with using the updated trained generative model (operation 1190), for example, the flowchart can be executed in using the updated generative model in a subsequent radiation therapy treatment.

[0180] Fig. 11B A flow chart of a process 1100B of exemplary operations for training a second generative model adapted to output a set of synthetic projection images based on a set of input graphic aperture images (e.g., real or training graphic aperture images). Process 1100B is shown from the perspective of a radiation therapy system 100 that uses a GAN or CycleGAN to train and utilize the second generative model, as discussed in the previous examples. However, the corresponding operations may be performed by other devices or systems (including in an offline training or validation setting separate from a particular image processing workflow or medical treatment).

[0181] As shown, the first stage of the flowchart workflow begins with operations (1110, 1120) to establish parameters for training and model operations. Process 1100B begins with an operation (operation 1110) of receiving (e.g., obtaining, extracting, identifying) training image data and an operation (operation 1120) of receiving (e.g., obtaining, extracting, identifying) constraints or conditions for training. In an example, the training image data may include image data from multiple human subjects, involving specific conditions, anatomical features, or anatomical regions, such as paired projection images and true graphic aperture images of target regions of the subject and various gantry angles. In addition, in an example, the constraints may involve imaging devices, treatment devices, patients, or medical treatment considerations. In an example, these constraints may include adversarial losses and cycle consistency-based losses.

[0182] The second stage of process 1100B continues the training operation, including adversarial training of the generative model and the discriminative model in the generative adversarial network (operation 1130). In the example, the adversarial training includes training the second generative model to generate simulated projection images by processing the set of input graphic aperture images (operation 1146). The set of the first simulated set projection images is provided to the discriminative model to train the discriminative model to classify the generated set of simulated projection images as simulated training data or real training data (operation 1148). In addition, in this adversarial training, the output of the second generative model is used to train the discriminative model, and the output of the discriminative model is used to train the second generative model. As combined Fig. 11B As described in process 1100B, the set of first simulated projection images are respectively passed to the first generator model. The first generator model generates a cyclic pattern aperture image from the set of first synthetic projection images. The cyclic pattern aperture image is used in the loss term to train the second generator model.

[0183] Process 1100B continues by outputting the second generative model for use in generating the set of synthetic projection images (operation 1152), as the second generative model is suitable for generating the set of projection images based on the set of input graphical aperture images of the object.

[0184] Process 1100B continues by utilizing the trained second generative model to generate a set of synthetic projection images based on the set of input graphic aperture images of the object (operation 1162).

[0185] Process 1100B ends with a final phase of implementing an update to the second generative model, including updating the second generative model based on the additional training data (operation 1172) and outputting the updated trained second generative model. In various examples, the update can be generated in conjunction with receiving additional training image data and constraints (e.g., in a manner similar to operations 1110, 1120), or performing additional adversarial training (e.g., in a manner similar to operations 1130, 1146, 1148). In other examples, the second generative model can be specifically updated based on approval, change, or use of the collection of projection images (e.g., caused by modification, verification, or change of image data by a medical professional).

[0186] Fig.12 1 is a flow chart illustrating example operations of the image processing device 112 when performing the process 1200 according to an example embodiment. The process 1200 may be embodied in computer-readable instructions for execution by one or more processors, such that the operations of the process 1200 may be performed in part or in whole by the functional components of the image processing device 112; therefore, the process 1200 is described below by reference to examples thereof. However, in other embodiments, at least some of the operations of the process 1200 may be deployed on various other hardware configurations. Therefore, the process 1200 is not intended to be limited to the image processing device 112, but may be implemented in whole or in part by any other component. Some or all of the operations of the process 1200 may be parallel, out of order, or omitted entirely.

[0187] At operation 1230, the image processing device 112 receives a first projection image representing a view of an anatomical structure of a subject from a first gantry angle of a radiation therapy treatment machine.

[0188] At operation 1250, the image processing device 112 generates a first graphic aperture image representation of the MLC blade position at a first gantry angle based on the first projection image using the generative model. The generative model is trained based on a generative adversarial network to process projection images representing different views of the patient's anatomy as input and provide corresponding graphic aperture image representations of the MLC blade position at different gantry angles corresponding to the different views as output.

[0189] At operation 1260, the image processing device 112 generates one or more radiation therapy device parameters based on the generated first graphical aperture image representation.

[0190] Fig.13A block diagram of an embodiment of a machine 1300 is shown, on which one or more of the methods discussed herein can be implemented. In one or more embodiments, one or more items of the image processing device 112 can be implemented by the machine 1300. In an alternative embodiment, the machine 1300 operates as a standalone device, or can be connected (e.g., networked) to other machines. In one or more embodiments, the image processing device 112 can include one or more of the items of the machine 1300. In a networked deployment, the machine 1300 can operate in the capacity of a server or client machine in a server-client network environment, or operate as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by the machine. Further, while a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0191] The example machine 1300 includes processing circuitry or processors 1302 (e.g., a CPU, a graphics processing unit (GPU), an ASIC, circuitry (e.g., one or more transistors, resistors, capacitors, inductors, diodes, logic gates, multiplexers, buffers, modulators, demodulators, radios (e.g., transmitting or receiving radios or transceivers)), sensors 1321 (e.g., transducers that convert one form of energy (e.g., light, heat, electricity, mechanical or other energy) to another form of energy), the like or a combination thereof), a main memory 1304, and a static memory 1306 that communicate with each other via a bus 1308. The machine 1300 (e.g., a computer system) may also include a video display device 1310 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The machine 1300 also includes an alphanumeric input device 1312 (e.g., a keyboard), a user interface (UI) navigation device 1314 (e.g., a mouse), a disk drive or mass storage unit 1316, a signal generating device 1318 (e.g., a speaker), and a network interface device 1320.

[0192] The disk drive unit 1316 includes a machine-readable medium 1322 on which is stored one or more sets of instructions and data structures (e.g., software) 1324 that embody or are utilized by any one or more of the methods or functions described herein. During execution of the instructions 1324 by the machine 1300, the instructions 1324 may also reside, in whole or in part, within the main memory 1304 and / or within the processor 1302, the main memory 1304 and the processor 1302 also constituting machine-readable media.

[0193] The machine 1300 as shown includes an output controller 1328. The output controller 1328 manages the flow of data to / from the machine 1300. The output controller 1328 is sometimes referred to as a device controller, where software that interacts directly with the output controller 1328 is referred to as a device driver.

[0194] Although in one embodiment the machine-readable medium 1322 is shown as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more instructions or data structures. The term "machine-readable medium" should also be considered to include any tangible medium capable of storing, encoding, or carrying instructions or capable of storing, encoding, or carrying data structures that are executed by a machine and cause the machine to perform any one or more of the methods of the present disclosure, the data structures being utilized by or associated with such instructions. Thus, the term "machine-readable medium" should be considered to include, but are not limited to, solid-state memory and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, which includes, by way of example: semiconductor storage devices, such as erasable programmable read-only memory (EPROM), EEPROM, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks and DVD-ROM disks.

[0195] Instructions 1324 may also be sent or received over a communication network 1326 using a transmission medium. Instructions 1324 may be sent using a network interface device 1320 and any of a number of well-known transmission protocols (e.g., HTTP). Examples of communication networks include LANs, WANs, the Internet, mobile phone networks, plain old telephone (POTS) networks, and wireless data networks (e.g., WiFi and WiMax networks). The term "transmission medium" shall be deemed to include any intangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine and includes digital or analog communications signals, or other intangible media that facilitates the communication of such software.

[0196] As used herein, "communicatively coupled between" means that entities on either side of the coupling must communicate through an item between them, and the entities cannot communicate with each other without communicating through the item.

[0197] Additional Notes

[0198] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings show specific embodiments in which the present disclosure can be practiced by way of illustration rather than by way of limitation. These embodiments are also referred to as "examples" in this article. Such examples may include elements other than those shown or described. However, the inventors have also contemplated providing examples of only those elements shown or described. In addition, or with respect to a specific example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) shown or described in this article, the inventors have also contemplated using examples of any combination or permutation (or one or more aspects thereof) of those elements shown or described.

[0199] All publications, patents, and patent documents referenced in this document are incorporated herein by reference in their entirety, as if individually incorporated by reference. In the event of inconsistent usages between this document and those documents incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for conflicting inconsistencies, the usage in this document controls.

[0200] In this document, when introducing elements of various aspects of the present disclosure or embodiments thereof, the terms "a", "an", "the", and "said" are used, as is common in patent literature, to include one or more than one or more of the elements, independent of any other instances or uses of "at least one" or "one or more". In this document, the term "or" is used to represent a non-exclusive, or, such that "A or B" includes "A but not B", "B but not A", and "A and B" unless otherwise specified.

[0201] In the following claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein." Further, in the following claims, the terms "comprising," "including," and "having" are intended to be open-ended, meaning that there may be additional elements in addition to the listed elements, such that anything following such a term in a claim (e.g., comprising, including, having) is still considered to fall within the scope of the claim. Moreover, in the following claims, the terms "first," "second," and "third," etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0202] The embodiments of the present disclosure can be implemented with computer executable instructions.Computer executable instructions (e.g., software code) can be organized into one or more computer executable components or modules.Any number of such components or modules and any organization of such components or modules can be used to implement the various aspects of the present disclosure.For example, the various aspects of the present disclosure are not limited to the specific computer executable instructions or specific components or modules shown in the drawings and described in this article.Other embodiments of the present disclosure may include different computer executable instructions or components with more or less functions than those shown and described herein.

[0203] The method examples described herein (e.g., operations and functions) may be at least partially machine or computer implemented (e.g., implemented as software code or instructions). Some examples may include a computer-readable medium or machine-readable medium encoded with instructions that can be operated to configure an electronic device to perform the method as described in the above examples. The implementation of such a method may include software code such as microcode, assembly language code, high-level language code, etc. (e.g., "source code"). Such software code may include computer-readable instructions (e.g., "objects" or "executable code") for performing various methods. The software code may form part of a computer program product. The software implementation of the embodiments described herein may be provided via an article having codes or instructions stored thereon, or via a method of operating a communication interface to send data via a communication interface (e.g., wirelessly, via the Internet, via satellite communications, etc.).

[0204] In addition, the software code may be tangibly stored on one or more volatile or non-volatile computer-readable storage media during execution or at other times. These computer-readable storage media may include any mechanism for storing information in a form accessible by a machine (e.g., a computing device, an electronic system, etc.), such as, but not limited to, a floppy disk, a hard disk, a removable disk, any form of disk storage media, a CD-ROM, a magneto-optical disk, a removable optical disk (e.g., a compact disk and a digital video disk), a flash memory device, a magnetic cassette, a memory card or stick (e.g., a secure digital card), a RAM (e.g., a CMOS RAM, etc.), a recordable / non-recordable medium (e.g., a read-only memory (ROM)), an EPROM, an EEPROM, or any type of medium suitable for storing electronic instructions, etc. Such computer-readable storage media are coupled to the computer system bus to be accessible by the processor and other parts of the OIS.

[0205] In one embodiment, the computer readable storage medium may have encoded a data structure for treatment planning, wherein the treatment plan may be adaptive. The data structure for the computer readable storage medium may be at least one of a Digital Imaging and Communications in Medicine (DICOM) format, an extended DICOM format, an XML format, and the like. DICOM is an international communication standard that defines a format for transmitting data related to medical images between various types of medical devices. DICOMRT refers to a communication standard specific to radiation therapy.

[0206] In various embodiments of the present disclosure, the methods of creating components or modules can be implemented in software, hardware, or a combination thereof. For example, the methods provided by various embodiments of the present disclosure can be implemented in software using standard programming languages ​​such as, for example, C, C++, Java, Python, etc., and combinations thereof. As used herein, the terms "software" and "firmware" are interchangeable and include any computer program stored in a memory for execution by a computer.

[0207] A communication interface includes any mechanism that interfaces with any of a hardwired medium, a wireless medium, an optical medium, etc. to communicate with another device, such as a memory bus interface, a processor bus interface, an Internet connection, a disk controller, etc. The communication interface may be configured by providing configuration parameters and / or sending signals to prepare the communication interface to provide data signals describing the software content. The communication interface may be accessed via one or more commands or signals sent to the communication interface.

[0208] The present disclosure also relates to a system for performing the operations herein. The system can be specially constructed for the desired purpose, or can include a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Unless otherwise stated, the operation or execution order of the operations in the embodiments of the present disclosure shown and described herein is not necessary. That is, unless otherwise stated, operations can be performed in any order, and compared with those operations disclosed herein, the embodiments of the present disclosure can include other operations or fewer operations. For example, it is conceivable that running or performing a specific operation before another operation, simultaneously with another operation, or after another operation falls within the scope of the various aspects of the present disclosure.

[0209] In view of the above, it will be seen that several objects of the present disclosure are achieved and other advantageous results are obtained. Having described various aspects of the present disclosure in detail, it will be apparent that modifications and variations are feasible without departing from the scope of various aspects of the present disclosure as defined in the appended claims. Since various changes can be made in the above constructions, products and methods without departing from the scope of various aspects of the present disclosure, it is intended that all content contained in the above description and shown in the accompanying drawings should be interpreted as illustrative, and not in a restrictive sense.

[0210] The above description is intended to be illustrative, rather than restrictive. For example, the above examples (or one or more aspects thereof) can be used in combination with each other. In addition, without departing from the scope of the present disclosure, many modifications can be made to adapt specific situations or materials to the teachings of the present disclosure. Although the size and type of the material and coating described herein are intended to limit the parameters of the present disclosure, they are by no means restrictive, but exemplary embodiments. After reviewing the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present disclosure should be determined with reference to the full scope of the equivalents to which the appended claims and these claims are assigned.

[0211] In addition, in the above detailed description, various features can be combined together to simplify the present disclosure. This should not be interpreted as an intention that unclaimed disclosed features are necessary for any one claim. Rather, the subject matter of the present invention may lie in less than all the features of a particular disclosed embodiment. Therefore, the appended claims are thereby incorporated into the detailed description, with each claim acting alone as a separate embodiment. The scope of the present disclosure should be determined with reference to the appended claims and the full scope of equivalents to which such claims are endowed. In addition, the limitations of the appended claims are not written in a device-plus-function format, and are not intended to be interpreted based on the sixth paragraph of 35 U.S.C. §112, unless and until the limitation of such a claim explicitly uses the phrase "device for...", followed by a functional statement without further structure.

[0212] The Abstract is provided to comply with 37 CFR §1.72(b), to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.

Claims

1. A computer-implemented method for generating one or more radiation therapy treatment plans, the method comprising: receiving a set of image data pairs for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of image data pairs comprises a given projection image representing a view of an anatomical structure of a subject from a given gantry angle and a given graphical aperture image of positions of leaves of a multi-leaf collimator (MLC) at the given gantry angle based on the given projection image; Training a Generative Adversarial Network (GAN) model based on a set of image data pairs for each gantry angle; and Using the trained GAN model, an aperture image of the MLC leaf positions for a desired gantry angle is predicted based on projection images representing views of the anatomical region of interest.

2. The method according to claim 1, wherein: The GAN includes a conditional adversarial network cGAN.

3. The method according to claim 1, wherein: The GAN includes a cycle-consistent generative adversarial network CycleGAN.

4. The method according to claim 1, wherein: The given gantry angle of a first pair in the set of image data pairs differs by a predetermined amount from the given gantry angle of a second pair in the set of image data pairs.

5. The method according to claim 1, wherein: The GAN is configured to train the generative model using the discriminative model; Using adversarial training between the discriminative model and the generative model to establish values ​​applied by the generative model and the discriminative model; as well as The generative model and the discriminative model include respective convolutional neural networks.

6. The method according to claim 5, wherein: The adversarial training comprises: training the generative model to generate a first synthetic graphical aperture image representation of the MLC blade position at a first gantry angle from projection images representing a view of a training subject's anatomy from the first gantry angle; and training the discriminant model to classify the first synthetic graphic aperture image as a synthetic training example graphic aperture image or a real training example graphic aperture image; and The output of the generative model is used to train the discriminative model, and the output of the discriminative model is used to train the generative model.

7. The method according to claim 6, wherein: The GAN is trained using a cycle-consistent generative adversarial network CycleGAN including the generative model and the discriminative model, wherein the generative model is a first generative model and the discriminative model is a first discriminative model, wherein the CycleGAN further comprises: The second generative model is trained to: processing as input a given graphical aperture image representation of the MLC blade position at a given gantry angle from a given pair of the set of image data pairs; and providing as output a synthetic projection image representing a view of the training subject's anatomy from the given gantry angle; and A second discriminant model is trained to classify the synthetic projection image as a synthetic projection image or a real projection image.

8. The method according to claim 7, wherein: The CycleGAN comprises a first part for training the first generative model, wherein the first part is trained to: a set of training projection images representing different views of the patient's anatomy obtained from a previous treatment, the set of training projection images being paired with a training graphic aperture image corresponding to each of the different views, each of the training graphic aperture images being aligned with a corresponding one of the training projection images; transmitting the set of training projection images to an input of the first generative model to output a first set of graphic aperture images; receiving the first set of graphic aperture images at an input of the first discriminant model to classify the first set of graphic aperture images into a set of synthetic training graphic aperture images or a set of real training graphic aperture images; as well as The first set of graphic aperture images is received at an input of the second generative model to generate a first set of cyclic projection images for computing a cycle consistency loss.

9. The method according to claim 8, wherein: The CycleGAN includes a second part, which is trained to: transmitting the training pattern aperture image corresponding to each of the different views to an input of the second generative model to output a first set of composite projection images; receiving the first set of composite projection images at an input of the second discriminant model to classify the first set of composite projection images as synthetic training projection images or real training projection images; as well as The first set of composite projection images is received at an input of the first generative model to generate a first set of cyclic pattern aperture images for computing a cycle consistency loss.

10. The method according to claim 9, wherein: generating the cycle consistency loss based on a comparison of the first set of cyclic projection images to the set of training projection images and a comparison of the first set of cyclic graphic aperture images to the training graphic aperture images; The first generative model is trained to: minimize or reduce a first loss term, the first loss term representing an expectation of differences between a plurality of synthetic graphic aperture images and respectively paired training graphic aperture images; as well as The second generative model is trained to minimize or reduce a second loss term representing an expectation of differences between a plurality of synthesized projection images and respectively paired training projection images.

11. The method according to claim 1, further comprising: obtaining radiation therapy treatment machine parameter information representing MLC leaf positions at gantry angles corresponding to each of the views represented by the projection images of the set of image data pairs and radiation therapy beam intensities corresponding to each of the projection images; generating a training graphical aperture image representation based on the obtained radiation therapy treatment machine parameter information; as well as Each of the generated training pattern aperture image representations is aligned with a corresponding projection image.

12. The method according to claim 11, wherein: The training graphic aperture image representation and the corresponding projection image are two-dimensional images or three-dimensional images comprising a stack of two-dimensional projection image and graphic aperture image pairs corresponding to an entire treatment fraction.

13. The method according to claim 1, further comprising: One or more radiation therapy treatment machine parameters are generated from the aperture image predicted by the trained GAN model.

14. The method according to claim 13, further comprising: A dose volume histogram or three-dimensional dose distribution is calculated based on the generated one or more radiation therapy treatment machine parameters.

15. The method according to claim 13, wherein: The one or more parameters include at least one of a gantry angle, an MLC jaw position, an MLC leaf position, or a radiotherapy beam intensity.

16. The method according to claim 13, further comprising: Dosimetry parameters are calculated based on the generated one or more radiation therapy treatment machine parameters.

17. The method according to claim 1, wherein: A given projection image in the set of image data pairs is generated by ray tracing or Fourier reconstruction.

18. A system for generating one or more radiation therapy treatment plans, the system comprising: One or more processors configured to perform operations comprising: receiving a set of image data pairs for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of image data pairs comprises a given projection image representing a view of an anatomical structure of a subject from a given gantry angle and a given graphical aperture image of positions of leaves of a multi-leaf collimator (MLC) at the given gantry angle based on the given projection image; Training a Generative Adversarial Network (GAN) model based on a set of image data pairs for each gantry angle; and Using the trained GAN model, an aperture image of the MLC leaf positions for a desired gantry angle is predicted based on projection images representing views of the anatomical region of interest.

19. The system of claim 18, wherein: The GAN includes a conditional adversarial network cGAN or a cycle-consistent generative adversarial network CycleGAN.

20. The system of claim 18, further comprising an operation for generating a plurality of two-dimensional projection images as projection images in the set of image data pairs, the plurality of two-dimensional projection images representing a plurality of two-dimensional views of the anatomical structure from a plurality of different gantry angles.

21. The system of claim 20, wherein: The plurality of two-dimensional projection images represent views of the anatomical structure at multiple angles across an arc of motion of a gantry of the radiation therapy treatment machine during treatment.

22. A non-transitory computer readable medium comprising non-transitory computer readable instructions for performing operations comprising: receiving a set of image data pairs for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of image data pairs comprises a given projection image representing a view of an anatomical structure of a subject from a given gantry angle and a given graphical aperture image of positions of leaves of a multi-leaf collimator (MLC) at the given gantry angle based on the given projection image; Training a Generative Adversarial Network (GAN) model based on a set of image data pairs for each gantry angle; and Using the trained GAN model, an aperture image of the MLC leaf positions for a desired gantry angle is predicted based on projection images representing views of the anatomical region of interest.