System and method for planning radiation therapy

Abstract

Embodiments of the present disclosure relate to systems and methods for planning radiation therapy. Methods and systems for planning radiation therapy are provided herein. In an example, at least one processor may be programmed to receive data associated with a first planned target volume and a first radiation pattern, provide the first planned target volume and the first radiation pattern to a first model to cause the first model to output data associated with at least one first beam position and at least one first beam intensity, and provide the first model to a second model to output data associated with at least one second beam position and at least one second beam intensity. A second radiation pattern is generated, and the first planned target volume and the second radiation pattern are provided to a second model such that the second model outputs data associated with the at least one second beam position and the at least one second beam intensity. The at least one processor may be further programmed to transmit data associated with the at least one second beam position and the at least one second beam intensity to cause the linear accelerator to deliver radiation.

Description

Technical Field

[0001] The present application generally relates to systems and methods for planning radiotherapy using a medical device that delivers energy to a patient. Background Art

[0002] Radiotherapy is one of the most common treatments used to treat cancer in the body. However, one of the main challenges faced by radiotherapy is how to plan the delivery of energy to a volume of tissue in the body such that the energy delivery is optimized and damage to healthy tissue is minimized. For example, when using a linear accelerator to deliver energy, a clinician needs to carefully configure a treatment plan that maximizes the energy delivery to the tumor while minimizing the energy delivery to the tissue surrounding the tumor (which, differently, is healthy tissue).

[0003] Currently available methods and systems for optimizing radiotherapy plans focus on data obtained from a patient on a case-by-case basis. For example, a CT scan can be initially generated for a patient, and a clinician can provide inputs to guide a system that controls a linear accelerator when delivering energy to a planned target volume of the patient. The inputs can specify the position of the beam, the intensity of the beam, etc. This is an extremely time-consuming process and depends on each clinician developing expertise and improving over time. And when analyzing a patient's CT scan and determining the position and intensity of the beam, traditional methods that automate aspects of the planning process typically rely on brute-force methods. This is both time-consuming and computationally intensive. Summary of the Invention

[0004] For the reasons above, a system that can improve radiotherapy planning is desired. For example, it is desired to improve the radiotherapy planning process such that radiotherapy plans developed using improved linear accelerator technology are more accurate, optimize the energy delivery to the planned target volume while minimizing the energy delivery to the surrounding tissue. It is also desired to have a technology that enables radiotherapy planning to develop consistently, with different patients having different planned target volumes.

[0005] Aspects of the present disclosure relate to systems, methods, devices, apparatuses, and non-transitory machine-readable media for planning radiotherapy. In an embodiment, at least one processor is programmed to: receive data associated with a first planned target volume and a first radiation map, the first planned target volume representing a region of a patient's body and a target mask, the target mask representing a portion of the region of the patient's body to receive radiation; provide the first planned target volume and the first radiation map to a first model to cause the first model to output data associated with at least one first beam position and at least one first beam intensity, the at least one first beam position corresponding to the at least one first beam intensity; generate a second radiation map based on the at least one first beam position, the at least one first beam intensity, and the first radiation map; provide the first planned target volume and the second radiation map to a second model to cause the second model to output data associated with at least one second beam position and at least one second beam intensity; and transmit data associated with the at least one second beam position and the at least one second beam intensity to cause a linear accelerator (LINAC) to deliver radiation to the patient.

[0006] The first model may be trained to output data associated with first beam positions of a first scale, and the second model may be trained to output data associated with second beam positions of a second scale. The first scale may be associated with a first set of candidate angles within a candidate angle space, the second scale may be associated with a second set of candidate angles within the candidate angle space, and an angular measurement associated with the first set of candidate angles may be greater than an angular measurement associated with the second set of candidate angles. The beams associated with the second set of candidate angles correspond to the beams associated with the first set of candidate angles.

[0007] The first model may be trained based on a reinforcement learning agent and training data associated with previously treated patients. The training data may represent planned target volumes and target radiation maps corresponding to previously treated patients. When training, at least one processor may also be programmed to: for each previously treated patient among the previously treated patients: generate data associated with beam positions and beam intensities based on a plurality of reinforcement learning agents, each reinforcement learning agent associated with a training model; provide the data associated with the beam positions and the beam intensities to a simulator to generate a set of simulated outputs; determine a reward for each reinforcement learning agent among the plurality of reinforcement learning agents based on the corresponding simulated outputs and the target radiation map; and select the first model based on the rewards for each reinforcement learning agent. The target radiation map represents a radiation map previously generated based on clinician input, the target radiation map representing the clinician's preference when delivering radiation to a previously treated patient.

[0008] In another embodiment, a method includes: receiving, by at least one processor, data associated with a first planned target volume and a first radiation map, the first planned target volume representing a region of a patient's body and a target mask, the target mask representing a portion of the region of the patient's body to receive radiation; providing, by at least one processor, the first planned target volume and the first radiation map to a first model to cause the first model to output data associated with at least one first beam position and at least one first beam intensity, the at least one first beam position corresponding to the at least one first beam intensity; generating, by at least one processor, a second radiation map based on the at least one first beam position, the at least one first beam intensity, and the first radiation map; providing, by at least one processor, the first planned target volume and the second radiation map to a second model to cause the second model to output data associated with at least one second beam position and at least one second beam intensity; and transmitting, by at least one processor, data associated with the at least one second beam position and the at least one second beam intensity to cause a LINAC to deliver radiation to the patient.

[0009] In yet another embodiment, a non-transitory machine-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: receiving data associated with a first planned target volume and a first radiation map, the first planned target volume representing a region of a patient's body and a target mask, the target mask representing a portion of the region of the patient's body to receive radiation; providing the first planned target volume and the first radiation map to a first model to cause the first model to output data associated with at least one first beam position and at least one first beam intensity, the at least one first beam position corresponding to the at least one first beam intensity, generating a second radiation map based on the at least one first beam position, the at least one first beam intensity, and the first radiation map; providing the first planned target volume and the second radiation map to a second model to cause the second model to output data associated with at least one second beam position and at least one second beam intensity; and transmitting data associated with the at least one second beam position and the at least one second beam intensity to cause a LINAC to deliver radiation to the patient.

[0010] By implementing the above techniques, systems and methods can incorporate machine learning-based techniques into the process of planning and delivering radiotherapy. In practice, a specific radiation dose is prescribed for a patient's tumor, and that dose delivery is spread over multiple days (or fractions). A common variant of radiotherapy is image-guided radiotherapy (IGRT), where daily (cone beam, megavoltage, or low-tesla MR) images of the patient's anatomy are acquired prior to each treatment fraction for patient setup verification. Additionally, intensity-modulated RT (IMRT) and volumetric arc therapy (VMAT) are techniques that ensure that radiation dose delivery is adjusted to be close to the target. By implementing the above techniques, compared to other techniques such as modeling (e.g., modeling before optimization (e.g., using a cost function that can map the relative importance of different criteria), modeling during optimization (e.g., allowing a clinician to actively interact with the optimization and adjust the relative importance of one criterion relative to another (e.g., treatment duration relative to radiation to at-risk organs)), and modeling after optimization (e.g., by pre-creating a pareto surface by re-running the optimization with different configurations and multi-criteria dynamically)), the radiation dose delivery to the planning target volume can be optimized, the radiation dose delivery to at-risk organs or structures near the planning target volume can be consistently minimized, and it can be confirmed that the planned therapy is deliverable (in terms of machine capabilities) and effective (in the shortest amount of time).

[0011] Aspects of the present disclosure relate to systems, methods, devices, apparatuses, and non-transitory machine-readable media for generating a fluence map when planning radiotherapy. In an embodiment, at least one processor is programmed to: receive data associated with a plurality of beams, each beam in the plurality of beams specifying a beam angle measured relative to a LINAC; generate a beam's-eye view (BEV) image based on the plurality of beams; determine a digitally reconstructed radiograph (DRR) based on the cascaded BEV images; and provide the DRR to a model to cause the model to generate an output associated with the fluence map.

[0012] When providing the DRR to the model, the at least one processor can be programmed to: provide the DRR to at least one encoder along a contraction path of the model to cause the at least one encoder to provide a first output to at least one second encoder and a second output to at least one decoder corresponding to the at least one encoder, where the at least one decoder is along an expansion path of the model. When providing the first output to the at least one second encoder, the at least one processor is programmed to: provide the first output to the at least one second encoder to cause the at least one second encoder to: provide a third output to at least one second decoder along an expansion path of the model, the at least one second decoder corresponding to the at least one second encoder, where the second decoder is configured to provide a fourth output to the at least one decoder based on the third output.

[0013] When receiving data associated with multiple beams, at least one processor is programmed to: receive data associated with the positions of multiple leaves corresponding to each of the multiple beams, the multiple positions representing a configuration of the multiple leaves, and wherein, when generating a BEV image, at least one processor is programmed to: generate a BEV image based on the radiation delivery of the LINAC when the multiple leaves are in the configuration. When generating a BEV image, at least one processor may be programmed to: generate a BEV image based on the power level at which the LINAC delivers radiation.

[0014] When receiving data associated with multiple leaves, at least one processor may be programmed to: receive, for each of the multiple beams, data associated with the positions of the multiple leaves across multiple time steps, wherein the positions of the multiple leaves at each time step are associated with a sub-configuration of the multiple leaves, and when generating a BEV image, at least one processor may be programmed to: generate each BEV image based on the radiation delivery of the LINAC at each of the multiple time steps. When generating a BEV image, at least one processor may be programmed to: generate a BEV image based on the power level at which the linac delivers radiation for each of the multiple beams.

[0015] In another embodiment, a non-transitory machine-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: receiving data associated with multiple beams, each of the multiple beams specifying a beam angle measured relative to the LINAC; generating a BEV image based on the multiple beams; determining a DRR based on the cascaded BEV images; and providing the DRR to a model to cause the model to generate an output associated with a fluence map.

[0016] In another embodiment, a method includes: receiving, by at least one processor, data associated with multiple beams, each of the multiple beams specifying a beam angle measured relative to the LINAC; generating, by at least one processor, a BEV image based on the multiple beams; determining, by at least one processor, a DRR based on the cascaded BEV images; and providing, by at least one processor, the DRR to a model to cause the model to generate an output associated with a fluence map.

[0017] By implementing the above techniques, the system and method can adopt a machine learning-based approach to develop a fluence map, which improves techniques such as inverse planning for fluence map development. The disclosed techniques can enable faster fluence map development, which in turn can shorten the entire radiotherapy planning process. As a result, power and computational resources can be saved and / or redirected to the treatment of additional patients. Additionally, the techniques of the present disclosure allow for careful handling of multiple beams without accessing the field dose distribution for each dose to support the training process.

[0018] Aspects of the present disclosure relate to systems, methods, devices, apparatuses, and non-transitory machine-readable media for generating a leaf sequence when planning radiotherapy. In an embodiment, at least one processor is programmed to: receive input data associated with a target fluence map that represents a target radiation dose to be delivered to a patient via a LINAC; provide the input data to a model to cause the model to generate an output that represents an accumulated fluence map and a leaf mask; and when a difference between the accumulated fluence map and the target fluence map satisfies a first threshold amount: generate updated input data associated with the target fluence map, the accumulated fluence map, and the leaf mask, provide the updated input data to the model to cause the model to generate an updated output that represents an updated accumulated fluence map and an updated leaf mask, determine that a difference between the updated accumulated fluence map and the target fluence map satisfies a second threshold amount; and determine a leaf sequence based on the leaf mask and the updated leaf mask.

[0019] When determining the leaf sequence, at least one processor may be programmed to: determine a difference between the leaf mask and the updated leaf mask; and determine the leaf sequence based on the difference between the leaf mask and the updated leaf mask.

[0020] When determining the difference between the leaf mask and the updated leaf mask, at least one processor may be programmed to: determine a first leaf position and a second leaf position associated with a leaf pair of the LINAC, the first leaf position and the second leaf position corresponding to the leaf mask; and determine a third leaf position and a fourth leaf position associated with the leaf pair, the third leaf position and the fourth leaf position corresponding to the updated leaf mask, wherein the difference between the leaf mask and the updated leaf mask is represented by a change in position between the first leaf position and the third leaf position and a change in position between the second leaf position and the fourth leaf position.

[0021] When determining the difference between the leaf mask and the updated leaf mask, at least one processor may be programmed to: determine a first movement rate of a first leaf of the leaf pair during a time period; determine a second movement rate of a second leaf of the leaf pair during the time period; and determine the difference between the leaf mask and the updated leaf mask based on the first movement rate of the first leaf and the second movement rate of the second leaf.

[0022] When determining the difference between the leaf mask and the updated leaf mask, at least one processor may be programmed to: determine at least one radiation dose delivered via the LINAC when the leaf pair transitions from the leaf mask to the updated leaf mask. The LINAC may include a plurality of leaf pairs, and the model may include a plurality of surrogate policies, each of the plurality of surrogate policies corresponding to one of the plurality of leaf pairs.

[0023] Each agent policy can be trained based on training data associated with multiple previously treated patients, where the training data includes multiple target dose metrics. At least one processor can also be programmed to, for each agent corresponding to a leaf pair and each target dose metric among the multiple agents, as the leaf pair moves over multiple time steps, generate motion data associated with the motion of the leaf pair and the radiation dose to be delivered via a LINAC, provide the motion data to a simulator for each agent to generate a simulated fluence map for each time step of the multiple time steps, determine a reward for each agent based on the simulated fluence map, and select a final agent from among the multiple agents based on the score of each agent among the multiple agents, where the score corresponds to the cumulative score of each agent among the multiple agents.

[0024] In another embodiment, a non - transitory machine - readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: providing input data to a model to cause the model to generate an output that represents a cumulative fluence map and a leaf mask; and when the difference between the cumulative fluence map and the target dose metric map satisfies a first threshold amount: generating updated input data associated with the target dose metric map, the cumulative fluence map, and the leaf mask, providing the updated input data to the model to cause the model to generate an updated output that represents an updated cumulative fluence map and an updated leaf mask, determining that the difference between the updated cumulative fluence map and the target dose metric map satisfies a second threshold amount; and determining a leaf sequence based on the leaf mask and the updated leaf mask.

[0025] In another embodiment, a method includes: receiving, by at least one processor, input data associated with a target dose metric map that represents a target radiation dose to be delivered to a patient via a LINAC; providing, by at least one processor, the input data to a model to cause the model to generate an output that represents a cumulative fluence map and a leaf mask; and when the difference between the cumulative fluence map and the target dose metric map satisfies a first threshold amount: generating, by at least one processor, updated input data associated with the target dose metric map, the cumulative fluence map, and the leaf mask, providing, by at least one processor, the updated input data to the model to cause the model to generate an updated output that represents an updated cumulative fluence map and an updated leaf mask, determining, by at least one processor, that the difference between the updated cumulative fluence map and the target dose metric map satisfies a second threshold amount; and determining, by at least one processor, a leaf sequence based on the leaf mask and the updated leaf mask.

[0026] With the implementation of the techniques described herein, a leaf sequence can be formulated as a decision problem suitable for solution by the deep reinforcement learning (DRL) techniques described herein. For example, a leaf sequence model can be implemented to handle different plan types (VMAT and IMRT). These models can also improve the total time required to calculate the leaf sequence, again reducing the computational resources and power requirements, which can be conserved or directed towards processing additional patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying drawings, which are schematic and not intended to be drawn to scale. Unless indicated as representing the background art, the drawings represent aspects of the present disclosure.

[0028] Figure 1 Illustrated is a diagram of a system for planning and delivering radiotherapy according to an embodiment.

[0029] Figure 2A Illustrated is a diagram of a process for planning radiotherapy according to an embodiment.

[0030] Figures 2B to 2D Illustrated is a flowchart of an implementation related to a process for planning and delivering radiotherapy according to an embodiment.

[0031] Figure 3A Illustrated is a diagram of a process for generating a fluence map when planning radiotherapy according to an embodiment.

[0032] Figures 3B to 3D Illustrated is a flowchart of an implementation related to a process for generating a fluence map when planning radiotherapy according to an embodiment.

[0033] Figure 4A Illustrated is a diagram of a process for generating a leaf sequence when planning radiotherapy according to an embodiment.

[0034] Figures 4B to 4F Illustrated is a flowchart of an implementation related to a process for generating a leaf sequence when planning radiotherapy according to an embodiment.

[0035] Figure 4G Illustrated is Figures 4B to 4F an example iteration of an implementation.

[0036] Figure 4H Illustrated is an example pair of a predicted fluence map and a target (e.g., cumulative) fluence map according to an embodiment.

[0037] Figure 5A and Figure 5B Illustrated is an example training pipeline involving multiple systems according to an embodiment. DETAILED DESCRIPTION

[0038] Reference will now be made to the illustrative embodiments depicted in the accompanying drawings, and specific language will be used herein to describe them. However, it should be understood that no limitation of the scope of the claims or the disclosure is thereby intended. Variations and further modifications of the features of the invention shown herein, as well as additional applications of the principles of the subject matter shown herein, will be considered within the scope of the subject matter disclosed herein, which will be implemented by those skilled in the relevant art and those having the technology of the present disclosure. Other embodiments may be used, and / or other changes may be made without departing from the spirit or scope of the present disclosure. The illustrative embodiments described in the detailed description are not meant to limit the subject matter presented.

[0039] Figure 1 Components of a system 100 for planning and delivering radiotherapy according to an embodiment are shown. System 100 may include an analysis server 114a, a system database 114b, an AI model 111, electronic data sources 120a-d (collectively referred to as electronic data sources 120), end-user devices 140a-c (collectively referred to as end-user devices 140), an administrator computing device 150, a medical device 160, and a medical device computer 162. Figure 1 The various components depicted in may belong to a radiotherapy clinic, and in some cases, a patient may receive radiotherapy treatment via one or more radiotherapy machines (e.g., medical device 160) located within the clinic. System 100 is not limited to the components described herein and may include additional components or other components (not shown for brevity) that will be considered within the scope of the embodiments described herein.

[0040] The above-described components may be connected to each other via a network 130. Examples of network 130 may include, but are not limited to, private or public local area networks (LANs), wireless LAN (WLAN) networks, metropolitan area networks (MANs), wide area networks (WANs), and the Internet. Network 130 may include wired and / or wireless communication according to one or more standards and / or via one or more transmission media. Communication over network 130 may be performed according to various communication protocols such as Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and IEEE communication protocols. In one example, network 130 may include wireless communication according to the Bluetooth specification set or another standard or proprietary wireless communication protocol. In another example, network 130 may also include communication via a cellular network, including, for example, GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), and EDGE (Enhanced Data rates for Global Evolution) networks.

[0041] The analysis server 114a can generate and display an electronic platform, which is configured to use an AI model 111 (including artificial intelligence and / or machine learning models) to receive patient information and output the execution results of the AI model 111. The electronic platform can include a graphical user interface (GUI) displayed on one or more electronic data sources 120, end-user devices 140, medical devices 160, and / or administrator computing devices 150. Examples of the electronic platform generated and hosted by the analysis server 114a can be a web-based application or a website configured to be displayed on different electronic devices (such as mobile devices, tablet computers, personal computers, etc.).

[0042] The information displayed by the electronic platform can include, for example, input elements to receive data associated with the patient being treated, synchronize one or more sensors, and display the prediction results generated by the AI model 111. For example, the analysis server 114a can execute the AI model 111 (e.g., as described herein, a machine learning model trained to generate fluence maps, leaf sequences, etc. for a patient being treated via the medical device 160). The analysis server 114a can then display the results for the clinician and / or directly revise one or more operating attributes of the medical device 160.

[0043] The analysis server 114a can be any computing device including a processor and a non-transitory machine-readable storage device capable of performing the various tasks and processes described herein. The analysis server 114a can employ various processors such as a central processing unit (CPU) and a graphics processing unit (GPU). Non-limiting examples of such computing devices can include workstation computers, laptop computers, server computers, etc. Although the system 100 includes a single analysis server 114a, the analysis server 114a can include any number of computing devices operating in a distributed computing environment (such as a cloud environment).

[0044] The electronic data source 120 can represent various computing devices that contain, retrieve, and / or access data associated with the medical device 160, such as operation information associated with a previously performed radiotherapy treatment (e.g., an electronic log file or an electronic profile), data associated with a previously monitored patient (e.g., CT scans, tumor location, deformation information, etc.), or research participants who trained the AI models discussed herein. For example, the analysis server 114a can use the clinic computer 120a, the medical professional device 120b, the server 120c (associated with a clinician and / or a clinic), and the database 120d (associated with a clinician and / or a clinic) to retrieve / receive data associated with the medical device 160. The analysis server 114a can retrieve data from the end-user device 120, generate a training data set, and train the AI model 111. The analysis server 114a can execute various algorithms to convert the raw data received / retrieved from the electronic data source 120 into a machine-readable object that can be stored and processed by other analysis processes described herein.

[0045] The end-user device 140 can be any computing device that includes a processor and a non-transitory machine-readable storage medium capable of performing the various tasks and processes described herein. Non-limiting examples of the end-user device 140 can be a workstation computer, a laptop computer, a tablet computer, or a server computer. In operation, various users can use the end-user device 140 to access the GUI operationally managed by the analysis server 114a or otherwise access the execution results of the AI model 111. Specifically, the end-user device 140 can include the clinic computer 140a, the clinic server 140b, and the medical professional device 140c. Even though it is called an "end-user" device herein, these devices are not always operated by an end user. For example, an end user cannot directly use the clinic server 140b. However, the results stored on the clinic server 140b can be used to populate the various GUIs accessed by the end user via the medical professional device 140c. In some embodiments, the end-user device 140 can be associated with one or more clinicians who are associated with one or more treatment plans for a patient (e.g., involving the preparation of one or more treatment plans).

[0046] The administrator computing device 150 can represent a computing device operated by a system administrator. The administrator computing device 150 can be configured to display radiotherapy treatment attributes generated by the analysis server 114a (e.g., various analysis metrics determined during the training of one or more machine learning models and / or systems); monitor the various models 111 used by the analysis server 114a, the electronic data source 120, and / or the end-user device 140; review feedback; and / or facilitate the training or retraining (calibration) of the AI model 111 maintained by the analysis server 114a.

[0047] In some embodiments, the medical device 160 can be a diagnostic imaging device or a treatment delivery device. For example, the medical device 160 can include one or more computed tomography (CT) scanners, a linear accelerator (LINAC) having a multi-leaf collimator (referred to herein as a collimator for convenience of description) composed of a plurality of small lead leaves, or other similar devices configured to transmit energy to a target tissue (referred to as a planned target volume) associated with a patient and, in some cases, measure the energy delivered to the target tissue. The plurality of small lead leaves can be individually moved to shape the radiation beam and deliver a dose to the tumor while minimizing the dose to surrounding healthy tissue. The medical device 160 can also include one or more sensors configured to monitor the patient being treated. That is, the medical device 160 and / or the analysis server 114a can communicate with various sensors that can monitor the patient's external biometric signals. Non-limiting examples of sensors can include 3D surface mechanisms and optical (or other) sensors configured to monitor the patient's movement (e.g., how the patient moves and / or breathes).

[0048] The AI model 111 can be stored in the system database 114b. The AI model 111 can be trained using data received / retrieved from the electronic data source 120 and can be executed using data received from the end-user device, the medical device 160, and / or the sensor 163. In some embodiments, the AI model 111 can reside locally at the clinic or within a clinic-specific data repository. In various embodiments, the AI model 111 can use one or more deep learning engines to develop a patient treatment plan using radiotherapy. For example, the analysis server 114a can transmit patient attributes from the sensor 163 and accordingly execute the AI model 111. The analysis server 114a can then display the results on one or more end-user devices 140. In some embodiments, the analysis server 114a can change one or more configurations of the medical device 160 based on the results predicted by the AI model 111.

[0049] Reference Figure 2A , shows an illustration of a process 200 for planning radiotherapy according to an embodiment. The process 200 can include operations 202-210. However, other embodiments can include additional or alternative operations, or one or more operations can be completely omitted. The process 200 is described as being executed by an analysis server 114a that is the same as or similar to Figure 1 . However, one or more steps of the process 200 can be executed (fully, partially, etc.) by any number of computing devices operating in the distributed computing system described in Figure 1 . For example, one or more computing devices can execute Figure 2ASome or all of the operations described in

[0050] In operation 202, the analysis server receives data associated with a first planned target volume and a first radiation map. For example, the first planned target volume may represent one or more regions within a patient that are identified (e.g., by a clinician, such as an oncologist, etc.) to receive energy (e.g., energy associated with a radiotherapy treatment such as high-energy X-rays). Additionally, the first planned target volume may be associated with a target mask. The target mask may represent a portion of one or more regions within the patient that are to receive energy during the patient's treatment. In some embodiments, the first radiation map may represent the amount of one or more energies to be delivered to one or more regions within the first planned target volume. The amount of one or more energies may be measured in counts per second, microsieverts per hour, etc. In some embodiments, the first planned target volume and the first radiation map may be at a resolution associated with a first scale. As described below, the resolution of the first scale may be lower (e.g., coarser) than the resolution of subsequent scales.

[0051] In operation 204, the analysis server provides the first planned target volume and the first radiation map to a first model such that the first model outputs data associated with at least one first beam position and at least one first beam intensity. For example, the analysis server may provide the first planned target volume and the first radiation map to the first model, where the first model is trained to output at least one first beam position and at least one first beam intensity. In some embodiments, the first beam position corresponds to one or more devices associated with a medical device (e.g., a medical device 160 similar to or the same as a medical device such as a LINAC). In some embodiments, the first beam position may correspond to one or more beam angles. One or more beam angles may be the angular position relative to an axis along which one or more energy delivery devices are configured to transmit energy when delivering energy to patient tissue. It should be understood that during the process of the medical device delivering energy to patient tissue, the axis is typically positioned parallel, and the points along the axis may be referred to as isocenters. Additionally or alternatively, one or more beam angles may be the angular position relative to the patient's planned target volume.

[0052] In some embodiments, one or more beam angles can be limited (e.g., corresponding to certain predetermined angles relative to the axis of the patient and / or the planned target volume), as represented within the candidate angle space. Additionally or alternatively, one or more beam angles can correspond to a range of angles. In some embodiments, one or more beam angles can correspond to one or more scales. For example, at a first scale, one or more beam angles can include eleven beam angles, where each beam angle corresponds to a position around the axis that is spaced approximately 33 degrees apart. In an example, at a second scale, one or more beam angles can include sixteen beam angles, where each beam angle corresponds to a position around the axis that is spaced approximately 22.5 degrees apart. In an example, at a third scale, one or more beam angles can include thirty-two beam angles, where each beam angle corresponds to a position around the axis that is spaced approximately 11.25 degrees apart. It should be understood that in the example, the first scale may be coarser than the second scale (e.g., providing fewer granularity options when selecting one or more beam angles at which energy is to be delivered), and the second scale is coarser than the third scale.

[0053] In yet another illustrative example, at a first scale, the candidate angles can include 0, 30, 60, 90, 120, 150, 180, 210, 240, 270, 300, and 330 degrees (with a 30-degree interval between the angles). In this example, at a second scale, the candidate angles can include 0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225, 240, 255, 270, 285, 300, 315, 330, and 345 degrees (with a 15-degree interval between the angles). It should be understood that certain angles within the candidate angle space (e.g., 0, 30, 60 degrees, etc.) can correspond to both the first scale and the second scale. And in the illustrative example above, the angles of the first scale are included as a subset of the angles of the second scale. It should be understood that with respect to the IMRT process, angles can be selected within the candidate angle space; and with respect to the VMAT process, the starting and ending points can be associated with the starting and ending points of a particular interval that corresponds to a scale associated with the candidate angle space.

[0054] In some embodiments, the first model is configured to provide data associated with at least one first beam position and at least one first beam intensity as an output of a first scale (e.g., as described above). In some embodiments, the first scale is defined by a first angular range formed within a candidate angular space (e.g., represented by the angle formed when extending two adjacent beams outward from the centroid within the candidate angular space). The candidate angular space may include one or more positions at which beams corresponding to positions along which the linear accelerator may be configured to transmit energy may be formed. In an illustrative example, and again referring to the first scale, a first angular range may be formed within the candidate angular space within which the LINAC may be configured to deliver energy. The first scale may also be associated with an angle that may be approximately 99 degrees in this illustrative example, the angle being measured between a first beam (extending from a point along an axis (referred to as the isocenter) to a first point along an axis that rotates about the isocenter, about which the LINAC may rotate when delivering energy) and a second beam (extending from the isocenter to a second point along an axis that rotates about the isocenter). It should be understood that the first angular range may be associated with different scales corresponding to different angles.

[0055] In operation 206, the analysis server generates a second radiation map based on at least one first beam position, at least one first beam intensity, and a first radiation map. For example, the analysis server may generate the second radiation map based on determining (e.g., simulating) the radiation delivered to a patient based on at least one first beam position and at least one first beam intensity. In an example, the analysis server may generate the second radiation map based on comparing the radiation delivered to the patient based on at least one first beam position and at least one first beam intensity with a planned target volume. Based on comparing the radiation delivered to the patient based on at least one first beam position and at least one first beam intensity with the planned target volume, the analysis server may determine the second radiation map, where the second radiation map represents one or more regions within the planned target volume that have not received a sufficient amount of energy. In some embodiments, the second radiation map may be associated with a resolution corresponding to a second scale. For example, the resolution associated with the second radiation map may be greater than the resolution associated with the first radiation map. In this way, the analysis server may determine beam angles and beam intensities at increasingly fine scales, thereby achieving more precise beam placement and power level selection at each scale.

[0056] In some embodiments, the analysis server iteratively repeats the functions performed with respect to operations 204 and 206 until convergence to an optimized first beam position and an optimized first beam intensity. For example, the analysis server may iteratively provide successive versions of the first planned target volume and the second radiation map to the model such that the model outputs successive versions of at least one first beam position and at least one first beam intensity. The analysis server may repeat these operations until the at least one first beam position and the at least one first beam intensity output by the model converge.

[0057] In operation 208, the analysis server provides the first planned target volume and the second radiation map to a second model such that the second model outputs data associated with at least one second beam position and at least one second beam intensity. For example, the analysis server may provide the first planned target volume and the second radiation map to the second model, where the second model is trained to output at least one second beam position and at least one second beam intensity. In this example, the at least one second beam position corresponds to the at least one second beam intensity. In some embodiments, the at least one second beam position corresponds to one or more devices associated with a medical device as described herein. In some embodiments, the second beam position may correspond to one or more beam angles at which the medical device is configured to transmit energy. In some embodiments, with respect to a second scale, the first planned target volume and the second radiation map may be associated with a resolution corresponding to the second scale. For example, the first planned target volume and the second radiation map may be associated with a resolution that is greater (e.g., higher) than the resolution of the first planned target volume and the first radiation map analyzed by the analysis server at the first scale. By increasing the resolution at each scale, the analysis server is able to achieve robust optimization of the beam position and beam intensity, as well as precision with respect to the position at which the beam is delivered to the planned target volume.

[0058] In some embodiments, the at least one second beam position may correspond to a position corresponding to a second scale, different from the position corresponding to the first scale. For example, the second scale at which the second model outputs one or more second beam angles may be finer than the scale at which the first model outputs one or more first beam angles (e.g., providing a greater granularity of choice when selecting one or more beam angles at which to deliver energy). In some embodiments, the second scale may be associated with a second set of candidate angles within a candidate angle space. For example, the second scale may be associated with an angle formed at the second scale (e.g., measured by projecting adjacent beams outward from a centroid), where the angle is less than the angle formed by two adjacent beams at the first scale (as described above). In this way, the second model may be configured to provide at least one second beam position as an output, where the at least one beam position may be located at a more precise position when compared to the at least one beam position that may be provided as an output by the first model.

[0059] In some embodiments, the second model is configured to provide data associated with at least one second beam position and at least one second beam intensity as an output on a second scale (e.g., as described above). In some embodiments, the second scale is associated with a second set of candidate angles formed within a candidate angle space. In an illustrative example, in a case where an angle is formed between two adjacent beams of a first scale, a similar angle formed between two adjacent beams of the second scale can be smaller, thereby causing a tighter grouping of possible beam angles on the second scale. In this way, a second beam can be transmitted at a position within a first range but where the first model cannot provide an output as such, enabling the second model to identify the second beam position from one or more beam positions within a restricted search space associated with the first scale, thereby focusing the output of the second model.

[0060] In some embodiments, the analysis server generates a third radiation map based on at least one second beam position, at least one second beam intensity, and a second radiation map. For example, the analysis server can generate the third radiation map based on the analysis server determining (e.g., simulating) the radiation delivered to a patient based on at least one second beam position and at least one second beam intensity. In an example, the analysis server can generate the third radiation map based on the analysis server comparing the radiation delivered to the patient based on at least one second beam position and at least one second beam intensity with a planned target volume. Based on comparing the radiation delivered to the patient based on at least one second beam position and at least one second beam intensity with the planned target volume, the analysis server can determine the third radiation map, where the third radiation map represents one or more regions within the planned target volume that have not received a sufficient amount of energy.

[0061] In some embodiments, the analysis server iteratively repeats the functions performed with respect to operation 208 until convergence to an optimized second beam position and an optimized second beam intensity. For example, the analysis server can iteratively provide the model with successive versions of a first planned target volume and a third radiation map so that the model outputs successive versions of at least one second beam position and at least one second beam intensity. The analysis server can repeat these operations until the at least one second beam position and the at least one second beam intensity output by the model converge.

[0062] At operation 210, the analytics server transmits data associated with at least one second beam position and at least one second beam intensity to cause a medical device to deliver energy to a patient. For example, the analytics server may transmit data associated with at least one second beam position and at least one second beam intensity to cause the medical device to deliver radiation to the patient, the data being configured to cause the medical device to operate an energy delivery device based on the at least one second beam position and the at least one second beam intensity. In some embodiments, the analytics server may update the data associated with the at least one second beam position and the at least one second beam intensity to include at least one first beam position and at least one first beam intensity. The analytics server may then transmit data associated with (1) the at least one first beam position and the at least one first beam intensity and (2) the at least one second beam position and the at least one second beam intensity to cause the medical device to operate the energy delivery device.

[0063] Training models on different scales

[0064] In some embodiments, one or more of the models described with respect to process 200 may be trained. For example, one or more models (e.g., a first model and a second model) may be trained based on reinforcement learning techniques, such as model-free methods that rely on sampling and world exploration (e.g., gradient policy optimization, deep Q-learning, etc.) and model-based methods that incorporate world models to improve exploration and learning efficiency. In an example, in the case where one or more agents are associated with corresponding training models (e.g., a deep neural network (DNN), etc., which is sometimes referred to as a “policy”), the analytics server may train the training model by initializing and / or updating the weights of the training model and use the environment to test the training model. Based on the simulated output of the environment corresponding to the training model, the agent may then update the corresponding training model by updating one or more weights of the corresponding training model to cause the training model to make more accurate predictions when tested. Training using reinforcement learning techniques will be described in more detail below.

[0065] During training, the analytics server may first obtain (e.g., receive) a dataset (which may be referred to as parametric model β). The dataset may represent (e.g., β may contain) data associated with a set of patients (N patients), each patient being associated with one or more computed tomography (CT) scans, organs at risk (OAR), planning target volume (PTV), and a radiation map (sometimes referred to as a dose prescription). In this example, the dataset may represent data associated with patients previously treated using a medical device as described herein. Additionally, the analytics server may obtain a dose prediction model, which is configured to receive a CT scan, OAR, and / or PTV as input and provide a radiation map as output.

[0066] The analysis server can then initialize an experience queue (“Q”) for each of one or more agents. Additionally or alternatively, the analysis server can initialize an experience queue shared across one or more agents. In some embodiments, the experience queue represents transitions associated with one or more training models (“θ”) that are associated with one or more scales (e.g., a set of scales representing M scale levels). In some embodiments, each scale of the one or more scales can be associated with one or more beam positions corresponding to a beam position range and / or one or more beam intensities corresponding to a beam intensity range.

[0067] In some embodiments, the analysis server constructs a scale space for each of the one or more scales, which can be used to separate transitions in the experience queue. For example, the analysis server can construct a scale space that includes a first scale, a second scale, and a third scale. It should be understood that more or fewer scale spaces can be constructed as desired. In some embodiments, when one or more agents generate one or more sample experience events (described below), the sample experience events can be associated with (e.g., separated by) one or more of the scales described herein.

[0068] In some embodiments, the analysis server can initialize multiple sample experience events. For example, the analysis server can initialize multiple sample experience events, where one or more of the sample experience events correspond to each of one or more agents and / or one or more of the scales of the scale space. During each sample experience event, the analysis server can provide data associated with a separate patient in the patient set to the agent. The agent can then provide the data associated with the separate patient to a training model associated with the agent to cause the training model to provide an output. The output can include one or more beam positions and one or more beam intensities for targeting tissue in the patient's PTV.

[0069] In some embodiments, an agent provides one or more beam positions and one or more beam intensities to a simulator that generates a simulated output. For example, the agent can provide one or more beam positions and one or more beam intensities to the simulator, which generates a simulated output representing the delivery of energy to patient tissue by a medical device (e.g., a LINAC). In some embodiments, the simulated output is compared with a target radiation map (sometimes referred to as a "dose prescription") to determine a reward. For example, the simulated output can be compared with the target radiation map to determine the extent to which the simulated output corresponds to the target radiation map, and this extent represents the reward. In some embodiments, an analysis server can store data associated with each experience event in an experience queue (e.g., the training model involved, the weights of the training model involved, data associated with the individual patient involved, one or more transformations in the weights of the training model involved during the experience event, one or more rewards associated with one or more transformations, etc.).

[0070] In some embodiments, an analysis server can sample one or more parameters of one or more training models represented in the experience queue to optimize a set of models. For example, the analysis server can sample one or more parameters of one or more training models represented in the experience queue to determine a set of parameters (e.g., weights) for a model to be used during inference. In this example, each model in the set of models can correspond to a respective scale of one or more scales in a scale space. In some embodiments, the analysis server can sample one or more parameters of one or more training models to determine a set of models associated with an optimized set of parameters based on one or more transformations. For example, in a case where a first training model is associated with a first transformation having a first reward and a second training model is associated with a second transformation having a second reward less than the first reward, the analysis server can determine a model (e.g., one or more weights of the model) based on one or more weights of the first training model. In this way, the analysis server can select parameters that optimize the output of each model in the set of models. In some embodiments, once the model is optimized (e.g., converges such that the performance of the model is stable), the analysis server outputs the model. In some embodiments, the analysis server can sample one or more parameters of one or more training models represented in the experience queue to optimize the set of models, where the set of models includes models associated with each scale.

[0071] During training, the analytics server limits the exploration that each agent can perform. For example, the analytics server can limit the exploration that each agent can perform to one or more scale spaces. In an example, when optimizing a model at a given scale space, the analytics server can limit the exploration when generating the model at the given scale space by sampling empirical events at the given scale. An illustrative limitation can include limiting the locations at which a medical device can generate one or more beams based on a given scale space.

[0072] By means of the training techniques described herein, the described models can be trained to incorporate one or more preferences and / or one or more machine-specific requirements. For example, in the case where a target radiation map is generated by a clinician, the models can be generated by the analytics server such that they approximate the target radiation map. Additionally or alternatively, during simulation, simulation output can be generated based on one or more machine-specific requirements. For example, these can include a minimum angle required between two adjacent beam angles, a minimum or maximum beam intensity that can be delivered, etc. In this way, the models output by the analytics server during training can capture specific clinician requirements at inference in a variety of different possible ways, such as by leveraging the randomness of the transition dynamics to generate different solutions to select from a particular space of the parameter space and constraining the inference to that particular space, thereby ensuring that the solution is consistent with the given requirements of the medical device and can be used with the given requirements when delivering energy to a patient.

[0073] And by means of an implementation of the model as described herein (e.g., multiple sets of models configured to generate outputs at different scales), the time and / or amount of resources required to perform inference using the model can be reduced. Subsequently, this can lead to faster generation of beam positions and corresponding beam intensities for developing treatment plans. Additionally, by limiting the space in which each model can operate the medical device (e.g., beam positions at each scale space), the computational resources required to generate such beam positions and corresponding beam intensities can be reduced, enabling faster analysis of data associated with an individual by a system the same as or similar to the analytics server described herein.

[0074] Now referring Figures 2B to 2D to, a flowchart of implementation 205 related to the process for planning and delivering radiotherapy is shown. As Figures 2B to 2D shown, implementation 205 includes an electronic data source 220 (which can be the same as or similar to the Figure 1 electronic data source 120), an end-user device 240 (which can be the same as or similar to the Figure 1 end-user device 140), and an analytics server 214a (which can be the same as or similar to the Figure 1 analytics server 114a).

[0075] At operation 270, the analysis server 214a receives data associated with a first planned target volume from the electronic data source 220. At operation 272, the analysis server 214a receives data associated with a first radiation map from the end user device 240. In some implementations, the data associated with the first radiation map can include data associated with one or more user inputs that are configured to cause the analysis server 214a to generate the first radiation map. More specifically, the one or more user inputs can correspond to one or more commands provided by the user for defining regions of the patient's PTV. In these implementations, the analysis server 214a can generate the first radiation map based on one or more CT scans (or other similar scans) and / or one or more treatment plans represented by the one or more user inputs.

[0076] At operation 274, the analysis server 214a provides data to the model. For example, the analysis server 214a can provide data associated with the first planned target volume and data associated with the first radiation map to the model 211a to cause the model 211a to generate an output. In this example, the model 211a can be trained to provide an output of a first scale, as described herein. At operation 276, the analysis server 214a causes the model 211a to output data associated with at least one first beam position and at least one first beam intensity. In some implementations, the data associated with at least one first beam position and at least one first beam intensity can represent one or more beams extending outward from the isocenter (toward the device emitting energy, such as the collimator of a LINAC).

[0077] At operation 278, the analysis server 214a generates a second radiation map. At operation 280, the analysis server 214a provides data associated with the first planned volume and data associated with the second radiation map to the second model 211b. In this example, the model 211b can be trained to provide an output of a second scale, as described herein. At operation 282, the analysis server 214a causes the model 211b to output data associated with at least one second beam position and at least one second beam intensity. In some implementations, the data associated with at least one second beam position and at least one second beam intensity can represent one or more beams extending outward from the isocenter (toward the device emitting energy, such as the collimator of a LINAC).

[0078] At operation 284, the analysis server 214a transmits data associated with at least one second beam position and at least one second beam intensity. For example, the analysis server 214a transmits data associated with at least one second beam position and at least one second beam intensity to a medical device computer (e.g., a medical device computer that is the same as or similar to the medical device computer 162) so that the medical device computer can control a medical device (e.g., a medical device that is the same as or similar to the Figure 1 medical device 160) when delivering energy to a patient.

[0079] Reference Figure 3A , shows an illustration of a process 300 for generating a fluence map when planning radiotherapy according to an embodiment. The process 300 may include operations 302-308. However, other embodiments may include additional or alternative operations, or may omit one or more operations entirely. The process 300 is described as being performed by an analysis server 114a that is the same as or similar to the Figure 1 . However, one or more steps of the process 300 may be performed by any number of computing devices operating in a distributed computing system described in Figure 1 (fully, partially, etc.). For example, one or more computing devices may perform Figure 3A some or all of the operations described therein.

[0080] At operation 302, the analysis server receives data associated with a plurality of beams. For example, the analysis server may receive data associated with a plurality of beams that may be generated by one or more devices (e.g., an electron gun and / or a waveguide) associated with a medical device (e.g., a medical device that is the same as or similar to the Figure 1 medical device 160). Although reference is made to a medical device that is the same as or similar to a LINAC, it should be understood that the techniques described herein may be implemented using any suitable medical device capable of delivering energy to at least a portion of a patient as described herein. In some embodiments, the plurality of beams is associated with a treatment plan for a patient. For example, the plurality of beams may be specified by a treatment plan generated using, for example, the techniques for planning radiotherapy discussed above with respect to process 200 (see Figure 2A ).

[0081] In some embodiments, the analysis server receives data associated with a plurality of leaves corresponding to each of a plurality of beams. For example, in the case where a LINAC is configured to deliver energy to a patient, the LINAC can be configured to set one or more positions of one or more leaves when generating a corresponding beam. In this example, the one or more leaves can be the leaves of a collimator of the LINAC, which is configured to cause the LINAC to generate a beam of a certain shape. In some embodiments, when arranged to cause the LINAC to generate a beam of a certain shape, the one or more positions of the one or more leaves can represent the configuration of the plurality of leaves. This configuration can also be associated with the shape of the beam intended to be transmitted to the body, which targets at least a portion of the patient's tissue (e.g., the tissue associated with the PTV). This configuration can be used to generate a beam's-eye view image as described herein.

[0082] In some embodiments, the analysis server receives data associated with a plurality of positions of a plurality of leaves, which are also associated with a plurality of time steps. For example, the analysis server can receive data associated with a plurality of positions of a plurality of leaves, and the set of positions at each time step is called a sub-configuration. In an illustrative example, in the case where there is a first leaf and a second leaf in a leaf pair, the plurality of positions can represent the movement of the first leaf and the second leaf along a common direction, forming a shape of iterative movement such that the corresponding beam travels across the PTV as it extends through the patient's body. In other examples, it should be understood that there can be multiple leaf pairs, and the leaf pairs can move synchronously with each other, in different directions from each other, and / or at different rates from each other across a plurality of time steps.

[0083] At operation 304, the analysis server generates a BEV image based on a plurality of beams. For example, the analysis server can generate a BEV image based on data that the analysis server receives and / or determines to be associated with a plurality of beams. For example, the analysis server can generate a BEV based on the beams (e.g., the relative position between the axis along which the beam is transmitted with respect to the patient and / or the point where the medical device generates energy and the points along the PTV). In addition, the analysis server can generate a BEV based on the power level associated with the beam. For example, the analysis server can generate a BEV based on the power level at which the device in the medical device that generates the beam is configured to generate power. In some embodiments, the power level for generating each beam can be constant. Alternatively, the power level for generating each beam is variable.

[0084] In some embodiments, the analysis server generates a BEV image based on the configuration of multiple leaves. For example, as described above, a medical device (e.g., a LINAC) may include a collimator having multiple leaves. In this example, the multiple leaves may be located at positions representing the configuration. Then, the analysis server may generate a BEV image based on the configuration. In this example, the BEV image may represent the beam shaped by the multiple leaves in the configuration. In some embodiments, the analysis server generates a BEV image based on the energy delivery of the medical device at each of multiple time steps. For example, in the case of delivering multiple beams at multiple corresponding time steps, the analysis server may generate a BEV image at each of the multiple time steps.

[0085] At operation 306, the analysis server determines a DRR based on the cascaded BEV images. For example, the analysis server may determine (e.g., generate) a DRR based on the BEV images cascaded over multiple time steps. In this example, the multiple time steps may be associated with a patient's treatment plan. The DRR may represent a multi-channel 2D image that can be provided to the model as described herein. For example, the DRR may include multiple channels, where each channel in the multiple channels corresponds to a BEV image in the set of BEV images cascaded by the analysis server.

[0086] At operation 308, the analysis server provides the DRR to the model to cause the model to generate an output associated with a fluence map. In some embodiments, the output of the model may include a fluence map prediction for each beam associated with the DRR, where the set of all beams is represented as K beams The model may also be associated with a loss function defined as where y i is the reference fluence map for the i-th beam, and is the distance between y i and . In some embodiments, each y i may also be a vector, where each element captures different characteristics (e.g., minimum dose, maximum dose, average dose, etc.) of the line projection through the 3D dose map. In this case, the goal of the network is to disentangle the information and calculate the expected fluence map for each beam angle given the total 3D dose. Although the above description is associated with the beams generated during intensity-modulated radiation therapy (IMRT), in the case of volumetric modulated arc therapy (VMAT), a finite set of angles from the arc can be considered as one or more of the beams described, and the same analysis is performed. In some embodiments, the analysis server may consider interpolation between discrete angles to calculate a continuous arc-based fluence map.

[0087] In some embodiments, the analysis server may provide DRRs to a model, where the model includes one or more encoders and one or more decoders. For example, in the case where the model is associated with a U-net architecture, the model may include one or more encoders associated with a contracting path and one or more corresponding decoders associated with an expanding path, the one or more encoders and one or more decoders being associated with one or more levels. In this example, the analysis server may provide DRRs to the model by providing individual BEV images to a first encoder at a first level of the model. The first encoder may be configured to receive the individual BEV image and provide two outputs: a first output that is provided by the first encoder along the contracting path to a second encoder; and a second output that is provided to a first decoder corresponding to the first encoder, the first decoder being associated with the expanding path. Depending on the number of layers of the model, this process may be repeated for corresponding encoders and decoders at each successive layer of the model. In some embodiments, at the last (or nth) layer of the model, the output of the encoder at the nth layer is provided as the input to the decoder at the nth layer. The decoder then provides an output to the decoder associated with the layer above the nth layer, which continues for each successive layer until the first decoder is reached.

[0088] In some embodiments, the output of each encoder may be provided to one or more convolutional layers associated with the encoder before being provided along the contracting path of the model to the next encoder, or in the case of the nth encoder, along the expanding path of the model to the nth decoder. Additionally or alternatively, the output of each encoder may be provided to a max-pooling layer (e.g., after passing through one or more convolutional layers) before being provided along the contracting path of the model to the next encoder. In some embodiments, the output of each decoder along the expanding path of the model may be provided to a transposed convolutional layer. Additionally or alternatively, the output of each decoder may be provided to one or more convolutional layers (e.g., after passing through the transposed convolutional layer). In some embodiments, the last decoder of the model may provide one or more fluence maps as output. For example, the last decoder of the model may provide one or more fluence maps as output, where the one or more fluence maps correspond to the one or more BEV images of the DRR provided as input to the model.

[0089] In some embodiments, the analysis server may generate a leaf motion sequence. For example, the analysis server may generate a leaf motion sequence based on the analysis server generating one or more fluence maps. In the example, the analysis server may generate a leaf motion sequence based on the one or more fluence maps and confirm whether the leaf sequence ultimately achieves a desired dose volume histogram (e.g., confirm whether a desired amount of energy is delivered to the patient's PTV).

[0090] In some embodiments, the analysis server may train a model. For example, the analysis server may train a model based on one or more BEV images and one or more corresponding fluence maps. In this example, one or more BEV images and one or more corresponding fluence maps may be generated at an earlier time point. In some embodiments, the analysis server trains a model based on one or more BEV images and one or more corresponding fluence maps, where the fluence map represents a measurement of the energy delivered to a patient during patient treatment.

[0091] In some embodiments, the analysis server trains a model based on the analysis server providing one or more BEV images to the model such that the model provides an output that represents a fluence map. The analysis server may then compare the output representing the fluence map with a corresponding fluence map (e.g., a fluence map generated at an earlier time point corresponding to the input BEV image). In this example, the analysis server may determine the difference between the output representing the fluence map and the corresponding fluence map (generated at an earlier time), and the analysis server may update the layers of the model. In some embodiments, the analysis server may update the layers of the model using one or more backpropagation techniques (e.g., stochastic gradient descent, root mean square propagation, etc.).

[0092] Now referring to Figures 3B to 3D , a flowchart of implementation 305 is shown, which relates to a process for generating a fluence map when planning radiotherapy. As Figures 3B to 3D shown, implementation 305 includes an electronic data source 320 (which may be the same as or similar to the Figure 1 electronic data source 120) and an analysis server 314a (which may be the same as or similar to the Figure 1 analysis server 114a).

[0093] In operation 370, the analysis server 314a receives data associated with a plurality of beams. In some implementations, the analysis server 314a may receive data associated with a plurality of beams to be generated and transmitted by a medical device (e.g., a medical device such as the Figure 1 medical device 160, such as a LINAC). In operation 372, the analysis server 314a generates a BEV image. In some implementations, the analysis server 314a generates a BEV image based on the data associated with the plurality of beams. Additionally, the analysis server 314a may generate a BEV image based on a predicted geometry associated with the medical device (e.g., based on a dosimetry model such as an analytical model, Monte Carlo simulation, deterministic model, etc.).

[0094] At operation 374, the analysis server 314a determines the DRR. In some implementations, the analysis server 314a determines the DRR based on the BEV image. For example, the analysis server 314a can determine the DRR based on analyzing server-cascaded BEV images to generate the DRR.

[0095] At operation 376, the analysis server 314a provides data associated with the DRR to the model 311. In some implementations, the analysis server 314a provides data associated with the DRR to the model 311, where the model 311 is associated with a U-net architecture. In an example, the model 311 can be trained as described herein to output a fluence map. At operation 378, the analysis server 314a causes the model 311 to output a fluence map.

[0096] Reference Figure 4A , shows an illustration of a process 400 for generating a fluence map when planning radiotherapy according to an embodiment. The process 400 can include operations 402-412. However, other embodiments can include additional or alternative operations, or can omit one or more operations entirely. The process 300 is described as being performed by an analysis server 114a that can be the same as or similar to Figure 1 the analysis server 114a. However, one or more steps of the process 300 can be performed by any number of computing devices operating in a distributed computing system described in Figure 1 For example, one or more computing devices can perform Figure 3A some or all of the operations described in

[0097] At operation 402, the analysis server receives input data associated with a target fluence map. For example, the analysis server can receive input data associated with a target fluence map, where the target fluence map represents a target amount of energy to be delivered to a patient. In this example, the target amount of energy to be delivered to the patient can be associated with energy delivery via a medical device such as a LINAC (e.g., a medical device the same as or similar to Figure 1 medical device 160). In some embodiments, the target amount of energy can be expressed as the intensity of energy (e.g., radiation) to be delivered per unit area of the patient's tissue (e.g., per unit area of the PTV). In some embodiments, the target fluence map corresponds to the angle at which the PTV can be viewed.

[0098] In some embodiments, the target fluence map is represented as a vector image. For example, the analysis server can receive a target fluence map represented as a vector image. Additionally or alternatively, the analysis server can receive a target fluence map represented as a two-dimensional image (e.g., a rasterized image). In this example, the analysis server can update the target fluence map such that the target fluence map is a vector image that represents (e.g., is very close to) the two-dimensional image. In this way, the operations discussed herein can be performed based on a smaller state space (e.g., using the target fluence map and the cumulative fluence map described herein, which can be, for example, one three-hundredth the size of when represented as a two-dimensional image), thereby saving a significant amount of computational resources during the inference and training phases described herein.

[0099] At operation 404, the analysis server provides input data to the model to cause the model to generate an output that represents a cumulative fluence map and a leaf mask. In some embodiments, the analysis server can provide input data to the model to cause the model to generate an output, where the model includes one or more second models. For example, the analysis server can provide input data to a model that includes one or more second models, where each of the one or more second models is associated with a leaf pair of a medical device. In this example, input can be provided to the one or more second models to cause the one or more second models to generate corresponding outputs that represent one or more leaf masks and a cumulative fluence map. It should be understood that the one or more second models can be configured to receive input data and provide, as its output, data associated with a fluence map and / or a leaf mask corresponding to a region that can be addressed by a corresponding leaf pair. The model and the one or more second models can be trained as described herein.

[0100] In some embodiments, when the input data is initially provided as an input to the model, the input data can also be associated with a cumulative fluence and a current leaf mask. For example, the input data can also be associated with a cumulative fluence and a current leaf mask, where the cumulative fluence is equal to zero or a zero value, and where the current leaf mask is associated with an initial position of each leaf of a leaf pair of a collimator that can be adjusted during LINAC operation. As described below, the cumulative fluence and the current leaf mask can be iteratively updated at each time step when planning a patient's treatment.

[0101] At operation 406, the analysis server generates updated input data associated with the target dose map, the cumulative dose map, and the leaf masks. For example, the analysis server can generate updated input data associated with the target dose map, the cumulative dose map, and the leaf masks based on the analysis server determining the difference between the cumulative dose map and the target dose map. In this example, the analysis server can determine the difference between the cumulative dose map and the target dose map at each time step when planning a patient treatment. In some embodiments, the analysis server can determine the difference between the cumulative dose map and the target dose map based on the analysis server comparing the intensity value of the energy per unit area of the target dose map with the intensity value of the energy per unit area of the cumulative dose map. Then, when compared with one or more intensity values of the target dose map, the analysis server can determine whether the difference between one or more intensity values of the intensity values of the cumulative dose map meets a first threshold amount. In the example, the first threshold amount can be a value associated with the difference between the aggregated intensity value of the cumulative dose map and the aggregated intensity value of the target dose map. Additionally or alternatively, the first threshold amount can be a value associated with the difference between one or more intensity values of the cumulative dose map and one or more corresponding intensity values of the target dose map.

[0102] At operation 408, the analysis server provides the updated input data to the model to cause the model to generate an updated output that represents the updated cumulative dose map and the updated leaf masks. For example, the analysis server can provide the updated input data to the model such that the model generates an updated output based on the analysis server updating the input data. In some embodiments, the analysis server provides the updated input data to the model to cause the model to generate a second updated output that represents a second updated cumulative dose map and a second updated leaf mask. In some embodiments, the analysis server can provide the updated input data to the model to cause the model to generate an output, where the model is the same or similar to the above model (e.g., with respect to operation 404).

[0103] At operation 410, the analysis server determines that the difference between the updated cumulative fluence map and the target fluence map satisfies a second threshold amount. For example, the analysis server can determine the difference between the updated cumulative fluence map and the target fluence map based on the analysis server comparing the intensity value of the energy per unit area of the updated cumulative fluence map and the intensity value of the energy per unit area of the target fluence map. Then, when compared with one or more intensity values of the target fluence map, the analysis server can determine whether the difference between one or more intensity values of the intensity values of the updated cumulative fluence map satisfies the second threshold amount. In an example, the second threshold amount can be a value associated with the difference between the aggregated intensity value of the updated cumulative fluence map and the aggregated intensity value of the target fluence map. Additionally or alternatively, the second threshold amount can be a value associated with the difference between one or more intensity values of the updated cumulative fluence map and one or more intensity values of the target fluence map. In some embodiments, the second threshold amount can be less than the first threshold amount. In this way, the analysis server can determine whether the energy delivered to the PTV meets the patient's treatment plan.

[0104] In some embodiments, the analysis server repeats operations 404 - 410. More specifically, the analysis server can repeat operations 404 - 410, where at operation 410, the difference between one or more intensity values of the updated cumulative fluence map and one or more intensity values of the target fluence map satisfies the first threshold amount rather than the second threshold amount. In this way, the analysis server can iteratively provide data to the model and generate output data that can be used to generate a leaf sequence that causes energy to be delivered during patient treatment that meets the patient's treatment plan.

[0105] At operation 412, the analysis server determines a leaf sequence based on the leaf mask and the updated leaf mask. For example, the analysis server can determine the leaf sequence based on the analysis server determining the difference between the leaf mask and the updated leaf mask. In this example, the analysis server can determine the leaf sequence based on the difference between the leaf mask and the updated leaf mask. In some embodiments, in the case of iteratively repeating operations 404 - 410 to address a larger PTV, the leaf mask and the updated leaf mask can be cascaded. In this case, the analysis server can determine the leaf sequence based on the analysis server determining the difference between the iteratively generated updated leaf masks.

[0106] In some embodiments, the analysis server can determine a leaf sequence based on the first leaf position and the second leaf position determined by the analysis server associated with a leaf pair of the LINAC. For example, the analysis server can determine the first leaf position and the second leaf position associated with the leaf pair, where the first leaf position and the second leaf position correspond to a leaf mask. In some embodiments, the analysis server can then determine the third leaf position and the fourth leaf position associated with the leaf pair. For example, the analysis server can determine the third leaf position and the fourth leaf position associated with the leaf pair, where the third leaf position and the fourth leaf position correspond to an updated leaf mask. The analysis server can then determine the difference between the leaf mask and the updated leaf mask, which is represented by the change in the position of each leaf of the leaf pair. For example, the analysis server can then determine the difference between the leaf mask and the updated leaf mask, which is represented by the following changes: (1) the change in the first leaf position when compared with the third leaf position, and (2) the change in the second leaf position when compared with the fourth leaf position. Similar to the above, this can be performed iteratively according to the number of generated updated leaf masks.

[0107] Training model for determining leaf sequence

[0108] In some embodiments, one or more of the models described with respect to process 400 can be trained. For example, one or more models (e.g., a first model and one or more second models) can be trained according to techniques based on reinforcement learning. Schulman et al. describe examples of reinforcement learning techniques, "Proximal Policy Optimization Algorithms", accessible at https: / / arxiv.org / abs / 1707.06347. Given a policy network π θ , state s t and action a t , the probability ratio can be expressed as The main objective may be where is an estimator of the advantage function at time step t, is the empirical average over a finite batch of samples. An alternative to this clipped surrogate objective is to use KL divergence as a penalty between the current policy and the old policy: Another network V θ (except for the policy network) can be used to estimate the value function, and this network is trained with a squared error loss .

[0109] In some embodiments, one or more of the models described with respect to process 400 may be trained based on one or more model-free methods as well as model-based methods that incorporate world models for improved exploration and learning efficiency. In an example, where one or more agents are associated with a corresponding trained model (e.g., a deep neural network (DNN), etc., which is sometimes referred to as a “policy”), each of the one or more agents may also be assigned to train the corresponding trained model based on one or more leaf pairs assigned to the agent. In an example, each leaf pair of a medical device (e.g., a LINAC) may be assigned to an agent from among the one or more agents. In some embodiments, the analysis server may train the trained model corresponding to each agent by initializing and / or updating the weights of the trained model and test the trained model using the environment. Based on the simulated output of the environment corresponding to the trained model, the agent may then update the corresponding trained model by updating one or more weights of the corresponding trained model to cause the trained model to make more accurate predictions during testing. Training using reinforcement learning-based techniques will be described in more detail below.

[0110] During training, the analysis server may first obtain (e.g., receive) a data set. The data set may represent data associated with a set of patients (N patients), each patient associated with one or more computed tomography (CT) scans, organs at risk (OARs), planning target volumes (PTVs), and / or radiation maps (sometimes referred to as dose prescriptions). In this example, the data set may represent data associated with patients previously treated using a medical device as described herein.

[0111] The analysis server may next initialize an experience queue (“Q”) for each of the one or more agents. Additionally or alternatively, the analysis server may initialize an experience queue shared across the one or more agents. In some embodiments, the experience queue represents one or more sample experience events associated with one or more trained models (“θ”), the sample experience events representing transitions that may also represent, for example, changes in leaf positions (e.g., a first pair of leaf positions and a second pair of leaf positions). Additionally, the experience queue may represent one or more power levels associated with the transitions, where energy is delivered to the patient at that power level when one or more leaves of the leaf pair are moved between the first pair of leaf positions, the second pair of leaf positions, and / or one or more intermediate positions.

[0112] In some embodiments, the empirical queue may represent one or more speeds at which one or more leaves move when transitioning from a first pair of leaf positions to a second pair of leaf positions. For example, when the leaf pair moves between the first pair of leaf positions and the second pair of leaf positions, the analysis server may determine the speed at which each leaf moves. In some examples, the minimum speed may be associated with a minimum movement unit (e.g., "1 cm / s"). In an example, the maximum speed may be associated with a maximum movement unit (e.g., "3 cm / s"). Based on the clinical situation, the leaves used during IMRT may move only in one direction, while the leaves used during VMAT may move in two directions. In an illustrative example, with respect to the left and right leaves, a 4x4 action may be performed for IMRT: {0, s}, where s ∈ {1, 2, 3} (0 is stationary, and s is the movement speed), and a 7x7 action may be performed for VMAT: {s, 0, s}, where s ∈ {1, 2, 3} (-s is backward movement). It should be understood that "s" may represent any positive integer such that the positive integer does not exceed the constraints of the LINAC.

[0113] In some embodiments, the empirical queue may represent the amount of energy delivered to the patient at a time step and with respect to a given leaf pair (sometimes referred to as the dose rate). For example, at each time step during the delivery of energy to the patient, the analysis server may determine the dose rate d. In this example, the analysis server may determine the dose rate based on data provided by the analysis server associated with the positions of the LINAC's leaves at one or more time steps, the power levels associated with the energy transfer of the LINAC at one or more time steps, and the position and orientation of the PTV relative to the LINAC's emitted energy device. In some embodiments, the analysis server may store each sampled empirical event in the empirical queue in association with a corresponding reference fluence map, cumulative fluence map, and / or leaf position before, during, and / or after the leaf movement during the transition.

[0114] In some embodiments, one or more sampled empirical events may be associated with a reward. For example, the analysis server may determine one or more rewards for each sampled empirical event based on the transitions involved in the sampled empirical event. Rewards may include, for example, a reward for a sampled empirical event in which the leaves in the leaf pair do not collide during the transition. In an illustrative example, in the case where the position of the first (e.g., left) leaf is greater than the position of the second (e.g., right) leaf, indicating that the first leaf has collided with the second leaf, the analysis server may assign a lower reward to the sampled empirical event. Alternatively, in the case where the position of the first (e.g., left) leaf is less than the position of the second (e.g., right) leaf, indicating that the first leaf has not collided with the second leaf, the analysis server may assign a higher reward to the sampled empirical event than the reward assigned for a collision.

[0115] In an example, the analysis server can determine a reward based on a comparison of the cumulative fluence represented by the cumulative fluence map of a given sample experience event with the target fluence represented by the target fluence map. For example, in a case where the difference between the cumulative fluence and the target fluence is less than the similar differences of different sample experience events, a higher reward can be assigned to the sample experience event. Alternatively, if the difference between the cumulative fluence and the target fluence is greater than the similar differences of different sample experience events, a lower reward can be assigned to the sample experience event. For example, compared with other sample experience events, in a case where a sample experience event causes an increase in energy delivery to one or more OARs closer to the PTV, the sample experience event that delivers more energy to the OAR can be assigned a lower reward. In this way, a higher reward can be assigned to the sample experience event that causes more precise energy delivery to at least a portion of the PTV.

[0116] In other examples, the analysis server can determine the reward of a sample experience event based on the movement speed of each leaf in a leaf pair. For example, compared with other example experience events, the analysis server can assign a higher reward to the example experience event in which one or both leaves move slower during the transition. In this way, the analysis server can increase the importance of the sample experience event with slower leaf movement. This can in turn reduce the chance of systematic errors (such as mechanical errors, errors in delivering energy to the correct part of the PTV, etc.).

[0117] In some embodiments, one or more sample experience events in the sample experience queue can be associated with a given PTV. For example, in a case where the training set represents multiple PTVs and corresponding target fluence maps, one or more sample experience events can be associated with the energy delivery to the specific PTV. In this way, groups of sample experience events (such as those representing operations performed by one or more agents during energy delivery to the PTV) can be grouped together before sampling.

[0118] In some embodiments, the analysis server may sample one or more parameters of one or more training models of a corresponding agent represented in an experience queue to optimize a set of models. For example, the analysis server may sample one or more parameters of one or more training models represented in an experience queue to determine a set of parameters of a model to be used during inference. In this example, each model in the set of models may correspond to a leaf pair of a LINAC. In some embodiments, the analysis server may sample one or more parameters of one or more training models to determine a set of models associated with an optimized parameter set based on one or more transformations. For example, in a case where a first training model is associated with a first transformation having a first reward and a second training model is associated with a second transformation having a second reward less than the first reward, the analysis server may determine a model (e.g., one or more weights of the model) based on one or more weights of the first training model. In this way, the analysis server may select parameters that optimize the output of each model in the set of models. In some embodiments, once the model is optimized (e.g., converges such that the performance of the model is stable), the analysis server outputs the model. In some embodiments, the analysis server may sample one or more parameters of one or more training models represented in an experience queue to optimize a set of models, where the set of models includes models associated with each scale.

[0119] In preliminary experiments, based on data from prostate and head and neck cancer patients, the total number of plans for each site was established (as cited in Table 1 below, the data was filtered due to missing data, quality issues, and incompatibility with the environment / software).

[0120] Table 1. Number of plans (e.g., patients) for different sites

[0121]

[0122] Each plan has several fluence maps. For an IMRT plan, each angle corresponds to a fluence map, and each plan typically has 4 - 10 angles. For a VMAT plan, the number of fluences depends on the scale. For example, given an arc in VMAT with 180 control points, if the scale is 18 (18 control points form a fluence map), then the arc can generate at least 180 / 18 = 10 samples for testing. During training, the data is augmented by randomly selecting starting control points. In the experiment, 10% of the plans were used for testing.

[0123] Leaf sequencing can be evaluated in the following ways: 1) comparing the target fluence and the predicted fluence map (generated by the predicted leaf sequence), 2) comparing the DVH of the downstream dose calculation with Acuros XB. The error of 1) is calculated by the following formula: where F and are the target fluence and the predicted fluence, and Y is the fluence magnitude of a single leaf pair, and X is the number of leaf pairs for that fluence. Preliminary experiments for both the prostate and head and neck sites are as follows:

[0124] Table 2: Quantitative results for the prostate and head and neck sites of VMAT

[0125]

[0126] Now referring to Figures 4B to 4E , a flowchart of implementation 405 related to a process for generating a fluence map when planning radiotherapy is shown. As Figures 4B to 4E shown, implementation 405 includes an electronic data source 420 (which may be the same as or similar to Figure 1 the electronic data source 120 of Figure 1 ), an end-user device 440 (which may be the same as or similar to Figure 1 the end-user device 140 of

[0127] In operation 470, the analysis server 414a receives data associated with a target fluence map from the electronic data source 420. In some implementations, one or more of the techniques described herein with respect to Figures 3A to 3D may be used to generate the target fluence map. In operation 472, the analysis server 414a provides input data associated with the target fluence map to the model 211a. In some implementations, the model 211a may include one or more sub-models, each sub-model corresponding to a component of a medical device (e.g., one or more leaves of a collimator of a medical device that is the same as or similar to Figure 1 the medical device 160 of

[0128] In operation 474, the analysis server 414a causes the model 211a to output data associated with a cumulative fluence map and a leaf mask. In some implementations, the cumulative fluence map is associated with the cumulative amount of energy delivered to the patient's PTV, and the leaf mask is associated with the position of each leaf of the medical device at a particular point in time.

[0129] At operation 476, the analysis server 414a provides updated input data to the model 211a. In some implementations, the analysis server 414a provides updated input data to the model based on the analysis server 414a determining the updated input data. In these implementations, the analysis server 414a may determine the updated input data (e.g., at operation 474) based on data associated with the cumulative fluence map and leaf masks output by the model 211a. At operation 478, the analysis server 414a causes the model 211a to output data associated with the updated cumulative fluence map and the updated leaf masks.

[0130] At operation 480, the analysis server 414a determines that the difference between the updated cumulative fluence map and the target fluence map meets a threshold amount. In some implementations, the analysis server 414a determines that the difference between the updated cumulative fluence map and the target fluence map meets the threshold amount based on the analysis server 414a comparing the intensity value of the energy per unit area of the updated cumulative fluence map with the intensity value of the energy per unit area of the target fluence map.

[0131] At operation 482, the analysis server 414a determines a leaf sequence. In some implementations, the analysis server 414a determines the leaf sequence based on the leaf masks and the updated leaf masks. For example, the analysis server 414a may determine the leaf sequence based on the position of each leaf in the leaves represented by the leaf mask and the position of each leaf in the leaves represented by the updated leaf mask. In some implementations, the analysis server 414a also determines one or more power levels at which to deliver energy corresponding to each leaf mask. At operation 484, the analysis server 414a outputs data associated with the leaf sequence. In some implementations, the data associated with the leaf sequence may be configured to cause control of the medical device when delivering energy to the patient.

[0132] Figure 4F Shows Figures 4B to 4E An example iteration of an implementation.

[0133] Figure 4G Shows an example comparison of the target (e.g., cumulative) fluence delivered to a patient and the predicted fluence delivered to the patient according to an embodiment.

[0134] Figure 4H Shows an example pair of a predicted fluence map and a target (e.g., cumulative) fluence map according to an embodiment.

[0135] Figure 5A And Figure 5B Shows an example training pipeline involving multiple systems according to an embodiment. As will be described, the training of one or more of the models described herein can be performed in an end-to-end manner. More specifically, as regarding Figures 2A to 2DAs discussed, one or more of the models described therein can be used when converting a planned target volume into a set or sets of beam positions (which can be described with respect to the beam angles formed by the beams during delivery of energy to a patient) and beam intensities. With respect to Figures 3A to 3D , different models can be used to convert a beam (e.g., a beam output by one or more of the models described with respect to Figures 2A to 2D ) into a fluence map. And with respect to Figures 4A to 4E , another different model can be used to convert a target fluence map (which can be the same as or similar to the fluence map output by the model of Figures 3A to 3D ) into a leaf sequence. These models can be trained in such an end-to-end manner to improve the overall performance of a system that is configured to initially receive a planned target volume and provide a leaf sequence as output to control a LINAC during patient treatment.

[0136] Continuing to refer to Figure 5A and Figure 5B , shown is the assembly of multiple models (referred to as an ensemble) when processing data associated with a planned target volume and a radiation map to generate a treatment plan that can be executed using a medical device described herein (e.g., a medical device the same as or similar to medical device 160 of Figure 1 ). As shown, a first system 502 can include one or more devices that are capable of receiving data associated with a planned target volume (e.g., data associated with a CT scan, PTV, and OAR masks) and determining a set or sets of beam angles and corresponding beam intensities. A second system 504 can include one or more devices that are capable of receiving data associated with multiple beams (e.g., data associated with one or more of the beam angles and corresponding beam intensities discussed with respect to the first system 502) and determining a fluence map. A third system 506 can include one or more devices that are capable of receiving data associated with a fluence map (e.g., a target fluence map) and determining a leaf sequence.

[0137] In some embodiments, the first system 502 can include one or more systems or devices that are the same as or similar to one or more systems or devices described with respect to Figures 2A to 2D ; the second system 504 can include one or more systems or devices that are the same as or similar to one or more systems or devices described with respect to Figures 3A to 3D ; and the third system 506 can include one or more systems or devices that are the same as or similar to one or more systems or devices described with respect to Figures 4A to 4H .

[0138] Continuing to refer to Figure 5A and Figure 5B , during training, the first system 502 (e.g., one or more models implemented by the first system 502) can be trained (as described above with respect to Figures 2A to 2Duntil the first system 502 converges. In some embodiments, when the first system 502 is pre-trained or after the first system 502 converges during training, the second system 504 (e.g., one or more models implemented by the second system 504) can be trained (as described above with respect to Figures 3A to 3D until the second system 504 converges. In some embodiments, the first system 502 and the second system 504 can be trained simultaneously based on the inputs and outputs of each respective system. For example, the first system 502 or the second system 504 can be fixed (e.g., not trained), while the other of the first system 502 or the second system 504 is trained. Once the first system 502 or the second system 504 is trained respectively, they can be locked (sometimes referred to as "fixed"), and the other of the first system 502 or the second system 504 is trained. In some embodiments, the second system 504 and / or the third system 506 can be fixed when training the first system 502. In some embodiments, the first system 502 or the second system 504 can be fixed when training the third system 506 (see Figure 5B ). It should be understood that when one system is fixed, the other systems can receive, process, and provide outputs that ultimately result in 3D dose calculations. These 3D dose calculations can then be compared with the initially received 3D dose calculations and the differences used to update the weights of one or more non-fixed systems to improve the overall performance of the corresponding systems. In this way, by fixing different systems during the end-to-end training process, the systems can coordinate with each other to globally find the optimal solution during the training process.

[0139] In some embodiments, the first system 502, the second system 504, or the third system 506 is pre-trained. For example, the first system 502, the second system 504, or the third system 506 can be pre-trained by separately training each system based on the techniques described with respect to Figures 2A to 2D , Figures 3A to 3D and Figures 4A to 4H respectively. In an example, the first system 502 and / or the second system 504 can be pre-trained, and pre-trained models involved in training another model during the end-to-end training of the model are in the first system 502, the second system 504, and / or the third system 506. In some embodiments, when training the second system 504 and / or the third system 506, the first system 502 can implement a physics-based dose calculation system.

[0140] In some embodiments, a fluence prediction module and a dose prediction module are used. The fluence prediction module may receive as inputs a CT scan, a PTV mask, an OAR mask (provided to the first system 502), and a beam geometry (output by the first system 502), and the fluence prediction module may output a fluence map representing each corresponding beam angle (for IMRT) or each sub - field (for VMAT). In this way, when training the second system 504 and / or the third system 506, the fluence prediction module can be used in place of the first system 502 or in cooperation with the first system.

[0141] In some embodiments, a dose calculation module receives a fluence map (which may be represented as a combination of a leaf sequence and / or corresponding power levels output by the third system 506) and outputs a 3D dose distribution (e.g., represented by a radiation map). The quality of the 3D dose distribution is evaluated (e.g., compared to a predetermined dose prescription), and the difference between 3D dose distributions is used to train the system (e.g., update one or more weights of a model implemented by the first system 502, the second system 504, and / or the third system 506).

[0142] Some non - limiting embodiments of the present disclosure are described herein in connection with thresholds. As used herein, meeting a threshold may refer to a value that is greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0143] Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, a hardware description language, or any combination thereof. A code segment or machine - executable instruction may represent a process, a function, a sub - routine, a program, a routine, a sub - routine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. By passing and / or receiving information, data, arguments, parameters, or memory contents, a code segment may be coupled to another code segment or a hardware circuit. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0144] The actual software code or dedicated control hardware used to implement these systems and methods does not limit the claimed features or the present disclosure. Thus, the operation and behavior of the systems and methods are described without reference to specific software code, understanding that software and control hardware can be designed based on the description herein to implement the systems and methods.

[0145] When implemented in software, the functionality can be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of the methods or algorithms disclosed herein can be embodied in a processor-executable software module that can reside on a computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable media include both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. A non-transitory processor-readable storage medium can be any available medium that is accessible by a computer. By way of example and not limitation, such non-transitory processor-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer or a processor. As used herein, disk and optical disks include compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), floppy disk, and Blu-ray disk, where disks typically reproduce data magnetically, while optical disks reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. In addition, the operations of a method or algorithm can reside as one or any combination or collection of code and / or instructions on a non-transitory processor-readable medium and / or a computer-readable medium, which can be incorporated into a computer program product.

[0146] The foregoing description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the embodiments and variations thereof described herein. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein can be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the claims and the principles and novel features disclosed herein.

[0147] Although various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various aspects and embodiments disclosed are for illustrative purposes and are not intended to be limiting, where the true scope and spirit are indicated by the appended claims.

Claims

1. A system, comprising: at least one processor programmed to: receive data associated with a first planned target volume and a first radiation map, the first planned target volume representing a region of a patient's body and a target mask, the target mask representing a portion of the region of the patient's body to receive radiation; provide the first planned target volume and the first radiation map to a first model to cause the first model to output data associated with at least one first beam position and at least one first beam intensity, the at least one first beam position corresponding to the at least one first beam intensity, generate a second radiation map based on the at least one first beam position, the at least one first beam intensity, and the first radiation map; provide the first planned target volume and the second radiation map to a second model to cause the second model to output data associated with at least one second beam position and at least one second beam intensity; and transmit data associated with the at least one second beam position and the at least one second beam intensity to cause a linear accelerator (LINAC) to deliver radiation to the patient.

2. The system of claim 1, wherein the first model is trained to output data associated with first beam positions of a first scale, wherein the second model is trained to output data associated with second beam positions of a second scale, and wherein the first radiation map is associated with a first resolution that is less than a resolution of the second radiation map.

3. The system of claim 2, wherein the first scale is associated with a first set of candidate angles within a candidate angle space, wherein the second scale is associated with a second set of candidate angles within the candidate angle space, and wherein an angular measurement associated with the first set of candidate angles is greater than an angular measurement associated with the second set of candidate angles.

4. The system of claim 3, wherein the beams associated with the second set of candidate angles correspond to the beams associated with the first set of candidate angles.

5. The system of claim 1, wherein the first model is trained based on a reinforcement learning agent and training data associated with previously treated patients.

6. The system of claim 5, wherein the training data represents planned target volumes and target radiation maps corresponding to the previously treated patients; and wherein when training, the at least one processor is further programmed to: for each previously treated patient among the previously treated patients: generate data associated with beam positions and beam intensities based on a plurality of reinforcement learning agents, each reinforcement learning agent being associated with a training model, provide the data associated with beam positions and beam intensities to a simulator to generate a set of simulated outputs, determine a reward for each reinforcement learning agent among the plurality of reinforcement learning agents based on the corresponding simulated outputs and the target radiation map, and select the first model based on the reward for each reinforcement learning agent.

7. The system according to claim 5, wherein the target radiation pattern represents a radiation pattern previously generated based on clinician input, and the target radiation pattern represents the clinician's preference when delivering radiation to the previously treated patient.

8. A method, comprising: receiving, by at least one processor, data associated with a first planned target volume and a first radiation pattern, the first planned target volume representing a region of a patient's body and a target mask, the target mask representing a portion of the region of the patient's body to receive radiation; providing, by the at least one processor, the first planned target volume and the first radiation pattern to a first model such that the first model outputs data associated with at least one first beam position and at least one first beam intensity, the at least one first beam position corresponding to the at least one first beam intensity, generating, by the at least one processor, a second radiation pattern based on the at least one first beam position, the at least one first beam intensity, and the first radiation pattern; providing, by the at least one processor, the first planned target volume and the second radiation pattern to a second model such that the second model outputs data associated with at least one second beam position and at least one second beam intensity; and transmitting, by the at least one processor, data associated with the at least one second beam position and the at least one second beam intensity such that a linear accelerator (LINAC) delivers radiation to the patient.

9. The method according to claim 8, wherein the first model is trained to output data associated with first beam positions of a first scale, wherein the second model is trained to output data associated with second beam positions of a second scale.

10. The method according to claim 9, wherein the first scale is associated with a first set of candidate angles within a candidate angle space, wherein the second scale is associated with a second set of candidate angles within the candidate angle space, and wherein the angular measurement associated with the first set of candidate angles is greater than the angular measurement associated with the second set of candidate angles.

11. The method according to claim 10, wherein the beams associated with the second set of candidate angles correspond to the beams associated with the first set of candidate angles.

12. The method according to claim 8, wherein the first model is trained based on a reinforcement learning agent and training data associated with previously treated patients.

13. The method according to claim 12, wherein the training data represents a planned target volume and a target radiation pattern corresponding to the previously treated patient; The method further comprises: for each previously treated patient among the previously treated patients: generating, based on a plurality of reinforcement learning agents, data associated with beam positions and beam intensities, each reinforcement learning agent being associated with a training model, providing, to a simulator, the data associated with beam positions and beam intensities to generate a set of simulated outputs, determining, based on the corresponding simulated outputs and the target radiation pattern, a reward for each of the plurality of reinforcement learning agents, and Select the first model based on the reward for each reinforcement learning agent.

14. The method of claim 12, wherein the target radiation map represents a radiation map previously generated by the at least one processor based on an input from a clinician, the target radiation map representing the clinician's preference when delivering radiation to the previous treatment patient.

15. A non-transitory machine-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: Receive data associated with a first planned target volume and a first radiation map, the first planned target volume representing a region of a patient's body and a target mask, the target mask representing a portion of the region of the patient's body to receive radiation; Provide the first planned target volume and the first radiation map to a first model to cause the first model to output data associated with at least one first beam position and at least one first beam intensity, the at least one first beam position corresponding to the at least one first beam intensity, Generate a second radiation map based on the at least one first beam position, the at least one first beam intensity, and the first radiation map; Provide the first planned target volume and the second radiation map to a second model to cause the second model to output data associated with at least one second beam position and at least one second beam intensity; And Transmit data associated with the at least one second beam position and the at least one second beam intensity to cause a linear accelerator (LINAC) to deliver radiation to the patient.

16. The non-transitory machine-readable medium of claim 15, wherein the first model is trained to output data associated with first beam positions at a first scale, wherein the second model is trained to output data associated with second beam positions at a second scale.

17. The non-transitory machine-readable medium of claim 16, wherein the first scale is associated with a first set of candidate angles within a candidate angle space, wherein the second scale is associated with a second set of candidate angles within the candidate angle space, and wherein the angular measurement associated with the first set of candidate angles is greater than the angular measurement associated with the second set of candidate angles.

18. The non-transitory machine-readable medium of claim 17, wherein the beams associated with the second set of candidate angles correspond to the beams associated with the first set of candidate angles.

19. The non-transitory machine-readable medium of claim 15, wherein the first model is trained based on a reinforcement learning agent and training data associated with a previous treatment patient.

20. The non-transitory machine-readable medium of claim 19, wherein the training data represents a planned target volume and a target radiation map corresponding to the previous treatment patient; and wherein the computer-executable instructions further cause the one or more processors to: For each previous treatment patient among the previous treatment patients: Generating data associated with beam positions and beam intensities based on multiple reinforcement learning agents, each reinforcement learning agent being associated with a training model, Providing the simulator with the data associated with beam positions and beam intensities to generate a set of simulated outputs, Determining a reward for each of the multiple reinforcement learning agents based on the corresponding simulated outputs and the target radiation pattern, and Selecting the first model based on the rewards for each reinforcement learning agent.

Images