Using reinforcement learning in radiotherapy plan optimization to position dose-volume objectives

Abstract

Reinforcement learning agents help optimize radiation delivery treatment plans. A reinforcement learning agent is configured to generate a radiation delivery treatment plan that can exceed the quality of one or more plans used to train the reinforcement learning agent. The reinforcement learning agent is trained to evaluate 805 a radiation delivery treatment plan output by an optimization software application, modify 807 one or more dose-volume objective parameters of the evaluated radiation delivery treatment plan, and then input the modified radiation delivery treatment plan to the optimization software application for further optimization. The reinforcement learning agent adaptively adjusts the one or more dose-volume objective parameters based on an action policy learned during a reinforcement learning training process.

Description

Background Technology

[0001] Unless otherwise stated herein, the methods described in this section are not prior art to the claims of this application and are not acknowledged as prior art by virtue of their inclusion in this section.

[0002] Radiation therapy is a localized treatment used on a specific target tissue (planned target volume), such as a cancerous tumor. Ideally, radiation therapy is performed on the planned target volume, which protects surrounding normal tissue from receiving doses exceeding specified tolerances, thereby minimizing the risk of damage to healthy tissue. Before administering radiation therapy, an imaging system is typically used to provide three-dimensional images of the target tissue and surrounding area. Based on this imaging, the size and mass of the target tissue can be estimated, the planned target volume can be determined, and an appropriate treatment plan can be generated. When implemented, such a treatment plan results in meeting certain predetermined treatment goals, such as the planned target volume receiving a radiation dose exceeding a minimum required threshold, surrounding tissues receiving a radiation dose not exceeding a maximum permissible threshold, and the homogeneity of the radiation dose relative to the planned target volume that meets the minimum threshold.

[0003] Treatment planning is a complex process due to the geometrical complexity of applying radiation therapy to three-dimensional targets that can be very close to one or more organs at risk (OARs), involving the involvement of multiple highly trained medical professionals and analysis using sophisticated software. As a result, generating a treatment plan that produces the correct dosing target volume without overdosing nearby OARs is a multi-step process, where each step can be time-consuming and often can only be performed by expert personnel. In intensity-modulated radiotherapy (IMRT), where photon and / or proton beams conform to the shape of the tumor, one such planning step is IMRT optimization.

[0004] In IMRT optimization, one or more treatment plans are optimized for a specific patient. That is, one or more beam geometries are determined for implementing the planned treatment, and the dose distribution for each beam geometry is optimized. In some cases, a qualified radiation dosimeter using a software application (sometimes called an "optimizer") determines one or more beam geometries and optimizes the associated dose distribution, and then the physician responsible for the patient typically approves one or more treatment plans. In some cases, the software application can generate one or more beam geometries and associated optimized dose distributions during IMRT optimization, and the radiation dosimeter can select, modify, and / or review the software application's output.

[0005] One drawback of conventional IMRT optimization is that scoring the trade-off between target coverage and OAR sparing largely depends on the location of the dose-volume histogram target. For a given patient, such a target location is unknown prior to planning. Typically, IMRT planners or radiation dosimeters begin with some initial guesses and iteratively adjust the target location for the planned target volume through several rounds of optimization until the treatment plan becomes clinically acceptable. For example, in such a plan, the minimum dose is applied to the planned target volume without exceeding the maximum dose tolerated by the patient for that specific OAR. Therefore, IMRT planners oversee the optimization of various IMRT treatment plans until one or more plans are generated that meet the clinical goals established for the specific patient. This means that not only is the time spent generating treatment plans considerable, but the quality of the plans can also vary among IMRT planners, depending on their experience level, the availability of clinical feedback, and the time available for optimization of a particular plan.

[0006] Another drawback of conventional IMRT optimization is that the quality of a particular optimized treatment plan is heavily dependent on multiple conflicting dose-volume targets associated with a specific plan target volume. For example, there is often a trade-off between how high the dose applied to the plan target volume and how low the synthetic dose received by the OAR is. In another example, relaxation of dose homogeneity within the plan target volume can result in a lower dose in the OAR. Therefore, thresholds for such dose-volume targets are often modified by the IMRT planner to produce higher quality treatment plans. However, balancing such competing dose-volume targets is generally considered an intuitive task; therefore, IMRT optimization does not involve a rigorous exploration of many possible treatment plans, which can be optimized based on many possible permutations of dose-volume targets that can be considered for a particular plan target volume. Summary of the Invention

[0007] In one aspect, the invention provides a computer-implemented method for generating a treatment plan for delivering radiotherapy to a target volume, as defined in claim 1. Optional features are specified in its dependent claims.

[0008] In another aspect, the present invention provides a non-transient computer-readable medium as defined in claim 13. Optional features are specified in its dependent claims.

[0009] According to at least some embodiments of this disclosure, reinforcement learning (RL) agents facilitate the optimization of radiodelivery treatment plans. Specifically, the RL agent is configured to generate radiodelivery treatment plans that can exceed the quality of one or more plans used to train the RL agent. The RL agent is trained to evaluate radiodelivery treatment plans output by an optimization software application, modify one or more dose-volume target parameters of the evaluated radiodelivery treatment plan, and then input the modified radiodelivery treatment plan into the optimization software application for further optimization. The RL agent adaptively adjusts one or more dose-volume target parameters based on action strategies learned during the reinforcement learning training process. During training, the RL agent develops action strategies for adjusting one or more dose-volume target parameters to maximize or otherwise increase rewards. Rewards can be defined according to multiple dose-volume targets and / or other treatment objectives, such as target coverage, avoidance of risky organs, uniformity of dose applied to the planned target volume, and consistency (consistency index) between the area receiving the prescribed dose and the planned target volume.

[0010] The foregoing description of the invention is merely illustrative and is not intended to be limiting in any way. Other aspects, embodiments, and features, besides the illustrative aspects, embodiments, and features described above, will become apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0011] The foregoing and other features of this disclosure will become more fully apparent from the accompanying drawings, the following description, and the appended claims. These drawings depict only a few embodiments according to this disclosure and are therefore not intended to limit its scope. The disclosure will be described in more specific and detailed manner by means of the drawings.

[0012] Figure 1 This is a perspective view of a radiotherapy system that can advantageously realize various aspects of the present disclosure.

[0013] Figure 2 Various embodiments according to this disclosure are illustrated schematically. Figure 1 The drive stand and gantry of the RT system.

[0014] Figure 3 The diagram schematically illustrates a digital volume constructed based on a projection image showing the tumor and the surrounding anatomical region.

[0015] Figure 4 The illustrations schematically depict gross tumor volume, clinical target volume, planned target volume, two organs at risk, and examples of key structures.

[0016] Figure 5 This is a block diagram of a treatment planning optimization system configured to generate optimized radiotherapy plans according to various embodiments of the present disclosure.

[0017] Figure 6 Dose-volume histograms (DVH) according to various embodiments of the present disclosure are shown.

[0018] Figure 7 The DVH and certain variable dose-volume target parameters are shown according to various embodiments of this disclosure.

[0019] Figure 8 A flowchart illustrating an example optimization process for a treatment plan according to one or more embodiments of the present disclosure is shown.

[0020] Figure 9 This is an illustration of a computing device configured to perform various embodiments of the present disclosure.

[0021] Figure 10 This is a block diagram illustrating one or more embodiments of a computer program product for implementing a method for segmenting images according to the present disclosure. Detailed Implementation

[0022] In the following detailed description, reference is made to the accompanying drawings, which form part of the specification. In the drawings, similar symbols generally identify similar components unless the context otherwise requires. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that aspects of the invention as generally described herein and illustrated in the figures can be arranged, substituted, combined, and designed in a variety of different configurations, all of which are expressly contemplated and form part of this disclosure.

[0023] System Overview

[0024] Figure 1This is a perspective view of a radiotherapy system 100 that can advantageously implement various aspects of the present disclosure. The radiotherapy (RT) system 100 is a radiation system configured to detect intra-fraction motion in near real-time using X-ray imaging techniques. Therefore, the RT system 100 is configured to provide stereotactic radiosurgery and precise radiotherapy to indicate damage, tumors, and conditions anywhere in the body that require radiotherapy. For example, in some embodiments, the RT system 100 is configured to perform intensity-modulated radiotherapy (IMRT). Thus, the RT system 100 may include one or more of a linear accelerator (LINAC) that generates a megavolt (MV) treatment beam of high-energy X-rays, a kilovolt (kV) X-ray source, an X-ray imager, and, in some embodiments, an electronic entry imaging device (EPID). As an example, the radiotherapy system 100 described herein is configured with a circular gantry. In other embodiments, the radiotherapy system 100 may be configured with a C-shaped gantry capable of infinite rotation via slip rings.

[0025] Typically, the RT system 100 is capable of kV imaging of the target volume during the application of an MV treatment beam, enabling the use of X-ray imaging to perform an image-guided radiotherapy (IGRT) procedure. Alternatively, in some embodiments, the RT system 100 is configured to perform an IMRT procedure without IGRT. The RT system 100 may include one or more touchscreens 101, a table motion control 102, an aperture 103, a pedestal positioning assembly 105, a table 107 disposed on the pedestal positioning assembly 105, and an image acquisition and treatment control computer 106, all disposed within a treatment room. The RT system 100 also includes a remote console 110 disposed outside the treatment room and capable of treatment delivery and patient monitoring from a remote location. The pedestal positioning assembly 105 is configured to precisely position the table 107 relative to the aperture 103, and the motion control 102 includes input devices, such as buttons and / or switches, enabling a user to operate the pedestal positioning assembly 105 to automatically and precisely position the table 107 relative to the aperture 103 to a predetermined position. Motion control 102 also allows the user to manually position the examination table 107 to a predetermined location. In some embodiments, the RT system 100 also includes one or more cameras (not shown) for patient monitoring in the treatment room.

[0026] Figure 2 The drive bracket 200 and rack 210 of the RT system 100 according to various embodiments of the present disclosure are schematically shown. For clarity, in Figure 2The cover, base positioning assembly 105, examination table 107, and other components of the RT system 100 are omitted. The drive support 200 is a fixed support structure for the components of the RT treatment system 110, including a gantry 210 and a drive system 201 for rotatably moving the gantry 210. The drive support 200 rests on and / or is fixed to a support surface outside the RT treatment system 110, such as the floor of the RT treatment facility. The gantry 210 is rotatably coupled to the drive support 200 and is the support structure on which various components of the RT system 100 are mounted, including a linear accelerator (LINAC) 204, an electronic entry imaging device (EPID) 205, an imaging X-ray source 206, and an X-ray imager 207. During operation of the RT treatment system 110, the gantry 210 rotates about the aperture 103 when actuated by the drive system 201.

[0027] The drive system 201 rotatably actuates the gantry 210. In some embodiments, the drive system 201 includes a linear motor that is fixed to the drive bracket 200 and interacts with a magnetic track (not shown) mounted on the gantry 210. In other embodiments, the drive system 201 includes another suitable drive mechanism for precisely rotating the gantry 210 about the aperture 201. The LINAC 204 generates a high-energy X-ray (or electrons in some embodiments) MV treatment beam 230, and the EPD 205 is configured to acquire X-ray images using the treatment beam 230. The imaging X-ray source 206 is configured to guide a conical beam of X-rays (here referred to as imaging X-ray 231) to the X-ray imager 207 via an isocenter 203 of the RT system 100, the isocenter 203 typically corresponding to the location of the target volume 209 to be treated. Figure 2 In the illustrated embodiment, the X-ray imager 207 is depicted as a planar device, while in other embodiments, the X-ray imager 207 may have a curved configuration.

[0028] X-ray imager 207 receives imaging X-rays 231 and generates appropriate projection images therefrom. According to some embodiments, such projection images can then be used to construct or update portions of imaging data corresponding to a digital volume comprising a three-dimensional (3D) region including the target volume 209. That is, a 3D image of such a 3D region is reconstructed from the projection images. In embodiments, cone-beam computed tomography (CBCT) and / or digital tomography synthesis (DTS) can be used to process the projection images generated by X-ray imager 207. CBCT is typically used to acquire projection images over a relatively long acquisition arc, such as at a rotation of 180° or greater over gantry 210. As a result, a high-quality 3D reconstruction of the imaging volume can be produced. CBCT is typically employed at the start of a radiotherapy session to generate the established 3D reconstruction. For example, CBCT can be used immediately before the application of the treatment beam 230 to generate a 3D reconstruction confirming that the target volume 209 has not moved or changed shape. Alternatively or additionally, in some embodiments, partial data reconstruction may be performed by the RT system 100 during a portion of the IGRT procedure, during which a portion of the image data is used to generate a 3D reconstruction of the target volume 209. For example, DTS image acquisition may be performed to generate image data for the target volume 209 as the treatment beam 230 is directed to the isogonal point 203 while the gantry 210 rotates through the treatment arc. Alternatively, CBCT may be employed during a portion of the IGRT procedure to generate a 3D reconstruction of the target volume 209 during treatment.

[0029] exist Figure 2 In the illustrated embodiment, the RT system 100 includes a single X-ray imager and a single corresponding imaging X-ray source. In other embodiments, the RT system 100 may include two or more X-ray imagers, each having a corresponding imaging X-ray source. Thus, in such an embodiment, the RT system 100 includes a first imaging X-ray source and a corresponding X-ray imager mounted on rack 210, and a second imaging X-ray source and a corresponding X-ray imager mounted on rack 210.

[0030] The projected images generated by the X-ray imager 207 are used to construct digital volumetric imaging data of the patient's anatomy within a 3D region including the target volume. Alternatively or additionally, such projected images can be used to update portions of existing imaging data corresponding to the digital volume of the 3D region. (See below for further details.) Figure 3 An example of such a digital volume is described.

[0031] Figure 3A digital volume 300 constructed based on a projection image showing the tumor and the surrounding anatomical region is schematically illustrated. For example, the digital volume 300 can be constructed based on one or more treatment planning computed tomography (CT) scans performed by an X-ray imager. The X-ray imager can be an imaging device separate from the RT system 100, or it can be an X-ray imager included within the RT system 100.

[0032] The digital volume 300 comprises multiple voxels 301 (dashed lines) of anatomical image data, where each voxel 301 corresponds to a different location within the digital volume 300. For clarity, in... Figure 3 Only a single voxel 301 is shown. The digital volume 300 corresponds to a 3D region including the target volume 310. Figure 3 In the diagram, digital volume 300 is depicted as an 8×8×8 voxel cube, but in practice, digital volume 300 typically includes more voxels, for example, more than 8×8×8 voxels. Figure 3 The numbers represent orders of magnitude greater than those shown.

[0033] For the purposes of discussion, target volume 310 may refer to the gross tumor volume (GTV), clinical target volume (CTV), or planned target volume (PTV) for a specific treatment. GTV depicts, for example, the location and extent of a gross tumor that can be seen or imaged; CTV includes the GTV and additional margins of subclinical disease spread that are typically not imageable; PTV is a geometric concept designed to ensure that an appropriate dose of radiation therapy is actually delivered to the CTV without adversely affecting nearby organs of risk (OARs). Therefore, PTV is generally larger than CTV, but in some cases it may be reduced in certain sections to provide a safety margin around the OAR. PTV is typically determined based on imaging performed prior to treatment.

[0034] Figure 4 Examples of GTV 410, CTV 420, PTV 430, two OARs 450, and key structure 460 are illustrated schematically. It should be noted that... Figure 4 The structures included are described in two dimensions, but are typically three-dimensional in nature. Thus, each of GTV 410, CTV420, PTV 430, OAR450, and key structures 460 is generally defined via a three-dimensional segmentation process, in which a specific voxel is included in a set of treatment planning CT scans. Figure 4 Regions (not shown in the diagram) are designated as specific structures. GTV410, CTV420, and PTV430 are regions or structures in which ideal treatment planning results in receiving at least the minimum threshold dose, while OAR450 and critical structure 460 are regions or structures in which ideal treatment planning results in receiving the smallest possible dose of normal tissue.

[0035] Typically, treatment planning is based on CT scans that include the patient's GTV, CTV, and / or PTV (e.g., Figure 3 The treatment planning instructions (GTV) are generated using a digital volume (GTV) of 300 to specify the treatment plan for a particular patient. These instructions typically describe an imaging study of the treatment site, including target and normal tissue structures defined through the imaging study. These target and normal tissue structures are then used in the treatment plan. To score multiple treatment plans during optimization, the GTV may also specify extensions to the target and normal tissue structures. Therefore, in addition to the GTV, the GTV may include the CTV, Internal Target Volume (ITV), PTV, OAR, critical structures, and / or planned OAR volume (PRV). The GTV may further specify radiotherapy prescribing guidelines, planning recommendations, and / or special instructions.

[0036] Treatment plan optimization using reinforcement learning agents

[0037] According to the various embodiments described below, an artificial intelligence (AI) agent is used to generate one or more radiotherapy plans based on information included in the planning instructions. The AI ​​agent is trained using reinforcement learning (RL) to determine certain actions to be performed to facilitate the generation of higher-quality radiotherapy plans by an optimization software application. More specifically, in embodiments, actions performed by the AI ​​agent include modifications to one or more dose-volume targets or dose-volume target parameters for candidate radiotherapy plans. For example, actions included in the AI ​​agent's action space may include: repositioning the dose-volume target curve for a specific dose-volume target, modifying the minimum dose distribution within the target volume, modifying the maximum dose distribution within an OAR close to the target volume, modifying factors (e.g., weighting factors) included in a cost function that quantifies the penalty for radiotherapy delivery plans that fail to achieve dose-volume targets, and so on.

[0038] The AI ​​agent selects actions based on an action policy learned by the AI ​​agent to maximize the reward function and improve radiotherapy delivery plans. As a result, optimization of candidate radiotherapy plans after the AI ​​agent executes the actions enables the optimization software application to generate higher-quality radiotherapy plans. In some embodiments, training the AI ​​agent includes learning an action policy that instructs actions to maximize or otherwise increase the value of the reward function associated with the radiotherapy plan. The reward function is based on one (or typically multiple) dose-volume objectives, such as target coverage, OAR avoidance, uniformity of dose applied to the planned target volume, and consistency between the area receiving the prescribed dose and the planned target volume. Therefore, the reward function can be a broader concept than the cost function used in conventional optimizer applications.

[0039] Figure 5 This is a block diagram of a treatment planning optimization system 500 configured to generate an optimized radiotherapy plan 540 according to various embodiments of the present disclosure. The treatment planning optimization system 500 includes a reinforcement learning (RL) agent 520 and an optimizer application 530. After appropriate training via a reinforcement learning process, the RL agent 520 is configured to interact with the optimizer application 530 to generate an optimized radiotherapy delivery plan 540. In some embodiments, the optimized radiotherapy delivery plan 540 generated by the treatment planning optimization system 500 may exceed the quality of one or more plans used to train the RL agent 520.

[0040] The optimizer application 530 may be or may include conventional treatment plan optimization software applications or algorithms, such as RapidPlan, which is available from Varian Medical Systems in Palo Alto, California. TM Thus, the optimizer application 530 is configured to enable radiation dosimeters or other treatment planners to generate new radiation delivery treatment plans for patients that achieve the objectives included in the patient's treatment planning instructions. These objectives may include one or more dose-volume objectives, such as tumor coverage, OAR, and normal tissue avoidance objectives. Typically, the optimizer application 530 employs an iterative process to determine one or more treatment beam geometries, which optimize and / or otherwise select a dose distribution that satisfies the objectives included in the patient's treatment planning instructions.

[0041] The RL agent 520 is configured to provide input 503 to the optimizer application 530, enabling the optimizer application 530 to generate one or more treatment plans for the patient that are of higher quality than those generated by a human planner interacting with a conventional optimizer application. More specifically, the one or more treatment plans generated by the optimizer application 530 can achieve more than just certain objectives included in the patient's treatment plan instructions. For example, in one embodiment, the treatment plan generated by the optimizer application 530 based on input 503 from the RL agent 520 can exceed one or more minimum threshold treatment objectives (e.g., minimum required dose in the target volume) and / or be below one or more maximum permissible target thresholds (e.g., maximum permissible dose in the OAR). Therefore, such a treatment plan has higher quality than a treatment plan that only meets some objectives.

[0042] In some embodiments, the RL agent 520 receives patient-specific plan information 501, generates an initial treatment plan for the patient based on this information, and provides the initial treatment plan as input 503 to the optimizer application 530. Alternatively or additionally, in some embodiments, the RL agent 520 receives an existing treatment plan 502 for the patient, such as a treatment plan generated at an earlier stage of the patient's treatment. In such embodiments, the RL agent 520 generates a modified treatment plan by modifying the existing treatment plan 502 according to embodiments described below, and then provides the modified treatment plan as input 503 to the optimizer software application 530. For example, detecting changes in one or more patient volumes of interest may indicate that modifications to existing treatment plans by the treatment plan optimization system 500 may be beneficial, such as when determining changes in the size and / or location of the GTV relative to a particular structure (e.g., OAR).

[0043] In some embodiments, patient-specific planning information 501 may include information enabling the construction of a treatment plan, such as delineation targets for planned radiotherapy based on a treatment planning CT and information included in the treatment planning instructions. For example, one or more of GTV, CTV, ITV, and / or PTV may be delineated on two-dimensional slices of a reconstructed volume imaged from a treatment planning CT. Additionally, one or more OARs may be delineated similarly. In some cases, radiation oncologists perform one or more of these delineations manually via a display screen. Typically, radiation oncologists delineate various target volumes based on visual cues in each two-dimensional slice of the reconstructed volume, as well as based on personal training and experience. Alternatively, in some cases, radiation oncologists may assist in segmenting one or more anatomical structures (GTV, CTV, ITV, PTV, OAR, etc.) using software applications configured to automatically generate segments of GTV, PTV, etc.

[0044] In some embodiments, patient-specific planning information 501 may also include one or more treatment goals, dose-volume goals, and / or dose-volume goal parameters for segmented target regions and / or OARs, such as information included in treatment planning instructions for a particular patient. Thus, in some embodiments, patient-specific planning information 501 may include upper dose-volume goals limiting the dose in a particular structure (e.g., "no more than 20% of the particular structure can receive more than 25 Gy"), lower dose-volume goals defining the desired dose level in a particular target structure (e.g., "at least 70% of the structure must receive at least 20 Gy"), dose homogeneity within the GTV, CTV, or PTV, consistency index values ​​indicating the consistency of the specific region receiving the prescribed dose with the PTV, etc.

[0045] In operation, RL agent 520 generates input 503, which may include a treatment plan to be optimized by optimizer application 530. Additionally or alternatively, input 503 includes one or more dose-volume targets, enabling optimizer application 530 to modify an existing treatment plan to be optimized. Optimizer application 530 then performs an optimization process on the treatment plan provided by RL agent 520. Alternatively, optimizer application 530 modifies an existing treatment plan based on input provided by RL agent 520. For example, in embodiments where input 503 includes one or more dose-volume targets, optimizer application 530 modifies an existing treatment plan and performs an optimization process on the modified treatment plan. That is, candidate treatment plans are generated by optimizer application 530 based on new dose-volume targets provided by RL agent 520.

[0046] During optimization, the optimizer application typically begins with an initial set of parameter settings, iteratively adjusting one or more of these settings, and quantitatively evaluating the relative value of the adjusted plan, for example, by weighting a cost function with respect to the relative value of each treatment objective of the treatment plan being optimized. In the iterations, aspects of the optimized treatment plan vary, such as the collimator blade settings for each of the multiple X-ray source angles, the X-ray source class for each of the multiple X-ray source angles, etc. This iterative approach allows for the exploration, at a certain level, of various solutions that can differently satisfy each treatment objective of the treatment plan. Note that the “optimized” plan result from this optimization process is not assumed to be the best possible (or optimal) plan superior to any other alternative. Rather, the term “optimized” plan, as used herein, refers to a treatment plan that is an improvement over the initial plan prior to the optimization process.

[0047] Upon completion of the optimization process, the optimizer application 530 generates a candidate treatment plan 504 and / or dose-volume information 505 associated with the candidate treatment plan 504. The candidate treatment plan 504 is a radiation delivery treatment plan for the patient, which is typically a clinically acceptable treatment plan but can be improved through one or more modifications by the RL agent 520. In some embodiments, the dose-volume information 505 includes one or more dose-volume histograms or data that can be formulated as one or more dose-volume histograms. The following is in conjunction with... Figure 6 The description may include an example of a dose-volume histogram in dose-volume information 505.

[0048] Figure 6A dose-volume histogram (DVH) 600 according to various embodiments of the present disclosure is illustrated. The DVH 600 correlates the predicted radiation dose for a particular treatment plan with a specific tissue volume (e.g., PTV, GTV, OAR, or other structure) in a two-dimensional format. For example, the DVH 600 can indicate the dose to be administered in a manner such as... Figure 5 The DVH 600 provides the predicted radiation dose to one or more structures following a specific treatment plan (candidate treatment plan 504). Therefore, the DVH 600 enables the comparison of predicted radiation doses to one or more structures used for multiple treatment plans.

[0049] DVH includes dose information from multiple dose bins and can be implemented as differential DVH or cumulative DVH. Figure 6 In the illustrated embodiment, DVH 600 is a cumulative DVH and includes values ​​for multiple dose bins from 0 to 70 Gy plotted along a horizontal axis. In DVH 600, the column height of each bin (e.g., 0 Gy, 1 Gy, 2 Gy, etc.) indicates the volume of a specific structure receiving a dose greater than or equal to that value. For example, the column height of the second bin (1 Gy) represents the volume of a specific structure receiving a dose greater than or equal to that value. Therefore, for a large number of small bin sizes, each dose-volume target included in DVH 600 essentially represents a smooth curve.

[0050] exist Figure 6In the illustrated embodiment, the DVH 600 includes dose-volume targets for several different structures: a dose-volume target 610 for a first OAR, a dose-volume target 620 for a second OAR, and a dose-volume target 630 for a target region such as a PTV. In other embodiments, the DVH 600 may include more or fewer dose-volume targets. Dose-volume targets 610 and 620 are each associated with an OAR and, accordingly, indicate the maximum permissible dose distribution within the associated OAR. That is, dose-volume target 610 indicates the maximum permissible dose distribution within an organ of risk or other structures near the target region, while dose-volume target 620 indicates the maximum permissible dose distribution within another organ of risk or other structures near the target volume. In contrast, dose-volume target 630 is associated with a target region and therefore indicates a target minimum dose threshold for radiotherapy of a PTV. Based on the above, the ideal treatment plan satisfies the dose-volume target 630 for all dose bins, while the OAR or structure associated with dose-volume targets 610 and 620 receives zero dose. In practice, the realization of a candidate treatment plan inevitably results in some non-zero dose being received by the OAR or structure associated with dose-volume targets 610 and 620. Furthermore, some or all dose bins of dose-volume target 630 may not be satisfied. That is, there is a trade-off between satisfying dose-volume target 630 and minimizing or otherwise reducing the dose received by the OAR and other critical structures. This trade-off is illustrated by dose curves 611, 621, and 631 described below.

[0051] For each of the dose-volume targets 610, 620, and 630, the DVH 600 also includes a dose curve (dashed line) that indicates when a specific treatment plan is implemented (e.g., Figure 5 The DVH 600 also includes dose curves 611 for regions or structures associated with dose-volume targets 610, 621 for regions or structures associated with dose-volume targets 620, and 631 for regions or structures associated with dose-volume targets 630.

[0052] As shown in the figure, Figure 6In the example shown, dose curve 631 is not ideal because dose-volume target 630 is not met for all dose chambers. Instead, there are deviations 632 and 633 (indicated by the cross-shaded areas) between dose curve 631 and dose-volume target 630, where dose curve 631 differs from dose-volume target 630. Deviation 632 corresponds to where dose curve 631 is below dose-volume target 630, while deviation 633 corresponds to where dose curve 631 exceeds dose-volume target 630. Furthermore, dose curve 611 is not ideal either. Specifically, even though dose curve 611 indicates that the candidate treatment plan associated with DVH600 is less than the maximum permissible dose distribution in the associated OAR across all dose chambers, at least some portions of the associated OAR receive at least some doses. Similarly, dose curve 621 is not ideal because at least some portions of the OAR associated with dose-volume target 620 receive at least some doses. Therefore, even if a candidate treatment plan associated with DVH 600 (such as candidate treatment plan 504) is clinically acceptable, the candidate treatment plan can potentially be further improved by the interaction of the RL agent 520 with the optimizer application 530 as described below.

[0053] In embodiments, the RL agent alters the cost function associated with the current candidate treatment plan by changing the dose corresponding to one or more dose-volume targets. This modification of the dose-volume target changes the cost function because each unique set of target parameters defines a different cost function. For example, in some embodiments, the dose-volume target can be shifted right or left in the DVH 600 by increasing or decreasing the appropriate dose parameter of the dose-volume target, respectively. Similarly, in some embodiments, the dose-volume target can be shifted up or down in the DVH 600 by increasing or decreasing the volume percentage of the dose-volume target.

[0054] Back Figure 5The RL agent 520 evaluates candidate treatment plan 504 and / or dose-volume information 505 to determine whether candidate treatment plan 504 can be further improved. Therefore, in some embodiments, the RL agent 520 evaluates candidate treatment plan 504 based on dose-volume information 505. In some embodiments, the RL agent 520 determines the current state of candidate treatment plan 504 provided by optimizer application 530. In such embodiments, dose-volume information 505 is used to determine the current state of candidate treatment plan 504. For example, in such embodiments, the state of candidate treatment plan 504 may be based on a dictionary containing OARs and target structures (e.g., GTV, CTV, PTV, etc.) as keys, where the value of each key constitutes the current state of candidate treatment plan 504. In some embodiments, the key values ​​are generated from corresponding dose-volume information (such as DVH curves) for each OAR and target structure. In such embodiments, the current state of candidate treatment plan 504 may be based on values ​​for each OAR and target structure, where each value is derived from the deviation of the dose curve from the corresponding dose-volume target. For example, in one embodiment, one value contributing to the current state of candidate treatment plan 504 is the deviation 632 and 633 from the dose curve 631 and the dose-volume target 630 (in... Figure 6 (As shown in the figure)

[0055] Alternatively or additionally, in some embodiments, the RL agent 520 evaluates the candidate treatment plan 504 based on other quantitative measures indicating the quality of the candidate treatment plan 504. For example, in some embodiments, the RL agent 520 evaluates the candidate treatment plan 504 based on a target volume consistency index, similar to the calculation of a reward function used during the training process of the RL agent 520, and / or other quantitative measures.

[0056] After the RL agent 520 evaluates the candidate treatment plan 504 based on dose-volume information 505, the RL agent 520 uses an action policy 521 learned by the RL agent 520 during the training process to select one or more actions to modify the candidate treatment plan 504. In some embodiments, the RL agent 520 selects one or more actions based on the current state of the candidate treatment plan 504, which is based on values ​​for each OAR and target structure. In such embodiments, the RL agent 520 identifies the current state of the candidate treatment plan 504 and selects the action policy 521 to indicate one or more actions that result in high rewards during the training process. That is, for each state of the candidate treatment plan tested during the reinforcement learning process, the RL agent 520 stores the resulting reward for each action performed on the candidate treatment plan. Thus, the action policy 521 is generated during the reinforcement learning process of the RL agent 520.

[0057] During reinforcement learning, when a candidate treatment plan is in a specific state, the resulting reward is observed for various actions performed on the candidate treatment plan by the RL agent 520. The resulting reward for a specific action is determined by performing an action on the candidate treatment plan (i.e., modifying the candidate treatment plan), performing an optimization process on the modified candidate treatment plan using the optimizer application 530, and quantifying the change in the quality of the newly optimized treatment plan. In some embodiments, the quality of the newly optimized treatment plan is quantified by reflecting or otherwise based on a reward function of some or all treatment objectives associated with the candidate treatment plan. This generates entries in the action policy 521. That is, each action and its associated reward function value contribute to the action policy 521.

[0058] In some embodiments, the reward function may be a function of one or more factors, including: target coverage; OAR avoidance; dose homogeneity within the GTV, CTV, and / or PTV; a consistency index value indicating the consistency of a specific region receiving a prescribed dose with the PTV; and computational efficiency, etc. In some embodiments, an improvement in a candidate treatment plan relative to one or more of the above factors is rewarded more highly than an improvement in a candidate treatment plan relative to other of the above factors. For example, some or all of the factors on which the reward function is based may have weighting factors associated with them, wherein some or all of these weighting factors may have different values. Thus, in such embodiments, one or more of the above factors have a weighted contribution to the reward function. Alternatively or additionally, in some embodiments, achieving a threshold condition for a specific factor through a candidate treatment plan is rewarded more highly than exceeding a threshold condition for a specific factor. For example, in one embodiment, achieving dosimetry criteria for target coverage (e.g., minimum dose threshold) and OAR avoidance (e.g., maximum dose threshold) is highly rewarded compared to exceeding such criteria.

[0059] In some embodiments, the uniformity of PTV is achieved by adjusting the smaller D5-D. 95 The value is explained by rewards, where D X This represents the dose corresponding to X% of the volume. In some embodiments, the consistency index is defined as the volume of the isosurface of the prescribed dose divided by the volume of the corresponding PTV. In some embodiments, smaller values ​​of |consistency index - 1| yield more rewards. In some embodiments, the efficiency of the candidate treatment plan is measured as the number of times the RL agent 520 triggers the optimizer application 530 before generating a clinically acceptable plan.

[0060] In some embodiments, different action strategies 521 are generated for different clinical scenarios. For example, in some embodiments, a unique action strategy 521 is developed via a reinforcement learning process for each different part of the anatomical structure. Additionally or alternatively, in some embodiments, a unique action strategy 521 is developed for each of a plurality of different categories of patients via a reinforcement learning process. For example, in such embodiments, different action strategies 521 may be developed based on patient gender, patient age, etc. Additionally or alternatively, in some embodiments, a unique action strategy 521 is developed via a reinforcement learning process for a specific radiotherapy system. Additionally or alternatively, in some embodiments, a unique action strategy 521 is developed for any combination of two or more of the above-described clinical scenarios.

[0061] In some embodiments, the action space of the RL agent 520 includes generating a modified dose-volume target and causing the optimizer application 530 to generate another candidate treatment plan 504 that satisfies the modified dose-volume target. That is, the RL agent 520 modifies at least one dose-volume target parameter of the current candidate treatment plan 504 and causes the optimizer application 530 to perform an optimization process on the current modified version of the candidate treatment plan 504. Various dose-volume target parameters that can be changed by the RL agent 520 are described below.

[0062] Figure 7 A DVH 700 and certain variable dose-volume target parameters according to various embodiments of the present disclosure are illustrated. The DVH 700 is the cumulative DVH and includes a dose-volume target 710 for OAR (not shown). According to various embodiments, one or more dose-volume target parameters may be derived from... Figure 5The RL agent 520 is modified to generate a modified dose-volume target, which is then included in the treatment plan for further optimization. For example, in some embodiments, the dose-volume target parameters of the dose-volume target 710 that can be modified include dose values ​​of one or more dose chambers of the DVH 700. Therefore, in such embodiments, the RL agent 520 generates a modified dose-volume target for the OAR by modifying the dose values ​​of one or more dose chambers of the DVH 700. In one such embodiment, the RL agent 520 modifies the dose values ​​of a portion of the dose chambers of the dose-volume target 710 (e.g., dose chamber 711). In doing so, the RL agent 520 generates a modified dose-volume target for the OAR that includes a modified segment 712 (dashed line) different from a portion of the dose-volume target 710. In another such embodiment, the RL agent 520 modifies the dose values ​​of the dose-volume target 710 for all dose chambers, thereby generating a completely different dose-volume target 720 for the OAR. In yet another such embodiment, the RL agent 520 moves the dose value of each dose cell of the dose-volume target 710 by the same number of dose cells and moves them in the same direction. In another such embodiment, the RL agent 520 generates different dose-volume targets 730 for the OAR, which are uniformly moved toward doses higher or lower than the dose-volume target 710.

[0063] Back Figure 5 In some embodiments, the dose-volume target parameters modified by the RL agent 520 are parameters that modify the cost function used during the optimization process performed by the optimizer application 530. In some embodiments, the dose value of the dose-volume target (as described above) Figure 7 The dose-volume target parameter is such that... Alternatively or additionally, in some embodiments, the dose-volume target parameter modified by the RL agent 520 is a weighting factor, wherein the weighting factor modifies the contribution of the cost function to a dose-volume target relative to the contribution to the cost function of another dose-volume target. Therefore, in such embodiments, the RL agent 520 modifies the subsequent optimization process performed by the optimizer application 530 by changing the relative importance (or weight) of different dose-volume targets. For example, by increasing the importance of the first dose-volume target relative to a second dose-volume target that conflicts with the first dose-volume target, the optimizer application 530 can generate a higher quality treatment plan.

[0064] In some embodiments, when the RL agent 520 evaluates the candidate treatment plan 504 and / or dose-volume information 505 and determines that the candidate treatment plan 504 meets one or more termination criteria, the RL agent 520 generates or outputs an optimized treatment plan 540. Such termination criteria may include the maximum threshold number of interactive iterations occurring between the RL agent 520 and the optimizer application 530, the determination that the candidate treatment plan 504 is a clinically acceptable plan, or the determination that the candidate treatment plan 504 meets one or more additional quality criteria (such as exceeding one or more clinical goals).

[0065] Figure 8 A flowchart illustrating an example optimization process for a treatment plan according to one or more embodiments of the present disclosure is shown. The method may include one or more operations, functions, or actions as shown in one or more of blocks 801-810. Although these blocks are shown in a sequential order, they may be performed in parallel and / or in an order different from that described herein. Furthermore, based on the desired implementation, the individual blocks may be combined into fewer blocks, divided into additional blocks, and / or eliminated. Despite the combination Figures 1-7 The process and system described herein represent the method; however, those skilled in the art will understand that any appropriately configured system can perform the optimized process described herein. In some cases, the method is performed prior to any radiotherapy the patient receives. In other cases, the method is performed in response to the detection of changes in the size, shape, and / or relative positioning of the target volume and / or the OAR, in which case the patient's current treatment plan may be appropriately modified.

[0066] The optimization process 800 begins at step 801, at which point the RL agent 520 receives patient-specific plan information 101 and / or the patient’s existing treatment plan 502.

[0067] In step 802, if no existing treatment plan is available 502, the RL agent 520 generates an initial treatment plan target, such as a dose-volume target. In some embodiments, the treatment plan target and / or dose-volume target generated in step 802 are based on a default treatment plan and / or on treatment targets included in the patient-specific plan information 101.

[0068] Note that such initial treatment plan objectives define a specific cost function for the optimizer application 530. That is, the cost function used by the optimizer application 530 to optimize a particular treatment plan is a function of a specific value included in each treatment plan objective. For example, one such treatment plan objective could be an upper dose-volume objective used to limit the dose in a given structure (e.g., "no more than 20% of the structure can receive more than 25 Gy"). In another example, one such treatment plan objective could be a lower dose-volume objective used to define the target dose level in the target structure (e.g., "at least 70% of the target structure must receive at least 20 Gy").

[0069] In step 803, RL agent 520 causes optimizer application 530 to perform iterative optimization of the current treatment plan using the current dose-volume target. For example, in a first iteration of step 803, RL agent 520 typically causes optimizer application 530 to perform iterative optimization of the existing treatment plan 502, if available. Conversely, in subsequent iterations of step 803, RL agent 520 typically causes optimizer application 530 to perform iterative optimization of the modified candidate treatment plan 504.

[0070] In step 804, optimizer application 530 determines whether one or more optimization termination criteria have been met. If yes, method 800 proceeds to step 805; if no, method 800 returns to step 803, and optimizer application 530 performs another iteration of optimization on the current candidate treatment plan.

[0071] In step 805, the RL agent 520 receives candidate treatment plan 504 and / or dose-volume information 505 associated with candidate treatment plan 504. The RL agent 520 then evaluates candidate treatment plan 504. In some embodiments, the RL agent 520 evaluates candidate treatment plan 504 based on its current state. For example, in some embodiments, the RL agent 520 performs the evaluation of the candidate treatment plan for the target volume by comparing at least one dose-volume distribution generated by the candidate treatment plan with a corresponding dose-volume target included in or otherwise associated with the candidate treatment plan.

[0072] In step 806, RL ​​agent 520 determines whether candidate treatment plan 504 meets one or more termination conditions. If yes, method 800 proceeds to step 810 and terminates, generating an optimized radiotherapy plan 540; if no, method 800 proceeds to step 807.

[0073] In step 807, based on the current state of candidate treatment plan 504, RL agent 520 determines at least one action to modify the dose-volume target parameter of the candidate treatment plan's dose-volume target. For example, in some embodiments, the dose-volume target parameter modified by the action may be one of the following: the minimum dose distribution within the target volume (e.g., GTV, CTV, ITV, PTV, etc.), the maximum dose distribution within the OAR close to the target volume, a factor included in a cost function quantifying the penalty for a radiotherapy delivery plan that fails to achieve the dose-volume target, etc. In some embodiments, RL agent 520 determines multiple actions, each modifying a dose-volume target parameter.

[0074] In some embodiments, the RL agent 520 determines at least one action based on the current state of the candidate treatment plan 504 and the action strategy 521. In such embodiments, the RL agent 520 selects at least one action from its action space. Examples of actions included in such an action space in some embodiments include: changing a threshold associated with a specific dose-volume target of the candidate treatment plan, repositioning the dose-volume curve associated with the specific dose-volume target of the candidate treatment plan on the dose-volume histogram, and modifying the weighting factor associated with the specific dose-volume target of the candidate treatment plan.

[0075] In step 808, the RL agent 520 generates a modified dose-volume target by performing the actions determined in step 807. Therefore, in step 808, the RL agent 520 updates at least one dose-volume target of the candidate treatment plan with the modified dose-volume target parameters. Method 800 then returns to step 803, where the optimizer application 530 performs iterative optimization on the now modified treatment plan.

[0076] Training reinforcement learning agent

[0077] According to the various embodiments described herein, an RL agent learns action policies to adaptively adjust the dose-volume targets of candidate treatment plans to maximize or otherwise increase the cumulative reward or other output from a reward function. As is known in the art, during RL agent training, the RL agent takes actions in an environment and observes the consequences of the actions, quantifying them as rewards associated with a specific state of the environment. The rewards and associated states are then fed back to the RL agent, which repeats this process with one or more different actions. According to the various embodiments, in order to facilitate... Figure 5 The RL agent 520 is trained using a reward function based on clinical goals of radiotherapy delivery planning, as described below.

[0078] In some embodiments, the action space of the RL agent includes, for example, moving a specific dose-volume target to the right or left. Additionally, in some embodiments, the action space also includes running or stopping the optimizer application 530. In an example embodiment, the dose-volume target can be moved between 0 and 100 Gy in steps of 0.1 Gy. In this embodiment, the agent is in state s, takes action a, and observes the next state s'. The immediate reward is then described by Equation 1: r = v(s') - v(s), where v(·) is the state-value function. An embodiment of the state-value function for a specific architecture such as OAR or PTV is described below.

[0079] For this structure, v(d) j ) represents the dose d in a specific dose-volume target curve. j Volume % at ∈[0, 1000](dGy) This represents the dose d corresponding to state s. j Volume percentage. Based on the upper clinical target for the structure (d cg v cg The mass measurement of ) is described by Equation 2:

[0080]

[0081] Based on the lower clinical goals of the structure (d) ch v cg The mass measurement of ) is described by Equation 3:

[0082]

[0083] When the structure is OAR, the mass measurement of DVH based on OAR is described by Equation 4:

[0084]

[0085] Where D represents the number of points in the dose axis after discretization. For example, if the dose axis is in the range of 0 to 100 Gy and the distance from the set point is 0.1, then D = 1000.

[0086] D pr The mass measurement of a target structure with a specified dose is described by Equation 5:

[0087]

[0088] Based on the above, the state-value function can be defined by Equation 6:

[0089]

[0090] Where n oar n ptv and ncg These represent the number, objectives, and clinical objectives of OARs, respectively, with weights w. oar w ptv and w cg As specified by the planner. In an alternative embodiment, the state value function may include a term based on a spatial assessment of the dose-volume target in the treatment plan and a term based on the dose distribution of the current solution.

[0091] Figure 9 This is an illustration of a computing device 900 configured to perform various embodiments of the present disclosure. The computing device 900 may be a desktop computer, laptop computer, smartphone, or any other type of computing device suitable for practicing one or more embodiments of the present disclosure. In operation, the computing device 900 is configured to perform RL agent 520, optimizer application 530, and / or reinforcement learning training process 911, as described herein. Note that the computing device described herein is illustrative, and any other technically feasible configuration falls within the scope of this disclosure.

[0092] As shown in the figure, the computing device 900 includes, but is not limited to, an interconnect (bus) 940 connecting the processing unit 950, an input / output (I / O) device interface 960 coupled to the input / output (I / O) device 980, a memory 910, a storage 930, and a network interface 970. In some embodiments, the action policy 521 resides in the memory 930. The processing unit 950 may be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of processing unit, or a combination of different processing units, such as a CPU configured to operate in conjunction with a GPU or a digital signal processor (DSP). Generally, the processing unit 950 may be any technically feasible hardware unit capable of processing data and / or executing software applications (including RL agent 520, optimizer application 530, and / or reinforcement learning training process 911).

[0093] I / O device 980 may include devices capable of providing input, such as a keyboard, mouse, touchscreen, etc., and devices capable of providing output, such as a display device, etc. Additionally, I / O device 980 may include devices capable of receiving input and providing output, such as a touchscreen, a Universal Serial Bus (USB) port, etc. I / O device 980 may be configured to receive various types of input from the end user of computing device 900 and also to provide various types of output to the end user of computing device 900, such as displayed digital images or digital video. In some embodiments, one or more I / O devices 980 are configured to couple computing device 900 to a network.

[0094] Memory 910 may include random access memory (RAM) modules, flash memory cells, or any other type of memory cell or a combination thereof. Processing unit 950, I / O device interface 960, and network interface 970 are configured to read data from memory 910 and write data to memory 910. Memory 910 includes various software programs executable by processor 950 and application data associated with said software programs, including RL agent 520, optimizer application 530, and / or reinforcement learning training process 911.

[0095] Figure 10 This is a block diagram of an illustrative embodiment of a computer program product 1000 for implementing one or more embodiments of the present disclosure. The computer program product 1000 may include a signal carrying medium 1005. The signal carrying medium 1004 may include one or more sets of executable instructions 1002, which, when executed by, for example, a processor of a computing device, may at least provide the above-mentioned... Figures 1-9 The described function.

[0096] In some embodiments, signal carrying medium 1004 may include non-transient computer-readable medium 1008, such as, but not limited to, hard disk drives, CDs, digital video discs (DVDs), digital magnetic tapes, memory, etc. In some embodiments, signal carrying medium 1004 may include recordable medium 1010, such as, but not limited to, memory, read / write (R / W) CDs, R / W DVDs, etc. In some embodiments, signal carrying medium 1004 may include communication medium 1006, such as, but not limited to, digital and / or analog communication media (e.g., optical fiber, waveguides, wired communication links, wireless communication links, etc.). Computer program product 1000 may be recorded on non-transient computer-readable medium 1008 or another similar recordable medium 1010.

[0097] In summary, embodiments of this disclosure facilitate the optimization of radiodelivery treatment planning utilizing an RL agent. The RL agent is configured to generate radiodelivery treatment plans that can exceed the quality of one or more plans used to train the RL agent. The RL agent is trained to evaluate radiodelivery treatment plans output by an optimization software application, modify one or more dose-volume target parameters of the evaluated radiodelivery treatment plans, and then input the modified radiodelivery treatment plans into the optimization software application for further optimization. The RL agent adaptively adjusts one or more dose-volume target parameters based on action policies learned during the reinforcement learning training process.

[0098] Various embodiments have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0099] Various aspects of this embodiment can be implemented as a system, method, or computer program product. Therefore, various aspects of the invention can take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as “circuit,” “module,” or “system.” Furthermore, various aspects of this disclosure can take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code implemented thereon.

[0100] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include the following: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0101] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The aspects and embodiments disclosed herein are for illustrative purposes and are not intended to be limiting, wherein the true scope and spirit are indicated by the appended claims.

Claims

1. A computer-implemented method for generating a treatment plan for delivering radiotherapy to a target volume using a reinforcement learning agent, the method comprising: The evaluation of the first treatment plan is performed by comparing the dose-volume distribution based on the first treatment plan against the target volume with dose-volume targets included in a set of one or more dose-volume targets; Based on the assessment and based on the action strategy for improving the treatment plan, an action is determined to modify at least one dose-volume target parameter for a dose-volume target included in the set of one or more dose-volume targets, wherein the agent selects the action based on the action strategy learned by the agent to maximize the reward function; A modified dose-volume target is generated by performing the action on the dose-volume target included in the set of one or more dose-volume targets; as well as The optimized application generates a second treatment plan that meets the modified dose-volume target.

2. The computer-implemented method of claim 1, wherein the dose-volume target parameter includes at least one of the following: the minimum dose distribution within the target volume, the maximum dose distribution within a risk organ close to the target volume, and a factor included in a cost function that quantifies the penalty for a treatment plan that fails to achieve the dose-volume target.

3. The computer-implemented method of claim 2, wherein the factor included in the cost function comprises a weighting factor associated with a first dose-volume target in the set of the one or more dose-volume targets.

4. The computer-implemented method of claim 3, wherein the weighting factor modifies the first contribution of the cost function of the dose-volume target included in the set of one or more dose-volume targets relative to a second contribution of the cost function of another dose-volume target included in the set of one or more dose-volume targets.

5. The computer-implemented method according to any one of claims 1 to 4, wherein determining the action comprises selecting at least one action from an action space, the action space comprising: The threshold associated with at least one dose-volume target included in the set of one or more dose-volume targets is changed; the dose-volume curve associated with at least one dose-volume target included in the set of one or more dose-volume targets is repositioned on the dose-volume histogram; and the weighting factor associated with at least one dose-volume target included in the set of one or more dose-volume targets is modified.

6. The computer-implemented method of claim 5, wherein selecting the action comprises selecting a plurality of actions from the action space.

7. The computer-implemented method according to any one of claims 1 to 4, further comprising: Before the optimized application generates the second treatment plan, a modified treatment plan is generated by modifying the first treatment plan using the modified dose-volume target parameters. The process of generating the second treatment through the optimized application includes performing an optimization process on the modified treatment plan.

8. The computer-implemented method of claim 7, wherein causing the optimization application to generate the second treatment plan comprises: The optimization application performs at least one iteration of the optimization process on the modified treatment plan.

9. The computer-implemented method according to any one of claims 1 to 4, wherein the action strategy is generated for a clinical scenario.

10. The computer-implemented method according to any one of claims 1 to 4, further comprising: Before performing the evaluation of the first treatment plan, entries in the action strategy are generated based on a first reward value associated with the action.

11. The computer-implemented method of claim 10, wherein the action is associated with satisfying a threshold condition of the dose-volume target included in the set of one or more dose-volume targets, and the first reward value is greater than a second reward value associated with an action that exceeds the threshold condition.

12. The computer-implemented method according to any one of claims 1 to 4, further comprising: This allows the second treatment plan to be implemented via a radiotherapy system.

13. A non-transient computer-readable medium storing instructions, which, when executed by a processor, cause the processor to generate a treatment plan for delivering radiotherapy to a target volume by performing the following steps by a reinforcement learning agent: The evaluation of the first treatment plan is performed by comparing the dose-volume distribution based on the first treatment plan against the target volume with dose-volume targets included in a set of one or more dose-volume targets; Based on the assessment and based on the action strategy for improving the treatment plan, an action is determined to modify at least one dose-volume target parameter for a dose-volume target included in the set of one or more dose-volume targets, wherein the agent selects the action based on the action strategy learned by the agent to maximize the reward function; A modified dose-volume target is generated by performing the action on the dose-volume targets included in the set of one or more dose-volume targets; and The optimized application generates a second treatment plan that meets the modified dose-volume target.

14. The non-transient computer-readable medium of claim 13, wherein the dose-volume target parameter comprises at least one of the following: a minimum dose distribution within the target volume, a maximum dose distribution within an organ of risk close to the target volume, and a factor included in a cost function that quantifies the penalty for a treatment plan that fails to achieve the dose-volume target.

15. The non-transient computer-readable medium of claim 14, wherein the factor included in the cost function comprises a weighting factor associated with a first dose-volume target in the set of the one or more dose-volume targets.

16. The non-transient computer-readable medium of claim 15, wherein the weighting factor modifies the first contribution of the cost function of the dose-volume target included in the set of one or more dose-volume targets relative to a second contribution of the cost function of another dose-volume target included in the set of one or more dose-volume targets.

17. The non-transient computer-readable medium according to any one of claims 13 to 16, wherein determining the action includes selecting at least one action from an action space, the action space comprising: The threshold associated with at least one dose-volume target included in the set of one or more dose-volume targets is changed; the dose-volume curve associated with at least one dose-volume target included in the set of one or more dose-volume targets is repositioned on the dose-volume histogram; and the weighting factor associated with at least one dose-volume target included in the set of one or more dose-volume targets is modified.

18. The non-transient computer-readable medium of claim 17, wherein selecting the action includes selecting a plurality of actions from the action space.

19. The non-transient computer-readable medium according to any one of claims 13 to 16, further comprising instructions that, when executed by the processor, cause the processor to perform the following steps: Before the optimized application generates the second treatment plan, a modified treatment plan is generated by modifying the first treatment plan using the modified dose-volume target parameters.

20. The non-transient computer-readable medium of claim 19, wherein causing the optimizing application to generate the second treatment plan comprises: The optimization application performs at least one iteration of the optimization process on the modified treatment plan.

Images