Method and system for adaptive radiation therapy planning using a deep learning engine

Abstract

An example method for adaptive radiation therapy planning using a deep learning engine is provided. An example method may include obtaining treatment image data associated with a first imaging modality. The treatment image data may be acquired during a treatment phase for a patient. Additionally, planning image data associated with a second imaging modality may be acquired prior to the treatment phase to generate a treatment plan for the patient. The method may also include, in response to determining that an update to the treatment plan is required, transforming the treatment image data associated with the first imaging modality to generate transformed image data associated with the second imaging modality. The method may also include processing the transformed image data using the deep learning engine to generate output data to update the treatment plan.

Description

[0001] This application is a divisional application of an invention patent application with an application date of September 26, 2019, a Chinese national application number of 201910918486.X, and an invention name of “A computer system for executing adaptive radiation therapy plans”. Technical Field

[0002] The present disclosure relates to methods and systems for adaptive radiation therapy planning using a deep learning engine. Background Art

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0004] Radiation therapy is an important part of treatment for reducing or eliminating unwanted tumors from patients. Unfortunately, applied radiation itself will not distinguish unwanted tumors from any proximal healthy structures, such as organs, etc. This requires careful administration to limit the radiation to the tumor (i.e., the target). Ideally, the goal is to deliver a lethal or disease-eliminating radiation dose to the tumor while maintaining an acceptable dose level in proximal healthy structures. However, to achieve this goal, conventional radiation therapy planning and / or adaptive radiation therapy planning can be time-consuming and labor-intensive. Summary of the invention

[0005] According to a first aspect of the present disclosure, an example method and system for performing radiation therapy planning using a deep learning engine is provided. Figures 1 to 5 to discuss various examples. The deep learning engine may include at least a first processing path, a second processing path, and a third processing path. An example method may include: obtaining first image data associated with a patient; generating first feature data by processing the first image data associated with a first resolution level using the first processing path; generating second feature data by processing the second image data associated with a second resolution level using the second processing path; and generating third feature data by processing the third image data associated with a third resolution level using the third processing path. The example method may also include: generating a first group and feature data set associated with the second resolution level based on the second feature data and the third feature data; and generating a second group and feature data set associated with the first resolution level based on the first feature data and the first combination set. In addition, the example method may include generating output data associated with radiation therapy of the patient. For example, the output data may include at least one of: structural data associated with the patient, dose data associated with the patient, and therapy delivery data for a therapy delivery system.

[0006] According to a second aspect of the present disclosure, an example method and system for adaptive radiation therapy planning using a deep learning engine is provided. Figure 6 and Figure 7 to discuss various examples. An example method may include obtaining treatment image data associated with a first imaging modality. The treatment image data may be acquired during a treatment phase for a patient. Additionally, planning image data associated with a second imaging modality may be acquired prior to the treatment phase to generate a treatment plan for the patient. The method may also include, in response to determining that the treatment plan needs to be updated, transforming the treatment image data associated with the first imaging modality to generate transformed image data associated with the second imaging modality. The method may also include processing the transformed image data using a deep learning engine to generate output data to update the treatment plan. For example, the output data may be at least one of: structural data associated with the patient, dose data associated with the patient, and treatment delivery data for a treatment delivery system.

[0007] According to a third aspect of the present disclosure, an example method and system for adaptive radiation therapy planning using a deep learning engine is provided. Figure 6 and Figure 8 Various examples are discussed below. An example method may include obtaining treatment image data associated with a first imaging modality and planning image data associated with a second imaging modality. The planning image data may be acquired prior to a treatment phase to generate a treatment plan for a patient. The method may also include, in response to determining that an update to the treatment plan is required, processing the treatment image data and the planning image data using a deep learning engine to generate output data to update the treatment plan. For example, the output data may be at least one of: structural data associated with the patient, dose data associated with the patient, and treatment delivery data for a treatment delivery system. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram showing an example process flow for radiation therapy;

[0009] Figure 2 is a schematic diagram illustrating an example deep learning engine having multiple processing paths for performing radiation therapy treatment planning;

[0010] Figure 3 is a flow chart of an example process for performing radiation therapy treatment planning by a computer system using a deep learning engine;

[0011] Figure 4 is a schematic diagram illustrating an example deep learning engine performing automatic segmentation of image data for radiation therapy planning;

[0012] Figure 5is a schematic diagram illustrating an example deep learning engine performing dose prediction for radiation therapy planning;

[0013] Figure 6 is a schematic diagram illustrating an example process flow for performing adaptive radiation therapy (ART) planning by a computer system using a deep learning engine;

[0014] Figure 7 It shows that according to Figure 6 A schematic diagram of a first example method for performing an ART program as in the example;

[0015] Figure 8 It shows that according to Figure 6 A schematic diagram of a second exemplary method for performing an ART program according to an example in FIG.

[0016] Fig. 9 It is shown based on Figures 1 to 8 a schematic diagram of an example treatment plan generated or improved based on the output data in the example; and

[0017] Fig.10 is a schematic diagram of an example computer system for performing radiation therapy planning and / or adaptive radiation therapy planning. DETAILED DESCRIPTION

[0018] The technical details set forth in the following description enable one skilled in the art to practice one or more embodiments of the present disclosure.

[0019] Figure 1 1 is a schematic diagram illustrating an example process flow 100 for radiation therapy. The example process 100 may include one or more operations, functions, or actions illustrated by one or more blocks. Various blocks may be combined into fewer blocks, divided into additional blocks, and / or deleted based on the desired implementation. Figure 1 In the example in , radiation therapy typically includes various stages, such as an imaging system performing image data acquisition for a patient (see 110); a radiation therapy planning system (see 130) generating a suitable treatment plan for the patient (see 156); and a treatment delivery system (see 160) delivering treatment according to the treatment plan.

[0020] In more detail, Figure 1At 110 in, image data acquisition may be performed using an imaging system to capture image data 120 associated with a patient, particularly an anatomical structure of the patient. Any one or more suitable medical image modalities may be used, such as computed tomography (CT), cone beam computed tomography (CBCT), positron emission tomography (PET), magnetic resonance imaging (MRI), single photon emission computed tomography (SPECT), any combination thereof, etc. For example, when CT or MRI is used, the image data 120 may include a series of two-dimensional (2D) images or slices, each 2D image or slice representing a cross-sectional view of the anatomical structure of the patient, or may include a volumetric or three-dimensional (3D) image of the patient, or may include a 2D or 3D time series image of the patient (e.g., four-dimensional (4D) CT or 4D CBCT).

[0021] exist Figure 1 At 130 in the figure, radiation therapy planning can be performed during the planning stage to generate a treatment plan 156 based on the image data 120. Any suitable number of treatment planning tasks or steps can be performed, such as segmentation, dose prediction, projection data prediction, treatment plan generation, etc. For example, segmentation can be performed to generate a data structure 140, which identifies various segments or structures that can form the image data 120. In practice, a three-dimensional (3D) volume of the patient's anatomical structure can be reconstructed from the image data 120. The 3D volume that will be subjected to radiation is called a treatment or irradiation volume, which can be divided into a plurality of smaller volume pixels (voxels) 142. Each voxel 142 represents a 3D element associated with a location (i, j, k) within the treatment volume. The structural data 140 can include any suitable data related to the contours, shapes, sizes, and locations of the patient's anatomical structure 144, the target 146, and any organs at risk (OARs) 148.

[0022] In another example, dose prediction may be performed to generate dose data 150 that specifies the radiation dose (denoted as “D TAR ”) and the radiation dose to OAR 148 (denoted as “D OAR In practice, the target 146 may represent a malignant tumor (e.g., a prostate tumor, etc.) that requires radiation therapy, and the OAR 148 represents a proximal healthy structure or non-target structure (e.g., rectum, bladder, etc.) that may be adversely affected by the treatment. The target 146 is also referred to as a planning target volume (PTV). Although Figure 1, but the treatment volume may include multiple targets 146 and OARs 148 having complex shapes and sizes. In addition, although shown as having a regular shape (e.g., a cube), the voxel 142 may have any suitable shape (e.g., an irregular shape). Depending on the desired implementation, the radiation therapy planning at box 130 may be performed based on any additional and / or alternative data, such as prescription, disease stage, bio- or radiomics data, genetic data, assay data, biopsy data, past treatment history or medical history, any combination thereof, etc.

[0023] Based on the structural data 140 and the dose data 150, a treatment plan 156 may be generated that includes 2D fluence map data regarding a set of beam orientations or beam angles. Each fluence map specifies the intensity and shape (e.g., as determined by a multi-leaf collimator (MLC)) of a radiation beam emitted from a radiation source at a specific beam orientation and at a specific time. For example, in practice, intensity modulated radiation therapy (IMRT) or any other treatment technique may involve changing the shape and intensity of the radiation beam while being at a constant gantry and bed angle. Alternatively or in addition, the treatment plan 156 may include machine control point data (e.g., jaw and leaf positions), volumetric modulated radiation therapy (VMAT) trajectory data for controlling a treatment delivery system, and the like. In practice, block 130 may be performed based on a target dose specified by a clinician (e.g., an oncologist, a radiation dosimeter, a planner, etc.), such as based on the clinician's experience, the type and size of the tumor, the patient's geometry and condition, and the like.

[0024] exist Figure 1 At 160 in the figure, treatment delivery is performed during a treatment session to deliver radiation to a patient according to the treatment plan 156. For example, the radiation treatment delivery system 160 may include a rotatable gantry 164 to which a radiation source 166 is attached. During treatment delivery, the gantry 164 rotates around a patient 170 supported on a structure 172 (e.g., a table) to emit radiation beams 168 at various beam orientations according to the treatment plan 156. The controller 162 can be used to retrieve the treatment plan 156 and control the gantry 164, the radiation source 166, and the radiation beams 168 to deliver the radiation treatment according to the treatment plan 156.

[0025] It should be understood that any suitable radiation therapy delivery system can be used, such as a robotic arm-based system, a spiral blade type system, brachy, sirex microsphere radiotherapy, any combination thereof, etc. In addition, examples of the present disclosure can be applied to particle delivery systems (e.g., protons, carbon ions, etc.). Such systems can use scattered particle beams that are then shaped by a device similar to MLC; or can alternatively use scanning beams with adjustable energy, spot size, and dwell time.

[0026] Generally speaking, in Figure 1 The radiation therapy plan at the box 130 in the image is time-consuming and laborious. For example, a highly skilled and well-trained team of oncologists and radiation dosimeters is usually required to manually outline the structure of interest by drawing contours or segments based on the image data 120. These structures are manually reviewed by the physician, which may require adjustment or redrawing. In many cases, the segmentation of critical organs may be the most time-consuming part of the radiation therapy plan. After the structure is agreed upon, there are additional laborious steps to process the structure to generate a clinically optimal treatment plan for specifying treatment delivery data, such as beam orientation and trajectory, and a corresponding 2D fluence map. Due to the lack of consensus between different physicians and / or clinical bases on what constitutes a "good" contour or segmentation, these steps are often complicated. In practice, there may be great variations in the way structures or segments are drawn by different clinical experts. The variations may result in uncertainty in the size and shape of the target volume, as well as the exact proximity, size and shape of the OAR that should receive the lowest radiation dose. Even for a specific clinical expert, there may be variations in the way segments are drawn on different days.

[0027] According to examples of the present disclosure, artificial intelligence (AI) techniques can be applied to alleviate various challenges associated with radiotherapy planning. In particular, a deep learning engine can be used to automate radiotherapy planning steps and / or adaptive radiotherapy planning steps. Examples of the present disclosure can be implemented to improve the efficiency of radiotherapy planning and potentially improve treatment outcomes, such as increasing the probability of tumor control and / or reducing the likelihood of health complications or death due to excessive radiation doses in healthy structures. For example, automatic segmentation of image data 120 will greatly benefit speeding up workflow and enabling various applications, such automatic treatment planning and radiotherapy adjustments.

[0028] Throughout this disclosure, the term "deep learning" may generally refer to a class of methods that use multi-layer or multi-stage nonlinear data processing for feature learning and pattern analysis and / or classification. Thus, the term "deep learning model" may refer to a hierarchical structure of nonlinear data processing layers, including an input layer, an output layer, and multiple (i.e., two or more) "hidden" layers between the input layer and the output layer. These layers may be trained end-to-end (e.g., from the input layer to the output layer) to extract features from the input, and classify the features to produce an output (e.g., a classification label or category).

[0029] Thus, the term "deep learning engine" may refer to any suitable hardware and / or software component of a computer system that is capable of executing an algorithm according to any suitable deep learning model. Depending on the desired implementation, any suitable deep learning model may be used, such as a convolutional neural network, a recursive neural network, a deep belief network, or any combination thereof, etc. In practice, a neural network is typically formed using a network of processing elements (referred to as "neurons", "nodes", etc.) interconnected via connections (referred to as "synapses", "weights", etc.).

[0030] Deep learning methods should be contrasted with machine learning methods that have been applied to, for example, automatic segmentation. In general, these methods involve extracting (artificially designed) feature vectors from images, such as for each voxel. The feature vectors can then be used as input to a machine learning model that classifies the category to which each voxel belongs. However, such machine learning methods typically do not utilize complete image data and may require additional constraints. Another challenge is that in order to accurately predict the class label of each voxel, these methods rely on high-dimensional artificially designed features. Solving high-dimensional classification problems is computationally expensive and requires a lot of memory. Some methods use low-dimensional features (e.g., using dimensionality reduction techniques) but they may reduce prediction accuracy.

[0031] Next, we will use Figures 1 to 10 Various examples are discussed. In particular, Figures 1 to 5 We will discuss radiation therapy treatment planning using a deep learning engine. Figures 6 to 9 The adaptive radiation therapy planning using a deep learning engine is discussed in detail below. Examples of the present disclosure may be implemented using any suitable computer system, one example of which is shown in FIG. Fig.10 middle.

[0032] Deep learning engine with multiple processing paths

[0033] According to a first aspect of the present disclosure, radiation therapy plans can be improved using a deep learning engine with multiple (K) processing paths to process medical image data at different resolution levels. Some examples will use Figure 2 and Figure 3 In particular, Figure 2 is a schematic diagram illustrating an example deep learning engine 200 having multiple processing paths for radiation therapy planning. Figure 3 is a flow chart of an example process 300 for performing radiation therapy treatment planning by a computer system using a deep learning engine 200. The example process 300 may include one or more operations, functions, or actions illustrated by one or more blocks, such as 310 through 370. Various blocks may be combined into fewer blocks, divided into additional blocks, and / or deleted based on the desired implementation.

[0034] exist Figure 2 In the example in FIG. 1 , the multi-resolution deep learning engine 200 includes at least three (K=3) processing paths 221-223 to process the image at different resolution levels (denoted as R k =R1, R2, R3) to process the corresponding image data 211-213 to generate output data 260. The image data 211-213 at multiple resolution levels can be separately transmitted to specially designed processing paths 221-223. In this way, compared with conventional methods that rely on a single processing path, the deep learning engine 200 can achieve a larger receptive field to improve prediction results. In particular, in order to achieve the same (i.e., larger) receptive field, a single-path deep learning engine will have to be deeper (e.g., more layers) and therefore require greater computing power.

[0035] In practice, in order to facilitate the extraction and analysis of both local feature data and global feature data in the image data 211-213 to produce better quality output data, a larger receptive field is preferred over a smaller receptive field. In general, deep neural networks can be difficult to adjust to operate appropriately on medical image data because the required accuracy and reliability are relatively high. By decomposing the image processing problem into multiple resolution levels, examples of the present disclosure can be implemented in a resource-efficient manner. In the case of users with limited processing resources, for example, in order to improve efficiency, memory-efficient methods are preferred. For example, the processing cost is lower at lower resolutions in the sense that the processing path can process more distant data (e.g., feature data at different physical distances) at the same cost as in the absence of downsampling.

[0036] Additional references Figure 3 , at 310, first image data 120 / 211 associated with the patient is obtained. Herein, the term "obtain" may generally refer to the computer system accessing or retrieving the image data 120 / 211 from any suitable source (e.g., another computer system), memory or storage (e.g., local or remote), etc. In practice, the first image data 211 may be 2D or 3D image data acquired using any one or more suitable imaging modalities.

[0037] exist Figure 3 At 320 in FIG. 1 , the first image data (I1) 211 associated with the first resolution level (R1) is processed using the first processing path 221 to generate the first feature data (F1) 241. Figure 3 At 330 in FIG. 2 , the second image data (I2) 212 associated with the second resolution level (R2) is processed using the second processing path 222 to generate second feature data (F2) 242. Figure 3At 340 in FIG. 1 , the third image data (I3) 213 associated with the third resolution level (R3) is processed using the third processing path 223 to generate third feature data (F3) 243. In practice, the "processing path" can be implemented using any suitable architecture, such as a convolutional block or convolutional layer, to generate feature data.

[0038] exist Figure 3 At 350 in FIG. 1 , a first set of feature data sets (C1) 251 associated with a second resolution level (R2) is generated based on the second feature data (F2) 242 and the third feature data (F3) 243. Figure 3 At 360 in FIG. 1 , a second set of sum feature data (C2) 252 associated with the first resolution level (R1) is generated based on the first set of sum data (C1) 251 and the first feature data (F1) 241. Figure 3 At 370, output data 260 associated with radiation treatment of the patient is generated based on the second group and set (C2) 252. Examples of the present disclosure may be implemented to facilitate better integration of feature data 241-243 at different resolution levels from respective processing paths 221-223.

[0039] Depending on the desired implementation, the deep learning engine 200 can be trained to perform automatic segmentation to produce output data = structure data (e.g., Figure 1 140), performing dose prediction to produce output = dose data (e.g., Figure 1 150), performing treatment delivery data estimation to produce output = treatment delivery data, or any combination thereof. For example, in the case of automatic segmentation, the structural data may include segmented and labeled images that specify structures or segments that can be identified from the image data 120 / 211, such as Figure 1 The dose data may specify the radiation doses to the targets 146 and OARs 148 (see also Figure 1 152-154 in ). In the case of treatment delivery data estimation, "treatment delivery data" can be structure projection data (e.g., beam trajectory and / or orientation), fluence map data, etc.

[0040] exist Figure 2 In the example of , the second image data (I2) 212 can be generated based on the first image data (I1) 211, and the third image data (I3) 213 can be generated based on the second image data (I2) 212 or the first image data (I1) 211. The deep learning engine 200 includes matching resampling (i.e., downsampling or upsampling) blocks 231-234. For example, in R1>R2>R3 (using Figure 4 and Figure 5 In the case of FIG. 2 , the downsampling blocks 231 - 232 may be used to reduce the resolution level, and the matched upsampling blocks 233 - 234 may be used to increase the resolution level.

[0041] In the following, we will use Figure 4 Describe examples related to automatic segmentation and use Figure 5 Examples related to dose prediction are described. Although illustrated using a deep convolutional neural network, it should be understood that any alternative and / or additional deep learning models may be used to implement the deep learning engine 200 / 400 / 500 according to examples of the present disclosure.

[0042] Automatic segmentation

[0043] Figure 4 is a schematic diagram showing an example deep learning engine 400 performing automatic segmentation of image data for radiation therapy planning. Figure 2 , the deep learning engine 400 includes K=3 processing paths (see 421-423) that are trained during the training phase 401. After training, the deep learning engine 400 can be used (e.g., by a clinician) to perform automatic segmentation on actual patients during the inference phase 402.

[0044] (a) Training Data

[0045] During the training phase 401, the deep learning engine 400 may be trained using any suitable training data 411-412 related to automatic segmentation. In practice, the training data 411-412 may include example input data = unsegmented image data 411, and example output data = structural data 412 (also referred to as segmentation data). The structural data 412 may identify any suitable contour, shape, size, and / or position of a patient's anatomical structure, such as a target, OAR, or the like, or a structure or segment. The image data 411 may include a 2D or 3D image of the patient's anatomical structure and may be captured using any one or more suitable imaging modalities. Depending on the desired implementation, the structural data 412 may be manually generated by a trained professional using any suitable method and clinically validated.

[0046] The purpose of the training phase 401 is to train the deep learning engine 400 to perform automatic segmentation by mapping input data = image data 411 to example output data = structure data 412. The training phase 401 may involve finding weights that minimize the training error between the trained structure data 412 and the estimated structure data 482 produced by the deep learning engine 400. In practice, the deep learning engine 200 may be trained to recognize multiple targets and OARs of any suitable shape and size.

[0047] For example, with respect to prostate cancer, image data 411 may include image data of a patient's prostate. In this case, structural data 412 may identify a target representing the patient's prostate, and OARs representing proximal healthy structures, such as the rectum or bladder. With respect to lung cancer treatment, image data 411 may include image data of a patient's lungs. In this case, structural data 412 may identify a target representing cancerous lung tissue, and OARs representing proximal healthy lung tissue, esophagus, heart, etc. With respect to brain cancer, image data 411 may include image data of a patient's brain, in which case structural data 412 may identify a target representing a brain tumor, and OARs representing proximal optic nerves, brainstem, etc.

[0048] In practice, the training data 411-412 can be generated by the user with the aid of observation and experience to facilitate supervised learning. For example, the training data 411-412 can be extracted from past treatment plans formulated for past patients. The training data 411-412 can be preprocessed using any suitable data enhancement method (e.g., rotation, flipping, translation, scaling, noise addition, cropping, any combination thereof, etc.) to generate a new data set with improved properties so as to improve model generalization using true value data. In practice, the 3D volume of the patient to be irradiated is referred to as the treatment volume, which can be divided into a plurality of smaller volume pixels (voxels). In this case, the structural data 412 can specify a class label (e.g., "target", "OAR", etc.) associated with each voxel in the 3D volume. Depending on the desired implementation, the structural data 412 can identify a plurality of targets and OARs of any suitable shape and size.

[0049] (b) Processing paths and layers

[0050] The deep learning engine 400 includes three processing paths 421-423 (k=1, 2, 3) to process images at different resolution levels (R k=R1, R2, R3). The first processing path 421 (k=1) is configured to process the first image data (I1) at a first resolution level R1 (e.g., 1x). The second processing path 422 (k=2) is configured to process the second image data (I2) at a second resolution level R2<R1 to expand the receptive field. The third processing path 423 (k=3) is configured to process the third image data (I3) at a third resolution level R3<R2<R1 to further expand the receptive field.

[0051] exist Figure 4 In the example of , the input of the first processing path 421 is image data 411. The input of the second processing path 422 is image data 411 that has been downsampled by downsampling block 431 (e.g., by a factor of 2x). The input of the third processing path 423 is image data 411 that has been downsampled by both downsampling blocks 431-432 (e.g., by a total factor of 4x). Downsampling blocks 431-432 have matching upsampling blocks 441-443 for upsampling before combining feature data (see F1, F2, F3) from corresponding processing paths 421-423. In other words, each downsampling step has a corresponding (i.e., matching) upsampling step to re-adjust the resolution level. Downsampling blocks 431-432 can be implemented using segmented sampling, pooling, etc., and upsampling blocks 441-443 can be implemented using transposed convolution, repetition, etc.

[0052] By processing the image data 411 at multiple resolution levels, the processing paths 421-423 provide different views into the image data 411 to achieve a larger receptive field. In practice, medical image data typically includes both local feature data and global feature data of the patient's anatomical structure, where the terms "local" and "global" are essentially related. For example, the local feature data can provide a microscopic view of the patient's anatomical structure, such as tissue texture, regardless of whether the structure has finite boundaries, etc. In contrast, the global feature data can provide a relatively macroscopic view of the patient's anatomical structure, such as in which region the anatomical structure is located (e.g., pelvis, abdomen, head and neck, etc.), in which orientation (e.g., on the left, on the right, in front, on the back), etc.

[0053] exist Figure 4In an example of, a first processing path 421 can process image data 411 at the highest resolution level (R1) to analyze local tissue texture. A second processing path 422 can process image data 411 at an intermediate resolution level (R2 < R1) to analyze evidence of nearby structural boundaries for tissue type changes. A third processing path 422 can process image data 411 at the coarsest resolution level (R3 < R2 < R1) to analyze landmarks such as bones and body contours. Processing image data 411 at lower resolution levels generally requires less processing. This is particularly important for 3D image data processing, where halving the resolution can cut processing costs to 1 / 8. This allows more resources to be invested in more accurate segmentation, such as more channels in processing paths 421-423.

[0054] Using, for example, a deep convolutional neural network, processing paths 421-423 can each include any suitable number of convolutional layers (e.g., 424-426) to extract feature data (F1, F2, F3) at different resolution levels from image data 411. In practice, each convolutional layer can be configured to extract feature data (e.g., 2D or 3D feature maps) at a specific resolution level by applying a filter or kernel to overlapping regions of its input. The values ​​of the parameters in the convolutional filters are obtained during the training phase 401. For example, the convolutional layer can create a 2D feature map including features that appear in 2D image data, or a 3D feature map of related 3D image data. This automatic feature extraction method should be distinguished from conventional methods that rely on manually designed features.

[0055] The deep learning engine 400 also includes additional convolutional layers or blocks 450-470 and a mixing block 480 (one is shown for simplicity) to combine the feature data (F1, F2, F3) from the processing paths 421-423 in a staged manner. In particular, the third feature data (F3) from the third processing path 423 can be upsampled from the lowest resolution level R3 to the intermediate resolution level R2 using the upsampling block 441. The upsampled third feature data (F3) is then combined with the second feature data (F2) from the second processing path 422 using the convolution block 450, thereby producing a first combination set (C1). As an optimization strategy, the convolution block 450 can be configured to "smooth" or "refine" the second feature data (F2) and the upsampled third feature data (F3) before performing another stage of upsampling (e.g., 2x) using subsequent upsampling blocks 442-443.

[0056] Additional convolution blocks 460-470 are then used to combine the feature data (F1, F2, F3) from all processing paths 421-423, thereby producing a second group and set (C2). In particular, the feature data can be combined by upsampling the lower resolution path to the resolution of the higher resolution path. In order to bring the different feature data to the same resolution level, upsampling blocks 442-443 can be used to upsample the first combined set (C1) from convolution block 450. In practice, the convolution blocks included in the processing paths 421-423 and the convolution blocks 450-470 can have any suitable configuration (e.g., 3x3x3 convolution).

[0057] The second combination set (C2) produced using the convolution blocks 460-470 is then processed using a mixing block 480 to produce output data = estimated structural data 482. The mixing block 480 is configured to separate the final feature set into a final segmentation decision (i.e., estimated structural data 482) (e.g., via a 1x1x1 convolution). The estimated structural data 482 can specify voxel-based classification data such as associated with a treatment volume identified from the image data 411. For example, a voxel can be classified as a target (e.g., label = "TAR") or an OAR (e.g., label = "OAR"). In practice, label = "OAR" can represent a larger set of labels, such as "rectum", "bladder", "brainstem", or any other automatically defined volume. In addition, label = "TAR" can represent a tumor or treatment volume.

[0058] The above training steps can be repeated during the training phase 401 to minimize the error between the expected results in the trained structure data 412 and the estimated structure data 482. Depending on the desired implementation, the deep learning engine 400 can be implemented using any suitable convolutional neural network architecture, such as U-net, LeNet, AlexNet, ResNet, V-net, DenseNet, etc. For example, the U-net architecture includes a contraction path (left side) and an expansion path (right side). The contraction path includes repeated application of convolutions, followed by rectified linear units (ReLU) and maximum pooling operations. Each step in the expansion path can include upsampling the feature map, followed by convolution, etc. It should be noted that the processing paths 421-423 can use the same architecture or different architectures.

[0059] (c) Inference stage

[0060] After training, the deep learning engine 400 can be used by a clinician during an inference phase 402 to perform segmentation to produce output data = patient structure data 260 / 492 based on the input data = image data 210 / 491 for a specific patient. The image data 210 / 491 can be processed by the processing paths 421-423 of the deep learning engine 400 at the corresponding resolution level to expand the receptive field. Figure 3 The example process explained (see blocks 310 - 370 ) may be applied to perform automatic segmentation and will not be repeated here for the sake of brevity.

[0061] Dose prediction

[0062] Figure 5 is a schematic diagram showing an example deep learning engine 500 performing dose prediction for a radiation therapy treatment plan. Figure 4 , the deep learning engine 500 includes K=3 processing paths (see 521-523) that are trained during the training phase 501. After training, the deep learning engine 500 can be used (e.g., by a clinician) to perform dose predictions on actual patients during the inference phase 502.

[0063] (a) Training Data

[0064] During the training phase 501, the deep learning engine 500 may be trained using any suitable training data 511-512 related to dose prediction. In practice, the training data 511-512 may include example input data = image data and structure data 511 (i.e., segmented image data), and example output data = dose data 512. The dose data 512 (e.g., 3D dose data) may specify a target dose distribution (denoted as “D TAR ”) and OAR dose distribution (denoted as “D OAR ”). In practice (not shown for simplicity) Figure 5 ), the dose data 512 may specify the dose distribution of the entire 3D volume, not just the target and OAR volumes. Depending on the desired implementation, the dose data 512 may include spatial bioeffect data (e.g., graded corrected dose) and / or cover only a portion of the treatment volume.

[0065] For example, with respect to prostate cancer, the dose data 512 may specify a target representing the patient's prostate, and a dose distribution representing proximal healthy structures, such as the rectum or bladder, in OARs. With respect to lung cancer treatment, the dose data 512 may specify a target representing cancerous lung tissue, and a dose distribution representing proximal healthy lung tissue, esophagus, heart, etc. With respect to brain cancer, the dose data 512 may specify a target representing a brain tumor, and a dose distribution representing proximal optic nerves, brain stem, etc., in OARs.

[0066] The purpose of the training phase 501 is to train the deep learning engine 500 to perform dose prediction by mapping input data = image data and corresponding structure data 511 to example output data = dose data 512. The training phase 501 may involve finding weights (e.g., kernel parameters) that minimize the training error between the trained dose data 512 and the estimated dose data 582 generated by the deep learning engine 500. Any suitable constraints may be used, such as limiting dose prediction only to the vicinity of the target or to certain dose levels.

[0067] (b) Processing paths and layers

[0068] Similar to Figure 4 In the example, Figure 5 The deep learning engine 500 in FIG. 5 includes three processing paths 521 - 523 (k = 1, 2, 3) to process images at different resolution levels (R k =R1, R2, R3). The first processing path 521 (k=1) is configured to process the first image data (I1) at a first resolution level R1 (e.g., 1x). The second processing path 522 (k=2) is configured to process the second image data (I2) at a second resolution level R2<R1 to expand the receptive field. The third processing path 523 (k-3) is configured to process the third image data (I3) at a third resolution level R3<R2<R1 to further expand the receptive field.

[0069] exist Figure 5 In the example of FIG. 5 , the input to the first processing path 521 is image data 511. The input to the second processing path 522 is image data (I2) that has been downsampled by downsampling block 531 (e.g., by a factor of 2x). The input to the third processing path 523 is image data (I3) that has been downsampled by both downsampling blocks 531-532 (e.g., by a total factor of 6x). The downsampling blocks 531-532 have matching upsampling blocks 541-543 for upsampling before combining the feature data (see F1, F2, F3) from the respective processing paths 521-523.

[0070] The deep learning engine 500 also includes additional convolutional layers or blocks 550-570 and a mixing block 580 (one is shown for simplicity) to combine the feature data (F1, F2, F3) from the processing paths 521-523 in stages. Similarly, the third feature data (F3) can be upsampled using the upsampling block 541 (e.g., by a factor of 4x) and then combined with the second feature data (F2) using the convolution block 550 to produce a first combination set (C1). In addition, the first combination set (C1) can be upsampled using the upsampling blocks 542-543 (e.g., by a factor of 2x) and then combined with the first feature data (F1) using the convolution blocks 560-570 to produce a second combination set (C2). The mixing block 580 is configured to sort the final feature set into a final dose prediction decision (i.e., estimated dose data 582) (e.g., using a 1x1x1 convolution).

[0071] (c) Inference stage

[0072] After training, the deep learning engine 500 can be used by a clinician during an inference phase 502 to perform dose prediction to generate output data = dose data 260 / 592 based on the input data = image data 210 / 591 for a specific patient. The image data 210 / 591 can be processed by the processing paths 521-523 of the deep learning engine 500 at the corresponding resolution level to expand the receptive field. Figure 3 The example process explained (see blocks 310 - 370 ) may be applied to perform dose prediction and will not be repeated here for the sake of brevity.

[0073] (d) Changes

[0074] In practice, the deep learning engine 200 / 400 / 500 may be trained to process data associated with any suitable number of resolution levels. In practice, the number of processing paths and corresponding resolution levels may depend on the input image data. For example, at a certain point, downsampling may not reveal additional features of interest because the data is too coarse. Medical image data resolution is often quite high, and to achieve efficiency gains, three or more resolution levels may be appropriate.

[0075] In the case of K=4, a fourth processing path may be used to process fourth image data (I4) associated with a fourth resolution level. For example, the fourth image data (I4) may be generated by downsampling the first image data (I1), the second image data (I2), or the third image data (I3) using any suitable downsampling factor. The feature data (F1, F2, F3, F4) from the corresponding K=4 processing paths may be combined in a staged manner to improve efficiency. For example, F4 and F3 may be combined first, followed by F2, and finally F1 (e.g., in a stepwise manner). H 、F H-1 , ..., F1 order).

[0076] Apart from Figure 4 Automatic segmentation and Figure 5 In addition to dose prediction in , examples of the present disclosure can also be implemented to perform treatment delivery data prediction. In this case, the treatment delivery data (i.e., output data) can include structure projection data, fluence map data, etc. For example, the deep learning engine 200 can be trained to perform structure projection data, such as based on image data, structure data, dose data, or any combination thereof. The structure projection data can include information related to the treatment delivery system (see Figure 1 In another example, the deep learning engine 200 can be trained to perform fluence map estimation, such as a 2D fluence map for a set of beam orientations / trajectories, machine control point data (e.g., jaw and blade positions, gantry and bed positions), etc. Fig. 9 Let's further explain the fluence diagram.

[0077] The input data and output data of the deep learning engine 200 / 400 / 500 may include any suitable additional and / or alternative data. For example, field geometry data may be an input or output for all applications. Other examples include monitor units (amount of radiation counted by the machine), plan assessment quality (acceptable or unacceptable), daily dose prescription (output), field size or other machine parameters, bed position parameters or isocenter location in the patient, treatment strategy (with or without motion control mechanism, pressurization or without pressurization), and treatment or non-treatment decisions.

[0078] Adaptive Radiation Therapy (ART)

[0079] In radiotherapy, the treatment goal is to be able to deliver a high dose to the target (e.g., to kill cancer cells) without harming healthy tissue (e.g., to minimize adverse effects on key OARs). Therefore, it is important to deliver to the correct site during a course of radiotherapy. However, the situation or condition of the patient's anatomical structure at the time of delivery may be significantly different from the situation or condition considered in the treatment plan. For example, the shape, size and position of key organs may have changed compared to those in the planning image data (e.g., CT images). The difference may be caused by various factors, such as internal organ movement (e.g., bladder filling, defecation), weight loss of the patient, shrinkage or expansion of the tumor, etc. In some cases, the existing treatment plan generated based on the planning image data may no longer meet the treatment goals, and a new treatment plan is required. This is referred to as ART.

[0080] For example, for treatment planning purposes, CT image data is typically acquired during the planning phase (i.e., before the treatment phase). The treatment plan can be generated based on manual segmentation of the CT image data. During the treatment phase (e.g., close to or at the time of treatment delivery), CBCT image data can be acquired to monitor any changes in the patient's condition. The clinician can compare the CBCT image data with the CT image data to assess whether the treatment plan is still applicable to produce accurate dose delivery. If the treatment plan no longer meets the treatment goals, the treatment plan needs to be adjusted.

[0081] Conventionally, ART usually involves clinicians repeating manual segmentation steps on newly acquired CBCT image data to improve the quality of the treatment plan. Depending on the case and / or treatment area, segmentation can easily become one of the most costly bottlenecks in ART because the number of structures and the complexity of their shapes may vary. For example, contouring may take minutes or hours. In some cases, it may not be possible to treat the patient in a timely manner because a rescan may be required to continue the planning process offline. The patient cannot continue treatment until the new plan is ready, which has the undesirable consequence of delayed treatment.

[0082] According to examples of the present disclosure, a deep learning engine can be used to improve ART planning. In the following, two example methods will be explained. Figure 6 and Figure 7 The first method may be implemented when the CBCT image data acquired during the treatment phase ("treatment image data") is significantly different from the CT image data acquired before the treatment phase ("planning image data"). Otherwise, the method according to Figure 6 and Figure 7 The second method.

[0083] In more detail, Figure 6600 is a schematic diagram showing an example process flow 600 for performing adaptive radiation therapy therapy planning using a deep learning engine by a computer system. The example process 600 may include one or more operations, functions, or actions illustrated by one or more blocks. Various blocks may be combined into fewer blocks, divided into additional blocks, and / or deleted based on the desired implementation. Examples of the present disclosure may be implemented using any suitable computer system, and one example will use Fig.10 To discuss.

[0084] exist Figure 6 At 610 and 620 in , treatment image data associated with a first imaging modality (e.g., CBCT image data), and planning image data associated with a second imaging modality (e.g., CT image data) may be obtained. Here, the term "obtaining" may refer to a computer system accessing or retrieving image data from any suitable source (e.g., another computer system, local memory / storage, remote memory / storage, etc.). The term "treatment image data" may generally refer to any suitable image data that may be acquired during treatment phase 601 (e.g., close to or on a scheduled treatment day) to determine whether ART is needed. The term "planning image data" may generally refer to any suitable image data that may be acquired during planning phase 602 (i.e., prior to treatment phase 601) for the purpose of generating a treatment plan (see 603) for a patient.

[0085] Next, the treatment image data 610 may be compared with the planning image data 620 to determine whether the treatment plan generated based on the planning image data needs to be updated. If yes (i.e., update is required), the first method (see 640-660) or the second method (see 670-690) may be implemented based on whether the difference exceeds a significance threshold. In particular, Figure 3 At 630 in FIG. 1 , in response to determining that the difference between the treatment image data 610 and the planning image data 620 exceeds a predetermined significance threshold, the first method may be implemented. Otherwise, at Figure 6 At 632 in FIG. 6 , in response to determining that the difference does not exceed a predetermined significance threshold, a second method may be implemented. If the patient's condition has changed significantly since the planning stage 602, the first method may be implemented based on the treatment image data 610. If the difference is not significant, the second method may be implemented to utilize information in both the treatment image data 610 and the planning image data 620.

[0086] The selection between the first method and the second method can be performed manually (e.g., by a clinician) or programmatically (e.g., by a computer system). The "predetermined significance threshold" can be associated with (e.g., set based on, associated with) at least one of the following: a change in the shape, size, or position of a target to which dose delivery is desired; and a change in the shape, size, or position of healthy tissue (e.g., OAR) proximal to the target. Depending on the relevant clinical specialty, any suitable quality metric data can be used to evaluate the distance or error map between the treatment image data 610 and the planning image data 620, such as target size, shift in tumor location (e.g., the location of voxels associated with the target in a 3D map), the distance from the target to the OAR (e.g., the distance to the body surface or centroid), dose measurement values ​​in the target and OAR when the original field settings are used in a new situation, etc.

[0087] It should be understood that Figures 6 to 8 The examples in apply to image data acquired using any one or more suitable imaging modalities (i.e., not limited to CT and CBCT image data). For example, treatment image data 610 may be in the form of CBCT image data, and planning image data 620 may be in the form of CT image data, ultrasound image data, MRI image data, PET image data, SPECT, or camera image data (e.g., using a time-of-flight camera to capture the patient's body contours), etc. In practice, there are typically limited options for acquiring treatment image data 610 given the limited capabilities of treatment delivery machines. For example, CBCT image data acquired during treatment has relatively poor image quality compared to CT image data acquired for treatment planning purposes. The area of ​​a patient's anatomy scanned by CBCT is typically smaller than that of CT, and therefore some structures may not be visible at all in the CBCT image data.

[0088] In the case where the treatment image data 610 = CT image data (e.g., associated with one energy level), the planning image data 620 may be in the form of CT image data, ultrasound image data, MRI image data, PET image data, SPECT image data, or camera image data associated with another energy level. In the case where the treatment image data 610 = MRI image data, the planning image data 620 may be in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, PET image data, SPECT image data, or camera image data. In the case where the treatment image data 610 = ultrasound image data, the planning image data 620 may be in the form of CT image data, CBCT image data, PET image data, MRI image data, SPECT image data, or camera image data. In the case where the treatment image data 610 = PET image data, the planning image data 620 may be in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, SPECT image data, or camera image data. Alternative and / or additional image data associated with any one or more suitable imaging modalities may be used.

[0089] In addition, it should be understood that Figures 6 to 8 The deep learning engine 650 / 680 in the example can be implemented using any suitable deep learning technology. Figures 1 to 5 A deep learning architecture with multiple processing paths in will be used as an example. Depending on the desired implementation, any alternative and / or additional deep learning models (i.e., single or multiple processing paths) may be used. Examples of the present disclosure may be implemented to improve the efficiency of ART, which may improve customer satisfaction, increase the number of patients that can be treated, and maintain a set level of plan quality for patients. In the following, CBCT image data will be used as an example of "treatment image data associated with a first imaging modality", and CT images will be used as an example of "treatment image data associated with a second imaging modality".

[0090] (a) First method (difference > significance threshold)

[0091] exist Figure 6 In the example of FIG. 6 , in response to determining that an update is needed to a treatment plan generated based on the planning image data 620, and the difference between the treatment image data 610 and the planning image data 620 exceeds a predetermined significance threshold, the first example method can be performed. Due to the presence of significant deviations, in order to avoid any adverse effects on treatment delivery, it is not necessary to rely on the planning image data 620 during ART.

[0092] In particular, Figure 6At 640 in , the treatment image data 610 (eg, CBCT image data) may be transformed to produce transformed image data (eg, synthetic CT image data) associated with a second imaging modality. Figure 6 At 650 in the figure, the transformed image data 640 may be processed using a deep learning engine to generate any suitable output data to update the treatment plan 603. The output data may be patient structure data associated with the patient (e.g., identifying one or more targets and / or OARs, etc.), dose data associated with the patient (e.g., dose distribution for one or more targets and / or OARs), treatment delivery data associated with the treatment delivery system (e.g., beam orientation and / or trajectory, machine control point data, fluence map, etc.), or any combination thereof.

[0093] according to Figure 6 An example implementation of the first method of blocks 640-660 in would use Figure 7 To explain, it is shown Figure 7 Used according to Figure 6 Schematic diagram of a first example method 700 for performing ART planning in an example of ART planning in . The example process 700 may include one or more operations, functions, or actions illustrated by one or more blocks. Various blocks may be combined into fewer blocks, divided into additional blocks, and / or deleted based on the desired implementation. The example process 700 may be implemented using any suitable computer system, and one example will use Fig.10 To discuss.

[0094] During the training phase 701, the deep learning engine 650 may be trained using any suitable training data to produce output data 660, such as training CT image data (see 731) and corresponding output data. In the case of automatic segmentation, the trained structural data 732 of the CT image data 731 may be used. Optionally (for simplicity in Figure 7 ), the deep learning engine 650 may be trained to perform automatic segmentation to generate structural data 732, to perform dose prediction to generate dose data, to perform treatment delivery data estimation to generate treatment delivery data, or any combination thereof. Figure 2 and Figure 3 In the example of FIG. 6 , the deep learning engine 650 may include multiple processing paths to process image data (I1, I2, I3) at different resolution levels (R1, R2, R3). Figures 1 to 5 The examples discussed will also apply here and will not be repeated for the sake of brevity.Using the deep learning engine 650 instead of manual methods can improve the efficiency of ART.

[0095] exist Figure 7In the example of , the transformed image data 640 can be generated using a deep learning engine 720 that is trained to map image data associated with one imaging modality (e.g., CBCT) to another imaging modality (e.g., CT). Figure 7 In the example of FIG. 7 , the deep learning engine 720 can be trained to map the CBCT image data to the CT image data. In one example, the deep learning engine 720 can be trained using training data including CT image data and corresponding structural data (see 711 ), and CBCT image data and corresponding structural data (see 712 ). Figures 1 to 5 The examples discussed will also apply here and will not be repeated for the sake of brevity.Alternatively, instead of the deep learning engine 720, an algorithmic approach may be used, such as a rigid or deformable registration algorithm or the like.

[0096] During the inference phase 702, the treatment planning data 610 (e.g., CBCT image data) may be processed using a trained deep learning engine 720 to produce transformed image data 640 (e.g., synthetic CT image data). The transformed image data 640 may then be processed using a deep learning engine 650 to produce output data 660. Figure 7 In the example in FIG. 6 , the deep learning engine 650 can be trained to perform automatic segmentation to produce output = structural data that identifies targets and OARs associated with the patient. Optionally, the deep learning engine 650 can be trained to perform dose prediction, projection data estimation, etc. The output data 660 can then be used to update the treatment plan 603 to reflect changes in the patient's condition, thereby improving the quality of treatment delivery. Figure 6 Treatment is delivered based on the improved treatment plan 604 in.

[0097] (b) Second method (difference ≤ significance threshold)

[0098] Reference again Figure 6 In response to determining that an update is needed to the treatment plan generated based on the planning image data 620, and the difference between the treatment image data 610 and the planning image data 620 does not exceed a predetermined significance threshold, a second example method may be performed. Since the difference is not very significant, ART benefits from two sets of image data, namely both the treatment image data 610 and the planning image data 620. This is because, for example, CT image data may include additional data because the CT image data does not suffer from severe artifacts compared to CBCT image data.

[0099] In more detail, Figure 6At 680 and 690 in FIG. 6 , a deep learning engine may be used to process the treatment image data 610 and the planning image data 620 to generate any suitable output data to update the treatment plan 603. Similarly, the output data 690 may include patient structure data associated with the patient (e.g., identifying one or more targets and OARs, etc.), dose data associated with the patient (e.g., dose distributions for one or more targets and OARs), treatment delivery data associated with the treatment delivery system (e.g., beam orientation and / or trajectory, machine control point data, fluence map data, etc.), or any combination thereof.

[0100] Prior to processing using the deep learning engine 680, the treatment image data 610 may be transformed, such as by performing image registration to register the treatment image data 610 relative to the planning image data 620, to produce transformed image data (see 670). Any suitable method for image registration may be used, such as an algorithmic method, a machine learning method, a deep learning method, etc. Image registration may be performed to obtain a correspondence between the treatment image data 610 and the planning image data 620.

[0101] For example, after the CBCT image data has been transformed to match the CT image data, they can be fed into the deep learning engine 680 to produce output data 690. In practice, any suitable method, such as a deep learning method, algorithm, etc., can be used to perform image registration. An example deep learning method for image registration is disclosed in a paper entitled "Quicksilver: Fast Predictive Image Registration-a Deep Learning Approach" (2017), authored by Xiao, Y., Kwitt, R., Styner, M., Niethammer, M. and published in NeuroImage (Volume 158, 2017, pages 378 to 396). This method can be implemented to perform deformable image registration using a piece-by-piece prediction of a deformation model based on image appearance. A deep encoder-decoder network can be used as a prediction model.

[0102] It should be noted that the transformed image data 670 in the second method is generated based on both the treatment image data 610 and the planning image data 620 (i.e., two inputs, such as CT and CBCT image data). This should be contrasted with the first method, in which the transformed image data 640 is generated based on one input = treatment image data 610 (e.g., CBCT image data of the day). Both methods can rely on image registration with respect to the transformation.

[0103] according to Figure 6An example implementation of the second method of blocks 670-690 in the example would use Figure 8 To explain, it is shown Figure 8 Used according to Figure 6 Schematic diagram of a second example method 800 for performing ART planning in an example of ART planning in . The example process 800 may include one or more operations, functions, or actions illustrated by one or more blocks. Various blocks may be combined into fewer blocks, divided into additional blocks, and / or deleted based on the desired implementation. The example process 800 may be implemented using any suitable computer system, and one example will use Fig.10 To discuss.

[0104] During the training phase 801, the deep learning engine 680 may be trained using any suitable training data to produce output data 690, such as training CT images and structure data 811, and training CBCT images and structure data 812. The goal is to train the deep learning engine 680 to generate output data 690 based on the use of different imaging modalities, such as Figure 8 The output data (for example, structural data in the case of automatic segmentation) is generated by using two sets of image data acquired by CT and CBCT in the present invention.

[0105] The deep learning engine 680 may be implemented using any suitable deep learning model. Figures 1 to 5 In the example of FIG. 6 , the deep learning engine 680 may include multiple processing paths to process two sets of CT and CBCT image data (I1, I2, I3) at different resolution levels (R1, R2, R3). Compared with using a single set of image data, the input image data of the deep learning engine 680 may have two values ​​representing the CT and CBCT image data, respectively. The convolutional layers in the deep learning engine 680 may be configured to transform the input image data into more or less abstract features by combining data from all modalities. Using Figures 1 to 5 The additional implementation details discussed will also apply here and will not be repeated for the sake of brevity.

[0106] During the inference phase 802, the trained deep learning engine 680 processes two sets of image data, namely the planning image data 620 and the transformed image data 670 generated using image registration or the like. In the case of automatic segmentation, the output data 690 may include structural data identifying targets and OARs associated with the patient. Optionally, the deep learning engine 680 may be trained to perform dose prediction, treatment delivery data estimation, etc. The output data 690 may then be used to update the treatment plan 603 to reflect changes in the patient's condition, thereby improving the quality of treatment delivery. The output data 690 may then be used to update the treatment plan 603 based on the patient's condition. Figure 6 Treatment is delivered based on the improved treatment plan 604 in.

[0107] Using multiple sets of image data acquired using different imaging modalities can produce improved output data (e.g., better quality contours) than having only one set of image data. Compared to the first approach, two different imaging techniques generally provide more information than one imaging technique. For example, a time-of-flight camera system provides information related to a large area of ​​the patient's body surface without providing information inside the patient, while CBCT provides information inside the patient, but within a limited field of view, the time-of-flight camera system can capture motion, while CBCT cannot capture motion. The two sets of image data can be interpreted by deep neural network technology to provide information in an agreed format (e.g., CT images, CT images and segmentation, segmentation, 3D density map, 3d density map in motion, segmentation in motion, etc.).

[0108] Sample Treatment Plan

[0109] Fig. 9 is based on Figures 1 to 8 Schematic diagram of an example treatment plan 156 / 900 generated or modified based on output data in the example of FIG. The treatment plan 156 may be delivered using any suitable treatment delivery system including a radiation source 910 for projecting radiation beams 920 at various beam angles 930 onto a treatment volume 960 representing the patient's anatomical structure. Although not shown for simplicity, Fig. 9 156, but the radiation source 910 may include a linear accelerator for accelerating the radiation beam 920 and a collimator (e.g., MLC) for modifying or adjusting the radiation beam 920. In another example, the radiation beam 920 may be conditioned by scanning it across a target patient in a specific pattern with various energies and dwell times (e.g., as in proton therapy). A controller (e.g., a computer system) may be used to control the operation of the radiation source 920 according to the treatment plan 156.

[0110] During treatment delivery, the radiation source 910 can be rotated around the patient using a gantry, or the patient can be rotated (as in some proton radiotherapy solutions) to emit radiation beams 920 at various beam orientations or beam angles relative to the patient. For example, five equally spaced beam angles 930A to 930E (also labeled "930A," "930B," "930C," "930D," and "930E") can be selected using a deep learning engine configured to perform treatment delivery data estimation. In practice, any suitable number of beam angles and / or table or chair angles 930 can be selected (e.g., five, seven, etc.). At each beam angle, the radiation beam 920 is associated with a fluence plane 940 (also called an intersection plane) located outside the patient envelope along a beam axis extending from the radiation source 910 to the treatment volume 960. As Fig. 9As shown, the fluence plane 940 is generally at a known distance relative to the isocenter.

[0111] During radiation therapy planning, treatment plans 156 / 900 can be based on the use of Figures 1 to 5 The output data 260 / 492 / 592 generated by the deep learning engine 200 / 400 / 500 in the example is generated. During ART planning, the treatment plan 156 / 900 can be based on the use of Figures 6 to 8 The output data 660 / 690 produced by the deep learning engine 650 / 680 in the example is improved.

[0112] Computer Systems

[0113] The above examples may be implemented by hardware, software or firmware, or a combination thereof. Fig.10 1 is a schematic diagram of an example computer system 1000 for radiation therapy planning and / or ART planning. In this example, a computer system 1005 (also referred to as a treatment planning system) may include a processor 1010, a computer readable storage medium 1020, an interface 1040 for interacting with a radiation therapy delivery system 160, and a bus 1030 that facilitates communication between these illustrated components and other components.

[0114] The processor 1010 is used to execute the Figures 1 to 9 The computer-readable storage medium 1020 may store any suitable information 1022, such as information related to training data, deep learning engines, image data, output data, etc. The computer-readable storage medium 1020 may further store computer-readable instructions 1024, which in response to being executed by the processor 1010 cause the processor 1010 to perform the processes described herein. Figure 1 The illustrated treatment planning system 160 is delivered according to the treatment plan 156, a description of which will not be repeated here for the sake of brevity.

[0115] The foregoing detailed description has been described various embodiments of the device and / or process using block diagrams, flow charts and / or examples. To the extent that such block diagrams, flow charts and / or examples include one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation within such block diagrams, flow charts or examples can be implemented individually and / or collectively by a wide range of hardware, software, firmware or substantially any combination thereof. Throughout this disclosure, the terms "first", "second", "third", etc. do not represent any order of importance, but are used to distinguish between individual elements.

[0116] Those skilled in the art will recognize that some aspects of the embodiments disclosed herein may be equivalently implemented in whole or in part in an integrated circuit as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as substantially any combination thereof, and that designing circuits and / or writing code for software and / or firmware will be well within the capabilities of those skilled in the art in view of this disclosure.

[0117] Although the present disclosure has been described with reference to specific exemplary embodiments, it will be appreciated that the present disclosure is not limited to the described embodiments, but can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense.

Claims

1. A non-transitory computer-readable storage medium comprising a set of instructions that, in response to being executed by a processor of a computer system, causes the processor to perform a method for adaptive radiation therapy planning using a deep learning engine, wherein the method comprises: obtaining therapy image data associated with a first imaging modality, wherein the therapy image data is acquired during a therapy session of a patient; obtaining planning image data associated with a second imaging modality, wherein the planning image data is acquired prior to the treatment session to generate a treatment plan for the patient; determining, based on a comparison between the treatment image data and the planning image data, that a difference between the treatment image data and the planning image data does not exceed a predetermined significance threshold associated with at least one of: a change in shape, size, or position of a target requiring dose delivery; and a change in shape, size, or position of healthy tissue proximate to the target; as well as In response to determining that the difference does not exceed the predetermined significance threshold, processing the treatment image data and the planning image data using the deep learning engine to generate output data to update the treatment plan.

2. The non-transitory computer-readable storage medium of claim 1, wherein the method further comprises: Prior to processing the treatment image data and the planning image data using the deep learning engine, image registration is performed by registering the treatment image data relative to the planning image data to produce transformed image data.

3. The non-transitory computer readable storage medium of claim 2, wherein processing the treatment image data and the planning image data comprises: The transformed image data and the planning image data are processed using the deep learning engine to generate the output data.

4. The non-transitory computer readable storage medium of claim 1 , wherein processing the treatment image data and the planning image data comprises: processing the treatment image data and the planning image data using a first processing path of the deep learning engine to generate first feature data associated with a first resolution level; as well as processing the treatment image data and the planning image data using a second processing path of the deep learning engine to generate second feature data associated with a second resolution level; processing the treatment image data and the planning image data using a third processing path of the deep learning engine to generate third feature data associated with a third resolution level; as well as The output data is generated based on the first feature data, the second feature data, and the third feature data.

5. The non-transitory computer-readable storage medium of claim 1 , wherein obtaining the treatment image data and the planning image data comprises one of: obtaining the treatment image data in the form of cone beam computed tomography (CBCT) image data and the planning image data in the form of computed tomography (CT) image data, ultrasound image data, magnetic resonance imaging (MRI) image data, positron emission tomography (PET) image data, single photon emission computed tomography (SPECT) or camera image data; obtaining the treatment image data in the form of CT image data and the planning image data in the form of CT image data, ultrasound image data, MRI image data, PET image data, SPECT image data or camera image data associated with different energy levels; obtaining the treatment image data in the form of MRI image data and the planning image data in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, PET image data, SPECT image data or camera image data; obtaining the treatment image data in the form of ultrasound image data and the planning image data in the form of CT image data, CBCT image data, PET image data, MRI image data, SPECT image data or camera image data; and The treatment image data are obtained in the form of PET image data, and the planning image data are obtained in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, SPECT image data or camera image data.

6. The non-transitory computer-readable storage medium of claim 1, wherein the method further comprises: The deep learning engine is trained using training data associated with past patients to perform one of: automatic segmentation for generating the output data in the form of structural data associated with the patient; dose prediction for generating the output data in the form of dose data associated with the patient; and a therapy delivery data estimate for generating said output data in the form of therapy delivery data for a therapy delivery system.

7. A computer system configured to perform a method of adaptive radiation therapy planning using a deep learning engine, the computer system comprising a processor and a non-transitory computer readable medium having instructions stored thereon, the instructions, when executed by the processor, causing the processor to: obtaining therapy image data associated with a first imaging modality, wherein the therapy image data is acquired during a therapy session of a patient; obtaining planning image data associated with a second imaging modality, wherein the planning image data is acquired prior to the treatment session to generate a treatment plan for the patient; determining, based on a comparison between the treatment image data and the planning image data, that a difference between the treatment image data and the planning image data does not exceed a predetermined significance threshold associated with at least one of: a change in shape, size, or position of a target requiring dose delivery; and a change in shape, size, or position of healthy tissue proximate to the target; as well as In response to determining that the difference does not exceed the predetermined significance threshold, processing the treatment image data and the planning image data using the deep learning engine to generate output data to update the treatment plan.

8. The computer system of claim 7, wherein the instructions further cause the processor to: Prior to processing the treatment image data and the planning image data using the deep learning engine, image registration is performed by registering the treatment image data relative to the planning image data to produce transformed image data.

9. The computer system of claim 8, wherein the instructions for processing the treatment image data and the planning image data cause the processor to: The transformed image data and the planning image data are processed using the deep learning engine to generate the output data.

10. The computer system of claim 7, wherein the instructions for processing the treatment image data and the planning image data cause the processor to: processing the treatment image data and the planning image data using a first processing path of the deep learning engine to generate first feature data associated with a first resolution level; as well as processing the treatment image data and the planning image data using a second processing path of the deep learning engine to generate second feature data associated with a second resolution level; processing the treatment image data and the planning image data using a third processing path of the deep learning engine to generate third feature data associated with a third resolution level; as well as The output data is generated based on the first feature data, the second feature data, and the third feature data.

11. The computer system of claim 7, wherein the instructions for obtaining the treatment image data and the planning image data cause the processor to perform one of the following: obtaining the treatment image data in the form of cone beam computed tomography (CBCT) image data and the planning image data in the form of computed tomography (CT) image data, ultrasound image data, magnetic resonance imaging (MRI) image data, positron emission tomography (PET) image data or camera image data; obtaining the treatment image data in the form of CT image data and the planning image data in the form of CT image data, ultrasound image data, MRI image data, PET image data or camera image data associated with different energy levels; obtaining the treatment image data in the form of MRI image data and the planning image data in the form of CT image data, CBCT image data, ultrasound image data, MRI image data, PET image data or camera image data; obtaining the treatment image data in the form of ultrasound image data and the planning image data in the form of CT image data, CBCT image data, PET image data, MRI image data or camera image data; and The treatment image data are obtained in the form of PET image data, and the planning image data are obtained in the form of CT image data, CBCT image data, ultrasound image data, MRI image data or camera image data.

12. The computer system of claim 7, wherein the instructions further cause the processor to: The deep learning engine is trained using training data associated with past patients to perform one of: automatic segmentation, which is used to generate the output data in the form of structural data associated with the patient; dose prediction, which is used to generate the output data in the form of dose data associated with the patient; and treatment delivery data estimation, which is used to generate the output data in the form of treatment delivery data for a treatment delivery system.

Images