Methods and systems for radiation therapy treatment planning based on deep transfer learning

Through deep transfer learning technology, the automated generation and optimization of radiation therapy treatment plans are solved, and the problems of time-consuming and uncertainty of radiation therapy treatment plans in the prior art are improved, and treatment efficiency and safety are improved.

CN113272909BActive Publication Date: 2024-11-29VARIAN MEDICAL SYST INT AG
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN201980084493.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-05-30
Publication Date
2024-11-29
Estimated Expiration
2039-05-30

AI Technical Summary

Technical Problem

Existing radiotherapy treatment programs are often time-consuming and labor-intensive, making it difficult to effectively distinguish between tumors and proximal health structures, resulting in uncertainty in radiation dose and inefficient treatment.

Method used

Deep transfer learning methods and systems are adopted to generate target deep learning engines through pre-trained basic deep learning engines, and the generation and optimization of radiotherapy processing plans are automated, including automatic segmentation and dose prediction, reducing dependence on clinicians and manual operations.

Benefits of technology

It improves the efficiency and accuracy of the radiotherapy treatment plan, reduces the requirements for clinician technical knowledge, reduces the computational cost and time of training deep learning engines, and improves the probability of tumor control and the radiation safety of healthy structures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113272909B_ABST
    Figure CN113272909B_ABST
Patent Text Reader

Abstract

Example methods and systems for deep transfer learning for radiotherapy treatment planning are provided. An example method may include: obtaining (310) a base deep learning engine that is pre-trained to perform a base radiotherapy treatment planning task; and generating a target deep learning engine based on the base deep learning engine to perform a target radiotherapy treatment planning task. The target deep learning engine may be generated by configuring (330) a variable base layer among multiple base layers of the base deep learning engine, and generating (340) one of multiple target layers of the target deep learning engine by modifying the variable base layer. Alternatively or additionally, the target deep learning engine may be generated by configuring (350) an invariant base layer among multiple base layers, and generating (360) one of multiple target layers of the target deep learning engine based on feature data generated using the invariant base layer.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 783,217 (Attorney Docket No. 2018-024US01), filed December 21, 2018. This application is related to U.S. Patent Application No. 16 / 145,461 (Attorney Docket No. 2018-012US01). The above application (including any appendices or attachments) is incorporated herein by reference in its entirety. Background Art

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0004] Radiotherapy is an important part of treatment used to reduce or eliminate unwanted tumors in patients. Unfortunately, the applied radiation cannot inherently distinguish between unwanted tumors and any proximal healthy structures (such as organs, etc.). This requires careful management to limit the radiation to the tumor (i.e., the target). Ideally, the goal is to provide a lethal or therapeutic radiation dose to the tumor while maintaining an acceptable dose level to proximal healthy structures. However, to achieve this goal, conventional radiotherapy treatment planning can be time consuming and labor intensive. Summary of the invention

[0005] According to an example of the present disclosure, a method and system for deep transfer learning of radiotherapy treatment planning are provided. An example method may include: obtaining a base deep learning engine that is pre-trained to perform a basic radiotherapy treatment planning task; and based on the base deep learning engine, generating a target deep learning engine to perform a target radiotherapy treatment planning task. The target deep learning engine may be generated by configuring a variable base layer among multiple base layers of the base deep learning engine, and generating one of the multiple target layers of the target deep learning engine by modifying the variable base layer. Alternatively or additionally, the target deep learning engine may be generated by configuring an invariant base layer among multiple base layers, and generating one of the multiple target layers of the target deep learning engine based on feature data generated using the invariant base layer. Various examples will be discussed below, such as references to Figures 1 to 9 and Fig.16 .

[0006] According to an example of the present disclosure, a method and system for radiotherapy treatment planning based on deep transfer learning are provided. An example method may include: providing a first user interface, the first user interface being used to send a request to generate a target deep learning engine to perform a target radiotherapy treatment planning task. In response to detecting the request to generate a target deep learning engine via the first user interface, a basic deep learning engine may be selected from a plurality of basic deep learning engines that are pre-trained to perform corresponding basic radiotherapy treatment planning tasks. Based on the selected basic deep learning engine and target training data associated with a plurality of past patients, a target deep learning engine may be generated to perform the target radiotherapy treatment planning task. The method may also include: providing a second user interface, the second user interface being used to access the target deep learning engine to perform the target radiotherapy treatment planning task for an individual patient. Various examples are discussed below, such as references to Figures 10 to 15B and Fig.16 . BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram illustrating an example process flow for radiation therapy treatment;

[0008] Figure 2 is a schematic diagram illustrating example deep transfer learning for radiation therapy treatment planning;

[0009] Figure 3 is a flow chart of an example process for a computer system to perform deep transfer learning for radiation therapy treatment planning;

[0010] Figure 4 It shows that according to Figure 3 A schematic diagram of an exemplary deep transfer learning of an example;

[0011] Figure 5 is a schematic diagram illustrating a first example deep transfer learning for automatic segmentation of image data;

[0012] Figure 6 is a schematic diagram illustrating a second example deep transfer learning for automatic segmentation of image data;

[0013] Figure 7 is a schematic diagram illustrating an example deep transfer learning for dose prediction;

[0014] Figure 8 It shows that according to Figure 5 , Figure 6 and Figure 7 A schematic diagram of an example inference stage of a target engine generated by an example of

[0015] Fig. 9 is based on Figure 8Schematic diagram of an example processing plan for generating or improving output data in an example;

[0016] Fig.10 is a schematic diagram illustrating an example network environment in which deep transfer learning for radiation therapy treatment planning may be implemented;

[0017] Fig.11 is used Fig.10 A flowchart of an example process for deep transfer learning for radiation therapy treatment planning;

[0018] Fig. 12A is a schematic diagram illustrating a first example interface for target engine generation during a training phase;

[0019] Fig. 12B is a schematic diagram illustrating a second example interface for target engine generation during a training phase;

[0020] Fig.13A is a schematic diagram illustrating a third example interface for target engine generation during a training phase;

[0021] Fig. 13B is a schematic diagram illustrating a fourth example interface for target engine generation during a training phase;

[0022] Fig.14A is a schematic diagram illustrating a first example interface for target engine generation during a verification phase;

[0023] Fig. 14B is a schematic diagram illustrating a second example interface for target engine generation during a verification phase;

[0024] Fig.15A is a schematic diagram showing an example planning interface for automatic segmentation during an inference phase;

[0025] Fig. 15B is a schematic diagram showing an example planning interface for dose prediction during the inference phase; and

[0026] Fig.16 is a schematic diagram of an example computer system that performs deep transfer learning for radiation therapy treatment planning. DETAILED DESCRIPTION

[0027] The technical details set forth in the following description enable those skilled in the art to implement one or more embodiments of the present disclosure.

[0028] Figure 11 is a schematic diagram illustrating an example process flow 100 for radiation therapy treatment. The example process flow 100 may include one or more operations, functions, or actions illustrated by one or more blocks. Various blocks may be combined into fewer blocks, divided into additional blocks, and / or eliminated based on the desired implementation. Figure 1 In the example of , a radiation therapy treatment typically includes various stages, such as an imaging system (see 110) that performs image data acquisition for a patient, a radiation therapy treatment planning system (see 130) that generates an appropriate treatment plan for the patient (see 156), and a treatment delivery system (see 160) that performs treatment delivery according to the treatment plan.

[0029] In more detail, Figure 1 At 110, an imaging system may be used to perform image data acquisition to capture image data 120 associated with a patient, particularly the patient's anatomical structure. Any suitable medical image modality may be used, such as computed tomography (CT), cone beam computed tomography (CBCT), positron emission tomography (PET), magnetic resonance imaging (MRI), single photon emission computed tomography (SPECT), any combination thereof, etc. For example, when CT or MRI is used, the image data 120 may include a series of two-dimensional (2D) images or slices, each of which represents a cross-sectional view of the patient's anatomical structure, or may include a volumetric or three-dimensional (3D) image of the patient, or may include a time series of 2D or 3D images of the patient (e.g., four-dimensional (4D) CT or 4D CBCT).

[0030] exist Figure 1 At 130 in the figure, radiation therapy treatment planning can be performed during the planning stage to generate a treatment plan 156 based on the image data 120. Any suitable number of treatment planning tasks or steps can be performed, such as segmentation, dose prediction, projection data prediction, treatment plan generation, etc. For example, segmentation can be performed to generate structural data 140 identifying various segments or structures from the image data 120. In practice, a three-dimensional (3D) volume of the patient's anatomy can be reconstructed from the image data 120. The 3D volume to be subjected to radiation is called a treatment or radiation volume, which can be divided into a plurality of smaller volume pixels (voxels) 142. Each voxel 142 represents a 3D element associated with a location (i, j, k) within the treatment volume. The structural data 140 may include any suitable data related to the contour, shape, size, and location of the patient's anatomy 144, a target 146, an organ at risk (OAR) 148, or any other structure of interest (e.g., tissue, bone). For example, using image segmentation, a line can be drawn around a portion of the image and marked as a target 146 (e.g., marked with a label = "prostate"). Everything inside the line will be considered as target 146, while everything outside the line will not.

[0031] In another example, dose prediction may be performed to generate dose data 150 that specifies a radiation dose (denoted at 152 as “D TAR ”) and the radiation dose to the OAR 148 (denoted at 154 as “D OAR In practice, target 146 may represent a malignant tumor (e.g., a prostate tumor, etc.) that requires radiotherapy treatment, and OAR 148 represents proximal healthy structures or non-target structures (e.g., rectum, bladder, etc.) that may be adversely affected by the treatment. Target 146 is also referred to as a planning target volume (PTV). Although Figure 1 146 and OARs 148 of complex shapes and sizes. In addition, although shown as having a regular shape (e.g., a cube), the voxel 142 may have any suitable shape (e.g., irregular). Depending on the desired implementation, the radiation therapy treatment plan at box 130 may be performed based on any additional and / or alternative data (such as prescription, disease stage, biological or radiation data, genetic data, laboratory data, biopsy data, past treatment or medical history, any combination thereof, etc.).

[0032] Based on the structural data 140 and the dose data 150, a treatment plan 156 can be generated, including 2D fluence map data for a set of beam directions or angles. Each fluence map specifies the intensity and shape of the radiation beam emitted from the radiation source at a specific beam direction and a specific time (e.g., determined by a multi-leaf collimator (MLC)). For example, in practice, an intensity modulated radiation therapy treatment (IMRT) or any other (multiple) treatment techniques may involve changing the shape and intensity of the radiation beam at a constant gantry and couch angle. Alternatively or additionally, the treatment plan 156 may include machine control point data (e.g., mandibular and leaf positions), volumetric modulated arc therapy (VMAT) trajectory data for controlling a treatment delivery system, and the like. In practice, frame 130 may be performed based on target dosing specified by a clinician (e.g., an oncologist, radiation dosimeter, planner, etc.), such as based on the clinician's experience, the type and extent of the tumor, the patient's geometry and condition, and the like.

[0033] exist Figure 1At 160 in the figure, treatment delivery is performed during a treatment phase to deliver radiation to a patient according to the treatment plan 156. For example, the radiation treatment delivery system 160 may include a rotatable gantry 164 to which a radiation source 166 is attached. During treatment delivery, the rotatable gantry 164 is rotated around a patient 170 supported on a structure 172 (e.g., a table) to emit radiation beams 168 in various beam directions according to the treatment plan 156. The controller 162 may be used to retrieve the treatment plan 156 and the rotatable gantry 164, the radiation source 166, and the radiation beams 168 to deliver the radiation therapy treatment according to the treatment plan 156.

[0034] It should be understood that any suitable (multiple) radiation treatment delivery system may be used, such as a robotic-based system, a tomotherapy-type system, brachytherapy, a sirex sphere, any combination thereof, etc. Additionally, examples of the present disclosure may be applicable to particle delivery systems (e.g., protons, carbon ions, etc.). Such systems may employ scattered particle beams that are subsequently shaped by a device similar to the MLC, or scanning beams with adjustable energy, spot size, and dwell time. In addition, OAR segmentation may be performed, and automatic segmentation of the treatment probe may be desired.

[0035] Traditionally, Figure 1 The radiotherapy treatment plan at the box 130 in is time and labor intensive. For example, a team of highly skilled and well-trained oncologists and dosimetrists is usually required to manually delineate the structures of interest by drawing contours or segmentations on the image data 120. These structures are manually checked by the physician and may need to be adjusted or redrawn. In many cases, the segmentation of critical organs can be the most time-consuming part of the radiotherapy treatment plan. After reaching a consensus on the structure, additional labor-intensive steps are required to process the structure to generate a clinically optimal treatment plan that specifies treatment delivery data, such as beam direction and trajectory and corresponding 2D fluence maps. These steps are often complicated by the lack of consensus between different physicians and / or clinical areas on what constitutes a "good" contour or segmentation. In practice, there can be significant differences in the way different clinical experts draw structures or segments. This difference can lead to uncertainty in the size and shape of the target volume and uncertainty in the exact proximity, size and shape of the OAR that should receive the minimum radiation dose. Even for a specific expert, the way the segmentation is drawn on different days may also be different.

[0036] According to examples of the present disclosure, artificial intelligence (AI) technology can be used to improve various challenges associated with radiotherapy treatment planning. In particular, (multiple) deep learning engines can be used to automate (multiple) radiotherapy treatment planning steps. Throughout the present disclosure, the term "deep learning" may generally refer to a class of methods that utilize many layers or stages of nonlinear data processing for feature learning and pattern analysis and / or classification. A "deep learning model" may refer to a hierarchy of "layers" of nonlinear data processing, including an input layer, an output layer, and multiple (i.e., two or more) "hidden" layers between the input layer and the output layer. These layers can be trained end-to-end (e.g., from the input layer to the output layer) to extract (multiple) features from the input and classify (multiple) features to produce outputs (e.g., classification labels or categories).

[0037] Therefore, the term "deep learning engine" may refer to any suitable hardware and / or (multiple) software components of a computer system capable of executing an algorithm according to any suitable (multiple) deep learning model. Depending on the desired implementation, any suitable (multiple) deep learning model may be used, such as a convolutional neural network, a recursive neural network, a deep belief network, a generative adversarial network (GAN), or any combination thereof. In practice, a neural network is typically formed using a network of processing elements (referred to as "neurons", "nodes", etc.) that are interconnected via connections (referred to as "synapses", "weights", etc.). For example, a convolutional neural network may be implemented using any suitable (multiple) architectures, such as U-net, LeNet, AlexNet, ResNet, V-net, DenseNet, etc. In this case, a "layer" of a convolutional neural network may be a convolutional layer, a pooling layer, a rectified linear unit (ReLU) layer, a fully connected layer, a loss layer, an activation layer, a loss layer, a transposed convolution layer, a cascade layer, or any combination thereof. In practice, the U-net architecture includes a contraction path (left side) and an expansion path (right side). The contracting path consists of repeated application of convolutions followed by ReLU layers and max pooling layers. Each step in the expanding path can consist of upsampling or transposed convolution of the feature map followed by convolution, etc.

[0038] Deep learning methods should be contrasted with machine learning methods, which have been applied, for example, to automatic segmentation. In general, these methods involve extracting (hand-crafted) feature vectors from the image, such as for each voxel. These feature vectors can then be used as input to a machine learning model, which classifies which class each voxel belongs to. However, such machine learning methods typically do not use the complete image data and may require additional constraints. Another challenge is that these methods rely on high-dimensional, hand-crafted features in order to accurately predict the class label for each voxel. Solving high-dimensional classification problems is computationally intensive and requires large amounts of memory. Some methods use lower-dimensional features (e.g., using dimensionality reduction techniques), but they reduce prediction accuracy.

[0039] Traditionally, there are many challenges associated with training a deep learning engine for a radiotherapy treatment plan. For example, if the training process is not standardized and / or a large amount of organized training data is not collected, it may be difficult to achieve the desired training results for the deep learning engine. This can lead to suboptimal results, or worse, the inability to create any working deep learning engine. In another example, if an inexperienced user (e.g., a clinician) is provided with a tool to train their own deep learning engine, there are challenges in training from scratch. For example, with respect to automatic segmentation, the computational cost of training a blank deep learning engine can be high, especially when a large number of anatomical structures need to be modeled.

[0040] While conceptually simple, training a deep learning engine typically requires significant technical expertise related to model architecture(s), optimization, convergence analysis, regularization, etc. A deep learning engine starting from a blank slate can deviate to suboptimal forms when trained with limited amounts of data. The complexity of this training process can discourage users from training and using a deep learning engine for radiation therapy treatment planning, which is undesirable.

[0041] Deep Transfer Learning

[0042] According to examples of the present disclosure, deep transfer learning can be performed to improve the training process of a deep learning engine for radiation therapy treatment planning. Instead of starting from scratch using a blank state with random weights, a pre-trained "base" deep learning engine can be used as a starting point to improve the training phase of a "target" deep learning engine, especially where resources in terms of computation and personnel are reduced. The target deep learning engine can be adapted from the base learning engine according to the needs and specifications of any suitable user.

[0043] As used herein, the term "deep transfer learning" may generally refer to a technique(s) in which a deep learning engine is adapted or redesigned (in whole or in part) as a starting point for another deep learning engine. In practice, deep transfer learning represents an optimization strategy that helps to speed up progress or improve performance during the training process. Improving the efficiency of radiation therapy treatment plans using deep transfer learning may also improve patient outcomes, such as increasing the probability of tumor control and / or reducing the likelihood of health complications or death in healthy structures due to excessive radiation.

[0044] In more detail, Figure 2 2 is a schematic diagram illustrating an example deep transfer learning 200 for radiation therapy treatment planning. In particular, deep transfer learning can be performed from a base deep learning engine (see 210) to a target deep learning engine (see 220, 230, and 240). The base deep learning engine 210 can represent a "pre-trained model" generated or trained based on base training data 203 during a pre-training phase 201 (e.g., performed at a provider's site). The target deep learning engines 220 / 230 / 240 can be generated or trained based on the base deep learning engine 210 and target training data available during a subsequent training phase 202 (e.g., performed at a user's site).

[0045] Embodiments of the present disclosure may be implemented to improve various challenges associated with training a deep learning engine for radiation therapy treatment planning. In one example, a user does not have to train a target deep learning engine 220 / 230 / 240 from scratch, especially when they only have a limited amount of target training data (e.g., limited in quantity or variation) which may lead to suboptimal results. Alternatively, a user may utilize better quality base training data 203 (e.g., more data, availability of expert curated data, more variation, etc.) for training a base deep learning engine 210.

[0046] In addition, by leveraging the knowledge already learned by the base deep learning engine 210, the computational cost of training the target deep learning engine 220 / 230 / 240 for radiotherapy treatment planning can be reduced. This helps to increase the improvement and convergence rate of the target deep learning engine 220 / 230 / 240 during the training phase 202. The risk of achieving suboptimal training results due to limited training data (e.g., limited quantity or variation) can also be reduced. Using the examples of the present disclosure, users (e.g., clinicians) do not have to have extensive knowledge about (multiple) deep learning model architectures, etc. For example, by providing a substantially stable base deep learning engine 210, the knowledge required by clinicians related to technical issues (such as convergence, local minima, or poor weight initialization during the training phase 202) can be reduced.

[0047] Will use Figure 3 Discuss some examples, Figure 3 3 is a flow chart of an example process 300 of a computer system for deep transfer learning of radiation therapy treatment plans. The example process 300 may include one or more operations, functions, or actions illustrated by one or more blocks, such as 310 to 360. Individual blocks may be combined into fewer blocks, divided into additional blocks, and / or omitted based on the desired implementation. The example process 300 may be implemented using any suitable computer system(s), and will be described using Fig.16 Discuss examples of this.

[0048] exist Figure 3 At 310 in , a base deep learning engine 210 can be obtained, which is pre-trained to perform basic radiation therapy treatment planning tasks. Here, the term "obtain" may generally refer to a computer system that accesses or retrieves data and / or computer-readable instructions associated with the base deep learning engine 210 from any suitable source (e.g., another computer system), memory or data storage (e.g., local or remote), etc. The base deep learning engine 210 may include multiple (N) base layers (denoted as Bi, where i=1, ..., N) to perform basic radiation therapy treatment planning tasks. In Figure 2 In the case where N=4, the basic deep learning engine 210 includes four basic layers 211-214.

[0049] exist Figure 3 At 320 in FIG. 1 , based on the base deep learning engine 210 , a target deep learning engine 220 / 230 / 240 may be generated to perform a target radiotherapy treatment planning task. The target deep learning engine 220 / 230 / 240 may include multiple (M) target layers (denoted as T j , where j = 1, ..., M). It should be noted that M (i.e., the number of target layers) can be the same as or different from N (i.e., the number of base layers). In this way, the base deep learning engine 210 can be further trained to adapt to the needs of the user, or fixed and used as is. The base layers 211 to 214 of the base engine 200 can be configured to be variable (see 330 and 340), invariant (see 350 and 360), or a combination of the two.

[0050] In a first example, based on the configuration of the variable base layer in the base layers 211-214, the Figure 3 330-340 in modifying the variable base layer to generate the target layer. Here, the term "variable" may refer to a layer whose weights are modifiable during the training phase 202. For example, in Figure 2In the example, the base layers 211-214 can be configured to be variable to generate the target deep learning engine 220. Based on the configuration of the variable base layers 211-214, the target layers 221-224 can be generated by modifying the corresponding variable base layers 211-214. See also 250 and the symbol "→" indicating modification.

[0051] In the second example, based on the configuration of the unchanged base layer in the base layers 211 to 214, it is possible to Figure 3 350 to 360 in generate the target layer using the feature data generated by the unchanged base layer. Here, the term "unchanged" may refer to a layer whose weights are fixed and not modified during the training phase 202. For example, in Figure 2 In the example, the base layers 211-214 can be configured to be unchanged to generate the second target deep learning 230. In this way, the target layers 231-233 can be based on the feature data (denoted as F) generated by the unchanged base layers 211-214. i , where i=1, ..., N=4) for training.

[0052] In a third example, a combination of mutable and invariant base layers can be used to generate Figure 2 In this case, the base layers 211-212 may be configured to be constant and the remaining base layers 213-214 may be variable. The latter variable base layers 213-214 may then be modified to generate corresponding target layers 241-242. Additional target layers 243-245 may be generated or trained based on feature data (denoted as F1 and F2) generated using the corresponding constant base layers 211-212. Figure 4-6 As discussed, successions of target deep learning engines can be generated using different combinations of mutable and invariant base layers.

[0053] Depending on the desired implementation, the base deep learning engine 210 and the target deep learning engine 220 / 230 / 240 may be trained to perform any appropriate (multiple) radiation therapy treatment planning tasks, such as automatic segmentation, dose prediction, treatment delivery data estimation, abnormal organ detection, treatment outcome prediction, or any combination thereof. In the case of automatic segmentation, the base deep learning engine 210 and the target deep learning engine 220 / 230 / 240 may be trained to generate a prediction of the treatment outcome based on the input = image data (e.g., Figure 1 120) generates output = structure data (e.g., Figure 1 140 in). In the case of dose prediction, the base deep learning engine 210 and the target deep learning engine 220 / 230 / 240 can be trained to generate output = dose data based on input = structural data and beam geometry data (e.g., Figure 1150 in). In the case of processing delivery data estimation, the base deep learning engine 210 and the target deep learning engine 220 / 230 / 240 can be trained to generate output = processing delivery data (e.g., fluence map data, structural projection data, etc.) based on input = structural data and / or dose data, etc.

[0054] Furthermore, both the base and target deep learning engines may be trained to perform radiation therapy treatment planning task(s) associated with any suitable anatomical site(s), rule(s), etc. Figure 4 Some examples are shown in Figure 4 It shows that according to Figure 3 A schematic diagram of an example of deep transfer learning 400. Figure 4 In the example of , the base deep learning engine 210 can be trained using the base training data 420 to perform a base radiation therapy treatment planning task associated with a base anatomical site = X (e.g., the bladder region) and a base rule R1 (according to which the task is performed).

[0055] Based on the base deep learning engine 210, target deep learning engines 410-440 can be generated to perform target radiotherapy treatment planning tasks associated with the same anatomical site = X (see 410-420) or different anatomical sites = Y (see 430-440). In addition, target deep learning engines 410-440 associated with the same rule = R1 (see 410, 430) or different rules = R2 or R3 (see 420, 440) can be generated. Here, the term "anatomical location" or "anatomical region" generally refers to a portion of a patient's anatomical structure captured using an imaging modality. The term "rule" can refer to any appropriate (multiple) clinical guidelines, strategies, and / or (multiple) planning practices associated with a specific radiotherapy treatment planning task. In order to reuse the base deep learning engine 210, target training data 421-424 associated with a target anatomical location (e.g., X or Y) and / or a rule (e.g., R1, R2, or R3) can be used during the training phase 402. In this way, the base deep learning engine 210 can be adjusted according to the needs and specifications of the user.

[0056] Next, we will use Figures 5 to 11 Various examples are discussed. In particular, Figures 5 and 6 Discuss deep transfer learning for automatic segmentation and use Figure 7 Dose prediction will be discussed. Figures 8 to 9 Explain the example inference phase and use Fig.16Explaining an example computer system. For simplicity, "deep learning engines" will be referred to as "engines" (eg, base engines and target engines), and "radiation therapy treatment planning tasks" will be referred to as "tasks" (eg, base tasks and target tasks).

[0057] Automatic segmentation

[0058] Figure 5 is a schematic diagram illustrating a first example deep transfer learning 500 for automatic segmentation of image data, and Figure 6 is a schematic diagram showing a second example deep transfer learning 600 for automatic segmentation of image data. In both examples, the base engine 510 / 610 can be pre-trained to perform a base task = automatic segmentation for a base anatomical location (e.g., X = breast region) according to a base segmentation rule (e.g., R1 = a segmentation margin of 2 mm). During a subsequent training phase 502 / 602, the knowledge learned by the base engine 510 / 610 can be transferred to the respective target engines.

[0059] exist Figure 5 In the example of , the target engines 540-550 are trained to perform the target task = automatic segmentation of the same anatomical location as the base engine 510, but according to different target rules (e.g., R2 = contour margin of 5 mm). Figure 6 In the example of , the target engines 640-650 are trained to perform target task = automatic segmentation of different target anatomical locations (e.g., Y = lung region) compared to the base engine 610. These examples are discussed below. In practice, the results of the automatic segmentation can be used for abnormal organ detection, dose prediction, treatment delivery data estimation, etc.

[0060] Although segmentation margins are used as examples in the following examples, it should be understood that the base and target segmentation rules (e.g., R1 and R2) can be configured based on any appropriate segmentation or contour guidelines, such as when a structure stops, etc. For example, contour guidelines specify when to stop outlining the upper and lower contours of a structure (e.g., a breast), or whether the contour ends at the skin boundary or extends into fat tissue (e.g., in the case of a pendulum breast). Any additional and / or base target segmentation rules may be used in the following examples.

[0061] (a) Basic Engine (Position = X, Rule R1)

[0062] First reference Figure 5In the first step in the pre-training phase 501, the base engine 510 may be trained using any appropriate base training data 521-522 associated with (position = X, rule R1). The purpose of the pre-training phase 501 is to train the base engine 510 to perform automatic segmentation by mapping the input training data = image data 521 to the output training data = structure data 522. In practice, the image data 521 may include a 2D or 3D image of an anatomical location of a patient and may be captured using any appropriate imaging modality. The structure data 522 may identify any appropriate contour, shape, size, and / or position of the structure(s) identified from the image data 521. Example structures may include target(s), OAR(s), or any other structure of interest (e.g., tissue, bone).

[0063] For example, with respect to prostate cancer, image data 521 may include image data for an X=prostate region. In this case, structure data 522 may identify a target representing the patient's prostate and OARs representing proximal healthy structures such as the rectum and bladder. With respect to lung cancer treatment, image data 521 may include image data for an X=lung region. In this case, structure data 522 may identify a target representing cancerous lung tissue and OARs representing proximal healthy lung tissue, esophagus, heart, etc. In connection with brain cancer, image data 521 may include image data for an X=brain region, in which case structure data 522 may identify a target representing a brain tumor and OARs representing proximal optic nerves, brainstem, etc. Depending on the desired implementation, structure data 522 may identify multiple targets and OARs of any suitable shapes and sizes.

[0064] In practice, the basic training data 521-522 can be generated by the user through observation and experience to promote supervised learning. For example, the basic training data 521-522 can be extracted from a treatment plan formulated for a previous patient in the past. The basic training data 521-522 can be preprocessed using any appropriate data enhancement method (e.g., rotation, flipping, translation, scaling, noise addition, cropping, any combination thereof, etc.) to produce a new data set with modified characteristics, thereby improving model generalization using ground truth. In practice, the 3D volume of the patient to be irradiated is called the treatment volume, which can be divided into multiple smaller volume pixels (voxels). In this case, the structural data 522 can specify a class label (e.g., "target", "OAR", etc.) associated with each voxel in the 3D volume.

[0065] Any (multiple) segmentation rule(s) or (multiple) guidelines may be used. For example, rule = R1 may specify a margin around an organ to be contoured. (Multiple) more advanced rules may be used, such as segmentation of specific structures at anatomical locations, selection of cutting planes for structures (e.g., spinal cord), application of different margins at different sides of an organ (e.g., a larger margin applied to the lower side than to the upper side of the organ). (Multiple) segmentation rules may be fine-tuned during a subsequent training phase 502 to achieve a more desired classification result. Thus, the base engine 510 represents a pre-trained model that may not be completely relevant to the desired classification result, but may be used as a starting point for the training phase 502. The anatomical location = X associated with the base engine 510 may be generic for medical image data, or specific to a particular treatment.

[0066] Any suitable deep learning model(s) may be used. For example, in Figure 5 In the example, the base engine 510 includes a plurality of (N>1) processing blocks or layers, each of which is labeled B i , where i=1, ..., N (see 511-51N). In this case, the pre-training stage 501 may involve finding the base weight data (denoted as wB) that minimizes the training error between the training structure data 522 and the estimated structure data (not shown for simplicity) generated by the base engine 510. i ). The training process is guided by estimating a loss associated with the classification error. A simple example of a loss function is the mean squared error between the true outcome and the predicted outcome, but loss functions can have more complex formulas (e.g., Dice loss, Jacquard loss, focal loss, etc.). This loss can be estimated at the output of the model or at any discrete point within the model.

[0067] The basic weight data wB of the i-th basic layer i It can be a scalar or a multidimensional vector. In the case of a convolutional neural network, the i-th base layer (B i ) may be a convolutional layer configured to use a convolution operation to extract information from the training data 520 or the (i-1)th base layer (B i-1 ) output to extract feature data (F i ). Alternatively and / or additionally, each i-th basis layer may be configured to perform other (multiple) operations related to activation functions, loss, concatenation, batch normalization, etc. For example, the first basis layer (B1) processes the input image data 521 to generate first feature data (F1). The second basis layer (B2) processes the first feature data (F1) to generate second feature data (F2), and so on. i ) may involve applying convolutional filters or kernels to overlapping locations of its input to learn corresponding basis weight data wBi .

[0068] The feature data generated by the i-th base layer (F i ) may include a 2D feature map for 2D image data or a 3D feature map for 3D image data. Feature data (F i ) can specify any suitable anatomical feature(s), such as borders, distance to centroid, distance to midline, distance to skin, distance to bone, laterality, presence of vertical and / or horizontal lines, shape-related parameters, texture type, any combination thereof. This automatic feature extraction approach should be distinguished from traditional approaches that rely on hand-crafted features.

[0069] (b) Target Engine (Position = X, Rule R2)

[0070] During the training phase 502, a target engine 540 / 550 associated with (position=X, rule R2) can be generated based on the base engine 510 associated with (position=X, rule R1) and the target training data 530. Compared to the base training data 520 used during the pre-training phase 501, the target training data 530 is also associated with position=X, but includes examples associated with rule=R2 instead of R1. For example, rule=R1 may include a general segmentation margin of 2mm, but the margin is increased to 5mm according to rule=R2. See also Figure 4 Related 420 in.

[0071] In practice, even when training with more limited target training data 530 than base training data 520, base engine 510 promotes faster convergence of target engine 540 / 550, thereby improving efficiency. Training phase 502 may involve finding target weight data (denoted as wT) that minimizes the training error between training structure data 532 and estimated structure data (not shown for simplicity) generated by target engine 540 / 550. j , where j=1, ..., M). Two example target engines 540-550 are discussed below.

[0072] In the first example (with Figure 2 220 in the example), the base engine 510 may be modified (e.g., further trained or fine-tuned) to generate the target engine 540. This may involve configuring the M≤N base layers (see 511-51M) of the base engine 510 as "mutable" so that they can be further trained to generate the corresponding target layers (see 541-54M) of the target engine 540. The modification may involve modifying the base weight data wB associated with the corresponding base layers 511-51M. i (i=1, ..., M) to obtain the corresponding target weight data wT j(j=1, ..., M). Once training is complete (see 533), each modified base layer (B i ) is used as the corresponding target layer (T j ). During the inference phase (which will use Figure 8 ), the target engine 550 can be used to perform automatic segmentation.

[0073] In the second example (with Figure 2 230 in the example), instead of modifying the base engine 510, the target engine 550 is built on top of the base engine 510. In this case, the K≤N base layers (see 511-51K) of the base engine 510 can be configured to be "unchanged" (i.e., fixed or frozen). In this way, the K unchanged base layers can be used to generate feature data (F) based on the target training data 530. K ).like Figure 5 As shown in 560, the feature data (F K ) is then fed to the target engine 550 to achieve the desired classification result, i.e., automatic segmentation for anatomical location = X according to rule = R2. The target engine 550 may include any appropriate number (M) of target layers (see 551-55M). During the inference phase (using Figure 8 ), the base engine 510 and the target engine 550 may be used sequentially to perform automatic segmentation.

[0074] In the third example (for simplicity in Figure 4 ), a combination of variable and invariant base layers can be configured (with Figure 2 In this case, the base engine 510 may be configured to include K≤N invariant base layers and at most NK variable base layers. During the training phase 502, the K invariant base layers are fixed and used to process the target training data 530 to generate feature data (F1, ..., F K ). The feature data is then used as input to modify the mutable base layer and generate any subsequent target layers.

[0075] (c) Target Engine (position = Y, rule R1 or R3)

[0076] Reference now Figure 6 , a target engine associated with different anatomical locations and / or rules can be generated based on the base engine 610. Figure 5In the example of , the base engine 610 can be trained to perform automatic segmentation using the base training data 620 associated with (position=X, rule R1). The purpose of the pre-training phase 601 is to train the base engine 610 to map input=image data 621 to output=structure data 622.

[0077] In a first example, the base engine 610 may be modified to generate a first target engine 640 to perform automatic segmentation for different target anatomical locations = Y according to a target rule = R1. For example, X represents the bladder region, and Y represents the lung region. In another example, a base engine 510 / 610 trained to perform a base task = breast segmentation may be obtained, and further trained to generate a target engine using example contours including both breasts and lymph nodes to perform a target task = breast and lymph node segmentation.

[0078] In this way, knowledge related to the structure(s) in the base anatomical location = X can be transferred from the base engine 610 to the target engine 640 for automatic segmentation in the target anatomical location = Y. During the training phase 602, the M≤N variable base layers (see 611-61M) of the base engine 610 can be modified to generate corresponding target layers (see 641-64M). The modification can involve modifying the base weight data wB associated with the corresponding base layers 611-61M. i (i=1, ..., M) to obtain the corresponding target weight data wT j (j=1, ..., M). Once training is complete (see 633), each modified base layer (B i ) is used as the corresponding target layer (T j ).

[0079] In a second example, a combination of invariant and variable base layers may be configured to generate a second target engine 650 to perform automatic segmentation on a target anatomical location = Y according to a different target rule = R3. For simplicity, four base layers 611-614 are considered, which have been reused for the target engine 650. The invariant base layers (see 611-612) are fixed and used to generate feature data (e.g., F1 and F2) based on the target training data 630. The remaining variable base layers are modified based on the feature data to generate corresponding target layers 651-652. The feature data (e.g., F3 and F4) from the variable base layers are then used to train additional target layers 653-655. In practice, it will be appreciated that any appropriate number of invariant and variable base layers may be configured.

[0080] In practice, the training process 502 / 602 may include validating the Figure 5The target engines 540-550 and Figure 6 For example, quality metrics such as dice scores, average surface distance (measures the error of contour surface positions relative to ground truth), etc. may be tracked for validation purposes.

[0081] Dose prediction and other planning tasks

[0082] Figure 7 is a schematic diagram showing an example deep transfer learning 700 for dose prediction. During the pre-training phase 701, a base engine 710 may be trained using any suitable base training data 720 to perform dose prediction for an anatomical location = X according to a prediction rule = D1. The goal is to train the base engine 710 to map example input data = image data and structure data 721 (i.e., segmented image data), and example output data = dose data 722. Any suitable constraint(s) may be used, such as limiting dose prediction to only the vicinity of the target(s) or to a specific dose level.

[0083] Dose data 722 (eg, 3D dose data) may be specified for a target (denoted as “D TAR ”) and OAR (denoted as “D OAR ”). For example, with respect to prostate cancer, the dose data 712 may specify a dose distribution for a target representing the patient's prostate and an OAR representing proximal healthy structures such as the rectum or bladder. In practice, the dose data 722 may specify a dose distribution for the entire 3D volume, not just the target and OAR volumes. Depending on the desired implementation, the dose data 722 may include spatial bioeffect data (e.g., fractionation correction dose) and / or cover only a portion of the treatment volume. In addition to the structural data 721, additional input data may also include beam geometry data associated with the treatment delivery system.

[0084] During the training phase 702, the knowledge learned by the base engine 710 can be transferred to the target engines 740-750. In the first example, a target engine 740 is generated that is associated with anatomical location = X according to rule = D2. This involves modifying the variable base layers 711-71M of the base engine 710 based on the target training data 730-732 to generate corresponding target layers 741-74M. In the second example, a target engine 750 is generated that is associated with anatomical location = Y according to rule = D1. In particular, based on the target training data 733-735, the invariant base layers 711-71K of the base engine 710 can be used to generate feature data (see 760) to train the target layers 751-75M.

[0085] In practice, any appropriate prediction rules D1 and D2 may be used, such as rules relating to organ preservation, target coverage (and dose prescription), and normal tissue dose. Additionally or alternatively, the prediction rule(s) may relate to treatment techniques (e.g., IMRT, VMAT, etc.), cancer type, machine specifications (e.g., energy and field shape, clinical practice for field placement, etc.). All of these will have an impact on the predicted dose data. The prediction rule(s) may be implicitly learned from the training data 720 / 730 / 733, or optionally provided as input parameters to a particular type of deep learning engine.

[0086] Apart from Figures 5 and 6 Automatic segmentation in Figure 7 In addition to dose prediction in the radiation therapy treatment planning task, deep transfer learning can be implemented for other radiation therapy treatment planning tasks such as treatment delivery data estimation, treatment outcome prediction, etc. In connection with treatment delivery data estimation, the estimated treatment delivery data (i.e., output data) can include structural projection data, fluence map data, etc. For example, the target engine can be trained to perform structural projection data, such as based on image data, structural data, dose data, or any combination thereof. The structural projection data can include data related to the beam direction and machine trajectory of the treatment delivery system (see Figure 1 and Fig.16 160 in the above table).

[0087] In another example, the target engine can be trained to perform fluence map estimation, such as a 2D fluence map for a set of beam directions / trajectories, machine control point data (e.g., jaw and lobe positions, gantry and couch positions), etc. Fig.10 Further interpretation of the fluence map. Any additional and / or alternative training data may be used, such as site geometry data, monitoring units (amount of radiation counted by the machine), quality of plan estimates (acceptable or not), daily dose prescription (output), site size or other machine parameters, patient lying position parameters or isocenter position, treatment strategy (whether to use motion control mechanisms, boost or not), treat or not treat decisions.

[0088] Although deep convolutional neural networks are used for illustration, it should be understood that any alternative and / or additional (multiple) deep learning models may be used to implement the base and target engines. For example, the base and target engines described in the present disclosure may include multiple processing paths described in related U.S. Patent Application No. 16 / 145,461 (Agent Docket No. 2018-012US01). The processing paths may be configured to process input data (e.g., image data) at different resolution levels. In this way, a larger receptive domain may be achieved.

[0089] In practice, medical image data usually includes local and global feature data of the patient's anatomical structure, where the terms "local" and "global" are relative in nature. For example, the local feature data can provide a microscopic view of the patient's anatomical structure, such as tissue texture, whether the structure has a bounding boundary, etc. In contrast, the global feature data can provide a relative macroscopic view of the patient's anatomical structure, such as in which region (e.g., prostate, etc.) the anatomical structure is located, the direction (e.g., left, right, forward, backward), etc.

[0090] Inference Phase

[0091] Figure 8 is shown for use according to Figure 5 , Figure 6 and Figure 7 Schematic diagram of an example inference stage 801 of a target engine generated by the example in . Once trained, the target engine 540 / 550 / 640 / 650 can be used by a clinician during the inference stage 801 to perform automatic segmentation, thereby generating output data = patient structure data 803 based on the input data = image data 802 of the current patient. In one example, the image data 802 is processed using the target layer of the target engine 540 / 640 generated by modifying the variable base layer of the base engine 510 / 610. In another example, the image data 802 is processed by both the base engine 510 / 610 and the target engine 550 / 650 arranged in sequence. In practice, it should be noted that the "current patient" is one of the multiple patients being processed simultaneously during the inference stage 801. In addition, automatic segmentation can be performed when the image data 802 is captured and transmitted to a storage system (e.g., a picture archiving and communication system (PACS) or the like).

[0092] Similarly, the clinician can use the target engine 640 / 650 to perform automatic segmentation during the inference phase 801 to generate output data = dose data 804 for the current patient based on the input data = image data and structural data 803 associated with the patient. In one example, the input data 803 is processed using the target layer of the target engine 740 generated by modifying the variable base layer of the base engine 510 / 610. In another example, the input data 803 is processed by the base engine 710 and the target engine 750 arranged in sequence. Figure 7 As discussed, any additional and / or alternative input data and output data may be used. During radiotherapy treatment planning, the Figure 8 The structural data 803 and dose data 804 generated during the inference phase 801 are used to generate a treatment plan.

[0093] Fig. 9 is based on Figure 8Schematic diagram of an example treatment plan 156 / 900 generated or improved from example output data.The treatment plan 156 may be delivered using any suitable treatment delivery system including a radiation source 910 to project radiation beams 920 at various beam angles 930 onto a treatment volume 960 representing the patient's anatomy.

[0094] Although for simplicity Fig. 9 156, but the radiation source 910 may include a linear accelerator for accelerating the radiation beam 920 and a collimator (e.g., MLC) for modifying or modulating the radiation beam 920. In another example, the radiation beam 920 may be modulated by scanning the target patient in a specific pattern with different energies and dwell times (e.g., in proton therapy). A controller (e.g., a computer system) may be used to control the operation of the radiation source 910 according to the treatment plan 156.

[0095] During treatment delivery, the radiation source 910 can be rotated about the patient using a gantry, or the patient can be rotated (as in some proton radiation therapy treatment protocols) to transmit the radiation beam 920 at various beam directions or angles relative to the patient. For example, five equally spaced beam angles 930A-E (also labeled "A," "B," "C," "D," and "E") can be selected using a deep learning engine configured to perform treatment delivery data estimation. In practice, any appropriate number of beam and / or table or chair angles 930 can be selected (e.g., five, seven, etc.). At each beam angle, the radiation beam 920 is associated with a fluence plane 940 (also called an intersection plane) located outside the patient envelope along a beam axis extending from the radiation source 910 to the treatment volume 960. As shown in FIG. Fig. 9 As shown, the fluence plane 940 is generally located at a known distance from the isocenter.

[0096] Sample Planning System

[0097] The examples of the present disclosure can be deployed in any suitable manner, such as a standalone system, a web-based Platform as a Service (PaaS) system, etc. Fig.10 Describe an example planning system (also referred to as a "computer system"), Fig.10 FIG. 1 is a schematic diagram illustrating an example network environment 1000 in which deep transfer learning for radiation therapy treatment planning may be implemented. Depending on the desired implementation, the network environment 1000 may include: Fig.10 The components shown in FIG. 5 are further additional or alternative components.

[0098] exist Fig.10In the example of , the network environment 1000 includes a planning system 1010 that can be accessed by multiple user devices 1001-1003 via any suitable physical network (e.g., a local area network, a wide area network, etc.). In practice, the user devices 1001-1003 can be operated by individual users located at a specific clinical location or different clinical locations. The planning system 1010 can be implemented using a multi-layer architecture, which includes: a network-based user interface (UI) layer 1011 to interact with the user devices 1001-1003; a training infrastructure 1012 (also known as an application layer) to perform deep transfer learning; and a data layer 1013 to facilitate data access to / from a data store 1020. Depending on the desired implementation, the planning system 1010 can be deployed in a cloud computing environment, in which case multiple virtualized computing instances (e.g., virtual machines, containers) can be configured to implement the various functions of the layers 1011-1013. The cloud computing environment can be supported by a local cloud infrastructure, a public cloud infrastructure, or a combination of the two.

[0099] In more detail, the UI layer 1011 may be configured to provide any suitable (multiple) interfaces for interacting with the user devices 1001-1003, such as a graphical user interface (GUI), a command line interface (CLI), an application programming interface (API) call, any combination thereof, etc. The training infrastructure may be configured to perform training of a base engine during a pre-training phase, and to perform deep transfer learning (i.e., model adaptation) during training and validation phases to generate a target engine.

[0100] The data layer 1013 can be configured to facilitate (via the UI layer 1011 and the training infrastructure 1012) access to multiple base and target engines stored in the data store 1020. Figures 5 to 7 In the example discussed, the data store 1020 may store the base engine 510 / 610 associated with (position = X, segmentation rule = R1) and the base engine 710 associated with (position = X, dose prediction rule = D1). In this way, the user devices 1001-1003 do not have to interact directly with the backend training infrastructure 1012 and data layer 1013, thereby providing abstraction and enhancing security.

[0101] (a) Pre-training phase

[0102] Fig.11 Is used for Fig.101100 may include one or more operations, functions, or actions illustrated by one or more blocks, such as 1105-1180. Individual blocks may be combined into fewer blocks, divided into additional blocks, and / or omitted based on a desired implementation. The example process 1100 may be implemented using any suitable computer system(s), and examples thereof will be described using Fig.16 Let's discuss.

[0103] exist Fig.11 At 1105 in the pre-training phase, multiple basic engines can be trained to perform corresponding basic tasks using basic training data. Each basic engine can be trained to perform a specific basic task associated with a specific location (e.g., location X = bladder region) according to a specific rule (e.g., segmentation margin of R1 = 2 mm). Each basic engine can be trained to perform any suitable basic task, such as segmentation (see Figures 5 and 6 ), dose prediction (see Figure 7 ), processing delivery data estimation, treatment outcome prediction, etc. Once pre-training is performed, a plurality of base engines (see 1021-1022) may be stored in data storage 1020 for later access during the training phase. Depending on the desired implementation, block 1105 may be implemented by training infrastructure 1012 (e.g. Fig.11 As shown), user equipment 1001 / 1002 / 1003, alternative systems or any combination thereof.

[0104] (b) Sample user interface

[0105] exist Fig.11 At 1110 in the plan system 1010, the planning system 1010 may provide a first UI supported by (multiple) UI layers 1011 for a user to send a request to generate a target engine to perform a target task. At 1120, the user may interact with (multiple) first UIs to send a request to generate a target engine based on a base engine and target training data. The request may be generated using an interface (e.g., GUI, API, CLI) supported by the UI layer 1011. For example, in Fig.10 , the first user device 1001 may send a first request to the UI layer 1011 by interacting with the GUI (see 1031). The second user device 1002 may send a second request by calling (multiple) API calls (see 1032), and the third user device 1003 may send a third request using (multiple) CLI commands (see 1033).

[0106] Will use Figures 12A to 12B and FIG. 13A to FIG. 13BExplaining several example UIs, these figures are schematic diagrams showing example interfaces 1200 generated by the target engine during the training phase. To facilitate deep transfer learning, requests 1031-1033 can each include any suitable target training data (see Fig.11 and Fig. 12A 1121 in ). Requests 1031-1033 may include any additional data for generating the corresponding target engine, such as basic engine selection parameter data (see Fig.11 and Fig. 12B 1122 in), training parameter data (see 1123 and Fig.13A ), verify parameter data (see 1124 and Fig. 13B ) etc. In practice, the training interface 1200 may include any suitable “UI elements” with which a user can interact, such as windows, panes, buttons, menus, text boxes, tabs, lists, application icons, menu bars, scroll bars, title bars, status bars, toolbars, drop-down lists, any combination thereof, etc.

[0107] exist Fig. 12A At 1210 in the training interface 1200 (see the "Target Engine" view), the training interface 1200 includes a UI element for a user to provide target training data associated with a plurality of past patients or to specify a link (e.g., a uniform resource locator (URL), a network path, etc.) to the target training data. The target training data may include image data in any suitable format that can be processed by the training infrastructure 1012, such as Digital Imaging and Communications in Medicine (DICOM) images. Figure 5 In the automatic segmentation example in FIG. 1 , the first request 1031 may include target training data 530 to generate a target engine 540 / 550 associated with (position=X, target rule=R2). The second request 1032 may include Figure 6 The target training data 630 in the target engine 640 / 650 is generated to generate the target engine 640 / 650 associated with (position=Y, target rule=R1). Figure 7 In the dose prediction in, the third request 1033 may include the target training data 730 to generate the target engine 740 / 750 associated with (position=X, target rule=D2).

[0108] exist Fig. 12BAt 1220 in the training interface 1200 (see the "Base Engine" view), the training interface 1200 includes (multiple) UI elements for a user to specify selection parameter data for selecting a base engine. In the left pane of the training interface 1200 (see 1222), base engines can be searched or filtered based on task type (e.g., segmentation, dose prediction), anatomical location, structure, base engine provider (e.g., providers P1 and P2), performance metrics, (multiple) keywords, etc. In the right pane (see 1224), a list of matching base engines is presented for selection, such as ID, anatomical location, base rule (e.g., segmentation margin = R1, dose prediction rule = D1), (multiple) structures, (multiple) performance metrics, etc. In practice, the user can first select the planned task and general anatomical location before specifying more specific requirements. In response to receiving the base engine selection parameter data, the UI layer 1011 can retrieve a list of candidate base engines and at Fig. 12B A list is presented on interface 1200 for the user to select.

[0109] exist Fig. 12B At 1226, the training interface 1200 includes (multiple) UI elements for the user to specify mapping data used during the training phase. Based on the mapping data, the training infrastructure 1012 can map the target name (or target ID) associated with the target engine to the corresponding base name (or base ID) associated with the base engine. For example, the base engine for automatic segmentation can provide a list of structures as its output. These structures can be called "Breast Left", "Right Chest" and "Heart" by the base engine. However, in the target training data provided by the user, different (target) names may be used for the same structure, such as "Brust Links", "Brust Rechts" and "Herz" (German) or "BR-L", "BR-R" and "H" (shorthand specific to clinical locations). In this case, the mapping data can be provided to the training infrastructure 1012 to convert the base name (e.g., "right chest" with ID=112) to its corresponding target name (e.g., "BR-R" with ID=131). Where possible, the planning system 1010 can also automatically handle the mapping through the base / target structure ID mapping. The mapping may also be based on the structure ID of any suitable clinical standard or reference ontology, such as the Foundation Model of Anatomy (FMA) ontology or the like.

[0110] exist Fig.13AAt 1310 (see the "Training" view), the training interface 1200 also includes (multiple) UI elements for the user to configure training parameter data to guide the training infrastructure 1012 during the training phase. Under "Base Layer Configuration" (see 1312), the base engine can be configured to include (multiple) variable base layers and / or (multiple) constant base layers. Depending on the desired implementation, the number (or percentage) of variable and constant base layers can be configured. Under "Training Hyperparameters" (see 1314), hyperparameters associated with deep transfer learning, such as loss rate, activation function, number of training cycles, learning rate, number of hidden units, convolution kernel width, etc., can be configured. In practice, the configuration of base layers and training parameters can be automatically performed by the training infrastructure 1012, rather than relying on user input. Depending on the changed training hyperparameters, a complete retraining may be required because the underlying graph associated with the base engine changes. For example, in response to changes in hyperparameters such as convolution kernel width or activation function, a complete retraining may be required. However, when updating parameters such as the loss function, learning rate, and number of epochs, a full retraining is usually not necessary.

[0111] exist Fig. 13B At 1320 (see "Validation" view), the training interface 1200 also includes (multiple) UI elements for the user to configure validation parameter data to guide the training infrastructure 1012 during the validation phase. Under "Validation Criteria" (see 1332), for example, thresholds for validation parameter data such as dice score, average surface difference, Hausdorff distance, etc. can be configured. Although using FIG. 12A to FIG. 12B and FIG. 13A to FIG. 13B It is discussed, but it should be understood that the request 1032 / 1033 generated using the API / CLI may also similarly include target training data, base engine selection data, training parameter data, verification parameter data, or any combination thereof.

[0112] (b) Training phase

[0113] refer to Fig.11 At 1125, the planning system 1010 (e.g., UI layer 1011) detects a request (e.g., 1031) from a user device (e.g., 1001) to generate a target engine. Fig.11 At 1130 of , in response to detecting the request, the planning system 1010 (e.g., the training infrastructure 1012) can select a base engine from the plurality of base engines pre-trained at block 1105. The selection at block 1130 can be based on a manual selection by a user or an automatic selection by the training infrastructure 1012. In the case of a manual selection, the request can specify (e.g., using a model or engine ID) a particular base engine selected by the user, such as using Fig. 12B1200. In the case of automatic selection, the request may specify selection parameter data for the training infrastructure 1012 to automatically select a base engine, such as anatomical location, structure(s) or organ(s), performance metric(s), base engine provider, or any combination thereof.

[0114] exist Fig.11 At 1135 of the request, the training infrastructure 1012 configures the base engine selected at block 1130 to include (multiple) variable base layers, (multiple) invariant base layers, or a combination of both. Block 1135 may be based on the base layer configuration specified in the request (see Fig.13A 1312) or the default settings (if applicable). Fig.11 At 1140 of FIG. 1 , the training infrastructure 1012 performs deep transfer learning to generate a target engine based on the base engine selected at block 1130. During the training phase, the target engine can be generated based on the mapping data (see Fig. 12B ) and / or training parameter data (see Fig.13A ) etc. to perform deep transfer learning. In practice, the base engine at box 1135 can be represented using a set of weights (i.e., variable or constant, depending on its configuration) that will be used as a starting point for generating the target engine.

[0115] In a first example, a base engine (e.g., Figure 5 510 and Figure 6 610 in ) is configured to include M≤N variable base layers. In this case, the training infrastructure 1012 can be based on the target training data (e.g., Figure 5 530 and Figure 6 630) to modify the variable base layer to generate multiple target layers of the target engine (e.g., Figure 5 540 and Figure 6 640 in ). See Figure 7 Of the 740.

[0116] In a second example, the base engine (e.g., Figure 5 510 in ) is configured to include K≤N invariant base layers that are not modified during training. In this case, the training infrastructure 1012 can generate multiple target layers (e.g., Figure 5 550), wherein the feature data is based on target training data (e.g., Figure 5 530) is generated by the unchanged base layer. Figure 7 750 of them.

[0117] In a third example, a base engine (e.g., Figure 6610 in ) is configured to include a combination of variable and invariant base layers. In this case, the training infrastructure 1012 can generate multiple target layers (e.g., Figure 6 Then, the remaining variable base layer is modified based on the feature data to generate an additional target layer.

[0118] use Figures 1 to 9 The example of deep transfer learning explained is applicable here and is not repeated in detail for simplicity. In practice, a user's request may cause the training infrastructure 1012 to: (a) train a target engine to meet specific requirements; or (b) adjust the base engine to better match the different characteristics present in a specific patient population. For case (a), the request may specify guidelines such as additional margin, selection of cutting planes for long organs (e.g., spinal cord), etc.

[0119] For case (b), retraining may be required if the anatomy of the patient population is different from the patient population used to train the original model (i.e., the base engine). Differences in patient populations may include differences in body mass index and size, or even the shape of contoured organs (e.g., breast size may be different in different geographic locations due to differences in the amount of adipose tissue). These differences may be related to anatomical location, such as a base engine developed for an intact breast may be retrained for a patient who has had a mastectomy. The base and target engines may be based on any appropriate medical imaging modality, such as CT, CBCT, PET, MRI, or any combination thereof.

[0120] (d) Verification phase

[0121] exist Fig.11 At 1145, the training infrastructure 1012 is based on, for example, using Fig. 13B The validation parameter data configured in the training interface 1200 in the target engine is used to perform validation of the target engine. Validation can be performed to improve the results, such as preventing overfitting or falling into a local minimum, providing convergence indication, etc. In practice, the target training data provided by the user can be divided into a training set for the training phase and a validation set for the validation phase.

[0122] In response to determining that a particular iteration of the target engine does not meet a particular validation criterion, the training phase continues at block 1140 for the next iteration. Otherwise, at 1150, in response (e.g., Fig.10 1041 / 1042 / 1043) is sent to a user device (e.g., 1001 / 1002 / 1003) to facilitate access to the generated target engine and any associated data. The generated target engine (e.g., Fig.101023 / 1024) can be stored in the data storage 1020.

[0123] Planning system 1010 can generate statistics and visual inspections about how much the target engine has changed compared to the base engine. Some examples are FIG. 14A to FIG. 14B 1410 ), which are schematic diagrams showing the training interface 1200 generated for the target engine during the validation phase. Fig.14A At 1420 and 1430 in the training interface 1200, the validation metrics view can present statistics or metrics associated with the generated target engine (e.g., ID=200). Example metrics include dice score (or sharded dice score), accuracy, sensitivity, average surface difference, Hausdorff distance, etc.

[0124] As a reference, quality statistics can also be provided by applying the target engine to a larger set of target training data trained with the base engine. In practice, the validation criteria can be specific to a particular application, target task, anatomical location, etc. For example, larger structures typically give higher Dice scores than smaller structures. These scores can be used to track whether the target engine is improving. Fig.14A In the example in , it is desired to make the "adaptive" (target) model perform better than the "non-adaptive" (base) model. Depending on the desired implementation, the training interface 1200 can provide a UI that presents how the target engine performs for the "base" verification case (i.e., used to verify the base engine) and any additional verification cases (i.e., used to verify the target engine). In this way, the user can distinguish between a reduction in generality and an improvement in location-specific performance.

[0125] exist Fig. 14B At 1440 in the training interface 1200, the visual inspection view can present the adaptation of the target engine. For example, a specific patient can be selected (see 1450) to compare the ground truth with the output of the target engine for that patient. The visual inspection can be presented in multiple anatomical planes, including the cross-sectional or axial plane, the sagittal plane, and the coronal plane. The axial plane is parallel to the ground and divides the body into a top and a bottom. The sagittal or lateral plane divides the body into left and right halves. The coronal or anterior plane divides the body into front and back halves.

[0126] (e) Inference stage

[0127] After the training phase, the generated target engine can be provided to users for local application or cloud-based planning application. Fig.11At 1155-1160 of the UI layer 1011, the UI layer 1011 is further configured to provide a planning interface for a user (e.g., a clinician) to interact with. At 1165, by interacting with the planning interface, the user can access and instruct the goal engine to perform a goal task for an individual patient. At 1170-1180, the planning system 1010 performs the goal task (e.g., automatic segmentation) for the individual patient and responds to any suitable output data (e.g., structural data) and statistics, which will be accessible via the interface(s) supported by the UI layer 1011.

[0128] The first example is Fig.15A As shown, Fig.15A 1 is a schematic diagram showing a planning interface 1500 for automatic segmentation during the inference phase. The planning interface 1500 includes UI elements for: a user to provide input associated with a patient = image data (see 1510); and to select and instruct a specific target engine (see 1520) to be trained to perform automatic segmentation (see 1530). For example, one may select Figure 5 The target engine 540 / 550 in the example performs the AND operation with (position X, rule R2) or Figure 6 1540 ). The planning interface 1500 includes UI elements to present the output structure data associated with the patient (see 1540 ) and for the user to download the output structure data (see 1550 ). In practice, the planning interface 1500 can be integrated with any suitable system, such as (multiple) treatment planning platforms, (multiple) patient databases, etc. An example of a treatment planning platform is Eclipse TM Treatment Planning Platform (available from Varian Medical Systems), etc.

[0129] The second example is Fig. 15B As shown, Fig. 15B 15 is a schematic diagram showing a planning interface 1500 for dose prediction during the inference phase. The planning interface 1500 includes UI elements for: a user to provide input associated with a patient = image data and structural data (see 1560); and to select and instruct a specific target engine (see 1570) to be trained to perform dose prediction (see 1580). Figure 7 In the example of , the target engine 740 / 750 may be instructed to perform a dose prediction associated with (location X, prediction rule D1 or D2). Once the dose prediction is completed, the planning interface 1500 includes a UI element to present the output = dose data associated with the patient (see 1590) and for the user to download the output dose data (see 1595).

[0130] Computer Systems

[0131] The above examples may be implemented by hardware, software or firmware, or a combination thereof. Fig.16 1 is a schematic diagram of an example computer system 1600 for deep transfer learning of radiation therapy treatment planning. In this example, the computer system 1600 (also referred to as a treatment planning system) may include a processor 1610, a computer-readable storage medium 1620, a system interface 1640 that interacts with the radiation treatment delivery system 160, and a bus 1630 that facilitates communication between these illustrated components and other components.

[0132] The processor 1610 is used to execute the Figures 1 to 15B The computer-readable storage medium 1620 may store any suitable information 1622, such as information related to training data, deep learning engines, image data, output data, etc. The computer-readable storage medium 1620 may further store computer-readable instructions 1624, which, in response to execution by the processor 1610, cause the processor 1610 to perform the processes described herein. Figure 1 The treatment planning system 160 explained here delivers treatment according to the treatment plan 156, and for the sake of brevity, its description is not repeated here. In practice, the computer system 1600 can be part of a computing cluster including multiple computer systems. The computer system 1600 can include any (multiple) alternative and / or additional components, such as a graphics processing unit (GPU), a message queue for communication, an object store or database, (multiple) load balancers, dedicated circuits, etc. Fig.10 As discussed, computer system 1600 may be deployed in any suitable manner, including as a service-type deployment in a local cloud infrastructure, a public cloud infrastructure, combinations thereof, and the like.

[0133] The foregoing detailed description has been described various embodiments of the device and / or processing by using block diagrams, flow charts and / or examples. As long as such block diagrams, flow charts and / or examples contain one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation in such block diagrams, flow charts or examples can be implemented individually and / or collectively by a wide range of hardware, software, firmware or any combination thereof. In the present invention, the terms "first", "second", "third" and the like do not represent any order of importance, but are used to distinguish one element from another.

[0134] Those skilled in the art will recognize that some aspects of the embodiments disclosed herein may be equivalently implemented in whole or in part in an integrated circuit as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), firmware, or any combination thereof, and that designing circuits and / or writing code for software and firmware will be well within the technical scope of those skilled in the art in accordance with the present disclosure.

[0135] Although the present disclosure has been described with reference to specific exemplary embodiments, it should be recognized that the present disclosure is not limited to the described embodiments, but can be modified and altered within the spirit and scope of the appended claims. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A method for a computer system to perform deep transfer learning for a radiation therapy treatment plan, wherein the method comprises: obtaining a base deep learning engine, the base deep learning engine being pre-trained based on base training data to perform a basic radiation therapy treatment planning task, wherein the base deep learning engine comprises a plurality of base layers; as well as Based on the base deep learning engine and the target training data, a target deep learning engine is generated to perform a target radiation therapy treatment planning task by performing one of the following: configuring a variable base layer among the multiple base layers of the base deep learning engine, and generating one of the multiple target layers of the target deep learning engine by modifying the variable base layer, wherein modifying the variable base layer comprises modifying base weight data associated with the corresponding base layer to obtain corresponding target weight data; as well as configuring an invariant base layer among the multiple base layers of the base deep learning engine, and generating one target layer among the multiple target layers of the target deep learning engine based on feature data generated using the invariant base layer and by training the target layer based on the feature data, Generating a processing plan based on data generated by the target deep learning engine, wherein generating the target deep learning engine comprises: Based on the base deep learning engine that is pre-trained to perform the base radiation therapy treatment planning task associated with a base anatomical position, the target deep learning engine is generated to perform the target radiation therapy treatment planning task associated with a target anatomical position, which is different from the base anatomical position.

2. The method according to claim 1, wherein generating the target deep learning engine comprises: Based on the base deep learning engine that is pre-trained to perform the base radiation therapy treatment planning task according to base rules, the target deep learning engine is generated to perform the target radiation therapy treatment planning task associated with target rules.

3. The method according to claim 1 or 2, wherein generating the target deep learning engine comprises: The target deep learning engine is generated to perform one of the following target radiation therapy treatment planning tasks: generating automatic segmentation of structure data based on image data; generating dose predictions of dose data based on the structural data and the image data; and generating a processing delivery data prediction for processing the delivery data.

4. The method according to any one of claims 1 to 2, wherein generating the target deep learning engine comprises one of the following: configuring the plurality of base layers of the base deep learning engine as variable base layers; and The plurality of target layers are generated by modifying corresponding variable base layers based on target training data associated with the target radiation therapy treatment plan task.

5. The method according to any one of claims 1 to 2, wherein generating the target deep learning engine comprises: Configuring the plurality of base layers of the base deep learning engine as invariant base layers; as well as The feature data is generated using the invariant base layer based on target training data associated with the target radiation therapy treatment planning task.

6. The method according to any one of claims 1 to 2, wherein generating the target deep learning engine comprises: The plurality of base layers of the base deep learning engine are configured to include both immutable base layers and variable base layers.

7. A non-transitory computer-readable storage medium comprising a set of instructions that, in response to being executed by a processor of a computer system, causes the processor to perform a method for deep transfer learning for radiation therapy treatment planning, wherein the method comprises: obtaining a base deep learning engine, the base deep learning engine being pre-trained based on base training data to perform a basic radiation therapy treatment planning task, wherein the base deep learning engine comprises a plurality of base layers; as well as Based on the base deep learning engine and the target training data, a target deep learning engine is generated to perform a target radiation therapy treatment planning task by performing one of the following: configuring a variable base layer among the multiple base layers of the base deep learning engine, and generating one of the multiple target layers of the target deep learning engine by modifying the variable base layer, wherein modifying the variable base layer comprises modifying base weight data associated with the corresponding base layer to obtain corresponding target weight data; as well as configuring an invariant base layer among the multiple base layers of the base deep learning engine, and generating one target layer among the multiple target layers of the target deep learning engine based on feature data generated using the invariant base layer and by training the target layer based on the feature data, Generating a processing plan based on data generated by the target deep learning engine, wherein generating the target deep learning engine comprises: Based on the base deep learning engine that is pre-trained to perform the base radiation therapy treatment planning task associated with a base anatomical position, the target deep learning engine is generated to perform the target radiation therapy treatment planning task associated with a target anatomical position, which is different from the base anatomical position.

8. The non-transitory computer-readable storage medium of claim 7, wherein generating the target deep learning engine comprises: Based on the base deep learning engine that is pre-trained to perform the base radiation therapy treatment planning task according to base rules, the target deep learning engine is generated to perform the target radiation therapy treatment planning task associated with target rules.

9. The non-transitory computer-readable storage medium according to claim 7 or 8, wherein generating the target deep learning engine comprises: The target deep learning engine is generated to perform one of the following target radiation therapy treatment planning tasks: generating automatic segmentation of structure data based on image data; generating dose predictions of dose data based on the structural data and the image data; and generating a processing delivery data prediction for processing the delivery data.

10. The non-transitory computer-readable storage medium according to any one of claims 7 to 8, wherein generating the target deep learning engine comprises one of the following: configuring the plurality of base layers of the base deep learning engine as variable base layers; and The plurality of target layers are generated by modifying corresponding variable base layers based on target training data associated with the target radiation therapy treatment plan task.

11. The non-transitory computer-readable storage medium according to any one of claims 7 to 8, wherein generating the target deep learning engine comprises: Configuring the plurality of base layers of the base deep learning engine as invariant base layers; as well as The feature data is generated using the invariant base layer based on target training data associated with the target radiation therapy treatment planning task.

12. The non-transitory computer-readable storage medium according to any one of claims 7 to 8, wherein generating the target deep learning engine comprises: The plurality of base layers of the base deep learning engine are configured to include both immutable base layers and variable base layers.

13. A computer system configured to perform deep transfer learning for radiation therapy treatment planning, wherein the computer system comprises a processor and a non-transitory computer readable medium having instructions stored thereon that, when executed by the processor, cause the processor to: obtaining a base deep learning engine, the base deep learning engine being pre-trained based on base training data to perform a basic radiation therapy treatment planning task, wherein the base deep learning engine comprises a plurality of base layers; as well as Based on the base deep learning engine and the target training data, a target deep learning engine is generated to perform a target radiation therapy treatment planning task by performing one of the following: configuring a variable base layer among the multiple base layers of the base deep learning engine, and generating one of the multiple target layers of the target deep learning engine by modifying the variable base layer, wherein modifying the variable base layer comprises modifying base weight data associated with the corresponding base layer to obtain corresponding target weight data; as well as configuring an invariant base layer among the multiple base layers of the base deep learning engine, and generating one target layer among the multiple target layers of the target deep learning engine based on feature data generated using the invariant base layer and by training the target layer based on the feature data, generating a processing plan based on data generated by the target deep learning engine, wherein the instructions for generating the target deep learning engine cause the processor to: Based on the base deep learning engine that is pre-trained to perform the base radiation therapy treatment planning task associated with a base anatomical position, the target deep learning engine is generated to perform the target radiation therapy treatment planning task associated with a target anatomical position, which is different from the base anatomical position.

14. The computer system of claim 13, wherein the instructions for generating the target deep learning engine cause the processor to: Based on the base deep learning engine that is pre-trained to perform the base radiation therapy treatment planning task according to base rules, the target deep learning engine is generated to perform the target radiation therapy treatment planning task associated with target rules.

15. The computer system of claim 13 or 14, wherein the instructions for generating the target deep learning engine cause the processor to: The target deep learning engine is generated to perform one of the following target radiation therapy treatment planning tasks: generating automatic segmentation of structure data based on image data; generating dose predictions of dose data based on the structural data and the image data; and generating a processing delivery data prediction for processing the delivery data.

16. The computer system according to any one of claims 13 to 14, wherein the instructions for generating the target deep learning engine cause the processor to: configuring the plurality of base layers of the base deep learning engine as variable base layers; and The plurality of target layers are generated by modifying corresponding variable base layers based on target training data associated with the target radiation therapy treatment plan task.

17. The computer system according to any one of claims 13 to 14, wherein the instructions for generating the target deep learning engine cause the processor to: configuring the plurality of base layers of the base deep learning engine as invariant base layers; and The feature data is generated using the invariant base layer based on target training data associated with the target radiation therapy treatment planning task.

18. The computer system according to any one of claims 13 to 14, wherein the instructions for generating the target deep learning engine cause the processor to: The plurality of base layers of the base deep learning engine are configured to include both immutable base layers and variable base layers.

19. A method for a computer system for deep transfer learning-based radiation therapy treatment planning, wherein the method comprises: providing a first user interface for sending a request to generate a target deep learning engine to perform a target radiation therapy treatment plan task; In response to detecting a request to generate the target deep learning engine via the first user interface, selecting a base deep learning engine from a plurality of base deep learning engines that are pre-trained to perform corresponding base radiation therapy treatment planning tasks; Based on the selected base deep learning engine and target training data associated with a plurality of past patients, using the method according to any one of claims 1 to 6, to generate the target deep learning engine to perform the target radiation therapy treatment planning task; as well as A second user interface is provided for accessing the target deep learning engine to perform the target radiation therapy treatment planning task for an individual patient.

20. The method of claim 19, wherein selecting the base deep learning engine comprises: The base deep learning engine is selected based on selection parameter data identifying one or more of: an identifier of the base deep learning engine, an anatomical location, a structure from the anatomical location, a performance metric, and a base engine provider.

21. The method according to claim 19 or 20, wherein generating the target deep learning engine comprises: generating the target deep learning engine based on at least one of the following data configurable via the first user interface: structure mapping data that maps a base name associated with a base engine to a target name associated with a target engine; The training parameter data for the guided training phase and the validation parameter data for the guided validation phase.

22. The method according to claim 21, wherein generating the target deep learning engine comprises: Based on the verification parameter data, verification of the target deep learning engine is performed.

23. The method according to any one of claims 19 to 20, wherein generating the target deep learning engine comprises: configuring a variable base layer among the selected multiple base layers of the base deep learning engine; as well as A target layer among a plurality of target layers of the target deep learning engine is generated by modifying the variable base layer based on the target training data.

24. The method according to any one of claims 19 to 20, wherein generating the target deep learning engine comprises: configuring an invariant base layer among the selected multiple base layers of the base deep learning engine; as well as A target layer among a plurality of target layers of the target deep learning engine is generated based on feature data, wherein the feature data is generated based on the target training data using the invariant base layer.

25. The method according to any one of claims 19 to 20, wherein the method further comprises: In response to detecting the request via the second user interface, instructing the target deep learning engine to perform one of the following target radiation therapy treatment planning tasks based on input data associated with the individual patient: generating automatic segmentation of structural data based on image data associated with the individual patient; generating a dose prediction of dose data based on structural data and image data associated with the individual patient; and generating a treatment delivery data prediction of treatment delivery data for the individual patient.

26. A non-transitory computer-readable storage medium comprising a set of instructions that, in response to being executed by a processor of a computer system, causes the processor to perform a method for deep transfer learning-based radiation therapy treatment planning, wherein the method comprises: providing a first user interface for sending a request to generate a target deep learning engine to perform a target radiation therapy treatment plan task; In response to detecting a request to generate the target deep learning engine via the first user interface selecting a base deep learning engine from a plurality of base deep learning engines that are pre-trained to perform corresponding base radiation therapy treatment planning tasks; Based on the selected base deep learning engine and target training data associated with a plurality of past patients, using the method according to any one of claims 1 to 6, to generate the target deep learning engine to perform the target radiation therapy treatment planning task; as well as A second user interface is provided for accessing the target deep learning engine to perform the target radiation therapy treatment planning task for an individual patient.

27. The non-transitory computer-readable storage medium of claim 26, wherein selecting the base deep learning engine comprises: The base deep learning engine is selected based on selection parameter data identifying one or more of: an identifier of the base deep learning engine, an anatomical location, a structure from the anatomical location, a performance metric, and a base engine provider.

28. The non-transitory computer-readable storage medium of claim 26 or 27, wherein generating the target deep learning engine comprises: generating the target deep learning engine based on at least one of the following data configurable via the first user interface: structure mapping data that maps a base name associated with a base engine to a target name associated with a target engine; The training parameter data for the guided training phase and the validation parameter data for the guided validation phase.

29. The non-transitory computer-readable storage medium of claim 28, wherein generating the target deep learning engine comprises: Based on the verification parameter data, verification of the target deep learning engine is performed.

30. The non-transitory computer-readable storage medium of any one of claims 26 to 27, wherein generating the target deep learning engine comprises: configuring a variable base layer among the selected multiple base layers of the base deep learning engine; as well as A target layer among a plurality of target layers of the target deep learning engine is generated by modifying the variable base layer based on the target training data.

31. The non-transitory computer-readable storage medium of any one of claims 26 to 27, wherein generating the target deep learning engine comprises: configuring an invariant base layer among the selected multiple base layers of the base deep learning engine; as well as A target layer among a plurality of target layers of the target deep learning engine is generated based on feature data, wherein the feature data is generated based on the target training data using the invariant base layer.

32. The non-transitory computer readable storage medium of any one of claims 26 to 27, wherein the method further comprises: In response to detecting the request via the second user interface, instructing the target deep learning engine to perform one of the following target radiation therapy treatment plans based on input data associated with the individual patient: generating automatic segmentation of structural data based on image data associated with the individual patient; generating dose prediction of dose data based on structural data and image data associated with the individual patient; and generating a treatment delivery data prediction of treatment delivery data for the individual patient.

33. A computer system configured to perform radiation therapy treatment planning based on deep transfer learning, wherein the computer system comprises a processor and a non-transitory computer readable medium having instructions stored thereon that, when executed by the processor, cause the processor to perform: providing a first user interface for sending a request to generate a target deep learning engine to perform a target radiation therapy treatment plan task; In response to detecting a request to generate the target deep learning engine via the first user interface, selecting a base deep learning engine from a plurality of base deep learning engines that are pre-trained to perform corresponding base radiation therapy treatment planning tasks; Based on the selected base deep learning engine and target training data associated with a plurality of past patients, using the method according to any one of claims 1 to 6, to generate the target deep learning engine to perform the target radiation therapy treatment planning task; as well as A second user interface is provided for accessing the target deep learning engine to perform the target radiation therapy treatment planning task for an individual patient.

34. The computer system of claim 33, wherein the instructions for selecting the base deep learning engine cause the processor to: The base deep learning engine is selected based on selection parameter data identifying one or more of: an identifier of the base deep learning engine, an anatomical location, a structure from the anatomical location, a performance metric, and a base engine provider.

35. The computer system of claim 33 or 34, wherein the instructions for generating the target deep learning engine cause the processor to: The target deep learning engine is generated based on at least one of the following data that can be configured via the first user interface: structural mapping data that maps a base name associated with a base engine to a target name associated with a target engine; training parameter data to guide a training phase and verification parameter data to guide a verification phase.

36. The computer system of claim 35, wherein the instructions for generating the target deep learning engine cause the processor to: Based on the verification parameter data, verification of the target deep learning engine is performed.

37. The computer system of any one of claims 33 to 34, wherein the instructions for generating the target deep learning engine cause the processor to: configuring a variable base layer among the selected multiple base layers of the base deep learning engine; and A target layer among a plurality of target layers of the target deep learning engine is generated by modifying the variable base layer based on the target training data.

38. The computer system of any one of claims 33 to 34, wherein the instructions for generating the target deep learning engine cause the processor to: configuring an invariant base layer among the selected multiple base layers of the base deep learning engine; and A target layer among a plurality of target layers of the target deep learning engine is generated based on feature data, wherein the feature data is generated based on the target training data using the invariant base layer.

39. The computer system of any one of claims 33 to 34, wherein the instructions further cause the processor to: In response to detecting the request via the second user interface, instructing the target deep learning engine to perform one of the following target radiation therapy treatment planning tasks based on input data associated with the individual patient: generating automatic segmentation of structural data based on image data associated with the individual patient; generating a dose prediction of dose data based on structural data and image data associated with the individual patient; and generating a treatment delivery data prediction of treatment delivery data for the individual patient.

Citation Information

Patent Citations

  • Methods and systems for radiotherapy treatment planning using deep learning engines

    US20200104695A1

  • Automated organ risk segmentation machine learning methods and systems

    US20180315188A1

  • Neural network for generating synthetic medical images

    WO2018048507A1