Pre-training method, pre-training system, training method and training system of dose distribution prediction model
Through pre-training and fine-tuning techniques, the problem of poor generalization between different targets by existing dose prediction models is solved, efficient dose prediction in different body parts is achieved, and the efficiency of radiotherapy plans is improved.
Patent Information
- Application Number
- CN202310533773.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-12
AI Technical Summary
Existing dose prediction models have poor generalization between different targets and are difficult to adapt to different treatment tasks, resulting in poor dose prediction in different body parts.
Pretrained models and fine-tuning methods are used to obtain pretrained models under multiple tasks through meta-learning or transfer learning, and fine-tune them through a small number of samples under the target task to generate a dose distribution prediction model for a specific task.
The good generalization of different target tasks was achieved under the condition of sample scarcity, which improved the accuracy and efficiency of dose prediction, and shortened the time for formulating radiotherapy plans.
Smart Images

Figure CN118942629A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning, and in particular to a pre-training method, a pre-training system, a training method and a training system for a dose distribution prediction model. Background Art
[0002] Radiotherapy (RT) is one of the important means of cancer treatment. Its main purpose is to ensure that the radiation dose received by the target area reaches the prescribed dose and reduce the radiation dose received by the surrounding normal organs. With the development of advanced treatment methods such as IMRT (Intensity-Modulated Radiation Therapy) and VMAT (Volumetric-Modulated Arc Therapy), the quality of radiotherapy plans has been significantly improved. However, the formulation of high-quality radiotherapy plans usually requires a lot of manual intervention. Physicists can repeatedly adjust the planning parameters based on their own experience and feedback from doctors, which leads to a longer planning time, which is not conducive to tumor control and patient survival. With the development of machine learning technology, a machine learning model (such as a deep learning model) can be trained to predict the dose distribution in the patient's body (referred to as dose prediction). Dose prediction can provide a reasonable initial optimization target, which can effectively shorten the formulation time of radiotherapy plans.
[0003] However, some current machine learning solutions for dose prediction have poor generalization and are difficult to adapt to different goals (or tasks). For example, the effect of applying the dose distribution prediction model originally used for the nasopharynx to the dose prediction of the prostate is often very poor. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a dose prediction method applicable to multiple targets (tasks).
[0005] One of the embodiments of this specification provides a method for pre-training a dose distribution prediction model, the method may include: for each of a plurality of tasks, obtaining one or more samples under the task; obtaining a pre-trained model based on the samples under the plurality of tasks, so that: adjusting the pre-trained model based on one or more samples under the target task, being able to obtain a dose distribution prediction model for the target task, and outputting the predicted dose distribution information under the target task for the dose distribution prediction model of the target task. The label of the sample under any task may include the annotated dose distribution information under the task.
[0006] In some embodiments, the pre-trained model can be obtained by methods such as meta-learning methods, transfer learning methods, etc.
[0007] In some embodiments, each task may correspond to a patient or a group of patients, and the dose distribution prediction model for any task outputs the predicted dose distribution information of the patient corresponding to the task. The model input of the sample under any task includes the reference information of the patient corresponding to the task, and the label of the sample under any task includes the annotated dose distribution information of the patient corresponding to the task.
[0008] In some embodiments, each task may correspond to a weight combination, each weight combination may include a planned target area weight and an organ at risk weight, the planned target area weight indicates the dose requirement of the planned target area, and the organ at risk weight indicates the dose requirement of the organ at risk. The model input of the sample under any task may include the weight combination corresponding to the task, and the label of the sample under any task may include the annotated dose distribution information that meets the dose requirement under the task.
[0009] In some embodiments, each task may correspond to a body part, and the dose distribution prediction model for any task may output the predicted dose distribution information of the body part corresponding to the task. The model input of the sample under any task may include an image of the body part corresponding to the task, and the label of the sample under any task may include the annotated dose distribution information of the body part corresponding to the task.
[0010] In some embodiments, the input of the dose distribution prediction model may include one or more of the following information: an image of a body part, contour information of a planned target volume and contour information of an organ at risk, dose requirements of the planned target volume, and dose requirements of an organ at risk.
[0011] In some embodiments, the pre-trained model may include a conditional generative adversarial network, the generator in the conditional generative adversarial network adopts a U-Net structure consisting of an encoder and a decoder, and the generator and discriminator in the conditional generative adversarial network have the same number of downsampling times.
[0012] One of the embodiments of this specification provides a pre-training system for a dose distribution prediction model, the system may include: a training data acquisition module, for obtaining one or more samples under each of multiple tasks; a pre-training module, for obtaining a pre-training model based on the samples under multiple tasks, so that: the pre-training model is adjusted based on one or more samples under a target task, so as to obtain a dose distribution prediction model for the target task, and the dose distribution prediction model for the target task outputs the predicted dose distribution information under the target task. The label of the sample under any task may include the annotated dose distribution information under the task.
[0013] One of the embodiments of the present specification provides a method for training a dose distribution prediction model, which may include: obtaining a pre-trained model; obtaining one or more samples under a target task, wherein the labels of the samples under the target task may include annotated dose distribution information under the target task; adjusting the pre-trained model based on the one or more samples under the target task to obtain a dose distribution prediction model for the target task, and outputting the predicted dose distribution information under the target task according to the dose distribution prediction model for the target task.
[0014] In some embodiments, the one or more samples can be obtained based on an existing treatment plan. The method can also include: obtaining reference information of a target patient; inputting the reference information of the target patient into a dose distribution prediction model for the target task to obtain predicted dose distribution information under the target task, and the predicted dose distribution information can be used to generate a new treatment plan for the target patient.
[0015] One of the embodiments of the present specification provides a training system for a dose distribution prediction model, the system may include: a model acquisition module, used to obtain a pre-trained model; a training data acquisition module, used to obtain one or more samples under a target task, the labels of the samples under the target task may include annotated dose distribution information under the target task; a model adjustment module, used to adjust the pre-trained model based on one or more samples under the target task to obtain a dose distribution prediction model for the target task, and the dose distribution prediction model for the target task can output the predicted dose distribution information under the target task.
[0016] One of the embodiments of the present specification provides a computer-readable storage medium, wherein the storage medium stores instructions. When a computer executes the instructions in the storage medium, the method described in any embodiment of the present specification can be implemented.
[0017] The embodiments of this specification provide a pre-training method, a pre-training system, a training method and a training system for a dose distribution prediction model. In the pre-training stage, the pre-training model can be obtained by a meta-learning method. In the fine-tuning stage, the pre-training model can be adjusted by a small number of samples under the target task to obtain a dose distribution prediction model for the target task. For different targets (such as dose prediction for different parts), the pre-training model can be fine-tuned using samples under different tasks in the fine-tuning stage. In this way, good generalization for different targets (tasks) can be achieved under the condition of scarce samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:
[0019] Figure 1 is a schematic diagram of an application scenario of a treatment planning system according to some embodiments of this specification;
[0020] Figure 2 is an exemplary flow chart of a pre-training method of a dose distribution prediction model according to some embodiments of this specification;
[0021] Figure 3 is an exemplary flow chart of the pre-training stage according to some embodiments of this specification;
[0022] Figure 4 is an exemplary flow chart of a method for training a dose distribution prediction model according to some embodiments of this specification;
[0023] Figure 5 is a data flow diagram of a conditional generative adversarial network according to some embodiments of this specification;
[0024] Figure 6 is a schematic diagram of the structure of a conditional adversarial network according to some embodiments of this specification;
[0025] Figure 7 is a schematic diagram of the structure of a generator according to some embodiments of this specification;
[0026] Figure 8 is an exemplary module diagram of a pre-training system for a dose distribution prediction model according to some embodiments of this specification;
[0027] Fig. 9 is an exemplary module diagram of a training system for a dose distribution prediction model according to some embodiments of the present specification. DETAILED DESCRIPTION
[0028] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.
[0029] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.
[0030] As shown in this specification, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.
[0031] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.
[0032] Figure 1 Schematic diagram of application scenarios of the treatment planning system according to some embodiments of this specification. Figure 1 As shown, the system 100 may include a data collection device 110 , a processing device 120 , a user terminal 130 and a network 140 .
[0033] The data acquisition device 110 is used to collect relevant data of the patient, where the relevant data may refer to data related to dose prediction. In some embodiments, the relevant data of the patient may include medical imaging data of the patient, for example, a whole body scan image of the patient, an image of a region of interest (ROI), etc.
[0034] In some embodiments, the data acquisition device 110 may include one or more medical devices such as CT (Computed Tomography) equipment, PET (Positron Emission Computed Tomography) equipment, MRI (Magnetic Resonance Imaging) equipment, and ultrasonic imaging equipment.
[0035] The processing device 120 can be used to generate a treatment plan, such as a radiotherapy plan. Specifically, the processing device 120 can obtain relevant data of the patient from the data acquisition device 110, input the relevant data of the patient into the trained dose distribution prediction model, and obtain the patient's predicted dose distribution information. Furthermore, the processing device 120 can generate a treatment plan for the patient based on the patient's predicted dose distribution information.
[0036] The processing device 120 can also be used to obtain a dose distribution prediction model under the meta-learning framework. The process of obtaining the dose distribution prediction model can be divided into a pre-training stage and a fine-tuning stage. For more details about the pre-training stage, please refer to Figure 2 , Figure 3 For more details about the fine-tuning phase, see Figure 4 and its related description.
[0037] In some embodiments, the processing device 120 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 120 may be local or remote. In some embodiments, the processing device 120 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof.
[0038] The user terminal 130 may include various devices with input / output functions, such as a smart phone 130-1, a tablet computer 130-2, a laptop computer 130-3, a desktop computer 130-4, etc. In some embodiments, the user terminal 130 may provide a service interface, such as a graphical user interface.
[0039] In some embodiments, the user may send one or more instructions of starting / stopping scanning, configuring scanning parameters, importing / exporting data, etc. to the data acquisition device 110 through the user terminal 130 .
[0040] In some embodiments, the user may set one or more parameters related to model training through the user terminal 130, such as the model category, structure, number of input / output channels, iteration termination conditions, etc. In some embodiments, the user may define a task set through the user terminal 130, and the task set may include multiple tasks. In some embodiments, the user may import a sample set through the user terminal 130, and the sample set may include multiple samples, each of which may include a model input (denoted as x) and a sample label (denoted as y).
[0041] In some embodiments, the user may obtain the predicted dose distribution information output by the dose distribution prediction model through the user terminal 130. In some embodiments, the user may view, modify, copy, and the like the treatment plan of the patient through the user terminal 130.
[0042] The network 140 connects the components of the system 100 so that the components can communicate with each other. In some embodiments, the processing device 120 can obtain relevant data of the patient from the data acquisition device 110 through the network 140 to achieve dose prediction. In some embodiments, the processing device 120 can send the measured dose distribution information output by the dose distribution prediction model to the user terminal 130 through the network 140 for the user to obtain. In some embodiments, the user terminal 130 can send the information and / or data input by the user, such as parameters, task sets, sample sets, etc. related to model training, to the processing device 120 through the network 140 for use by the processing device 120.
[0043] The network between the parts of the system 100 may include a wired network and / or a wireless network. For example, the network 140 may include a cable network, a wired network, an optical fiber network, a telecommunication network, an internal network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network (ZigBee), a near field communication (NFC), a bus within a device, a line within a device, a cable connection, etc. or any combination thereof. The network connection between each two parts may be in one of the above-mentioned ways, or in multiple ways.
[0044] Figure 2 is an exemplary flow chart of a pre-training method of a dose distribution prediction model according to some embodiments of this specification. In some embodiments, process 200 may be performed by Figure 1 The processing device 120 shown in the figure may be executed by Figure 8 The pre-training system 800 shown is implemented on the processing device 120. Figure 2 As shown, process 200 may include the following steps.
[0045] Step 210 , for each of the multiple tasks, obtain one or more samples under the task. In some embodiments, step 210 may be performed by the training data acquisition module 810 .
[0046] Meta-learning is based on tasks. Tasks refer to the goal of model training. In the pre-training stage, they are called training tasks, and in the fine-tuning stage, they are called testing tasks or prediction tasks.
[0047] As a non-limiting example, under the meta-learning framework, a corresponding dose distribution prediction model can be obtained based on any task, that is, a dose distribution prediction model for a single task can be obtained through meta-learning. Specifically, assuming there are N tasks, denoted as T 1 ~T N , for task T i (1≤i≤N), through meta-learning, we can obtain i The dose distribution prediction model (denoted as M i ). In addition, the dose distribution prediction model of each of the multiple tasks can be obtained by transfer learning and other methods. The dose distribution prediction model for any task outputs the predicted dose distribution information under the task. For example, for task T i Dose distribution prediction model M i Output task T i Accordingly, the label of a sample under any task may include the annotated dose distribution information under the task, for example, task T i The labels of the samples under task T can include i The labeled dose distribution information under this specification is not specifically limited to the labeling method of the sample. As an example only, in actual applications, the labeled dose distribution information can be the dose distribution information calculated based on the treatment plan using clinically verified simulation software, and of course it can also be the real dose distribution information (for example, actually measured by any known or future technical means).
[0048] The plurality of tasks in step 210 may be selected from a user-defined task set.
[0049] In some embodiments, each task may correspond to a patient, for example, task one may correspond to patient A, task two may correspond to patient B, and so on. At this time, a single task may be understood as performing radiotherapy dose prediction for a specific patient, that is, the dose distribution prediction model for any task may output the predicted dose distribution information of the patient corresponding to the task. When each task corresponds to a patient, the model input of the sample under any task may include the reference information of the patient corresponding to the task (for example, a patient image), and the label of the sample under any task may include the annotated dose distribution information of the patient corresponding to the task. In addition, in some embodiments, each task may correspond to a group of patients, for example, task one may correspond to patient group A, task two may correspond to patient group B, and so on. At this time, a single task may be understood as performing radiotherapy dose prediction for patients belonging to a specific patient group, that is, the dose distribution prediction model for any task may output the predicted dose distribution information of the patient belonging to the patient group corresponding to the task (one of the patients corresponding to the task). When each task corresponds to a group of patients, the model input of the sample under any task may include the reference information of one of the patients corresponding to the task (for example, a patient image), and the label of the sample under any task may include the annotated dose distribution information of one of the patients corresponding to the task.
[0050] In some embodiments, each task may correspond to a body part (also referred to as a part of interest), for example, task one may correspond to the lungs, task two may correspond to the nasopharynx, task three may correspond to the cervix, and so on. At this point, a single task may be understood as radiotherapy dose prediction for a specific body part, that is, the dose distribution prediction model for any task outputs the predicted dose distribution information of the body part corresponding to the task. When each task corresponds to a body part, the model input of the sample under any task may include an image of the body part corresponding to the task, and the label of the sample under any task includes the annotated dose distribution information of the body part (such as the lungs) corresponding to the task.
[0051] In some embodiments, each task may correspond to a weight combination. The weight combination includes a planning target volume (PTV) weight and an organ at risk (OAR) weight, which may also be referred to as a PTV weight and an OAR weight. The planning target volume (PTV) refers to the irradiation range that is expanded during radiotherapy planning due to changes in organ movement during irradiation and target position and target volume during positioning. Organs at risk (OAR) refer to organs that need to be protected around the planning target volume. The PTV weight indicates the dose requirement of the planning target volume (PTV), and the OAR weight indicates the dose requirement of the organ at risk (OAR). The units of these two weights may be radiation dose units Gy. At this point, a single task may be understood as predicting radiotherapy doses according to specific dose requirements. When each task corresponds to a weight combination, the model input of the sample under any task may include the weight combination corresponding to the task, and the label of the sample under any task may include annotated dose distribution information that meets the dose requirements under the task.
[0052] Depending on individual differences in cases or the style of doctors, doctors may have different trade-offs between target area prescription coverage and organ-at-risk limit during treatment. Some cases (or doctors) may place more emphasis on target area coverage, while some cases (doctors) may place more emphasis on organ-at-risk limit. Therefore, predicting only one certain dose distribution is difficult to meet the diverse treatment needs of doctors. Therefore, this patent proposes a multi-target dose prediction model. By combining meta-learning technology, the model proposed in this patent can predict the 3D dose distribution corresponding to the input weights based on the different weights of the input target area and organ-at-risk.
[0053] It should be understood that the present specification does not impose any specific restrictions on the definition of tasks. Referring to the above embodiments, a single task may correspond to any combination of patients, body parts, and dose requirements. For example, a single task may be understood as predicting the radiotherapy dose for a specific body part of a specific patient according to a specific dose requirement.
[0054] Step 220 , based on samples under multiple tasks, obtain a pre-training model. In some embodiments, step 220 may be performed by the pre-training module 820 .
[0055] For example, through various methods such as meta-learning methods and transfer learning methods, it is hoped to obtain such a pre-trained model: the pre-trained model is adjusted based on one or more samples under the target task, so as to obtain a dose distribution prediction model for the target task. As an example, the purpose of meta-learning is to learn a set of "good" initialization parameters, which are the parameters of the pre-trained model, so that the model can converge quickly on a small-scale sample (even a single sample).
[0056] For more details about the pre-training phase, please refer to Figure 3 and its related description.
[0057] Figure 3 is an exemplary flow chart of the pre-training phase according to some embodiments of this specification. Figure 3 As shown, process 300 may include the following steps.
[0058] Step 310, initializing model parameters.
[0059] Step 320: extract a group of tasks from the task set.
[0060] In some embodiments, the task set may be divided into several subsets, and then a task is randomly selected from each subset to extract a group of tasks. In some embodiments, tasks may be extracted from the task set in multiple times, with one task being extracted each time, to extract a group of tasks. In some embodiments, a group of tasks may also be extracted from the task set at one time.
[0061] As a non-limiting example, the difference from conventional machine learning methods is that meta-learning uses the sample set under each task as a piece of training data, rather than each sample as a piece of training data. Therefore, when the batch stochastic gradient descent method is used to adjust the model parameters, the set of tasks extracted in step 320 corresponds to a batch of training data.
[0062] For the convenience of description, the extracted tasks can be recorded as , the tasks in the task set follow the distribution Right now
[0063] For each task in the set of tasks , the processing device 120 may perform a first gradient calculation and a first parameter update, namely step 330 and step 340 .
[0064] Step 330, from the task Several samples are extracted from the support set to perform the first gradient calculation.
[0065] The gradient of the first calculation can be recorded as , where θ represents the model parameter (f θ i.e. model), represents the loss function.
[0066] In the pre-training stage, the sample set of each training task can be divided into a support set and a query set. The support set can be used for the first gradient calculation, and the query set can be used for the second gradient calculation.
[0067] Step 340: Perform a first parameter update based on the gradient calculated for the first time.
[0068] The first parameter update can be written as Among them, θ′ i represents the updated parameters, θ represents the parameters before the update, and α represents the learning rate of the first parameter update.
[0069] Step 350 , extract a number of samples from the query set of the task group and perform a second gradient calculation.
[0070] The result of the first parameter update θ′ i For the second gradient calculation, the gradient of the second calculation can be recorded as Among them, the tasks involved in summation For all tasks in this group of tasks It can be seen that the gradient calculated for the first time is obtained based on the loss function of a single task, and the gradient calculated for the second time is obtained based on the sum of the loss functions of multiple tasks.
[0071] Step 360: Perform a second parameter update based on the gradient calculated for the second time.
[0072] The second parameter update can be written as Among them, β represents the learning rate of the second parameter update. It should be noted that the starting point of the second parameter update is the model parameter θ before the first parameter update.
[0073] In some embodiments, process 300 may be iteratively executed, and the model parameters obtained in each iteration (model parameters updated in step 360) may be used as the initialization model parameters for the next iteration until the expected model parameters are obtained. In particular, in the first iteration, the model parameters may be randomly initialized.
[0074] The loss function in process 300 can be set according to the model properties. For example, for a regression model, the loss function can be constructed based on mean square error. For another example, for a classification model, the loss function can be constructed based on cross entropy.
[0075] Just as an example, for a generative adversarial network, the loss function can be calculated as follows:
[0076]
[0077]
[0078]
[0079] in, represents the loss function, represents the generative adversarial loss, represents the mean square error (MSE) loss; G represents the generator, D represents the discriminator, x represents the input of the generative adversarial network, y represents the sample label (i.e., the labeled dose distribution information); E represents the expectation, log represents the logarithmic function, ‖ ‖ 2 represents the 2-norm.
[0080] Figure 4 is an exemplary flow chart of a method for training a dose distribution prediction model according to some embodiments of this specification. In some embodiments, process 400 may be performed by Figure 1 The processing device 120 shown in the figure may be executed by Fig. 9 The training system 900 shown is implemented on the processing device 120. Figure 4 As shown, process 400 may include the following steps.
[0081] Step 410 , obtaining a pre-trained model. In some embodiments, step 410 may be performed by the model acquisition module 910 .
[0082] The pre-trained model can be obtained by various methods such as a meta-learning method, a transfer learning method, etc. More details about the pre-trained model and meta-learning can be found in the relevant description of process 200.
[0083] In the prediction scenario, the model training party (such as a software service provider) can train the pre-trained model in advance and send the pre-trained model to one or more prediction parties (such as hospitals). One or more prediction parties can fine-tune the same pre-trained model according to their respective task requirements to obtain a dose distribution prediction model for the prediction task (i.e., the target task).
[0084] Step 420 , obtaining one or more samples under the target task. In some embodiments, step 420 may be performed by the training data acquisition module 920 .
[0085] In the fine-tuning stage, several tasks (called test tasks) can be extracted from the task set to test the effect of the pre-trained model. Specifically, the test can refer to the prediction accuracy of the pre-trained model on the new task after fine-tuning. That is, in the test scenario, the target task can refer to the test task.
[0086] In the prediction scenario, that is, after the dose distribution prediction model is launched, the target task can refer to the prediction task. For example, the target task can refer to providing dose prediction services for patients who are about to receive radiotherapy.
[0087] In the fine-tuning phase, each test task (denoted as ) can also be divided into a support set and a query set. The support set can be used for fine-tuning the model parameters (i.e., step 430), and the query set can be used to test the fine-tuned model (i.e., the dose distribution prediction model) in the task The prediction accuracy on .
[0088] In some embodiments, the one or more samples may be obtained based on an existing treatment plan for the target patient.
[0089] In some embodiments, when each task corresponds to a patient, it is necessary to use the sample of the target patient to fine-tune the pre-trained model to obtain a dose distribution prediction model for the target patient. Generally, the sample of the target patient can be queried in the database. When the database lacks the data of the target patient, for example, the target patient has no historical treatment plan and cannot provide the real dose distribution information as a sample label, the training data acquisition module 920 can determine the samples of other patients whose similarity with the reference information of the target patient meets the preset conditions in the database, and use them as the samples of the target patient. As an example only, the reference information may include image data, and the preset conditions may include that the similarity between the image data and the image data of the target patient exceeds a first similarity threshold (such as 90%) and the dose requirements (such as PTV weight and OAR weight) and the dose requirements of the target patient exceed a second similarity threshold (such as 90%). However, without limitation thereto, the reference information may also include other data or information such as characteristic information of the target patient (such as age, height, weight, etc.), historical diagnostic data (such as historical illness), physiological indicators (such as heartbeat, pulse, blood oxygen saturation, blood pressure, etc.).
[0090] Step 430 , adjusting the pre-trained model based on one or more samples under the target task to obtain a dose distribution prediction model for the target task. In some embodiments, step 430 may be performed by a fine-tuning module 930 .
[0091] Since the pre-trained model has good generalization ability, the model can be quickly adapted to different tasks with a small number of samples.
[0092] In some embodiments, the input of the dose distribution prediction model may include one or more of the following information: an image of a body part (part of interest, such as the lungs), contour information of the planned target area (or referred to as delineation information), contour information of organs at risk, dose requirements of the planned target area (such as PTV weight), and dose requirements of organs at risk (such as OAR weight). The contour information of the planned target area (PTV) / organ at risk (OAR) may indicate the contour of the PTV / OAR. Specifically, the contour information of the planned target area (PTV) / organ at risk (OAR) may be expressed as a segmented image of the PTV / OAR, which may be segmented from an image of the part of interest (such as the lungs). With regard to the input of the pre-trained model and the model input of the sample, reference may be made to the above examples.
[0093] According to the foregoing, in some embodiments, the one or more samples may be obtained based on an existing treatment plan for the target patient. Accordingly, the processing device 120 may also obtain reference information (such as image data) of the target patient, input the reference information of the target patient into a dose distribution prediction model for the target task, and obtain predicted dose distribution information under the target task, and the predicted dose distribution information may be used to generate a new treatment plan for the target patient.
[0094] It should be noted that the principles of this specification are applicable to any machine learning model, for example, linear regression model, logistic regression model, vector support machine, decision tree, random forest, Bayesian model, neural network (such as deep learning model), etc.
[0095] In some embodiments, the pre-trained model may include a conditional generative adversarial network. The generative adversarial network includes a generator and a discriminator. Compared with an ordinary generative adversarial network, the input of the conditional generative adversarial network also contains conditional information. For the dose prediction problem, the input of the conditional generative adversarial network (i.e., the input of the generator) may include images of body parts, contour information of the planned target area, contour information of the organs at risk, weights of the planned target area and weights of the organs at risk, and the output of the generative adversarial network (i.e., the output of the generator) may include predicted dose distribution information. It is worth noting that in the prediction scenario, the dose distribution prediction model may only retain the generator in the (conditional) generative adversarial network, or shield the discriminator. However, not limited to this, the pre-trained model may also include other pre-trained models, such as end-to-end convolutional neural networks (such as ResNet, DenseNet, U-Net, etc.), Bayesian neural networks, etc.
[0096] Figure 5 It is a data flow diagram of generating an adversarial network based on the conditions shown in some embodiments of this specification.
[0097] like Figure 5As shown in the figure, the data flow is divided into a generation stage and a discriminant stage. In the generation stage, the image of the body part (ROI image), PTV / OAR weight, and PTV / OAR contour information are sent to the generator, and the generator generates predicted dose distribution information under the guidance of these inputs. In the discriminant stage, on the one hand, the image of the body part, PTV / OAR weight, PTV / OAR contour information, and the generated predicted dose distribution information are sent to the discriminator as negative samples, and on the other hand, the image of the body part, PTV / OAR weight, PTV / OAR contour information, and the annotated dose distribution information are sent to the discriminator as positive samples. During the training process, the parameters of the generator are continuously optimized to generate dose distribution information that tends to be true (the annotated dose distribution information can be regarded as the true dose distribution information) to deceive the discriminator, and the parameters of the discriminator are continuously optimized to distinguish the annotated dose distribution information from the generated predicted dose distribution information under certain conditions. Finally, the generator can generate predicted dose distribution information close to the annotated dose distribution information (in layman's terms, "fake is real"), and the discriminator can accurately distinguish the annotated dose distribution information from the generated predicted dose distribution information.
[0098] Figure 6 It is a schematic diagram of the structure of a conditional adversarial network according to some embodiments of this specification.
[0099] refer to Figure 6 , the generator in the conditional generative adversarial network can include a downsampling part and an upsampling part. The rectangular bars in the figure represent image features. The longer the rectangular bars, the larger the size of the image features. During the downsampling process (solid arrows), the size of the image features continues to decrease; during the upsampling process (solid arrows), the size of the image features continues to increase. In the generator, the number of downsampling times is the same as the number of upsampling times, for example, both are Figure 6 4 times as shown. Correspondingly, the discriminator in the conditional generative adversarial network can have the same number of downsampling, for example, Figure 6 4 times shown.
[0100] In some embodiments, the generator in the conditional generative adversarial network may further include a feature concatenation part, and the image features in the downsampling process may be concatenated layer by layer with the image features in the upsampling process. Figure 6As shown in the figure, the image features before the first downsampling can be spliced with the features after the fourth upsampling (dashed arrow), the image features before the second downsampling can be spliced with the features after the third upsampling (dashed arrow), the image features before the third downsampling can be spliced with the features after the second upsampling (dashed arrow), and the image features before the fourth downsampling can be spliced with the features after the first upsampling (dashed arrow). By splicing features, the details lost in the downsampling process, such as shallow features, edge features, etc., can be retained, thereby improving the accuracy of dose prediction.
[0101] Figure 7 It is a schematic diagram of the structure of the generator shown in some embodiments of this specification.
[0102] like Figure 7 As shown, the generator can adopt a U-Net structure consisting of an encoder (or encoding part) and a decoder (or decoding part). The encoding process may include multiple convolutions and multiple (such as 4) downsampling. First, after performing a preset number of (such as 2) convolutions (horizontal arrows), the first downsampling (downward arrow) is performed. Then, after each downsampling is performed, the preset number of (such as 2) convolutions are performed. The decoding process may include multiple (such as 4) upsampling and multiple convolutions. After each upsampling (upward arrow), the preset number of (such as 2) convolutions (horizontal arrows) are performed. In addition to receiving the high-dimensional features output by the encoder, the decoder can also receive the features of each layer of the encoder in a cascaded manner. Specifically, the features output by the kth layer (counting from top to bottom in the figure, such as the 1st layer) in the encoder are clipped (refer to Figure 7 , retaining the part in the dotted box), it can be concatenated with the features sent to the N+1-kth layer (counting from bottom to top in the figure, such as the 4th layer) in the decoder, and the concatenated result is used as the input of the convolution operation of this layer. In some embodiments, the processing logic of the discriminator can also include N times of downsampling (not shown), that is, the generator and the discriminator can have the same number of downsampling times.
[0103] It should be understood that Figure 6 and Figure 7 The model structure in is only for example and description. For example, the number of sampling times can also be set to other values, such as N=3 or 5.
[0104] In some embodiments, each convolution can be implemented by a 3D convolution layer and a subsequent rectified linear unit (RectifiedLinear Unit, ReLU function), downsampling can use a pooling operation (such as maximum pooling), and upsampling can use a deconvolution operation.
[0105] As an example only, the input of the encoder may include a tensor of size (shape) B×C×Z×H×W, where B is the batch size (i.e., the number of tasks contained in a set of tasks extracted in step 320), C is the number of channels, for example, C can be 3, the 3 channels are the image of the region of interest (ROI), PTV contour information and OAR contour information, and Z, H, and W are the depth, height, and width of the ROI, respectively. To ensure the consistency of the samples, Z, H, and W can all be set to 128. The output of the generator may include a predicted dose distribution map of size (shape) B×1×Z×H×W (e.g., 2×1×128×128×128). The input of the discriminator may include a tensor of size (shape) B×(C+1)×Z×H×W (e.g., 2×4×128×128×128), and the final discriminator may output 0 or 1 to indicate whether the input dose distribution map is a generated predicted dose distribution map or a true dose distribution map.
[0106] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.
[0107] Figure 8 800 is an exemplary module diagram of a pre-training system for a dose distribution prediction model according to some embodiments of the present specification. In some embodiments, the pre-training system 800 may be implemented on the processing device 120 .
[0108] like Figure 8 As shown, the pre-training system 800 may include a training data acquisition module 810 and a pre-training module 820 .
[0109] The training data acquisition module 810 may be used to: for each of the multiple tasks, acquire one or more samples under the task.
[0110] The pre-training module 820 can be used to: obtain a pre-training model based on samples under multiple tasks, so that: the pre-training model is adjusted based on one or more samples under the target task, so as to obtain a dose distribution prediction model for the target task, and the dose distribution prediction model for the target task outputs the predicted dose distribution information under the target task.
[0111] The label of a sample under any task may include the annotated dose distribution information under the task.
[0112] For more details about the pre-training system 800 and its modules, please refer to Figure 2 and its related description.
[0113] Fig. 9 900 is an exemplary module diagram of a training system for a dose distribution prediction model according to some embodiments of the present specification. In some embodiments, the training system 900 may be implemented on the processing device 120 .
[0114] like Fig. 9 As shown, the training system 900 may include a model acquisition module 910 , a training data acquisition module 920 , and a fine-tuning module 930 .
[0115] The model acquisition module 910 can be used to acquire a pre-trained model.
[0116] The training data acquisition module 920 may be used to acquire one or more samples under a target task, and the labels of the samples under the target task may include annotated dose distribution information under the target task.
[0117] The fine-tuning module 930 can be used to adjust the pre-trained model based on one or more samples under the target task to obtain a dose distribution prediction model for the target task. The dose distribution prediction model for the target task can output the predicted dose distribution information under the target task.
[0118] For more details about the training system 900 and its modules, please refer to Figure 4 and its related description.
[0119] It should be understood that Figure 8 , Fig. 9 The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or included in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the system and its modules of this specification. Not only can the hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc. be implemented, it can also be implemented with software such as executed by various types of processors, and it can also be implemented by a combination of the above hardware circuits and software (for example, firmware).
[0120] It should be noted that the above description of the system and its modules is only for the convenience of description, and this specification cannot be limited to the scope of the embodiments cited. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the various modules or form a subsystem to connect with other modules without deviating from this principle. For example, in some embodiments, the model acquisition module 910 and the training data acquisition module 920 can be different modules in a system, or one module can implement the functions of these two modules. For example, in some embodiments, the pre-training system 800 and the training system 900 can be two systems, or they can be combined into one system. Such variations are within the scope of protection of this specification.
[0121] The beneficial effects that may be brought about by the embodiments of this specification include but are not limited to: (1) Based on the meta-learning method, the model can quickly adapt to new tasks (such as new patients, new parts) by fine-tuning with a small number of samples or even a single sample, and achieve multi-target dose prediction under the condition of few samples; (2) Within the generative adversarial network, the features of each layer of the encoder are retained in a cascade manner to improve the model accuracy. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other possible beneficial effects.
[0122] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation on the embodiments of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to the embodiments of this specification. Such modifications, improvements and corrections are suggested in the embodiments of this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.
[0123] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.
[0124] In addition, it will be appreciated by those skilled in the art that the various aspects of the embodiments of this specification may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, the various aspects of the embodiments of this specification may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, the various aspects of the embodiments of this specification may be represented as a computer product located in one or more computer-readable media, the product including computer-readable program code.
[0125] A computer storage medium may include a propagated data signal containing computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination. A computer storage medium may be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit the program for use. The program code on the computer storage medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.
[0126] The computer program coding required for the operation of each part of the embodiment of this specification can be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as C language, VisualBasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages, etc. The program coding can be run completely on the user's computer, or run on the user's computer as an independent software package, or partly run on the user's computer and partly run on the remote computer, or run completely on the remote computer or processing equipment. In the latter case, the remote computer can be connected to the user's computer by any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., by the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).
[0127] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in the embodiments of this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of the embodiments of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing processing device or mobile device.
[0128] Similarly, it should be noted that in order to simplify the description of the embodiments disclosed in this specification, thereby helping to understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the objects of the embodiments of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.
[0129] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification is hereby incorporated by reference in its entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this application (currently or later attached to this application) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.
[0130] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of the embodiments of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.
Claims
1. A pre-training method for a dose distribution prediction model, characterized in that: include: For each of the multiple tasks, obtain one or more samples under the task; A pre-trained model is obtained based on samples under multiple tasks, so that: the pre-trained model is adjusted based on one or more samples under a target task to obtain a dose distribution prediction model for the target task, and the dose distribution prediction model for the target task outputs predicted dose distribution information under the target task; The label of a sample under any task includes the annotated dose distribution information under the task.
2. The method according to claim 1, characterized in that The pre-trained model is obtained through a meta-learning method.
3. The method according to claim 1, characterized in that Each task corresponds to a patient or a group of patients. The dose distribution prediction model for any task outputs the predicted dose distribution information of the patient corresponding to the task; the model input of the sample under any task includes the reference information of the patient corresponding to the task, and the label of the sample under any task includes the annotated dose distribution information of the patient corresponding to the task.
4. The method according to claim 1, characterized in that Each task corresponds to a weight combination, each weight combination includes a planned target weight and an organ at risk weight, the planned target weight indicates the dose requirement of the planned target, and the organ at risk weight indicates the dose requirement of the organ at risk; the model input of the sample under any task includes the weight combination corresponding to the task, and the label of the sample under any task includes the annotated dose distribution information that meets the dose requirement under the task.
5. The method according to claim 1, characterized in that Each task corresponds to a body part, and the dose distribution prediction model for any task outputs the predicted dose distribution information of the body part corresponding to the task; the model input of the sample under any task includes the image of the body part corresponding to the task, and the label of the sample under any task includes the annotated dose distribution information of the body part corresponding to the task.
6. The method according to claim 1, characterized in that The input of the dose distribution prediction model includes one or more of the following information: an image of a body part, contour information of a planned target volume and contour information of an organ at risk, a dose requirement of the planned target volume, and a dose requirement of an organ at risk.
7. A pre-training system for a dose distribution prediction model, characterized in that: include: A training data acquisition module is used to: for each of the multiple tasks, obtain one or more samples under the task; A pre-training module is used to: obtain a pre-training model based on samples under multiple tasks, so that: the pre-training model is adjusted based on one or more samples under a target task to obtain a dose distribution prediction model for the target task, and the dose distribution prediction model for the target task outputs predicted dose distribution information under the target task; The label of a sample under any task includes the annotated dose distribution information under the task.
8. A method for training a dose distribution prediction model, characterized in that: include: Get the pre-trained model; Acquire one or more samples under a target task, wherein labels of the samples under the target task include annotated dose distribution information under the target task; The pre-trained model is adjusted based on one or more samples under the target task to obtain a dose distribution prediction model for the target task, and the dose distribution prediction model for the target task outputs predicted dose distribution information under the target task.
9. A training system for a dose distribution prediction model, characterized in that: include: Model acquisition module, used to: obtain pre-trained models; A training data acquisition module, used to acquire one or more samples under a target task, wherein labels of the samples under the target task include annotated dose distribution information under the target task; A model adjustment module is used to adjust the pre-trained model based on one or more samples under the target task to obtain a dose distribution prediction model for the target task, and the dose distribution prediction model for the target task outputs the predicted dose distribution information under the target task.
10. A computer-readable storage medium storing instructions, characterized in that: When a computer executes the instructions in the storage medium, the method according to any one of claims 1 to 6 and 8 is implemented.
