Tumor field angle optimization method, device and medium based on deep reinforcement learning

Abstract

The application belongs to the field of intelligent medical treatment, and particularly relates to a tumor field angle optimization method, device and medium based on reinforcement learning. The method comprises the following steps: acquiring a target area image to be planned and an initial state; inputting the initial state into a tumor field angle planning model to output an optimized field planning, and the construction method of the tumor field angle planning model is as follows: S1: acquiring training set images and an initial action; S2: obtaining an initial state and an initial state score based on the initial action; S3: inputting the initial state into DRL to obtain a second action by sampling a new action; S4: obtaining a second state and a second state score based on the second action; S5: obtaining a reward of the second action based on the second state score; and S6: iteratively optimizing according to a state-action pair and a target reward until a stop condition is met to obtain a tumor field angle planning model. The DRL of the application can recommend a field planning with a high planning score according to a current state.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent medical treatment, and more particularly, to a tumor field angle optimization method, device, medium and program product based on deep reinforcement learning. BACKGROUND

[0002] There are many types of brain tumors, and different types of tumors have different responses to treatment. The heterogeneity of tumors increases the complexity and difficulty of treatment. Surgery is the main means of treating brain tumors, but due to the special location of tumors, some tumors are difficult to completely resect, and the risk of surgery is high. Chemotherapy drugs are difficult to penetrate the blood-brain barrier, limiting the effectiveness of chemotherapy. Radiotherapy plays an important role in the treatment of brain tumors, especially for tumors that cannot be completely resected and postoperative residual tumor cells, radiotherapy is an important adjuvant treatment method. Advanced precision radiotherapy techniques, such as intensity-modulated radiation therapy (IMRT), can more accurately act on tumors and reduce damage to surrounding normal tissues, improving treatment effectiveness.

[0003] The basic steps of radiotherapy plan design include: ①preprocessing the target area and critical organs, and outlining auxiliary structures; ②the physicist designs the field angle according to experience; ③the physicist sets the optimization conditions in the planning system, and the planning system executes dose optimization according to the position information of the target area and critical organs and the set field angle; ④the physicist adjusts the field angle and optimization conditions according to the optimization results and experience until the clinical requirements are met.

[0004] The prior art has made a lot of in-depth research on dose optimization, for example:

[0005] Patent CN202180034806.0 discloses using reinforcement learning to position dose-volume objectives in radiotherapy plan optimization, which aims to optimize dose-volume objectives through the method of reinforcement learning;

[0006] Patent CN202011575203.5 discloses a radiotherapy automatic planning system, an automatic planning method and a storage medium, which uses a deep learning neural network to iteratively optimize the dose parameters in the radiotherapy plan according to the error of the radiotherapy plan and dose prediction generated by the TPS;

[0007] Among them, the optimization of the field angle has received less attention.

[0008] However, the field angle parameter is not paid enough attention, and the radiation plan is often set by the physicist according to experience, and then the subsequent dose optimization is carried out, which often cannot achieve the best effect, especially for brain tumors, the radiation area is the human brain, and the requirement for the radiation dose distribution is higher. In clinical work, most plans can meet the clinical needs by adjusting the dose optimization parameters, especially other tumors, such as nasopharyngeal carcinoma, cervical cancer, prostate cancer, etc., and the field angle is divided. However, due to the particularity of the nervous system, the field angle is very important in the treatment of brain tumors, and a good field angle scheme can greatly reduce the dose of normal organs. And due to the multiple options of the field angle and the complex influencing factors, there is a lack of clear optimization index, and the trial and error time cost is high, so it has not become the main parameter of plan optimization. Therefore, in order to further improve the quality of brain tumor radiotherapy plan, an automatic method is urgently needed to generate a high-quality field angle scheme. SUMMARY

[0009] In view of the above problems, the present application provides a tumor field angle optimization method based on deep reinforcement learning, which uses a deep reinforcement learning algorithm to find the best angle of the field, improves the design efficiency of the radiotherapy plan, and reduces the radiation damage.

[0010] The present application (first aspect) discloses a method for constructing a tumor field angle plan model based on deep reinforcement learning, which comprises:

[0011] S1: acquiring a training set image and an initial action, wherein the image set contains a target region, and the action is a field angle plan;

[0012] S2: obtaining an initial state based on the initial action, and evaluating the initial state to obtain an initial state score;

[0013] S3: inputting the initial state into DRL to obtain a second action by sampling a new action;

[0014] S4: obtaining a second state based on the second action, and evaluating the second state to obtain a second state score;

[0015] S5: obtaining the reward of the second action based on the difference between the second state score and the initial state score;

[0016] S6: according to the initial state-second action pair and the target reward, the DRL repeatedly optimizes S3-S5 based on the maximization of the target reward until the stopping condition is met, to obtain a tumor field angle plan model, wherein the target reward includes the reward of at least one second action, and the initial state-second action pair is replaced by the second state-next second action pair after the iteration starts.

[0017] Further, the initial state and the second state fuse image position features, the image position features including positions of target regions and organs;

[0018] Optionally, the specific steps of obtaining the DRL initial state based on the initial action include:

[0019] Step 1: obtaining a position matrix based on the image, numbers in the position matrix representing target regions or organs;

[0020] Step 2: inputting the image and the initial action into a dose calculation model to obtain an initial dose distribution matrix;

[0021] Step 3: fusing the position matrix and the initial dose distribution matrix to obtain the initial state;

[0022] Optionally, the method of fusing the image position matrix and the initial dose distribution matrix is adding the image position matrix and the initial dose distribution matrix to obtain the initial state.

[0023] Further, obtaining the position matrix based on the image specifically includes: elements of the position matrix corresponding to voxels of the image; different numbers representing different target regions and organs, respectively; and filling the numbers of the target regions or organs corresponding to the voxels into corresponding elements of the matrix;

[0024] Optionally, obtaining the position matrix based on the image further includes delineating the image to obtain positions and boundaries of the target regions and the organs;

[0025] Optionally, when the target regions and the organs intersect in the image, the sum of the numbers corresponding to the target regions and the organs is filled into the corresponding elements of the position matrix;

[0026] Optionally, when the target regions intersect in the image, the number of the high-dose target region is filled into the corresponding elements of the position matrix;

[0027] Optionally, the method of fusing the initial dose distribution matrix and the position matrix is: scaling the initial dose distribution matrix to obtain a scaled dose matrix, and adding the scaled dose matrix and the position matrix to obtain the initial state;

[0028] Optionally, the position matrix represents multiple target regions, organs, extracorporeal parts, and intracorporeal non-target region non-organ parts by different numbers, respectively;

[0029] Optionally, the organs include: eyeballs, optic nerves, optic chiasm, and brain stems;

[0030] Optionally, the dose calculation model is a clinical planning system;

[0031] Optionally, the scaling is: based on the target zone dose, scaling to the decimal part at most 120% of the maximum prescription dose target zone;

[0032] Optionally, the target reward is the sum of sample rewards sampled by the deep reinforcement learning;

[0033] Optionally, the target reward is the sum of sample rewards sampled by the deep reinforcement learning from the experience replay pool.

[0034] Further, the Buo plan is the field angle of each irradiation field;

[0035] Optionally, the action space of DRL is the change interval of the field angle, and the field angle change interval is [0, 360];

[0036] Optionally, the field angle corresponds to N irradiation fields;

[0037] Optionally, the action space of DRL is the combination of the field angle change possibilities of N irradiation fields;

[0038] Optionally, the initial action is to divide the field angle change interval equally to N irradiation fields;

[0039] Optionally, the action space of deep reinforcement learning is discrete values in the field angle change interval according to the value interval;

[0040] Optionally, the value range of the field angle is represented as:

[0041]

[0042] Wherein a n is the nth real number of the optimal field angle of the output, An is the angle of the nth field, A n-1 is the angle of the (n-1)th field, A1 is the angle of the first field, the function F(x) unifies the field angle to 0-360°, and is discretized to an integer multiple of 5 as close as possible; N represents the number of irradiation fields, interval represents the value interval of the field angle; f represents the angle difference between A n-1 and A1.

[0043] Optionally, the value interval of the field angle is 30°;

[0044] Optionally, the state score includes an organ score, and the organ score is obtained based on an initial dose distribution matrix;

[0045] Optionally, the organ score is obtained based on the initial state;

[0046] Optionally, the organ score is one or more of the following organ at risk scores: lens, optic nerve, chiasma, brainstem.

[0047] Optionally, the organ score is calculated according to the radiation dose of the organ at risk;

[0048] Optionally, the organ score is calculated in the following manner:

[0049]

[0050] wherein Score represents the corresponding organ score, ScoreRange is the highest score that the current organ can obtain, Max is the maximum dose of the current organ, Low represents the zero-volume percentage of the organ score being 0, and High represents the full-volume percentage of the organ score being full marks;

[0051] Optionally, the organ score is calculated in the following manner: the radiation dose of each organ is obtained by scaling the maximum prescribed dose of the organ dose in proportion;

[0052] Optionally, the state score further includes one or more of the following: consistency index, uniformity index, and drop score;

[0053] The DRL includes one or more of the following: TD3, PPO, and SAC algorithms.

[0054] A tumor field angle optimization method based on deep reinforcement learning, comprising:

[0055] Obtaining a target region image to be planned and an initial action;

[0056] Obtaining an initial state based on the target region image to be planned and the initial action;

[0057] Inputting the initial state into a tumor field angle planning model to output an optimized action, wherein the action is a field planning, and the tumor field angle planning model is obtained based on the construction method of the deep reinforcement learning-based tumor field angle planning model.

[0058] A tumor field angle adaptive optimization method based on transfer learning, comprising:

[0059] S31: Obtaining a target region image to be planned and an initial action;

[0060] S32: Obtaining an initial state based on the target region image to be planned and the initial action;

[0061] S33: Performing transfer learning on the tumor field angle planning model to obtain a transfer DRL model, wherein the tumor field angle planning model is obtained based on the construction method of the deep reinforcement learning-based tumor field angle planning model;

[0062] S34: Updating the action by using the transfer DRL model to obtain a third action;

[0063] S35: obtaining a third state based on the to-be-planned target region image and the third action, and evaluating the third state to obtain a third state score;

[0064] S36: obtaining a reward of the third action based on the third state score, the reward of the third action being the third state score or a difference between the third state score and an initial state score;

[0065] S37: migrating a DRL model based on a maximum target reward according to an initial state-third action pair and a target reward, repeating steps S34-S36 iteratively until a stopping condition is met to obtain a tumor field angle planning model, the target reward including at least one reward of the third action, and replacing the initial state-third action pair with a third state-next third action pair after the iteration starts.

[0066] A method for accelerating tumor field angle optimization based on reinforcement learning, which uses multi-agent parallel sampling to complete the iteration process of the above method;

[0067] Optionally, the multi-agent parallel sampling is implemented by parallel adoption of a main thread and multi-agent threads through reinforcement learning optimization to achieve parallel sampling on the same planning system.

[0068] Optionally, the main thread sends a sampling field angle request to an agent of each agent thread, the agent detects the sampling request from the main thread, initializes a local environment, generates an action according to a current state using a local model, obtains a field dose distribution according to the generated action by a local planning system, stores the sampled field angle and the corresponding reward in agent thread sampling data after calculating the reward based on the field dose distribution, and updates the parameters based on the sampling data of each agent thread, and sends the updated parameters to each agent thread, and each agent thread updates the local model after receiving the updated parameters.

[0069] Optionally, each agent thread is deployed through a virtual machine.

[0070] The application also discloses a method for accelerating tumor field angle optimization based on reinforcement learning, which uses multi-agent parallel sampling to complete the iteration process of the above method;

[0071] Optionally, the multi-agent parallel sampling is implemented by parallel adoption of a main thread and multi-agent threads through reinforcement learning optimization to achieve parallel sampling on the same planning system.

[0072] Optionally, the main thread sends a sampling field angle request to the agent of each agent thread, the agent detects the sampling request from the main thread, initializes the local environment, generates an action based on the current state using the local model, obtains a field dose distribution based on the generated action by the local planning system, stores the sampled field angle and the corresponding reward in the agent thread sampling data after calculating the reward based on the field dose distribution, and updates the parameters after the main thread updates the parameters based on the sampling data of each agent thread and sends the updated parameters to each agent thread, and each agent thread updates the local model after receiving the updated parameters.

[0073] Optionally, each agent thread is deployed through a virtual machine.

[0074] The second aspect of the present application discloses a construction system of a tumor field angle planning model based on deep reinforcement learning, comprising:

[0075] An acquisition module is configured to acquire a training set image and an initial action, wherein the image set contains a target region, and the action is a field planning;

[0076] An initial state module is configured to obtain an initial state based on the initial action, and evaluate the initial state to obtain an initial state score;

[0077] A DRL sampling action module is configured to input the initial state into a DRL, and sample a new action to obtain a second action;

[0078] A state score updating module is configured to obtain a second state based on the second action, and evaluate the second state to obtain a second state score;

[0079] A reward calculation module is configured to obtain a reward of the second action based on the difference between the second state score and the initial state score;

[0080] An iterative training module is configured to repeatedly schedule the sampling module, the state score updating module, and the reward calculation module to maximize the target reward iteratively until a stop condition is met, and obtain a trained tumor field angle planning model, wherein the target reward includes the reward of at least one second action.

[0081] The present application also discloses a tumor field angle optimization system based on deep reinforcement learning, comprising:

[0082] A second acquisition module is configured to acquire a target region image to be planned and an initial action;

[0083] A second initialization module is configured to obtain an initial state based on the target region image to be planned and the initial action;

[0084] an output module configured to output an optimized action in the initial state input into the tumor angle of field planning model, the action being a field planning.

[0085] The application also discloses a tumor angle of field adaptive optimization system based on transfer learning, comprising:

[0086] a third acquisition module configured to acquire a target image to be planned and an initial action;

[0087] a third initialization module configured to obtain an initial state based on the target image to be planned and the initial action;

[0088] a transfer learning module configured to perform transfer learning on the tumor angle of field planning model to obtain a transfer DRL model;

[0089] a third sampling module configured to input the initial state into the transfer DRL model to update an action to obtain a third action;

[0090] a third state score updating module configured to obtain a reward of the third action based on the third state score, the reward of the third action being the third state score or a difference between the third state score and an initial state score;

[0091] a third reward calculation module configured to obtain a reward of the transfer DRL model based on the third state score and the initial state score;

[0092] a third training module configured to schedule the third sampling module, the third state score updating module and the third reward calculation module to repeatedly sample state-action pairs and iteratively optimize based on the reward maximization until a stop condition is met, so as to obtain a transfer adaptive DRL model;

[0093] a third output module configured to output an adaptive optimal angle of field output by the trained transfer DRL model.

[0094] The third aspect of the application discloses a computer device, comprising a memory and a processor; the memory is configured to store program instructions; the processor is configured to invoke the program instructions, and when the program instructions are executed, the steps of the above method are executed.

[0095] The fourth aspect of the application discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0096] The fifth aspect of the application discloses a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0097] The application has the following beneficial effects:

[0098] (1) This application uses a deep reinforcement learning algorithm to optimize the radiation field angle, a parameter that has not been given enough attention and has been designed inaccurately in previous radiation plans; and the optimized field arrangement achieves better results than manual field arrangement by physicists;

[0099] (2) The field angle optimization model of this application using the deep reinforcement learning algorithm found a field angle that was inconsistent with clinical experience. However, medical examinations found that the field angle optimized by deep reinforcement learning was more effective. Therefore, the deep reinforcement learning field angle planning model of this application can be used to guide physiotherapists in clinical field arrangement; it helps physiotherapists find field arrangement plans that are inconsistent with clinical experience but can achieve better results.

[0100] (3) In this application, the radiation field dose distribution and the target volume and organ location are fused and then input into the DRL for training, so that the DRL learns the strategy of field placement based on the target volume location and the location of the organ at risk. This strategy can be generalized and applied to new field placement requirements.

[0101] (4) After using a deep reinforcement learning-based algorithm to optimize the parameters of the radiation plan for transfer learning, this application can further optimize the individualized optimal solution based on adaptive deep learning to achieve a better individualized targeted design;

[0102] (5) The design of multi-agent sampling in this scheme realizes reinforcement learning using a real clinical commercial planning system rather than a simple dose calculation model, so that the strategies learned by reinforcement learning can be better transferred to clinical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0104] Figure 1 This is a schematic diagram of the method flow provided by the first aspect of the embodiment of the present invention;

[0105] Figure 2 is a schematic diagram of a program product provided by the second aspect of an embodiment of the present invention;

[0106] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0107] Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention;

[0108] Figure 5is a schematic view of a storage medium provided by an embodiment of the present application;

[0109] Figure 6 is a network structure diagram of a SAC algorithm provided by an embodiment of the present application;

[0110] Figure 7 is a schematic view of a network layer of an actor network in a SAC algorithm provided by an embodiment of the present application;

[0111] Figure 8 is a schematic view of a network layer of a critic network in a SAC algorithm provided by an embodiment of the present application;

[0112] Figure 9 is a schematic view of an interaction process between a main training optimization thread and multi-agent sampling provided by an embodiment of the present application;

[0113] Figure 10 is a schematic view of a training process of a reward function when multi-agent sampling is provided by an embodiment of the present application;

[0114] Figure 11 is a schematic view of a training process of a loss function of a critic network when multi-agent sampling is provided by an embodiment of the present application;

[0115] Figure 12 is a comparison diagram of plan scores of a training set after reinforcement learning training provided by an embodiment of the present application;A diagram is a dot diagram, and B diagram is a box diagram;

[0116] Figure 13 is a comparison diagram of plan scores of a test set after reinforcement learning training provided by an embodiment of the present application;

[0117] Figure 14 is a schematic view of a scheme provided by a model being better than a scheme of field placement of a clinical physicist provided by an embodiment of the present application;

[0118] Figure 15 is a schematic view of another scheme provided by a model being better than a scheme of field placement of a clinical physicist provided by an embodiment of the present application;

[0119] Figure 16 is a schematic view of a reason analysis of a scheme provided by a model being better than a scheme of field placement of a clinical physicist provided by an embodiment of the present application;

[0120] Figure 17 is a comparison diagram of dose distribution under an initial field angle and dose distribution under an optimized field angle provided by an embodiment of the present application;

[0121] Figure 18An iterative intermediate process schematic diagram of a deep reinforcement learning model provided by an embodiment of the present application is provided.

[0122] Figure 19 An illustration of constructing a position matrix according to a target region and an organ provided by an embodiment of the present application is provided.

[0123] Figure 20 A flowchart of DRL iterative optimization provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0124] In order to enable personnel in the technical field to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application.

[0125] In some of the processes described in the specification and claims of the present application and in the above-mentioned drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed or in parallel without the order in which they appear in this text, and the serial numbers of the operations such as S101, S102, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this text are used to distinguish different messages, devices, modules, etc. and do not represent the order of precedence. Also, "first" and "second" are not different types.

[0126] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0127] Figure 1 An evaluation PATENTNAME method flowchart provided by an embodiment of the present application is provided. Specifically, the method comprises the following steps:

[0128] S101: Obtain a training set image and an initial action of DRL, the image set contains a target region, and the action is a field planning;

[0129] S102: Obtain an initial state of DRL based on the initial action, and evaluate the initial state to obtain an initial state score;

[0130] S103: DRL samples a new action to obtain a second action;

[0131] S104: obtaining a second state based on the second action, evaluating the second state to obtain a second state score;

[0132] S105: obtaining a reward of the second action based on a difference between the second state score and the initial state score;

[0133] S106: based on the second action-second state pair and the target reward, the DRL repeatedly optimizes S103-S105 iteratively until a stopping condition to obtain a tumor field angle planning model, the target reward including the reward of at least one second action.

[0134] Optionally, the target reward is the sum of the sample rewards of the deep reinforcement learning from the experience replay pool.

[0135] In some embodiments, the stopping condition is that the target reward no longer increases after a certain number of iterations, at which time the stopping condition is reached.

[0136] First, the reinforcement learning algorithm used in this application is clearly described in the prior art of this application, although the reinforcement learning algorithm is prior art, but its application to the field of field angle optimization is the first time of this application, the background technology of the reinforcement learning algorithm does not affect the creativity of this application.

[0137] Reinforcement learning consists of two subjects and four parts, two subjects are:

[0138] Agent: Agent, i.e. individual taking action;

[0139] Environment: Environment, which can provide feedback to the action;

[0140] The four parts include: A, S, R, P; A represents the action space, i.e. all action spaces of Agent taking action, S represents the state space, i.e. the feedback state after Agent taking action, R represents the reward, which is a real number, positive value represents reward, negative value represents punishment, P represents the strategy, i.e. what action Agent will take under state s∈S a∈A. (Sometimes A represents the strategy)

[0141] Deep reinforcement learning is a process of generating a strategy P by Agent according to state S to take action A in environment E in order to obtain maximum reward R.

[0142] Algorithm development: from the Q-learning algorithm of reinforcement learning to the DQN algorithm of deep reinforcement learning, the DQN algorithm develops into the DDPG algorithm, and the DDPG algorithm develops into the TD3 algorithm, and the TD3 algorithm develops into the SAC algorithm.

[0143] Symbolic representation in deep reinforcement learning:

[0144] π: function symbol, represents the policy function, the parameter is usually the state returned by the environment, and the output is a specific action, that is, a = π(s) ∈ A.

[0145] r: the immediate reward obtained after taking action a.

[0146] G: cumulative reward, G t represents the cumulative reward from time t to the end of the action;

[0147] Q π (s, a): state-action value function (state-action value function), sometimes also referred to as action value function, that is, at time t, in state s, taking action a, using policy π, the expected value of the cumulative reward expected to be obtained.

[0148] V π (s): state value function (state value function). That is, at time t, in state s, taking action a, using policy π, the expected value of the cumulative reward expected to be obtained.

[0149] The SAC algorithm can be divided into two parts: policy evaluation and policy improvement:

[0150] In the policy evaluation part, the state value function Vsoft and the action value function Qsoft are defined as follows:

[0151] V soft (s t )=E[Q(s t ,a t )]-lgπ(a t |s t )

[0152] Q soft (s t ,a t )=r(s t ,a t )+γE[V_soft(s t+1 )

[0153] In the policy improvement part, the policy is optimized using relative entropy. The sampling policy and the reward function: the sampling policy refers to the selection of the input DNN sample in the deep reinforcement learning process, and the current environment state s t In order to improve the convergence and fitting ability of deep reinforcement learning,

[0154] SAC (Soft Actor-Critic) is a model-based deep reinforcement learning algorithm that combines the advantages of Actor-Critic algorithms and model-based policy optimization algorithms. SAC algorithm can efficiently solve reinforcement learning problems in continuous action space, especially suitable for problems with high-dimensional state space and action space.

[0155] I. The basic principles of SAC algorithm are as follows:

[0156] (1) Establish a value function (Value Function) to evaluate the pros and cons of taking different actions in a given state, i.e. evaluate the value of "state-action pairs";

[0157] (2) Establish a policy function (Policy Function) to determine which action to take in a given state.

[0158] (3) Learn the value function and policy function by maximizing the expected return.

[0159] (4) Use the learned value function to evaluate the policy, and use the learned policy function to select the action with the maximum expected return.

[0160] The basic process of SAC algorithm to solve reinforcement learning problems:

[0161] 1. Initialization: Define the architecture of the neural network, initialize the parameters of the value function and the policy function.

[0162] 2. Collect experience: Execute the current policy in the environment and record the state, action, reward, next state, etc.

[0163] 3. Update the value function: Use the collected experience to update the value function using gradient descent method.

[0164] 4. Update the policy function: Use the collected experience to update the policy function using gradient ascent method.

[0165] 5. Select action: Select an action according to the current state and the learned policy function and execute it.

[0166] 6. Repeat steps 2-5 until the termination condition is met, such as reaching the maximum number of iterations or the policy converges.

[0167] Finally, SAC algorithm outputs the learned optimal action, i.e. the field angle, according to the learned policy function.

[0168] In order to solve the technical problems proposed in this application, the following element definitions are based on the problems of this application:

[0169] In some embodiments, when only one field is included, the action space includes any action in the [0, 360] degree range, for example, 60 means the action taken is to turn to the 60 degree direction of the field.

[0170] In some embodiments, when multiple fields are included, the action space includes the combination of all fields in the [0, 360] degree range; wherein the angle of the kth field must be greater than the angle of the previous k-1 fields, that is, the action interval of the kth field is reduced to (the angle of the kth field, 360 degrees]. An action in the action space is represented as: [a1, a2,..., a k ,..., a N ], N represents the total number of fields, and a k represents the angle of the kth field.

[0171] In some embodiments, when multiple fields are included, the action space includes any action in the [0, 360] degree range, [a1, a2,..., a k ,..., a N ] represents the path of an action, N represents the total number of fields, and a k represents the angle of the kth field.

[0172] In some embodiments, the state in the state space is defined as the field dose distribution containing position information: the position includes the target region position and the organ.

[0173] In some embodiments, the region where the PTV is located is assigned a positive value, the region between the PTV and the Body is assigned a negative value, and the region outside the Body is assigned a value of 0.

[0174] In some embodiments, the field is divided into n regions according to the position of the PTV, and the first to nth regions / positions are represented by the numbers 2-n+1, respectively. The region assignment of the target region is divided into an integer part and a decimal part, with the integer part representing the target region and the decimal part representing the dose of the target region.

[0175] In some embodiments, when obtaining the position of the target region and the organ based on the image, the position and boundary of the target region and the organ are obtained by manual delineation.

[0176] In some embodiments, when obtaining the position of the target region and the organ based on the image, the position and boundary of the target region and the organ are obtained by AI recognition.

[0177] In some embodiments, according to the position of the brain tumor target region in the brain, the target region is divided into target regions 1-n, and the state of the whole brain is represented by a continuous decimal, for example, Figure 19 as shown on the left, the purple part delineates target region 1; the yellow part is target region 2; then

[0178] Target 1 is in the range of [2.0-2.1], where the integer part 2 indicates that the PTV is in target 1 (purple area), and the decimal part is set as follows: 0.1 is set as the maximum prescription dose of the target area in the current plan, and other doses are scaled proportionally.

[0179] Target 2 is in the range of [3.0-3.1];

[0180] Target 3 is in the range of [4.0-4.1];

[0181] If there is overlap between the target areas, the high-dose target area is preferred, such as Figure 19 As shown on the left, where the purple and yellow target areas overlap, the high-dose purple area is used.

[0182] Through such design, the state value reflects both the target area position and the dose received by the target area.

[0183] Assuming the maximum prescription dose is 60 Gy, the specific meaning of 3.08 is that the pixel point is located in target 2, and the corresponding dose is 60*1.2*0.8=57.6 Gy.

[0184] In some embodiments, when characterizing the state, all OARs are valued from 0.1 to 1, and the specific value range of each organ depends on the number of OARs. If the number of OARs does not exceed 9, the value range of each organ is 0.1, and if the number of organs increases, further subdivision is required, for example:

[0185] Organ 1: 0.1-0.2; Organ 2: 0.2-0.3; Organ 3: 0.3-0.4; the decimal part is valued according to the same rule as the decimal part of the target area, with 120% of the maximum prescription dose scaled proportionally in the value range of each organ.

[0186] If an organ and a target area overlap, the integer part uses the target area and the decimal part uses the organ. For example, 3.38, the integer part corresponds to target 3, and the decimal part is in the value range of organ 3, which means that the pixel point is located in the overlapping part of target 3 and organ 3, and the dose is 60*1.2*0.8=57.6 Gy, as shown in Figure 19 As shown on the left, the green part represents the brainstem and the yellow part represents the target area, so the integer part 3 represents target 2 and the decimal part 3 represents organ 3 (brainstem), Figure 19 The right side shows the depiction of the crystalline organ.

[0187] The position information is fused into the state representation of the state space by the method described above, and the deep reinforcement learning updates Q through learning of the "state-action pair", and further updates the policy (such as Figure 20 The policy of the DRL learns the position-based action scheme, that is, the field design scheme.

[0188] In some embodiments, the state score (also called the plan score) is calculated by the organ score;

[0189] In some embodiments, the state score is calculated by the organ score, the consistency index, the uniformity index, and the drop score.

[0190] In some embodiments, the reward is represented as the difference between the state score after the sampling of the field angle updates the state and the state score corresponding to the initial state.

[0191] In some embodiments, the ultimate goal of the reinforcement learning is to learn the action [a1, a2, …, a k ,…,a N ] that maximizes the reward, where a1 corresponds to the angle of the first field (the first step action), a2 corresponds to the angle of the second field (the second step action), …, and a7 corresponds to the angle of the seventh field (the seventh step action). After the kth field takes the action a k , the irradiation field dose distribution matrix is obtained by the planning system, and the state corresponding to the action is obtained after the fusion of the target position. The state score is calculated based on the state, and the loss between the new state score and the initial state score is taken as the reward obtained. Through the optimization and iteration of the reinforcement learning, the action combination of the 7 steps that can obtain the best reward is learned.

[0192] In some embodiments, the study collects the target information of 236 patients with brain tumors, uses the three-dimensional dose distribution as the input, interacts with the ECLIPSE planning system through the script written by the ESAPI, and uses the multi-Agent parallel sampling to greatly improve the sampling efficiency. The Soft Actor-Critic (SAC) algorithm is used for iterative optimization to obtain the reinforcement learning field angle optimization model. The target of the training set is input into the model, the field angle of the test set is obtained, and the final field design scheme is generated based on the field angle and the optimized dose by the planning system. The results show that the field design scheme generated by the model in the training set is better than the initial scheme of 7-field equal division, the average plan score is improved by 18.9 points, and the average score of the manual plan of the physicist is improved by 3.4 points. In the test set, the average score is improved by 7.3 points compared with the 7-field equal division scheme. It is concluded that the field angle selection algorithm based on DRL designed in the application can consider various factors, explore and master the methods and skills beyond the clinic, and generate high-quality field design schemes.

[0193] In some embodiments, the present study makes the following adjustments when applying DRL to solve the problem of field optimization radiotherapy planning: (1) Expand the size of the training set; (2) Use 3D dose distribution as input and consider the spatial position information of target organs, use SAC algorithm which performs well in continuous action space, and prepare for larger scale action space expansion in the future; (3) Design a multi-agent multi-thread sampling framework interconnected through local area network to improve sampling efficiency.

[0194] In some embodiments, 236 cases of central nervous system tumors treated in the Department of Radiation Oncology of the Second Hospital of Hebei Medical University from January 2022 to April 2024 are selected, all of which have only one target area, and are divided into left upper, left lower, right upper, right lower, and central five categories according to the approximate location of the target area, and are evenly distributed to 7 parallel trained agents (Agents), ensuring uniform sampling of target areas in different positions. All cases use the same prescription dose, with a planned target volume (PTV) of 60Gy / 30 fractions. Of these, 212 cases are used for training, and 24 cases are used for testing. The present study uses the AAA (Anisotropic Analytical Algorithm) algorithm of version 15.6 Eclipse for dose calculation. In order to more directly reflect the influence of field angle on plan quality, a set of relatively loose and fixed optimization conditions are set when optimizing the dose according to the field angle, as shown in Table 1:

[0195] Table 1 Optimization conditions for dose planning

[0196] Type Vol [ % ] Dose [Gy] Priority PTV Lower 99 60 200 PTV Upper 0 63 150 BrainStem Upper 0 52 80 Lens_L / R Upper 0 6 80 OpticChiasm Upper 0 48 80 OpticNerve_L / R Upper 0 48 80 External Upper 0 63 300

[0197] SAC (Soft Actor-Critic): The SAC algorithm is selected for the present study (as shown in the attached Figure 6 PADDLE 2.5.1 and PARL 2.2.1 are used to build the model.

[0198] As Figure 6 shown, the SAC algorithm includes an actor network and four critic networks, which are:

[0199] ① State value (V) estimation network:

[0200] ② Target value (target V) network:

[0201] ③ Action-state value estimation network Q0:

[0202] ④ Action-state value estimation network Q1:

[0203] In SAC algorithm, we model two action value functions Critic and one policy function Actor. Based on the idea of Double DQN, SAC uses two Critic networks, but each time we use a Critic network, we will choose a network with a small value to alleviate the problem of overestimation. The pseudo code is shown in Table 2:

[0204] Table 2 SAC algorithm (Soft Actor-Critic)

[0205]

[0206]

[0207] In some embodiments, the iterative update process of the SAC algorithm shown in Table 2 is as shown in Table 2, and the interpretation of action and space in this application is shown in Table 3 DRL iteration table: Figure 20

[0208] Table 3 DRL iteration table

[0209]

[0210] In some embodiments: we first use a customized MIM (7.3.2) workflow to unify the image size and delineation name of all patients. The image resolution is 512*512*128, and the pixel is 0.78125*0.78125*3(mm). The initial field angle is 7 fields, and the optimized dose distribution is obtained. Then, based on the image information, a position matrix with the same three-dimensional dose dimension is constructed: according to the number of target regions, the PTV target region is divided into 3 regions, and the position of each case PTV target region in the position matrix is set to the corresponding number representing the target region; At the same time, the organs at risk include the left eyeball, the right eyeball, the optic nerve, the chiasm, and the brain stem; then give a small number less than 1 to these 5 regions, 0.1 represents the left eyeball, 0.2 represents the right eyeball, 0.3 represents the optic nerve, 0.4 represents the chiasm, and 0.5 represents the brain stem. After such assignment, the remaining places in the three-dimensional position matrix are set to 1 for intracerebral and 0 for external; thus, the three-dimensional position matrix is constructed; the three-dimensional dose distribution matrix is scaled and summed with the three-dimensional position matrix to obtain the three-dimensional state matrix, which is then down-sampled to 128*128*128 and translated to make the treatment isocenter (MARK) located in the middle of the space. Finally, the processed three-dimensional matrix of the dose distribution is input into the model as the observation of reinforcement learning.

[0211] Network structure: based on the input and output dimensions of this study, the actor network and critic network model structure designed by us is as follows: Figure 7 ​and Figure 8 As shown in the figure, the final output of the Critic network is [8, 1], which represents the Q value of the 8 plans sampled at the current time; Figure 7 The final output of the Actor network is [8, 7], where 8 represents the batch size (i.e., 8 plans are randomly selected from the cache pool for training at a time), and 7 represents 7 actions (in reinforcement learning, it is called action, which can be mapped to 7 real numbers of field angles);

[0212] In some embodiments, 8 samples are selected for each model update, and the rewards of the 8 samples are added to obtain a total reward. The total-reward curve records the change of the total reward, and once there is no increasing trend, the iteration is stopped to obtain the field angle planning model.

[0213] Figure 8 The final output of the Critic network shown in the figure is [8, 1], which represents the Q value of the 8 plans sampled at the current time;

[0214] Because the commercial planning system is expensive to use and has limited deployment resources, in order to complete the sampling of reinforcement learning using the few commercial planning systems, the present application achieves parallel sampling by interacting with the main thread through distributed deployment of multiple Agent threads, thereby improving the efficiency of sampling.

[0215] Because the radiotherapy planning system is a commercial software, it is limited by the license, which is not conducive to the training of reinforcement learning in the present application. Taking the Eclipse planning system used in the present research as an example, only 2 plans can be calculated at the same time, and the planning system also needs to be used clinically, so it cannot bear the research work of reinforcement learning. Therefore, the corresponding research is carried out on the Eclips Tbox workstation (for clinical learning) configured by the company, and a set of planning system is installed on the workstation. The related research of reinforcement learning here will not delay the clinical work, but still cannot meet the demand of high-quality research. Because of this limitation, researchers have adopted two schemes: 1. Simplify the modeling of the planning system and simplify the dose calculation process to improve the sampling efficiency. This will undoubtedly bring errors and unpredictable problems. 2. Use a single planning system for sampling, but significantly simplify the clinical problem, but the simplified clinical problem no longer has clinical application value, and is more a simple verification of feasibility.

[0216] In order to solve the problem of slow sampling, the present research first adopts a multi-Agent parallel sampling framework. We use virtual machine software to virtualize the operating system on the Tbox, so that multiple virtual machines can be opened at the same time for calculation.

[0217] Virtual machines consume a lot of resources, and the number of virtual machines that a computer can run is still not enough to meet the research needs, so we distribute the virtual machines in two computers, each running 3 virtual machines, and sampling at the same time, plus the local planning system of Tbox, there are currently 7 agents sampling. This is because of the lack of hardware resources, otherwise the number of agents is unlimited.

[0218] Because the program runs on different computers, data interaction is also a problem that must be solved. Because the script that interacts with the planning system uses C# language, and the DRL algorithm uses python language, the solution finally adopted is to use LAN interconnection and json file interaction. The flow chart is shown in Figure 9

[0219] As can be seen from the flow chart, each Agent runs independently in its own environment, and the flow only reads and writes in the corresponding two folders. These two operations will not affect other threads, nor will they be affected by other threads. Similarly, the main thread does not interact with each Agent thread, so this achieves decoupling. Even if the Agent stops running for various reasons, the main thread will not detect the sampling data in the corresponding folder, and will directly skip, without affecting the entire training process. The only problem that may be encountered is that two threads operate on a file at the same time. Here there is also a corresponding error capture process, which only needs to pause the error thread for a few seconds until it does not report an error.

[0220] In some embodiments, multiple Agent parallel sampling is used in the training process to speed up the training speed of reinforcement learning. Each Agent interacts with the Eclispe planning system through the ESAPI script interface, and the Agent is located in multiple computers and connected to each other through the LAN. The simple flow chart is shown in Figure 9

[0221] ​​The main thread sends a field angle sampling request to the agent in the agent thread, and the agent receives the instruction of the main thread and starts to perform sampling: first, the agent in the agent thread initializes the local environment, generates an action (i.e. field angle) according to the current state using the local model, and calculates the reward after re-dose optimization according to the generated action by the local planning system. The agent thread stores the action and the corresponding reward in the sampling data folder, and the main thread traverses the sampling data according to the environment where each agent thread is located, stores the sampling data in the experience replay pool, and the reinforcement learning main thread randomly samples from the experience replay pool for training, updates the model parameters, and sends the updated parameters to each agent thread to update the local model; wherein the local planning system in the above process refers to the same business planning system, but based on the scheme design of the present application, parallel access to the commercial planning system to obtain accurate reward calculation can be realized.

[0222] In some embodiments, the agent inputs the initial planned field angle through a script when sampling, and the planning system optimizes the dose distribution according to the input field angle, and then the reinforcement learning obtains the optimized field angle based on the obtained dose distribution, and replaces the initial planned field angle with the optimized field angle to input the planning system to re-optimize the dose distribution and obtain the optimized radiotherapy plan.

[0223] In some embodiments, the number of fields is 7, and the action path is represented as a t ={a1,a2,a3,a4,a5,a6,a7}, the model output a t includes 7 float types, and the value of the real number is -1-1, and the range becomes 0-2 after being uniformly added 1. According to experience, a small field interval usually has no meaning, so we stipulate that the minimum field interval is 30°. Each field angle is calculated by the following formula:

[0224]

[0225] wherein a n ∈[-1,1] is the nth real number of the output optimal field angle, An is the angle of the nth field, A n-1 is the angle of the (n-1)th field, A1 is the angle of the first field, the function F(x) unifies the field angle to 0-360°, and is discretized to 5 integer multiples; N represents the number of fields, interval represents the interval of the field angle; f represents the angle difference between A n-1 and A1.

[0226] where an is the nth real number output of the model, also represents the action space of the nth field, ranging from [-1, 1], mapping to the angle range [0, 360]. An is the angle of the nth field. Function F(x) is responsible for discretizing the field angle to the nearest integer multiple of 5, reducing the training complexity, and making the correct transformation when encountering a field angle greater than 360°.

[0227] At the same time, since the angles of the 7 fields cannot coincide, the action a n The constraint conditions include: the values of the 7 actions cannot be equal.

[0228] Reward function: use the difference between the initial plan score (i.e. initial state score) as the reward function. The plan score / state score is the sum of the following items, and the specific scoring rules of each item are as follows:

[0229] (1) Lens, Optic Nerv, Optic Chiasm, Brain Stem scoring formula:

[0230]

[0231] where Score is the organ score, ScoreRange is the highest score that the current organ can get, and the highest score of each organ is different, representing the different weights in the organ score. The lens is very fragile, and the algorithm is expected to better learn to protect the lens, so it is given a high weight. Max is the maximum dose of the current organ, and Low and High are the dividing lines between full marks and 0 marks. Specific to each organ as follows:

[0232]

[0233] (2) Conformity Index (CI)

[18] :

[0234]

[0235] where TV RI represents the target volume covered by 95% of the prescribed dose, TV represents the target volume; V RI represents the volume of 95% of the prescribed dose.

[0236] (3) Homogeneity Index (HI)

[19] :

[0237]

[0238] D2 and D98 represent the highest and lowest doses received in the target region, respectively, and Dprescription is the prescribed dose.

[0239] (4) fall score

[0240]

[0241] where V20initial is the volume covered by the initial plan of 20 Gy, and V20 is the volume covered by the current plan of 20 Gy.

[0242] This study uses both Python (DRL model training) and C# (Agent and plan system interaction sampling) programming languages, uses a Json file as a data transmission medium, decouples each thread, and runs independently without interference. The accidental stop of any Agent will not affect the running of the entire framework, greatly enhancing the stability of the system.

[0243] A complete sampling takes about 70s on average. According to the scale of this study, a total of 80000 samples need to be collected. If only one commercial planning system is used for sampling, it will take about 64 days. The study uses 7 Agents for parallel sampling, which shortens the time to about two weeks. The training curve is shown in Figure 10 and Figure 11 .

[0244] In some embodiments, the state space is only assigned positive values in the region where the PTV is located, negative values between the PTV and the Body, and 0 outside the Body. We randomly selected 50 cases from the training set. The final score of the plan given by the model (Drl group) was higher than that of the initial plan with 7 field uniformity (Initial group). Then we manually designed the field for 16 cases in the training set that did not meet the clinical field habit (the physicist only designed the field plan according to the environment of this model, without knowing the training results of this model). By comparing the manual field plan (Manual), only one plan score was slightly higher than that of the DRL group. The data is shown in Table 4:

[0245] Table 4 Comparison of scores of initial plan and DRL group plan in training set

[0246]

[0247] The two groups of data were tested by Kolmogorov-Smirnov test and found that the data did not meet the normal distribution, so Wilcoxon paired signed rank test was used to compare the differences between the Initial group and the Drl group, and the Drl group and the Manual group. The difference between the two groups was statistically significant (P<0.001).

[0248] There are 24 cases in the test set, of which 18 cases have a score higher than the initial state, and the remaining cases have a score lower than the initial plan. The data is shown in Table 5 andFigure 13

[0249] Table 5 Score comparison of initial plan and DRL group plan in test set

[0250]

[0251] Both groups of data were found to meet the normal distribution after Kolmogorov-Smirnov test, so paired T test was used to compare the difference between the two groups, and it was found that there was a significant statistical difference (P = 0.001) between the two groups of data.

[0252] The examples shown in Tables 4 and 5 only show the illustrative effect of a simple state space setting (the region where the PTV is assigned positive, and the region between the PTV and the body is assigned negative), and not the best effect of the method of the present application.

[0253] In the above research of the present application, the trained model exhibits strong exploration and learning ability by combining the SAC algorithm. Since a higher ScoreRange is designed for the organ score of the crystal in the present application, the organ score of the crystal occupies a higher weight. At the same time, the target region position, the position of the organ at risk, the in-body part, and the out-body part are distinguished in the state representation. Therefore, the field arrangement scheme of the field angle optimization in the present application learns to avoid the crystal; based on the fall-off function, the field arrangement scheme learns to arrange the field nearby; based on the design of the field angle mapping algorithm, there is enough exploration space to fully consider the details ignored in the clinic and learn and master the skills not mastered in the clinic. When observing the field arrangement results of the training set, several cases were found which were completely opposite to the clinical field arrangement habits, but after comparative analysis, it was found that under the current scoring rules, the scheme given by the model was better than the field arrangement scheme of the clinical physicist.

[0254] A typical case is shown in Figure 1. Figure 14 As shown in the figure, the target region of the case is located in the upper right, Figure 14 -A is the field arrangement scheme given by reinforcement learning, which does not follow the principle of arranging the field nearby from the opposite side. Figure 14 -B is arranged nearby according to clinical experience. However, after comparison, it was found that according to the scoring rules in the research, the field arrangement scheme of reinforcement learning has a higher score than the clinical scheme because it has fewer low-dose regions and fewer total MUs. Finally, after analysis, we found that the target region of the case is close to the skin, and arranging the field nearby requires increasing the total MU to cover the area under the skin, which also leads to the expansion of the low-dose region, resulting in a lower comprehensive score.

[0255] Another typical case is shown in Figure 2. Figure 15 As shown in the figure, the target region of the case is close to the last layer of the foot side, which is located in the layer where the crystal is located. Figure 15 -A is the field arrangement scheme of reinforcement learning, and it can be seen that the field directly passes through the crystal,​Figure 15 -B is the clinical arrangement, which passes through the lens from the far side. Theoretically, the arrangement of the lens in Figure B should have a lower dose, but in fact, the arrangement of the lens given by reinforcement learning has a lower dose.

[0256] The reason is as shown in Figure 16 : Because the rays are conical, the farther the distance from the machine head, the larger the range covered by the rays, and the arrangement of the depth reinforcement learning makes the lens closer to the machine head, which can completely avoid the lens under the premise of covering the target area, while the clinically commonly used arrangement is just the opposite, which exposes the lens to the field, ultimately leading to the increase of the dose of the lens.

[0257] In some embodiments, the tumor planning model designed and trained by the present application is used in the field angle setting stage in the clinic, and the field angle output by the model is directly used for arrangement.

[0258] In some embodiments, the tumor field angle planning model designed and trained by the present application is used in the field angle auxiliary setting stage in the clinic, and the physicist compares the field angle output by the model with his own experience, and selects a better scheme for arrangement.

[0259] In one embodiment, the physicist finds that the recommended arrangement plan output by the tumor field angle planning model is quite different from the previous experience, and after research, the model's recommended setting of the field angle is adopted for subsequent planning.

[0260] DRL combines the advantages of deep learning and reinforcement learning, has strong learning and generalization ability, and has great potential to surpass the level of human experts, but this requires the support of massive data. The application of DRL in the game field is undoubtedly the most successful, and the main reasons are two, one is that the algorithm can directly interact with the game, and does not need to model and simulate the environment, which avoids the introduction of modeling errors, and the second is that massive data can be quickly obtained by stacking computing power.

[0261] However, the application of DRL in the field of radiotherapy planning is different from the field of electronic games. In this scheme, we simulate the interaction of the algorithm in the game field through the script interface provided by the planning system and the system, but the biggest bottleneck of this interaction in the field of radiation planning is that the sampling efficiency is extremely low, which seriously restricts the clinical landing of DRL.

[0262] To break this bottleneck, the current research institute adopts the solution mainly into three categories, first, using their own established simple model to calculate the dose, which in the dose calculation of simple model, their own dose calculation program, accuracy, precision can not be compared with the clinic, but the advantage of simple, high sampling efficiency, but can improve the efficiency, but also will introduce modeling error. Second, as far as possible to simplify the clinical problem, reduce the exploration range of DRL algorithm, so that it can converge in the case of small sample, but this method learned strategy is too simple, not much value to the clinic. Third, increase the number of network, each network is responsible for the training of a parameter, which will greatly reduce the difficulty of network training, allow greater exploration space, is the current better research direction.

[0263] Unlike previous DRL research, the algorithm used in this study does not use a simple dose calculation model, but directly interacts with a commercial planning system, although it will reduce the sampling efficiency, but can ensure that the training results can be seamlessly migrated to the clinic, will not bring additional error.

[0264] In order to solve the problem of low sampling efficiency, we have created a multi-agent parallel sampling framework, which is distributed in the local area network of multiple computers. Agent interacts with the local Eclipse planning system for sampling, and the data is collected to the computer where the algorithm main thread is located for algorithm training. Although limited by hardware computing power, this study only uses 7 agents, but also controls the training time within an acceptable range. As long as the computing power is sufficient, the number of agents used in theory has no upper limit. The multi-agent parallel sampling framework solves the core problem of DRL sampling difficulty, which is also the bottleneck of the current DRL unable to land in the clinic. This framework lays a solid foundation for future larger-scale research.

[0265] In order to better verify the potential of DRL algorithm, this study is guided by clinical landing, and does not choose to simplify the clinical problem for verification, but chooses the field angle optimization of central nervous system tumors as the verification problem, because the spatial relationship between the target and the organ at risk of this type of disease is complex, and the field angle combination is variable, there is enough learning and exploration space. In order to ensure learning a good strategy in a complex environment, the study uses 212 cases for training, and the three-dimensional dose distribution is used as the input state, which is different from the DVH graph used by other researchers as input data. This scheme not only provides dose information for the model, but also provides PTV shape, position information, MARK position information, and head contour information, so that the model can consider multiple information and learn a better field strategy.

[0266] In some embodiments, the study first uses the SAC algorithm for end-to-end training, and has reliable performance in more complex clinical problems than previous studies. Moreover, because SAC is an optimization algorithm in continuous action space, it can assign more meanings to the actions generated by it and solve more complex clinical problems.

[0267] In some embodiments, the TD3 algorithm is used as the DRL model.

[0268] In some embodiments, the PPO algorithm is used as the DRL model to complete the training of the field angle planning model.

[0269] As Figure 17 The figure shows a dose distribution at an initial field angle and a dose distribution at an optimized field angle. The left side is the dose distribution in the initial state, and the right side is the dose distribution corresponding to the state of the field angle learned by deep reinforcement learning. According to the dose contours in the figure, it can be seen that the low dose area on the right is much lower than that on the left.

[0270] As Figure 18 The figure shows an intermediate process diagram of the iteration of a deep reinforcement learning model. Figure 18 There are two stored plans in the upper left corner, one is DRL-reset, which represents the initial 7-field uniform plan. The RTStruct in the lower left corner is the state score corresponding to the radiotherapy plan, and the score of the initial plan is 75.4 (the reward at this time is 0). The Best-reward is the plan with the highest reward, which will be updated with the highest score as the training progresses, Figure 18 The figure shows the current highest reward state score of 971.36 (the reward at this time is 971.36-75.4).

[0271] At the same time, considering the amazing performance of DRL in the training set, we further adapt DRL, which limits the DRL environment to a single case, uses the model trained by the current large number of cases for transfer learning, and then trains it for the single case to generate the best field planning scheme under the current scoring rule.

[0272] The purpose of adaptation is that the original data volume or parameter setting is not enough for the model to learn all the position information and have the best generalization effect. Therefore, transfer and individualized further training are used to make up for this deficiency.

[0273] In summary, this study first constructed a multi-agent parallel sampling framework based on the Eclipse planning system, solving the problem of low sampling efficiency. The model selected in this study directly uses the commercial Eclipse planning system for training, avoiding environmental bias, and after training, it can be directly applied to clinical practice. The design of this study is universal and can be quickly migrated to automatic planning, laying the foundation for the study of automatic planning based on deep reinforcement learning. Most importantly, the study has truly shown the huge potential of DRL surpassing clinical practice, learning skills that professional physicists cannot master but are very practical. As the study deepens, the complexity gradually increases, and DRL will learn more practical skills. In summary, this study has laid a solid foundation for the application of deep reinforcement learning in clinical practice.

[0274] Action a t In the Acotr network, the input is a 128*128*128 three-dimensional matrix, and the output is 7 real numbers between -1 and 1, corresponding to the angles of the 7 fields.

[0275] In some embodiments, we use adaptive DRL, which limits the DRL environment to a single case and uses the model trained on a large number of cases to do transfer learning, and then trains on the single case. The SAC algorithm is naturally suitable for adaptation, that is, after training on a large number of samples, it can quickly learn the optimal field arrangement scheme for the current scoring rule; all agents only sample on one case (that is, the training set has only one case, which uses a case that has not appeared in the previous training set), and the model parameters used by the agent are the model parameters after the first step of training. In this way, the model will give a similar scheme at the beginning, and then use the strong learning and exploration ability of the SAC algorithm to find the optimal solution for the current case.

[0276] First, the outstanding performance of the training set proves that under the corresponding scoring rule, the scheme given by reinforcement learning completely surpasses the scheme given by clinical personnel. Therefore, it can be concluded that if the target plan is placed in the training set, the target plan will get the best scheme under the current scoring rule, and the only problem to be solved is efficiency.

[0277] Second, the environment is complex, there is not enough training samples and computing power to support the model to learn completely in various situations. Although the experiment uses more than 200 samples, far more than the one or two samples of other researchers in related fields, the model generalization ability is still poor. Therefore, it cannot be guaranteed that continuing to expand sample data and continuing to adjust the model can obtain an excellent clinical acceptable model. So change the way, only use one sample, greatly reduce the complexity of the environment, let the model quickly explore and learn the optimal solution of the current sample. The main difference between the two schemes is the use efficiency. Training is extremely time-consuming. In this experiment, it took about 10 days to train about 200 samples, while the trained model gave the scheme in a very short time, about two or three minutes. But if the sample is controlled to one, the training time will definitely be greatly shortened. The application of transfer learning will further compress the time.

[0278] Finally, the use of a multi-agent parallel sampling framework can compress the training time to a clinically acceptable level. The current hardware resources are very limited, and only 7 agents are used. If the computing power is expanded to a server level of about 100,000 yuan, the number of available agents will expand to dozens, and the training time will be further improved.

[0279] As can be seen, the process is basically unchanged compared to the previous experiment. The only change is that the model parameters use the last trained parameters, that is, transfer learning. The training set is replaced from the original large sample database to the target plan. The rest is to observe the score of the target plan. The algorithm saves the plan with the highest reward and the plan with the highest state score. By observing the change curve of the state score, once it becomes flat, the training can be terminated, and the plan with the highest score is selected. The plan used is the final plan.

[0280] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present application, as Figure 3 shown, the device can include one or more processors and one or more memories; wherein the memory has computer readable code stored therein, which when executed by the one or more processors, can perform the method as described above.

[0281] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. It can implement or execute the various methods, operations, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor can be a microprocessor or any conventional processor, etc., and can be an X86 architecture or an ARM architecture.

[0282] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0283] For example, the method or apparatus according to the embodiment of the present disclosure may also be implemented by Figure 4 The architecture of the computing device 3000 shown in FIG. Figure 4 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided in the present disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of a computing device are shown.

[0284] The embodiment of the present invention further provides a computer-readable storage medium, such as Figure 5As shown, the computer storage medium 4020 stores computer readable instructions 4010. The computer readable instructions 4010, when executed by a processor, perform a method according to embodiments of the present disclosure, as described above with reference to the accompanying drawings. The computer readable storage medium in the embodiments of the present disclosure can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SynchBurst Dynamic Random Access Memory (SLDRAM), and Direct Rambus Dynamic Random Access Memory (DR RAM). It is to be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It is to be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0285] The embodiments of the present disclosure also provide a computer program product or computer program, which, when executed by a processor, implements the steps of the above method. Figure 2 As shown, the computer program product or computer program includes:

[0286] The acquisition module 201 is configured to acquire a training set image and an initial action, wherein the image set contains a target region, and the action is a field planning;

[0287] The initialization state module 202 is configured to obtain an initial state based on the initial action, and evaluate the initial state to obtain an initial state score;

[0288] The DRL sampling action module 203 is configured to input the initial state into a DRL, sample a new action from the DRL, and obtain a second action;

[0289] The state score updating module 204 is configured to obtain a second state based on the second action, evaluate the second state to obtain a second state score;

[0290] The reward calculation module 205 is configured to obtain a reward of the second action based on a difference between the second state score and the initial state score;

[0291] The iteration training module 206 is configured to repeatedly schedule the sampling module, the state score updating module, and the reward calculating module to iteratively optimize the tumor angle planning model based on the target reward maximization until a stop condition is met, wherein the target reward comprises a reward of at least one second action.

[0292] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that carries out the specified functions or operations, or a combination of hardware and software.

[0293] In general, the various example embodiments of the present disclosure can be implemented in hardware or special-purpose circuits, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software which can be executed by a controller, microprocessor or other computing device, Although the various aspects of the disclosure can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0294] The skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0295] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the embodiments of the device described above are merely schematic; for example, the division of the units is only a logical function division; there can be another division manner for the actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0296] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0297] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0298] The example embodiments of the present disclosure described in detail above are merely illustrative, rather than restrictive. Those skilled in the art should understand that various modifications and combinations of these embodiments or their features can be made without departing from the principles and spirits of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for constructing a tumor field angle planning model based on deep reinforcement learning, characterized in that: The method comprises: S1: Acquire training set images and initial actions, wherein the training set images include a target area, and the initial action is a field layout plan; S2: obtaining an initial state based on the initial action, and evaluating the initial state to obtain an initial state score; S3: The initial state is input into DRL, and DRL samples the new action to obtain the second action; S4: obtaining a second state based on the second action, and evaluating the second state to obtain a second state score; S5: Obtaining a reward for a second action based on a difference between the second state score and the initial state score; S6: Based on the initial state-second action pair and the target reward, DRL repeats S3-S5 iterative optimization based on maximizing the target reward until the stopping condition is reached to obtain a tumor field angle planning model. The target reward includes the reward of at least one second action. After the iteration starts, the initial state-second action pair is replaced by the second state-next second action pair.

2. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: The initial state and the second state fuse image position features, and the image position features include the positions of the target area and the organ.

3. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: The specific steps to obtain the initial state based on the initial action include: Step 1: obtaining a position matrix based on the image, wherein the numbers in the position matrix represent target areas or organs; Step 2: Inputting the image and the initial action into a dose calculation model to obtain an initial dose distribution matrix; Step 3: Fusing the position matrix and the initial dose distribution matrix to obtain the initial state; the fusion method is: adding the position matrix and the initial dose distribution matrix to obtain the initial state, or scaling the initial dose distribution matrix to obtain a scaled dose matrix, and adding the scaled dose matrix and the position matrix to obtain the initial state.

4. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 3, characterized in that: The position matrix obtained based on the image is specifically as follows: the elements of the position matrix correspond to the voxels of the image; different numbers are used to represent different target areas and organs, and the numbers of the target areas or organs corresponding to the voxels are filled into the corresponding elements of the matrix.

5. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 4, characterized in that: Obtaining a position matrix based on an image also includes depicting the position and boundaries of the target area and organ from the image; when the target area and the organ intersect in the image, filling the sum of the numbers corresponding to the target area and the organ into the corresponding element of the position matrix; when the target area and the target area intersect in the image, filling the number of the high-dose target area into the corresponding element of the position matrix.

6. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 3, characterized in that: The position matrix represents multiple target areas, organs, external parts, and internal non-target areas and non-organ parts with different numbers.

7. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 6, characterized in that: The organs include: eyeball, optic nerve, optic chiasm, and brain stem.

8. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 3, characterized in that: The dose calculation model is a clinical planning system.

9. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 3, characterized in that: The scaling is as follows: based on the target dose, the upper limit is 120% of the maximum prescribed dose target, and the dose is scaled to the decimal part in equal proportion.

10. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: The target reward is the sum of the sample rewards sampled by deep reinforcement learning, or the sum of the sample rewards sampled by deep reinforcement learning from the experience replay pool.

11. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: The field distribution plan is the field angle of each irradiation field, and the field angle corresponds to N irradiation fields.

12. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 11, characterized in that: The action space of DRL is the range of changes in the field angle, and the range of changes in the field angle is [0,360].

13. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 12, characterized in that: The action space of DRL is a combination of possible changes in the angles of N irradiation fields.

14. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 12, characterized in that: The initial action is to evenly divide the irradiation field angle variation interval into N irradiation fields.

15. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 12, characterized in that: The action space of deep reinforcement learning is a discrete value taken according to the value interval in the variation range of the field angle.

16. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 11, characterized in that: The range of the field angle is expressed as: in is the nth real number of the output optimal field angle, An is the angle of the nth field, is the angle of the n-1th field, is the angle of the first field, the function F(x) unifies the field angle to between 0 and 360 degrees, and discretizes it to an integer multiple of 5; N represents the number of irradiation fields, interval represents the interval of the field angle; f represents the first and angular difference.

17. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 3, characterized in that: The state score includes an organ score, which is obtained based on an initial dose distribution matrix or an initial state.

18. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 17, characterized in that: The organ score is the sum of the scores of one or more of the following organs at risk: lens, optic nerve, optic chiasm, and brainstem.

19. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 17, characterized in that: The organ score is calculated based on the radiation dose to the organ at risk; The organ scores were calculated as follows: Among them, Score represents the corresponding organ score, ScoreRange is the highest score that the current organ can obtain, Max is the maximum dose of the current organ, Low represents the volume percentage of the organ with a score of 0, and High represents the volume percentage of the organ with a score of full.

20. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 19, characterized in that: The organ scores are calculated by proportionally scaling the maximum prescription dose of the organ to obtain the radiation dose of each organ.

21. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: The status score also includes one or more of the following: consistency index, uniformity index, and drop score.

22. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: DRL includes one or more of the following: TD3, PPO, SAC algorithms.

23. A tumor field angle optimization method based on deep reinforcement learning, characterized in that: The method comprises: Obtain the target area image and initial action to be planned; An initial state is obtained based on the target area image to be planned and the initial action; The initial state is input into a tumor field angle planning model to output an optimized action, where the action is a field plan. The tumor field angle planning model is obtained based on the method according to any one of claims 1 to 22.

24. A tumor field angle adaptive optimization method based on transfer learning, characterized in that: The method comprises: S31: Acquire the target area image and initial action to be planned; S32: obtaining an initial state based on the target area image to be planned and the initial action; S33: performing transfer learning on the tumor radiation field angle planning model to obtain a transfer DRL model, wherein the tumor radiation field angle planning model is obtained based on the method according to any one of claims 1 to 22; S34: The initial state is input into the migration DRL model, and the action is updated to obtain a third action; S35: obtaining a third state based on the target area image to be planned and the third action, and evaluating the third state to obtain a third state score; S36: Obtaining a reward for a third action based on the third state score, where the reward for the third action is the third state score or the difference between the third state score and the initial state score; S37: Based on the initial state-third action pair and the target reward, the migration DRL model repeats steps S34-S6 based on maximizing the target reward until the stopping condition to obtain a tumor field angle planning model, wherein the target reward includes a reward for at least one third action. After the iteration starts, the initial state-third action pair is replaced with the third state-next third action pair.

25. A method for accelerating tumor field angle optimization based on reinforcement learning, characterized in that: The iterative process of the method described in any one of claims 1 to 22 is completed using multi-agent parallel sampling.

26. The method for accelerating tumor field angle optimization based on reinforcement learning according to claim 25, characterized in that: The multi-agent system optimizes the main thread and the multi-agent threads through reinforcement learning to achieve parallel sampling of the same planning system.

27. The method for accelerating tumor field angle optimization based on reinforcement learning according to claim 26, characterized in that: The main thread sends a sampling field angle request to the agent of each agent thread. After detecting the sampling request from the main thread, the agent initializes the local environment and uses the local model to generate an action according to the current state. The local planning system obtains the irradiation field dose distribution according to the generated action. After the reward is calculated based on the irradiation field dose distribution, the sampled field angle and the corresponding reward are stored in the agent thread sampling data. The main thread iteratively optimizes and updates the parameters based on the sampling data of each agent thread and sends the updated parameters to each agent thread. After receiving the updated parameters, each agent thread updates the local model.

28. The method for accelerating tumor field angle optimization based on reinforcement learning according to claim 27, characterized in that: Each Agent thread is deployed through a virtual machine.

29. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 28.

30. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 28 are implemented.

31. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 28 are implemented.

Images