Tumor radiation field angle optimization method and device based on deep reinforcement learning and medium

Abstract

The invention belongs to the field of intelligent medical treatment, and particularly relates to a tumor radiation field angle optimization method and device based on reinforcement learning and a medium. The method comprises the following steps: acquiring a to-be-planned target region image and an initial state; inputting the initial state into a tumor radiation field angle planning model to output an optimized field arrangement plan, wherein a construction method of the tumor radiation field angle planning model comprises the following steps: S1, obtaining a training set image and an initial action; s2, obtaining an initial state and an initial state score based on the initial action; s3, inputting the initial state into DRL, and sampling a new action by the DRL to obtain a second action; s4, obtaining a second state and a second state score based on the second action; s5, obtaining a reward of a second action based on the second state score; and S6, according to the state-action pair and the target reward, iterative optimization is carried out until the condition is stopped, and then a tumor radiation field angle planning model is obtained. The DRL can recommend the field arrangement plan with the high plan score according to the current state.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical care, and more specifically, to a tumor field angle optimization method, device, medium and program product based on deep reinforcement learning. Background Art

[0002] There are many types of brain tumors, and different types of tumors respond very differently to treatment. The heterogeneity of tumors increases the complexity and difficulty of treatment. Surgery is the main means of treating brain tumors, but due to the particularity of the tumor's location, some tumors are difficult to completely remove and the surgical risk is high. Chemotherapy drugs have difficulty penetrating the blood-brain barrier, which limits the effectiveness of chemotherapy. Radiotherapy plays an important role in the treatment of brain tumors, especially for tumors that cannot be completely removed and residual tumor cells after surgery. Radiotherapy is an important auxiliary treatment method. Advanced precision radiotherapy techniques, such as intensity-modulated radiation therapy (IMRT), can act on tumors more accurately, reduce damage to surrounding normal tissues, and improve treatment outcomes.

[0003] The basic steps of radiation plan design include: ① pre-treatment of the target area and organs at risk, and outlining of auxiliary structures; ② physicists design the field angle based on experience; ③ the physicists set the optimization conditions in the planning system, and the planning system performs dose optimization based on the location information of the target area and organs at risk and the set field angle; ④ the physicists adjust the field angle and optimization conditions appropriately based on the optimization results and experience until they meet clinical needs.

[0004] Existing technologies have done a lot of in-depth research on dose optimization, for example:

[0005] Patent "CN202180034806.0" discloses the use of reinforcement learning to locate dose-volume targets in radiotherapy plan optimization, and the reinforcement learning method aims to optimize the dose-volume target;

[0006] Patent "CN202011575203.5" discloses a radiotherapy automatic planning system, automatic planning method and storage medium, which uses a deep learning neural network to iteratively optimize the dose parameters in the radiotherapy plan based on the radiotherapy plan generated by TPS and the error of dose prediction;

[0007] Among them, the optimization of the field angle has received less attention.

[0008] However, the field angle parameter has not received enough attention. The radiation plan is often set by the physicist based on experience, and then the dose optimization of the subsequent steps is often not able to achieve the best effect. Especially for craniocerebral tumors, the radiation area is the human brain, and the requirements for radiation dose distribution are higher. In clinical work, most plans can meet clinical needs only by adjusting the dose optimization parameters. Especially for other tumors, such as nasopharyngeal carcinoma, cervical cancer, prostate cancer, etc., the field angle is evenly divided. However, due to the particularity of the nervous system, craniocerebral tumors need to be targeted according to the location of the target area and the relative position relationship with nearby organs. The field angle is very important in the treatment of craniocerebral tumors. A good field angle plan can significantly reduce the dose of normal organs. However, due to the large number of optional options for the field angle, complex influencing factors, lack of clear optimization indicators, and high trial and error time cost, it has not become the main parameter for plan optimization. Therefore, in order to further improve the quality of craniocerebral tumor radiotherapy plans, an automated method is urgently needed to generate high-quality field angle plans. Summary of the invention

[0009] In view of the above problems, the present invention provides a tumor radiation field angle optimization method based on deep reinforcement learning, which uses a deep reinforcement learning algorithm to find the optimal angle of the radiation field, improve the design efficiency of radiotherapy plans, and reduce radiation damage.

[0010] The present application (first aspect) discloses a method for constructing a tumor field angle planning model based on deep reinforcement learning, the method comprising:

[0011] S1: Acquire training set images and initial actions, wherein the image set includes a target area, and the action is a field layout plan;

[0012] S2: obtaining an initial state based on the initial action, and evaluating the initial state to obtain an initial state score;

[0013] S3: The initial state is input into DRL, and DRL samples a new action to obtain a second action;

[0014] S4: obtaining a second state based on the second action, and evaluating the second state to obtain a second state score;

[0015] S5: obtaining a reward for a second action based on a difference between the second state score and the initial state score;

[0016] S6: According to the initial state-second action pair and the target reward, DRL repeats S3-S5 iterative optimization based on maximizing the target reward until the stopping condition is reached to obtain the tumor field angle planning model, wherein the target reward includes at least one reward for the second action. After the iteration starts, the initial state-second action pair is replaced by the second state-next second action pair.

[0017] Furthermore, the initial state and the second state fuse image position features, and the image position features include the positions of the target area and the organ;

[0018] Optionally, the specific steps of obtaining the DRL initial state based on the initial action include:

[0019] Step 1: obtaining a position matrix based on the image, wherein the numbers in the position matrix represent the target area or organ;

[0020] Step 2: Inputting the image and the initial action into a dose calculation model to obtain an initial dose distribution matrix;

[0021] Step 3: Fusing the position matrix and the initial dose distribution matrix to obtain the initial state;

[0022] Optionally, the method for fusing the image position matrix and the initial dose distribution matrix is: adding the image position matrix and the initial dose distribution matrix to obtain the initial state.

[0023] Furthermore, the position matrix is ​​obtained based on the image: the elements of the position matrix correspond to the voxels of the image; different numbers are used to represent different target areas and organs, and the numbers of the target areas or organs corresponding to the voxels are filled into the corresponding elements of the matrix;

[0024] Optionally, obtaining the position matrix based on the image further includes describing the image to obtain the position and boundary of the target area and the organ;

[0025] Optionally, when the target area and the organ intersect in the image, the sum of the numbers corresponding to the target area and the organ is filled into the corresponding element of the position matrix;

[0026] Optionally, when the target volume and target volume intersect in the image, the number of the high-dose target volume is filled into the corresponding element of the position matrix;

[0027] Optionally, the method for fusing the initial dose distribution matrix and the position matrix is: scaling the initial dose distribution matrix to obtain a scaled dose matrix, and adding the scaled dose matrix and the position matrix to obtain the initial state;

[0028] Optionally, the position matrix represents multiple target areas, organs, external parts, and non-target areas and non-organ parts in the body with different numbers;

[0029] Optionally, the organs include: eyeball, optic nerve, optic chiasm, brain stem;

[0030] Optionally, the dose calculation model is a clinical planning system;

[0031] Optionally, the scaling is: based on the target dose, taking 120% of the maximum prescription dose target as the upper limit, scaling to the decimal part in equal proportion;

[0032] Optionally, the target reward is the sum of sample rewards sampled by deep reinforcement learning;

[0033] Optionally, the target reward is the sum of sample rewards sampled by deep reinforcement learning from the experience replay pool.

[0034] Furthermore, the field distribution plan is the field angle of each irradiation field;

[0035] Optionally, the action space of DRL is the range of changes in the field angle, and the range of changes in the field angle is [0,360];

[0036] Optionally, the field angle corresponds to N irradiation fields;

[0037] Optionally, the action space of the DRL is a combination of possible changes in the angles of the N irradiation fields;

[0038] Optionally, the initial action is to evenly distribute the irradiation field angle variation interval to N irradiation fields;

[0039] Optionally, the action space of deep reinforcement learning is discrete values ​​in the range of field angle variation according to the value interval;

[0040] Optionally, the range of the field angle is expressed as:

[0041]

[0042] where a n ∈[-1,1] is the nth real number of the best field angle output, An is the angle of the nth field, A n-1 is the angle of the n-1th field, A 1 is the angle of the first radiation field, the function F(x) unifies the radiation field angle to between 0 and 360°, and discretizes it to an integer multiple of 5; N represents the number of radiation fields, interval represents the interval of the radiation field angle; f represents the Ath n-1 and A 1 The angle difference

[0043] Optionally, the field angle is set at an interval of 30°;

[0044] Optionally, the state score includes an organ score, and the organ score is obtained based on an initial dose distribution matrix;

[0045] Optionally, the organ score is obtained based on an initial state;

[0046] Optionally, the organ score is the sum of the scores of one or more of the following organs at risk: lens, optic nerve, optic chiasm, brain stem;

[0047] Optionally, the organ score is calculated based on the radiation dose to the organ at risk;

[0048] Optionally, the organ score is calculated as follows:

[0049]

[0050] Among them, Score represents the corresponding organ score, ScoreRange is the highest score that the current organ can get, Max is the maximum dose of the current organ, Low represents the volume percentage of the organ with a score of 0, and High represents the volume percentage of the organ with a score of full.

[0051] Optionally, the organ score is calculated by scaling the maximum prescription dose of the organ dose in proportion to obtain the radiation dose of each organ;

[0052] Optionally, the state score also includes one or more of the following: consistency index, uniformity index, drop score;

[0053] The DRL includes one or more of the following: TD3, PPO, SAC algorithm.

[0054] A tumor radiation field angle optimization method based on deep reinforcement learning, comprising:

[0055] Obtain the target area image and initial action to be planned;

[0056] An initial state is obtained based on the target area image to be planned and the initial action;

[0057] The initial state is input into the tumor field angle planning model to output an optimized action, wherein the action is a field layout plan. The tumor field angle planning model is obtained based on the above-mentioned method for constructing a tumor field angle planning model based on deep reinforcement learning.

[0058] A tumor radiation field angle adaptive optimization method based on transfer learning, the method comprising:

[0059] S31: Acquire the target area image and initial action to be planned;

[0060] S32: obtaining an initial state based on the target area image to be planned and the initial action;

[0061] S33: performing transfer learning on the tumor radiation field angle planning model to obtain a transfer DRL model, wherein the tumor radiation field angle planning model is obtained based on a method for constructing a tumor radiation field angle planning model using deep reinforcement learning;

[0062] S34: Migrate the DRL model update action to obtain the third action;

[0063] S35: obtaining a third state based on the target area image to be planned and the third action, and evaluating the third state to obtain a third state score;

[0064] S36: obtaining a reward for a third action based on the third state score, where the reward for the third action is the third state score or the difference between the third state score and the initial state score;

[0065] S37: According to the initial state-third action pair and the target reward, the migration DRL model repeats steps S34-S6 iteratively based on maximizing the target reward until the stopping condition is reached to obtain a tumor field angle planning model, wherein the target reward includes a reward for at least one third action, and after the iteration starts, the initial state-third action pair is replaced by the third state-next third action pair.

[0066] A method to accelerate the optimization of tumor radiation angle based on reinforcement learning, using multi-agent parallel sampling to complete the iterative process of the above method;

[0067] Optionally, the multi-agents optimize the main thread and the multi-agent threads through reinforcement learning to achieve parallel sampling of the same planning system;

[0068] Optionally, the main thread sends a sampling field angle request to the Agent of each Agent thread. After detecting the sampling request from the main thread, the Agent initializes the local environment, generates an action according to the current state using the local model, and the local planning system obtains the irradiation field dose distribution according to the generated action. After the reward is calculated based on the irradiation field dose distribution, the sampled field angle and the corresponding reward are stored in the Agent thread sampling data. The main thread iteratively optimizes and updates the parameters based on the sampling data of each Agent thread, and then sends the updated parameters to each Agent thread. After receiving the updated parameters, each Agent thread updates the local model.

[0069] Optionally, each Agent thread is deployed through a virtual machine.

[0070] The present application also discloses a method for accelerating optimization of tumor radiation field angle based on reinforcement learning, using multi-agent parallel sampling to complete the iterative process of the above method;

[0071] Optionally, the multi-agents optimize the main thread and the multi-agent threads through reinforcement learning to achieve parallel sampling of the same planning system;

[0072] Optionally, the main thread sends a sampling field angle request to the Agent of each Agent thread. After detecting the sampling request from the main thread, the Agent initializes the local environment, generates an action according to the current state using the local model, and the local planning system obtains the irradiation field dose distribution according to the generated action. After the reward is calculated based on the irradiation field dose distribution, the sampled field angle and the corresponding reward are stored in the Agent thread sampling data. The main thread iteratively optimizes and updates the parameters based on the sampling data of each Agent thread, and then sends the updated parameters to each Agent thread. After receiving the updated parameters, each Agent thread updates the local model.

[0073] Optionally, each Agent thread is deployed through a virtual machine.

[0074] The second aspect of the present application discloses a system for constructing a tumor field angle planning model based on deep reinforcement learning, comprising:

[0075] Acquisition module: used to acquire training set images and initial actions, the image set includes the target area, and the action is the field layout plan;

[0076] Initialization state module: used to obtain an initial state based on the initial action, and evaluate the initial state to obtain an initial state score;

[0077] DRL sampling action module: used for the initial state input DRL, DRL sampling new action to obtain the second action;

[0078] A state score updating module: obtaining a second state based on the second action, and evaluating the second state to obtain a second state score;

[0079] Reward calculation module: obtaining a reward for the second action based on the difference between the second state score and the initial state score;

[0080] Iterative training module: used to repeatedly schedule the sampling module, the state scoring update module, and the reward calculation module to iteratively optimize based on the target reward maximization until the stopping condition is obtained to obtain a trained tumor field angle planning model, and the target reward includes a reward for at least one second action.

[0081] The present application also discloses a tumor field angle optimization system based on deep reinforcement learning, comprising:

[0082] The second acquisition module is used to acquire the target area image and initial action to be planned;

[0083] The second initialization module: obtaining an initial state based on the target area image to be planned and the initial action;

[0084] Output module: used to input the initial state into the tumor field angle planning model to output the optimized action, which is the field angle plan.

[0085] The present application also discloses a tumor radiation field angle adaptive optimization system based on transfer learning, comprising:

[0086] The third acquisition module is used to acquire the target area image and initial action to be planned;

[0087] The third initialization module is used to obtain an initial state based on the target area image to be planned and the initial action;

[0088] Transfer learning module: used for performing transfer learning on the tumor radiation field angle planning model to obtain a transfer DRL model;

[0089] The third sampling module: the initial state inputs the migration DRL model, and updates the action to obtain the third action;

[0090] A third state score updating module: used to obtain a reward for a third action based on the third state score, where the reward for the third action is the third state score or the difference between the third state score and the initial state score;

[0091] A third reward calculation module: used to obtain a reward for migrating the DRL model based on the third state score and the initial state score;

[0092] The third training module is used to schedule the third sampling module, the third state scoring update module, and the third reward calculation module to repeatedly sample state-action pairs and then iteratively optimize based on reward maximization until the stopping condition is reached to obtain a migration adaptive DRL model;

[0093] The third output module is used to output the adaptive optimal field angle output according to the trained migration DRL model.

[0094] The third aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, it is used to execute the steps of the above method.

[0095] A fourth aspect of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0096] A fifth aspect of the present application discloses a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0097] This application has the following beneficial effects:

[0098] (1) This application uses a deep reinforcement learning algorithm to optimize the radiation field angle, a parameter that has not received enough attention and has been designed inaccurately in previous radiation plans; and the optimized field arrangement scheme achieves better results than the manual field arrangement by physicists;

[0099] (2) The application uses the deep reinforcement learning algorithm to optimize the field angle and finds a field angle that is inconsistent with clinical experience. However, medical examinations have found that the field angle optimized by deep reinforcement learning is more effective. Therefore, the deep reinforcement learning field angle planning model of this application can be used to guide physicists in clinical field layout; help physicists find field layout plans that are inconsistent with clinical experience but can achieve better results;

[0100] (3) In this application, the irradiation field dose distribution and the target area and organ position are fused and input into the DRL for training, so that the DRL learns the strategy of field arrangement according to the target area position and the position of the organ at risk. This strategy can be generalized and applied to new field arrangement requirements.

[0101] (4) After using a deep reinforcement learning-based algorithm to optimize the parameters of the radiation plan for transfer learning, this application can further optimize the individualized optimal solution based on adaptive deep learning to achieve a better individualized targeted design;

[0102] (5) The design of multi-agent sampling in this scheme realizes reinforcement learning using a real clinical commercial planning system rather than a simple dose calculation model, so that the strategies learned by reinforcement learning can be better transferred to clinical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0104] Figure 1 is a schematic diagram of a method flow chart provided by the first aspect of an embodiment of the present invention;

[0105] Figure 2 is a schematic diagram of a program product provided by the second aspect of an embodiment of the present invention;

[0106] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0107] Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention;

[0108] Figure 5is a schematic diagram of a storage medium provided by an embodiment of the present invention;

[0109] Figure 6 is a network structure diagram of a SAC algorithm provided by an embodiment of the present invention;

[0110] Figure 7 is a schematic diagram of a network layer of an actor network in a SAC algorithm provided by an embodiment of the present invention;

[0111] Figure 8 is a schematic diagram of a network layer of a critic network in a SAC algorithm provided by an embodiment of the present invention;

[0112] Fig. 9 It is a schematic diagram of the interaction process between the main training optimization thread and the multi-agent sampling during multi-agent sampling provided by an embodiment of the present invention;

[0113] Fig.10 Schematic diagram of a training process of a reward function during multi-agent sampling provided by an embodiment of the present invention;

[0114] Fig.11 It is a schematic diagram of a training process of a loss function of a Critic network during multi-agent sampling provided by an embodiment of the present invention;

[0115] Fig.12 It is a comparison diagram of the training set plan scores after reinforcement learning training provided by an embodiment of the present invention; Figure A is a dot graph, and Figure B is a box graph;

[0116] Fig.13 It is a comparison chart of test set plan scores after reinforcement learning training provided by an embodiment of the present invention;

[0117] Fig.14 It is a schematic diagram showing that a solution provided by a model provided by an embodiment of the present invention is superior to a field arrangement solution of a clinical physicist;

[0118] Fig.15 It is a schematic diagram showing that a solution provided by another model provided by an embodiment of the present invention is superior to a field arrangement solution of a clinical physicist;

[0119] Fig.16 It is a schematic diagram for analyzing the reasons why a solution provided by a model provided by an embodiment of the present invention is superior to a field arrangement solution provided by a clinical physicist;

[0120] Fig.17 A comparison diagram of dose distribution under an initial field angle and a dose distribution under an optimized field angle provided by an embodiment of the present invention;

[0121] Fig.18A schematic diagram of an iterative intermediate process of a deep reinforcement learning model provided by an embodiment of the present invention;

[0122] Fig.19 A schematic diagram of constructing a position matrix according to target areas and organs provided in an embodiment of the present invention;

[0123] Fig. 20 A schematic diagram of a DRL iterative optimization process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0124] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0125] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The sequence numbers of the operations, such as S101, S102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0126] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0127] Figure 1 1 is a flow chart of a method for evaluating PATENTNAME provided by an embodiment of the present invention. Specifically, the method comprises the following steps:

[0128] S101: Acquire training set images and initial actions of the DRL, wherein the image set includes a target area, and the action is a field layout plan;

[0129] S102: obtaining an initial state of the DRL based on the initial action, and evaluating the initial state to obtain an initial state score;

[0130] S103: DRL samples the new action to obtain the second action;

[0131] S104: obtaining a second state based on the second action, and evaluating the second state to obtain a second state score;

[0132] S105: Obtaining a reward for a second action based on a difference between the second state score and the initial state score;

[0133] S106: According to the second action-second state pair and the target reward, DRL repeats S103-S105 iterative optimization based on maximizing the target reward until the stopping condition is reached to obtain a tumor field angle planning model, wherein the target reward includes a reward for at least one second action.

[0134] Optionally, the target reward is the sum of sample rewards sampled by deep reinforcement learning from the experience replay pool.

[0135] In some embodiments, the stopping condition is that the target reward no longer increases after a certain number of iterations, at which point the stopping condition is reached.

[0136] First of all, the reinforcement learning algorithm used in this application is able to clearly explain the prior art of reinforcement learning of this application. Although the reinforcement learning algorithm is a prior art, its application to the field of field angle optimization is the first of its kind in this application, and the background technology of the reinforcement learning algorithm does not affect the creativity of this application.

[0137] Reinforcement learning consists of two themes and four parts. The two main parts are:

[0138] Agent: An agent is an individual who takes action.

[0139] Environment: Environment, which can provide feedback to behavioral actions;

[0140] The four parts include: A, S, R, and P; A represents the action space, that is, the space of all actions in which the agent takes action; S represents the state space, which is the feedback state after the agent takes action; R represents the reward, which is a real number. Positive values ​​represent rewards, and negative values ​​represent penalties; P represents the strategy, that is, what action a∈A the agent will take in the state s∈S. (Sometimes A is used to represent the strategy)

[0141] Deep reinforcement learning is the process in which the agent takes action A according to the state S in the environment E and continuously trains to generate strategy P in order to obtain the maximum reward R.

[0142] Algorithm development: From the Q-learning algorithm of reinforcement learning to the DQN algorithm of deep reinforcement learning, the DQN algorithm developed into the DDPG algorithm, and the DDPG algorithm developed into the TD3 algorithm, and the TD3 algorithm developed into the SAC algorithm.

[0143] Symbolic representation in deep reinforcement learning:

[0144] π: Function symbol, representing the policy function. The parameter is often the state returned by the environment, and the output is a specific action. It can be seen that a=π(s)∈A.

[0145] r: The immediate reward obtained after taking action a.

[0146] G: cumulative reward, G t represents the cumulative reward from time t to the end of the action;

[0147] Q π (s,a): state-action value function, sometimes also referred to as action value function, which is the expected value of the cumulative reward expected to be obtained by taking action a and using strategy π at time t in state s.

[0148] V π (s): state value function, which is the expected value of the cumulative reward obtained by taking action a and using strategy π at time t and in state s.

[0149] The SAC algorithm can be divided into two parts: strategy evaluation and strategy improvement:

[0150] In the strategy evaluation part, the state value function Vsoft and the action value function Qsoft are defined as:

[0151] V soft (s t )=E[Q(s t ,a t )]-lgπ(a t |s t )

[0152] Q soft (s t ,a t )=r(s t ,a t )+γE[V_soft(s t+1 )

[0153] In the strategy improvement part, the relative entropy optimization strategy is used. Sampling strategy and reward function: The sampling strategy refers to the selection of DNN samples input during deep reinforcement learning. Usually, the current environment state s is selected. t .In order to improve the convergence and fitting ability of deep reinforcement learning,

[0154] SAC (Soft Actor-Critic) is a model-based deep reinforcement learning algorithm that combines the advantages of the Actor-Critic algorithm and the model-based policy optimization algorithm. The SAC algorithm can efficiently solve reinforcement learning problems in continuous action spaces, especially for problems with high-dimensional state spaces and action spaces.

[0155] 1. The basic principles of the SAC algorithm are as follows:

[0156] (1) Establish a value function to evaluate the pros and cons of taking different actions in a given state, that is, to evaluate the value of the “state-action pair”;

[0157] (2) Establish a policy function to determine what action to take under a given state.

[0158] (3) Learn the value function and policy function by maximizing the expected return.

[0159] (4) Use the learned value function to evaluate the policy and use the learned policy function to select the action with the maximum expected reward.

[0160] The basic process of the SAC algorithm to solve reinforcement learning problems:

[0161] 1. Initialization: Define the architecture of the neural network and initialize the parameters of the value function and policy function.

[0162] 2. Collect experience: Execute the current strategy in the environment and record information such as state, action, reward, and next state.

[0163] 3. Update the value function: Using the collected experience, the value function is updated using the gradient descent method.

[0164] 4. Update the policy function: Using the collected experience, the policy function is updated using the gradient ascent method.

[0165] 5. Select an action: Select an action and execute it based on the current state and the learned policy function.

[0166] 6. Repeat steps 2-5 until the termination condition is met, such as reaching the maximum number of iterations or the strategy converges.

[0167] Finally, the SAC algorithm outputs the learned optimal action, i.e., the field of view angle, based on the learned strategy function.

[0168] In order to solve the technical problems raised by this application, the following elements are defined based on the problems of this application:

[0169] In some embodiments, when only one irradiation field is included, the action space includes any action within the angle range of [0,360]. For example, 60 indicates that the action taken is to rotate a certain irradiation field angle to a direction of 60 degrees.

[0170] In some embodiments, when multiple fields are included, the action space includes a combination of all irradiation fields within the angle range of [0,360]; wherein the angle of the kth field must be greater than the angle of the first k-1 fields, that is, the action interval of the kth field is reduced to (the angle of the kth field, 360 degrees]. An action in the action space is represented as: [a 1 ,a 2 , ..., a k , ..., a N ], N represents the total number of irradiation fields, a k represents the angle of the kth field.

[0171] In some embodiments, when multiple fields are included, the action space includes any action within the angle range of [0,360], [a 1 ,a 2 , ..., a k , ..., a N ] represents the path of an action, N represents the total number of irradiation fields, a k represents the angle of the kth field.

[0172] In some embodiments: a state in the state space is defined as an irradiation field dose distribution including position information: the position includes the target area position and the organ.

[0173] In some embodiments, a positive value is assigned to the region where the PTV is located, a negative value is assigned to the region between the PTV and the Body, and a value of 0 is assigned to the region outside the Body.

[0174] In some embodiments, the irradiation field is divided into n regions according to the position of the PTV, and the numbers 2 to n+1 are used to represent the 1st to nth regions / positions respectively, and the region of the target region is assigned a value, and the region where the target region is located starts from 2.0 and is divided into an integer part and a decimal part. The integer part represents the target region, and the decimal part represents the dose of the target region.

[0175] In some embodiments, when acquiring the position of a target area or organ based on an image, the position and boundary of the target area or organ need to be manually traced.

[0176] In some embodiments, when acquiring the target area and organ position based on images, it is necessary to use AI to identify the position and boundaries of the target area and organ.

[0177] In some embodiments, the target area is divided into target areas 1-n according to the possible location of the brain tumor target area in the brain, and a continuous decimal is used to represent the state of the whole brain, such as Fig.19 As shown on the left, the purple part marks the target area 1; the yellow part is the target area 2;

[0178] The target area 1 value range is [2.0~2.1], where the integer part of the number 2 indicates that the PTV is in the target area 1 (purple area), and the value strategy of the decimal part is: 120% of the current planned maximum prescription dose target area is set to 0.1 (this value is not absolute and will be set according to actual conditions), and other doses are scaled proportionally.

[0179] The value range of target area 2 is [3.0~3.1];

[0180] The value range of target area 3 is [4.0~4.1];

[0181] If there is overlap between target areas, the high-dose target area will be given priority, such as Fig.19 Shown on the left, where the purple and yellow target areas overlap, the high-dose purple area is used to represent it.

[0182] Through this design, the state value reflects both the target location and the dose received by the target.

[0183] Assuming that the maximum prescription dose is 60 Gy, the specific meaning of 3.08 is that the pixel point is located in target area 2, and the corresponding dose is 60*1.2*0.8=57.6 Gy.

[0184] In some embodiments, when characterizing the status, all organs at risk take values ​​from 0.1 to 1, and the specific value range of each organ depends on the number of organs at risk. If the number of organs at risk does not exceed 9, the value range of each organ is 0.1. If the number of organs increases, it is further subdivided, for example:

[0185] Organ 1: 0.1~0.2; Organ 2: 0.2~0.3; Organ 3: 0.3~0.4; The value-taking rule for the decimal part is the same as the decimal rule for the target area, and 120% of the maximum prescription dose is proportionally scaled within the value range of each organ.

[0186] If the organ and the target area overlap, the integer part uses the target area and the decimal part uses the organ. For example, 3.38, the integer part corresponds to target area 3, and the decimal part is in organ 3. This means that the pixel is located in the overlapped part of target area 3 and organ 3, and its dose is 60*1.2*0.8=57.6Gy. Fig.19 As shown in the left figure, the green part represents the brainstem, and the yellow part represents the target area. The integer part 3 represents the target area 2, and the decimal part 3 represents the organ 3 (brainstem). Fig.19 The right side of the image shows a tracing of the lens organ.

[0187] Through the above method, the position information is integrated into the state representation of the state space. Deep reinforcement learning updates Q by learning the "state-action pair" and further updates the strategy (such as Fig. 20 As shown in the figure, the DRL strategy learns the position-based action plan, which is also the field design plan.

[0188] In some embodiments, a state score (also called a plan score) is calculated by an organ score;

[0189] In some embodiments, the state score is calculated by combining the organ score, the consistency index, the uniformity index, and the drop score;

[0190] In some embodiments, the reward is represented by the difference between the state score after the state is updated by sampling the field angle and the state score corresponding to the initial state.

[0191] In some embodiments, the ultimate goal of reinforcement learning is to learn the action that maximizes the reward. 1 ,a 2 ,…,a k ,…,a N ], a 1 Corresponding to the angle of the first field (the first step), a 2 Corresponding to the angle of the second field (step 2), ...a 7 Corresponding to the angle of the 7th field (the 7th action), the kth field takes action a k After that, the irradiation field dose distribution matrix is ​​obtained through the planning system, and the state corresponding to the action is obtained after fusion with the target area position. The state score is obtained based on the state calculation, and the loss between the new state score and the initial state score is used as the reward. Through the optimization iteration of reinforcement learning, the action combination of 7 steps that can obtain the best reward is learned.

[0192] In some embodiments, the target area information of craniocerebral tumors of 236 patients was collected, and three-dimensional dose distribution was used as input. The script written by ESAPI was used to interact with the ECLIPSE planning system, and multi-agent parallel sampling was used to greatly improve the sampling efficiency. The reinforcement learning field angle optimization model was obtained after iterative optimization using the Soft Actor-Critic (SAC) algorithm. The target area of ​​the training set was input into the model to obtain the field angle of the test set. The final field layout plan was generated after the planning system optimized the dose based on the field angle. The results showed that the field layout plans generated by the model in the training set were better than the initial plan with 7 fields evenly divided, and the average plan score increased by 18.9 points, and the average score of the manual plan of the physicist was increased by 3.4 points. The average score of the 7-field evenly divided plan in the test set was increased by 7.3 points. It is concluded that the DRL-based field angle selection algorithm designed in this application can comprehensively consider multiple factors, explore and master methods and techniques beyond clinical practice, and generate high-quality field layout plans.

[0193] In some embodiments, when applying DRL to solve the problem of field optimization radiotherapy planning, this study makes the following adjustments to the method: (1) the size of the training set is expanded; (2) 3D dose distribution is used as input, and the spatial position information of the target organ is considered. The SAC algorithm with good performance in continuous action space is adopted to prepare for the expansion of larger-scale action space in the future; (3) a multi-agent multi-threaded sampling framework interconnected by a local area network is designed to improve sampling efficiency.

[0194] In some embodiments, 236 cases of central nervous system tumors that have completed radiotherapy in the Department of Radiotherapy of the Second Hospital of Hebei Medical University from January 2022 to April 2024 were selected. All cases had only one target area, and were divided into five categories according to the approximate location of the target area: upper left, lower left, upper right, lower right, and center. They were evenly distributed to 7 parallel trained agents to ensure uniform sampling of targets in different positions. All cases used the same prescription dose, with a planned target volume (PTV) of 60Gy / 30 times. Among them, 212 cases were used for training and 24 cases were used for testing. This study used the AAA (Anisotropic Analytical Algorithm) algorithm of Eclipse version 15.6 for dose calculation. In order to more directly reflect the image of the field angle on the quality of the plan, this study set a set of relatively loose and fixed optimization conditions when optimizing the dose according to the field angle, as shown in Table 1:

[0195] Table 1 Optimization conditions for dose planning

[0196] Type Vol[%] Dose[Gy] Priority PTV Lower 99 60 200 PTV Upper 0 63 150 BrainStem Upper 0 52 80 Lens_L / R Upper 0 6 80 OpticChiasm Upper 0 48 80 OpticNerve_L / R Upper 0 48 80 External Upper 0 63 300

[0197] SAC (Soft Actor-Critic): This study uses SAC. Click or click here to enter text. Algorithm (see attached) Figure 6 The model was constructed using PADDLE 2.5.1 and PARL 2.2.1.

[0198] like Figure 6 As shown, the SAC algorithm includes an actor network and four critic networks. The four critic networks are:

[0199] ① State value (V) estimation network:

[0200] ②Target value (target V) network:

[0201] ③Action-state value estimation network Q0:

[0202] ④Action-state value estimation network Q1:

[0203] In the SAC algorithm, we model two action value functions Critic and one policy function Actor. Based on the idea of ​​Double DQN, SAC uses two Critic networks, but each time the Critic network is used, a network with a small value is selected to alleviate the problem of overestimation of the value. The pseudo code is shown in Table 2:

[0204] Table 2 SAC algorithm (Soft Actor-Critic)

[0205]

[0206]

[0207] In some embodiments, the iterative update process of the SAC algorithm shown in Table 2 is as follows: Fig. 20 As shown, the meanings of action and space in this application are shown in the DRL iteration schematic table in Table 3:

[0208] Table 3 DRL iteration diagram

[0209]

[0210] In some embodiments: We first use a customized MIM (7.3.2) workflow to unify the image size and outline the names of all patients. The image resolution is 512*512*128, and the pixel is 0.78125*0.78125*3 (mm). The initial field angle is divided into 7 fields equally. After optimization, a three-dimensional matrix of dose distribution is obtained. Then, a position matrix with the same three-dimensional dose dimension is constructed based on the image information: the PTV target area is divided into 3 areas according to the number of target areas, and the position of the PTV target area of ​​each case in the position matrix is ​​set to the corresponding number representing the target area; at the same time, the organs at risk include the left eyeball, right eyeball, optic nerve, optic chiasm, and brainstem; then the decimals within 1 are divided into these 5 areas, with 0.1 representing the left eyeball, 0.2 representing the right eyeball, 0.3 representing the optic nerve, and 0.4 representing the optic chiasm. 0.5 represents the brainstem. After such assignment, the rest of the three-dimensional position matrix is ​​set to 1 inside the brain and 0 outside the brain according to the values ​​inside and outside the brain. Thus, the construction of the three-dimensional position matrix is ​​completed. The three-dimensional dose distribution matrix is ​​scaled and summed with the three-dimensional position matrix to obtain the three-dimensional state matrix, which is then downsampled to 128*128*128, and the treatment isocenter (MARK) is located in the center of the space through translation. Finally, the processed three-dimensional dose distribution matrix is ​​input into the model as the observation of reinforcement learning.

[0211] Network structure: Based on the input and output dimensions of this study, we designed the actor network and critic network model structure as follows Figure 7 and Figure 8 As shown, Figure 7 The dimension of the final output of the actor network is [8,7]; 8 represents the batch size (that is, I randomly selected 8 planned data from the cache pool for training in a single training), and 7 represents 7 actions (called actions in reinforcement learning, that is, 7 real numbers that can be mapped to field angles);

[0212] In some embodiments, 8 samples are selected each time the model is updated, and the rewards of the 8 samples are added together to obtain a total reward, and then a curve records the change of total-reward. Once there is no increasing trend, the iteration is stopped to obtain the field angle planning model.

[0213] Figure 8 The final output of the network in the Critic network shown in the figure [8,1] represents the Q value of the 8 plans currently sampled;

[0214] Since commercial planning systems are expensive to use and have limited deployment resources, in order to utilize the few commercial planning systems to complete the sampling of reinforcement learning, this application implements parallel sampling by distributing multiple Agent threads to interact with the main thread, thereby improving the efficiency of sampling.

[0215] Because the radiotherapy planning system is commercial software, there will be license restrictions, which is not conducive to the training of reinforcement learning in this application. Taking the Eclipse planning system used in this study as an example, it can only calculate 2 plans at the same time, and the planning system usually needs clinical use, so it cannot undertake the research work of reinforcement learning. Therefore, the corresponding research is carried out on the Eclipse Tbox workstation configured by the company (for clinical learning use). A planning system is installed on the workstation. The relevant research on reinforcement learning here will not delay clinical work, but it still cannot meet the needs of high-quality research. Because of this limitation, the relevant researchers adopted two solutions: 1. Simplify the modeling of the planning system and simplify the dose calculation process to improve the sampling efficiency. This will undoubtedly bring errors and unpredictable problems. 2. Use a single planning system for sampling, but greatly simplify the clinical problems, but the simplified clinical problems no longer have clinical application value, but are more of a simple verification of feasibility.

[0216] In order to solve the problem of slow sampling, this study used a multi-agent parallel sampling framework for the first time. We virtualized the operating system on the Tbox using virtual machine software, so that multiple virtual machines can be opened for simultaneous calculations.

[0217] The operation of virtual machines consumes a lot of resources. The number of virtual machines that can be run on one computer still cannot meet the research needs, so we distributed the virtual machines on two computers, each running 3 virtual machines, sampling at the same time, plus the local planning system of Tbox, there are currently 7 agents doing sampling work. This is also due to insufficient hardware resources, otherwise there is no upper limit to the number of agents.

[0218] Because the program runs on different computers, data interaction is also a problem that must be solved. Because the script that interacts with the planning system uses C# language and the DRL algorithm uses Python language, we finally adopted the solution of LAN interconnection and json file interaction. The flow chart is as follows Fig. 9 As shown:

[0219] As can be seen from the flowchart, each Agent runs independently in its own environment, and the process will only perform read and write operations in the corresponding two folders. These two operations will not affect other threads, nor will they be affected by other threads. Similarly, the main thread does not interact with each Agent thread, so this achieves decoupling. Even if the Agent stops running for various reasons, the main thread cannot detect the sampled data in the corresponding folder and will skip it directly without affecting the entire training process. The only problem that may be encountered is that two threads operate on the same file at the same time. There is also corresponding error capture processing here. You only need to pause the error-reporting thread for a few seconds until no error is reported.

[0220] In some embodiments, multi-agent parallel sampling is used during the training process to speed up the training of reinforcement learning. Each agent interacts with the Eclipse planning system through the ESAPI script interface. The agents are located in multiple computers and connected to each other through a local area network. See the simplified flowchart for details. Fig. 9 :

[0221] The main thread sends a field angle sampling request to the agent in the Agent thread. After receiving the instruction from the main thread, the agent starts to execute sampling: first, the agent in the Agent thread initializes the local environment, uses the local model to generate an action (i.e., field angle) according to the current state, and the local planning system re-optimizes the dose according to the generated action and calculates the reward. The Agent thread stores the action and the corresponding reward in the sampling data folder. The main thread traverses the sampling data according to the environment where each Agent thread is located, and stores the sampling data in the experience replay pool. The reinforcement learning main thread randomly samples from the experience replay pool for training, updates the model parameters, and sends the updated parameters to each Agent thread to update the local model. Among them, the local planning system in the above process refers to the same commercial planning system, but based on the design of the solution of this application, it is possible to access the commercial planning system in parallel to obtain accurate reward calculation.

[0222] In some embodiments, when the Agent samples, the initially planned field angle is input into the Eclipse planning system through a script. After the planning system optimizes the calculation based on the input field angle to obtain the dose distribution, reinforcement learning is used to obtain the optimized field angle. The optimized field angle replaces the initially planned field angle and is input into the planning system to re-optimize the dose distribution to obtain an optimized radiotherapy plan.

[0223] In some embodiments, the number of fields is 7, and the motion path is represented by a t ={a 1 ,a 2 ,a 3 ,a4 ,a 5 ,a 6 ,a 7}, the model outputs a t It includes 7 float types, real numbers with values ​​ranging from -1 to 1, and after adding 1, the range becomes 0 to 2. According to experience, too small field intervals are usually meaningless, so we stipulate that the minimum field interval is 30°. The angles of each field are calculated by the following formula:

[0224]

[0225] where a n ∈[-1,1] is the nth real number of the best field angle output, An is the angle of the nth field, A n-1 is the angle of the n-1th field, A 1 is the angle of the first radiation field, the function F(x) unifies the radiation field angle to between 0 and 360°, and discretizes it to an integer multiple of 5; N represents the number of radiation fields, interval represents the interval of the radiation field angle; f represents the Ath n-1 and A 1 The angle difference

[0226] Where an is the nth real number output by the model, which also represents the action space of the nth field, ranging from [-1, 1], mapped to the angle range [0, 360]. An is the angle of the nth field. The function F(x) is responsible for discretizing the field angle to an integer multiple of 5 to reduce the training complexity, and at the same time, making the correct conversion when the field angle is greater than 360°.

[0227] At the same time, since the angles of the seven shooting fields cannot overlap, action a n The constraints include: the values ​​of the 7 actions cannot be equal.

[0228] Reward function: The difference between the plan score and the initial plan score (that is, the initial state score) is used as the reward function. The plan score / state score is the sum of the following items. The specific scoring rules for each item are as follows:

[0229] (1) Lens, OpticNerv, OpticChiasm, BrainStem scoring formula:

[0230]

[0231] Among them, Score is the organ score, ScoreRange is the highest score that the current organ can get, and the different highest scores of each organ represent different weights in the organ score. The lens is very fragile, and we hope that the algorithm can learn to protect the lens better, so it is given a high weight. Max is the maximum dose of the current organ, and Low and High are the dividing lines between full score and 0 points. The specifics for each organ are as follows:

[0232]

[0233] (2) Conformity Index (CI)

[18] :

[0234]

[0235] Among them TV RI represents the target volume covered by 95% of the prescribed dose, TV represents the target volume; V RI The volume representing 95% of the prescribed dose;

[0236] (3)Homogeneity Index(HI)

[19] :

[0237]

[0238] D2 and D98 represent the highest and lowest doses received in the target area, respectively, and Dprescription is the prescribed dose.

[0239] (4) Drop score

[0240]

[0241] V20initial is the volume covered by the initial plan of 20 Gy, and V20 is the volume covered by the current plan of 20 Gy.

[0242] This study used two programming languages, Python (DRL model training) and C# (Agent and planning system interactive sampling), and used Json files as the medium for data transmission to decouple each thread so that their operations do not interfere with each other. The accidental stop of any Agent will not affect the operation of the entire framework, greatly enhancing the stability of the system operation.

[0243] A complete sampling takes about 70 seconds on average. According to the scale of this study, a total of 80,000 samples need to be collected. If only one commercial planning system is used for interactive sampling, it will take about 64 days. However, the study uses 7 agents to conduct parallel sampling, which shortens the time to about two weeks. The training curve is as follows Fig.10 and Fig.11 shown.

[0244] In some embodiments, only the area where the PTV is located in the state space is assigned a positive value, the area between the PTV and the Body is assigned a negative value, and the area outside the Body is assigned a value of 0. 50 cases in the training set were randomly selected, and the final scores of the plans given by the model (Drl group) were all higher than the initial plans (Initial group) with an average score of 7 fields. Then, we manually arranged the field for 16 cases in the training set that did not conform to the clinical field arrangement habits (the physicists only designed the field arrangement plans according to the environment of this model and did not know the training results of this model). By comparing the manual field arrangement plans (Manual) of the physicists, only one plan scored slightly higher than the plan of the DRL group. The data are shown in Table 4:

[0245] Table 4 Comparison of scores between the initial plan and the DRL group plan in the training set

[0246]

[0247] The Kolmogorov-Smirnov test showed that the data of the two groups did not meet the normal distribution. Therefore, the Wilcoxon matched pairs signed rank test was used to compare the differences between the Initial group and the Drl group, and between the Drl group and the Manual group. The differences between the two groups were statistically significant (P<0.001).

[0248] The test set has a total of 24 cases, of which 18 cases scored higher than the initial state, and the remaining cases scored lower than the initial plan. The data are shown in Table 5 and Fig.13 As shown:

[0249] Table 5 Comparison of scores between the initial plan and the DRL group plan in the test set

[0250]

[0251] The Kolmogorov-Smirnov test showed that both groups of data were normally distributed, so the paired T test was used to compare the differences between the two groups, and it was found that there was a significant statistical difference between the two groups (P=0.001).

[0252] The examples shown in Table 4 and Table 5 only show the schematic effects when the state space is simply set (the area where the PTV is located is assigned a positive value, and the area between the PTV and the Body is assigned a negative value), which is not the optimal effect of the method of the present application.

[0253] In the above research of the present application, by combining the SAC algorithm, the trained model demonstrated its powerful exploration and learning capabilities. Since a higher ScoreRange is designed for the organ score of the lens in the present application, the organ score of the lens occupies a higher weight. At the same time, the target area position, the position of the endangered organ, the internal part, and the external part are distinguished in the state representation. Therefore, the field layout scheme for optimizing the field angle in the present application has learned to avoid the crystal; based on the drop score, it has learned to layout the field nearby. Based on the design of the field angle mapping algorithm, there is enough space for exploration, which can comprehensively consider the details ignored by the clinic, and learn and master the skills that have not been mastered in the clinic. When observing the field layout results of the training set, several cases were found that were completely opposite to the clinical field layout habits, but after comparative analysis, it was found that under the current scoring rules, the schemes given by the model were better than the field layout schemes of clinical physicists.

[0254] A typical case Fig.14 Shown: The target area in this case is located in the upper right. Fig.14 -A is the field deployment plan given by reinforcement learning. It does not follow the principle of proximity and deploys the field from the opposite side. Fig.14 -B is to arrange the field nearby according to clinical experience. However, after comparison, it was found that according to the scoring rules in the study, the score of the field arrangement scheme based on reinforcement learning was higher than that of the clinical scheme because it had fewer low-dose areas and fewer total MUs. Finally, after analysis, we found that the target area of ​​this case was close to the skin, and the total MU needed to be increased to allow the prescribed dose to cover the area under the skin, which also led to the expansion of the low-dose area and a lower overall score.

[0255] Another typical case Fig.15 Shown: The last layer of the target area near the foot side is located at the level where the lens is located. Fig.15 -A is the field layout method of reinforcement learning. You can see that the field of view directly passes through the crystal. Fig.15 -B is the clinical field arrangement method, which passes through the lens from the far side. In theory, the field arrangement method in Figure B should have a lower dose to the lens, but in reality, the field arrangement method given by reinforcement learning has a lower dose to the lens.

[0256] Reasons such as Fig.16 As shown: Because the rays are cone-shaped, the farther away from the handpiece, the larger the range of the rays. The field layout method of deep reinforcement learning makes the crystal closer to the handpiece. Under the premise of covering the target area, the crystal can be completely avoided. The commonly used field layout method in clinical practice is just the opposite, exposing the crystal to the radiation field, which ultimately leads to an increase in the dose of the crystal.

[0257] In some embodiments, the tumor planning model designed and trained in the present application is used in the clinical field angle setting stage, and the field angle output by the model is directly used for field arrangement.

[0258] In some embodiments, the tumor field angle planning model designed and trained in the present application is used in the clinical field angle auxiliary setting stage. The physicist compares the field angle output by the model with his own experience and selects a better plan for field arrangement.

[0259] In one embodiment, the physicist finds that the suggested field layout plan output by the tumor field angle planning model is significantly different from previous experience. After research, the physicist adopts the model's suggestion to set the field angle and then conducts subsequent planning.

[0260] DRL combines the advantages of deep learning and reinforcement learning, has strong learning and generalization capabilities, and has great potential to surpass the level of human experts, but this requires the support of massive data. DRL is undoubtedly the most successful in the field of games. There are two main reasons. First, the algorithm can interact directly with the game, without the need to model and simulate the environment, thus avoiding the introduction of modeling errors. Second, it can quickly obtain massive data by superimposing computing power.

[0261] The application of DRL in the field of radiotherapy plan design is different from that in the field of electronic games. In this plan, we will interact with the system through the script interface provided by the planning system to simulate the interaction of algorithms in the game field. However, the biggest bottleneck of this interaction in the field of radiation planning is the extremely low sampling efficiency, which seriously restricts the clinical implementation of DRL.

[0262] In order to break through this bottleneck, the solutions currently adopted by the research institutes are mainly divided into three categories. First, use the simple model established by oneself to perform dose calculation. The simple model for dose calculation and the dose calculation program built by oneself cannot be compared with the clinical one in terms of accuracy and precision, but it is simple and has high sampling efficiency. It can improve efficiency, but it will also introduce modeling errors. Second, simplify clinical problems as much as possible and narrow the exploration scope of the DRL algorithm so that it can converge with small samples. However, the strategies learned by this method are too simple and do not have much clinical application value. Third, increase the number of networks, with each network only responsible for the training of one parameter. This will greatly reduce the difficulty of network training and allow for a larger exploration space. It is currently a better research direction.

[0263] Unlike previous DRL studies, the algorithm used in this study did not use a simple dose calculation model, but directly interacted with the commercial planning system. Although this would reduce the sampling efficiency, it could ensure that the training results could be seamlessly transferred to the clinic without introducing additional errors.

[0264] In order to solve the problem of low sampling efficiency, we pioneered a multi-agent parallel sampling framework. Agents distributed on multiple computers in the local area network interact with the local Eclipse planning system to sample data, and aggregate the data to the computer where the main thread of the algorithm is located for algorithm training. Although limited by hardware computing power, this study only used 7 agents, but the training time was also kept within an acceptable range. As long as the computing power is sufficient, there is no upper limit to the number of agents used in theory. The multi-Agnet parallel sampling framework solves the core problem of DRL sampling, which is also the bottleneck that currently prevents DRL from being implemented in clinical practice. The framework lays a solid foundation for conducting larger-scale research in the future.

[0265] In order to better verify the potential of the DRL algorithm, this study was guided by clinical implementation. Instead of choosing a simplified clinical problem for verification, it chose the difficult problem of optimizing the radiation field angle of central nervous system tumors for verification. This is because the spatial relationship between the target area and the organs at risk of this type of disease is complex, and the radiation field angle combinations are varied, which provides ample space for learning and exploration. In order to ensure that good strategies are learned in a complex environment, the study used 212 cases for training, with three-dimensional dose distribution as the input state. Unlike other researchers who use DVH maps as input data, this solution not only provides dose information for the model, but also provides the shape and position information of PTV, the position information of MARK, and the outer contour information of the head, allowing the model to comprehensively consider multiple information and learn better field layout strategies.

[0266] In some embodiments, the study used the SAC algorithm for the first time, performing end-to-end training, and still had reliable performance in clinical problems that were more complex than previous studies. And because SAC is an optimization algorithm for continuous action space, it can give more meaning to the actions it generates and solve more complex clinical problems.

[0267] In some embodiments, the TD3 algorithm is used as the DRL model.

[0268] In some embodiments, the PPO algorithm is used as the DRL model to complete the training of the field angle planning model.

[0269] like Fig.17 Shown is a comparison diagram of the dose distribution under an initial field angle and the dose distribution under an optimized field angle; the left side is the dose distribution under the initial state, and the right side is the dose distribution of the state corresponding to the field angle learned by deep reinforcement learning. According to the dose contour lines in the figure, it can be seen that the low-dose area on the right is much lower than that on the left.

[0270] like Fig.18 Shown is a schematic diagram of the iterative intermediate process of a deep reinforcement learning model; Fig.18There are two stored plans in the upper left corner. One is DRL-reset, which indicates the initial 7-field evenly divided plan. The RTStruct in the lower left corner is the status score corresponding to the radiation plan. The score of the initial plan is 75.4 points (the reward at this time is 0). Best-reward is the plan with the highest reward. The highest score will be continuously updated as the training progresses. Fig.18 The current highest reward status score is shown as 971.36 points (the reward at this time is 971.36-75.4).

[0271] Taking into account the amazing performance of DRL in the training set, we further performed adaptive DRL, which not only limited the DRL environment to a single case, but also used the model trained with a large number of cases for transfer learning, and then trained the single case to generate the best field layout plan under the current scoring rules.

[0272] The purpose of adaptation is that the original amount of data or parameter settings are not enough for the model to learn all the position information and achieve the best generalization effect. Therefore, transfer is used, and then individual training is continued to make up for this deficiency.

[0273] In summary, this study constructed a multi-agent parallel sampling framework based on the Eclipse planning system for the first time, solving the problem of low sampling efficiency. The model selected in this study was directly trained using the commercial Eclipse planning system to avoid environmental bias, and it can be directly applied to clinical practice after training. The design of the study is universal and can be quickly migrated to the direction of automatic planning, laying the foundation for the research of automatic planning based on deep reinforcement learning. Most importantly, the study truly shows the great potential of DRL to go beyond clinical practice, and has learned very practical skills that professional physicists cannot master. As the research deepens and the complexity gradually increases, DRL will learn more practical skills. In short, this study has laid a solid foundation for the implementation of deep reinforcement learning in clinical practice.

[0274] Action a t The output in the Acotr network is a three-dimensional matrix of 128*128*128, and the output is 7 real numbers ranging from -1 to 1, corresponding to the angles of the 7 fields.

[0275] In some embodiments, we use adaptive DRL, which limits the DRL environment to a single case, uses the model trained with the current large batch of cases for transfer learning, and then trains on the single case. The SAC algorithm is naturally suitable for adaptation, that is, after the original large-scale samples are trained, replacing new samples can also quickly learn that the scheme will produce the best field layout scheme under the current scoring rules; all agents only sample on one case (that is, the training set has only one case, and this case uses cases that did not appear in the previous training set), and the model parameters used by the agent are the model parameters after the first step of training. In this way, the model will give a similar solution at the beginning, and then use the powerful learning and exploration capabilities of the SAC algorithm to find the optimal solution for the current case.

[0276] First, the outstanding performance of the training set proves that under the corresponding scoring rules, the solutions given by reinforcement learning comprehensively surpass those given by clinical staff. Therefore, it can be concluded that if the target plan is put into the training set, then the target plan will get the best solution under the current scoring rules, and the only problem that needs to be solved is the efficiency problem.

[0277] Secondly, the environment is complex, and there are not enough training samples and computing power support to allow the model to fully learn solutions in various situations. Although this experiment used more than 200 samples, far more than the 10 or 20 samples currently used by researchers in other related directions, the model's generalization ability is still poor. Therefore, it is not guaranteed that continuing to expand the sample data and continue to adjust the model parameters will result in an excellent clinically acceptable model. So change your mindset and use only one sample to greatly reduce the complexity of the environment, allowing the model to quickly explore and learn the optimal solution for the current sample. The main difference between the two solutions is the efficiency of use. Training is extremely time-consuming. This experiment took about 10 days to train with a sample size of about 200, and the trained model gave a solution in a very short time, about two or three minutes. However, if the sample is controlled to one case, the training time will definitely be greatly shortened. The application of transfer learning will further compress the time.

[0278] Finally, the use of a multi-agent parallel sampling framework can shorten the training time to a level acceptable for clinical use. Currently, hardware resources are very limited, with only 7 agents used. If the computing power is expanded to a server level of about 100,000 yuan, the number of available agents will be expanded to dozens, and the training time will be further improved.

[0279] It can be seen that the process is basically the same as the previous experiment. The only change is that the model parameters use the last trained parameters, which is transfer learning. The training set is changed from the original large sample database to the target plan. The rest is to observe the score of the target plan. The algorithm saves the plan with the highest reward and the plan with the highest status score at the same time. By observing the change curve of the status score, once it becomes flat, the training can be terminated, and the plan with the highest score is selected as the final plan.

[0280] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention, such as Figure 3 As shown, the device may include: one or more processors, and one or more memories; wherein the memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method described above may be executed.

[0281] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0282] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0283] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 4 The architecture of the computing device 3000 shown in FIG. Figure 4As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as ROM 3030 or hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of a computing device are shown.

[0284] The embodiment of the present invention also provides a computer-readable storage medium, such as Figure 5 As shown, it is a schematic diagram of a storage medium provided by an embodiment of the present invention, and a computer readable instruction 4010 is stored on the computer storage medium 4020. When the computer readable instruction 4010 is executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0285] The present disclosure also provides a computer program product or a computer program, which implements the steps of the above method when executed by a processor, such as Figure 2 As shown, the computer program product or computer program comprises:

[0286] Acquisition module 201: used to acquire training set images and initial actions, the image set includes the target area, and the action is a field layout plan;

[0287] Initialization state module 202: used to obtain an initial state based on the initial action, and evaluate the initial state to obtain an initial state score;

[0288] DRL sampling action module 203: used for the initial state input DRL, DRL sampling new action to obtain the second action;

[0289] State score updating module 204: obtaining a second state based on the second action, and evaluating the second state to obtain a second state score;

[0290] Reward calculation module 205: obtaining a reward for the second action based on the difference between the second state score and the initial state score;

[0291] Iterative training module 206: used to repeatedly schedule the sampling module, the state scoring update module, and the reward calculation module to iteratively optimize based on the maximum target reward until the stopping condition is reached to obtain a trained tumor field angle planning model, wherein the target reward includes a reward for at least one second action.

[0292] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0293] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0294] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0295] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0296] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0297] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0298] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for constructing a tumor field angle planning model based on deep reinforcement learning, characterized in that: The method comprises: S1: Acquire training set images and initial actions, wherein the image set includes a target area, and the action is a field layout plan; S2: obtaining an initial state based on the initial action, and evaluating the initial state to obtain an initial state score; S3: The initial state is input into DRL, and DRL samples a new action to obtain a second action; S4: obtaining a second state based on the second action, and evaluating the second state to obtain a second state score; S5: obtaining a reward for a second action based on a difference between the second state score and the initial state score; S6: According to the initial state-second action pair and the target reward, DRL repeats S3-S5 iterative optimization based on maximizing the target reward until the stopping condition is reached to obtain the tumor field angle planning model, wherein the target reward includes at least one reward for the second action. After the iteration starts, the initial state-second action pair is replaced by the second state-next second action pair.

2. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: The initial state and the second state combine the image position features. The image position features include the positions of the target area and the organ; Optionally, the specific steps of obtaining the DRL initial state based on the initial action include: Step 1: obtaining a position matrix based on the image, wherein the numbers in the position matrix represent the target area or organ; Step 2: Inputting the image and the initial action into a dose calculation model to obtain an initial dose distribution matrix; Step 3: Fusing the position matrix and the initial dose distribution matrix to obtain the initial state; Optionally, the method for fusing the image position matrix and the initial dose distribution matrix is: adding the image position matrix and the initial dose distribution matrix to obtain the initial state.

3. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 2, characterized in that: The position matrix is ​​obtained based on the image in detail: the elements of the position matrix correspond to the voxels of the image; different numbers are used to represent different target areas and organs, and the numbers of the target areas or organs corresponding to the voxels are filled into the corresponding elements of the matrix; Optionally, obtaining the position matrix based on the image further includes describing the image to obtain the position and boundary of the target area and the organ; Optionally, when the target area and the organ intersect in the image, the sum of the numbers corresponding to the target area and the organ is filled into the corresponding element of the position matrix; Optionally, when the target volume and target volume intersect in the image, the number of the high-dose target volume is filled into the corresponding element of the position matrix; Optionally, the method for fusing the initial dose distribution matrix and the position matrix is: scaling the initial dose distribution matrix to obtain a scaled dose matrix, and adding the scaled dose matrix and the position matrix to obtain the initial state; Optionally, the position matrix represents multiple target areas, organs, external parts, and non-target areas and non-organ parts in the body with different numbers; Optionally, the organs include: eyeball, optic nerve, optic chiasm, brain stem; Optionally, the dose calculation model is a clinical planning system; Optionally, the scaling is: based on the target dose, taking 120% of the maximum prescription dose target as the upper limit, scaling to the decimal part in equal proportion; Optionally, the target reward is the sum of sample rewards sampled by deep reinforcement learning; Optionally, the target reward is the sum of sample rewards sampled by deep reinforcement learning from the experience replay pool.

4. The method for constructing a tumor field angle planning model based on deep reinforcement learning according to claim 1, characterized in that: The field distribution plan is the field angle of each irradiation field; Optionally, the action space of DRL is the range of changes in the field angle, and the range of changes in the field angle is [0,360]; Optionally, the field angle corresponds to N irradiation fields; Optionally, the action space of the DRL is a combination of possible changes in the angles of the N irradiation fields; Optionally, the initial action is to evenly distribute the irradiation field angle variation interval to N irradiation fields; Optionally, the action space of deep reinforcement learning is discrete values ​​in the range of field angle change according to the value interval; Optionally, the range of the field angle is expressed as: where a n ∈[-1,1] is the nth real number of the best field angle output, An is the angle of the nth field, A n-1 is the angle of the n-1th field, A1 is the angle of the 1st field, and the function F(x) unifies the field angle to between 0 and 360° and discretizes it to an integer multiple of 5. N represents the number of irradiation fields, interval represents the interval of the field angle, and f represents the A-th field. n-1 The angle difference with A1; Optionally, the field angle is set at an interval of 30°; Optionally, the state score includes an organ score, and the organ score is obtained based on an initial dose distribution matrix; Optionally, the organ score is obtained based on an initial state; Optionally, the organ score is the sum of the scores of one or more of the following organs at risk: lens, optic nerve, optic chiasm, brain stem; Optionally, the organ score is calculated based on the radiation dose to the organ at risk; Optionally, the organ score is calculated as follows: Among them, Score represents the corresponding organ score, ScoreRange is the highest score that the current organ can get, Max is the maximum dose of the current organ, Low represents the volume percentage of the organ with a score of 0, and High represents the volume percentage of the organ with a score of full. Optionally, the organ score is calculated by scaling the maximum prescription dose of the organ dose in proportion to obtain the radiation dose of each organ; Optionally, the state score also includes one or more of the following: consistency index, uniformity index, drop score; The DRL includes one or more of the following algorithms: TD3, PPO, and SAC algorithms.

5. A tumor field angle optimization method based on deep reinforcement learning, characterized in that: The method comprises: Obtain the target area image and initial action to be planned; An initial state is obtained based on the target area image to be planned and the initial action; The initial state is input into a tumor field angle planning model to output an optimized action, wherein the action is a field layout plan. The tumor field angle planning model is obtained based on the method described in claims 1-4.

6. A tumor field angle adaptive optimization method based on transfer learning, characterized in that: The method comprises: S31: Acquire the target area image and initial action to be planned; S32: obtaining an initial state based on the target area image to be planned and the initial action; S33: performing transfer learning on the tumor radiation field angle planning model to obtain a transfer DRL model, wherein the tumor radiation field angle planning model is obtained based on the method described in claims 1-4; S34: the initial state is input into the migration DRL model, and the action is updated to obtain a third action; S35: obtaining a third state based on the target area image to be planned and the third action, and evaluating the third state to obtain a third state score; S36: obtaining a reward for a third action based on the third state score, where the reward for the third action is the third state score or the difference between the third state score and the initial state score; S37: According to the initial state-third action pair and the target reward, the migration DRL model repeats steps S34-S6 iteratively based on maximizing the target reward until the stopping condition is reached to obtain a tumor field angle planning model, wherein the target reward includes a reward for at least one third action, and after the iteration starts, the initial state-third action pair is replaced by the third state-next third action pair.

7. A method for accelerating optimization of tumor field angles based on reinforcement learning, characterized in that: Use multi-agent parallel sampling to complete the iterative process of the method described in claims 1-6; Optionally, the multi-agents optimize the main thread and the multi-agent threads through reinforcement learning to achieve parallel sampling of the same planning system; Optionally, the main thread sends a sampling field angle request to the Agent of each Agent thread. After detecting the sampling request from the main thread, the Agent initializes the local environment, generates an action according to the current state using the local model, and the local planning system obtains the irradiation field dose distribution according to the generated action. After the reward is calculated based on the irradiation field dose distribution, the sampled field angle and the corresponding reward are stored in the Agent thread sampling data. The main thread iteratively optimizes and updates the parameters based on the sampling data of each Agent thread, and then sends the updated parameters to each Agent thread. After receiving the updated parameters, each Agent thread updates the local model. Optionally, each Agent thread is deployed through a virtual machine.

8. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images