Model training method, treatment plan generation method, device, and medium

Abstract

The present disclosure provides a deep reinforcement learning model training method for generating a treatment plan, a treatment plan generation method, equipment and media, relating to the technical field of medical treatment, especially to the technical field of radiotherapy. The method comprises: obtaining initial dose distribution state data of a target target region; determining target data based on the initial dose distribution state data of the target target region, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer; updating the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the target data, completing the current training of the deep reinforcement learning model; iteratively performing the above training process until the training of the deep reinforcement learning model reaches a preset number of times, and obtaining the trained deep reinforcement learning model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of medical technology, in particular to the technical field of radiotherapy, and specifically to a model training method, a treatment plan generation method, a device and a medium. BACKGROUND

[0002] In the technical field of medical technology, radiotherapy is one of the important means for treating tumors. Before radiotherapy is used to treat a subject, a treatment plan usually needs to be designed in advance.

[0003] At present, the design of a treatment plan usually relies on a physicist (manual) to repeatedly adjust according to his own experience and professional skills, so as to design a more reasonable treatment plan. Therefore, the existing treatment plan method requires higher clinical experience of the physicist, and needs to be continuously adjusted by trial and error, which is time-consuming and laborious, and has low efficiency. SUMMARY

[0004] The present disclosure provides a model training method, a treatment plan generation method, a device and a medium.

[0005] In a first aspect, a deep reinforcement learning model training method for generating a treatment plan is provided, the deep reinforcement learning model is configured to include: a plurality of action prediction network layers and an evaluation prediction network layer, different action prediction network layers are used to output different types of target point parameters included in the treatment plan, and the method includes: obtaining initial dose distribution state data of a target target region; determining target data based on the initial dose distribution state data of the target target region, current policy data of the plurality of action prediction network layers and current policy data of the evaluation prediction network layer; updating the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the target data, completing a current training of the deep reinforcement learning model; iterating the above training process until the training number of the deep reinforcement learning model reaches a preset number, and obtaining a trained deep reinforcement learning model.

[0006] The target data includes: a final dose distribution and a plurality of dose distribution state data of the target target region, a plurality of action sets output by the plurality of action prediction network layers corresponding one-to-one to the plurality of dose distribution state data, a plurality of predicted values output by the evaluation prediction network layer corresponding one-to-one to the plurality of action sets, a plurality of actual reward values corresponding one-to-one to the plurality of action sets, a predicted value and an actual reward value of the target target region; each action set in the plurality of action sets includes a target point parameter combination composed of a plurality of different types of target point parameters.

[0007] In some embodiments, the plurality of action prediction network layers includes at least two of: a target point size prediction network layer, a target point position prediction network layer, and a target point weight prediction network layer.

[0008] In some embodiments, based on the initial dose distribution state data of the target region, the current policy data of each of the plurality of action prediction network layers, and the current policy data of the evaluation prediction network layer, determining the target data comprises: determining, based on the current dose distribution state data of the target region and the current policy data of each of the plurality of action prediction network layers, a set of actions corresponding to the current dose distribution state data output by the plurality of action prediction network layers, and determining, based on the current dose distribution state data of the target region and the current policy data of the evaluation prediction network layer, a predicted value corresponding to the set of actions output by the evaluation prediction network layer; determining a dose distribution of the target region based on the set of actions corresponding to the current dose distribution state data, and determining an actual reward value corresponding to the set of actions based on the dose distribution of the target region; when the dose distribution of the target region does not satisfy the preset prescribed dose and the number of target points in the target region is less than the preset maximum number of target points, updating the current dose distribution state data of the target region based on the dose distribution of the target region; when the dose distribution of the target region satisfies the preset prescribed dose and / or the number of target points in the target region is equal to the preset maximum number of target points, determining a final dose distribution of the target region and a plurality of dose distribution state data of the target region, a plurality of sets of actions output by the plurality of action prediction network layers corresponding one-to-one to the plurality of dose distribution state data, a plurality of predicted values output by the evaluation prediction network layer corresponding one-to-one to the plurality of sets of actions, and a plurality of actual reward values corresponding one-to-one to the plurality of sets of actions; determining an actual reward value and a predicted value of the target region based on the final dose distribution of the target region and the plurality of predicted values corresponding one-to-one to the plurality of sets of actions.

[0009] In some embodiments, when the plurality of action prediction network layers comprises a first action prediction network layer and a second action prediction network layer deployed from top to bottom, determining, based on the current dose distribution state data of the target region and the current policy data of each of the plurality of action prediction network layers, a set of actions corresponding to the current dose distribution state data output by the plurality of action prediction network layers comprises: determining, based on the current dose distribution state data and the current policy data of the first action prediction network layer, a first action; and determining, based on the current dose distribution state data, the first action, and the current policy data of the second action prediction network layer, a second action corresponding to the first action.

[0010] In some embodiments, when the plurality of action prediction network layers comprises a first action prediction network layer, a second action prediction network layer and a third action prediction network layer deployed from top to bottom, the determining, based on the current dose distribution state data of the target volume and the current policy data of each action prediction network layer, the set of actions corresponding to the current dose distribution state data output by the plurality of action prediction network layers comprises: determining a first action based on the current dose distribution state data and the current policy data of the first action prediction network layer; determining a second action corresponding to the first action based on the current dose distribution state data, the first action and the current policy data of the second action prediction network layer; and determining a third action corresponding to the first action based on the current dose distribution state data, the first action, the second action and the current policy data of the third action prediction network layer.

[0011] In some embodiments, the updating, based on the target data, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer comprises: when the final dose distribution of the target volume obtained in the current training satisfies the preset prescribed dose, determining whether an actual reward value of the target volume obtained in the current training is greater than a dynamic reward threshold; the dynamic reward threshold being an actual reward value of the target volume used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer were last updated; when the actual reward value of the target volume obtained in the current training is greater than the dynamic reward threshold, determining a loss value of the target volume corresponding to the current training based on the actual reward value of the target volume obtained in the current training and the predicted value; and when the loss value of the target volume corresponding to the current training is less than a dynamic loss value, updating the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the plurality of dose distribution state data obtained in the current training, the plurality of sets of actions corresponding to the plurality of dose distribution state data, the plurality of actual reward values corresponding to the plurality of sets of actions, the plurality of predicted values corresponding to the plurality of sets of actions and the actual reward value of the target volume; the dynamic loss value being a loss value of the target volume used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer were last updated.

[0012] In some embodiments, based on the plurality of dose distribution state data obtained in the current training, the plurality of action sets corresponding to the plurality of dose distribution state data, the plurality of actual reward values corresponding to the plurality of action sets, the plurality of predicted values corresponding to the plurality of action sets, and the actual reward value of the target region, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated, including: determining the actual cumulative reward values of the plurality of action sets based on the plurality of actual reward values corresponding to the plurality of action sets and the actual reward value of the target region, and updating the current policy data of the plurality of action prediction network layers based on the plurality of dose distribution state data, the plurality of action sets corresponding to the plurality of dose distribution state data, and the actual cumulative reward values of the plurality of action sets; updating the current policy data of the evaluation prediction network layer based on the plurality of dose distribution state data, the plurality of action sets corresponding to the plurality of dose distribution state data, and the plurality of predicted values corresponding to the plurality of action sets.

[0013] In a second aspect, a method for generating a treatment plan is provided, including: obtaining image data and contour data of a target region to be treated; determining dose distribution state data of the target region to be treated based on the image data and the contour data of the target region to be treated; inputting the dose distribution state data of the target region to be treated into a deep reinforcement learning model to obtain a target point parameter combination of the target region to be treated composed of a plurality of different types of target point parameters; the deep reinforcement learning model is trained by the method for training a deep reinforcement learning model for generating a treatment plan according to any one of the embodiments of the first aspect; and generating a treatment plan for the target region to be treated according to the target point parameter combination.

[0014] In a third aspect, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of the first aspect or the second aspect.

[0015] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the storage medium stores a computer program for executing the method of any one of the first aspect or the second aspect.

[0016] The embodiment of the present disclosure provides a deep reinforcement learning model training method for generating a treatment plan, a treatment plan generation method, equipment and a medium, the deep reinforcement learning model is configured to include: a plurality of action prediction network layers and an evaluation prediction network layer, and different action prediction network layers are used to output different types of target parameters included in the treatment plan. In this way, after obtaining the initial dose distribution state data of the target target region, the target data can be determined based on the initial dose distribution state data of the target target region, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer. Then, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer can be updated based on the target data, and the current training of the deep reinforcement learning model is completed. The training process is iterated until the training times of the deep reinforcement learning model reach the preset times, and the trained deep reinforcement learning model is obtained.

[0017] The target data includes: a final dose distribution and a plurality of dose distribution state data of the target target region, a plurality of action sets output by the plurality of action prediction network layers corresponding to the plurality of dose distribution state data one by one, a plurality of predicted values output by the evaluation prediction network layer corresponding to the plurality of action sets one by one, a plurality of actual reward values corresponding to the plurality of action sets one by one, a predicted value and an actual reward value of the target target region; each action set in the plurality of action sets includes a target parameter combination composed of a plurality of different types of target parameters.

[0018] As can be seen from the above, the present disclosure can divide the overall task of determining the action set of the different types of target parameters into a plurality of sub-tasks for determining the action of the different types of target parameters, and each action prediction network layer included in the deep reinforcement learning model completes one sub-task, that is, the complex overall task is decomposed into each sub-task for calculation, so that the determination efficiency of the target parameters can be improved through the plurality of action prediction network layers. Secondly, since the deep reinforcement learning conforms to the characteristics of the gamma knife treatment plan design, the processor in the computer system can realize the repeated trial and error process, so that the target parameters with good effect can be automatically generated through the deep reinforcement learning model, thereby generating the treatment plan with good effect according to the target parameters, reducing the dependence on clinical experience, and improving the effect of the treatment plan without manually setting the target parameters, thereby improving the efficiency of the physicist in designing the treatment plan.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:

[0021] Figure 1 is a schematic diagram of an implementation environment of a method for training a deep reinforcement learning model for generating a treatment plan and a method for generating a treatment plan according to an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of a method for training a deep reinforcement learning model according to an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of another method for training a deep reinforcement learning model according to an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of a method for generating a treatment plan implemented by a processor according to an embodiment of the present disclosure;

[0025] Figure 5 is a block diagram of an electronic device for implementing a method for generating a treatment plan according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0028] Before the present disclosure is described in detail, the application scenarios involved in the embodiments of the present disclosure are described.

[0029] The method for training a deep reinforcement learning model for generating a treatment plan, the method for generating a treatment plan, the device and the medium provided by the embodiments of the present disclosure can be applied in the field of medical technology, and specifically can be applied in the scenario of clinical radiotherapy (referred to as radiotherapy), such as the scenario of gamma knife treatment plan making.

[0030] Before a treatment object is treated by radiotherapy, a treatment plan usually needs to be designed in advance.

[0031] Currently, the design of a treatment plan usually relies on a physicist (manual) to repeatedly adjust according to his own experience and professional skills to design a more reasonable treatment plan. Therefore, the existing treatment plan method requires higher clinical experience of the physicist, and needs to be continuously adjusted by trial and error, which is time-consuming and laborious, and has low efficiency.

[0032] Based on this, the deep reinforcement learning model training method for generating a treatment plan, the method for generating a treatment plan, the equipment and the medium provided by the embodiments of the present disclosure are provided, the deep reinforcement learning model is configured to include: a plurality of action prediction network layers and an evaluation prediction network layer, and different action prediction network layers are used to output different types of target point parameters included in the treatment plan. In this way, after obtaining the initial dose distribution state data of the target target region, the target data can be determined based on the initial dose distribution state data of the target target region, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer. Then, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer can be updated based on the target data, and the current training of the deep reinforcement learning model is completed. Iterate the above training process until the training times of the deep reinforcement learning model reach the preset times, and obtain the trained deep reinforcement learning model.

[0033] Among them, the target data includes: the final dose distribution and a plurality of dose distribution state data of the target target region, a plurality of action sets output by the plurality of action prediction network layers corresponding to the plurality of dose distribution state data one by one, a plurality of prediction values output by the evaluation prediction network layer corresponding to the plurality of action sets one by one, a plurality of actual reward values corresponding to the plurality of action sets one by one, the prediction value and the actual reward value of the target target region; each action set in the plurality of action sets includes a target point parameter combination composed of a plurality of different types of target point parameters.

[0034] As can be seen from the above, the present disclosure can divide the overall task of determining the action set of different types of target point parameters into a plurality of sub-tasks for determining the action of different types of target point parameters, and each action prediction network layer included in the deep reinforcement learning model completes one sub-task, that is, the complex overall task is divided into each sub-task for calculation, therefore, the determination efficiency of the target point parameter can be improved through the plurality of action prediction network layers. Secondly, since the deep reinforcement learning conforms to the characteristics of the Gamma Knife treatment plan design, the processor in the computer system can realize the repeated trial and error process, therefore, the target point parameter with good effect can be automatically generated through the deep reinforcement learning model, so as to generate the treatment plan with good effect according to the target point parameter, reduce the dependence on clinical experience, and improve the efficiency of the physicist in designing the treatment plan without manually setting the target point parameter.

[0035] The method provided in the present application mainly relates to two methods, i.e., a model training method and a treatment plan generation method. The implementation environment of the two methods is introduced as follows.

[0036] Figure 1 (a) in FIG. 1 is a schematic diagram of an implementation environment of a model training method according to an embodiment of the present disclosure. Referring to (a) in FIG. 1, the implementation environment includes an image scanning device a1 and an electronic device a2. Figure 1

[0037] The image scanning device a1 is a device for scanning and displaying a tumor site and surrounding normal tissue of a subject to be treated. In some embodiments, the image scanning device a1 can be at least one of a computed tomography (CT) device, an emission computed tomography (ECT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasonic examination device.

[0038] In the present embodiment, the image scanning device a1 is configured to obtain a medical scan image (e.g., image data of different target regions, etc.) and upload the medical scan image to the electronic device a2, so that the electronic device a2 performs a subsequent model training method based on the medical scan image of the subject to be treated.

[0039] The electronic device a2 is a device for training a deep reinforcement learning model. In some embodiments, the electronic device a2 can be at least one of a smartphone, a smart watch, a desktop computer, a laptop computer, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer.

[0040] Further, in some embodiments, the electronic device a2 can run a computer system including a processor configured to implement the training method of the deep reinforcement learning model.

[0041] In some embodiments, the implementation environment further includes a server a3. In some embodiments, the server a3 is configured to provide a background communication service for the image scanning device a1 and the electronic device a2.

[0042] ​The server a3 can be a stand-alone physical server, or a server cluster or distributed file system composed of multiple physical servers, or at least one of a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network, and big data or artificial intelligence platform, and the like, without limitation of the embodiments of the present disclosure. In some embodiments, the number of the above-mentioned server a3 can be more or less, without limitation of the embodiments of the present disclosure. Of course, the server a3 can also include other functions in order to provide more comprehensive and diversified services.

[0043] Figure 1 (b) in (a) is a schematic diagram of an implementation environment of a treatment plan generation method according to an embodiment of the present disclosure. Referring to (b) in (a), Figure 1 The implementation environment includes an image scanning device b1, an electronic device b2, and a radiotherapy device b3.

[0044] The image scanning device b1 and Figure 1 The device form of the image scanning device a1 in (a) is similar, and the description of the image scanning device a1 can be referred to, which will not be repeated here.

[0045] In the embodiments of the present disclosure, the image scanning device b1 is configured to obtain a medical scanning image of a subject to be treated (for example, image data of a target region to be treated), and upload the medical scanning image of the subject to be treated to the electronic device b2, so that the electronic device b2 performs a subsequent treatment plan generation method based on the medical scanning image of the subject to be treated.

[0046] The electronic device b2 is a device for applying a deep reinforcement learning model to generate a treatment plan.

[0047] The electronic device b2 and Figure 1 The device form of the electronic device a2 in (a) is similar, and the description of the electronic device a2 can be referred to, which will not be repeated here.

[0048] Further, in some embodiments, the electronic device b2 can run a computer system, which includes a processor configured to implement a method of applying a deep reinforcement learning model to generate a treatment plan.

[0049] The radiotherapy device b3 is a device for radiotherapy. In some embodiments, the radiotherapy device b3 can be at least one of a gamma knife, a linear accelerator, a neutron knife, and an X-ray therapy machine.

[0050] In the embodiments of the present disclosure, the radiotherapy device b3 is configured to receive a treatment plan from the electronic device b2, and perform radiotherapy on the subject to be treated according to the treatment plan.

[0051] In some embodiments, the implementation environment further includes a server b4. In some embodiments, the server b4 is configured to provide background communication services for the above-mentioned image scanning device b1, the electronic device b2 and the radiotherapy device b3.

[0052] The server b4 and Figure 1 The device form of the server a3 in (a) in the implementation environment shown in

[0053] The following describes a model training method provided by the embodiments of the present disclosure based on the implementation environment shown in (a) in Figure 1

[0054] The model training method provided by the embodiments of the present disclosure can be applied to the electronic device a2 in (a) in Figure 1 The electronic device a2 can run a computer system including a processor configured to implement the model training method.

[0055] It should be understood that, since the training method of the deep reinforcement learning model involves multiple different processing and computing processes, the processor can also implement the model training method by invoking various appropriate processing threads, which is not limited in the embodiments of the present disclosure. For ease of description, the embodiments of the present disclosure take the processor as an execution subject for illustration.

[0056] Figure 2 is a flowchart of a deep reinforcement learning model training method for generating a treatment plan according to an embodiment of the present disclosure. The deep reinforcement learning model is configured to include a plurality of action prediction network layers and an evaluation prediction network layer, and different action prediction network layers are configured to output different types of target parameters included in the treatment plan. As shown in Figure 2 The method includes:

[0057] S201, obtaining initial dose distribution state data of a target target region.

[0058] The target target region includes a to-be-treated region of a to-be-treated object.

[0059] Optionally, the target target region can also be referred to as a planning target volume (PTV).

[0060] For example, the to-be-treated object can be a phantom, a human body, an animal, etc. The target target region can be a tumor region of the to-be-treated object, etc.

[0061] For example, in combination with Figure 1 ​In (a) of FIG. 1, the image scanning device a1 can collect image data of the subject to be treated, and transmit the image data to the electronic device a2.

[0062] Optionally, after receiving the image data, the electronic device a2 can perform contouring on the image data to obtain contour data of different target regions of the subject to be treated.

[0063] For example, the contour data can be the contour of a tumor region.

[0064] Optionally, the electronic device a2 can also be connected with a third-party software program. The third-party software program is used to perform contouring on the image data to obtain the contour data.

[0065] Optionally, when performing contouring on the image data, a doctor can perform contouring operation on the electronic device a2. The electronic device a2 can obtain the contour data in response to the contouring operation of the doctor.

[0066] Alternatively, the electronic device a2 can also automatically perform contouring on the contour of different target regions of the subject to be treated in the image data through contouring software.

[0067] After determining the image data and the contour data of the target region, the processor in the electronic device a2 can determine initial dose distribution state data of the target region based on the image data and the contour data of the target region.

[0068] The initial dose distribution state data is used to represent the dose distribution state data of the target region without placing a target point.

[0069] The dose distribution state data includes mask data of the target region, dose distribution of the target region, volume of a region with insufficient dose in the target region, and volume of a region with excessive dose in the target region.

[0070] Since the initial dose distribution state data is used to represent the dose distribution state data of the target region without placing a target point, the dose distribution, the volume of the region with insufficient dose, and the volume of the region with excessive dose in the initial dose distribution state data are all 0. In this case, the processor in the electronic device a2 only needs to determine the mask data of the target region based on the image data and the contour data of the target region, and then the initial dose distribution state data of the target region can be determined.

[0071] Specifically, the image data of the target region can include the contour data of the target region and image data of an organ at risk (OAR).

[0072] The organ at risk refers to a normal organ around the target region, i.e., an organ without lesions.

[0073] The processor in the electronic device a2 can process the image data and the contour data of the target target region to generate mask data of the target target region. In the process of generating the mask data, the processor in the electronic device a2 can perform different operations on the region corresponding to the target target region and the region corresponding to the OAR based on the contour data, so as to distinguish the region corresponding to the target target region and the region corresponding to the OAR.

[0074] Meanwhile, the subsequent target point position search space (i.e., the placement position of the target point) can be limited to the region corresponding to the target target region, so as to avoid the target point position falling into the region corresponding to the OAR or other tissues, thereby avoiding the treatment damage to the region corresponding to the OAR or other tissues. Moreover, limiting the subsequent target point position search space to the region corresponding to the target target region can reduce the search space for the specific shape of the target target region, thereby improving the speed of model training and generating a treatment plan.

[0075] For example, the processor in the electronic device a2 can construct a three-dimensional matrix of a uniform size. In this case, the image parameters of the region corresponding to the target target region can be set to 1, the image parameters of the region corresponding to the OAR can be set to -1, and the image parameters of other tissues can be set to 0. In this way, a mask data distinguishing the region corresponding to the target target region and the region corresponding to the OAR can be obtained.

[0076] S202, determining target data based on the initial dose distribution state data of the target target region, the current policy data of the plurality of action prediction network layers, and the current policy data of the evaluation prediction network layer.

[0077] The plurality of action prediction network layers are arranged from top to bottom, i.e., the plurality of action prediction network layers are connected in sequence. For example, as shown in FIG. 8, the plurality of action prediction network layers can include action prediction network layer 1 to action prediction network layer n arranged from top to bottom; n is an integer greater than or equal to 2. Figure 2

[0078] In actual application, since the target point parameters usually include the target point size, the target point weight, and the target point position, the plurality of action prediction network layers usually include at least two of the target point size prediction network layer, the target point position prediction network layer, and the target point weight prediction network layer. Of course, when the number of target point parameters is other, the number of the plurality of action prediction network layers can also be adaptively adjusted according to the number of target point parameters.

[0079] ​In the embodiment of the present application, the target data comprises: a plurality of dose distribution state data of the target target region, a plurality of action sets output by the plurality of action prediction network layers corresponding to the plurality of dose distribution state data one by one, a plurality of predicted values output by the evaluation prediction network layer corresponding to the plurality of action sets one by one, a plurality of actual reward values corresponding to the plurality of action sets one by one, the predicted value and the actual reward value of the target target region; each of the plurality of action sets comprises a target point parameter combination composed of a plurality of different types of target point parameters.

[0080] Since the generation process of the gamma knife treatment plan is a process of sequentially arranging target points, after the arrangement of one target point is completed, the dose distribution state of the target target region will change, so in the generation process of the complete treatment plan of the target target region, the dose distribution state of the target target region will change with the placement of one target point after another.

[0081] Therefore, in the embodiment of the present application, the target data can be obtained when a complete treatment plan of the target target region is completed.

[0082] Specifically, after obtaining the initial dose distribution state data, the processor in the electronic device a2 can input the initial dose distribution state data into each of the plurality of action prediction network layers. At the same time, since the plurality of action prediction network layers are arranged from top to bottom, the processor in the electronic device a2 can also input the output data of the action prediction network layer of the upper layer into the action prediction network layer of the lower layer.

[0083] Different action prediction network layers can output different types of target point parameters selected for the first target point according to their initial strategy data and the initial dose distribution state data of the target target region. In this way, the first action set can be obtained through the output data of the plurality of action prediction network layers.

[0084] Since the deep reinforcement learning model also includes the evaluation prediction network layer, after obtaining the initial dose distribution state data, the processor in the electronic device a2 can also input the initial dose distribution state data into the evaluation prediction network layer at the same time, so that the evaluation prediction network layer can also determine the predicted value corresponding to the first action set according to its initial strategy data and the initial dose distribution state data of the target target region while determining the first action set.

[0085] After the first target point is placed, the dose distribution state of the target target region changes, at this time, the processor in the electronic device a2 can also determine the dose distribution of the target target region based on the first action set, and determine the actual reward value corresponding to the first action set based on the dose distribution of the target target region.

[0086] Then, the processor in the electronic device a2 can update the dose distribution state data of the target target region based on the dose distribution of the target target region, and repeat the above-mentioned target point placement process according to the updated dose distribution state data of the target target region until the generation of the complete treatment plan of the target target region is completed, obtaining the final dose distribution of the target target region, the plurality of dose distribution state data of the target target region, the plurality of action sets output by the plurality of action prediction network layers corresponding to the plurality of dose distribution state data one by one, the plurality of prediction values output by the evaluation prediction network layer corresponding to the plurality of action sets one by one, the plurality of actual reward values corresponding to the plurality of action sets one by one, the actual reward value and the prediction value of the target target region. In this way, the processor in the electronic device a2 can obtain the target data.

[0087] S203, update the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the target data, and complete the current training of the deep reinforcement learning model.

[0088] Specifically, after obtaining the target data, it is indicated that the treatment plan of the target target region has been completed. In this case, the processor in the electronic device a2 can update the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the target data.

[0089] It should be noted that the training process of the deep reinforcement learning model is equivalent to a trial-and-error process of treatment plan generation, and therefore, there may be a case where the obtained target data is discarded because it does not meet the update condition. Therefore, in the embodiment of the present application, completing the training of the deep reinforcement learning model does not mean that the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer will be updated.

[0090] S204, iterate the above-mentioned training process until the training number of the deep reinforcement learning model reaches the preset number, and obtain the trained deep reinforcement learning model.

[0091] Specifically, iterate the above-mentioned training process of the deep reinforcement learning model (i.e., S201 to S203) until the training number of the deep reinforcement learning model reaches the preset number, and obtain the trained deep reinforcement learning model.

[0092] Optionally, the electronic device can further include an experience memory unit (buffer_memory). The experience memory unit is used to store the final dose distribution of the target region in each update stage, the plurality of dose distribution state data, the plurality of action sets output by the plurality of action prediction network layers corresponding to the plurality of dose distribution state data, the plurality of predicted values output by the evaluation prediction network layer corresponding to the plurality of action sets, the plurality of actual reward values corresponding to the plurality of action sets, the predicted value and the actual reward value of the target region (i.e., target data) in the current policy data update process. Subsequently, by using the corresponding relationship stored in the experience memory unit, various required data determined in each training can be found, and model training can be performed.

[0093] The model training method provided by the present application will be described in detail below. Figure 3 The model training method provided by the present application will be described in detail below. Figure 3 An embodiment of the model training method provided by the present application is shown in FIG. 1. As shown in FIG. 1, the model training method specifically includes the following steps. Figure 3 An embodiment of the model training method provided by the present application is shown in FIG. 1. As shown in FIG. 1, the model training method specifically includes the following steps.

[0094] S1, obtaining initial dose distribution state data of a target region.

[0095] The specific implementation process of this step can refer to the description of S201, which will not be described here.

[0096] S2, determining, based on the current dose distribution state data of the target region and the current policy data of each action prediction network layer, an action set corresponding to the current dose distribution state data output by the plurality of action prediction network layers, and determining, based on the current dose distribution state data of the target region and the current policy data of the evaluation prediction network layer, a predicted value corresponding to the action set output by the evaluation prediction network layer.

[0097] Each action prediction network layer and the evaluation prediction network layer can be pre-provisioned with an initial policy data.

[0098] For the action prediction network layer, the policy data refers to the selection strategy of the target point parameters when placing the target point in the target region (which can also be referred to as the action selection strategy), for example, the strategy of selecting the size of the target point when placing the target point.

[0099] For the evaluation prediction network layer, the policy data is the value prediction of the action selection under a certain dose distribution state of the target region, and the policy data is the strategy for determining the predicted value corresponding to the action selection result.

[0100] Since the generation process of the Gamma Knife treatment plan is a process of sequentially arranging target points, the dose distribution state of the target target region will change after the arrangement of one target point is completed. Therefore, in the generation process of the complete treatment plan of the target target region, the current dose distribution state of the target target region will change with the placement of one target point after another.

[0101] Therefore, after obtaining the current dose distribution state data of the target target region (for the first target point, the current dose distribution state data of the target target region is the initial dose distribution state data of the target target region), the processor in the electronic device can input the current dose distribution state data of the target target region into each action prediction network layer in the plurality of action prediction network layers respectively. The first action prediction network layer can determine a first action according to its initial strategy data and the current dose distribution state data of the target target region, and the first action can represent a first type of target point parameter selected when placing the current target point, such as a target point size selected when placing the current target point.

[0102] Then, the processor in the electronic device can input the output data (i.e., the first action) of the first action prediction network layer into the second action prediction network layer. The second action prediction network layer can determine a second action according to its initial strategy data, the current dose distribution state data of the target target region, and the output data (i.e., the first action) of the first action prediction network layer, and the second action can represent a second type of target point parameter selected when placing the current target point, such as a target point position selected when placing the current target point.

[0103] By analogy, the processor in the electronic device can obtain a set of actions corresponding to the current dose distribution state data, which is composed of a plurality of actions, and the set of actions includes a plurality of different types of target point parameters selected for the current target point when placing the current target point, such as a target point size, a target point position, and a target point weight selected for the current target point when placing the current target point.

[0104] Since the deep reinforcement learning model further includes an evaluation prediction network layer, after obtaining the current dose distribution state data of the target target region, the processor in the electronic device can input the initial dose distribution state data into the evaluation prediction network layer while determining the set of actions corresponding to the current dose distribution state data output by the plurality of action prediction network layers. The evaluation prediction network layer can determine a predicted value corresponding to the set of actions according to its initial strategy data and the current dose distribution state data of the target target region.

[0105] S3, determining the dose distribution of the target target region based on the set of actions corresponding to the current dose distribution state data, and determining an actual reward value corresponding to the set of actions based on the dose distribution of the target target region.

[0106] Specifically, after determining the action set corresponding to the current dose distribution state data, the processor in the electronic device can place a target point in the target volume according to the action set. In this way, after placing the target point, the dose distribution of the target volume will change. In this case, the processor in the electronic device can determine the dose distribution of the changed target volume.

[0107] In some embodiments, in the case that the action set only includes two types of target point parameters, for example, the action set only includes target point size and target point weight, in order to place the current target point, the target point position also needs to be determined, at this time, the processor in the electronic device can call a shape matching algorithm to determine the target point position.

[0108] After obtaining all parameters of the target point, i.e., target point size, target point position, and target point weight, the processor in the electronic device determines the dose distribution of the target volume according to these target point parameters.

[0109] It can be understood that when the multiple action prediction network layers include two layers, the deep reinforcement learning model can only generate two types of target point parameters, and when the types of target point parameters exceed two, the determination of the remaining types of target point parameters can be performed by the processor in the electronic device calling a corresponding algorithm to determine the remaining types of target point parameters.

[0110] Secondly, in order to obtain the deep reinforcement learning model through subsequent training, the processor in the electronic device can also determine an actual reward value corresponding to the action set based on the dose distribution of the target volume.

[0111] Specifically, since the dose distribution of the target volume is determined after the target point is placed, the dose distribution of the target volume can truly reflect the contribution of an action set to the dose distribution of the target volume. In this way, after obtaining an action set, the processor in the electronic device can determine an actual reward value corresponding to the action set based on the dose distribution of the target volume.

[0112] For example, the actual reward value corresponding to the action set includes a positive reward and a negative reward. According to the growth values of the dose coverage rate and the conformance degree of the target volume before and after the target point placement corresponding to the action set, the corresponding weight values are multiplied as the positive reward; and according to the growth value of the dose overflow rate of the target volume before and after the target point placement corresponding to the action set, the corresponding weight value is multiplied as the negative reward.

[0113] S4, when the dose distribution of the target volume does not satisfy the preset prescription dose, and the number of target points in the target volume is less than the preset maximum number of target points, updating the current dose distribution state data of the target volume based on the dose distribution of the target volume.

[0114] Specifically, after determining the dose distribution of the target target region, it can be judged whether the dose distribution of the target target region meets the preset prescribed dose, and / or whether the number of target points in the target target region is less than the preset maximum number of target points. When the dose distribution of the target target region does not meet the preset prescribed dose, and the number of target points in the target target region is less than the preset maximum number of target points, it indicates that the treatment plan of the target target region has not been completed. In this case, the processor of the electronic device can update the current dose distribution state data of the target target region based on the dose distribution of the target target region, that is, update the mask data of the target target region, the dose distribution of the target target region, the volume of the dose distribution insufficient region in the target target region, and the volume of the dose distribution overflow region in the target target region.

[0115] Optionally, when the processor of the electronic device updates the current dose distribution state data of the target target region based on the dose distribution of the target target region, the dose distribution can be feature extracted to obtain the updated current dose distribution state data.

[0116] Then, the processor in the electronic device can repeatedly execute the above S2-S4 according to the updated dose distribution state data to select the action set corresponding to the subsequent target point and determine the predicted reward value and the actual reward value corresponding to the action set.

[0117] S5, when the dose distribution of the target target region meets the preset prescribed dose, and / or the number of target points in the target target region is equal to the preset maximum number of target points, determining the final dose distribution of the target target region, the plurality of dose distribution state data of the target target region, the plurality of action sets output by the plurality of action prediction network layers corresponding to the plurality of dose distribution state data one by one, the plurality of predicted values output by the evaluation prediction network layer corresponding to the plurality of action sets one by one, and the plurality of actual reward values corresponding to the plurality of action sets one by one.

[0118] Specifically, when the dose distribution of the target target region meets the preset prescribed dose, and / or the number of target points in the target target region is equal to the preset maximum number of target points, it indicates that the target target region has completed the formulation of the treatment plan. In this case, the target target region has placed a plurality of target points forming the treatment plan. Correspondingly, the final dose distribution of the target target region, the plurality of dose distribution state data of the target target region, the plurality of action sets output by the plurality of action prediction network layers corresponding to the plurality of dose distribution state data one by one, the plurality of predicted values output by the evaluation prediction network layer corresponding to the plurality of action sets one by one, and the plurality of actual reward values corresponding to the plurality of action sets one by one can be obtained.

[0119] S6, determining the actual reward value and the predicted value of the target target region based on the final dose distribution of the target target region and the plurality of predicted values corresponding to the plurality of action sets.

[0120] Specifically, when the dose distribution in the target area meets the preset prescription dose, and / or the number of target points in the target area is equal to the preset maximum number of target points, the final dose distribution of the target area and multiple predicted values ​​corresponding one-to-one with multiple action sets can be obtained. In this case, the processor in the electronic device can determine the actual reward value of the target area based on the final dose distribution, and determine the predicted value of the target area based on the multiple predicted values ​​corresponding one-to-one with multiple action sets.

[0121] Optionally, after determining the final dose distribution of the target area, the dose coverage and conformity of the target area can be determined based on the final dose distribution of the target area, and the actual reward value of the target area can be determined based on the dose coverage and conformity of the target area.

[0122] Optionally, after determining multiple predicted values ​​corresponding to multiple action sets, the multiple predicted values ​​corresponding to multiple action sets can be summed to obtain the predicted value of the target area.

[0123] In one feasible approach, the method described above, which updates the current policy data of multiple action prediction network layers and the current policy data of the evaluation prediction network layer based on target data to complete the current training of the multiple action prediction network layers and the evaluation prediction network layer, specifically includes:

[0124] S7. When the final dose distribution of the target area obtained in the current training meets the preset prescription dose, determine whether the actual reward value of the target area obtained in the current training is greater than the dynamic reward threshold.

[0125] The final dose distribution of the target area is the dose distribution corresponding to the treatment plan of the target area after the treatment plan of the target area, which consists of multiple action sets, is generated.

[0126] It should be noted that since the treatment plan for the target area is completed when the dose distribution in the target area meets the preset prescription dose, and / or the number of targets in the target area is equal to the preset maximum number of targets, the processor in the electronic device does not need to determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose before proceeding to subsequent judgment steps. Of course, the processor in the electronic device can also re-determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose.

[0127] When the treatment plan of the target region is obtained when the number of target points in the target region is equal to the preset maximum number of target points, the processor in the electronic device needs to determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose, since the treatment plan only meets the preset maximum number of target points, but does not necessarily meet the preset prescription dose.

[0128] If the dose distribution corresponding to the treatment plan does not meet the preset prescription dose, it means that the treatment plan is not a good treatment plan. Therefore, the processor in the electronic device can discard the target data corresponding to the treatment plan, i.e., there is no need to update the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer according to the target data corresponding to the treatment plan.

[0129] Correspondingly, if the dose distribution corresponding to the treatment plan meets the preset prescription dose, it means that the treatment plan can meet the requirement of the preset prescription dose. In this case, the processor in the electronic device can determine whether the actual reward value of the target region is greater than the dynamic reward threshold.

[0130] The dynamic reward threshold is the actual reward value of the target region used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated last time.

[0131] By determining whether the actual reward value of the target region obtained in the current training is greater than the dynamic reward threshold, it can be determined whether the quality of the treatment plan is better than the quality of the treatment plan used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated last time.

[0132] When the actual reward value of the target region obtained in the current training is less than or equal to the dynamic reward threshold, it means that the quality of the treatment plan is worse than or equal to the quality of the treatment plan used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated last time. Therefore, the processor in the electronic device can discard the target data corresponding to the treatment plan, i.e., there is no need to update the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer according to the target data corresponding to the treatment plan.

[0133] Correspondingly, when the actual reward value of the target region obtained in the current training is greater than the dynamic reward threshold, it means that the quality of the treatment plan is better than the quality of the treatment plan used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated last time, i.e., the quality of the treatment plan is better. In this case, the electronic device can perform S8.

[0134] S8, when the actual reward value of the target target region obtained in the current training is greater than the dynamic reward threshold, determining a loss value of the target target region corresponding to the current training based on the actual reward value and the predicted value of the target target region obtained in the current training.

[0135] Specifically, when the actual reward value of the target target region obtained in the current training is greater than the dynamic reward threshold, it indicates that the quality of the treatment plan is better. In this case, the processor in the electronic device can determine the loss value of the target target region corresponding to the current training based on the actual reward value and the predicted value of the target target region obtained in the current training.

[0136] Optionally, the processor in the electronic device can use a preset loss function to determine the loss value of the target target region corresponding to the current training according to the predicted value and the actual reward value of the target target region obtained in the current training.

[0137] Optionally, the preset loss function can be a relative advantage parameter, or other general loss function, which is not limited in the present application.

[0138] Then, the processor in the electronic device can determine whether the loss value of the target target region corresponding to the current training is less than the dynamic loss value.

[0139] The dynamic loss value is the loss value of the target target region used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated last time.

[0140] If the loss value of the target target region corresponding to the current training is greater than or equal to the loss value of the target target region used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated last time, it indicates that the deep reinforcement learning model is inaccurate in estimating the quality of the treatment plan (i.e., predicting the contribution value of the selection of each action set to the treatment plan). Therefore, the processor in the electronic device can discard the target data corresponding to the treatment plan, i.e., without updating the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer according to the target data corresponding to the treatment plan.

[0141] If the loss value of the target target region corresponding to the current training is less than the loss value of the target target region used when the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer are updated last time, it indicates that the deep reinforcement learning model is accurate in estimating the quality of the treatment plan (i.e., predicting the contribution value of the selection of each action set to the treatment plan). In this case, the processor in the electronic device can perform S9.

[0142] S9, when the loss value of the target target region corresponding to the current training is less than the dynamic loss value, based on the plurality of dose distribution state data obtained by the current training, the plurality of action sets corresponding to the plurality of dose distribution state data, the plurality of actual reward values corresponding to the plurality of action sets, the plurality of predicted values corresponding to the plurality of action sets, and the actual reward value of the target target region, update the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer.

[0143] In an implementable manner, the method of updating the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the plurality of dose distribution state data obtained by the current training, the plurality of action sets corresponding to the plurality of dose distribution state data, the plurality of actual reward values corresponding to the plurality of action sets, the plurality of predicted values corresponding to the plurality of action sets, and the actual reward value of the target target region, specifically includes: determining the actual cumulative reward values of the plurality of action sets based on the plurality of actual reward values corresponding to the plurality of action sets and the actual reward value of the target target region, and updating the current policy data of the plurality of action prediction network layers based on the plurality of dose distribution state data, the plurality of action sets corresponding to the plurality of dose distribution state data, and the actual cumulative reward values of the plurality of action sets, and updating the current policy data of the evaluation prediction network layer based on the plurality of dose distribution state data, the plurality of action sets corresponding to the plurality of dose distribution state data, and the plurality of predicted values corresponding to the plurality of action sets.

[0144] Specifically, since the quality of the treatment plan of the target target region is the result of the joint action of the plurality of action sets, under a certain dose distribution state of the target target region, the actual reward value of the selected action set is high, which does not mean that the quality of the treatment plan of the entire target target region will be good. If the quality of the treatment plan of the target target region is also good, the reference value of the selection of the action set will be high, and if the quality of the treatment plan of the target target region is not good, the reference value of the selection of the action set will be small. Therefore, after obtaining the plurality of action sets corresponding to the plurality of dose distribution state data, the actual cumulative reward values of the plurality of action sets can also be determined according to the plurality of actual reward values corresponding to the plurality of action sets and the actual reward value of the target target region.

[0145] The actual cumulative reward value can include an actual reward value of each action set and a delayed reward value after completion of the entire treatment plan. The delayed reward value is determined based on the actual reward value of the target target region. That is, after completion of the treatment plan of the target target region, the actual reward value of the target target region is determined according to the dose distribution of the treatment plan, and the actual reward value is distributed to each action set constituting the treatment plan according to a weight to form a delayed reward value of the action set; the actual reward value and the delayed reward value of each action set are accumulated to obtain the actual cumulative reward value of each action set.

[0146] When the actual cumulative reward value is high, it is said that the reference value of the action set selected in the dose distribution state is high. In this case, the current policy data of the multiple action prediction network layers can be updated based on the multiple dose distribution state data, the multiple action sets corresponding to the multiple dose distribution state data, and the actual cumulative reward values of the multiple action sets.

[0147] Meanwhile, since the loss value of the target target region is obtained based on the predicted value of the target target region, and the predicted value of the target target region is the accumulation of the predicted values of the multiple action sets, when the loss value of the target target region is less than the dynamic loss value, it indicates that the predicted values of the multiple action sets have high reference values. In this case, the current policy data of the evaluation prediction network layer can be updated based on the multiple dose distribution state data, the multiple action sets corresponding to the multiple dose distribution state data, and the multiple predicted values corresponding to the multiple action sets.

[0148] S10, iteratively performing the training process until the training of the deep reinforcement learning model reaches a preset number of times, and obtaining a trained deep reinforcement learning model.

[0149] The specific implementation process of this step can refer to the description of S204, which will not be repeated here.

[0150] In some embodiments, the multiple action prediction network layers include at least two of a target point size prediction network layer, a target point position prediction network layer, and a target point weight prediction network layer. When the multiple action prediction network layers include a first action prediction network layer and a second action prediction network layer deployed from top to bottom, in S1, based on the current dose distribution state data of the target target region and the current policy data of each action prediction network layer, the action set corresponding to the current dose distribution state data output by the multiple action prediction network layers is determined, including:

[0151] determining a first action based on the current dose distribution state data and the current policy data of the first action prediction network layer; and determining a second action corresponding to the first action based on the current dose distribution state data, the first action, and the current policy data of the second action prediction network layer.

[0152] Optionally, the first action prediction network layer and the second action prediction network layer can be any two different action prediction network layers for selecting a target point size, for selecting a target point position, or for selecting a target point weight.

[0153] Correspondingly, the first action and the second action can be any two different actions of selecting a target point size, selecting a target point position, or selecting a target point weight when placing a target point under the current dose distribution state data.

[0154] It should be noted that the first action prediction network layer corresponds to the first action, and the second action prediction network layer corresponds to the second action. For example, when the first action prediction network layer is an action prediction network layer for selecting a target point size, the first action is an action of selecting a target point size.

[0155] For example, assuming that the first action prediction network layer is an action prediction network layer for selecting a target point size, and the second action prediction network layer is an action prediction network layer for selecting a target point weight, then the first action is an action of selecting a target point size, and the second action is an action of selecting a target point weight.

[0156] When placing a target point under the current dose distribution state data, the processor in the electronic device can input the current dose distribution state data into the first action prediction network layer for selecting a target point size. The first action prediction network layer can determine, according to its current policy data, an action of selecting a target point size when placing a target point under the current dose distribution state data, i.e., a target point size when placing a target point under the current dose distribution state data.

[0157] In an implementable manner, after determining the target point size when placing a target point under the current dose distribution state data, the processor in the electronic device can further call a shape matching algorithm to determine a target point position of a target point corresponding to the target point size in the target volume according to the target point size when placing a target point under the current dose distribution state data.

[0158] Then, the processor in the electronic device can input the current dose distribution state data and the action of selecting a target point size when placing a target point under the current dose distribution state data into the second action prediction network layer for selecting a target point weight. The second action prediction network layer can determine, according to its current policy data, an action of selecting a target point weight when placing a target point under the current dose distribution state data, i.e., a target point weight when placing a target point under the current dose distribution state data.

[0159] In this way, the processor in the electronic device can obtain an action set composed of the first action and the second action.

[0160] It can be understood that when the plurality of action prediction network layers include two layers, the deep reinforcement learning model can only generate two types of target parameters, and when the types of target parameters exceed two, the determination of the remaining types of target parameters can be performed by calling corresponding algorithms by the processor in the electronic device.

[0161] In some embodiments, when the plurality of action prediction network layers include a first action prediction network layer, a second action prediction network layer and a third action prediction network layer deployed from top to bottom, in S1, the determination of the set of actions corresponding to the current dose distribution state data output by the plurality of action prediction network layers based on the current dose distribution state data of the target volume and the current policy data of each action prediction network layer, includes:

[0162] determining a first action based on the current dose distribution state data and the current policy data of the first action prediction network layer; determining a second action corresponding to the first action based on the current dose distribution state data, the first action and the current policy data of the second action prediction network layer; and determining a third action corresponding to the first action based on the current dose distribution state data, the first action, the second action and the current policy data of the third action prediction network layer.

[0163] Optionally, the first action prediction network layer, the second action prediction network layer and the third action prediction network layer can be different action prediction network layers for selecting a target size, a target position and a target weight.

[0164] Correspondingly, the first action, the second action and the third action can be different actions of selecting a target size, a target position and a target weight when placing a target under the current dose distribution state data.

[0165] It should be noted that the first action prediction network layer corresponds to the first action, the second action prediction network layer corresponds to the second action, and the third action prediction network layer corresponds to the third action. For example, when the first action prediction network layer is an action prediction network layer for selecting a target size, the first action is an action of selecting a target size.

[0166] For example, assuming that the first action prediction network layer is an action prediction network layer for selecting a target size, the second action prediction network layer is an action prediction network layer for selecting a target position, and the third action prediction network layer is an action prediction network layer for selecting a target weight, then the first action is an action of selecting a target size, the second action is an action of selecting a target position, and the third action is an action of selecting a target weight.

[0167] When placing the target point under the current dose distribution state data, the processor in the electronic device can input the current dose distribution state data into a first action prediction network layer for selecting a target point size. The first action prediction network layer can determine, according to its current policy data, an action of selecting a target point size when placing the target point under the current dose distribution state data, that is, a target point size when placing the target point under the current dose distribution state data.

[0168] Then, the processor in the electronic device can input the current dose distribution state data, the action of selecting a target point size when placing the target point under the current dose distribution state data into a second action prediction network layer for selecting a target point position. The second action prediction network layer can determine, according to its current policy data, an action of selecting a target point position when placing the target point under the current dose distribution state data, that is, a target point position when placing the target point under the current dose distribution state data.

[0169] Then, the processor in the electronic device can input the current dose distribution state data, the action of selecting a target point size when placing the target point under the current dose distribution state data, and the action of selecting a target point position when placing the target point under the current dose distribution state data into a third action prediction network layer for selecting a target point weight. The third action prediction network layer can determine, according to its current policy data, an action of selecting a target point weight when placing the target point under the current dose distribution state data, that is, a target point weight when placing the target point under the current dose distribution state data.

[0170] In this way, the processor in the electronic device can obtain an action set composed of the first action, the second action and the third action.

[0171] The function of the electronic device to implement the treatment plan will be further described below.

[0172] Figure 4 is a flowchart of a processor implementing a method for generating a treatment plan according to an embodiment of the present disclosure. As shown in Figure 4 , the method comprises:

[0173] S401, obtaining image data and contour data of a target region to be treated.

[0174] The specific process of obtaining the image data and the contour data of the target region to be treated can refer to the specific process of obtaining the image data and the contour data of different target regions in S201, which will not be described here.

[0175] For example, as shown in Figure 4 , the image data of the target region to be treated can be a head CT image of a subject to be treated, and the contour data can be contour data of the target region to be treated in the head of the subject to be treated.

[0176] S402, determine the dose distribution state data of the target region to be treated based on the image data and the contour data of the target region to be treated.

[0177] The specific process of determining the dose distribution state data of the target region to be treated can refer to the specific process of determining the initial dose distribution state data of the target region to be treated in S201, which will not be described here.

[0178] S403, input the dose distribution state data of the target region to be treated into the deep reinforcement learning model to obtain the target point parameter combination of the target region to be treated composed of multiple different types of target point parameters.

[0179] The deep reinforcement learning model is obtained by training the model training method shown in Figure 2 or Figure 3 .

[0180] For example, as shown in Figure 4 , the target point parameter combination of the target region to be treated composed of multiple different types of target point parameters output by the deep reinforcement learning model can include a target point parameter combination composed of at least two of the target point size, the target point position and the target point weight.

[0181] S404, generate a treatment plan for the target region to be treated according to the target point parameter combination.

[0182] It should be noted that when the above target point parameter combination includes two types of target point parameters (for example: target point size and target point weight), the processor can determine the remaining target point parameters for generating the treatment plan by calling the relevant algorithm (for example: calling a position matching algorithm to determine the target point position according to the target point parameter combination composed of the target point size and the target point weight).

[0183] Now, taking the example of the target point parameter combination including the target point size, the target point position and the target point weight, a process description of a processor implementation of a treatment plan generation method is performed. It can be understood that the target point parameter combination can also be at least two of the target point size, the target point position and the target point weight.

[0184] First, the processor obtains the image data and the contour data of the target region to be treated, and then the processor determines the dose distribution state data of the target region to be treated according to the obtained image data and the contour data of the target region to be treated, and inputs the dose distribution state data of the target region to be treated into the deep reinforcement learning model trained by the model training method shown in Figure 2 or Figure 3 , to obtain the target point size, the target point position and the target point weight of the first target point output by the deep reinforcement learning model. That is, the relevant parameter determination of the first target point in the treatment plan is completed.

[0185] Then, the processor calculates the dose distribution of the target region to be treated according to the related parameters of the first target point, determines the current dose distribution state data of the target region to be treated according to the dose distribution of the target region to be treated, and inputs the current dose distribution state data of the target region to be treated into the deep reinforcement learning model to obtain the target point size, target point position and target point weight of the second target point output by the deep reinforcement learning model.

[0186] The determination process of the target point size, target point position and target point weight is repeatedly iterated to determine the related parameters of the remaining target points until the dose distribution of the target region to be treated meets the prescription dose, and / or the number of target points in the target region to be treated is equal to the preset maximum number of target points, the determination of the target point parameter combination of the target region to be treated is completed, and a treatment plan is generated according to the target point parameter combination. The target point parameter combination of the target region to be treated includes the related parameters of multiple target points, that is, it includes multiple sets of target point size (S), target point position (P) and target point weight (W) composed of { (S1, P1, W1), (S2, P2, W2), … (S n , n , n )}.

[0187] It can be understood that the target point weight can also be obtained without the deep reinforcement learning model, and can be set as 1 by default.

[0188] The treatment plan can be displayed in the form of a display of an image corresponding to the image data of the target object. In the image corresponding to the image data of the target object, the target point parameter combination of the target region to be treated can be clearly marked.

[0189] Optionally, after determining the treatment plan of the target region to be treated, a physicist (manual) can also adjust each target point parameter in the target point parameter combination of the treatment plan of the target region to be treated according to experience, so as to determine the treatment plan of the target region to be treated according to the adjusted target point parameter combination.

[0190] The above mainly introduces the scheme provided by the embodiments of the present application from the perspective of a computer system. In order to realize the above functions, it contains the hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed in the present text, the present application can be realized in the form of hardware or the combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0191] The embodiments of the present application can divide the functional modules according to the above computer system, for example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The above integrated module can be realized in the form of hardware or in the form of software functional module. Optionally, the division of the modules in the embodiments of the present application is illustrative, and only a logical function division, and another division mode can be used in actual implementation.

[0192] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for generating a treatment plan provided by the present disclosure.

[0193] According to the embodiments of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable an electronic device to perform the method for generating a treatment plan provided by the present disclosure.

[0194] According to the embodiments of the present disclosure, the present disclosure also provides a computer program product comprising a computer program, which, when executed by a processor, implements the deep reinforcement learning model training method for generating a treatment plan, the method for generating a treatment plan provided by the present disclosure.

[0195] In some embodiments, the electronic device can be the electronic device a2 or b2 shown in the above Figure 1 . Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device 500 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device 500 can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0196] As Figure 5As shown, the electronic device 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0197] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including an input unit 506 such as a keyboard, a mouse, and the like, an output unit 507 such as various types of displays, a speaker, and the like, a storage unit 508 such as a magnetic disk, an optical disk, and the like, and a communication unit 509 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0198] The computing unit 501 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 501 performs various methods and processes described above, such as the method of generating a treatment plan. For example, in some embodiments, the method of generating a treatment plan can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method of generating a treatment plan described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the method of generating a treatment plan by any other appropriate means, such as by means of firmware.

[0199] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Parts (ASSP), a System on Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0200] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general or special purpose computer, such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow charts and / or block diagrams to be implemented. The program code can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0201] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0202] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a Cathode Ray Tube (CRT) or a Liquid Crystal Display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0203] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0204] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0205] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the flow. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0206] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a deep reinforcement learning model for generating treatment plans, the deep reinforcement learning model being configured to include: The method comprises multiple action prediction network layers and an evaluation prediction network layer, wherein different action prediction network layers are used to output different types of target parameters included in the treatment plan, characterized in that the method includes: Acquire initial dose distribution data of the target area; Based on the initial dose distribution state data of the target area, the current strategy data of the plurality of action prediction network layers, and the current strategy data of the evaluation prediction network layer, target data is determined, including: based on the current dose distribution state data of the target area and the current strategy data of each of the action prediction network layers, determining the action set output by the plurality of action prediction network layers corresponding to the current dose distribution state data; and based on the current dose distribution state data of the target area and the current strategy data of the evaluation prediction network layer, determining the prediction value output by the evaluation prediction network layer corresponding to the action set. Based on the action set corresponding to the current dose distribution status data, the dose distribution of the target area is determined, and based on the dose distribution of the target area, the actual reward value corresponding to the action set is determined; When the dose distribution in the target area does not meet the preset prescription dose, and the number of target points in the target area is less than the preset maximum number of target points, the current dose distribution status data of the target area is updated based on the dose distribution of the target area. When the dose distribution in the target area meets the preset prescription dose, and / or the number of target points in the target area is equal to the preset maximum number of target points, the final dose distribution of the target area and multiple dose distribution state data of the target area, multiple action sets output by the multiple action prediction network layers corresponding one-to-one with the multiple dose distribution state data, multiple predicted values ​​output by the evaluation prediction network layer corresponding one-to-one with the multiple action sets, and multiple actual reward values ​​corresponding one-to-one with the multiple action sets are determined. Based on the final dose distribution of the target area and the multiple predicted values ​​corresponding to the multiple action sets, the actual reward value and predicted value of the target area are determined. Based on the target data, update the current policy data of the multiple action prediction network layers and the current policy data of the evaluation prediction network layer to complete the current training of the deep reinforcement learning model. The above training process is iterated until the deep reinforcement learning model is trained a preset number of times, and then the trained deep reinforcement learning model is obtained. The target data includes: the final dose distribution of the target area and multiple dose distribution state data, multiple action sets output by the multiple action prediction network layers corresponding one-to-one with the multiple dose distribution state data, multiple predicted values ​​output by the evaluation prediction network layer corresponding one-to-one with the multiple action sets, multiple actual reward values ​​corresponding one-to-one with the multiple action sets, and the predicted value and actual reward value of the target area; each action set in the multiple action sets includes a target parameter combination composed of multiple different types of target parameters.

2. The method according to claim 1, characterized in that, The plurality of action prediction network layers include at least two of the following: target size prediction network layer, target position prediction network layer, and target weight prediction network layer.

3. The method according to claim 1, characterized in that, When the plurality of action prediction network layers includes a first action prediction network layer and a second action prediction network layer deployed from top to bottom, the step of determining the action set output by the plurality of action prediction network layers corresponding to the current dose distribution state data based on the current dose distribution state data of the target region and the current strategy data of each action prediction network layer includes: Based on the current dose distribution status data and the current strategy data of the first action prediction network layer, the first action is determined; Based on the current dose distribution status data, the current strategy data of the first action and the second action prediction network layer, the second action corresponding to the first action is determined.

4. The method according to claim 1, characterized in that, When the plurality of action prediction network layers includes a first action prediction network layer, a second action prediction network layer, and a third action prediction network layer deployed from top to bottom, the step of determining the action set output by the plurality of action prediction network layers corresponding to the current dose distribution state data based on the current dose distribution state data of the target region and the current strategy data of each action prediction network layer includes: Based on the current dose distribution status data and the current strategy data of the first action prediction network layer, the first action is determined; Based on the current dose distribution status data and the current strategy data of the first action and the second action prediction network layer, determine the second action corresponding to the first action; Based on the current dose distribution status data, the first action, the second action, and the current strategy data of the third action prediction network layer, the third action corresponding to the first action is determined.

5. The method according to claim 1, characterized in that, The step of updating the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the target data includes: When the final dose distribution of the target area obtained in the current training satisfies the preset prescription dose, it is determined whether the actual reward value of the target area obtained in the current training is greater than the dynamic reward threshold; the dynamic reward threshold is the actual reward value of the target area used when the current policy data of the multiple action prediction network layers and the current policy data of the evaluation prediction network layer were last updated. When the actual reward value of the target region obtained in the current training is greater than the dynamic reward threshold, the loss value of the target region corresponding to the current training is determined based on the actual reward value and the predicted value of the target region obtained in the current training. When the loss value of the target region corresponding to the current training iteration is less than the dynamic loss value, the current policy data of the multiple action prediction network layers and the current policy data of the evaluation prediction network layer are updated based on the multiple dose distribution state data obtained in the current training iteration, the multiple action sets corresponding to the multiple dose distribution state data, the multiple actual reward values ​​corresponding to the multiple action sets, the multiple predicted values ​​corresponding to the multiple action sets, and the actual reward value of the target region. The dynamic loss value is the loss value of the target region used when the current policy data of the multiple action prediction network layers and the current policy data of the evaluation prediction network layer were last updated.

6. The method according to claim 5, characterized in that, The step of updating the current policy data of the multiple action prediction network layer and the current policy data of the evaluation prediction network layer based on the multiple dose distribution state data obtained in the current training, the multiple action sets corresponding one-to-one with the multiple dose distribution state data, the multiple actual reward values ​​corresponding one-to-one with the multiple action sets, the multiple predicted values ​​corresponding one-to-one with the multiple action sets, and the actual reward value of the target area includes: Based on the multiple actual reward values ​​corresponding one-to-one with the multiple action sets and the actual reward value of the target area, the actual cumulative reward value of the multiple action sets is determined, and based on the multiple dose distribution state data, the multiple action sets corresponding one-to-one with the multiple dose distribution state data and the actual cumulative reward value of the multiple action sets, the current policy data of the multiple action prediction network layer is updated; Based on the multiple dose distribution state data, the multiple action sets corresponding to the multiple dose distribution state data, and the multiple prediction values ​​corresponding to the multiple action sets, the current policy data of the evaluation prediction network layer is updated.

7. A method for generating a treatment plan, characterized in that, The method includes: Acquire image and contour data of the target area to be treated; Based on the image data and contour data of the target area to be treated, the dose distribution status data of the target area to be treated is determined; The dose distribution data of the target area to be treated is input into a deep reinforcement learning model to obtain a combination of target parameters of multiple different types in the target area to be treated; the deep reinforcement learning model is trained by the deep reinforcement learning model training method for generating treatment plans according to any one of claims 1-6. A treatment plan for the target area to be treated is generated based on the combination of target parameters.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6, or the method of claim 7.

9. A non-volatile readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-6, or the method described in claim 7.

Images