Model training method, treatment plan generation method, device, and medium

Abstract

The present disclosure provides a model training method, a treatment plan generation method, equipment and a medium, relating to the technical field of medical treatment, in particular to the technical field of radiotherapy. The deep reinforcement learning model is configured to include an action selection network and an evaluation network. The method comprises: obtaining initial dose distribution state data of a target target region; based on the initial dose distribution state data of the target target region and the current policy data of the action selection network and the evaluation network, using multiple sub-threads to determine the output data of the multiple sub-threads in parallel; based on the output data of each sub-thread in the multiple sub-threads in turn, updating the current policy data of the action selection network and the evaluation network, completing the current training of the deep reinforcement learning model; iteratively performing the above training process until the number of training of the deep reinforcement learning model reaches a preset number, obtaining the trained deep reinforcement learning model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of medical technology, in particular to the technical field of radiotherapy, and more particularly to a model training method, a treatment plan generation method, a device and a medium. BACKGROUND

[0002] In the technical field of medical technology, radiotherapy is one of the important means for treating tumors. Before radiotherapy is performed on a subject to be treated, a treatment plan usually needs to be designed in advance.

[0003] At present, the design of a treatment plan usually relies on a physicist (manual) to repeatedly adjust according to his / her own experience and professional skills, so as to design a relatively reasonable treatment plan. Therefore, the existing treatment plan method requires a higher clinical experience of the physicist, and needs to be continuously adjusted by trial and error, which is time-consuming and laborious, and has a low efficiency. SUMMARY

[0004] The present disclosure provides a model training method, a treatment plan generation method, a device and a medium.

[0005] In a first aspect, a deep reinforcement learning model training method for generating a treatment plan is provided, the deep reinforcement learning model is configured to include an action selection network and an evaluation network, the method comprising: obtaining initial dose distribution state data of a target target region; based on the initial dose distribution state data of the target target region and the current policy data of the action selection network and the evaluation network, using a plurality of sub-threads to determine the output data of the plurality of sub-threads in parallel; sequentially updating the current policy data of the action selection network and the evaluation network based on the output data of each sub-thread in the plurality of sub-threads, completing the current training of the deep reinforcement learning model; iteratively performing the above training process until the training times of the deep reinforcement learning model reach a preset number, obtaining a trained deep reinforcement learning model; wherein the output data of the sub-thread includes: a final dose distribution and a plurality of dose distribution state data of the target target region, a plurality of target point parameters corresponding to the plurality of dose distribution state data, a plurality of predicted values and a plurality of actual reward values corresponding to the plurality of target point parameters, a predicted value and an actual reward value of the target target region.

[0006] In some embodiments, when the target target region only includes one target region, based on the initial dose distribution state data of the target target region and the current policy data of the action selection network and the evaluation network, the output data of the plurality of sub-threads is determined in parallel using the plurality of sub-threads, including: using each sub-thread in the plurality of sub-threads to perform the following operations: based on the current dose distribution state data of the target target region and the current policy data of the action selection network and the evaluation network, determining the target point parameters corresponding to the current dose distribution state data of the target target region and the predicted values corresponding to the target point parameters; based on the target point parameters corresponding to the current dose distribution state data of the target target region, determining the dose distribution of the target target region; based on the dose distribution of the target target region, determining the actual reward value corresponding to the target point parameters; when the dose distribution of the target target region does not satisfy the preset prescribed dose, and the number of target points in the target target region is less than the preset maximum number of target points, based on the dose distribution of the target target region, updating the current dose distribution state data of the target target region; when the dose distribution of the target target region satisfies the preset prescribed dose, and / or the number of target points in the target target region is equal to the preset maximum number of target points, determining the final dose distribution of the target target region and the plurality of dose distribution state data of the target target region, the plurality of target point parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values corresponding to the plurality of target point parameters, and the plurality of actual reward values; based on the final dose distribution of the target target region and the plurality of predicted values corresponding to the plurality of target point parameters, determining the actual reward value and the predicted value of the target target region.

[0007] In some embodiments, when the target target area includes multiple target areas, based on the initial dose distribution state data of the target target area and the current policy data of the action selection network and the evaluation network, the output data of the multiple sub-threads are determined in parallel using the multiple sub-threads, including: using each of the multiple sub-threads to perform the following operations: based on the current dose distribution state data of the current target area and the current policy data of the action selection network and the evaluation network, determining the target point parameters corresponding to the current dose distribution state data of the current target area and the predicted values corresponding to the target point parameters; based on the target point parameters corresponding to the current dose distribution state data of the current target area, determining the dose distribution of the current target area; based on the dose distribution of the current target area, determining the actual reward value corresponding to the target point parameters; when the dose distribution of the current target area does not satisfy the preset prescribed dose, and the number of target points in the current target area is less than the preset maximum number of target points, updating the current dose distribution state data of the current target area based on the dose distribution of the current target area; when the dose distribution of the current target area satisfies the preset prescribed dose, and / or the number of target points in the current target area is equal to the preset maximum number of target points, determining the final dose distribution of the current target area and the multiple dose distribution state data of the current target area, the multiple target point parameters corresponding to the multiple dose distribution state data, the multiple predicted values corresponding to the multiple target point parameters, and the multiple actual reward values; based on the final dose distribution of the current target area and the multiple predicted values corresponding to the multiple target point parameters, determining the actual reward value and the predicted value of the current target area; when the current target area is not the last target area, updating the current target area and determining the current dose distribution state data of the current target area based on the final dose distribution of the last target area of the current target area; when the current target area is the last target area, based on the final dose distribution of each of the multiple target areas and the predicted value of each of the multiple target areas, determining the actual reward value and the predicted value of the target target area.

[0008] In some embodiments, the current policy data of the action selection network and the evaluation network is updated based on the output data of each of the plurality of sub-threads in sequence, comprising: sequentially performing the following operations on the output data of each of the plurality of sub-threads: when the final dose distribution of the target region obtained by the current training of the sub-thread meets the preset prescription dose, determining whether the actual reward value of the target region obtained by the current training of the sub-thread is greater than a dynamic reward threshold; the dynamic reward threshold is the actual reward value of the target region used when the current policy data of the action selection network and the evaluation network is updated last time; when the actual reward value of the target region obtained by the current training of the sub-thread is greater than the dynamic reward threshold, determining the loss value of the target region corresponding to the current training of the sub-thread based on the actual reward value and the predicted value of the target region obtained by the current training of the sub-thread; when the loss value of the target region corresponding to the current training of the sub-thread is less than a dynamic loss value, updating the current policy data of the action selection network and the evaluation network based on the plurality of dose distribution state data obtained by the current training of the sub-thread, the plurality of target parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values and the plurality of actual reward values corresponding to the plurality of target parameters, and the actual reward value of the target region; the dynamic loss value is the loss value of the target region used when the current policy data of the action selection network and the evaluation network is updated last time.

[0009] In some embodiments, the current policy data of the action selection network and the evaluation network is updated based on the plurality of dose distribution state data obtained by the current training of the sub-thread, the plurality of target parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values and the plurality of actual reward values corresponding to the plurality of target parameters, and the actual reward value of the target region, comprising: updating the current policy data of the action selection network and the evaluation network using a proximal policy optimization algorithm based on the plurality of dose distribution state data obtained by the current training of the sub-thread, the plurality of target parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values and the plurality of actual reward values corresponding to the plurality of target parameters, and the actual reward value of the target region.

[0010] In some embodiments, the target parameter comprises one of the following target parameters: target point size, target point position, target point weight.

[0011] In some embodiments, the dose distribution state data comprises: mask data of the target region, dose distribution of the target region, volume of the area with insufficient dose in the target region, and volume of the area with overflow dose in the target region.

[0012] In a second aspect, a method for generating a treatment plan is provided, including: obtaining image data and contour data of a target region to be treated; determining dose distribution state data of the target region to be treated according to the image data and the contour data; inputting the dose distribution state data of the target region to be treated into a deep reinforcement learning model to obtain target point parameters of the target region to be treated; the deep reinforcement learning model is trained according to the method of any one of the first aspect, and generates a treatment plan of the target region to be treated according to the target point parameters.

[0013] In a third aspect, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of the first aspect or the second aspect.

[0014] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the storage medium stores a computer program for executing the method of any one of the first aspect or the second aspect.

[0015] In the technical solution provided by the present disclosure, the deep reinforcement learning model for generating a treatment plan is trained in parallel by multiple sub-threads, and since the multiple sub-threads can try and error in parallel, the speed of model training is improved. Since the deep reinforcement learning model meets the characteristics of Gamma Knife treatment plan design, the processor in the computer system can realize the repeated trial and error process, therefore, the target point parameters with good effect can be automatically generated through the deep reinforcement learning model, so that a treatment plan with good effect can be generated according to the target point parameters, the dependence on clinical experience is reduced, the effect of the treatment plan can be improved without manually setting the target point parameters, and the efficiency of the physicist in designing the treatment plan is improved.

[0016] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are used to better understand the present solution and do not limit the present disclosure. Among them:

[0018] Figure 1 is an implementation environment schematic diagram of a deep reinforcement learning model training method and a treatment plan generation method for generating a treatment plan according to an embodiment of the present disclosure;

[0019] Figure 2 is a flowchart of a deep reinforcement learning model training method according to an embodiment of the present disclosure;

[0020] Figure 3 is a flowchart of another deep reinforcement learning model training method according to an embodiment of the present disclosure;

[0021] Figure 4 is a flowchart of another deep reinforcement learning model training method according to an embodiment of the present disclosure;

[0022] Figure 5 is a flowchart of a treatment plan generation method according to an embodiment of the present disclosure;

[0023] Figure 6 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of the embodiments of the present disclosure, and are taken to illustrate the preferred embodiments of the present disclosure, merely. Accordingly, those of ordinary skill in the art will recognize that there are numerous variations and modifications that can be made to the embodiments described and that those variations and modifications can be made without departing from the scope and spirit of the present disclosure. Additionally, for the purpose of clarity and the brevity, the description below omits the description of well-known functions and structures.

[0025] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0026] Before the present disclosure is described in detail, the application scenarios involved in the embodiments of the present disclosure are described.

[0027] The deep reinforcement learning model training method for generating a treatment plan, the treatment plan generation method, the device and the medium provided by the embodiments of the present disclosure can be applied in the field of medical technology, and specifically can be applied in the scenario of clinical radiotherapy (referred to as radiotherapy), such as the scenario of Gamma Knife treatment plan making.

[0028] Before radiotherapy is performed on a to-be-treated object, a treatment plan usually needs to be designed in advance.

[0029] At present, the design of a treatment plan usually depends on a physicist (manual) to repeatedly adjust according to his / her own experience and professional skills, so as to design a relatively reasonable treatment plan. Therefore, the existing treatment plan method requires high clinical experience of the physicist, and needs to be continuously adjusted through trial and error, which is time-consuming and laborious and has low efficiency.

[0030] Based on this, the embodiments of the present disclosure provide a deep reinforcement learning model training method for generating a treatment plan, a treatment plan generation method, a device and a medium, and the deep reinforcement learning model includes an action selection network and an evaluation network.

[0031] The method comprises the following steps: firstly, obtaining initial dose distribution state data of a target target region. Then, based on the initial dose distribution state data of the target target region and current policy data of an action selection network and an evaluation network, output data of a plurality of sub-threads is determined in parallel using the plurality of sub-threads. Subsequently, based on the output data of each of the plurality of sub-threads, the current policy data of the action selection network and the evaluation network is updated in sequence, and a current training of the deep reinforcement learning model is completed. The above training process is iterated until the number of training of the deep reinforcement learning model reaches a preset number, and a trained deep reinforcement learning model is obtained.

[0032] The output data of the sub-thread comprises: a final dose distribution and a plurality of dose distribution state data of the target target region, a plurality of target point parameters corresponding to the plurality of dose distribution state data, a plurality of predicted values and a plurality of actual reward values corresponding to the plurality of target point parameters, a predicted value and an actual reward value of the target target region.

[0033] As can be seen from the above, the present disclosure trains the deep reinforcement learning model for generating a treatment plan in parallel through a plurality of sub-threads. Since the plurality of sub-threads can try and error in parallel, the speed of model training is improved. Since the deep reinforcement learning model meets the characteristics of gamma knife treatment plan design, the processor in the computer system can realize the repeated trial and error process, therefore, the deep reinforcement learning model can automatically generate target point parameters with good effect, so as to generate a treatment plan with good effect according to the target point parameters, reduce the dependence on clinical experience, and improve the efficiency of the physicist in designing the treatment plan without manually setting the target point parameters.

[0034] The method provided in the present application mainly relates to two methods of model training method and treatment plan generation method. The implementation environment of the above two methods will be introduced first.

[0035] Figure 1 (a) in (a) is a schematic diagram of the implementation environment of the model training method according to the embodiments of the present disclosure. Referring to Figure 1 As shown in (a) in FIG. 1, the implementation environment comprises an image scanning device a1 and an electronic device a2.

[0036] The image scanning device a1 is used to scan and display the tumor site and the surrounding normal tissue of the subject to be treated. In some embodiments, the image scanning device a1 can be at least one of a computed tomography (CT) device, an emission computed tomography (ECT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasonic examination device.

[0037] In the embodiments of the present disclosure, the image scanning device a1 is used to obtain a medical scan image (e.g., image data of a target region, etc.) and upload the medical scan image to the electronic device a2, so that the electronic device a2 performs a subsequent model training method based on the medical scan image of the subject to be treated.

[0038] The electronic device a2 is a device used to train a deep reinforcement learning model. In some embodiments, the electronic device a2 can be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer.

[0039] Further, in some embodiments, the electronic device a2 can run a computer system including a processor for implementing a training method of a deep reinforcement learning model.

[0040] In some embodiments, the implementation environment further includes a server a3. In some embodiments, the server a3 is used to provide background communication services for the above-mentioned image scanning device a1 and the electronic device a2.

[0041] The server a3 can be at least one of a standalone physical server, a server cluster or a distributed file system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks, and big data or artificial intelligence platforms. The embodiments of the present disclosure do not limit this. In some embodiments, the number of the above-mentioned server a3 can be more or less, and the embodiments of the present disclosure do not limit this. Of course, the server a3 can also include other functions to provide more comprehensive and diversified services.

[0042] Figure 1 (b) in FIG. 1 is a schematic diagram of an implementation environment of a treatment plan generation method according to an embodiment of the present disclosure. Referring to FIG. 1, Figure 1(b) of FIG. 1B, the implementation environment includes an image scanning device b1, an electronic device b2, and a radiotherapy device b3.

[0043] The image scanning device b1 and Figure 1 The device form of the image scanning device a1 in (a) of FIG. 1A is similar, and reference can be made to the description of the image scanning device a1, which will not be repeated here.

[0044] In the embodiments of the present disclosure, the image scanning device b1 is configured to acquire a medical scanning image (for example, image data of a target region to be treated) of a subject to be treated, and upload the medical scanning image of the subject to be treated to the electronic device b2, so that the electronic device b2 performs a subsequent treatment plan generation method based on the medical scanning image of the subject to be treated.

[0045] The electronic device b2 is a device configured to apply a deep reinforcement learning model to generate a treatment plan.

[0046] The electronic device b2 and Figure 1 The device form of the electronic device a2 in (a) of FIG. 1A is similar, and reference can be made to the description of the electronic device a2, which will not be repeated here.

[0047] Further, in some embodiments, the electronic device b2 can run a computer system including a processor configured to implement a method of applying a deep reinforcement learning model to generate a treatment plan.

[0048] The radiotherapy device b3 is a device configured to perform radiotherapy. In some embodiments, the radiotherapy device b3 can be at least one of a gamma knife, a linear accelerator, a neutron knife, and an X-ray therapy machine.

[0049] In the embodiments of the present disclosure, the radiotherapy device b3 is configured to receive a treatment plan from the electronic device b2, and perform radiotherapy on a subject to be treated according to the treatment plan.

[0050] In some embodiments, the implementation environment further includes a server b4. In some embodiments, the server b4 is configured to provide a background communication service for the above-mentioned image scanning device b1, electronic device b2, and radiotherapy device b3.

[0051] The server b4 and Figure 1 The device form of the server a3 in (a) of FIG. 1A is similar, and reference can be made to the description of the server a3, which will not be repeated here.

[0052] Based on the implementation environment shown in (a) of FIG. 1B, the model training method provided by the embodiments of the present disclosure will be introduced first. Figure 1

[0053] ​The deep reinforcement learning model training method for generating a treatment plan provided by the embodiments of the present disclosure can be applied to Figure 1 The electronic device a2 in (a) in the above method can run a computer system including a processor for implementing the deep reinforcement learning model training method for generating a treatment plan.

[0054] The deep reinforcement learning model includes an action selection network and an evaluation network.

[0055] The action selection network includes an action decision parameter. The action decision parameter is used to represent the probability of selecting a target parameter from a plurality of selectable target parameters in a specific dose distribution state. The higher the probability, the more optimal it is to select the target parameter in the specific dose distribution state.

[0056] For example, when the target parameter is a target size, in a certain dose distribution state, the selectable target sizes are , and , and the corresponding selection probabilities are 10%, 20% and 70% respectively. Then, the optimal target size selection in the dose distribution state is .

[0057] The evaluation network includes a prediction decision parameter. The prediction decision parameter is used to determine the predicted value corresponding to each target parameter. The predicted value is used to represent the selection of a certain target parameter in a specific dose distribution state.

[0058] Figure 2 is a flowchart of a deep reinforcement learning model training method for generating a treatment plan according to an embodiment of the present disclosure. As shown in Figure 2 , the method includes:

[0059] S201, obtaining initial dose distribution state data of a target target region.

[0060] The target target region includes a treatment area of a subject to be treated. The target target region can include one or more target regions. When the target target region includes multiple target regions, the multiple target regions can be target regions of different shapes.

[0061] Optionally, the subject to be treated can be a phantom, a human body, an animal, etc. The target target region can be a tumor region of the subject to be treated, etc. The target target region can also be referred to as a planning target volume (PTV).

[0062] For example, as shown in Figure 2 , the image data and contour data of different target regions can include image data and contour data of a target region 1, image data and contour data of a target region 2, etc.

[0063] For example, in combination Figure 1 In (a), the image scanning device a1 can acquire image data of the object to be treated and transmit the image data to the electronic device a2.

[0064] Optionally, after receiving image data, electronic device a2 can delineate the image data to obtain the contours of different target areas of the object to be treated in the image data, that is, contour data.

[0065] For example, the contour data could be the contour of a tumor region.

[0066] Optionally, the electronic device a2 can also connect to a third-party software program. This third-party software program is used to outline the image data to obtain contour data.

[0067] Optionally, when outlining the image data, the physician can perform the outlining operation on the electronic device a2. The electronic device a2 can respond to the physician's outlining operation and obtain the outline data.

[0068] Alternatively, the electronic device a2 can also automatically delineate the contours of different target areas of the object to be treated in the image data using contour drawing software.

[0069] After determining the image data and contour data of the target area, the processor in electronic device a2 can determine the initial dose distribution state data of the target area based on the image data and contour data of the target area.

[0070] The initial dose distribution state data is used to represent the dose distribution state data of the target area when no target point is placed.

[0071] The dose distribution status data includes: mask data of the target area, dose distribution of the target area, volume of the insufficient dose distribution area in the target area, and volume of the overflow dose area in the target area.

[0072] Since the initial dose distribution state data is used to represent the dose distribution state data of the target area when no target point is placed, the dose distribution, the volume of the insufficient dose distribution area, and the volume of the overflow dose area in the initial dose distribution state data are all 0. In this case, the processor in electronic device a2 only needs to determine the mask data of the target area based on the image data and contour data of the target area to determine the initial dose distribution state data of the target area.

[0073] Specifically, the image data of the target area may include the contour data of the target area and the image data of the organ at risk (OAR).

[0074] The OAR refers to a normal organ around the target region, i.e., an organ that has not been diseased.

[0075] The processor in the electronic device a2 can process the image data and the contour data of the target region to generate mask data of the target region. In the process of generating the mask data, the processor in the electronic device a2 can perform different operations on the region corresponding to the target region and the region corresponding to the OAR based on the contour data, so as to distinguish the region corresponding to the target region from the region corresponding to the OAR.

[0076] Meanwhile, the search space of the subsequent target position (i.e., the placement position of the target point) can be limited to the region corresponding to the target region, so as to avoid the target position from falling into the region corresponding to the OAR or the region corresponding to other tissues, thereby avoiding the treatment damage to the region corresponding to the OAR or the region corresponding to other tissues. Moreover, limiting the search space of the subsequent target position to the region corresponding to the target region can reduce the search space according to the specific shape of the target region, thereby improving the speed of model training and generating a treatment plan.

[0077] For example, the processor in the electronic device a2 can construct a three-dimensional matrix with a uniform size. The image parameters of the region corresponding to the target region can be set to 1, the image parameters of the region corresponding to the OAR can be set to -1, and the image parameters of other tissues can be set to 0. In this way, the mask data distinguishing the region corresponding to the target region from the region corresponding to the OAR can be obtained.

[0078] S202, based on the initial dose distribution state data of the target region and the current policy data of the action selection network and the evaluation network, using multiple sub-threads to determine the output data of the multiple sub-threads in parallel.

[0079] For example, as shown in Figure 2 The multiple sub-threads can include a sub-thread 1, a sub-thread 2, and the like.

[0080] The output data of the sub-thread includes the final dose distribution of the target region and the multiple dose distribution state data, the multiple target point parameters corresponding to the multiple dose distribution state data, the multiple predicted values and the multiple actual reward values corresponding to the multiple target point parameters, and the predicted value and the actual reward value of the target region.

[0081] For example, after determining the initial dose distribution state data of the target region, the processor of the electronic device a2 can input the initial dose distribution state data into each sub-thread, so that each sub-thread formulates a complete treatment plan for the target region according to the initial dose distribution state data and the current policy data of the action selection network and the evaluation network, and then determines the output data of each sub-thread.

[0082] When the deep reinforcement learning model is trained for the first time, the current policy data of the action selection network and the evaluation network are initial policy data of the action selection network and the evaluation network. Each sub-thread can formulate a complete treatment plan for the target target region according to the initial policy data of the action selection network and the evaluation network and the initial dose distribution state data of the target target region, and then determine the output data of the first sub-thread of each sub-thread.

[0083] When the deep reinforcement learning model is trained for the first time, the current policy data of the action selection network and the evaluation network are initial policy data of the action selection network and the evaluation network. Each sub-thread can formulate a complete treatment plan for the target target region according to the initial policy data of the action selection network and the evaluation network and the initial dose distribution state data of the target target region, and then determine the output data of the first sub-thread of each sub-thread.

[0084] Since the generation process of the gamma knife treatment plan is a process of sequentially arranging target points, the dose distribution state of the target target region will change after the arrangement of a target point is completed. Therefore, in the generation process of the complete treatment plan of the target target region, the dose distribution state of the target target region will change with the placement of one target point after another.

[0085] Therefore, in the embodiment of the present application, for each sub-thread, the output data of the sub-thread can be obtained only when a complete treatment plan for the target target region is completed.

[0086] Specifically, after obtaining the initial dose distribution state data, the processor in the electronic device a2 can input the initial dose distribution state data into a plurality of sub-threads respectively. Since each sub-thread can share the action selection network, each sub-thread can output a first type of target point parameter selected for a first target point based on the initial policy data of the action selection network and the initial dose distribution state data of the target target region. Then, each sub-thread can call different algorithms to determine other types of target point parameters, thereby obtaining a target point parameter combination for placing the first target point. The target point parameter combination includes the first type of target point parameter selected for the first target point output by the action selection network and the other types of target point parameters selected for the first target point determined by calling different algorithms.

[0087] Since the deep reinforcement learning model also includes the evaluation network, and each sub-thread can also share the evaluation network, while determining the first type of target point parameter of the first target point, the evaluation network can also determine a predicted value corresponding to the first type of target point parameter of the first target point according to its initial policy data and the initial dose distribution state data of the target target region.

[0088] After the first target point is placed, the dose distribution state of the target target region changes, at this time, each sub-thread can also determine the dose distribution of the target target region based on the target point parameter combination of the first target point, and determine the actual reward value corresponding to the first type of target point parameter of the first target point based on the dose distribution of the target target region.

[0089] Then, each sub-thread can update the dose distribution state data of the target target region based on the dose distribution of the target target region, and repeat the above target point placement process according to the updated dose distribution state data of the target target region until the generation of the complete treatment plan of the target target region is completed, obtaining the final dose distribution of the target target region and the multiple dose distribution state data of the target target region, the multiple target point parameters corresponding to the multiple dose distribution state data, the multiple predicted values corresponding to the multiple target point parameters, the multiple actual reward values corresponding to the multiple target point parameters, the actual reward value and the predicted value of the target target region. In this way, each sub-thread can obtain its output data.

[0090] S203, sequentially update the current policy data of the action selection network and the evaluation network based on the output data of each sub-thread in the multiple sub-threads, complete the current training of the deep reinforcement learning model.

[0091] Specifically, after each sub-thread obtains its output data, it means that the treatment plan of each sub-thread for the target target region has been completed. In this case, the current policy data of the action selection network and the evaluation network can be updated based on the output data of each sub-thread in the multiple sub-threads, and the current training of the deep reinforcement learning model is completed.

[0092] It should be noted that the training process of the deep reinforcement learning model is equivalent to a trial-and-error process of treatment plan generation, that is, the treatment plan generated by each sub-thread is not necessarily a good plan, that is, the training process of the deep reinforcement learning model is equivalent to a trial-and-error process of treatment plan generation, therefore, there may be a case that the output data obtained by one or more sub-threads is discarded because it does not meet the update condition.

[0093] Therefore, in the embodiment of the present application, in the case that the output data of the multiple sub-threads does not meet the update condition, the current policy data of the action selection network and the evaluation network does not need to be updated, that is, the training of the deep reinforcement learning model is completed once, which does not mean that the current policy data of the action selection network and the evaluation network will be updated.

[0094] Of course, in the case that the output data of the multiple sub-threads meets the update condition, the training of the deep reinforcement learning model is completed once, which means that the current policy data of the action selection network and the evaluation network is updated multiple times (i.e. the same number of times as the number of sub-threads).

[0095] Correspondingly, in a case where the output data of at least one sub-thread (i.e., part of the plurality of sub-threads) satisfies the update condition, one training of the deep reinforcement learning model is completed, and the number of times of updating the current policy data of the action selection network and the evaluation network is at least one (i.e., the same as the number of the at least one sub-thread).

[0096] When the current policy data of the action selection network and the evaluation network is updated, the plurality of sub-threads can package the output data of the plurality of sub-threads into a message buffer, and return the message buffer to the deep reinforcement learning model through a communication pipe, so that the deep reinforcement learning model updates the current policy data of the action selection network and the evaluation network, and thus completes the current training of the deep reinforcement learning model.

[0097] S204, iteratively performing the training process until the number of times of training the deep reinforcement learning model reaches a preset number, to obtain a trained deep reinforcement learning model.

[0098] In this way, the training process of the deep reinforcement learning model is iteratively performed (i.e., S201 to S203) until the number of times of training the deep reinforcement learning model reaches the preset number, to obtain a trained deep reinforcement learning model.

[0099] Optionally, the electronic device can further include an experience memory unit (buffer_memory). The experience memory unit is used to store the final dose distribution and the plurality of dose distribution state data of the target target area in each update stage in the current policy data update process, the plurality of target point parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values and the plurality of actual reward values corresponding to the plurality of target point parameters, and the predicted value and the actual reward value of the target target area (i.e., the output data of each sub-thread). Subsequently, by using the corresponding relationship stored in the experience memory unit, various required data determined in each training can be found, and then the model training is performed.

[0100] In some embodiments, the target target area can include only one target area (also referred to as a single target area), or can include a plurality of target areas (also referred to as a multi-target area). In this case, the embodiments of the present application provide two model training methods: an embodiment of a model training method implemented in a single target area scenario and an embodiment of a model training method implemented in a multi-target area scenario.

[0101] The two model training methods provided by the embodiments of the present application will be described in detail below. Figure 3 and Figure 4 The two model training methods provided by the embodiments of the present application will be described in detail below. Figure 3 An embodiment of a model training method implemented in a single target area scenario provided by the present application is shown in FIG. 2. As shown in FIG. 2, the model training method implemented in a single target area scenario includes the following steps. Figure 3As shown, the model training method implemented in the single target region scenario specifically includes:

[0102] A1, obtain initial dose distribution state data of the target region.

[0103] The specific implementation process of this step can refer to the description of S201, which will not be repeated here.

[0104] After obtaining the initial dose distribution state data of the target region, based on the initial dose distribution state data of the target region and the current policy data of the action selection network and the evaluation network, each sub-thread of the plurality of sub-threads can be used to perform the following A2-A7 to obtain the output data of the plurality of sub-threads.

[0105] A2, based on the current dose distribution state data of the target region and the current policy data of the action selection network and the evaluation network, determining the target point parameter corresponding to the current dose distribution state data of the target region and the predicted value corresponding to the target point parameter.

[0106] Since the generation process of the Gamma knife treatment plan is a process of sequentially arranging target points, after completing the arrangement of a target point, the dose distribution state of the target region will change, therefore, in the generation process of the complete treatment plan of the target region, the current dose distribution state of the target region will change with the placement of one target point after another.

[0107] Therefore, after obtaining the current dose distribution state data of the target region (for the first target point, the current dose distribution state data of the target region is the initial dose distribution state data of the target region), since the deep reinforcement learning model includes an action selection network, the action selection network can make action selection decisions according to its current policy data and the current dose distribution state data of the target region, i.e., output action decision parameters to the sub-thread; after receiving the action decision parameters sent by the action selection network, the sub-thread can determine the target point parameter according to the action decision parameters.

[0108] Optionally, the above-mentioned target point parameter includes one of the following target point parameters: target point size, target point position, target point weight. The target point weight is used to determine the dose of the target point.

[0109] For example, when the target point parameter is the target point size, the sub-thread can determine the target point size of the current target point according to the action decision parameters sent by the action selection network, so as to obtain the target point parameter corresponding to the current dose distribution state data of the target region, i.e., the target point size.

[0110] Secondly, since the deep reinforcement learning model further comprises the evaluation network, while the action selection network selects the target point parameters, the evaluation network can predict the value of the target point parameter selection action according to the current policy data and the current dose distribution state data of the target region, that is, output the predicted decision parameters to the sub-thread; after receiving the predicted decision parameters sent by the evaluation network, the sub-thread can determine the predicted value corresponding to the target point parameters according to the predicted decision parameters.

[0111] A3, determine the dose distribution of the target region based on the target point parameters corresponding to the current dose distribution state data of the target region.

[0112] Wherein, the dose distribution of the target region refers to the dose distribution of the target region after placing the current target point (i.e. the target point corresponding to the target point parameters determined in step A2) in the target region.

[0113] Specifically, after determining the target point parameters (for example: target point size) corresponding to the current dose distribution state data of the target region, the sub-thread can call the shape matching algorithm to match the selected target point size with the contour of the target region to determine the best placement position of the current target point in the target region, that is, the target point position of the current target point; then, the sub-thread can determine the weight of the current target point according to the size, position and preset prescribed dose of the current target point. Optionally, after determining the target point position and target point size, the target point weight can also be defaulted to 1.

[0114] It can be understood that when the target point parameters are the target point position or the target point weight, the sub-thread can call other algorithms to determine other target point parameters (for example: target point size and target point weight, target point size and target point position) included in the treatment plan.

[0115] After determining the target point size, target point position and target point weight, the sub-thread can execute the target point placement action in the target region according to the target point size, target point position and target point weight. In this way, after placing the target point, the dose distribution of the target region will change. In this case, the sub-thread can determine the dose distribution of the target region.

[0116] A4, determine the actual reward value corresponding to the target point parameters based on the dose distribution of the target region.

[0117] After determining the dose distribution of the target region, in order to subsequently train the deep reinforcement learning model, the sub-thread can determine the actual reward value corresponding to the target point parameters (for example: target point size) in the current dose distribution state based on the dose distribution of the target region.

[0118] Since the dose distribution of the target target region is determined after the current target point is placed, the dose distribution of the target target region can truly reflect the contribution of the selection action of the target point parameter (for example, the target point size) in step A2 to the dose distribution of the target target region, so that the actual reward value corresponding to the target point parameter (for example, the target point size) in the current dose distribution state can be obtained based on the dose distribution of the target target region.

[0119] For example, the actual reward value corresponding to the target point parameter includes a positive reward and a negative reward. The growth value of the dose coverage and the conformity of the target target region before and after the execution of the current target point is multiplied by the corresponding weight as the positive reward; and the growth value of the dose overflow rate of the target target region before and after the execution of the current target point is multiplied by the corresponding weight as the negative reward.

[0120] A5, when the dose distribution of the target target region does not satisfy the preset prescribed dose, and the number of target points in the target target region is less than the preset maximum number of target points, updating the current dose distribution state data of the target target region based on the dose distribution of the target target region.

[0121] When the dose distribution of the target target region does not satisfy the preset prescribed dose, and the number of target points in the target target region is less than the preset maximum number of target points, it indicates that the treatment plan for the target target region has not been completed. In this case, the sub-thread can determine the current dose distribution state data of the target target region according to the dose distribution of the target target region, and update the current dose distribution state data of the target target region, that is, update the mask data of the target target region, the dose distribution of the target target region, the volume of the region with insufficient dose distribution in the target target region, and the volume of the region with overflow dose in the target target region.

[0122] Then, the sub-thread repeats steps A2-A4 to determine the subsequent multiple target point parameters, the predicted values corresponding to the multiple target point parameters, and the actual reward values based on the updated current dose distribution state data of the target target region and the current policy data of the action selection network and the evaluation network.

[0123] Optionally, when the sub-thread updates the current dose distribution state data of the target target region based on the dose distribution of the target target region, the dose distribution of the target target region can be feature extracted to obtain the updated current dose distribution state data.

[0124] A6, when the dose distribution of the target target region satisfies the preset prescribed dose, and / or the number of target points in the target target region is equal to the preset maximum number of target points, determining the final dose distribution of the target target region, the multiple dose distribution state data of the target target region, the multiple target point parameters corresponding to the multiple dose distribution state data, the multiple predicted values corresponding to the multiple target point parameters, and the multiple actual reward values.

[0125] Specifically, when the dose distribution of the target target region meets the preset prescribed dose, and / or the number of target points in the target target region is equal to the preset maximum number of target points, it is indicated that the target target region has completed the treatment plan. In this case, the sub-thread can determine the final dose distribution of the target target region and the plurality of dose distribution state data of the target target region, the plurality of target point parameters corresponding to the plurality of dose distribution state data, the plurality of prediction values corresponding to the plurality of target point parameters, and the plurality of actual reward values.

[0126] A7, based on the final dose distribution of the target target region and the plurality of prediction values corresponding to the plurality of target point parameters, determine the actual reward value and the prediction value of the target target region.

[0127] Specifically, when the dose distribution of the target target region meets the preset prescribed dose, and / or the number of target points in the target target region is equal to the preset maximum number of target points, the final dose distribution of the target target region and the plurality of prediction values corresponding to the plurality of target point parameters can be obtained. In this case, the sub-thread can determine the actual reward value of the target target region based on the final dose distribution, and determine the prediction value of the target target region based on the plurality of prediction values corresponding to the plurality of target point parameters.

[0128] Optionally, after determining the final dose distribution of the target target region, the dose coverage and the conformality of the target target region can be determined according to the final dose distribution of the target target region, and the actual reward value of the target target region can be determined according to the dose coverage and the conformality of the target target region.

[0129] Optionally, after determining the plurality of prediction values corresponding to the plurality of target point parameters, the plurality of prediction values corresponding to the plurality of target point parameters can be summed to obtain the prediction value of the target target region.

[0130] At this point, after the plurality of sub-threads are executed in parallel, the determination of the output data of the plurality of sub-threads is completed.

[0131] After determining the output data of the plurality of sub-threads, since each sub-thread needs to determine whether the treatment plan determined by it is a good treatment plan, i.e. each sub-thread needs to determine whether its output data can be used to update the current strategy data of the action selection network and the evaluation network, the following A8-A10 can be executed on the output data of each sub-thread in the plurality of sub-threads in turn, thereby completing the current training of the deep reinforcement learning model.

[0132] A8, when the final dose distribution of the target target region obtained by the current training of the sub-thread meets the preset prescribed dose, determine whether the actual reward value of the target target region obtained by the current training of the sub-thread is greater than the dynamic reward threshold.

[0133] The dynamic reward threshold is the actual reward value of the target region used when the current policy data of the action selection network and the evaluation network is last updated.

[0134] It should be noted that since the treatment plan of the target region is completed when the dose distribution of the target region meets the preset prescription dose, and / or the number of target points in the target region is equal to the preset maximum number of target points, when the treatment plan of the target region is obtained when the dose distribution of the target region meets the preset prescription dose, the sub-thread does not need to determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose, and can directly proceed to the subsequent determination step. Of course, the sub-thread can also determine again whether the dose distribution corresponding to the treatment plan meets the preset prescription dose.

[0135] When the treatment plan of the target region is obtained when the number of target points in the target region is equal to the preset maximum number of target points, since the treatment plan only meets the preset maximum number of target points, but does not necessarily meet the preset prescription dose, the sub-thread needs to determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose.

[0136] If the dose distribution corresponding to the treatment plan does not meet the preset prescription dose, it means that the treatment plan is not a good treatment plan. Therefore, the sub-thread can discard the output data of the sub-thread, that is, it is not necessary to update the current policy data of the action selection network and the evaluation network according to the output data of the sub-thread.

[0137] Correspondingly, if the dose distribution corresponding to the treatment plan meets the preset prescription dose, it means that the treatment plan can meet the requirement of meeting the preset prescription dose. In this case, the sub-thread can determine whether the actual reward value of the target region is greater than the dynamic reward threshold.

[0138] The dynamic reward threshold is the actual reward value of the target region used when the current policy data of the action selection network and the evaluation network is last updated.

[0139] By determining whether the actual reward value of the target region obtained in the current training is greater than the dynamic reward threshold, it can be determined whether the quality of the treatment plan is better than the quality of the treatment plan used when the current policy data of the action selection network and the evaluation network is last updated.

[0140] When the actual reward value of the target target region obtained in the current training is less than or equal to the dynamic reward threshold, it indicates that the quality of the treatment plan is worse than or equal to the quality of the treatment plan used when the current policy data of the action selection network and the evaluation network is updated last time. Therefore, the sub-thread can discard the output data of the sub-thread, that is, the current policy data of the action selection network and the evaluation network is not updated according to the output data of the sub-thread.

[0141] Correspondingly, when the actual reward value of the target target region obtained in the current training is greater than the dynamic reward threshold, it indicates that the quality of the treatment plan is better than the quality of the treatment plan used when the current policy data of the action selection network and the evaluation network is updated last time, that is, the quality of the treatment plan is better. In this case, the sub-thread can perform A9.

[0142] A9, when the actual reward value of the target target region obtained in the current training of the sub-thread is greater than the dynamic reward threshold, the loss value of the target target region corresponding to the current training of the sub-thread is determined based on the actual reward value and the predicted value of the target target region obtained in the current training of the sub-thread.

[0143] Specifically, when the actual reward value of the target target region obtained in the current training is greater than the dynamic reward threshold, it indicates that the quality of the treatment plan is better. In this case, the sub-thread can determine the loss value of the target target region corresponding to the current training based on the actual reward value and the predicted value of the target target region obtained in the current training.

[0144] Optionally, the sub-thread can use a preset loss function to determine the loss value of the target target region corresponding to the current training according to the predicted value and the actual reward value of the target target region obtained in the current training.

[0145] Optionally, the preset loss function can be a relative advantage parameter, or other general loss function, which is not limited in the present application.

[0146] Then, the sub-thread can determine whether the loss value of the target target region corresponding to the current training is less than the dynamic loss value.

[0147] The dynamic loss value is the loss value of the target target region used when the current policy data of the action selection network and the evaluation network is updated last time.

[0148] If the loss value of the target target region corresponding to the current training is greater than or equal to the loss value of the target target region used when the current policy data of the action selection network and the evaluation network is updated last time, it indicates that the deep reinforcement learning model is inaccurate in estimating the quality of the treatment plan. Therefore, the sub-thread can discard the output data of the sub-thread, that is, the current policy data of the action selection network and the evaluation network is not updated according to the output data of the sub-thread.

[0149] If the loss value of the target target region corresponding to the current training is less than the loss value of the target target region used when the current policy data of the action selection network and the evaluation network is updated last time, it indicates that the quality estimation of the treatment plan of the deep reinforcement learning model is accurate. In this case, the sub-thread can execute A10.

[0150] A10, when the loss value of the target target region corresponding to the current training of the sub-thread is less than the dynamic loss value, based on the plurality of dose distribution state data obtained by the current training of the sub-thread, the plurality of target point parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values corresponding to the plurality of target point parameters, the plurality of actual reward values, the actual reward value of the target target region, update the current policy data of the action selection network and the evaluation network.

[0151] In an implementable manner, the method for updating the current policy data of the action selection network and the evaluation network based on the plurality of dose distribution state data obtained by the current training of the sub-thread, the plurality of target point parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values corresponding to the plurality of target point parameters, and the actual reward value of the target target region, specifically includes:

[0152] Based on the plurality of actual reward values corresponding to the plurality of target point parameters, the actual reward value of the target target region, determine the actual cumulative reward value of the plurality of target point parameters, and based on the plurality of dose distribution state data, the plurality of target point parameters corresponding to the plurality of dose distribution state data, and the actual cumulative reward value of the plurality of target point parameters, update the current policy data of the action selection network, and based on the plurality of dose distribution state data, the plurality of target point parameters corresponding to the plurality of dose distribution state data, and the plurality of predicted values corresponding to the plurality of target point parameters, update the current policy data of the evaluation network.

[0153] Specifically, since the quality of the treatment plan of the target target region is the result of the combination of the plurality of target point parameters corresponding to the plurality of dose distribution state data selected by the action selection network, in a certain dose distribution state of the target target region, the actual reward value of the target point parameter selected by the action selection network is high, which does not mean that the quality of the treatment plan of the entire target target region will be good. If the quality of the final treatment plan of the target target region is also good, the reference value of the selection of the target point parameter by the action selection network will be high, and if the quality of the final treatment plan of the target target region is not good, the reference value of the selection of the target point parameter by the action selection network will be small. Therefore, after obtaining the plurality of target point parameters corresponding to the plurality of dose distribution state data, the actual cumulative reward value of the plurality of target point parameters can also be determined according to the plurality of actual reward values corresponding to the plurality of target point parameters and the actual reward value of the target target region.

[0154] The actual cumulative reward value can include an actual reward value of each target point parameter selected by the action selection network and a delayed reward value after completion of the entire treatment plan. The delayed reward value is determined based on the actual reward value of the target target volume. That is, after completion of the treatment plan of the target target volume, the actual reward value of the target target volume is determined according to the dose distribution of the treatment plan, and the actual reward value is distributed to the target point parameter selected by the action selection network in each target point parameter combination constituting the treatment plan according to a weight, so as to form a delayed reward value of the target point parameter selected by the action selection network; and the actual reward value and the delayed reward value of each target point parameter selected by the action selection network are accumulated to obtain an actual cumulative reward value of each target point parameter selected by the action selection network.

[0155] When the actual cumulative reward value is high, it is said that the reference value of the target point parameter selected by the action selection network in the dose distribution state is high. In this case, the current policy data of the action selection network can be updated based on the plurality of dose distribution state data, the plurality of target point parameters corresponding to the plurality of dose distribution state data, and the actual cumulative reward values of the plurality of target point parameters.

[0156] Meanwhile, since the loss value of the target target volume is obtained based on the predicted value of the target target volume, and the predicted value of the target target volume is the accumulation of the predicted values of the plurality of target point parameters selected by the action selection network, when the loss value of the target target volume is less than the dynamic loss value, it is said that the predicted values of the plurality of target point parameters selected by the action selection network have high reference values. In this case, the current policy data of the evaluation network can be updated based on the plurality of dose distribution state data, the plurality of target point parameters corresponding to the plurality of dose distribution state data, and the plurality of predicted values corresponding to the plurality of target point parameters.

[0157] In yet another implementable manner, the method for updating the current policy data of the action selection network and the evaluation network based on the plurality of dose distribution state data obtained by the current training of the sub-thread, the plurality of target point parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values corresponding to the plurality of target point parameters, the plurality of actual reward values, and the actual reward value of the target target volume, specifically includes:

[0158] The plurality of dose distribution state data obtained by the current training of the sub-thread, the plurality of target point parameters corresponding to the plurality of dose distribution state data, the plurality of predicted values corresponding to the plurality of target point parameters, the plurality of actual reward values, and the actual reward value of the target target volume are used to update the current policy data of the action selection network and the evaluation network by using a proximal policy optimization algorithm.

[0159] The proximal policy optimization algorithm (PPO) can be efficiently trained in a large-scale environment, and can handle continuous action space and high-dimensional state space, which meets the characteristics of gamma knife treatment planning design, can realize repeated trial and error process, reduces the dependence on clinical experience, and improves the efficiency of the physicist in designing the treatment plan.

[0160] It can be understood that other algorithms can also be used, for example: A2C algorithm (Advantage Actor-Critic), Trust Region Policy Optimization (TPRO) updates the current policy data of the action selection network and the evaluation network.

[0161] A11、iterating the training process until the number of times of training the deep reinforcement learning model reaches a preset number of times, and obtaining the trained deep reinforcement learning model.

[0162] The specific implementation process of this step can refer to the description of S204, which will not be repeated here.

[0163] The above Figure 3 is an embodiment of a model training method implemented in a single target region scenario. However, multiple target regions are common in clinical radiotherapy. Therefore, Figure 4 An embodiment of a model training method implemented in a multi-target region scenario is shown, and the model training method implemented in the multi-target region scenario specifically includes: Figure 4

[0164] B1, obtaining initial dose distribution state data of a target region.

[0165] The specific implementation process of this step can refer to the description of S201, which will not be repeated here.

[0166] After obtaining the initial dose distribution state data of the target region, the current policy data of the action selection network and the evaluation network can be used to execute B2-B8 using each sub-thread in the plurality of sub-threads based on the initial dose distribution state data of the target region and the current policy data of the action selection network and the evaluation network, to obtain output data of the plurality of sub-threads.

[0167] B2, based on the current dose distribution state data of the current target region and the current policy data of the action selection network and the evaluation network, determining the target point parameters corresponding to the current dose distribution state data of the current target region and the predicted value corresponding to the target point parameters.

[0168] It should be noted that if the current target region is the first target region, the current dose distribution state data of the current target region is the initial dose distribution state data obtained in B1.​

[0169] If the current target region is not the first target region, since interaction occurs between adjacent target regions, each sub-thread can determine the dose distribution state data corresponding to the final dose distribution of the previous target region as the current dose distribution state data of the current target region.

[0170] The specific implementation process of this step can refer to the description of A2, which will not be described here.

[0171] B3, determining the dose distribution of the current target region based on the target point parameters corresponding to the current dose distribution state data of the current target region.

[0172] The specific implementation process of this step can refer to the description of A3, which will not be described here.

[0173] B4, determining the actual reward value corresponding to the target point parameters based on the dose distribution of the current target region.

[0174] The specific implementation process of this step can refer to the description of A4, which will not be described here.

[0175] B5, when the dose distribution of the current target region does not satisfy the preset prescribed dose, and the number of target points in the current target region is less than the preset maximum number of target points, updating the current dose distribution state data of the current target region based on the dose distribution of the current target region.

[0176] The specific implementation process of this step can refer to the description of A5, which will not be described here.

[0177] B6, when the dose distribution of the current target region satisfies the preset prescribed dose, and / or, the number of target points in the current target region is equal to the preset maximum number of target points, determining the final dose distribution of the current target region and a plurality of dose distribution state data of the current target region, a plurality of target point parameters corresponding to the plurality of dose distribution state data, a plurality of predicted values corresponding to the plurality of target point parameters, and a plurality of actual reward values, and determining the actual reward value and the predicted value of the current target region based on the final dose distribution of the current target region and the plurality of predicted values corresponding to the plurality of target point parameters.

[0178] The specific implementation process of this step can refer to the description of A6, which will not be described here.

[0179] After determining the actual reward value and the predicted value of the current target region, since the target target region includes a plurality of target regions, the sub-thread needs to determine whether the current target region is the last target region.

[0180] B7, when the current target region is not the last target region, updating the current target region, and determining the current dose distribution state data of the current target region based on the final dose distribution of the previous target region of the current target region.

[0181] B8, when the current target region is the last target region, determining the actual reward value and the predicted value of the target target region based on the final dose distribution of each of the plurality of target regions and the predicted value of each of the plurality of target regions.

[0182] Specifically, after determining the actual reward value and the predicted value of a target region, the sub-thread can perform the same operation (iteration steps B2~ B6) on the next target region until the determination of the actual reward value and the predicted value of the last target region in the plurality of target regions is completed.

[0183] Among them, the actual reward value of the plurality of target regions is the overall reward for the selection of the plurality of target point parameters of the plurality of target regions obtained after the above iteration is completed, for example: the target target region includes 3 target regions, after the determination of the target point parameters of the 3rd target region is completed, all the target point parameters of the 3 target regions are determined as a group of selection, and the overall reward is given to this group of selection.

[0184] Therefore, the actual reward value of the target target region can be determined by the sub-thread according to the final dose distribution of the plurality of target regions. That is, different weights can be assigned to the actual reward values of different target regions according to the importance of different target regions, and then the actual reward values of different target regions are weighted and summed to obtain the actual reward value of the target target region.

[0185] The predicted value of the target target region can be determined by the sub-thread by summing the predicted values of the plurality of target regions.

[0186] At this point, the determination of the output data of the plurality of sub-threads is completed.

[0187] After determining the output data of the plurality of sub-threads, since each sub-thread needs to determine whether the treatment plan determined by it is a good treatment plan, that is, each sub-thread needs to determine whether the output data can be used to update the current strategy data of the action selection network and the evaluation network, therefore, the following B9-B11 can be performed on the output data of each sub-thread in the plurality of sub-threads in turn, thereby completing the current training of the deep reinforcement learning model.

[0188] B9, when the final dose distribution of the target target region obtained by the current training of the sub-thread meets the preset prescription dose, determining whether the actual reward value of the target target region obtained by the current training of the sub-thread is greater than the dynamic reward threshold.

[0189] The specific implementation process of this step can refer to the description of A8, which will not be repeated here.

[0190] It should be noted that since the number of target regions is multiple, when the final dose distribution of each target region in the multiple target regions meets the preset prescribed dose, the sub-thread can determine that the final dose distribution of the target region obtained by the current training of the sub-thread meets the preset prescribed dose.

[0191] B10, when the actual reward value of the target region obtained by the current training of the sub-thread is greater than the dynamic reward threshold, determining the loss value of the target region corresponding to the current training of the sub-thread based on the actual reward value and the predicted value of the target region obtained by the current training of the sub-thread.

[0192] The specific implementation process of this step can refer to the description of A9, which will not be repeated here.

[0193] B11, when the loss value of the target region corresponding to the current training of the sub-thread is less than the dynamic loss value, updating the current policy data of the action selection network and the evaluation network based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target point parameters corresponding to the multiple dose distribution state data, the multiple predicted values and the multiple actual reward values corresponding to the multiple target point parameters, and the actual reward value of the target region.

[0194] The specific implementation process of this step can refer to the description of A10, which will not be repeated here.

[0195] B12, iteratively performing the training process until the number of training of the deep reinforcement learning model reaches the preset number, and obtaining the trained deep reinforcement learning model.

[0196] The specific implementation process of this step can refer to the description of S204, which will not be repeated here.

[0197] The function of the processor in generating the treatment plan will be introduced below.

[0198] Figure 5 is a flowchart of a processor-implemented treatment plan generation method according to an embodiment of the present disclosure. As shown in Figure 5 The method comprises:

[0199] S501, obtaining image data and contour data of a target region to be treated.

[0200] The specific process of obtaining the image data and the contour data of the target region to be treated can refer to the specific process of obtaining the image data and the contour data of the target region in S201, which will not be repeated here.

[0201] For example, as shown in Figure 5As shown, the image data of the target area to be treated can be the head CT image of the subject to be treated, and the contour data can be the contour data of the target area 1 and the contour data of the target area 2 in the head of the subject to be treated.

[0202] S502. Based on the image data and contour data of the target area to be treated, determine the dose distribution status data of the target area to be treated.

[0203] For details on the specific process of determining the dose distribution status data of the target area to be treated, please refer to S201, which describes the specific process of determining the initial dose distribution status data of the target area to be treated. It will not be repeated here.

[0204] S503. Input the dose distribution data of the target area to be treated into the deep reinforcement learning model to obtain the target parameters of the target area to be treated.

[0205] Among them, the deep reinforcement learning model is based on Figure 2 , Figure 3 or Figure 4 The model was trained using the method shown.

[0206] For example, such as Figure 5 As shown, the target parameters of the target region to be treated output by the deep reinforcement learning model can include the target parameters of target region 1 and target region 2. For example, the target parameters of the target region to be treated output by the deep reinforcement learning model are: target size.

[0207] S504. Generate a treatment plan for the target area based on the target parameters.

[0208] For example, such as Figure 5 As shown, after determining the target parameters of target region 1 and target region 2, the processor can determine target parameter combination 1 (consisting of multiple different types of target parameters for target region 1) and target parameter combination 2 (consisting of multiple different types of target parameters for target region 2) based on the target parameters of target region 1 and target region 2. Then, the processor can generate a treatment plan for the target regions based on target parameter combination 1 and target parameter combination 2.

[0209] The target parameter combination includes: target size, target location, and target weight.

[0210] Now, taking target size as an example, we will describe the process of a processor-based method for generating treatment plans. It is understood that target parameters can also be target location or target weight.

[0211] Firstly, the processor acquires image data and contour data of a target region to be treated, and then inputs the acquired image data and contour data of the target region to be treated into a deep reinforcement learning model trained by the model training method shown in Figure 2 , Figure 3 or Figure 4 , to obtain a first target point size output by the deep reinforcement learning model. Then, a thread in the processor calls a shape matching algorithm to perform shape matching between the first target point size output by the deep reinforcement learning model and the contour of the target region to be treated, to determine a best placement position of the first target point size in the target region to be treated, i.e., to determine a first target point position. Then, the thread in the processor calls a weight optimization algorithm to determine a first target point weight according to the first target point size, the first target point position, and a prescribed dose. Thus, the determination of the target point parameter combination of the first target point in the treatment plan is completed.

[0212] Then, the thread in the processor calculates a dose distribution of the target region to be treated according to the target point parameter combination of the first target point, determines current dose distribution state data of the target region to be treated according to the dose distribution of the target region to be treated, inputs the current dose distribution state data of the target region to be treated into the deep reinforcement learning model, obtains a second target point size output by the deep reinforcement learning model, and determines a target point parameter combination of a second target point based on the second target point size (the determination methods of the second target point position and the second target point weight are the same as those of the first target point position and the first target point weight, and are not described herein).

[0213] The determination process of the target point size, the target point position, and the target point weight is repeatedly iterated to determine the target point parameter combinations of the remaining target points, until the dose distribution of the target region to be treated meets the prescribed dose, and / or the number of target points in the target region to be treated is equal to a preset maximum number of target points, to complete the determination of the target point parameter combinations of the target region to be treated. The target point parameter combinations of the target region to be treated include target point parameter combinations of multiple target points, i.e., include multiple sets of collections {(S1, P1, W1), (S2, P2, W2), …, (S n , P n , W n )} composed of a target point size (S), a target point position (P), and a target point weight (W).

[0214] It can be understood that the target point weight can also be obtained without calling an algorithm, and can be set as 1 by default.

[0215] The treatment plan can be displayed in a display form of an image corresponding to the image data of the target object. In the image corresponding to the image data of the target object, the target point parameter combinations of the target region 1 and the target region 2 can be explicitly marked.

[0216] Optionally, after the treatment plan of the target region to be treated is determined, the physicist (manually) can also adjust each target point parameter in the target point parameter combination of the treatment plan of the target region to be treated according to experience, so as to determine the treatment plan of the target region to be treated according to the adjusted target point parameter combination.

[0217] The above describes the solutions provided by the embodiments of the present application mainly from the perspective of the computer system. To implement the above functions, it contains the hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0218] The embodiments of the present application can divide the functional modules according to the above computer system, for example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or software functional module. Optionally, the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, there can be another division manner.

[0219] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the treatment plan generation method provided by the present disclosure.

[0220] According to the embodiments of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable an electronic device to execute the treatment plan generation method provided by the present disclosure.

[0221] According to the embodiments of the present disclosure, the present disclosure also provides a computer program product comprising a computer program, which, when executed by a processor, implements the deep reinforcement learning model training method for generating a treatment plan and the treatment plan generation method provided by the present disclosure.

[0222] In some embodiments, the electronic device can be the electronic device a2 or b2 shown in the above Figure 1 . Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device 600 can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0223] As shown in Figure 6 The electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 602 or a computer program loaded into a random access memory (RAM) 603 from a storage unit 608. Various programs and data required for the operation of the electronic device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0224] Various components in the electronic device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, and the like; an output unit 607, such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, a magneto-optical disk, and the like; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0225] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the treatment plan generation method. For example, in some embodiments, the treatment plan generation method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the treatment plan generation method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the treatment plan generation method by any other appropriate means, such as by means of firmware.

[0226] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0227] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0228] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory, read-only memory, erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0229] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device, e.g., a Cathode Ray Tube (CRT) or Liquid Crystal Display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0230] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0231] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between a client and a server is one of client-server. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0232] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure are achieved, and the present disclosure is not limited herein.

[0233] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for training a deep reinforcement learning model for generating treatment plans, wherein the deep reinforcement learning model is configured to include an action selection network and an evaluation network, characterized in that, The method includes: Acquire initial dose distribution data of the target area; Based on the initial dose distribution data of the target area and the current strategy data of the action selection network and the evaluation network, multiple sub-threads are used in parallel to determine the output data of the multiple sub-threads, including: Perform the following operations using each of the plurality of child threads: Based on the current dose distribution status data of the current target area and the current strategy data of the action selection network and the evaluation network, the target parameters corresponding to the current dose distribution status data of the current target area and the prediction value corresponding to the target parameters are determined. The dose distribution of the current target area is determined based on the target parameters corresponding to the current dose distribution status data of the current target area; Based on the dose distribution of the current target area, determine the actual reward value corresponding to the target parameters; When the dose distribution in the current target area does not meet the preset prescription dose, and the number of target points in the current target area is less than the preset maximum number of target points, the current dose distribution status data of the current target area is updated based on the dose distribution of the current target area. When the dose distribution in the current target area meets the preset prescription dose, and / or the number of target points in the current target area is equal to the preset maximum number of target points, the final dose distribution in the current target area and multiple dose distribution state data of the current target area, multiple target point parameters corresponding one-to-one with the multiple dose distribution state data, multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one with the multiple target parameters are determined. Based on the final dose distribution of the current target area and the multiple predicted values ​​corresponding one-to-one with the multiple target parameters, the actual reward value and predicted value of the current target area are determined. When the current target region is not the last target region, update the current target region and determine the current dose distribution status data of the current target region based on the final dose distribution of the previous target region. When the current target area is the last target area, the actual reward value and the predicted value of the target target area are determined based on the final dose distribution of each of the multiple target areas and the predicted value of each of the multiple target areas; Based on the output data of each of the plurality of sub-threads, the current policy data of the action selection network and the evaluation network are updated sequentially to complete the current training of the deep reinforcement learning model; The above training process is iterated until the deep reinforcement learning model is trained a preset number of times, and the trained deep reinforcement learning model is obtained. The output data of the sub-thread includes: the final dose distribution of the target area and multiple dose distribution state data, multiple target parameters corresponding to the multiple dose distribution state data, multiple predicted values ​​and multiple actual reward values ​​corresponding to the multiple target parameters, and the predicted value and actual reward value of the target area.

2. The method according to claim 1, characterized in that, The step of updating the current policy data of the action selection network and the evaluation network based on the output data of each of the plurality of sub-threads includes: Perform the following operations on the output data of each of the plurality of sub-threads in sequence: When the final dose distribution of the target area obtained in the current training of the sub-thread meets the preset prescription dose, it is determined whether the actual reward value of the target area obtained in the current training of the sub-thread is greater than the dynamic reward threshold; the dynamic reward threshold is the actual reward value of the target area used when the current policy data of the action selection network and the evaluation network were last updated. When the actual reward value of the target region obtained in the current training of the sub-thread is greater than the dynamic reward threshold, the loss value of the target region corresponding to the current training of the sub-thread is determined based on the actual reward value and the predicted value of the target region obtained in the current training of the sub-thread. When the loss value of the target region corresponding to the current training iteration of the sub-thread is less than the dynamic loss value, the current policy data of the action selection network and the evaluation network are updated based on the multiple dose distribution state data obtained from the current training iteration of the sub-thread, the multiple target point parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and multiple actual reward values ​​corresponding to the multiple target point parameters, and the actual reward value of the target region; the dynamic loss value is the loss value of the target region used when the current policy data of the action selection network and the evaluation network were last updated.

3. The method according to claim 2, characterized in that, The process of updating the current policy data of the action selection network and the evaluation network based on the multiple dose distribution state data obtained in the current training of the sub-thread, the multiple target parameters corresponding one-to-one with the multiple dose distribution state data, the multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one with the multiple target parameters, and the actual reward value of the target area includes: Based on the multiple dose distribution state data obtained in the current training of the sub-thread, the multiple target parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value of the target area, the current policy data of the action selection network and the evaluation network are updated using the proximal policy optimization algorithm.

4. The method according to any one of claims 1-3, characterized in that, The target parameters include one of the following: target size, target location, and target weight.

5. The method according to any one of claims 1-3, characterized in that, The dose distribution status data includes: mask data of the target area, dose distribution of the target area, volume of the region with insufficient dose distribution in the target area, and volume of the region with overflow dose in the target area.

6. A method for generating a treatment plan, characterized in that, The method includes: Acquire image and contour data of the target area to be treated; Based on the image data and contour data, determine the dose distribution status data of the target area to be treated; The dose distribution data of the target area to be treated is input into a deep reinforcement learning model to obtain the target parameters of the target area to be treated; the deep reinforcement learning model is trained by the method according to any one of claims 1-5; A treatment plan for the target area to be treated is generated based on the target parameters.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1 to 5, or the method of claim 6.

8. A non-volatile readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-5, or the method described in claim 6.

Images