Model training method, treatment planning method, equipment and medium

Abstract

The invention provides a model training method, a treatment planning method, equipment and a medium, and relates to the technical field of medical treatment, in particular to the technical field of radiotherapy. The deep reinforcement learning model is configured to comprise an action selection network and an evaluation network. The method comprises the following steps: acquiring initial dose distribution state data of a target region; determining output data of a plurality of sub-threads in parallel by using the plurality of sub-threads based on the initial dose distribution state data of the target region and the current strategy data of the action selection network and the evaluation network; updating the current strategy data of the action selection network and the evaluation network based on the output data of each sub-thread in the plurality of sub-threads in sequence, and completing the current training of the deep reinforcement learning model; and iterating the training process until the number of times of training the deep reinforcement learning model reaches a preset number of times, and obtaining the trained deep reinforcement learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of medical technology, in particular to the field of radiotherapy technology, and specifically to a model training method, a treatment planning method, a device and a medium. Background Art

[0002] In the field of medical technology, radiotherapy (abbreviated as radiotherapy) is one of the important means of treating tumors. Before using radiotherapy to treat a subject, a treatment plan is usually required to be designed in advance.

[0003] At present, the design of treatment plans usually relies on physiotherapists (manuals) to make repeated adjustments based on their own experience and professional skills in order to design a more reasonable treatment plan. Therefore, the existing treatment planning method requires high clinical experience of physiotherapists and requires continuous trial and error adjustments, which is time-consuming, labor-intensive and inefficient. Summary of the invention

[0004] The present invention provides a model training method, a treatment planning method, a device and a medium.

[0005] In a first aspect, a deep reinforcement learning model training method for generating a treatment plan is provided, wherein the deep reinforcement learning model is configured to include an action selection network and an evaluation network, and the method includes: obtaining initial dose distribution state data of a target area; based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, using multiple sub-threads to determine output data of multiple sub-threads in parallel; updating the current strategy data of the action selection network and the evaluation network based on the output data of each of the multiple sub-threads in turn, to complete the current training of the deep reinforcement learning model; iterating the above training process until the number of training times for the deep reinforcement learning model reaches a preset number, thereby obtaining a trained deep reinforcement learning model; wherein the output data of the sub-threads include: a final dose distribution and multiple dose distribution state data of the target area, multiple target parameters corresponding one-to-one to the multiple dose distribution state data, multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and predicted value and actual reward value of the target area.

[0006] In some embodiments, when the target area includes only one target area, based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, multiple sub-threads are used to determine the output data of multiple sub-threads in parallel, including: using each of the multiple sub-threads to perform the following operations: based on the current dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, determine the target parameters corresponding to the current dose distribution state data of the target area and the predicted values ​​corresponding to the target parameters; based on the target parameters corresponding to the current dose distribution state data of the target area, determine the dose distribution of the target area; based on the dose distribution of the target area, determine the actual reward value corresponding to the target parameters; When the dose distribution of the target target area does not meet the preset prescription dose and the number of target points in the target target area is less than the preset maximum number of target points, the current dose distribution status data of the target target area is updated based on the dose distribution of the target target area; when the dose distribution of the target target area meets the preset prescription dose and / or the number of target points in the target target area is equal to the preset maximum number of target points, the final dose distribution of the target target area and multiple dose distribution status data of the target target area, multiple target parameters corresponding one-to-one to the multiple dose distribution status data, multiple predicted values ​​corresponding one-to-one to the multiple target parameters, and multiple actual reward values ​​are determined; based on the final dose distribution of the target target area and the multiple predicted values ​​corresponding one-to-one to the multiple target parameters, the actual reward value and the predicted value of the target target area are determined.

[0007] In some embodiments, when the target target area includes multiple target areas, based on the initial dose distribution state data of the target target area and the current strategy data of the action selection network and the evaluation network, multiple sub-threads are used to determine the output data of multiple sub-threads in parallel, including: using each of the multiple sub-threads to perform the following operations: based on the current dose distribution state data of the current target area and the current strategy data of the action selection network and the evaluation network, determine the target parameters corresponding to the current dose distribution state data of the current target area and the predicted value corresponding to the target parameters; based on the target parameters corresponding to the current dose distribution state data of the current target area, determine the dose distribution of the current target area; based on the dose distribution of the current target area, determine the actual reward value corresponding to the target parameters; when the dose distribution of the current target area does not meet the preset prescription dose, and the number of targets in the current target area is less than the preset maximum number of targets, based on the dose distribution of the current target area, update the current dose of the current target area; dose distribution status data; when the dose distribution of the current target area satisfies the preset prescription dose, and / or the number of target points in the current target area is equal to the preset maximum number of target points, determine the final dose distribution of the current target area and multiple dose distribution status data of the current target area, multiple target parameters corresponding to the multiple dose distribution status data, multiple predicted values ​​corresponding to the multiple target parameters, and multiple actual reward values; based on the final dose distribution of the current target area and the multiple predicted values ​​corresponding to the multiple target parameters, determine the actual reward value and predicted value of the current target area; when the current target area is not the last target area, update the current target area, and determine the current dose distribution status data of the current target area based on the final dose distribution of the previous target area of ​​the current target area; when the current target area is the last target area, determine the actual reward value and predicted value of the target target area based on the final dose distribution of each of the multiple target areas and the predicted value of each of the multiple target areas.

[0008] In some embodiments, based on the output data of each of the multiple sub-threads, the current strategy data of the action selection network and the evaluation network are updated, including: performing the following operations on the output data of each of the multiple sub-threads in turn: when the final dose distribution of the target target area obtained by the current training of the sub-thread meets the preset prescription dose, determining whether the actual reward value of the target target area obtained by the current training of the sub-thread is greater than the dynamic reward threshold; the dynamic reward threshold is the actual reward value of the target target area used when the current strategy data of the action selection network and the evaluation network were last updated; when the actual reward value of the target target area obtained by the current training of the sub-thread is greater than the dynamic reward threshold, based on The actual reward value and predicted value of the target target area obtained by the current training of the sub-thread are used to determine the loss value of the target target area corresponding to the current training of the sub-thread; when the loss value of the target target area corresponding to the current training of the sub-thread is less than the dynamic loss value, the current policy data of the action selection network and the evaluation network are updated based on multiple dose distribution state data obtained by the current training of the sub-thread, multiple target parameters corresponding one-to-one to the multiple dose distribution state data, multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and the actual reward value of the target target area; the dynamic loss value is the loss value of the target target area used when the current policy data of the action selection network and the evaluation network were last updated.

[0009] In some embodiments, based on multiple dose distribution state data obtained from the current training of the sub-thread, multiple target parameters corresponding one-to-one to the multiple dose distribution state data, multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and the actual reward value of the target target area, the current strategy data of the action selection network and the evaluation network are updated, including: based on the multiple dose distribution state data obtained from the current training of the sub-thread, multiple target parameters corresponding one-to-one to the multiple dose distribution state data, multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and the actual reward value of the target target area, the current strategy data of the action selection network and the evaluation network are updated using a proximal strategy optimization algorithm.

[0010] In some embodiments, the target parameters include one of the following target parameters: target size, target position, and target weight.

[0011] In some embodiments, the dose distribution status data includes: mask data of the target area, dose distribution of the target area, volume of an area with insufficient dose distribution in the target area, and volume of an area with overflow dose in the target area.

[0012] In a second aspect, a treatment planning method is provided, comprising: acquiring image data and contour data of a target area to be treated; determining dose distribution state data of the target area to be treated based on the image data and contour data; inputting the dose distribution state data of the target area to be treated into a deep reinforcement learning model to obtain target parameters of the target area to be treated; the deep reinforcement learning model is trained according to any one of the methods according to the first aspect of claim to generate a treatment plan for the target area to be treated based on the target parameters.

[0013] In a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods of the first aspect or the second aspect mentioned above.

[0014] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein a computer program is stored on the storage medium, and the computer program is used to execute any one of the methods of the first aspect or the second aspect above.

[0015] In the technical solution provided by the present disclosure, the present disclosure trains the deep reinforcement learning model used to generate the treatment plan in parallel through multiple sub-threads. Since multiple sub-threads can be tried and failed in parallel, the speed of model training is improved. Since the deep reinforcement learning model conforms to the characteristics of the gamma knife treatment plan design, the processor in the computer system can implement a repeated trial and error process. Therefore, the deep reinforcement learning model can automatically generate target parameters with better effects, thereby generating a treatment plan with better effects based on the target parameters, reducing the dependence on clinical experience, and improving the effect of the treatment plan without manually setting the target parameters, thereby improving the efficiency of the physicist in designing the treatment plan.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0018] Figure 1 It is a schematic diagram of an implementation environment of a deep reinforcement learning model training method and a treatment plan method for generating a treatment plan shown in an embodiment of the present disclosure;

[0019] Figure 2 It is a flowchart of a deep reinforcement learning model training method shown in an embodiment of the present disclosure;

[0020] Figure 3 It is a flowchart of another deep reinforcement learning model training method shown in an embodiment of the present disclosure;

[0021] Figure 4 It is a flowchart of another deep reinforcement learning model training method shown in an embodiment of the present disclosure;

[0022] Figure 5 is a flowchart of a treatment planning method shown in an embodiment of the present disclosure;

[0023] Figure 6 It is a block diagram of an electronic device used to implement the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] Before introducing the present disclosure in detail, application scenarios involved in the embodiments of the present disclosure are described first.

[0027] The deep reinforcement learning model training method, treatment planning method, device and medium for generating treatment plans provided by the embodiments of the present disclosure can be applied in the field of medical technology, specifically in the scenario of clinical radiotherapy (abbreviated as radiotherapy), such as the scenario of gamma knife treatment plan formulation.

[0028] Before using radiotherapy to treat a subject, a treatment plan is usually required to be designed in advance.

[0029] At present, the design of treatment plans usually relies on physiotherapists (manuals) to make repeated adjustments based on their own experience and professional skills in order to design a more reasonable treatment plan. Therefore, the existing treatment planning method requires high clinical experience of physiotherapists and requires continuous trial and error adjustments, which is time-consuming, labor-intensive and inefficient.

[0030] Based on this, the embodiments of the present disclosure provide a deep reinforcement learning model training method, a treatment plan method, a device and a medium for generating a treatment plan, wherein the deep reinforcement learning model includes an action selection network and an evaluation network.

[0031] The method includes: first, obtaining the initial dose distribution state data of the target area. Then, based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, multiple sub-threads are used to determine the output data of multiple sub-threads in parallel. Subsequently, based on the output data of each sub-thread in the multiple sub-threads, the current strategy data of the action selection network and the evaluation network are updated to complete the current training of the deep reinforcement learning model. The above training process is iterated until the number of training times of the deep reinforcement learning model reaches a preset number, and a trained deep reinforcement learning model is obtained.

[0032] Among them, the output data of the sub-thread includes: the final dose distribution of the target area and multiple dose distribution status data, multiple target parameters corresponding to the multiple dose distribution status data, multiple predicted values ​​and multiple actual reward values ​​corresponding to the multiple target parameters, and the predicted value and actual reward value of the target area.

[0033] As can be seen from the above, the present disclosure trains the deep reinforcement learning model for generating treatment plans in parallel through multiple sub-threads. Since multiple sub-threads can be tried and failed in parallel, the speed of model training is improved. Since the deep reinforcement learning model conforms to the characteristics of gamma knife treatment plan design, it allows the processor in the computer system to implement a repeated trial and error process. Therefore, the deep reinforcement learning model can automatically generate target parameters with better effects, thereby generating a treatment plan with better effects based on the target parameters, reducing the dependence on clinical experience, and improving the effect of the treatment plan without manually setting the target parameters, thereby improving the efficiency of the physicist in designing the treatment plan.

[0034] The method provided in this application mainly involves two methods: a model training method and a treatment planning method. The implementation environment of the above two methods is first introduced below.

[0035] Figure 1 (a) is a schematic diagram of an implementation environment of a model training method shown in an embodiment of the present disclosure. Figure 1 In (a), the implementation environment includes an image scanning device a1 and an electronic device a2.

[0036] The image scanning device a1 is a device for scanning and displaying the tumor site and surrounding normal tissue of the subject to be treated. In some embodiments, the image scanning device a1 can be at least one of a computed tomography (CT) device, an emission computed tomography (ECT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasound examination device.

[0037] In the disclosed embodiment, the image scanning device a1 is used to acquire a medical scan image (such as image data of a target area, etc.) and upload the medical scan image to an electronic device a2 so that the electronic device a2 can execute a subsequent model training method based on the medical scan image of the object to be treated.

[0038] The electronic device a2 is a device for training a deep reinforcement learning model. In some embodiments, the electronic device a2 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer.

[0039] Furthermore, in some embodiments, the electronic device a2 may run a computer system including a processor for implementing a training method for a deep reinforcement learning model.

[0040] In some embodiments, the implementation environment further includes a server a3. In some embodiments, the server a3 is used to provide background communication services for the image scanning device a1 and the electronic device a2.

[0041] Among them, server a3 can be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or at least one of cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data or artificial intelligence platforms, etc., which is not limited in the embodiments of the present disclosure. In some embodiments, the number of the above-mentioned servers a3 can be more or less, which is not limited in the embodiments of the present disclosure. Of course, server a3 can also include other functions to provide more comprehensive and diversified services.

[0042] Figure 1 (b) is a schematic diagram of an implementation environment of a treatment planning method shown in an embodiment of the present disclosure. Figure 1In (b), the implementation environment includes an image scanning device b1, an electronic device b2 and a radiotherapy device b3.

[0043] Among them, image scanning equipment b1 and Figure 1 The device form of the image scanning device a1 in (a) is similar, and you can refer to the description of the image scanning device a1, which will not be repeated here.

[0044] In the disclosed embodiment, the image scanning device b1 is used to obtain a medical scanning image of the object to be treated (such as image data of the target area to be treated, etc.), and upload the medical scanning image of the object to be treated to the electronic device b2, so that the electronic device b2 executes a method for generating a subsequent treatment plan based on the medical scanning image of the object to be treated.

[0045] The electronic device b2 is a device for applying a deep reinforcement learning model to generate a treatment plan.

[0046] Electronic equipment b2 and Figure 1 The device form of the electronic device a2 in (a) is similar, and you can refer to the description of the electronic device a2, which will not be repeated here.

[0047] Furthermore, in some embodiments, the electronic device b2 may run a computer system including a processor for implementing a method for generating a treatment plan using a deep reinforcement learning model.

[0048] The radiotherapy device b3 is a device used for radiotherapy. In some embodiments, the radiotherapy device b3 can be at least one of a gamma knife, a linear accelerator, a neutron knife, an X-ray therapy machine and the like.

[0049] In the embodiment of the present disclosure, the radiotherapy device b3 is used to receive the treatment plan from the electronic device b2, and perform radiotherapy on the object to be treated according to the treatment plan.

[0050] In some embodiments, the implementation environment further includes a server b4. In some embodiments, the server b4 is used to provide background communication services for the above-mentioned image scanning device b1, electronic device b2 and radiotherapy device b3.

[0051] Server b4 and Figure 1 In (a), the device form of server a3 is similar, and you can refer to the description of server a3, which will not be repeated here.

[0052] The following is based on Figure 1 The implementation environment shown in (a) is first introduced to the model training method provided by the embodiment of the present disclosure.

[0053] The deep reinforcement learning model training method for generating a treatment plan provided in the embodiments of the present disclosure can be applied to Figure 1 The electronic device a2 in (a) may be capable of running a computer system, wherein the computer system includes a processor for implementing a deep reinforcement learning model training method for generating a treatment plan.

[0054] The deep reinforcement learning model includes: an action selection network and an evaluation network.

[0055] The action selection network includes action decision parameters. The action decision parameters are used to characterize the probability of the selectable target parameters being selected under a specific dose distribution state. The higher the probability of being selected, the better the target parameters are under the specific dose distribution state.

[0056] For example, when the target parameter is the target size, under a certain dose distribution state, the target sizes that can be selected are: and The corresponding selection probabilities are: 10%, 20% and 70%. Then, under this dose distribution state, the optimal target size is

[0057] The evaluation network includes prediction decision parameters. The prediction decision parameters are used to determine the prediction value corresponding to each target parameter. The prediction value is used to indicate the quality of a target parameter selection under a specific dose distribution state.

[0058] Figure 2 FIG. 1 is a flow chart of a deep reinforcement learning model training method for generating a treatment plan according to an embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0059] S201, obtaining initial dose distribution state data of the target area.

[0060] Wherein, the target area includes the area to be treated of the object to be treated. The target area may include one or more target areas. When the target area includes multiple target areas, the multiple target areas may be target areas of different shapes.

[0061] Optionally, the object to be treated may be a phantom, a human body, an animal, etc. The target volume may be a tumor region of the object to be treated, etc. The target volume may also be referred to as a planning target volume (Planning Target Volume, PTV).

[0062] For example, Figure 2 As shown, the image data and contour data of different target areas may include: image data and contour data of target area 1, image data and contour data of target area 2, etc.

[0063] Exemplary, combined Figure 1 In (a), the image scanning device a1 can collect image data of the object to be treated and transmit the image data to the electronic device a2.

[0064] Optionally, after receiving the image data, the electronic device a2 may perform contour outlining on the image data to obtain contours of different target areas of the object to be treated in the image data, that is, contour data.

[0065] Exemplarily, the contour data may be the contour of a tumor region.

[0066] Optionally, the electronic device a2 may also be connected to a third-party software program. The third-party software program is used to outline the image data to obtain outline data.

[0067] Optionally, when outlining the image data, the physician may perform an outlining operation on the electronic device a2. The electronic device a2 may obtain the outline data in response to the physician's outlining operation.

[0068] Alternatively, the electronic device a2 may also automatically outline the contours of different target areas of the object to be treated in the image data using contour drawing software.

[0069] After determining the image data and contour data of the target area, the processor in the electronic device a2 may determine the initial dose distribution state data of the target area based on the image data and contour data of the target area.

[0070] The initial dose distribution state data is used to represent the dose distribution state data of the target area when no target point is placed.

[0071] The dose distribution state data includes: mask data of the target area, dose distribution of the target area, volume of an area with insufficient dose distribution in the target area, and volume of an area with overflow dose in the target area.

[0072] Since the initial dose distribution state data is used to represent the dose distribution state data of the target target area when no target point is placed, the dose distribution, the volume of the insufficient dose distribution area, and the volume of the overflow dose area in the initial dose distribution state data are all 0. In this case, the processor in the electronic device a2 only needs to determine the mask data of the target target area according to the image data and contour data of the target target area to determine the initial dose distribution state data of the target target area.

[0073] Specifically, the image data of the target area may include contour data of the target area and image data of an organ at risk (OAR).

[0074] Among them, organs at risk refer to normal organs around the target area, that is, organs that have not developed lesions.

[0075] The processor in the electronic device a2 can process the image data and contour data of the target area to generate mask data of the target area. In the process of generating the mask data, the processor in the electronic device a2 can perform different operations on the area corresponding to the target area and the area corresponding to the OAR based on the contour data, thereby distinguishing the area corresponding to the target area and the area corresponding to the OAR.

[0076] At the same time, the subsequent target position search space (i.e., the placement position of the target) can be limited to the area corresponding to the target target area, so as to avoid the target position falling into the OAR area or the area corresponding to other tissues, thereby avoiding treatment damage to the OAR area or the area corresponding to other tissues. In addition, limiting the subsequent target position search space to the area corresponding to the target target area can reduce the search space for the specific shape of the target target area, thereby improving the speed of model training and generating treatment plans.

[0077] Exemplarily, the processor in the electronic device a2 can construct a three-dimensional matrix of uniform size, wherein the image parameter of the area corresponding to the target area can be set to 1, the image parameter of the area corresponding to the OAR can be set to -1, and the image parameter of other tissues can be set to 0. In this way, mask data can be obtained to distinguish the area corresponding to the target area and the area corresponding to the OAR.

[0078] S202: Based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, use multiple sub-threads to determine the output data of multiple sub-threads in parallel.

[0079] For example, Figure 2 As shown, the multiple sub-threads may include: sub-thread 1, sub-thread 2, etc.

[0080] The output data of the sub-thread includes: the final dose distribution of the target area and multiple dose distribution status data, multiple target parameters corresponding to the multiple dose distribution status data, multiple predicted values ​​and multiple actual reward values ​​corresponding to the multiple target parameters, and the predicted value and actual reward value of the target area.

[0081] For example, after determining the initial dose distribution state data of the target area, the processor of the electronic device a2 can input the initial dose distribution state data into each sub-thread, so that each sub-thread can formulate a complete treatment plan for the target area based on the initial dose distribution state data and the current strategy data of the action selection network and the evaluation network, and then determine the output data of each sub-thread.

[0082] When the deep reinforcement learning model is first trained, the current policy data of the action selection network and the evaluation network are the initial policy data of the action selection network and the evaluation network. Each sub-thread can formulate a complete treatment plan for the target area based on the initial policy data of the action selection network and the evaluation network and the initial dose distribution state data of the target area, thereby determining the output data of the first sub-thread of each sub-thread.

[0083] When the deep reinforcement learning model is not trained for the first time, each sub-thread can formulate a complete treatment plan for the target area based on the current strategy data and initial dose distribution state data of the action selection network and the evaluation network, and then determine the output data of the current sub-sub-thread of each sub-thread.

[0084] Since the process of generating a Gamma Knife treatment plan is a process of sequentially arranging targets, after the arrangement of one target is completed, the dose distribution state of the target area will change. Therefore, in the process of generating a complete treatment plan for the target area, the dose distribution state of the target area will change as the targets are placed one by one.

[0085] Therefore, in the embodiment of the present application, for each sub-thread, the output data of the sub-thread can be obtained only when a complete treatment plan is generated for the target area.

[0086] Specifically, after obtaining the initial dose distribution state data, the processor in the electronic device a2 can input the initial dose distribution state data into multiple sub-threads respectively. Since each sub-thread can share the action selection network, each sub-thread can output the first type of target parameters selected for the first target based on the initial strategy data of the action selection network and the initial dose distribution state data of the target target area. Then, each sub-thread can call different algorithms to determine other types of target parameters, thereby obtaining a target parameter combination for placing the first target. The target parameter combination includes the first type of target parameters selected for the first target output by the action selection network, and other types of target parameters selected for the first target determined by calling different algorithms.

[0087] Since the deep reinforcement learning model also includes an evaluation network, and each sub-thread can also share the evaluation network, while determining the target parameters of the first type of the first target, the evaluation network can also determine the prediction value corresponding to the target parameters of the first type of the first target based on its initial strategy data and the initial dose distribution state data of the target area.

[0088] After the first target is placed, the dose distribution state of the target area changes. At this time, each sub-thread can also determine the dose distribution of the target area based on the target parameter combination of the first target, and determine the actual reward value corresponding to the first type of target parameter of the first target based on the dose distribution of the target area.

[0089] Then, each sub-thread can update the dose distribution state data of the target target area based on the dose distribution of the target target area, and repeat the above target point placement process according to the updated dose distribution state data of the target target area until the generation of the complete treatment plan of the target target area is completed, and the final dose distribution of the target target area and multiple dose distribution state data of the target target area, multiple target point parameters corresponding to the multiple dose distribution state data, multiple predicted values ​​corresponding to the multiple target parameters, multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value and predicted value of the target target area are obtained. In this way, each sub-thread can obtain its output data.

[0090] S203. Based on the output data of each sub-thread in the multiple sub-threads, update the current strategy data of the action selection network and the evaluation network in turn to complete the current training of the deep reinforcement learning model.

[0091] Specifically, after each sub-thread obtains its output data, it indicates that the treatment plan for the target area of ​​each sub-thread has been completed. In this case, the current strategy data of the action selection network and the evaluation network can be updated based on the output data of each sub-thread in the multiple sub-threads in turn to complete the current training of the deep reinforcement learning model.

[0092] It should be noted that the training process of the deep reinforcement learning model is equivalent to a trial-and-error process for generating a treatment plan, that is, the treatment plan generated by each sub-thread is not necessarily a good plan. In other words, the training process of the deep reinforcement learning model is equivalent to a trial-and-error process for generating a treatment plan. Therefore, there may be a situation where the output data obtained by one or more sub-threads is discarded because it does not meet the update conditions.

[0093] Therefore, in an embodiment of the present application, when the output data of multiple sub-threads do not meet the update conditions, there is no need to update the current policy data of the action selection network and the evaluation network. That is, completing the training of the deep reinforcement learning model once does not mean that the current policy data of the action selection network and the evaluation network will be updated.

[0094] Of course, when the output data of multiple sub-threads all meet the update conditions, completing a training of the deep reinforcement learning model means that the current policy data of the action selection network and the evaluation network have been updated multiple times (i.e., the same number of times as the number of sub-threads).

[0095] Correspondingly, when the output data of at least one sub-thread (i.e., some of the sub-threads among the multiple sub-threads) meets the update conditions, the training of the deep reinforcement learning model is completed once, which means that the current policy data of the action selection network and the evaluation network are updated at least once (i.e., the same number as the number of at least one sub-thread).

[0096] When updating the current policy data of the action selection network and the evaluation network, multiple sub-threads can package the output data of the multiple sub-threads into a message queue (buffer), and transmit it back to the deep reinforcement learning model through a communication pipe (pipe), so that the deep reinforcement learning model can update the current policy data of the action selection network and the evaluation network, thereby completing the current training of the deep reinforcement learning model.

[0097] S204. Iterate the above training process until the deep reinforcement learning model is trained a preset number of times, thereby obtaining a trained deep reinforcement learning model.

[0098] In this way, the above-mentioned training process of the deep reinforcement learning model (i.e., S201 to S203) is iterated until the deep reinforcement learning model is trained a preset number of times, thereby obtaining a trained deep reinforcement learning model.

[0099] Optionally, the electronic device may also include an experience memory unit (buffer_memory). The experience memory unit is used to store the final dose distribution and multiple dose distribution state data of the target target area at each update stage during the current strategy data update process, multiple target parameters corresponding to the multiple dose distribution state data, multiple predicted values ​​and multiple actual reward values ​​corresponding to the multiple target parameters, and the predicted value and actual reward value of the target target area (i.e., the output data of each sub-thread). Subsequently, through the corresponding relationship stored in the experience memory unit, the various required data determined for each training can be found, and then the model training can be carried out.

[0100] In some embodiments, the target area may include only one target area (also referred to as a single target area) or multiple target areas (also referred to as multiple target areas). In this case, the present application provides two embodiments of the model training method: an embodiment of the model training method implemented in a single target area scenario and an embodiment of the model training method implemented in a multiple target area scenario.

[0101] Combine the following Figure 3 and Figure 4 , the two model training methods provided in the embodiments of the present application are introduced in detail. Figure 3 An embodiment of the model training method provided by the present application in a single target area scenario is shown, such as Figure 3As shown in the figure, the model training method implemented in the single target area scenario specifically includes:

[0102] A1. Obtain the initial dose distribution status data of the target area.

[0103] For the specific implementation process of this step, please refer to the description of S201, which will not be repeated here.

[0104] After obtaining the initial dose distribution state data of the target area, each of the multiple sub-threads can be used to execute the following A2-A7 based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network to obtain the output data of the multiple sub-threads.

[0105] A2. Based on the current dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, determine the target parameters corresponding to the current dose distribution state data of the target area and the prediction value corresponding to the target parameters.

[0106] Since the process of generating a Gamma Knife treatment plan is a process of sequentially arranging targets, after the arrangement of one target is completed, the dose distribution state of the target area will change. Therefore, in the process of generating a complete treatment plan for the target area, the current dose distribution state of the target area will change as the targets are placed one by one.

[0107] Therefore, after obtaining the current dose distribution state data of the target area (for the first target, the current dose distribution state data of the target area is: the initial dose distribution state data of the target area), since the deep reinforcement learning model includes an action selection network, the action selection network can make an action selection decision based on its own current strategy data and the current dose distribution state data of the target area, that is, output action decision parameters to the child thread; after receiving the action decision parameters sent by the action selection network, the child thread can execute the determination of the target parameters according to the action decision parameters.

[0108] Optionally, the target parameter includes one of the following target parameters: target size, target position, and target weight. The target weight is used to determine the dose of the target.

[0109] Exemplarily, when the target parameter is the target size, the child thread can determine the target size of the current target according to the action decision parameters sent by the action selection network, so that the target parameter corresponding to the current dose distribution state data of the target area can be obtained, that is, the target size.

[0110] Secondly, since the deep reinforcement learning model also includes an evaluation network, while the action selection network is selecting target parameters, the evaluation network can make value predictions for the target parameter selection action based on its own current strategy data and the current dose distribution status data of the target area, that is, output the predicted decision parameters to the child thread; after receiving the predicted decision parameters sent by the evaluation network, the child thread can determine the predicted value corresponding to the target parameters based on the predicted decision parameters.

[0111] A3. Determine the dose distribution of the target area based on the target point parameters corresponding to the current dose distribution state data of the target area.

[0112] The dose distribution of the target area refers to the dose distribution of the target area after the current target point (ie, the target point corresponding to the target point parameters determined in step A2) is placed in the target area.

[0113] Specifically, after determining the target parameters (e.g., target size) corresponding to the current dose distribution state data of the target area, the child thread can call the shape matching algorithm to match the selected target size with the contour of the target area, and determine the best placement position of the current target in the target area, that is, the target position of the current target; then, the child thread can determine the weight of the current target according to the size, position, and preset prescription dose of the current target. Optionally, after determining the target position and target size, the target weight can also be set to 1 by default.

[0114] It is understandable that when the target parameter is the target position or target weight, the sub-thread can call other algorithms to determine other target parameters included in the treatment plan (for example: target size and target weight, target size and target position).

[0115] After the target size, target position and target weight are determined, the sub-thread can perform the target placement action in the target area according to the target size, target position and target weight. In this way, after the target is placed, the dose distribution of the target area will change. In this case, the sub-thread can determine the dose distribution of the target area.

[0116] A4. Based on the dose distribution of the target area, determine the actual reward value corresponding to the target parameters.

[0117] After determining the dose distribution of the target area, in order to obtain the deep reinforcement learning model for subsequent training, the sub-thread can determine the actual reward value corresponding to the target parameters (for example, target size) under the current dose distribution state based on the dose distribution of the target area.

[0118] Since the dose distribution of the target target area is determined after the current target is placed, the dose distribution of the target target area can truly reflect the contribution of the target parameter selection action (e.g., target size) in step A2 to the dose distribution of the target target area. Therefore, based on the dose distribution of the target target area, the actual reward value corresponding to the target parameter (e.g., target size) under the current dose distribution state can be obtained.

[0119] For example, the actual reward values ​​corresponding to the target parameters include: positive reward and negative reward. The increase in the dose coverage and conformality of the target area before and after the current target is executed is multiplied by the corresponding weight as a positive reward; the increase in the dose overflow rate of the target area before and after the current target is executed is multiplied by the corresponding weight as a negative reward.

[0120] A5. When the dose distribution of the target area does not meet the preset prescription dose and the number of target points in the target area is less than the preset maximum number of target points, the current dose distribution status data of the target area is updated based on the dose distribution of the target area.

[0121] When the dose distribution of the target target area does not meet the preset prescription dose, and the number of target points in the target target area is less than the preset maximum number of target points, it means that the treatment plan of the target target area has not been completed. In this case, the subthread can determine the current dose distribution state data of the target target area according to the dose distribution of the target target area, and update the current dose distribution state data of the target target area, that is, update the mask data of the target target area, the dose distribution of the target target area, the volume of the insufficient dose distribution area in the target target area, and the volume of the overflow dose area in the target target area.

[0122] Afterwards, the sub-thread repeats steps A2 to A4 according to the updated current dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network to determine the subsequent multiple target parameters, the one-to-one corresponding prediction values ​​of the multiple target parameters, and the actual reward value.

[0123] Optionally, when the sub-thread updates the current dose distribution state data of the target target area based on the dose distribution of the target target area, it may perform feature extraction on the dose distribution of the target target area, thereby obtaining updated current dose distribution state data.

[0124] A6. When the dose distribution of the target target area satisfies the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, determine the final dose distribution of the target target area and multiple dose distribution status data of the target target area, multiple target point parameters corresponding one-to-one to the multiple dose distribution status data, multiple predicted values ​​corresponding one-to-one to the multiple target parameters, and multiple actual reward values.

[0125] Specifically, when the dose distribution of the target target area meets the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, it means that the target target area has completed the treatment plan. In this case, the subthread can determine the final dose distribution of the target target area and multiple dose distribution state data of the target target area, multiple target point parameters corresponding to the multiple dose distribution state data, multiple predicted values ​​corresponding to the multiple target parameters, and multiple actual reward values.

[0126] A7. Based on the final dose distribution of the target area and multiple predicted values ​​corresponding to multiple target point parameters, the actual reward value and predicted value of the target area are determined.

[0127] Specifically, when the dose distribution of the target target area satisfies the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, the final dose distribution of the target target area and multiple prediction values ​​corresponding to multiple target parameters can be obtained. In this case, the subthread can determine the actual reward value of the target target area based on the final dose distribution, and determine the prediction value of the target target area based on the multiple prediction values ​​corresponding to multiple target parameters.

[0128] Optionally, after determining the final dose distribution of the target area, the dose coverage and conformality of the target area can be determined according to the final dose distribution of the target area, and the actual reward value of the target area can be determined according to the dose coverage and conformality of the target area.

[0129] Optionally, after determining a plurality of prediction values ​​corresponding one-to-one to a plurality of target parameters, the plurality of prediction values ​​corresponding one-to-one to the plurality of target parameters may be summed to obtain the prediction value of the target area.

[0130] At this point, after multiple sub-threads have completed executing A2-A7 in parallel, the output data of the multiple sub-threads have been determined.

[0131] After determining the output data of multiple sub-threads, since each sub-thread needs to determine whether the treatment plan it determines is a good treatment plan, that is, each sub-thread needs to determine whether its output data can be used to update the current policy data of the action selection network and the evaluation network, the following A8-A10 can be executed on the output data of each of the multiple sub-threads in turn, thereby completing the current training of the deep reinforcement learning model.

[0132] A8. When the final dose distribution of the target area obtained by the current training of the sub-thread meets the preset prescription dose, determine whether the actual reward value of the target area obtained by the current training of the sub-thread is greater than the dynamic reward threshold.

[0133] The dynamic reward threshold is the actual reward value of the target area used when the current strategy data of the action selection network and the evaluation network were last updated. The final dose distribution of the target area is the dose distribution corresponding to the treatment plan of the target area after the generation of the treatment plan of the target area is completed.

[0134] It should be noted that, since the treatment plan of the target target area is completed when the dose distribution of the target target area meets the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, therefore, when the treatment plan of the target target area is obtained when the dose distribution of the target target area meets the preset prescription dose, the sub-thread does not need to determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose, and can directly proceed to the subsequent judgment steps. Of course, the sub-thread can also re-determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose.

[0135] When the treatment plan of the target area is obtained when the number of target points in the target area is equal to the preset maximum number of target points, since the treatment plan only meets the preset maximum number of target points but does not necessarily meet the preset prescription dose, the sub-thread needs to determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose.

[0136] If the dose distribution corresponding to the treatment plan does not meet the preset prescription dose, it means that the treatment plan is not a good treatment plan. Therefore, the child thread can discard the output data of the child thread, that is, there is no need to update the current strategy data of the action selection network and the evaluation network according to the output data of the child thread.

[0137] Accordingly, if the dose distribution corresponding to the treatment plan satisfies the preset prescription dose, it means that the treatment plan can meet the requirement of the preset prescription dose. In this case, the child thread can determine whether the actual reward value of the target area is greater than the dynamic reward threshold.

[0138] Among them, the dynamic reward threshold is the actual reward value of the target area used when the current strategy data of the action selection network and the evaluation network were last updated.

[0139] By judging whether the actual reward value of the target area obtained in the current training is greater than the dynamic reward threshold, it can be determined whether the quality of the treatment plan is better than the quality of the treatment plan used when the current strategy data of the action selection network and the evaluation network were last updated.

[0140] When the actual reward value of the target area obtained in the current training is less than or equal to the dynamic reward threshold, it means that the quality of the treatment plan is inferior to or equal to the quality of the treatment plan used when the current policy data of the action selection network and the evaluation network were last updated. Therefore, the child thread can discard the output data of the child thread, that is, there is no need to update the current policy data of the action selection network and the evaluation network according to the output data of the child thread.

[0141] Correspondingly, when the actual reward value of the target area obtained by the current training is greater than the dynamic reward threshold, it means that the quality of the treatment plan is better than the quality of the treatment plan used when the current strategy data of the action selection network and the evaluation network were last updated, that is, the quality of the treatment plan is better. In this case, the child thread can execute A9.

[0142] A9. When the actual reward value of the target target area obtained by the current training of the sub-thread is greater than the dynamic reward threshold, the loss value of the target target area corresponding to the current training of the sub-thread is determined based on the actual reward value and predicted value of the target target area obtained by the current training of the sub-thread.

[0143] Specifically, when the actual reward value of the target area obtained in the current training is greater than the dynamic reward threshold, it indicates that the quality of the treatment plan is better. In this case, the child thread can determine the loss value of the target area corresponding to the current training based on the actual reward value and predicted value of the target area obtained in the current training.

[0144] Optionally, the child thread may use a preset loss function to determine the loss value of the target area corresponding to the current training according to the predicted value and actual reward value corresponding to the target area obtained from the current training.

[0145] Optionally, the preset loss function may be a relative advantage parameter or other general loss function, which is not limited in this application.

[0146] Next, the child thread can determine whether the loss value of the target area corresponding to the current training is less than the dynamic loss value.

[0147] Among them, the dynamic loss value is the loss value of the target area used when the current strategy data of the action selection network and the evaluation network were last updated.

[0148] If the loss value of the target area corresponding to the current training is greater than or equal to the loss value of the target area used in the last update of the current policy data of the action selection network and the evaluation network, it means that the quality estimation of the treatment plan by the deep reinforcement learning model is inaccurate. Therefore, the child thread can discard the output data of the child thread, that is, there is no need to update the current policy data of the action selection network and the evaluation network based on the output data of the child thread.

[0149] If the loss value of the target area corresponding to the current training is less than the loss value of the target area used in the last update of the current policy data of the action selection network and the evaluation network, it means that the deep reinforcement learning model accurately estimates the quality of the treatment plan. In this case, the child thread can execute A10.

[0150] A10. When the loss value of the target area corresponding to the current training of the sub-thread is less than the dynamic loss value, the current strategy data of the action selection network and the evaluation network are updated based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and the multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value of the target area.

[0151] In an achievable manner, the method for updating the current strategy data of the action selection network and the evaluation network based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and the multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value of the target area specifically includes:

[0152] Based on multiple actual reward values ​​corresponding one-to-one to multiple target parameters and the actual reward value of the target target area, the actual cumulative reward value of the multiple target parameters is determined, and based on the multiple dose distribution state data, the multiple target parameters corresponding one-to-one to the multiple dose distribution state data and the actual cumulative reward value of the multiple target parameters, the current strategy data of the action selection network is updated, and based on the multiple dose distribution state data, the multiple target parameters corresponding one-to-one to the multiple dose distribution state data and the multiple predicted values ​​corresponding one-to-one to the multiple target parameters, the current strategy data of the evaluation network is updated.

[0153] Specifically, the quality of the treatment plan for the target target area is the result of the combined action of multiple target parameter combinations selected by the action selection network that correspond one-to-one to multiple dose distribution state numbers. Under a certain dose distribution state of the target target area, the actual reward value of the target parameter selected by the action selection network is high, which does not mean that the quality of the treatment plan for the entire target target area will be good. If the quality of the treatment plan for the final target target area is also very good, the reference value of the action selection network for the selection of the target parameter will be very high. If the quality of the treatment plan for the final target target area is not good, the reference value of the action selection network for the selection of the target parameter will become smaller. Therefore, after obtaining multiple target parameters corresponding one-to-one to multiple dose distribution state data, the actual cumulative reward value of the multiple target parameters can also be determined based on the multiple actual reward values ​​corresponding one-to-one to the multiple target parameters and the actual reward value of the target target area.

[0154] The actual cumulative reward value may include the actual reward value of each target parameter selected by the action selection network and the delayed reward value after the entire treatment plan is completed. The delayed reward value is determined based on the actual reward value of the target target area. That is, after the treatment plan for the target target area is completed, the actual reward value of the target target area is determined according to the dose distribution of the treatment plan, and the actual reward value is distributed according to the weight to the target parameter selected by the action selection network in each target parameter combination constituting the treatment plan, thereby forming the delayed reward value of the target parameter selected by the action selection network; the actual reward value and the delayed reward value of each target parameter selected by the action selection network are accumulated to obtain the actual cumulative reward value of each target parameter selected by the action selection network.

[0155] When the actual cumulative reward value is high, it is said that the reference value of the target parameter selected by the action selection network in the dose distribution state is high. In this case, the current strategy data of the action selection network can be updated based on multiple dose distribution state data, multiple target parameters corresponding to the multiple dose distribution state data, and the actual cumulative reward values ​​of the multiple target parameters.

[0156] At the same time, since the loss value of the target area is obtained based on the predicted value of the target area, and the predicted value of the target area is the accumulation of the predicted values ​​of multiple target parameters selected by the action selection network, when the loss value of the target area is less than the dynamic loss value, it means that the predicted values ​​of the multiple target parameters selected by the action selection network have a higher reference value. In this case, the current strategy data of the evaluation network can be updated based on multiple dose distribution state data, multiple target parameters corresponding to the multiple dose distribution state data, and multiple predicted values ​​corresponding to the multiple target parameters.

[0157] In another achievable manner, the method for updating the current strategy data of the action selection network and the evaluation network based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and the multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value of the target area specifically includes:

[0158] Based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and the multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value of the target area, the proximal strategy optimization algorithm is used to update the current strategy data of the action selection network and the evaluation network.

[0159] The Proximal Policy Optimization (PPO) algorithm can be trained efficiently in a large-scale environment and can process continuous action space and high-dimensional state space. It conforms to the characteristics of Gamma Knife treatment plan design, can realize repeated trial and error process, reduces dependence on clinical experience, and improves the efficiency of physicists in designing treatment plans.

[0160] It is understandable that other algorithms, such as the Advantage Actor-Critic (A2C) algorithm and the Trust Region Policy Gradient Optimization (TPRO) algorithm, may also be used to update the current policy data of the action selection network and the evaluation network.

[0161] A11. Iterate the above training process until the deep reinforcement learning model is trained a preset number of times to obtain a trained deep reinforcement learning model.

[0162] For the specific implementation process of this step, please refer to the description of S204, which will not be repeated here.

[0163] Above Figure 3 This is an example of a model training method implemented in a single target volume scenario. However, multiple target volumes are a common situation in clinical radiotherapy. Therefore, Figure 4 An embodiment of a model training method implemented in a multi-target area scenario provided by the present application is shown, such as Figure 4 As shown in FIG. 1 , the model training method implemented in the multi-target area scenario specifically includes:

[0164] B1. Obtain the initial dose distribution status data of the target area.

[0165] For the specific implementation process of this step, please refer to the description of S201, which will not be repeated here.

[0166] After obtaining the initial dose distribution state data of the target area, each of the multiple sub-threads can be used to execute the following B2-B8 based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network to obtain the output data of the multiple sub-threads.

[0167] B2. Based on the current dose distribution state data of the current target area and the current strategy data of the action selection network and the evaluation network, determine the target parameters corresponding to the current dose distribution state data of the current target area and the prediction value corresponding to the target parameters.

[0168] It should be noted that if the current target area is the first target area, the current dose distribution state data of the current target area is the initial dose distribution state data obtained in B1.

[0169] If the current target area is not the first target area, since adjacent target areas may interact with each other, each sub-thread may determine the dose distribution state data corresponding to the final dose distribution of the previous target area as the current dose distribution state data of the current target area.

[0170] For the specific implementation process of this step, please refer to the description of A2, which will not be repeated here.

[0171] B3. Determine the dose distribution of the current target area based on the target point parameters corresponding to the current dose distribution state data of the current target area.

[0172] For the specific implementation process of this step, please refer to the description of A3, which will not be repeated here.

[0173] B4. Based on the dose distribution of the current target area, determine the actual reward value corresponding to the target parameters.

[0174] For the specific implementation process of this step, please refer to the description of A4, which will not be repeated here.

[0175] B5. When the dose distribution of the current target area does not meet the preset prescription dose and the number of target points in the current target area is less than the preset maximum number of target points, the current dose distribution status data of the current target area is updated based on the dose distribution of the current target area.

[0176] For the specific implementation process of this step, please refer to the description of A5, which will not be repeated here.

[0177] B6. When the dose distribution of the current target area satisfies the preset prescription dose, and / or the number of target points in the current target area is equal to the preset maximum number of target points, determine the final dose distribution of the current target area and multiple dose distribution status data of the current target area, multiple target parameters corresponding one-to-one to the multiple dose distribution status data, multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and determine the actual reward value and predicted value of the current target area based on the final dose distribution of the current target area and the multiple predicted values ​​corresponding one-to-one to the multiple target parameters.

[0178] For the specific implementation process of this step, please refer to the description of A6, which will not be repeated here.

[0179] After determining the actual reward value and predicted value of the current target area, since the target target area includes multiple target areas, the child thread needs to determine whether the current target area is the last target area.

[0180] B7. When the current target area is not the last target area, the current target area is updated, and the current dose distribution state data of the current target area is determined based on the final dose distribution of the previous target area of ​​the current target area.

[0181] B8. When the current target area is the last target area, based on the final dose distribution of each of the multiple target areas and the predicted value of each of the multiple target areas, determine the actual reward value and the predicted value of the target target area.

[0182] Specifically, after determining the actual reward value and predicted value of a target area, the sub-thread can perform the same operation on the next target area (iterative steps B2 to B6) until the actual reward value and predicted value of the last target area among the multiple target areas are determined.

[0183] Among them, the actual reward value of multiple target areas is the overall reward for the selection of multiple target parameters of multiple target areas obtained after completing the above iterations. For example, the target target area includes 3 target areas. After the target parameters of the third target area are determined, all target parameters of the 3 target areas are determined as a group of selections, and this group of selections is rewarded as a whole.

[0184] Therefore, the actual reward value of the target area can be jointly determined by the subthread according to the final dose distribution of multiple target areas. In other words, different weights can be assigned to the actual reward values ​​of different target areas according to their importance, and then the actual reward values ​​of different target areas are weighted and summed to obtain the actual reward value of the target area.

[0185] The predicted value of the target area can be determined by summing the predicted values ​​of multiple target areas by the sub-thread.

[0186] At this point, the determination of the output data of the multiple sub-threads is completed.

[0187] After determining the output data of multiple sub-threads, since each sub-thread needs to determine whether the treatment plan it determines is a good treatment plan, that is, each sub-thread needs to determine whether its output data can be used to update the current strategy data of the action selection network and the evaluation network, the following B9-B11 can be executed on the output data of each of the multiple sub-threads in turn, thereby completing the current training of the deep reinforcement learning model.

[0188] B9. When the final dose distribution of the target area obtained by the current training of the sub-thread meets the preset prescription dose, determine whether the actual reward value of the target area obtained by the current training of the sub-thread is greater than the dynamic reward threshold.

[0189] For the specific implementation process of this step, please refer to the description of A8, which will not be repeated here.

[0190] It should be noted that since there are multiple target areas, when the final dose distribution of each of the multiple target areas meets the preset prescription dose, the sub-thread can determine that the final dose distribution of the target area obtained by the current training meets the preset prescription dose.

[0191] B10. When the actual reward value of the target target area obtained by the current training of the sub-thread is greater than the dynamic reward threshold, the loss value of the target target area corresponding to the current training of the sub-thread is determined based on the actual reward value and predicted value of the target target area obtained by the current training of the sub-thread.

[0192] For the specific implementation process of this step, please refer to the description of A9, which will not be repeated here.

[0193] B11. When the loss value of the target area corresponding to the current training of the sub-thread is less than the dynamic loss value, the current strategy data of the action selection network and the evaluation network are updated based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and the multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value of the target area.

[0194] For the specific implementation process of this step, please refer to the description of A10, which will not be repeated here.

[0195] B12. Iterate the above training process until the deep reinforcement learning model is trained a preset number of times to obtain a trained deep reinforcement learning model.

[0196] For the specific implementation process of this step, please refer to the description of S204, which will not be repeated here.

[0197] Next, the processor's function of generating treatment plans is introduced.

[0198] Figure 5 FIG. 1 is a flowchart of a method for implementing a treatment plan by a processor according to an embodiment of the present disclosure. Figure 5 As shown, the method includes:

[0199] S501, acquiring image data and contour data of a target area to be treated.

[0200] The specific process of obtaining the image data and contour data of the target area to be treated can refer to the specific process of obtaining the image data and contour data of the target area in S201, which will not be repeated here.

[0201] For example, Figure 5As shown, the image data of the target area to be treated may be a CT image of the head of the subject to be treated, and the contour data may be contour data of the target area 1 to be treated and contour data of the target area 2 to be treated in the head of the subject to be treated.

[0202] S502: Determine dose distribution state data of the target area to be treated based on the image data and contour data of the target area to be treated.

[0203] The specific process of determining the dose distribution state data of the target area to be treated may refer to the specific process of determining the initial dose distribution state data of the target area to be treated in S201, which will not be described in detail here.

[0204] S503. Input the dose distribution state data of the target area to be treated into the deep reinforcement learning model to obtain the target point parameters of the target area to be treated.

[0205] Among them, the deep reinforcement learning model is based on Figure 2 , Figure 3 or Figure 4 The model training method shown is trained.

[0206] For example, Figure 5 As shown, the target parameters of the target area to be treated output by the deep reinforcement learning model may include the target parameters of the target area to be treated 1 and the target parameters of the target area to be treated 2. For example, the target parameters of the target area to be treated output by the deep reinforcement learning model are: target size.

[0207] S504: Generate a treatment plan for the target area to be treated according to the target point parameters.

[0208] For example, Figure 5 As shown, after determining the target parameters of the target area 1 to be treated and the target parameters of the target area 2 to be treated, the processor can determine a target parameter combination 1 consisting of a plurality of different types of target parameters for the target area 1 to be treated and a target parameter combination 2 consisting of a plurality of different types of target parameters for the target area 2 to be treated according to the target parameters of the target area 1 to be treated and the target parameters of the target area 2 to be treated. Then, the processor can generate a treatment plan for the target area to be treated according to the target parameter combination 1 and the target parameter combination 2.

[0209] Among them, the target parameter combination includes: target size, target position and target weight.

[0210] Now, taking the target parameter as the target size as an example, a process description of a method for implementing a treatment plan by a processor is given. It is understandable that the target parameter can also be the target position or the target weight.

[0211] First, the processor obtains the image data and contour data of the target area to be treated, and then the processor inputs the obtained image data and contour data of the target area to be treated into the Figure 2 , Figure 3 or Figure 4 In the deep reinforcement learning model trained by the model training method shown, the first target size output by the deep reinforcement learning model is obtained. Then, the thread in the processor calls the shape matching algorithm to match the first target size output by the deep reinforcement learning model with the contour of the target area to be treated, and determines the best placement position of the first target size in the target area to be treated, that is, determines the first target position. Next, the thread in the processor calls the weight optimization algorithm to determine the first target weight based on the determined first target size, first target position and prescription dose. At this point, the target parameter combination determination of the first target in the treatment plan is completed.

[0212] Next, the processing thread calculates the dose distribution of the target area to be treated according to the target parameter combination of the first target, and determines the current dose distribution state data of the target area to be treated according to the dose distribution of the target area to be treated, and inputs the current dose distribution state data of the target area to be treated into the deep reinforcement learning model to obtain the second target size output by the deep reinforcement learning model, and determines the target parameter combination of the second target based on the second target size (the second target position and weight are determined in the same way as the first target position and weight, which will not be repeated here).

[0213] The above-mentioned determination process of target size, target position and target weight is repeatedly iterated to determine the target parameter combination of the remaining targets until the dose distribution of the target area to be treated meets the prescribed dose and / or the number of targets in the target area to be treated is equal to the preset maximum number of targets, and the determination of the target parameter combination of the target area to be treated is completed. The target parameter combination of the target area to be treated includes the target parameter combination of multiple targets, that is, including multiple sets of target sizes (S), target positions (P) and target weights (W) {(S1, P1, W1), (S2, P2, W2), ... (S n ,P n ,W n )}.

[0214] It is understandable that the above target weights may not be obtained by calling the algorithm and may be set to 1 by default.

[0215] The treatment plan can be displayed in the form of an image corresponding to the image data of the object to be treated. In the image corresponding to the image data of the object to be treated, the target parameter combination of the target area 1 to be treated and the target parameter combination of the target area 2 to be treated can be clearly marked.

[0216] Optionally, after determining the treatment plan for the target area to be treated, the physicist (manual) may also adjust each target parameter in the target parameter combination of the treatment plan for the target area to be treated based on experience, thereby determining the treatment plan for the target area to be treated based on the adjusted target parameter combination.

[0217] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of a computer system. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in conjunction with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0218] The embodiment of the present application can divide the functional modules according to the above-mentioned computer system. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0219] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a treatment plan provided by the present disclosure.

[0220] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable an electronic device to execute the method for generating a treatment plan provided by the present disclosure.

[0221] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the deep reinforcement learning model training method and treatment planning method provided by the present disclosure for generating a treatment plan.

[0222] In some embodiments, the electronic device may be the above Figure 1 The electronic device a2 or b2 shown in FIG. Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 600 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 600 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0223] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0224] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0225] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the method for generating a treatment plan. For example, in some embodiments, the method for generating a treatment plan may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for generating a treatment plan described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for generating a treatment plan in any other appropriate manner (eg, by means of firmware).

[0226] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0227] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0228] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0229] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0230] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0231] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0232] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document is not limited here.

[0233] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A deep reinforcement learning model training method for generating a treatment plan, wherein the deep reinforcement learning model is configured to include an action selection network and an evaluation network, characterized in that: The method comprises: Obtaining initial dose distribution state data of the target area; Based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, using multiple sub-threads to determine the output data of the multiple sub-threads in parallel; Based on the output data of each of the multiple sub-threads, the current policy data of the action selection network and the evaluation network are updated in sequence to complete the current training of the deep reinforcement learning model; Iterate the above training process until the number of training times for the deep reinforcement learning model reaches a preset number of times, thereby obtaining the trained deep reinforcement learning model; Among them, the output data of the sub-thread includes: the final dose distribution and multiple dose distribution status data of the target area, multiple target parameters corresponding one-to-one to the multiple dose distribution status data, multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and the predicted value and actual reward value of the target area.

2. The method according to claim 1, characterized in that: When the target area includes only one target area, the method of using multiple sub-threads to determine output data of the multiple sub-threads in parallel based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network includes: Perform the following operations using each of the plurality of child threads: Determine, based on the current dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network, a target parameter corresponding to the current dose distribution state data of the target area and a prediction value corresponding to the target parameter; Determining the dose distribution of the target target area based on the target point parameters corresponding to the current dose distribution state data of the target target area; Determining an actual reward value corresponding to the target point parameter based on the dose distribution of the target area; When the dose distribution of the target target area does not satisfy the preset prescription dose and the number of target points in the target target area is less than the preset maximum number of target points, updating the current dose distribution state data of the target target area based on the dose distribution of the target target area; When the dose distribution of the target target area satisfies the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, determining the final dose distribution of the target target area and a plurality of dose distribution state data of the target target area, a plurality of target point parameters corresponding one-to-one to the plurality of dose distribution state data, a plurality of predicted values ​​corresponding one-to-one to the plurality of target parameters, and a plurality of actual reward values; Based on the final dose distribution of the target area and the multiple predicted values ​​corresponding to the multiple target point parameters, the actual reward value and the predicted value of the target area are determined.

3. The method according to claim 1, characterized in that When the target area includes multiple target areas, the method of using multiple sub-threads to determine output data of the multiple sub-threads in parallel based on the initial dose distribution state data of the target area and the current strategy data of the action selection network and the evaluation network includes: Using each of the plurality of child threads, perform the following operations: Determine, based on the current dose distribution state data of the current target area and the current strategy data of the action selection network and the evaluation network, a target parameter corresponding to the current dose distribution state data of the current target area and a prediction value corresponding to the target parameter; Determining the dose distribution of the current target area based on the target point parameters corresponding to the current dose distribution state data of the current target area; Determining an actual reward value corresponding to the target point parameter based on the dose distribution of the current target area; When the dose distribution of the current target area does not satisfy the preset prescription dose and the number of target points in the current target area is less than the preset maximum number of target points, updating the current dose distribution state data of the current target area based on the dose distribution of the current target area; When the dose distribution of the current target area satisfies the preset prescription dose, and / or the number of target points in the current target area is equal to the preset maximum number of target points, determining the final dose distribution of the current target area and multiple dose distribution state data of the current target area, multiple target point parameters corresponding one-to-one to the multiple dose distribution state data, multiple predicted values ​​corresponding one-to-one to the multiple target parameters, and multiple actual reward values; Determining an actual reward value and a predicted value of the current target area based on a final dose distribution of the current target area and a plurality of predicted values ​​corresponding one-to-one to the plurality of target point parameters; When the current target area is not the last target area, updating the current target area, and determining current dose distribution state data of the current target area based on the final dose distribution of the previous target area of ​​the current target area; When the current target volume is the last target volume, the actual reward value and the predicted value of the target volume are determined based on the final dose distribution of each of the multiple target volumes and the predicted value of each of the multiple target volumes.

4. The method according to claim 1, characterized in that: The updating of the current strategy data of the action selection network and the evaluation network based on the output data of each sub-thread in the multiple sub-threads in turn includes: The following operations are performed on the output data of each of the multiple sub-threads in turn: When the final dose distribution of the target area obtained by the current training of the sub-thread satisfies the preset prescription dose, determining whether the actual reward value of the target area obtained by the current training of the sub-thread is greater than a dynamic reward threshold; the dynamic reward threshold is the actual reward value of the target area used when the current strategy data of the action selection network and the evaluation network were last updated; When the actual reward value of the target target area obtained by the current training of the sub-thread is greater than the dynamic reward threshold, determining the loss value of the target target area corresponding to the current training of the sub-thread based on the actual reward value and the predicted value of the target target area obtained by the current training of the sub-thread; When the loss value of the target target area corresponding to the current training of the sub-thread is less than the dynamic loss value, the current strategy data of the action selection network and the evaluation network are updated based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding one-to-one to the multiple dose distribution state data, the multiple predicted values ​​and the multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and the actual reward value of the target target area; the dynamic loss value is the loss value of the target target area used when the current strategy data of the action selection network and the evaluation network were last updated.

5. The method according to claim 4, characterized in that The method updates the current strategy data of the action selection network and the evaluation network based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding to the multiple dose distribution state data, the multiple predicted values ​​and the multiple actual reward values ​​corresponding to the multiple target parameters, and the actual reward value of the target target area, including: Based on the multiple dose distribution state data obtained by the current training of the sub-thread, the multiple target parameters corresponding one-to-one to the multiple dose distribution state data, the multiple predicted values ​​and multiple actual reward values ​​corresponding one-to-one to the multiple target parameters, and the actual reward value of the target target area, the current policy data of the action selection network and the evaluation network are updated using a proximal policy optimization algorithm.

6. The method according to any one of claims 1 to 5, characterized in that: The target parameters include one of the following target parameters: target size, target position, and target weight.

7. The method according to any one of claims 1 to 5, characterized in that: The dose distribution state data includes: mask data of the target area, dose distribution of the target area, volume of an area with insufficient dose distribution in the target area, and volume of an area with overflow dose in the target area.

8. A treatment planning method, characterized in that: The method comprises: Acquire image data and contour data of the target area to be treated; Determining dose distribution state data of the target area to be treated according to the image data and the contour data; Inputting the dose distribution state data of the target area to be treated into a deep reinforcement learning model to obtain target point parameters of the target area to be treated; the deep reinforcement learning model is trained according to the method according to any one of claims 1 to 7; A treatment plan for the target area to be treated is generated according to the target point parameters.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

10. A non-volatile readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 8.

Images