Model training method, treatment planning method, equipment and medium

Abstract

The invention provides a deep reinforcement learning model training method for generating a treatment plan, a treatment plan method, equipment and a medium, and relates to the technical field of medical treatment, in particular to the technical field of radiotherapy. The method comprises the following steps: acquiring initial dose distribution state data of a target region; determining target data based on the initial dose distribution state data of the target region, the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer; based on the target data, updating the current strategy data of the plurality of action prediction network layers and the current strategy data of the evaluation prediction network layer, and completing the current training of the deep reinforcement learning model; and iterating the training process until the number of times of training the deep reinforcement learning model reaches a preset number of times, thereby obtaining the trained deep reinforcement learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of medical technology, in particular to the field of radiotherapy technology, and specifically to a model training method, a treatment planning method, a device and a medium. Background Art

[0002] In the field of medical technology, radiotherapy (abbreviated as radiotherapy) is one of the important means of treating tumors. Before using radiotherapy to treat a subject, a treatment plan is usually required to be designed in advance.

[0003] At present, the design of treatment plans usually relies on physiotherapists (manuals) to make repeated adjustments based on their own experience and professional skills in order to design a more reasonable treatment plan. Therefore, the existing treatment planning method requires high clinical experience of physiotherapists and requires continuous trial and error adjustments, which is time-consuming, labor-intensive and inefficient. Summary of the invention

[0004] The present invention provides a model training method, a treatment planning method, a device and a medium.

[0005] In a first aspect, a deep reinforcement learning model training method for generating a treatment plan is provided, wherein the deep reinforcement learning model is configured to include: multiple action prediction network layers and an evaluation prediction network layer, and different action prediction network layers are used to output different types of target parameters included in the treatment plan, and the method includes: obtaining initial dose distribution state data of a target target area; determining target data based on the initial dose distribution state data of the target target area, current strategy data of multiple action prediction network layers, and current strategy data of the evaluation prediction network layer; updating current strategy data of multiple action prediction network layers and current strategy data of the evaluation prediction network layer based on the target data, and completing current training of the deep reinforcement learning model; iterating the above training process until the number of training times of the deep reinforcement learning model reaches a preset number of times, thereby obtaining a trained deep reinforcement learning model;

[0006] Among them, the target data includes: the final dose distribution and multiple dose distribution status data of the target area, multiple action sets corresponding to the multiple dose distribution status data output by multiple action prediction network layers, multiple prediction values ​​corresponding to the multiple action sets output by the evaluation prediction network layer, multiple actual reward values ​​corresponding to the multiple action sets, and the predicted value and actual reward value of the target area; each action set in the multiple action sets includes a target parameter combination composed of multiple different types of target parameters.

[0007] In some embodiments, the multiple action prediction network layers include: at least two of: a target size prediction network layer, a target position prediction network layer, and a target weight prediction network layer.

[0008] In some embodiments, the target data is determined based on the initial dose distribution state data of the target area, the current strategy data of multiple action prediction network layers, and the current strategy data of the evaluation prediction network layer, including: determining the action set corresponding to the current dose distribution state data output by multiple action prediction network layers based on the current dose distribution state data of the target area and the current strategy data of each action prediction network layer, and determining the prediction value corresponding to the action set output by the evaluation prediction network layer based on the current dose distribution state data of the target area and the current strategy data of the evaluation prediction network layer; determining the dose distribution of the target area based on the action set corresponding to the current dose distribution state data, and determining the actual reward value corresponding to the action set based on the dose distribution of the target area; and determining the actual reward value corresponding to the action set when the dose distribution of the target area is not satisfactory. When the preset prescription dose is satisfied and the number of target points in the target area is less than the preset maximum number of target points, based on the dose distribution of the target area, the current dose distribution state data of the target area is updated; when the dose distribution of the target area satisfies the preset prescription dose and / or the number of target points in the target area is equal to the preset maximum number of target points, the final dose distribution of the target area and multiple dose distribution state data of the target area, multiple action sets corresponding one-to-one to the multiple dose distribution state data output by the multiple action prediction network layers, multiple prediction values ​​corresponding one-to-one to the multiple action sets output by the evaluation prediction network layer, and multiple actual reward values ​​corresponding one-to-one to the multiple action sets are determined; based on the final dose distribution of the target area and the multiple prediction values ​​corresponding one-to-one to the multiple action sets, the actual reward value and the prediction value of the target area are determined.

[0009] In some embodiments, when multiple action prediction network layers include a first action prediction network layer and a second action prediction network layer deployed from top to bottom, based on the current dose distribution state data of the target area and the current strategy data of each action prediction network layer, a set of actions corresponding to the current dose distribution state data output by multiple action prediction network layers is determined, including: determining a first action based on the current dose distribution state data and the current strategy data of the first action prediction network layer; determining a second action corresponding to the first action based on the current dose distribution state data, the first action and the current strategy data of the second action prediction network layer.

[0010] In some embodiments, when multiple action prediction network layers include a first action prediction network layer, a second action prediction network layer, and a third action prediction network layer deployed from top to bottom, based on the current dose distribution state data of the target area and the current strategy data of each action prediction network layer, a set of actions corresponding to the current dose distribution state data output by multiple action prediction network layers is determined, including: determining a first action based on the current dose distribution state data and the current strategy data of the first action prediction network layer; determining a second action corresponding to the first action based on the current dose distribution state data, the first action, and the current strategy data of the second action prediction network layer; determining a third action corresponding to the first action based on the current dose distribution state data, the first action, the second action, and the current strategy data of the third action prediction network layer.

[0011] In some embodiments, based on the target data, the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer are updated, including: when the final dose distribution of the target target area obtained in the current training meets the preset prescription dose, determining whether the actual reward value of the target target area obtained in the current training is greater than the dynamic reward threshold; the dynamic reward threshold is the actual reward value of the target target area used when the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated; when the actual reward value of the target target area obtained in the current training is greater than the dynamic reward threshold, based on the actual reward value and predicted value of the target target area obtained in the current training, determining Determine the loss value of the target target area corresponding to the current training; when the loss value of the target target area corresponding to the current training is less than the dynamic loss value, based on the multiple dose distribution state data obtained in the current training, the multiple action sets corresponding to the multiple dose distribution state data one-to-one, the multiple actual reward values ​​corresponding to the multiple action sets, the multiple predicted values ​​corresponding to the multiple action sets and the actual reward value of the target target area, update the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer; the dynamic loss value is the loss value of the target target area used when the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated.

[0012] In some embodiments, based on multiple dose distribution state data obtained from the current training, multiple action sets corresponding one-to-one to the multiple dose distribution state data, multiple actual reward values ​​corresponding one-to-one to the multiple action sets, multiple predicted values ​​corresponding one-to-one to the multiple action sets, and the actual reward value of the target area, the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer are updated, including: based on the multiple actual reward values ​​corresponding one-to-one to the multiple action sets and the actual reward value of the target area, the actual cumulative reward value of the multiple action sets is determined, and based on the multiple dose distribution state data, the multiple action sets corresponding one-to-one to the multiple dose distribution state data, and the actual cumulative reward value of the multiple action sets, the current strategy data of the multiple action prediction network layers is updated; based on the multiple dose distribution state data, the multiple action sets corresponding one-to-one to the multiple dose distribution state data, and the multiple predicted values ​​corresponding one-to-one to the multiple action sets, the current strategy data of the evaluation prediction network layer is updated.

[0013] In a second aspect, a treatment planning method is provided, comprising: acquiring image data and contour data of a target area to be treated; determining dose distribution state data of the target area to be treated based on the image data and contour data of the target area to be treated; inputting the dose distribution state data of the target area to be treated into a deep reinforcement learning model to obtain a target parameter combination of the target area to be treated consisting of multiple different types of target parameters; the deep reinforcement learning model is trained according to the deep reinforcement learning model training method for generating a treatment plan in any embodiment of the first aspect; and generating a treatment plan for the target area to be treated according to the target parameter combination.

[0014] In a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods of the first aspect or the second aspect mentioned above.

[0015] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein a computer program is stored on the storage medium, and the computer program is used to execute any one of the methods of the first aspect or the second aspect above.

[0016] The disclosed embodiments provide a deep reinforcement learning model training method, a treatment plan method, a device and a medium for generating a treatment plan, wherein the deep reinforcement learning model is configured to include: multiple action prediction network layers and an evaluation prediction network layer, and different action prediction network layers are used to output different types of target parameters included in the treatment plan. In this way, after obtaining the initial dose distribution state data of the target target area, the target data can be determined based on the initial dose distribution state data of the target target area, the current strategy data of the multiple action prediction network layers, and the current strategy data of the evaluation prediction network layer. Then, based on the target data, the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer can be updated to complete the current training of the deep reinforcement learning model. The above training process is iterated until the number of training times of the deep reinforcement learning model reaches a preset number of times, and a trained deep reinforcement learning model is obtained.

[0017] Among them, the target data includes: the final dose distribution and multiple dose distribution status data of the target area, multiple action sets corresponding to the multiple dose distribution status data output by multiple action prediction network layers, multiple prediction values ​​corresponding to the multiple action sets output by the evaluation prediction network layer, multiple actual reward values ​​corresponding to the multiple action sets, and the predicted value and actual reward value of the target area; each action set in the multiple action sets includes a target parameter combination composed of multiple different types of target parameters.

[0018] From the above, it can be seen that the present disclosure can divide the overall task of a set of actions for determining different types of target parameters into multiple subtasks of actions for determining different types of target parameters, and each action prediction network layer in the multiple action prediction network layers included in the deep reinforcement learning model completes a subtask, that is, the complex overall task is decomposed into each subtask for calculation. Therefore, the efficiency of determining the target parameters can be improved through multiple action prediction network layers. Secondly, because deep reinforcement learning conforms to the characteristics of gamma knife treatment plan design, it allows the processor in the computer system to implement a repeated trial and error process. Therefore, through the deep reinforcement learning model, better target parameters can be automatically generated, thereby generating a better treatment plan based on the target parameters, reducing dependence on clinical experience, and improving the effect of the treatment plan without manually setting the target parameters, thereby improving the efficiency of the physicist in designing the treatment plan.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 It is a schematic diagram of an implementation environment of a deep reinforcement learning model training method and a treatment plan method for generating a treatment plan shown in an embodiment of the present disclosure;

[0022] Figure 2 It is a flowchart of a deep reinforcement learning model training method shown in an embodiment of the present disclosure;

[0023] Figure 3 It is a flowchart of another deep reinforcement learning model training method shown in an embodiment of the present disclosure;

[0024] Figure 4 is a flowchart of a method for implementing a treatment plan by a processor shown in an embodiment of the present disclosure;

[0025] Figure 5 It is a block diagram of an electronic device used to implement the embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Before introducing the present disclosure in detail, application scenarios involved in the embodiments of the present disclosure are described first.

[0029] The deep reinforcement learning model training method, treatment planning method, device and medium for generating treatment plans provided by the embodiments of the present disclosure can be applied in the field of medical technology, specifically in the scenario of clinical radiotherapy (abbreviated as radiotherapy), such as the scenario of gamma knife treatment plan formulation.

[0030] Before radiation therapy is given to a patient, a treatment plan is usually designed in advance.

[0031] At present, the design of treatment plans usually relies on physiotherapists (manuals) to make repeated adjustments based on their own experience and professional skills in order to design a more reasonable treatment plan. Therefore, the existing treatment planning method requires high clinical experience of physiotherapists and requires continuous trial and error adjustments, which is time-consuming, labor-intensive and inefficient.

[0032] Based on this, the embodiments of the present disclosure provide a deep reinforcement learning model training method, a treatment plan method, a device and a medium for generating a treatment plan, wherein the deep reinforcement learning model is configured to include: multiple action prediction network layers and an evaluation prediction network layer, and different action prediction network layers are used to output different types of target parameters included in the treatment plan. In this way, after obtaining the initial dose distribution state data of the target target area, the target data can be determined based on the initial dose distribution state data of the target target area, the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer. Then, based on the target data, the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer can be updated to complete the current training of the deep reinforcement learning model. The above training process is iterated until the number of training times of the deep reinforcement learning model reaches a preset number of times, and a trained deep reinforcement learning model is obtained.

[0033] Among them, the target data includes: the final dose distribution and multiple dose distribution status data of the target area, multiple action sets corresponding to the multiple dose distribution status data output by multiple action prediction network layers, multiple prediction values ​​corresponding to the multiple action sets output by the evaluation prediction network layer, multiple actual reward values ​​corresponding to the multiple action sets, and the predicted value and actual reward value of the target area; each action set in the multiple action sets includes a target parameter combination composed of multiple different types of target parameters.

[0034] From the above, it can be seen that the present disclosure can divide the overall task of a set of actions for determining different types of target parameters into multiple subtasks of actions for determining different types of target parameters, and each action prediction network layer in the multiple action prediction network layers included in the deep reinforcement learning model completes a subtask, that is, the complex overall task is decomposed into each subtask for calculation. Therefore, the efficiency of determining the target parameters can be improved through multiple action prediction network layers. Secondly, because deep reinforcement learning conforms to the characteristics of gamma knife treatment plan design, it allows the processor in the computer system to implement a repeated trial and error process. Therefore, through the deep reinforcement learning model, better target parameters can be automatically generated, thereby generating a better treatment plan based on the target parameters, reducing dependence on clinical experience, and improving the effect of the treatment plan without manually setting the target parameters, thereby improving the efficiency of the physicist in designing the treatment plan.

[0035] The method provided in this application mainly involves two methods: a model training method and a treatment planning method. The implementation environment of the above two methods is first introduced below.

[0036] Figure 1 (a) is a schematic diagram of an implementation environment of a model training method shown in an embodiment of the present disclosure. Figure 1 In (a), the implementation environment includes an image scanning device a1 and an electronic device a2.

[0037] The image scanning device a1 is a device for scanning and displaying the tumor site and surrounding normal tissue of the subject to be treated. In some embodiments, the image scanning device a1 can be at least one of a computed tomography (CT) device, an emission computed tomography (ECT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasound examination device.

[0038] In the disclosed embodiment, the image scanning device a1 is used to acquire medical scanning images (such as image data of different target areas, etc.) and upload the medical scanning images to the electronic device a2 so that the electronic device a2 can execute subsequent model training methods based on the medical scanning images of the object to be treated.

[0039] The electronic device a2 is a device for training a deep reinforcement learning model. In some embodiments, the electronic device a2 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer.

[0040] Furthermore, in some embodiments, the electronic device a2 may run a computer system including a processor for implementing a training method for a deep reinforcement learning model.

[0041] In some embodiments, the implementation environment further includes a server a3. In some embodiments, the server a3 is used to provide background communication services for the image scanning device a1 and the electronic device a2.

[0042] Among them, server a3 can be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or at least one of cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data or artificial intelligence platforms, etc., which is not limited in the embodiments of the present disclosure. In some embodiments, the number of the above-mentioned servers a3 can be more or less, which is not limited in the embodiments of the present disclosure. Of course, server a3 can also include other functions to provide more comprehensive and diversified services.

[0043] Figure 1 (b) is a schematic diagram of an implementation environment of a treatment planning method shown in an embodiment of the present disclosure. Figure 1 In (b), the implementation environment includes an image scanning device b1, an electronic device b2 and a radiotherapy device b3.

[0044] Among them, image scanning equipment b1 and Figure 1 The device form of the image scanning device a1 in (a) is similar, and you can refer to the description of the image scanning device a1, which will not be repeated here.

[0045] In the disclosed embodiment, the image scanning device b1 is used to obtain a medical scanning image of the object to be treated (such as image data of the target area to be treated, etc.), and upload the medical scanning image of the object to be treated to the electronic device b2, so that the electronic device b2 executes a method for generating a subsequent treatment plan based on the medical scanning image of the object to be treated.

[0046] The electronic device b2 is a device for applying a deep reinforcement learning model to generate a treatment plan.

[0047] Electronic equipment b2 and Figure 1 The device form of the electronic device a2 in (a) is similar, and you can refer to the description of the electronic device a2, which will not be repeated here.

[0048] Furthermore, in some embodiments, the electronic device b2 may run a computer system including a processor for implementing a method for generating a treatment plan using a deep reinforcement learning model.

[0049] The radiotherapy device b3 is a device used for radiotherapy. In some embodiments, the radiotherapy device b3 can be at least one of a gamma knife, a linear accelerator, a neutron knife, an X-ray therapy machine and the like.

[0050] In the embodiment of the present disclosure, the radiotherapy device b3 is used to receive the treatment plan from the electronic device b2, and perform radiotherapy on the subject to be treated according to the treatment plan.

[0051] In some embodiments, the implementation environment further includes a server b4. In some embodiments, the server b4 is used to provide background communication services for the above-mentioned image scanning device b1, electronic device b2 and radiotherapy device b3.

[0052] Server b4 and Figure 1 In (a), the device form of server a3 is similar, and you can refer to the description of server a3, which will not be repeated here.

[0053] The following is based on Figure 1 The implementation environment shown in (a) above first introduces the model training method provided by the embodiment of the present disclosure.

[0054] The model training method provided in the embodiments of the present disclosure can be applied to Figure 1 The electronic device a2 in (a) may be equipped with a computer system, the computer system including a processor, and the processor is used to implement the model training method.

[0055] It should be understood that since the training method of the deep reinforcement learning model involves multiple different processing and calculation processes, the processor can also implement the model training method by calling various suitable processing threads, and the embodiment of the present application does not limit this. For ease of description, the embodiment of the present application is described by taking the processor as the execution subject as an example.

[0056] Figure 2 1 is a flow chart of a deep reinforcement learning model training method for generating a treatment plan shown in an embodiment of the present disclosure. The deep reinforcement learning model is configured to include: multiple action prediction network layers and evaluation prediction network layers, and different action prediction network layers are used to output different types of target parameters included in the treatment plan. Figure 2 As shown, the method includes:

[0057] S201, obtaining initial dose distribution state data of the target area.

[0058] The target area includes the area to be treated of the subject to be treated.

[0059] Optionally, the target volume may also be referred to as a planning target volume (Planning Target Volume, PTV).

[0060] For example, the subject to be treated may be a phantom, a human body, an animal, etc. The target region may be a tumor region of the subject to be treated, etc.

[0061] Exemplary, combined Figure 1In (a), the image scanning device a1 can collect image data of the object to be treated and transmit the image data to the electronic device a2.

[0062] Optionally, after receiving the image data, the electronic device a2 may perform contour outlining on the image data to obtain contours of different target areas of the object to be treated in the image data, that is, contour data.

[0063] Exemplarily, the contour data may be the contour of a tumor region.

[0064] Optionally, the electronic device a2 may also be connected to a third-party software program. The third-party software program is used to outline the image data to obtain outline data.

[0065] Optionally, when outlining the image data, the physician may perform an outlining operation on the electronic device a2. The electronic device a2 may obtain the outline data in response to the physician's outlining operation.

[0066] Alternatively, the electronic device a2 may also automatically outline the contours of different target areas of the object to be treated in the image data using contour drawing software.

[0067] After determining the image data and contour data of the target area, the processor in the electronic device a2 may determine the initial dose distribution state data of the target area based on the image data and contour data of the target area.

[0068] The initial dose distribution state data is used to represent the dose distribution state data of the target area when no target point is placed.

[0069] The dose distribution state data includes: mask data of the target area, dose distribution of the target area, volume of an area with insufficient dose distribution in the target area, and volume of an area with overflow dose in the target area.

[0070] Since the initial dose distribution state data is used to represent the dose distribution state data of the target target area when no target point is placed, the dose distribution, the volume of the insufficient dose distribution area, and the volume of the overflow dose area in the initial dose distribution state data are all 0. In this case, the processor in the electronic device a2 only needs to determine the mask data of the target target area according to the image data and contour data of the target target area to determine the initial dose distribution state data of the target target area.

[0071] Specifically, the image data of the target area may include contour data of the target area and image data of an organ at risk (OAR).

[0072] Among them, organs at risk refer to normal organs around the target area, that is, organs that have not developed lesions.

[0073] The processor in the electronic device a2 can process the image data and contour data of the target area to generate mask data of the target area. In the process of generating the mask data, the processor in the electronic device a2 can perform different operations on the area corresponding to the target area and the area corresponding to the OAR based on the contour data, thereby distinguishing the area corresponding to the target area and the area corresponding to the OAR.

[0074] At the same time, the subsequent target position search space (i.e., the placement position of the target) can be limited to the area corresponding to the target target area, so as to avoid the target position falling into the OAR area or the area corresponding to other tissues, thereby avoiding treatment damage to the OAR area or the area corresponding to other tissues. In addition, limiting the subsequent target position search space to the area corresponding to the target target area can reduce the search space for the specific shape of the target target area, thereby improving the speed of model training and generating treatment plans.

[0075] Exemplarily, the processor in the electronic device a2 can construct a three-dimensional matrix of uniform size, wherein the image parameter of the area corresponding to the target area can be set to 1, the image parameter of the area corresponding to the OAR can be set to -1, and the image parameter of other tissues can be set to 0. In this way, mask data can be obtained to distinguish the area corresponding to the target area and the area corresponding to the OAR.

[0076] S202 , determining target data based on the initial dose distribution state data of the target area, the current strategy data of multiple action prediction network layers, and the current strategy data of the evaluation prediction network layer.

[0077] Among them, multiple action prediction network layers are deployed from top to bottom, that is, multiple action prediction network layers are connected in sequence. For example, Figure 2 As shown, the multiple action prediction network layers may include action prediction network layer 1 ... action prediction network layer n deployed from top to bottom; n is an integer greater than or equal to 2.

[0078] In practical applications, since the target parameters usually include target size, target weight and target position, the multiple action prediction network layers usually include at least two of the target size prediction network layer, the target position prediction network layer and the target weight prediction network layer. Of course, when the number of target parameters is other numbers, the number of multiple action prediction network layers can also be adaptively adjusted according to the number of target parameters.

[0079] In an embodiment of the present application, the target data includes: multiple dose distribution state data of the target area, multiple action sets corresponding one-to-one to the multiple dose distribution state data output by multiple action prediction network layers, multiple prediction values ​​corresponding one-to-one to the multiple action sets output by the evaluation prediction network layer, multiple actual reward values ​​corresponding one-to-one to the multiple action sets, and the predicted value and actual reward value of the target area; each action set in the multiple action sets includes a target parameter combination composed of multiple different types of target parameters.

[0080] Since the process of generating a Gamma Knife treatment plan is a process of sequentially arranging targets, after the arrangement of one target is completed, the dose distribution state of the target area will change. Therefore, in the process of generating a complete treatment plan for the target area, the dose distribution state of the target area will change as the targets are placed one by one.

[0081] Therefore, in the embodiment of the present application, the target data can only be acquired when a complete treatment plan is generated for the target area.

[0082] Specifically, after obtaining the initial dose distribution state data, the processor in the electronic device a2 can input the initial dose distribution state data into each of the multiple motion prediction network layers. At the same time, since the multiple motion prediction network layers are deployed from top to bottom, the processor in the electronic device a2 can also input the output data of the upper motion prediction network layer into the lower motion prediction network layer.

[0083] Different action prediction network layers can output different types of target parameters selected for the first target according to their initial strategy data and the initial dose distribution state data of the target area. In this way, the first action set can be obtained through the output data of multiple action prediction network layers.

[0084] Since the deep reinforcement learning model also includes an evaluation and prediction network layer, after obtaining the initial dose distribution state data, the processor in the electronic device a2 can also simultaneously input the initial dose distribution state data into the evaluation and prediction network layer. Therefore, while determining the first action set, the evaluation and prediction network layer can also determine the prediction value corresponding to the first action set based on its initial strategy data and the initial dose distribution state data of the target area.

[0085] After the first target point is placed, the dose distribution state of the target area changes. At this time, the processor in the electronic device a2 can also determine the dose distribution of the target area based on the first action set, and determine the actual reward value corresponding to the first action set based on the dose distribution of the target area.

[0086] Then, the processor in the electronic device a2 can update the dose distribution state data of the target target area based on the dose distribution of the target target area, and repeat the above target point placement process according to the updated dose distribution state data of the target target area until the generation of the complete treatment plan of the target target area is completed, and the final dose distribution of the target target area and multiple dose distribution state data of the target target area, multiple action sets corresponding to multiple dose distribution state data output by multiple action prediction network layers, multiple prediction values ​​corresponding to multiple action sets output by the evaluation prediction network layer, multiple actual reward values ​​corresponding to multiple action sets, and actual reward values ​​and prediction values ​​of the target target area are obtained. In this way, the processor in the electronic device a2 can obtain the target data.

[0087] S203. Based on the target data, update the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer to complete the current training of the deep reinforcement learning model.

[0088] Specifically, after the target data is obtained, it indicates that the treatment plan of the target area has been completed. In this case, the processor in the electronic device a2 can update the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer based on the target data.

[0089] It should be noted that the training process of the deep reinforcement learning model is equivalent to a trial-and-error process for generating a treatment plan. Therefore, there may be a situation where the target data obtained is discarded because it does not meet the update conditions. Therefore, in the embodiment of the present application, completing the training of the deep reinforcement learning model once does not mean that the current policy data of multiple action prediction network layers and the current policy data of the evaluation prediction network layer will be updated.

[0090] S204, iterating the above training process until the deep reinforcement learning model is trained a preset number of times, thereby obtaining a trained deep reinforcement learning model.

[0091] Specifically, the above-mentioned training process of the deep reinforcement learning model (i.e., S201 to S203) is iterated until the deep reinforcement learning model is trained a preset number of times, thereby obtaining a trained deep reinforcement learning model.

[0092] Optionally, the electronic device may also include an experience memory unit (buffer_memory). The experience memory unit is used to store the final dose distribution and multiple dose distribution state data of the target area at each update stage during the current strategy data update process, multiple action sets corresponding to multiple dose distribution state data output by multiple action prediction network layers, multiple predicted values ​​corresponding to multiple action sets output by the evaluation prediction network layer, multiple actual reward values ​​corresponding to multiple action sets, predicted values ​​and actual reward values ​​(i.e., target data) of the target area. Subsequently, through the corresponding relationship stored in the experience memory unit, various required data determined in each training can be found, and then model training can be performed.

[0093] Combine the following Figure 3 , the model training method provided in the embodiment of the present application is introduced in detail. Figure 3 An embodiment of the model training method provided by the present application is shown, such as Figure 3 As shown, the model training method specifically includes:

[0094] S1. Obtain the initial dose distribution state data of the target area.

[0095] For the specific implementation process of this step, please refer to the description of S201, which will not be repeated here.

[0096] S2. Based on the current dose distribution state data of the target area and the current strategy data of each action prediction network layer, determine the action set corresponding to the current dose distribution state data output by multiple action prediction network layers, and based on the current dose distribution state data of the target area and the current strategy data of the evaluation prediction network layer, determine the prediction value corresponding to the action set output by the evaluation prediction network layer.

[0097] Among them, each action prediction network layer and evaluation prediction network layer can be preset with an initial strategy data.

[0098] For the action prediction network layer, the strategy data refers to the strategy for selecting target parameters (also called action selection strategy) when placing the target in the target area, such as the strategy for selecting the target size when placing the target.

[0099] For the evaluation prediction network layer, the strategy data is a value prediction of the action selection under a certain dose distribution state of the target area. The strategy data is a strategy for determining the predicted value corresponding to the action selection result.

[0100] Since the process of generating a Gamma Knife treatment plan is a process of sequentially arranging targets, after the arrangement of one target is completed, the dose distribution state of the target area will change. Therefore, in the process of generating a complete treatment plan for the target area, the current dose distribution state of the target area will change as the targets are placed one by one.

[0101] Therefore, after obtaining the current dose distribution state data of the target target area (for the first target, the current dose distribution state data of the target target area is: the initial dose distribution state data of the target target area), the processor in the electronic device can input the current dose distribution state data of the target target area into each action prediction network layer in the multiple action prediction network layers. The first action prediction network layer can determine the first action based on its initial strategy data and the current dose distribution state data of the target target area, and the first action can represent the first type of target parameters selected when placing the current target, such as the target size selected when placing the current target.

[0102] Then, the processor in the electronic device can input the output data of the first action prediction network layer (i.e., the first action) into the second action prediction network layer. The second action prediction network layer can determine the second action based on its initial strategy data, the current dose distribution state data of the target area, and the output data of the first action prediction network layer (i.e., the first action), and the second action can represent the second type of target parameters selected when placing the current target, such as the target position selected when placing the current target.

[0103] By analogy, the processor in the electronic device can obtain an action set consisting of multiple actions corresponding to the current dose distribution state data, and the action set includes multiple different types of target parameters selected for the current target when placing the current target, for example, when placing the current target, the target size, target position and target weight selected for the current target are represented.

[0104] Since the deep reinforcement learning model also includes an evaluation prediction network layer, when determining the action set corresponding to the current dose distribution state data output by the multiple action prediction network layers, after obtaining the current dose distribution state data of the target area, the processor in the electronic device can input the initial dose distribution state data into the evaluation prediction network layer. The evaluation prediction network layer can determine the prediction value corresponding to the above action set based on its initial strategy data and the current dose distribution state data of the target area.

[0105] S3. Determine the dose distribution of the target area based on the action set corresponding to the current dose distribution state data, and determine the actual reward value corresponding to the action set based on the dose distribution of the target area.

[0106] Specifically, after determining the action set corresponding to the current dose distribution state data, the processor in the electronic device can place a target point in the target area according to the action set. In this way, after the target point is placed, the dose distribution of the target area will change. In this case, the processor in the electronic device can determine the dose distribution of the target area after the change.

[0107] In some embodiments, when the action set only includes two types of target parameters, for example, when the action set only includes target size and target weight, in order to place the current target, it is also necessary to determine the target position. At this time, the processor in the electronic device can call a shape matching algorithm to determine the target position.

[0108] After obtaining all parameters of the target, namely, target size, target position, and target weight, the processor in the electronic device determines the dose distribution of the target area according to these target parameters.

[0109] It can be understood that when the multiple action prediction network layers include two layers, the deep reinforcement learning model can only generate two types of target parameters. When the types of target parameters exceed two, the determination of the remaining types of target parameters can be performed by calling the corresponding algorithm through the processor in the electronic device to determine the remaining types of target parameters.

[0110] Secondly, in order to obtain a deep reinforcement learning model for subsequent training, the processor in the electronic device can also determine the actual reward value corresponding to the action set based on the dose distribution of the target area.

[0111] Specifically, since the dose distribution of the target area is determined after the target point is placed, the dose distribution of the target area can truly reflect the contribution of an action set to the dose distribution of the target area. In this way, after obtaining an action set, the processor in the electronic device can determine the actual reward value corresponding to the action set based on the dose distribution of the target area.

[0112] Exemplarily, the actual reward value corresponding to the action set includes: positive reward and negative reward. The positive reward is obtained by multiplying the increase in the dose coverage and conformality of the target area before and after the target point corresponding to the action set by the corresponding weight; the negative reward is obtained by multiplying the increase in the dose overflow rate of the target area before and after the target point corresponding to the action set by the corresponding weight.

[0113] S4. When the dose distribution of the target area does not satisfy the preset prescription dose and the number of target points in the target area is less than the preset maximum number of target points, the current dose distribution state data of the target area is updated based on the dose distribution of the target area.

[0114] Specifically, after determining the dose distribution of the target target area, it can be determined whether the dose distribution of the target target area meets the preset prescription dose, and / or whether the number of target points in the target target area is less than the preset maximum number of target points. When the dose distribution of the target target area does not meet the preset prescription dose, and the number of target points in the target target area is less than the preset maximum number of target points, it means that the treatment plan of the target target area has not been completed. In this case, the processor of the electronic device can update the current dose distribution status data of the target target area based on the dose distribution of the target target area, that is, update the mask data of the target target area, the dose distribution of the target target area, the volume of the area with insufficient dose distribution in the target target area, and the volume of the area with overflow dose in the target target area.

[0115] Optionally, when the processor in the electronic device updates the current dose distribution state data of the target area based on the dose distribution of the target area, it can extract features of the dose distribution to obtain updated current dose distribution state data.

[0116] Next, the processor in the electronic device may repeatedly execute the above S2-S4 according to the updated dose distribution state data, select an action set corresponding to the subsequent target point, and determine the predicted reward value and actual reward value corresponding to the action set.

[0117] S5. When the dose distribution of the target area satisfies the preset prescription dose, and / or the number of target points in the target area is equal to the preset maximum number of target points, determine the final dose distribution of the target area and multiple dose distribution status data of the target area, multiple action sets corresponding to the multiple dose distribution status data output by the multiple action prediction network layers, multiple prediction values ​​corresponding to the multiple action sets output by the evaluation prediction network layer, and multiple actual reward values ​​corresponding to the multiple action sets.

[0118] Specifically, when the dose distribution of the target target area meets the preset prescription dose, and / or the number of targets in the target target area is equal to the preset maximum number of targets, it means that the target target area has completed the formulation of the treatment plan. In this case, the target target area has placed multiple targets that form the treatment plan. Accordingly, the final dose distribution of the target target area and multiple dose distribution state data of the target target area, multiple action sets corresponding to multiple dose distribution state data output by multiple action prediction network layers, multiple prediction values ​​corresponding to multiple action sets output by the evaluation prediction network layer, and multiple actual reward values ​​corresponding to multiple action sets can be obtained.

[0119] S6. Determine the actual reward value and the predicted value of the target area based on the final dose distribution of the target area and the multiple predicted values ​​corresponding to the multiple action sets.

[0120] Specifically, when the dose distribution of the target target area satisfies the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, the final dose distribution of the target target area and multiple prediction values ​​corresponding to multiple action sets can be obtained. In this case, the processor in the electronic device can determine the actual reward value of the target target area based on the final dose distribution, and determine the prediction value of the target target area based on the multiple prediction values ​​corresponding to multiple action sets.

[0121] Optionally, after determining the final dose distribution of the target area, the dose coverage and conformality of the target area can be determined according to the final dose distribution of the target area, and the actual reward value of the target area can be determined according to the dose coverage and conformality of the target area.

[0122] Optionally, after determining a plurality of prediction values ​​corresponding one-to-one to a plurality of action sets, the plurality of prediction values ​​corresponding one-to-one to the plurality of action sets may be summed to obtain the prediction value of the target area.

[0123] In one achievable manner, the method for updating the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer based on the target data to complete the current training of multiple action prediction network layers and the evaluation prediction network layer specifically includes:

[0124] S7. When the final dose distribution of the target area obtained in the current training meets the preset prescription dose, determine whether the actual reward value of the target area obtained in the current training is greater than the dynamic reward threshold.

[0125] The final dose distribution of the target area is the dose distribution corresponding to the treatment plan of the target area after the generation of the treatment plan of the target area consisting of multiple action sets is completed.

[0126] It should be noted that, since the treatment plan of the target target area is completed when the dose distribution of the target target area satisfies the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, therefore, when the treatment plan of the target target area is obtained when the dose distribution of the target target area satisfies the preset prescription dose, the processor in the electronic device does not need to determine whether the dose distribution corresponding to the treatment plan satisfies the preset prescription dose, and can directly proceed to the subsequent judgment steps. Of course, the processor in the electronic device can also re-judge whether the dose distribution corresponding to the treatment plan satisfies the preset prescription dose.

[0127] When the treatment plan of the target area is obtained when the number of target points in the target area is equal to the preset maximum number of target points, since the treatment plan only meets the preset maximum number of target points but does not necessarily meet the preset prescription dose, the processor in the electronic device needs to determine whether the dose distribution corresponding to the treatment plan meets the preset prescription dose.

[0128] If the dose distribution corresponding to the treatment plan does not meet the preset prescription dose, it means that the treatment plan is not a good treatment plan. Therefore, the processor in the electronic device can discard the target data corresponding to the treatment plan, that is, there is no need to update the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer according to the target data corresponding to the treatment plan.

[0129] Accordingly, if the dose distribution corresponding to the treatment plan satisfies the preset prescription dose, it means that the treatment plan can meet the requirement of the preset prescription dose. In this case, the processor in the electronic device can determine whether the actual reward value of the target area is greater than the dynamic reward threshold.

[0130] Among them, the dynamic reward threshold is the actual reward value of the target area used when the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated.

[0131] By judging whether the actual reward value of the target area obtained in the current training is greater than the dynamic reward threshold, it can be determined whether the quality of the treatment plan is better than the quality of the treatment plan used when the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were updated last time.

[0132] When the actual reward value of the target area obtained in the current training is less than or equal to the dynamic reward threshold, it means that the quality of the treatment plan is inferior to or equal to the quality of the treatment plan used when the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated. Therefore, the processor in the electronic device can discard the target data corresponding to the treatment plan, that is, there is no need to update the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer according to the target data corresponding to the treatment plan.

[0133] Accordingly, when the actual reward value of the target area obtained in the current training is greater than the dynamic reward threshold, it indicates that the quality of the treatment plan is better than the quality of the treatment plan used when the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated, that is, the quality of the treatment plan is better. In this case, the electronic device can execute S8.

[0134] S8. When the actual reward value of the target target area obtained in the current training is greater than the dynamic reward threshold, the loss value of the target target area corresponding to the current training is determined based on the actual reward value and the predicted value of the target target area obtained in the current training.

[0135] Specifically, when the actual reward value of the target target area obtained in the current training is greater than the dynamic reward threshold, it indicates that the quality of the treatment plan is better. In this case, the processor in the electronic device can determine the loss value of the target target area corresponding to the current training based on the actual reward value and the predicted value of the target target area obtained in the current training.

[0136] Optionally, the processor in the electronic device may use a preset loss function to determine the loss value of the target area corresponding to the current training according to the predicted value and actual reward value corresponding to the target area obtained from the current training.

[0137] Optionally, the preset loss function may be a relative advantage parameter or other general loss function, which is not limited in this application.

[0138] Next, the processor in the electronic device can determine whether the loss value of the target area corresponding to the current training is less than the dynamic loss value.

[0139] Among them, the dynamic loss value is the loss value of the target area used when the current strategy data of multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated.

[0140] If the loss value of the target area corresponding to the current training is greater than or equal to the loss value of the target area used when the current policy data of the multiple action prediction network layers and the current policy data of the evaluation prediction network layer were updated last time, it means that the quality estimate of the treatment plan by the deep reinforcement learning model (that is, the prediction of the contribution value of each action set selection to the treatment plan) is inaccurate. Therefore, the processor in the electronic device can discard the target data corresponding to the treatment plan, that is, there is no need to update the current policy data of the multiple action prediction network layers and the current policy data of the evaluation prediction network layer according to the target data corresponding to the treatment plan.

[0141] If the loss value of the target area corresponding to the current training is less than the loss value of the target area used when the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were updated last time, it means that the quality estimation of the treatment plan by the deep reinforcement learning model (that is, the prediction of the contribution value of each action set selection to the treatment plan) is accurate. In this case, the processor in the electronic device can execute S9.

[0142] S9. When the loss value of the target area corresponding to the current training is less than the dynamic loss value, based on the multiple dose distribution state data obtained in the current training, the multiple action sets corresponding to the multiple dose distribution state data, the multiple actual reward values ​​corresponding to the multiple action sets, the multiple predicted values ​​corresponding to the multiple action sets and the actual reward value of the target area, update the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer.

[0143] In one achievable manner, the above-mentioned method for updating current strategy data of multiple action prediction network layers and current strategy data of evaluation prediction network layers based on multiple dose distribution state data obtained from the current training, multiple action sets corresponding one-to-one to the multiple dose distribution state data, multiple actual reward values ​​corresponding one-to-one to the multiple action sets, multiple predicted values ​​corresponding one-to-one to the multiple action sets and the actual reward value of the target area specifically includes: determining the actual cumulative reward values ​​of multiple action sets based on the multiple actual reward values ​​corresponding one-to-one to the multiple action sets and the actual reward value of the target area, and updating the current strategy data of multiple action prediction network layers based on the multiple dose distribution state data, the multiple action sets corresponding one-to-one to the multiple dose distribution state data and the actual cumulative reward values ​​of the multiple action sets, and updating the current strategy data of the evaluation prediction network layer based on the multiple dose distribution state data, the multiple action sets corresponding one-to-one to the multiple dose distribution state data and the multiple predicted values ​​corresponding one-to-one to the multiple action sets.

[0144] Specifically, since the quality of the treatment plan for the target area is the result of the joint action of multiple action sets. Under a certain dose distribution state of the target area, the actual reward value of the selected action set is high, which does not mean that the quality of the treatment plan for the entire target area will be good. If the quality of the treatment plan for the final target area is also very good, the reference value of the selection of the action set will be very high. If the quality of the treatment plan for the final target area is not good, the reference value of the selection of the action set will become smaller. Therefore, after obtaining multiple action sets corresponding to multiple dose distribution state data, the actual cumulative reward values ​​of the multiple action sets can also be determined according to the multiple actual reward values ​​corresponding to the multiple action sets and the actual reward value of the target area.

[0145] The actual cumulative reward value may include the actual reward value of each action set and the delayed reward value after the entire treatment plan is completed. The delayed reward value is determined based on the actual reward value of the target area. That is, after the treatment plan for the target area is completed, the actual reward value of the target area is determined according to the dose distribution of the treatment plan, and the actual reward value is distributed to each action set that constitutes the treatment plan according to the weight to form the delayed reward value of the action set; the actual reward value and the delayed reward value of each action set are accumulated to obtain the actual cumulative reward value of each action set.

[0146] When the actual cumulative reward value is high, it is said that the reference value of the action set selected in the dose distribution state is high. In this case, the current strategy data of the multiple action prediction network layers can be updated based on the multiple dose distribution state data, the multiple action sets corresponding to the multiple dose distribution state data, and the actual cumulative reward values ​​of the multiple action sets.

[0147] At the same time, since the loss value of the target area is obtained based on the predicted value of the target area, and the predicted value of the target area is the accumulation of the predicted values ​​of multiple action sets, when the loss value of the target area is less than the dynamic loss value, it means that the predicted values ​​of multiple action sets have a higher reference value. In this case, the current strategy data of the evaluation prediction network layer can be updated based on multiple dose distribution state data, multiple action sets corresponding to multiple dose distribution state data, and multiple predicted values ​​corresponding to multiple action sets.

[0148] S10. Iterate the above training process until the deep reinforcement learning model is trained a preset number of times, thereby obtaining a trained deep reinforcement learning model.

[0149] For the specific implementation process of this step, please refer to the description of S204, which will not be repeated here.

[0150] In some embodiments, the multiple action prediction network layers include: at least two of: a target size prediction network layer, a target position prediction network layer, and a target weight prediction network layer. When the multiple action prediction network layers include a first action prediction network layer and a second action prediction network layer deployed from top to bottom, in the above S1, based on the current dose distribution state data of the target area and the current strategy data of each action prediction network layer, the action set corresponding to the current dose distribution state data output by the multiple action prediction network layers is determined, including:

[0151] Based on the current dose distribution state data and the current strategy data of the first action prediction network layer, a first action is determined; based on the current dose distribution state data, the first action and the current strategy data of the second action prediction network layer, a second action corresponding to the first action is determined.

[0152] Optionally, the first action prediction network layer and the second action prediction network layer can be any two different action prediction network layers among an action prediction network layer for selecting a target size, an action prediction network layer for selecting a target position, and an action prediction network layer for selecting a target weight.

[0153] Correspondingly, the first action and the second action may be any two different actions among the actions of selecting the target size, selecting the target position, and selecting the target weight when placing the target under the current dose distribution state data.

[0154] It should be noted that the first action prediction network layer corresponds to the first action, and the second action prediction network layer corresponds to the second action. For example, when the first action prediction network layer is an action prediction network layer for selecting the target size, the first action is an action for selecting the target size.

[0155] Exemplarily, assuming that the first action prediction network layer is an action prediction network layer for selecting target size, and the second action prediction network layer is an action prediction network layer for selecting target weight, then the first action is an action for selecting target size, and the second action is an action for selecting target weight.

[0156] When placing a target under the current dose distribution state data, the processor in the electronic device can input the current dose distribution state data into the first action prediction network layer for selecting the target size. The first action prediction network layer can determine the action of selecting the target size when placing the target under the current dose distribution state data according to its current strategy data, that is, determine the target size when placing the target under the current dose distribution state data.

[0157] In one achievable method, after determining the target size when the target is placed under the current dose distribution state data, the processor in the electronic device can also call a shape matching algorithm to determine the target position of the target corresponding to the target size within the target area based on the target size when the target is placed under the current dose distribution state data.

[0158] Next, the processor in the electronic device can input the current dose distribution state data and the action of selecting the target size when placing the target under the current dose distribution state data into the second action prediction network layer for selecting the target weight. The second action prediction network layer can determine the action of selecting the target weight when placing the target under the current dose distribution state data according to its current strategy data, that is, determine the target weight when placing the target under the current dose distribution state data.

[0159] In this way, the processor in the electronic device can obtain an action set consisting of the first action and the second action.

[0160] It can be understood that when the multiple action prediction network layers include two layers, the deep reinforcement learning model can only generate two types of target parameters. When the types of target parameters exceed two, the determination of the remaining types of target parameters can be performed by calling the corresponding algorithm through the processor in the electronic device to determine the remaining types of target parameters.

[0161] In some further embodiments, when the multiple action prediction network layers include a first action prediction network layer, a second action prediction network layer, and a third action prediction network layer deployed from top to bottom, in the above S1, based on the current dose distribution state data of the target area and the current strategy data of each action prediction network layer, determining the action set corresponding to the current dose distribution state data output by the multiple action prediction network layers includes:

[0162] Based on the current dose distribution state data and the current strategy data of the first action prediction network layer, determine the first action; based on the current dose distribution state data, the first action and the current strategy data of the second action prediction network layer, determine the second action corresponding to the first action; based on the current dose distribution state data, the first action, the second action and the current strategy data of the third action prediction network layer, determine the third action corresponding to the first action.

[0163] Optionally, the first action prediction network layer, the second action prediction network layer, and the third action prediction network layer can be different action prediction network layers among the action prediction network layers for selecting target size, the action prediction network layers for selecting target position, and the action prediction network layers for selecting target weight.

[0164] Correspondingly, the first action, the second action and the third action may be different actions among the actions of selecting the target size, selecting the target position and selecting the target weight when placing the target under the current dose distribution state data.

[0165] It should be noted that the first action prediction network layer corresponds to the first action, the second action prediction network layer corresponds to the second action, and the third action prediction network layer corresponds to the third action. For example, when the first action prediction network layer is an action prediction network layer for selecting the target size, the first action is an action for selecting the target size.

[0166] Exemplarily, assuming that the first action prediction network layer is an action prediction network layer for selecting target size, the second action prediction network layer is an action prediction network layer for selecting target position, and the third action prediction network layer is an action prediction network layer for selecting target weight, then the first action is an action for selecting target size, the second action is an action for selecting target position, and the third action is an action for selecting target weight.

[0167] When placing a target under the current dose distribution state data, the processor in the electronic device can input the current dose distribution state data into the first action prediction network layer for selecting the target size. The first action prediction network layer can determine the action of selecting the target size when placing the target under the current dose distribution state data according to its current strategy data, that is, determine the target size when placing the target under the current dose distribution state data.

[0168] Next, the processor in the electronic device can input the current dose distribution state data and the action of selecting the target size when placing the target under the current dose distribution state data into the second action prediction network layer for selecting the target position. The second action prediction network layer can determine the action of selecting the target position when placing the target under the current dose distribution state data according to its current strategy data, that is, determine the target position when placing the target under the current dose distribution state data.

[0169] Next, the processor in the electronic device can input the current dose distribution state data, the action of selecting the target size when placing the target under the current dose distribution state data, and the action of selecting the target position when placing the target under the current dose distribution state data into the third action prediction network layer for selecting the target weight. The third action prediction network layer can determine the action of selecting the target weight when placing the target under the current dose distribution state data according to its current strategy data, that is, determine the target weight when placing the target under the current dose distribution state data.

[0170] In this way, the processor in the electronic device can obtain an action set consisting of the first action, the second action and the third action.

[0171] Next, the functions of electronic devices in implementing treatment plans are introduced.

[0172] Figure 4 FIG. 1 is a flowchart of a method for implementing a treatment plan by a processor according to an embodiment of the present disclosure. Figure 4 As shown, the method includes:

[0173] S401, acquiring image data and contour data of the target area to be treated.

[0174] The specific process of obtaining the image data and contour data of the target area to be treated can refer to the specific process of obtaining the image data and contour data of different target areas in S201, which will not be repeated here.

[0175] For example, Figure 4 As shown, the image data of the target area to be treated may be a CT image of the head of the subject to be treated, and the contour data may be contour data of the target area to be treated in the head of the subject to be treated.

[0176] S402: Determine dose distribution state data of the target area to be treated based on the image data and contour data of the target area to be treated.

[0177] The specific process of determining the dose distribution state data of the target area to be treated may refer to the specific process of determining the initial dose distribution state data of the target area to be treated in S201, which will not be described in detail here.

[0178] S403, inputting the dose distribution state data of the target area to be treated into the deep reinforcement learning model to obtain a target parameter combination of the target area to be treated consisting of multiple different types of target parameters.

[0179] Among them, the deep reinforcement learning model is based on Figure 2 or Figure 3 The model training method shown is trained.

[0180] For example, Figure 4 As shown, the target parameter combination composed of multiple different types of target parameters of the target area to be treated output by the deep reinforcement learning model may include: a target parameter combination composed of at least two of target size, target position and target weight.

[0181] S404: Generate a treatment plan for the target area to be treated according to the target point parameter combination.

[0182] It should be noted that when the above-mentioned target parameter combination includes two types of target parameters (for example, target size and target weight), the processor can determine the remaining target parameters that constitute the treatment plan by calling relevant algorithms (for example, calling a position matching algorithm to determine the target position based on the target parameter combination consisting of target size and target weight) to generate a treatment plan.

[0183] Now, taking the target parameter combination including target size, target position and target weight as an example, a process description of a processor implementing a treatment planning method is performed. It is understandable that the target parameter combination can also be at least two of the target size, target position and target weight.

[0184] First, the processor obtains image data and contour data of the target area to be treated, and then the processor determines the dose distribution state data of the target area to be treated according to the image data and contour data of the target area to be treated, and inputs the dose distribution state data of the target area to be treated into the Figure 2 or Figure 3 In the deep reinforcement learning model trained by the model training method shown, the target size, target position and target weight of the first target output by the deep reinforcement learning model are obtained. That is, the relevant parameters of the first target in the treatment plan are determined.

[0185] Next, the processor calculates the dose distribution of the target area to be treated based on the relevant parameters of the first target, determines the current dose distribution state data of the target area to be treated based on the dose distribution of the target area to be treated, and inputs the current dose distribution state data of the target area to be treated into the deep reinforcement learning model to obtain the target size, target position and target weight of the second target output by the deep reinforcement learning model.

[0186] The above target size, target position and target weight determination process is iterated repeatedly to determine the relevant parameters of the remaining targets until the dose distribution of the target area to be treated meets the prescribed dose and / or the number of targets in the target area to be treated is equal to the preset maximum number of targets, then the target parameter combination of the target area to be treated is determined, and a treatment plan is generated according to the target parameter combination. The target parameter combination of the target area to be treated includes the relevant parameters of multiple targets, namely, multiple sets of target sizes (S), target positions (P) and target weights (W) {(S1, P1, W1), (S2, P2, W2), ... (S n ,P n ,W n )}.

[0187] It is understandable that the above target weights may not be obtained through a deep reinforcement learning model and may be set to 1 by default.

[0188] The treatment plan can be displayed in the form of a display of an image corresponding to the image data of the object to be treated. In the image corresponding to the image data of the object to be treated, a target parameter combination of the target area to be treated can be clearly marked.

[0189] Optionally, after determining the treatment plan for the target area to be treated, the physicist (manual) may also adjust each target parameter in the target parameter combination of the treatment plan for the target area to be treated based on experience, thereby determining the treatment plan for the target area to be treated based on the adjusted target parameter combination.

[0190] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of a computer system. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in conjunction with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0191] The embodiment of the present application can divide the functional modules according to the above-mentioned computer system. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0192] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a treatment plan provided by the present disclosure.

[0193] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable an electronic device to execute the method for generating a treatment plan provided by the present disclosure.

[0194] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the deep reinforcement learning model training method and treatment planning method provided by the present disclosure for generating a treatment plan.

[0195] In some embodiments, the electronic device may be the above Figure 1 The electronic device a2 or b2 shown in FIG. Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 500 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0196] like Figure 5As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0197] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0198] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the method for generating a treatment plan. For example, in some embodiments, the method for generating a treatment plan may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method for generating a treatment plan described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method for generating a treatment plan in any other appropriate manner (eg, by means of firmware).

[0199] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0200] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0201] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0202] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0203] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0204] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0205] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document is not limited here.

[0206] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A deep reinforcement learning model training method for generating a treatment plan, the deep reinforcement learning model being configured to include: A plurality of action prediction network layers and evaluation prediction network layers, different action prediction network layers are used to output different types of target parameters included in the treatment plan, characterized in that the method comprises: Obtaining initial dose distribution state data of the target area; Determining target data based on the initial dose distribution state data of the target area, the current strategy data of the multiple action prediction network layers, and the current strategy data of the evaluation prediction network layer; Based on the target data, updating the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer to complete the current training of the deep reinforcement learning model; Iterate the above training process until the number of training times for the deep reinforcement learning model reaches a preset number, thereby obtaining the trained deep reinforcement learning model; Among them, the target data includes: the final dose distribution and multiple dose distribution status data of the target area, multiple action sets corresponding one-to-one to the multiple dose distribution status data output by the multiple action prediction network layers, multiple predicted values ​​corresponding one-to-one to the multiple action sets output by the evaluation prediction network layer, multiple actual reward values ​​corresponding one-to-one to the multiple action sets, and the predicted value and actual reward value of the target area; each action set in the multiple action sets includes a target parameter combination composed of multiple different types of target parameters.

2. The method according to claim 1, characterized in that: The multiple action prediction network layers include: at least two of a target size prediction network layer, a target position prediction network layer, and a target weight prediction network layer.

3. The method according to claim 2, characterized in that The determining of target data based on the initial dose distribution state data of the target area, the current strategy data of the plurality of action prediction network layers, and the current strategy data of the evaluation prediction network layer includes: Based on the current dose distribution state data of the target area and the current strategy data of each of the action prediction network layers, determine the action set corresponding to the current dose distribution state data output by the multiple action prediction network layers, and based on the current dose distribution state data of the target area and the current strategy data of the evaluation prediction network layer, determine the prediction value corresponding to the action set output by the evaluation prediction network layer; Determining the dose distribution of the target area based on the action set corresponding to the current dose distribution state data, and determining the actual reward value corresponding to the action set based on the dose distribution of the target area; When the dose distribution of the target target area does not satisfy the preset prescription dose and the number of target points in the target target area is less than the preset maximum number of target points, updating the current dose distribution state data of the target target area based on the dose distribution of the target target area; When the dose distribution of the target target area satisfies the preset prescription dose, and / or the number of target points in the target target area is equal to the preset maximum number of target points, determine the final dose distribution of the target target area and multiple dose distribution state data of the target target area, multiple action sets corresponding to the multiple dose distribution state data output by the multiple action prediction network layers, multiple prediction values ​​corresponding to the multiple action sets output by the evaluation prediction network layer, and multiple actual reward values ​​corresponding to the multiple action sets; Based on the final dose distribution of the target area and the multiple predicted values ​​corresponding to the multiple action sets, the actual reward value and the predicted value of the target area are determined.

4. The method according to claim 3, characterized in that When the multiple action prediction network layers include a first action prediction network layer and a second action prediction network layer deployed from top to bottom, the step of determining, based on the current dose distribution state data of the target area and the current strategy data of each of the action prediction network layers, a set of actions corresponding to the current dose distribution state data output by the multiple action prediction network layers includes: Determining a first action based on the current dose distribution state data and current strategy data of the first action prediction network layer; Based on the current dose distribution state data, the first action and the current strategy data of the second action prediction network layer, a second action corresponding to the first action is determined.

5. The method according to claim 3, characterized in that: When the multiple action prediction network layers include a first action prediction network layer, a second action prediction network layer, and a third action prediction network layer deployed from top to bottom, the action set corresponding to the current dose distribution state data output by the multiple action prediction network layers based on the current dose distribution state data of the target area and the current strategy data of each of the action prediction network layers is determined, including: Determining the first action based on the current dose distribution state data and current strategy data of the first action prediction network layer; Determine a second action corresponding to the first action based on the current dose distribution state data, the first action, and the current strategy data of the second action prediction network layer; Based on the current dose distribution state data, the first action, the second action and the third action prediction network layer's current strategy data, a third action corresponding to the first action is determined.

6. The method according to claim 1, characterized in that The updating of the current strategy data of the plurality of action prediction network layers and the current strategy data of the evaluation prediction network layer based on the target data includes: When the final dose distribution of the target target area obtained by the current training meets the preset prescription dose, determining whether the actual reward value of the target target area obtained by the current training is greater than a dynamic reward threshold; the dynamic reward threshold is the actual reward value of the target target area used when the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated; When the actual reward value of the target target area obtained in the current training is greater than the dynamic reward threshold, determining the loss value of the target target area corresponding to the current training based on the actual reward value and the predicted value of the target target area obtained in the current training; When the loss value of the target target area corresponding to the current training is less than the dynamic loss value, the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer are updated based on the multiple dose distribution state data obtained from the current training, the multiple action sets corresponding one-to-one to the multiple dose distribution state data, the multiple actual reward values ​​corresponding one-to-one to the multiple action sets, the multiple prediction values ​​corresponding one-to-one to the multiple action sets and the actual reward value of the target target area; the dynamic loss value is the loss value of the target target area used when the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer were last updated.

7. The method according to claim 6, characterized in that The updating of the current strategy data of the multiple action prediction network layers and the current strategy data of the evaluation prediction network layer based on the multiple dose distribution state data obtained by the current training, the multiple action sets corresponding one-to-one to the multiple dose distribution state data, the multiple actual reward values ​​corresponding one-to-one to the multiple action sets, the multiple predicted values ​​corresponding one-to-one to the multiple action sets and the actual reward value of the target area comprises: Determine the actual cumulative reward value of the multiple action sets based on the multiple actual reward values ​​corresponding one-to-one to the multiple action sets and the actual reward value of the target area, and update the current strategy data of the multiple action prediction network layers based on the multiple dose distribution state data, the multiple action sets corresponding one-to-one to the multiple dose distribution state data and the actual cumulative reward values ​​of the multiple action sets; Based on the multiple dose distribution state data, the multiple action sets corresponding one-to-one to the multiple dose distribution state data, and the multiple prediction values ​​corresponding one-to-one to the multiple action sets, the current strategy data of the evaluation prediction network layer is updated.

8. A treatment planning method, characterized in that: The method comprises: Acquire image data and contour data of the target area to be treated; Determining dose distribution state data of the target area to be treated based on the image data and contour data of the target area to be treated; Inputting the dose distribution state data of the target area to be treated into a deep reinforcement learning model to obtain a target parameter combination of the target area to be treated consisting of a plurality of different types of target parameters; the deep reinforcement learning model is trained according to the deep reinforcement learning model training method for generating a treatment plan according to any one of claims 1 to 7; A treatment plan for the target area to be treated is generated according to the target point parameter combination.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

10. A non-volatile readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 8.

Images