Radiotherapy plan generation method and system based on large model

Abstract

The invention discloses a radiotherapy plan generation method and system based on a large model. The method comprises the steps that medical images such as CT, MRI and PET-CT are acquired; constructing a plan generation model which comprises a feature extraction module, a sparse attention module and a large language model; the feature extraction module is used for extracting morphological features of tumors and organs at risk from the medical image to obtain image features; the sparse attention module is used for dynamically fusing the image features and the text data and generating interactive feature vectors; the large language model is used for generating radiotherapy plan parameters according to the fused feature vectors; and inputting the actual medical image into the plan generation model, and outputting radiotherapy plan parameters. The system comprises an image acquisition unit, a model construction unit and an application unit. According to the invention, the consistency and the accuracy of automatically generating the radiation treatment plan are improved. The method can be widely applied to the field of treatment plan design.

Description

Technical Field

[0001] The present invention relates to the field of treatment planning design, and particularly to a method and system for generating radiotherapy plans based on large models. Background Art

[0002] Radiotherapy is a treatment method that uses high-energy rays (such as X-rays, γ-rays, protons, electrons, etc.) to damage the DNA of cancer cells and inhibit their growth and division. Radiotherapy is currently one of the main means of cancer treatment. The existing intensity modulated radiation therapy (IMRT) technology has now become the main radiotherapy method, and the planning of radiotherapy plans is particularly important. Optimizing the irradiation plan to ensure that the tumor receives sufficient dose while protecting healthy tissues. Currently, the process of formulating radiotherapy plans is cumbersome, time-consuming, and highly dependent on manual experience. Summary of the Invention

[0003] In view of this, in order to solve the technical problem that most of the existing radiotherapy plan generation methods rely on manual editing, resulting in a cumbersome and time-consuming formulation process, on the one hand, the present invention proposes a method for generating radiotherapy plans based on large models, and proposes a three-stage fusion architecture. The method includes the following steps:

[0004] Obtain medical images such as CT, MRI, PET-CT;

[0005] Construct a plan generation model, including a feature extraction module, a sparse attention module, and a large language model;

[0006] Among them, the feature extraction module is used to extract the morphological features of tumors and organs at risk from medical images, that is, to obtain image features; the sparse attention module is used to dynamically fuse image features and text data (such as ASTRO reports) and generate interactive feature vectors; the large language model is used to generate radiotherapy plan parameters according to the fused feature vectors.

[0007] Input the actual medical image into the plan generation model and output radiotherapy plan parameters.

[0008] Furthermore, define a reward function and use a deep reinforcement learning algorithm (such as PPO, DQN) to optimize the dose distribution and gradually approach the optimal radiotherapy plan.

[0009] Furthermore, the feature extraction module is a hybrid architecture of a graph convolutional network and a graph neural network, combining the advantages of CNN and GNN to capture local features and global spatial dependencies.

[0010] The present invention also proposes a system for generating radiotherapy plans based on large models. The system includes:

[0011] An image acquisition unit for acquiring medical images such as CT, MRI, PET-CT;

[0012] A model construction unit for constructing a plan generation model, including a feature extraction module, a sparse attention module, and a large language model;

[0013] Among them, the feature extraction module is used to extract the morphological features of tumors and organs at risk from medical images, that is, to obtain image features; the sparse attention module is used to dynamically fuse image features and text data (such as ASTRO reports) and generate interactive feature vectors; the large language model is used to generate radiotherapy plan parameters according to the fused feature vectors.

[0014] An application unit for inputting actual medical images into the plan generation model and outputting radiotherapy plan parameters.

[0015] Based on the above solution, the present invention provides a large model-based radiotherapy plan generation method and system, which provides a new technical path for radiotherapy plan generation through multi-modal feature fusion and reinforcement learning optimization. This architecture not only improves the accuracy and personalization level of radiotherapy plans, but also lays a foundation for the in-depth application of artificial intelligence in the field of radiotherapy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of the steps of a large model-based radiotherapy plan generation method of the present invention;

[0017] Figure 2 is a structural block diagram of a large model-based radiotherapy plan generation system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In recent years, the progress of computer science and radiation oncology has promoted the application of knowledge-based planning (KBP) methods in DVH prediction. The KBP method: utilizes a previous radiotherapy plan database and draws on the planning experience of doctors and physicists. For example: Commercial tools such as RapidPlan have been widely used in clinical and research, but this method is limited by the quality and scale of the database, difficult to cover all patient situations, and the adaptability to new cases needs to be improved.

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0020] It should be noted that for ease of description, only parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0021] It should be understood that the "system", "device", "unit" and / or "module" used in the present application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0022] As shown in the present application and the claims, unless the context clearly indicates an exception, words such as "a", "an", "one" and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. An element defined by the statement "comprising one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0023] In the description of the embodiments of the present application, "a plurality" means two or more than two. The following terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0024] In addition, flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the previous or subsequent operations do not necessarily need to be executed precisely in sequence. On the contrary, the steps can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or more steps can be removed from these processes.

[0025] Referring to Figure 1 , which is a schematic flowchart of an optional example of the radiotherapy plan generation method based on a large model proposed by the present invention. This method can be applied to a computer device. The generation method proposed in this embodiment may include but is not limited to the following steps:

[0026] Step S1, obtaining medical images;

[0027] Step S2: Construct a plan generation model based on a feature extraction module, a sparse attention module, and a pre-trained large language model; wherein, the feature extraction module is used to extract features from the medical image to obtain image features; the sparse attention module is used to fuse and process the image features and text data to obtain a dynamic interactive feature vector; the pre-trained large language model is used to process the dynamic interactive feature vector to generate radiotherapy plan parameters.

[0028] Step S3: Input the medical image into the plan generation model and output radiotherapy plan parameters.

[0029] In some feasible embodiments, it further includes:

[0030] Optimize the dose distribution through Reinforcement Learning.

[0031] Define a reward function, comprehensively consider the Tumor Control Probability (TCP) and the Normal Tissue Complication Probability (NTCP), and gradually approximate the optimal radiotherapy plan in combination with the reinforcement learning algorithm.

[0032] In some feasible embodiments, the feature extraction module specifically includes a graph convolutional network and a graph neural network, where:

[0033] Graph data construction, objective: Model the anatomical structures in the medical image as graph data. Node definition: Each organ (such as the prostate, rectum, bladder) is used as a node. Node features include the morphological features of the organ (such as volume, shape), position information (such as central coordinates), and dose distribution features. Edge definition: Construct edges based on the spatial relationships between organs (such as distance, contact area). The edge weights can reflect the interaction intensity between organs (such as the adjacent relationship between the prostate and the rectum). Graph structure: Construct an undirected graph or a directed graph to represent the topological relationship between organs.

[0034] Combine the advantages of CNN and GNN to capture local features and global spatial dependencies. CNN module: Use 3D CNN to extract local features of organs (such as texture, edges) from the medical image. Output a high-dimensional feature map as a supplement to the node features. GNN module: Use a graph convolutional network (GCN) or a graph attention network (GAT) to process the graph data. Aggregate neighbor node information through a message passing mechanism to capture the spatial interaction between organs. Feature fusion: Fuse the local features extracted by CNN and the global features extracted by GNN (such as concatenation, weighted summation). Output the fused feature vector for dose distribution prediction.

[0035] Through this specific embodiment, the spatial relationships between organs are explicitly modeled by GNN, making up for the deficiencies of CNN in global feature extraction. For example, in prostate cancer radiotherapy, GNN can capture the adjacent relationships between the prostate, rectum, and bladder, and optimize the dose distribution to reduce damage to the rectum and bladder. Using the message passing mechanism of GNN, the spatial interactions between organs (such as distance and contact area) are analyzed. For example, in the adjacent area between the prostate and rectum, GNN can dynamically adjust the dose distribution to ensure that the tumor area receives sufficient dose while protecting the rectum.

[0036] In some feasible embodiments, after CNN feature extraction, a Sparse Attention Module is used to replace the standard Transformer, focusing only on key regions related to radiotherapy planning (such as tumor target areas and high-risk critical organs), reducing computational redundancy. Its formula is as follows:

[0037]

[0038] where Q is the query matrix, K is the key matrix, V is the value matrix, d k represents the correction factor, M represents the sparse mask matrix, and important feature regions are screened through a dynamic threshold.

[0039] In some feasible embodiments, the pre-training process of the large language model includes:

[0040] Based on radiotherapy domain knowledge (such as NRG, ASTRO guidelines, radiotherapy dosimetry literature), domain-adaptive pre-training is performed on LLMs to construct a medical large language model dedicated to radiotherapy planning (Radiotherapy-Specific LLM, RT-LLM).

[0041] Data collection: Data sources: Clinical guidelines: Radiotherapy guidelines published by NRG Oncology, ASTRO, NCCN, etc. Scientific literature: Relevant papers on radiotherapy dosimetry, radiobiology, clinical research, etc. (such as PubMed, IEEE Xplore). Textbooks: Classic textbooks and reference books in the field of radiotherapy. Clinical reports: Radiotherapy planning reports, treatment summaries, follow-up records, etc. Dosimetry data: Data such as dose distribution, tumor control probability (TCP), and normal tissue complication probability (NTCP).

[0042] Data preprocessing: Text cleaning: Remove irrelevant content (such as references and chart descriptions). Word segmentation and annotation: Use word segmentation tools in the medical field (such as MetaMap) for term recognition and annotation. Structured processing: Convert unstructured text (such as guidelines and literature) into structured data (such as JSON, XML).

[0043] Domain Adaptation Pretraining: Based on general LLMs, perform domain adaptation pre-training with radiotherapy domain data to construct RT-LLM. Selection of the base model: Select open-source large language models (such as LLaMA, GPT-3, BERT) as the base model. Pretraining method: Continue training the base model on radiotherapy domain data to make it adapt to the language patterns and knowledge in the radiotherapy domain. Domain-Adaptive Fine-tuning: Use tasks in the radiotherapy domain (such as radiotherapy plan generation, dose prediction) for fine-tuning.

[0044] In some feasible embodiments, it also includes evaluating the plan generation model:

[0045] Select three large language models (LLMs), namely ChatGPT 3.5, ChatGPT4.0, and Google Bard.

[0046] Intra-LLM Agreement: Three different large language models were independently evaluated three times each. The first model is Google Bard, which was evaluated for the First, Second, and Third times. The second model is ChatGPT 3.5, which was also evaluated three times. The third model is ChatGPT 4.0, which was also evaluated three times. The three evaluation results of each model will be used to calculate the KappaValue (Kappa value) to measure the consistency degree of each model itself in multiple evaluations. The higher the Kappa value, the more stable and reliable the model's judgment for the same data is.

[0047] Inter-LLM Agreement: By comparing the output results of these three models, calculate the Kappa Value between them to evaluate the consistency degree of different large language models for the plan design of the same data. This can help understand the similarities and differences among different large language models when dealing with related tasks.

[0048] Specifically, for the same case, input the plan parameters generated by Google Bard, ChatGPT 3.5 / 4.0 respectively to generate 3 groups of plan parameters. Three senior physicists blindly evaluate the plan rationality, and use the majority voting method to determine the final negotiation result.

[0049] Calculate the Cohen's Kappa coefficient between different models:

[0050]

[0051] Among them, Po is the observed agreement, and Pe is the expected agreement.

[0052] The evaluation performance metrics include: AUC (Area Under Curve): used to comprehensively evaluate the accuracy of the model; Accuracy: the proportion of correctly designed ones; Sensitivity: true positive rate, the ability of the model to design correct radiotherapy plans; Specificity: true negative rate.

[0053] The beneficial effects of the present invention specifically include:

[0054] Improving model performance and reliability: Consistency evaluation can discover the stability and reliability issues of large language models in the automatic planning process, provide directions for model improvement and optimization, and make them more reliable in practical applications. Performance evaluation can screen out models or strategy combinations with better performance, further improving the accuracy of automatic planning.

[0055] Promoting the combination of medicine and artificial intelligence: This research provides practical experience and reference cases for the combination of the medical field and artificial intelligence technology, helps to promote the development of related research such as medical image analysis and the application of artificial intelligence in radiotherapy, promotes the intelligent process of the medical industry, and provides new ideas and technical support for future radiotherapy new technologies.

[0056] Assisting doctors in decision-making and training: These strategies and evaluation results can assist doctors, especially inexperienced junior doctors, in making decisions, provide reference bases, and can also provide materials and cases for doctors' training, helping them better understand and apply advanced planning design techniques and methods, and improving the overall medical level.

[0057] Such as Figure 2 shown, a radiotherapy plan generation system based on a large model includes:

[0058] An image acquisition unit for acquiring medical images;

[0059] A model construction unit for constructing a plan generation model based on a feature extraction module, a sparse attention module, and a pre-trained large language model;

[0060] An application unit for inputting the medical images into the plan generation model and outputting radiotherapy plan parameters.

[0061] The content in the above method embodiments is applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0062] A radiotherapy plan generation device based on a large model:

[0063] At least one processor;

[0064] At least one memory for storing at least one program;

[0065] When the at least one program is executed by the at least one processor, the at least one processor implements a method for generating a radiotherapy plan based on a large model as described above.

[0066] The content in the above method embodiments is applicable to the device embodiments of the present invention. The functions specifically implemented by the device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0067] A storage medium storing processor-executable instructions, the processor-executable instructions being used to implement a method for generating a radiotherapy plan based on a large model as described above when executed by a processor.

[0068] The content in the above method embodiments is applicable to the storage medium embodiments of the present invention. The functions specifically implemented by the storage medium embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0069] The above is a specific description of the preferred embodiments of the present invention. However, the present invention is not limited to the described embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for generating a radiotherapy plan based on a large model, characterized in that: The following steps are involved: Acquiring medical images; Build a plan generation model based on feature extraction module, sparse attention module and pre-trained large language model; inputting the medical image into the plan generation model; Extracting features from the medical image using the feature extraction module to obtain image features; Based on the sparse attention module, the image features and text data are fused to obtain a dynamic interactive feature vector; The dynamic interactive feature vector is processed based on the pre-trained large language model to generate radiotherapy plan parameters.

2. A method for generating a radiotherapy plan based on a large model according to claim 1, characterized in that: Also includes: The dose distribution in the radiotherapy planning parameters is optimized by reinforcement learning.

3. A method and system for generating a radiation therapy plan based on a large model according to claim 1, characterized in that: The feature extraction module is a hybrid architecture of graph convolutional network and graph neural network, where: The spatial interaction between organs is analyzed by the graph neural network, and the features of the medical image are extracted in combination with the graph convolutional network.

4. The method for generating a radiation therapy plan based on a large model according to claim 1, characterized in that: The formula of the sparse attention module is as follows: Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k represents the correction factor, and M represents the sparse mask matrix.

5. The method for generating a radiation therapy plan based on a large model according to claim 1, characterized in that: Also includes: The performance of the plan generation model was evaluated using the area under the curve, accuracy, sensitivity and specificity as evaluation indicators.

6. The method for generating a radiation therapy plan based on a large model according to claim 1, characterized in that: The training process of the large language model specifically includes: Obtain text data in the field of radiotherapy; Cleaning, segmenting, annotating and structuring the text data to obtain preprocessed text data; Select the base model; The basic model is trained for domain adaptability based on the preprocessed text data, and fine-tuned in combination with a preset radiotherapy task to obtain a pre-trained large language model.

7. A method for generating a radiation therapy plan based on a large model according to claim 6, characterized in that: Also includes: Select different large language models and calculate the Cohen's Kappa coefficient between different models to complete the consistency assessment.

8. A radiation therapy plan generation system based on a large model, characterized in that: include: An image acquisition unit, used for acquiring medical images; A model building unit, which builds a plan generation model based on a feature extraction module, a sparse attention module and a pre-trained large language model; extracts features from the medical image through the feature extraction module to obtain image features; Based on the sparse attention module, the image features and text data are fused to obtain a dynamic interactive feature vector; based on the pre-trained large language model, the dynamic interactive feature vector is processed to generate radiotherapy plan parameters. An application unit inputs the medical image into the plan generation model.

9. A radiotherapy plan generation device based on a large model, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the large model-based radiation therapy plan generation method as described in any one of claims 1 to 7.

Images