Multi-modal image fusion method and system for improving radiotherapy accuracy
Through multimodal image fusion method and deep learning algorithm, the problem of insufficient positioning accuracy of traditional radiotherapy is solved, high-accuracy fusion of images and adaptive optimization of radiotherapy plans are achieved, and the accuracy and efficiency of radiotherapy are improved.
Patent Information
- Application Number
- CN202510153027.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-12
- Publication Date
- 2025-05-13
AI Technical Summary
Traditional radiotherapy positioning relies on single-modal images, making it difficult to fully and accurately reflect the anatomical structure and functional information of the tumor, affecting the accuracy of radiotherapy, and cannot systematically give the optimal radiotherapy plan, resulting in doctors needing to give the plan based on experience, with certain deviations.
A multimodal image fusion method is adopted to establish an image fusion model based on deep learning algorithms, fuse images of different modes, handle complex nonlinear mappings, improve the accuracy and robustness of image registration fusion, and establish a radiotherapy planning model, introduce deep learning algorithms, and adaptively adjust the radiotherapy dosage and direction.
It improves the accuracy and personalization of radiotherapy, optimizes the radiotherapy planning process, reduces the side effects and risks of treatment, shortens the treatment preparation time, and improves the treatment efficiency.
Smart Images
Figure CN119971342A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of radiotherapy image processing, and in particular to a multimodal image fusion method and system for improving radiotherapy accuracy. Background Art
[0002] Radiotherapy is one of the important means of tumor treatment, and its therapeutic effect depends largely on the accuracy of radiotherapy. Traditional radiotherapy positioning mainly relies on single-modality images, such as CT or MRI, but single-modality images are often difficult to fully and accurately reflect the anatomical structure and functional information of the tumor, thereby affecting the accuracy of radiotherapy. At the same time, it is currently impossible to systematically give the optimal radiotherapy plan, and doctors still need to give radiotherapy plans based on their own experience. This method will also have certain deviations. Therefore, the present invention proposes a multi-modality image fusion method and system for improving the accuracy of radiotherapy to solve the problems existing in the prior art. Summary of the invention
[0003] In response to the above problems, the purpose of the present invention is to propose a multimodal image fusion method and system for improving the accuracy of radiotherapy. The multimodal image fusion method and system for improving the accuracy of radiotherapy establish an image fusion model based on a deep learning algorithm to fuse images of different modalities. It can handle complex nonlinear mappings and improve the accuracy and robustness of image registration and fusion. At the same time, a radiotherapy planning model is established and a deep learning algorithm is introduced. The radiotherapy dose and direction can be adaptively adjusted to ensure that the best radiotherapy plan is always provided throughout the treatment process, reduce the side effects and risks of treatment, shorten the treatment preparation time, and improve the treatment efficiency.
[0004] To achieve the purpose of the present invention, the present invention is implemented by the following technical solution: a multimodal image fusion method for improving the accuracy of radiotherapy, comprising the following steps:
[0005] Step 1: Image acquisition and processing: First, different modality image data of the patient's tumor in different directions are obtained through CT, MRI and PET equipment, and then denoising and enhancing are performed respectively and annotated according to the acquisition direction, and finally sorted into different data sets;
[0006] The datasets include CT image datasets, MRI image datasets, and PET image datasets;
[0007] Step 2: Image fusion processing: establish an image fusion model based on a deep learning algorithm, use different data sets as input to the image fusion model, extract image features and perform feature relationship recognition, and finally use the feature relationship between images of different modalities to align and fuse the corresponding annotated images in different data sets to obtain a fused image set.
[0008] Step 3: Radiotherapy effect judgment: obtain radiotherapy data precedents of related tumors based on network big data, retrieve the patient's previous radiotherapy data and the fusion image set of the tumor, and judge the radiotherapy effect of the previous radiotherapy data by combining the radiotherapy data and tumor change graphics;
[0009] Step 4: Radiotherapy planning and adjustment. Based on machine learning technology, a radiotherapy planning model is established and a deep learning algorithm is introduced. The radiotherapy planning model is used in combination with the efficacy data in step 3 to intelligently plan the radiotherapy course. The fused image set obtained in step 2 and the previous radiotherapy data are used to analyze and optimize the radiotherapy dose distribution and radiotherapy range to obtain high-precision radiotherapy data.
[0010] Further improvements are: when acquiring image data in different directions in step one, the patient is asked to perform the same set of actions on different devices to ensure that the directions of the acquired tumor images are the same; the image denoising in step one uses the wavelet denoising method, and the image enhancement uses the Laplace enhancement method.
[0011] Further improvements are as follows: after the image fusion model in step 2 is established, images with good fusion quality and original images are obtained based on network big data to form training sets and verification sets for model training; the image fusion model fuses images by first using the convolutional neural network in the deep learning algorithm to extract features from the input image, then learning the spatiotemporal relationship between different modalities through a recurrent neural network, and finally using the spatiotemporal relationship and image features to align and fuse images of different modalities with the same annotations.
[0012] Further improvements are as follows: in step three, for patients who are undergoing radiotherapy for the first time, the corresponding radiotherapy data of other patients with the same type of tumor are obtained based on network big data for judgment; for patients with a history of multiple radiotherapy, the data of the patient's previous multiple radiotherapy treatments are obtained based on network big data for judgment.
[0013] Further improvements are as follows: after the radiotherapy planning model is established in step 4, radiotherapy data and effect information of the same type of tumors are obtained based on network big data to establish a training set and a validation set for training, and a deep learning algorithm is introduced into the analysis and optimization module of the trained radiotherapy planning model to adaptively train the radiotherapy planning model during the radiotherapy treatment.
[0014] A multimodal image fusion system for improving radiotherapy accuracy includes an image acquisition system and a radiotherapy planning system, wherein the image acquisition system includes an image acquisition module, an image processing module and an image fusion module, wherein the image acquisition module acquires different modal image data of a patient's tumor in different directions based on CT, MRI and PET equipment, wherein the image processing module is used to denoise and enhance the acquired images and organize them into corresponding data sets according to the type of imaging equipment, and wherein the image fusion module performs fusion processing on different modal image data with the same annotations in different data sets based on an image fusion model; wherein the radiotherapy planning system includes an efficacy judgment module, a radiotherapy planning module and an optimization adjustment module, wherein the efficacy judgment module acquires historical radiotherapy data based on network big data and judges the radiotherapy effect in combination with the radiotherapy dose and range, wherein the radiotherapy planning module performs a course planning for later radiotherapy based on the radiotherapy planning model in combination with the patient's physical condition and radiotherapy effect, and wherein the optimization adjustment module is used to adjust radiotherapy parameters in real time in combination with the changes in the patient's physical condition and tumor images during radiotherapy.
[0015] A further improvement is that the image processing module includes a denoising and enhancement submodule, an image annotation submodule and an image sorting submodule, the denoising and enhancement submodule performs denoising and enhancement processing on the acquired image based on wavelet denoising and Laplace enhancement method, the image annotation submodule is used to annotate the image according to the direction of the tumor in the action of image acquisition, and the image sorting submodule is used to classify and sort the processed images according to different types of acquired equipment.
[0016] Further improvements are: the image fusion module includes a feature extraction submodule, a spatiotemporal relationship recognition submodule and an alignment fusion submodule, the feature extraction submodule is used to extract tumor features from the input image, the spatiotemporal relationship recognition submodule recognizes the spatiotemporal relationship between the input images of different modalities based on the spatiotemporal relationship between the images of different modalities learned and trained, and the alignment fusion submodule is used to align and fuse tumor images of different modalities in the same direction according to the extracted features and spatiotemporal relationships.
[0017] Further improvements are as follows: the radiotherapy judgment module includes an initial efficacy judgment submodule and a historical efficacy judgment submodule. The initial efficacy judgment submodule predicts the efficacy of patients receiving initial radiotherapy based on network big data, and the historical efficacy judgment submodule predicts the efficacy of patients with a history of multiple radiotherapy based on network big data combined with historical data of previous radiotherapy.
[0018] The beneficial effects of the present invention are as follows: the present invention establishes an image fusion model based on a deep learning algorithm to fuse images of different modalities, can handle complex nonlinear mappings, and improve the accuracy and robustness of image registration and fusion. At the same time, a radiotherapy planning model is established and a deep learning algorithm is introduced, which can adaptively adjust the radiotherapy dose and direction to ensure that the best radiotherapy plan is always provided throughout the treatment process, improve the personalization level of radiotherapy, optimize the radiotherapy planning process, reduce the side effects and risks of treatment, shorten the treatment preparation time, and improve the treatment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is a flow chart of the method of Example 1 of the present invention.
[0020] Figure 2 This is a system architecture diagram of Example 2 of the present invention. DETAILED DESCRIPTION
[0021] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.
[0022] Example 1
[0023] according to Figure 1 As shown, this embodiment provides a multimodal image fusion method for improving radiotherapy accuracy, comprising the following steps:
[0024] Step 1: Image acquisition and processing. First, different modality image data of the patient's tumor in different directions are obtained through CT, MRI and PET equipment. When obtaining image data in different directions, the patient is asked to perform the same set of actions on different devices to ensure that the direction of the tumor image obtained is the same. Then, wavelet denoising and Laplace enhancement methods are used for denoising and enhancement processing, and the data are marked according to the acquisition direction, and finally organized into different data sets.
[0025] The datasets include CT image datasets, MRI image datasets, and PET image datasets.
[0026] Step 2: Image fusion processing. An image fusion model is established based on a deep learning algorithm. After the model is established, images with good fusion quality and original images are obtained based on network big data to form a training set and a validation set for model training. Different data sets are then used as inputs to the image fusion model to extract image features and identify feature relationships. Finally, the feature relationships between images of different modalities are used to align and fuse the correspondingly labeled images in different data sets to obtain a fused image set.
[0027] Specifically, the convolutional neural network in the deep learning algorithm is first used to extract features from the input image, and then the recurrent neural network is used to learn the spatiotemporal relationship between different modalities. Finally, the spatiotemporal relationship and image features are used to align and fuse images of different modalities with the same annotations.
[0028] Step 3: Radiotherapy effect judgment: obtain radiotherapy data precedents of related tumors based on network big data, retrieve the patient's previous radiotherapy data and the fusion image set of the tumor, and judge the radiotherapy effect of the previous radiotherapy data by combining the radiotherapy data and tumor change graphics;
[0029] Specifically, for patients who are undergoing radiotherapy for the first time, the corresponding radiotherapy data of other patients with the same type of tumor are obtained based on the big data on the Internet for judgment; for patients with a history of multiple radiotherapy, the data of the patient's previous multiple radiotherapy treatments are obtained based on the big data on the Internet for judgment.
[0030] Step 4: Radiotherapy planning and adjustment: A radiotherapy planning model is established based on machine learning technology and a deep learning algorithm is introduced. The radiotherapy planning model is used in combination with the efficacy data in step 3 to intelligently plan the radiotherapy course. The fused image set obtained in step 2 and the previous radiotherapy data are used to analyze and optimize the radiotherapy dose distribution and radiotherapy range to obtain high-precision radiotherapy data.
[0031] After the radiotherapy planning model is established, the radiotherapy data and effect information of the same type of tumors are obtained based on network big data to establish a training set and a validation set for training. A deep learning algorithm is introduced into the analysis and optimization module of the trained radiotherapy planning model, and the radiotherapy planning model is adaptively trained during the radiotherapy treatment.
[0032] Example 2
[0033] according to Figure 2 As shown, this embodiment provides a multimodal image fusion system for improving radiotherapy accuracy, including an image acquisition system and a radiotherapy planning system.
[0034] The specific image acquisition system includes an image acquisition module, an image processing module and an image fusion module. The image acquisition module acquires different modal image data of the patient's tumor in different directions based on CT, MRI and PET devices, and makes the patient perform the same set of actions on different devices to acquire tumor images in the same direction. The image processing module is used to denoise and enhance the acquired images and organize them into corresponding data sets according to the type of imaging equipment, including CT image data set, MRI image data set and PET image data set. The image fusion module fuses the different modal image data with the same annotations in different data sets based on the image fusion model.
[0035] The image processing module also includes a denoising and enhancement submodule, an image annotation submodule and an image sorting submodule. The denoising and enhancement submodule performs denoising and enhancement processing on the acquired images based on the wavelet denoising method and the Laplace enhancement method. The image annotation submodule is used to annotate the images according to the direction of the tumor in the action of image acquisition. The image sorting submodule is used to classify and sort the processed images according to the different types of equipment used for acquisition, and divide them into CT images, MRI images and PET images.
[0036] The image fusion module also includes a feature extraction submodule, a spatiotemporal relationship recognition submodule and an alignment fusion submodule. The feature extraction submodule is used to extract tumor features from the input image. The spatiotemporal relationship recognition submodule recognizes the spatiotemporal relationship between the input images of different modalities based on the spatiotemporal relationship between the images of different modalities trained by learning. The alignment fusion submodule is used to align and fuse tumor images of different modalities in the same direction according to the extracted features and spatiotemporal relationship.
[0037] The fusion process is to first use the convolutional neural network in the model's deep learning algorithm to extract features from the input image, then use the recurrent neural network to learn the spatiotemporal relationship between different modalities, and finally use the spatiotemporal relationship and image features to align and fuse images of different modalities with the same annotations.
[0038] The specific radiotherapy planning system includes an efficacy judgment module, a radiotherapy planning module and an optimization and adjustment module. The efficacy judgment module obtains historical radiotherapy data based on network big data and judges the radiotherapy effect in combination with the radiotherapy dose and range. The radiotherapy planning module plans the course of later radiotherapy based on the radiotherapy planning model in combination with the patient's physical condition and radiotherapy effect. The plan includes the number of radiotherapy times, the dosage and angle parameters of each radiotherapy, and the prediction of the efficacy after each radiotherapy. The optimization and adjustment module is used to adjust the radiotherapy parameters in real time during the radiotherapy process in combination with the patient's physical condition and changes in tumor images.
[0039] The radiotherapy judgment module includes the initial efficacy judgment submodule and the historical efficacy judgment submodule. The initial efficacy judgment submodule predicts the efficacy of patients receiving initial radiotherapy based on network big data, and the historical efficacy judgment submodule predicts the efficacy of patients with a history of multiple radiotherapy based on network big data combined with historical data of previous radiotherapy.
[0040] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A multimodal image fusion method for improving radiotherapy accuracy, characterized in that: The following steps are involved: Step 1: Image acquisition and processing: First, different modality image data of the patient's tumor in different directions are obtained through CT, MRI and PET equipment, and then denoising and enhancing are performed respectively and annotated according to the acquisition direction, and finally sorted into different data sets; Step 2: Image fusion processing: establish an image fusion model based on a deep learning algorithm, use different data sets as input to the image fusion model, extract image features and perform feature relationship recognition, and finally use the feature relationship between images of different modalities to align and fuse the corresponding annotated images in different data sets to obtain a fused image set. Step 3: Radiotherapy effect judgment: obtain radiotherapy data precedents of related tumors based on network big data, retrieve the patient's previous radiotherapy data and the fusion image set of the tumor, and judge the radiotherapy effect of the previous radiotherapy data by combining the radiotherapy data and tumor change graphics; Step 4: Radiotherapy planning and adjustment. Based on machine learning technology, a radiotherapy planning model is established and a deep learning algorithm is introduced. The radiotherapy planning model is used in combination with the efficacy data in step 3 to intelligently plan the radiotherapy course. The fused image set obtained in step 2 and the previous radiotherapy data are used to analyze and optimize the radiotherapy dose distribution and radiotherapy range to obtain high-precision radiotherapy data.
2. A multimodal image fusion method for improving radiotherapy accuracy according to claim 1, characterized in that: When acquiring image data in different directions in step one, the patient is asked to perform the same set of actions on different devices to ensure that the directions of the acquired tumor images are the same; in step one, the image denoising method is adopted by the wavelet denoising method, and the image enhancement method is adopted by the Laplace enhancement method.
3. The multimodal image fusion method for improving radiotherapy accuracy according to claim 1, characterized in that: After the image fusion model is established in the step 2, images with good fusion quality and original images are obtained based on network big data to form a training set and a verification set for model training; the image fusion model fuses images by first using the convolutional neural network in the deep learning algorithm to extract features in the input image, then learning the spatiotemporal relationship between different modalities through a recurrent neural network, and finally using the spatiotemporal relationship and image features to align and fuse images of different modalities with the same annotation.
4. The multimodal image fusion method for improving radiotherapy accuracy according to claim 1, characterized in that: Specifically, step three includes obtaining the corresponding radiotherapy data of other patients with the same type of tumor based on network big data for patients who are undergoing radiotherapy for the first time and making judgments; and obtaining the corresponding radiotherapy data of the patient's previous multiple radiotherapy based on network big data for patients with a history of multiple radiotherapy.
5. The multimodal image fusion method for improving radiotherapy accuracy according to claim 1, characterized in that: After the radiotherapy planning model is established in the step 4, the radiotherapy data and effect information of the same type of tumors are obtained based on the network big data to establish a training set and a validation set for training, and a deep learning algorithm is introduced into the analysis and optimization module of the trained radiotherapy planning model to perform adaptive training on the radiotherapy planning model during the radiotherapy treatment.
6. A multimodal image fusion system for improving radiotherapy accuracy, characterized in that: It includes an image acquisition system and a radiotherapy planning system. The image acquisition system includes an image acquisition module, an image processing module and an image fusion module. The image acquisition module acquires different modality image data of the patient's tumor in different directions based on CT, MRI and PET equipment. The image processing module is used to denoise and enhance the acquired images and organize them into corresponding data sets according to the type of imaging equipment. The image fusion module performs fusion processing on the different modality image data with the same annotations in different data sets based on the image fusion model; the radiotherapy planning system includes an efficacy judgment module, a radiotherapy planning module and an optimization and adjustment module. The efficacy judgment module acquires historical radiotherapy data based on network big data and judges the radiotherapy effect in combination with the radiotherapy dose and range. The radiotherapy planning module plans the course of later radiotherapy based on the radiotherapy planning model in combination with the patient's physical condition and radiotherapy effect. The optimization and adjustment module is used to adjust the radiotherapy parameters in real time in combination with the patient's physical condition and changes in tumor images during the radiotherapy process.
7. A multimodal image fusion system for improving radiotherapy accuracy according to claim 6, characterized in that: The image processing module includes a denoising and enhancement submodule, an image annotation submodule and an image sorting submodule. The denoising and enhancement submodule performs denoising and enhancement processing on the acquired image based on wavelet denoising and Laplace enhancement methods. The image annotation submodule is used to annotate the image according to the direction of the tumor in the action of image acquisition. The image sorting submodule is used to classify and sort the processed images according to different types of acquired equipment.
8. The multimodal image fusion system for improving radiotherapy accuracy according to claim 6, characterized in that: The image fusion module includes a feature extraction submodule, a spatiotemporal relationship recognition submodule and an alignment fusion submodule. The feature extraction submodule is used to extract tumor features from the input image. The spatiotemporal relationship recognition submodule recognizes the spatiotemporal relationship between the input images of different modalities based on the spatiotemporal relationship between the images of different modalities trained through learning. The alignment fusion submodule is used to align and fuse tumor images of different modalities in the same direction according to the extracted features and spatiotemporal relationships.
9. The multimodal image fusion system for improving radiotherapy accuracy according to claim 6, characterized in that: The radiotherapy judgment module includes an initial efficacy judgment submodule and a historical efficacy judgment submodule. The initial efficacy judgment submodule predicts the efficacy of patients receiving initial radiotherapy based on network big data, and the historical efficacy judgment submodule predicts the efficacy of patients with a history of multiple radiotherapy based on network big data combined with historical data of previous radiotherapy.