Method for predicting morphological and positional changes of moving tumor, and electronic device

Abstract

The present disclosure relates to the technical field of medical treatment. Provided are a method for predicting morphological and positional changes of a moving tumor, and an electronic device. A specific implementation solution comprises: acquiring a real-time image and physiological signal data of a target subject at a moment T0; determining the position of a tumor feature point and the shape of a tumor target volume in the real-time image of the target subject at the moment T0; and on the basis of the physiological signal data of the target subject at the moment T0, the position of the tumor feature point in the real-time image of the target subject at the moment T0, the shape of the tumor target volume in the real-time image of the target subject at the moment T0, a position sequence of the tumor feature point and a shape sequence of the tumor target volume in historical images of the target subject, a historical physiological signal data sequence of the target subject, and a pre-trained real-time continuous prediction model for tumor target volumes, predicting tumor morphological and positional data of the target subject at a moment T0+1. In this way, the present disclosure can effectively deal with dynamic changes of tumors during radiotherapy, thereby realizing continuous prediction of morphological and positional changes of tumors and ensuring accurate irradiation for tumors.

Description

A method and electronic device for predicting the shape and position changes of moving tumors Technical Field

[0001] This disclosure relates to the field of medical technology, and in particular to a method and electronic device for predicting changes in the shape and position of moving tumors. Background Technology

[0002] With the continuous advancement of radiotherapy technology, real-time adaptive radiotherapy (RT-ART) has become one of the key technologies in tumor treatment. The goal of RT-ART is to dynamically adjust the radiation dose distribution by monitoring the morphology and location changes of the patient's tumor in real time, ensuring precise irradiation of the tumor and minimizing damage to normal tissues. However, tumors are affected by various factors during treatment, such as periodic and non-periodic motion, leading to changes in their position and morphology. Therefore, accurately predicting changes in tumor shape and position is crucial for the success of radiotherapy.

[0003] Existing technologies can only predict changes in tumor shape and position based on static medical images, failing to fully consider the continuous changes of tumors over time. This makes it difficult to track and predict the dynamic changes of tumors in real time during actual radiotherapy.

[0004] Summary of the Invention

[0005] This disclosure provides a method and electronic device for predicting the shape and position changes of moving tumors, which can predict the shape and position changes of moving tumors in time series based on the image information and physiological signal information of the target object.

[0006] According to one aspect of this disclosure, a method for predicting changes in the shape and position of a moving tumor is provided, the method comprising:

[0007] Acquire real-time image and physiological signal data of the target object at time T0;

[0008] Determine the location of tumor feature points and the shape of the tumor target area in the real-time image of the target object at time T0, wherein the location of the tumor feature points is used to characterize the location of the tumor target area;

[0009] Based on the physiological signal data of the target object at time T0, the location of tumor feature points in the real-time image of the target object at time T0, the shape of the tumor target area in the real-time image of the target object at time T0, the sequence of tumor feature point locations and the sequence of tumor target area shapes in the historical images of the target object, the historical physiological signal data sequence of the target object, and the pre-trained real-time continuous prediction model of the tumor target area, the tumor shape and position data of the target object at time T0+1 are predicted.

[0010] In some embodiments, the location of tumor feature points in the real-time image of the target object at time T0 is determined based on the real-time image and physiological signal data of the target object at time T0, as well as a pre-trained tumor feature point prediction model.

[0011] In some embodiments, a deep convolutional neural network is trained using multi-task training based on meta-learning theory using different modal data of different types of tumors to obtain the tumor feature point prediction model; the different modal data includes image data and physiological signal data of different modalities.

[0012] In some embodiments, the step of using different modal data of different types of tumors to perform multi-task training on a deep convolutional neural network based on meta-learning theory to obtain the tumor feature point prediction model includes:

[0013] Acquire multimodal data, which is data related to multiple different tumor types. The multimodal data includes image data and physiological signal data of different modalities, and the images of different modalities include the identified tumor feature points.

[0014] Features are extracted from each data point in the multimodal data to obtain single-modal image features and physiological signal features;

[0015] The single-modal image features and physiological signal features are fused to obtain multimodal feature fusion data;

[0016] By utilizing multimodal feature fusion data, a deep convolutional neural network is trained using multi-task training based on meta-learning theory to obtain a general tumor feature point prediction model. This general tumor feature point prediction model is then defined as the tumor feature point prediction model. The general tumor feature point prediction model can predict tumor feature points for different modal images of different types of tumors.

[0017] One of the tasks in the multi-task set is the tumor feature point prediction task for a specific tumor type.

[0018] In some embodiments, after using multimodal feature fusion data to perform multi-task training on a deep convolutional neural network based on meta-learning theory to obtain a general tumor feature point prediction model, the method further includes:

[0019] When using the general tumor feature point prediction model to predict the location of tumor feature points, the general tumor feature point prediction model is optimized based on meta-learning theory to obtain the tumor feature point prediction model.

[0020] In some embodiments, determining the shape of the tumor target region in the real-time image of the target object at time T0 includes:

[0021] A pre-planned image of the target object is obtained, the pre-planned image of the target object including identified tumor feature points and tumor target area contour;

[0022] The real-time image of the target object at time T0, including tumor feature points, and the pre-planned image of the target object are input into a pre-trained image mapping model, which outputs the shape of the tumor target area in the real-time image of the target object at time T0.

[0023] In some embodiments, a deep convolutional neural network is trained using multimodal pre-planned images and multimodal real-time images with identified tumor feature points and tumor target area contours to form an image mapping relationship model between the tumor target area contours in the multimodal pre-planned images and the tumor target area contours in the multimodal real-time images. The trained deep convolutional neural network is then determined as the image mapping relationship model.

[0024] In some embodiments, the tumor feature points include at least one of: tumor-related anatomical landmarks, tumor-related functional features, and tumor-related feature points that can directly or indirectly reflect changes in tumor morphology.

[0025] In some embodiments, the physiological signal data includes at least one of respiratory signals, diaphragmatic reflex signals, and surface imaging signals.

[0026] In some embodiments, the real-time image includes at least one of: tomographic CBCT image, dynamic MRI image, and KV projection image.

[0027] In some embodiments, the pre-planned image includes at least one of: a CT image, a CBCT image, and an MRI image. According to another aspect of this disclosure, an electronic device is provided, comprising:

[0028] At least one processor; and

[0029] The memory is communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the motion tumor shape and position change prediction method provided in this disclosure.

[0031] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the electronic device to perform the method for predicting the shape and position changes of a moving tumor provided in this disclosure.

[0032] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method for predicting the shape and position changes of moving tumors provided in this disclosure.

[0033] The method for predicting the shape and position changes of moving tumors disclosed herein integrates image information and physiological signal information of the target object, and realizes the prediction of tumor shape and position changes in time series. It can effectively cope with the dynamic changes of tumors during radiotherapy, realize continuous prediction of tumor shape and position changes, and ensure accurate irradiation of tumors.

[0034] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0035] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0036] Figure 1 is a schematic diagram of the implementation environment of a method for predicting the shape and position changes of a moving tumor according to an embodiment of this disclosure;

[0037] Figure 2 is a flowchart illustrating a method for predicting the shape and position changes of a mobile tumor according to an embodiment of this disclosure;

[0038] Figure 3 is a flowchart illustrating a training method for a tumor feature point prediction model according to an embodiment of this disclosure;

[0039] Figure 4 is a flowchart illustrating a training method for an image mapping relationship model according to an embodiment of this disclosure;

[0040] Figure 5 is a block diagram of an electronic device used to implement the method for predicting the shape and position changes of a moving tumor according to an embodiment of the present disclosure. Detailed Implementation

[0041] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0042] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0043] First, the application scenarios involved in the embodiments of this disclosure are described. The method for predicting the shape and position changes of moving tumors provided in the embodiments of this disclosure can be applied to the field of medical technology, specifically to the scenario of adaptive radiotherapy.

[0044] Adaptive radiotherapy (ART) is a novel radiotherapy technique developed from image-guided radiotherapy (IGRT). Its core technology lies in dynamically adjusting the radiotherapy plan based on real-time data acquired by image acquisition equipment during radiotherapy, such as changes in tumor morphology (including size, shape, and other morphological characteristics) and location, as well as the irradiation status of surrounding normal tissues. The radiotherapy plan is a file used to control the delivery of radiation by the radiotherapy equipment.

[0045] Adaptive radiotherapy typically divides the radiotherapy process into multiple fractions, i.e., multi-fraction radiotherapy. Each fraction delivers a specific radiation dose to the patient, and the radiotherapy plan is dynamically adjusted based on the patient's current condition and changes in the tumor to ensure that each radiotherapy session achieves the best therapeutic effect.

[0046] However, current adaptive radiotherapy techniques can only adapt to changes in tumor morphology and location between fractionation sessions (i.e., between adjacent fractionation sessions) to a certain extent, eliminating some dose errors between fractionation sessions. However, they cannot eliminate the impact of real-time changes in tumor morphology within fractionation sessions, especially during radiation. Therefore, how to continuously optimize and adjust the treatment plan in real-time during adaptive radiotherapy, particularly within fractionation sessions, is a problem worthy of attention.

[0047] In order to continuously optimize treatment plans within fractionated radiotherapy, it is necessary to rapidly predict changes in tumor shape and location during the fractionated radiotherapy process.

[0048] Based on this, this disclosure proposes a method for predicting the shape and position changes of moving tumors. It integrates the image information and physiological signal information of the target object to achieve the prediction of tumor shape and position changes in time series. It can effectively cope with the dynamic changes of tumors during radiotherapy, realize the continuous prediction of tumor shape and position changes, and ensure accurate irradiation of tumors.

[0049] Figure 1 is a schematic diagram of the implementation environment of a method for predicting the shape and position changes of a moving tumor as disclosed in this disclosure. Referring to Figure 1, the implementation environment includes: an image acquisition device 101, a physiological signal monitoring device 102, a tumor shape and position change prediction device 103, and a radiotherapy planning device 104.

[0050] The image acquisition device 101 is used to acquire images of the tumor site (i.e., the target area) and surrounding normal tissue of the radiotherapy subject (such as a patient). In some embodiments, the image acquisition device 101 can be at least one of the following: a computed tomography (CT) device, an emission computed tomography (ECT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasound examination device.

[0051] The physiological signal monitoring device 102 is a device used to collect physiological signals from radiotherapy subjects. In some embodiments, the physiological monitoring device 102 may be at least one of a respiratory monitoring device, a body surface imaging device, or the like.

[0052] The tumor shape and position change prediction device 103 is a device used to acquire real-time images of the radiotherapy subject from the imaging device 101, acquire physiological signal data of the radiotherapy subject from the physiological signal monitoring device 102, and predict tumor shape and position changes based on the acquired real-time images and physiological signal data of the radiotherapy subject.

[0053] In some embodiments, the tumor shape and position change prediction device 103 may include a tumor shape and position change prediction client 1031 and a tumor shape and position change prediction server 1032.

[0054] The tumor shape and position change prediction client 1031 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. For example, in some embodiments, the user runs tumor shape and position change prediction instructions on the tumor shape and position change prediction server 1032 through the tumor shape and position change prediction client 1031.

[0055] The tumor location change prediction server 1032 can be a standalone physical server, a server cluster composed of multiple physical servers, a distributed file system, or at least one of the following cloud servers providing basic cloud computing services: cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data or artificial intelligence platforms. This disclosure does not limit the specific implementation. In some embodiments, the number of the tumor location change prediction servers 1032 can be more or less, and this disclosure does not limit the implementation. Of course, the tumor location change prediction server 1032 can also include other functions to provide more comprehensive and diversified services.

[0056] The radiotherapy planning device 104 is used to acquire the predicted tumor position changes of the radiotherapy subject from the tumor position change prediction device 103, in order to optimize the radiotherapy plan. The radiotherapy planning device 102 can run a radiotherapy planning system (TPS), which provides functions for developing, optimizing, and evaluating radiotherapy plans. For example, the RT pro TPS system.

[0057] In some embodiments, the radiotherapy planning device 104 may include a TPS client 1041 and a TPS server 1042.

[0058] The TPS client 1041 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. For example, in some embodiments, a user can trigger the TPS server 1042 to execute an adaptive radiotherapy planning optimization process via the TPS client 1041 on the radiotherapy planning system running on the TPS server 1042, and display the optimized radiotherapy plan. This effectively saves user time and provides a more intuitive presentation of the optimized treatment plan, allowing users to evaluate the radiotherapy plan.

[0059] The TPS server 1042 can be a standalone physical server, a server cluster consisting of multiple physical servers, a distributed file system, or at least one of the following cloud servers providing basic cloud computing services: cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data or artificial intelligence platforms. This disclosure does not limit the specific implementation of these services. In some embodiments, the number of TPS servers 1042 can be greater or less, and this disclosure does not limit the implementation of these services. Of course, the TPS server 1042 can also include other functions to provide more comprehensive and diverse services. In some embodiments, the TPS server 1042 is used to provide background services for the TPS client 1041, such as performing an adaptive radiotherapy planning optimization process.

[0060] The following section introduces a method for predicting the shape and position changes of a moving tumor, based on the implementation environment shown in Figure 1.

[0061] Figure 2 is a flowchart illustrating a method for predicting the shape and position changes of a moving tumor according to this disclosure. In some embodiments, this method is performed by an electronic device. For example, the electronic device may be the device shown in Figure 1 above. As shown in Figure 2, the method for predicting the shape and position changes of a moving tumor provided by this disclosure predicts the shape and position state of the moving tumor at time T0+1 based on the shape and position state of the moving tumor at time T0 and the shape and position data and physiological signal data of the moving tumor before time T0, including the following steps:

[0062] S201: Acquire real-time image and physiological signal data of the target object at time T0;

[0063] During radiotherapy, real-time images of the target subject are acquired using image acquisition equipment at time T0, and physiological signal data of the target subject at time T0 are obtained using physiological signal monitoring devices such as respiratory monitoring equipment and body surface imaging equipment.

[0064] In this disclosure, physiological signal data includes periodic or non-periodic motion signals such as respiratory signal data, diaphragmatic reflex signal data, and surface imaging signal data; real-time images include at least one of tomographic CBCT images, dynamic MRI images, and KV projection images.

[0065] In this disclosure, the target object can be a patient, an organ, or a tumor.

[0066] S202: Determine the location of tumor feature points and the shape of the tumor target area in the real-time image of the target object at time T0;

[0067] After obtaining the real-time image and physiological signal data of the target object at time T0, it is necessary to determine the location of tumor feature points and the shape of the tumor target area in the real-time image of the target object at time T0, and based on this, predict the changes in the shape and position of the tumor of the target object at time T0+1.

[0068] In this disclosure, tumor feature points are used to characterize the location of the tumor target region. Tumor feature points include:

[0069] (1) Anatomical landmarks associated with tumors, such as:

[0070] For lung tumors, the relevant anatomical landmarks may include: the apex of the lung (the uppermost point of the lung), the base of the lung (the lower edge of the lung, the area in contact with the diaphragm), the hilum of the lung (the location where the main bronchus and pulmonary artery enter the lung), and the bifurcation point of the trachea (the location where the left and right main bronchi bifurcate).

[0071] For breast tumors, the relevant anatomical landmarks may include: the center point of the nipple, the edge of the upper outer quadrant of the breast, the edge of the lower inner quadrant of the breast, and the center point of the tumor.

[0072] For head and neck tumors, pharyngeal tumors, and upper esophageal tumors, the relevant anatomical landmarks may include: the angle of the mandible (the angle of the mandible), the tip of the mandible (the anterior tip of the mandible), the central point of the hyoid body (the center of the main body of the hyoid bone), the Adam's apple, the top of the cricoid cartilage (located below the thyroid cartilage), the upper border of the epiglottis, the posterior border of the soft palate, and the midpoint of the hard palate.

[0073] For prostate tumors, the relevant anatomical landmarks may include: the base of the prostate (where it connects to the bladder neck), the apex of the prostate (where it connects to the membranous part of the urethra), and the midpoint of the outer edge of the prostate (including: the left prostate, the right prostate, the anterior prostate, and the posterior prostate).

[0074] For cervical tumors, the relevant anatomical landmarks may include: the external os of the cervix (where the cervix connects to the vagina), the internal os of the cervix (where the cervix connects to the uterine body), the midpoint of the lateral margin of the cervix (including the midpoint of the left lateral margin of the cervix and the midpoint of the right lateral margin of the cervix), and the midpoint of the cervical wall (including the midpoint of the anterior cervical wall and the midpoint of the posterior cervical wall).

[0075] (2) Functional features related to tumors, such as: for head and neck tumors, pharyngeal tumors, and upper esophageal tumors, the functional features related to them may include: the base of the tongue, the cricopharyngeal muscle region at the esophageal inlet, the position of the vocal cords at the glottis level, etc.

[0076] (3) Tumor-related feature points that can directly reflect changes in tumor morphology, such as tumor edge points located in different directions of the tumor, tumor centroid (center of the tumor), etc.

[0077] (4) Tumor-related feature points that can indirectly reflect changes in tumor morphology, such as:

[0078] For lung tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the apex of the diaphragm, the midpoint of the clavicle, etc.

[0079] For breast tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the manubrium of the sternum (located in the upper part of the sternum), the medial end of the clavicle, etc.

[0080] For head and neck tumors, pharyngeal tumors, and upper esophageal tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the anterior tubercle of the first cervical vertebra and the spinous process of the seventh cervical vertebra.

[0081] For prostate tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the apex of the seminal vesicle and the midpoint of the anterior rectal wall.

[0082] For cervical cancer, tumor-related feature points that can indirectly reflect changes in tumor morphology may include: the midpoint of the posterior wall of the bladder, the midpoint of the anterior wall of the rectum, etc.

[0083] This disclosure uses positional changes of the aforementioned tumor feature points to track and / or predict the shape and position of moving tumors.

[0084] S203: Based on the physiological signal data of the target object at time T0, the tumor feature point positions in the real-time image of the target object at time T0, the tumor target area shape in the real-time image of the target object at time T0, the tumor feature point position sequence and tumor target area shape sequence in the historical images of the target object, the historical physiological signal data sequence of the target object, and the pre-trained real-time continuous prediction model for the tumor target area, predict the tumor shape and position data of the target object at time T0+1.

[0085] After obtaining the location of tumor feature points and the shape of the tumor target area in the real-time image of the target object at time T0, the physiological signal data of the target object at time T0, the location of tumor feature points in the real-time image of the target object at time T0, the shape of the tumor target area in the real-time image of the target object at time T0, as well as the sequence of tumor feature point locations and the sequence of tumor target area shapes in the images of the target object obtained before time T0, and the sequence of physiological signal data of the target object corresponding to the time before time T0, are input into the pre-trained real-time continuous prediction model of the tumor target area. The tumor shape and position data of the target object at time T0+1 are then predicted by the real-time continuous prediction model of the tumor target area.

[0086] In this disclosure, the tumor geomorphic data of the target object at time T0+1 includes at least one of the following: the location of tumor feature points of the target object at time T0+1, the shape of the tumor target area of ​​the target object at time T0+1, and the pixel offset matrix of the tumor target area between time T0 and time T0+1.

[0087] The tumor target area pixel offset matrix characterizes the offset in position and shape of the tumor target area caused by periodic and non-periodic motion.

[0088] In this disclosure, the historical images of the target object include: images of the target object acquired before time T0; for example, they can be images of the target object acquired at all times before time T0, or images of the target object acquired at a portion of times before time T0.

[0089] The historical physiological signal data sequence of the target object includes: physiological signal data of the target object corresponding to the time before time T0; for example, it can be physiological signal data of the target object corresponding to all the time before time T0, or it can be physiological signal data of the target object corresponding to a portion of the time before time T0.

[0090] The method for predicting the shape and position changes of moving tumors disclosed herein integrates image information and physiological signal information of the target object, and realizes the prediction of tumor shape and position changes in time series. It can effectively cope with the dynamic changes of tumors during radiotherapy, realize continuous prediction of tumor shape and position changes, and ensure accurate irradiation of tumors.

[0091] This disclosure also provides a training method for a real-time continuous prediction model of tumor target area. By establishing a correlation between tumor movement and physiological signals based on time series, real-time continuous prediction of the shape and position of the tumor target area is achieved. Specifically, by establishing a correlation between the location of tumor feature points, the shape of the tumor target area, and physiological signals (e.g., respiratory phase, diaphragm position information) based on time series, time-series-based tumor shape and position prediction training data is obtained. Based on the aforementioned time-series-based tumor shape and position prediction training data, a recurrent neural network (e.g., a GRU network (Gated Recurrent Unit) or an LSTM network (Long Short-Term Memory)) is trained to obtain the real-time continuous prediction model of the tumor target area. This real-time continuous prediction model of the tumor target area can predict the shape and position changes of the tumor target area at the next moment based on the current and historical shape and position data, and the current and historical physiological signal data.

[0092] This disclosure achieves efficient prediction of tumor feature points and tumor target shape over time using a pre-trained real-time continuous prediction model for tumor target areas, providing strong support for the implementation of real-time radiotherapy.

[0093] This disclosure also provides a method for determining the location of tumor feature points in a real-time image of a target object, comprising: determining the location of tumor feature points in the real-time image of the target object based on the real-time image and physiological signal data of the target object, and a pre-trained tumor feature point prediction model.

[0094] In this disclosure, a pre-trained tumor feature prediction model is able to predict the location of tumor feature points in different modal images of different types of tumors.

[0095] For example, when determining the location of tumor feature points in the real-time image of the target object at time T0, the real-time image and physiological signal data of the target object at time T0 are input into a pre-trained tumor feature point prediction model, and the pre-trained tumor feature point prediction model outputs the location of tumor feature points in the real-time image of the target object at time T0.

[0096] This disclosure also provides a training method for a tumor feature point prediction model. Considering the diversity of tumor types and image modalities, to improve the generalization ability of the tumor feature point prediction model, enabling it to be applicable to multiple tumor types, effectively integrate multimodal data, and achieve higher real-time performance, this disclosure utilizes limited, different modal data (including image data and physiological signal data of different tumor types) to perform multi-task training on a deep convolutional neural network based on meta-learning theory, thereby obtaining the tumor feature point prediction model. Here, meta-learning is a learning method that learns how to learn; its goal is to improve the model's generalization ability on new tasks by learning learning strategies with limited sample data.

[0097] As shown in Figure 3, this disclosure provides a method for obtaining a tumor feature point prediction model by using different modal data of different types of tumors to perform multi-task training on a deep convolutional neural network based on meta-learning theory, including:

[0098] S301: Acquire multimodal data;

[0099] Multimodal data refers to data related to multiple different tumor types, including multimodal image data and physiological signal data. For example, multimodal data may include: 4D CT image data of lung tumors, CBCT image data of lung tumors, physiological signal data related to lung tumors, 4D MRI image data of prostate tumors, and physiological signal data related to prostate tumors, etc.

[0100] In this disclosure, the multimodal images include identified tumor feature points, which may be manually annotated by clinical experts or machine-annotated. In this disclosure, the multimodal image data includes, but is not limited to: 4D CT images, 4D MRI images, CBCT images, 3D MRI images, dynamic MRI images, real-time KV projection images, and tomographic CBCT images.

[0101] In this disclosure, in order to identify and predict tumor feature points in certain unclear modal images, physiological signal data is introduced and combined with multimodal images to train a tumor feature point prediction model. The aim is to use physiological signal data as a guide and tumor feature point information in clear modal images to infer and predict tumor feature points in unclear modal images.

[0102] S302: Extract the features of each data point in the multimodal data to obtain single-modal image features and physiological signal features;

[0103] After acquiring the multimodal data, feature extraction is performed on each data point. Because multimodal data includes both image data from different modalities and physiological signal data, the extracted feature data includes both single-modal image features from different modalities and physiological signal features.

[0104] In this disclosure, when extracting features from image data of different modalities, a convolutional neural network (CNN) can be designed for each modality of image to perform single-modality feature extraction; when extracting features from physiological signal data, a fully connected layer or a recurrent neural network (RNN) can be used to extract physiological signal features.

[0105] S303: Fusion of single-modal image features and physiological signal features;

[0106] After extracting single-modal image features and physiological signal features, these extracted features are fused to obtain multimodal feature fusion data.

[0107] In this disclosure, multimodal feature fusion data can be obtained by stitching together various modal features (different single-modal image features and physiological signal features) and fusing various modal features based on an attention mechanism (calculating the attention weight of each modal feature and emphasizing important modalities).

[0108] S304: By using multimodal feature fusion data, a deep convolutional neural network is trained on a multi-task basis based on meta-learning theory to obtain a general tumor feature point prediction model.

[0109] After obtaining the multimodal feature fusion data, the tumor feature point prediction task for each specific tumor type is defined as a separate task, resulting in multiple tasks. For these multiple tasks, the deep convolutional neural network is sequentially trained using the multimodal feature fusion data based on meta-learning theory to obtain a general tumor feature point prediction model. This general tumor feature point prediction model can predict tumor feature points for different modalities of images of different types of tumors.

[0110] In this disclosure, the deep convolutional neural network can be any of a variety of neural network models such as U-Net, V-Net, and SAM.

[0111] This disclosure enables rapid and accurate prediction of tumor feature point locations in different imaging modalities of different tumor types by constructing a tumor feature point prediction model based on meta-learning theory and multimodal data fusion.

[0112] In some embodiments of this disclosure, in order to enable the above-mentioned general tumor feature point prediction model to quickly adapt to new tumor types and image modalities, the following step S305 may also be performed after step S304:

[0113] S305: When using a general tumor feature point prediction model to predict the location of tumor feature points, the general tumor feature point prediction model is optimized based on meta-learning theory to obtain the tumor feature point prediction model.

[0114] After obtaining a general tumor feature point prediction model, when using this model to predict tumor feature point locations, the parameters in the general tumor feature point prediction model are updated based on meta-learning theory to obtain a new tumor feature point prediction model (i.e., the idea of ​​optimizing while using). This tumor feature point prediction model can quickly adapt to new tumor types and image modalities, and has the ability to predict tumor feature point locations in different modalities of images with different types of tumors.

[0115] This disclosure also provides a method for determining the shape of a tumor target region in a real-time image of a target object. The method involves performing tumor feature point-based registration between a real-time image of the target object including tumor feature points and a pre-planned image of the target object (which includes identified tumor feature points and a tumor target region outline), determining an offset matrix between the real-time image and the pre-planned image, and mapping the identified tumor target region outline in the pre-planned image of the target object to the real-time image of the target object based on the offset matrix, thereby determining the shape of the tumor target region in the real-time image of the target object.

[0116] In this disclosure, the pre-planned image of the target object refers to the image corresponding to the treatment plan on which the current fractionation treatment is based, and the pre-planned image includes identified tumor feature points and the outline of the tumor target area.

[0117] To improve the efficiency of determining the shape of the tumor target region in real-time images, a pre-trained image mapping relationship model is used to achieve registration based on tumor feature points and mapping of the tumor target region contour between the real-time image of the target object and the pre-planned image of the target object.

[0118] In other embodiments of this disclosure, a pre-trained image mapping model is used to input a real-time image of the target object, including tumor feature points, and a pre-planned image of the target object. The image mapping model then performs registration based on tumor feature points and mapping of the tumor target region contour between the real-time image and the pre-planned image of the target object, thereby determining the shape of the tumor target region in the real-time image. In this embodiment, the pre-trained image mapping model is used to indicate the image mapping relationship from the tumor target region contour in the pre-planned image to the tumor target region contour in the real-time image.

[0119] For example, when determining the shape of the tumor target area in the real-time image of the target object at time T0, a pre-planned image of the target object is obtained, which includes identified tumor feature points and the outline of the tumor target area. The real-time image of the target object at time T0, including the tumor feature points, and the pre-planned image of the target object are input into a pre-trained image mapping model, and the shape of the tumor target area in the real-time image of the target object at time T0 is output.

[0120] As shown in Figure 4, this disclosure also provides a training method for an image mapping relationship model. This method utilizes multimodal pre-planned images and multimodal real-time images with identified tumor feature points and tumor target region contours to train a deep convolutional neural network to model the image mapping relationship between the tumor target region contours in the multimodal pre-planned images and the tumor target region contours in the multimodal real-time images, based on tumor feature points. The trained deep convolutional neural network is then identified as the image mapping relationship model. The method includes the following steps:

[0121] S401: Multimodal image data acquisition and processing;

[0122] We collected multimodal pre-planned images and real-time images from multiple patients with different tumor types, and accurately labeled the tumor target area, organs at risk, and tumor feature points on the collected multimodal image data to establish a high-quality training dataset.

[0123] In this disclosure, the precise annotation of tumor target areas, organs at risk, and tumor feature points in the aforementioned multimodal image data can be performed manually by clinical experts or by machines.

[0124] Different types of tumors include tumors in different locations, such as head and neck tumors, lung tumors, breast tumors, bladder tumors, rectal tumors, prostate tumors, and cervical tumors.

[0125] Multimodal image data includes pre-planned images (including simulated positioning image data and fractionated treatment image data) and real-time treatment image data (e.g., tomographic CBCT images, dynamic MRI images, KV projection images, etc.) of various modalities (e.g., CT images, 4D CT images, CBCT images, MRI images (including 3D MRI images, 4D MRI images, etc.).

[0126] S402: Image mapping relationship model training;

[0127] Using the high-quality training dataset established in step S401, a deep convolutional neural network (e.g., U-Net, V-Net, SAM, etc.) is trained to establish an image mapping relationship model between the tumor target region contour in the multimodal pre-planned image and the tumor target region contour in the multimodal real-time image based on tumor feature points. Specifically, based on the tumor feature points in the multimodal real-time image and the multimodal pre-planned image, an offset matrix is ​​established between the multimodal real-time image and the multimodal pre-planned image. Based on the relationship between the tumor target region contours in the multimodal pre-planned image and the offset matrix, an image mapping relationship is established between the tumor target region contours in the multimodal pre-planned image and the tumor target region contours in the multimodal real-time image. The trained deep convolutional neural network is then identified as the image mapping relationship model.

[0128] In addition, to ensure the efficient operation of the real-time continuous prediction model of tumor target area, tumor feature point prediction model and image mapping relationship model established in this disclosure under real-time requirements, further optimizations such as pruning and quantization can be performed on the real-time continuous prediction model of tumor target area, tumor feature point prediction model and image mapping relationship model to reduce the number of model parameters and computational load, and meet real-time requirements.

[0129] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the motion tumor shape and position change prediction method provided in the present disclosure.

[0130] According to embodiments of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause an electronic device to execute the method for predicting the shape and position changes of moving tumors provided in the present disclosure.

[0131] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method for predicting the shape and position changes of moving tumors provided in this disclosure.

[0132] In some embodiments, the electronic device may be a tumor shape and position change prediction device as shown in FIG1 above. FIG5 shows a schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure. The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 500 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0133] As shown in Figure 5, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 can also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0134] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the method for predicting changes in the shape and position of a moving tumor. For example, in some embodiments, the method for generating a radiotherapy plan can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or mounted on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the method for predicting changes in the shape and position of a moving tumor described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform a method for predicting changes in the shape and position of a moving tumor by any other suitable means (e.g., by means of firmware).

[0136] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for predicting changes in the shape and position of a moving tumor, characterized in that, include: Acquire real-time image and physiological signal data of the target object at time T0; Determine the location of tumor feature points and the shape of the tumor target area in the real-time image of the target object at time T0, wherein the location of the tumor feature points is used to characterize the location of the tumor target area; Based on the physiological signal data of the target object at time T0, the location of tumor feature points in the real-time image of the target object at time T0, the shape of the tumor target area in the real-time image of the target object at time T0, the sequence of tumor feature point locations and the sequence of tumor target area shapes in the historical images of the target object, the historical physiological signal data sequence of the target object, and the pre-trained real-time continuous prediction model of the tumor target area, the tumor shape and position data of the target object at time T0+1 are predicted. The historical images of the target object include: images of the target object acquired before time T0; The historical physiological signal data sequence of the target object includes: the physiological signal data of the target object corresponding to the time before time T0; The tumor geomorphic data of the target object at time T0+1 includes at least one of the following: the location of tumor feature points of the target object at time T0+1, the shape of the tumor target area of ​​the target object at time T0+1, and the pixel offset matrix of the tumor target area between time T0 and time T0+1.

2. The method according to claim 1, characterized in that, Determining the location of tumor feature points in the real-time image of the target object at time T0 includes: Based on the real-time image and physiological signal data of the target object at time T0, and the pre-trained tumor feature point prediction model, the location of tumor feature points in the real-time image of the target object at time T0 is determined. The pre-trained tumor feature point prediction model can predict the location of tumor feature points in different modal images of different types of tumors.

3. The method according to claim 2, characterized in that, By utilizing different modal data of different types of tumors, a deep convolutional neural network is trained using multi-task training based on meta-learning theory to obtain the tumor feature point prediction model; the different modal data includes image data and physiological signal data of different modalities.

4. The method according to claim 3, characterized in that, The method involves using different modalities of different types of tumors to train a deep convolutional neural network using multi-task training based on meta-learning theory, resulting in the tumor feature point prediction model, including: Acquire multimodal data, which is data related to multiple different tumor types. The multimodal data includes image data and physiological signal data of different modalities, and the images of different modalities include the identified tumor feature points. Features are extracted from each data point in the multimodal data to obtain single-modal image features and physiological signal features; The single-modal image features and physiological signal features are fused to obtain multimodal feature fusion data; By utilizing multimodal feature fusion data, a deep convolutional neural network is trained using multi-task training based on meta-learning theory to obtain a general tumor feature point prediction model. This general tumor feature point prediction model is then defined as the tumor feature point prediction model. The general tumor feature point prediction model can predict tumor feature points for different modal images of different types of tumors. One of the tasks in the multi-task set is the tumor feature point prediction task for a specific tumor type.

5. The method according to claim 4, characterized in that, After obtaining a general tumor feature point prediction model by utilizing multimodal feature fusion data to perform multi-task training on a deep convolutional neural network based on meta-learning theory, the method further includes: When using the general tumor feature point prediction model to predict the location of tumor feature points, the general tumor feature point prediction model is optimized based on meta-learning theory to obtain the tumor feature point prediction model.

6. The method according to claim 1, characterized in that, Determining the shape of the tumor target region in the real-time image of the target object at time T0 includes: A pre-planned image of the target object is obtained, the pre-planned image of the target object including identified tumor feature points and tumor target area contour; The real-time image of the target object at time T0, including tumor feature points, and the pre-planned image of the target object are input into a pre-trained image mapping model, which outputs the shape of the tumor target area in the real-time image of the target object at time T0. The pre-trained image mapping model is used to indicate the mapping relationship between the tumor target region contour in the pre-planned image of the target object and the tumor target region contour in the real-time image.

7. The method according to claim 6, characterized in that, Using multimodal pre-planned images and multimodal real-time images with identified tumor feature points and tumor target area contours, a deep convolutional neural network is trained to perform an image mapping relationship model between the tumor target area contour in the multimodal pre-planned images and the tumor target area contour in the multimodal real-time images based on tumor feature points. The trained deep convolutional neural network is then determined as the image mapping relationship model.

8. The method according to claim 1, characterized in that, The tumor feature points include at least one of the following: tumor-related anatomical landmarks, tumor-related functional feature points, and tumor-related feature points that can directly or indirectly reflect changes in tumor morphology.

9. The method according to claim 1, characterized in that, The physiological signal data includes at least one of respiratory signals, diaphragmatic reflex signals, and surface imaging signals.

10. The method according to claim 1, characterized in that, The real-time images include at least one of the following: tomographic CBCT images, dynamic MRI images, and KV projection images.

11. The method according to claim 6, characterized in that, The planned images include at least one of CT images, CBCT images, and MRI images.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 11.

Images