A real-time online tumor target segmentation method and electronic equipment

Abstract

This disclosure provides a real-time online tumor target region segmentation method and electronic device, relating to the field of medical technology. The specific implementation scheme is as follows: Real-time images and physiological signal data of the target object, as well as a pre-planned image of the target object, are acquired. The pre-planned image of the target object includes identified tumor feature points and the contour of the tumor target region. Based on the real-time image of the target object, the physiological signal data, and a pre-trained tumor feature point prediction model, the positions of the tumor feature points in the real-time image of the target object are determined. The positions of the tumor feature points are used to characterize the position of the tumor target region. The pre-trained tumor feature point prediction model can predict the positions of tumor feature points in different modalities of images of different types of tumors. Based on the real-time image of the target object including the tumor feature points and the pre-planned image of the target object, the tumor target region in the real-time image of the target object is segmented. Thus, real-time and accurate segmentation of the tumor target region in the real-time image of the target object is achieved.

Description

A real-time online tumor target region segmentation method and electronic device Technical Field

[0001] This disclosure relates to the field of medical technology, and in particular to a real-time online tumor target segmentation method and electronic device. Background Technology

[0002] In radiotherapy, accurate prediction of the tumor target volume is crucial to its effectiveness, especially in real-time adaptive radiotherapy (RT-ART). Real-time prediction of the tumor target volume is essential for reducing radiation damage to healthy tissues and improving tumor dose coverage. Tumors undergo significant morphological and positional changes during treatment due to factors such as respiration and changes in body position. Therefore, accurate and real-time segmentation of the tumor target volume has become a significant technical challenge in radiotherapy.

[0003] Summary of the Invention

[0004] This disclosure provides a real-time online tumor target region segmentation method and electronic device. Based on the image information and physiological signal information of the target object, a pre-trained tumor feature point prediction model is used to determine the location of tumor feature points in the real-time image of the target object, and based on the location of tumor feature points in the real-time image of the target object, the tumor target region in the real-time image of the target object is segmented in real time and accurately.

[0005] According to one aspect of this disclosure, a real-time online tumor target region segmentation method is provided, the method comprising:

[0006] Acquire real-time images and physiological signal data of the target object, as well as a pre-planned image of the target object, wherein the pre-planned image of the target object includes identified tumor feature points and tumor target area contours;

[0007] Based on the real-time images and physiological signal data of the target object, as well as the pre-trained tumor feature point prediction model, the location of tumor feature points in the real-time images of the target object is determined; the location of the tumor feature points is used to characterize the location of the tumor target area, and the pre-trained tumor feature point prediction model can predict the location of tumor feature points in different modal images of different types of tumors.

[0008] Based on the real-time image of the target object including tumor feature points and the pre-planned image of the target object, the tumor target area in the real-time image of the target object is segmented.

[0009] In some embodiments, the segmentation of the tumor target region in the real-time image of the target object based on a real-time image of the target object including tumor feature points and a pre-planned image of the target object includes:

[0010] The real-time image of the target object, including tumor feature points, and the pre-planned image of the target object are input into a pre-trained image mapping model to achieve tumor target region segmentation in the real-time image of the target object.

[0011] In some embodiments, a deep convolutional neural network is trained using multi-task training based on meta-learning theory using different modal data of different types of tumors to obtain the tumor feature point prediction model. The different modal data includes image data and physiological signal data of different modalities.

[0012] In some embodiments, the step of using different modalities of different types of tumors to perform multi-task training on a deep convolutional neural network based on meta-learning theory to obtain the tumor feature point prediction model includes:

[0013] Acquire multimodal data, which is data related to multiple different tumor types. The multimodal data includes image data and physiological signal data of different modalities, and the images of different modalities include the identified tumor feature points.

[0014] Features are extracted from each data point in the multimodal data to obtain single-modal image features and physiological signal features;

[0015] The single-modal image features and physiological signal features are fused to obtain multimodal feature fusion data;

[0016] By utilizing multimodal feature fusion data, a deep convolutional neural network is trained using multi-task training based on meta-learning theory to obtain a general tumor feature point prediction model. This general tumor feature point prediction model is then defined as the tumor feature point prediction model. This general tumor feature point prediction model can predict tumor feature points for different modal images of different types of tumors.

[0017] One of the tasks in the multi-task set is the tumor feature point prediction task for a specific tumor type.

[0018] In some embodiments, after using multimodal feature fusion data to perform multi-task training on a deep convolutional neural network based on meta-learning theory to obtain a general tumor feature point prediction model, the method further includes:

[0019] When using the general tumor feature point prediction model to predict the location of tumor feature points, the general tumor feature point prediction model is optimized based on meta-learning theory to obtain the tumor feature point prediction model.

[0020] In some embodiments, a deep convolutional neural network is trained using multimodal pre-planned images and multimodal real-time images with identified tumor feature points and tumor target area contours to form an image mapping relationship model between the tumor target area contours in the multimodal pre-planned images and the tumor target area contours in the multimodal real-time images. The trained deep convolutional neural network is then determined as the image mapping relationship model.

[0021] In some embodiments, the real-time image includes at least one of: tomographic CBCT image, dynamic MRI image, and KV projection image.

[0022] In some embodiments, the pre-planned images include at least one of CT images, CBCT images, and MRI images.

[0023] In some embodiments, the physiological signal data includes at least one of respiratory signals, diaphragmatic reflex signals, and surface imaging signals.

[0024] In some embodiments, the tumor feature points include at least one of the following: tumor-related anatomical landmarks, tumor-related functional features, and tumor-related features that can directly or indirectly reflect changes in tumor morphology.

[0025] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0026] At least one processor; and

[0027] The memory is communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the real-time online tumor target segmentation method provided in this disclosure.

[0029] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the electronic device to perform the real-time online tumor target segmentation method provided in this disclosure.

[0030] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the real-time online tumor target segmentation method provided in this disclosure.

[0031] This disclosure proposes a real-time online tumor target region segmentation method. By using a pre-trained tumor feature point prediction model based on the image information and physiological signal information of the target object, the method determines the location of tumor feature points in the real-time image of the target object, and achieves real-time and accurate segmentation of the tumor target region in the real-time image of the target object based on the location of the tumor feature points in the real-time image of the target object.

[0032] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0033] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0034] Figure 1 is a schematic diagram of the implementation environment of a real-time online tumor target region segmentation method according to an embodiment of this disclosure;

[0035] Figure 2 is a flowchart illustrating a real-time online tumor target region segmentation method according to an embodiment of this disclosure;

[0036] Figure 3 is a flowchart illustrating a training method for a tumor feature point prediction model according to an embodiment of this disclosure;

[0037] Figure 4 is a flowchart illustrating a training method for an image mapping relationship model according to an embodiment of this disclosure;

[0038] Figure 5 is a block diagram of an electronic device used to implement the real-time online tumor target segmentation method according to embodiments of the present disclosure. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0041] First, the application scenarios involved in the embodiments of this disclosure are described. The real-time online tumor target segmentation method provided in the embodiments of this disclosure can be applied to the field of medical technology, specifically to the scenario of adaptive radiotherapy.

[0042] Adaptive radiotherapy (ART) is a novel radiotherapy technique developed from image-guided radiotherapy (IGRT). Its core technology lies in dynamically adjusting the radiotherapy plan based on real-time data acquired by image acquisition equipment during radiotherapy, such as changes in tumor morphology (including size, shape, and other morphological characteristics) and location, as well as the irradiation status of surrounding normal tissues. The radiotherapy plan is a file used to control the delivery of radiation by the radiotherapy equipment.

[0043] Adaptive radiotherapy typically divides the radiotherapy process into multiple fractions, i.e., multi-fraction radiotherapy. Each fraction delivers a certain radiation dose to the patient, and the radiotherapy plan is dynamically adjusted according to the patient's current condition and changes in the tumor to ensure that each radiotherapy session achieves the best therapeutic effect.

[0044] However, current adaptive radiotherapy techniques can only adapt to changes in tumor morphology and location between fractionation sessions (i.e., between adjacent fractionation sessions) to a certain extent, eliminating some dose errors between fractionation sessions. However, they cannot eliminate the impact of real-time changes in tumor morphology within fractionation sessions, especially during radiation. Therefore, how to continuously optimize and adjust the treatment plan in real-time during adaptive radiotherapy, particularly within fractionation sessions, is a problem worthy of attention.

[0045] In order to continuously optimize the treatment plan within fractionated radiotherapy, it is necessary to quickly achieve real-time online segmentation of the tumor target area during the fractionated radiotherapy process.

[0046] Based on this, this disclosure proposes a real-time online tumor target region segmentation method. By using a pre-trained tumor feature point prediction model based on the image information and physiological signal information of the target object, the method determines the location of tumor feature points in the real-time image of the target object, and achieves real-time and accurate segmentation of the tumor target region in the real-time image of the target object based on the location of the tumor feature points in the real-time image of the target object.

[0047] Figure 1 is a schematic diagram of the implementation environment of a real-time online tumor target segmentation method disclosed in this disclosure. Referring to Figure 1, the implementation environment includes: an image acquisition device 101, a physiological signal monitoring device 102, a real-time online tumor target segmentation device 103, and a radiotherapy planning device 104.

[0048] The image acquisition device 101 is used to acquire images of the tumor site (i.e., the target area) and surrounding normal tissue of the radiotherapy subject (such as a patient). In some embodiments, the image acquisition device 101 can be at least one of the following: a computed tomography (CT) device, an emission computed tomography (ECT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasound examination device.

[0049] The physiological signal monitoring device 102 is a device used to collect physiological signals from radiotherapy subjects. In some embodiments, the physiological monitoring device 102 may be at least one of a respiratory monitoring device, a body surface imaging device, or the like.

[0050] The real-time online tumor target segmentation device 103 is a device used to acquire real-time images of the radiotherapy subject from the imaging device 101, acquire physiological signal data of the radiotherapy subject from the physiological signal monitoring device 102, and segment the tumor target area based on the acquired real-time images and physiological signal data of the radiotherapy subject.

[0051] In some embodiments, the real-time online tumor target segmentation device 103 may include a real-time online tumor target segmentation client 1031 and a real-time online tumor target segmentation prediction server 1032.

[0052] The real-time online tumor target segmentation client 1031 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. For example, in some embodiments, the user runs real-time online tumor target segmentation instructions on the real-time online tumor target segmentation server 1032 through the real-time online tumor target segmentation client 1031.

[0053] The real-time online tumor target segmentation server 1032 can be a standalone physical server, a server cluster composed of multiple physical servers, a distributed file system, or at least one of the following cloud servers providing basic cloud computing services: cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data or artificial intelligence platforms. This disclosure does not limit the specific implementation. In some embodiments, the number of the aforementioned real-time online tumor target segmentation servers 1032 can be more or fewer, and this disclosure does not limit the implementation. Of course, the real-time online tumor target segmentation server 1032 can also include other functions to provide more comprehensive and diversified services.

[0054] The radiotherapy planning device 104 is used to acquire real-time target segmentation results of the radiotherapy subject from the real-time online tumor target segmentation device 103 to optimize the radiotherapy plan. The radiotherapy planning device 102 may run a radiotherapy planning system (TPS), which provides functions for developing, optimizing, and evaluating radiotherapy plans. For example, the RT pro TPS system.

[0055] In some embodiments, the radiotherapy planning device 104 may include a TPS client 1041 and a TPS server 1042.

[0056] The TPS client 1041 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. For example, in some embodiments, a user can trigger the TPS server 1042 to execute an adaptive radiotherapy planning optimization process via the TPS client 1041 on the radiotherapy planning system running on the TPS server 1042, and display the optimized radiotherapy plan. This effectively saves user time and provides a more intuitive presentation of the optimized treatment plan, allowing users to evaluate the radiotherapy plan.

[0057] The TPS server 1042 can be a standalone physical server, a server cluster consisting of multiple physical servers, a distributed file system, or at least one of the following cloud servers providing basic cloud computing services: cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data or artificial intelligence platforms. This disclosure does not limit the specific implementation of these services. In some embodiments, the number of TPS servers 1042 can be greater or less, and this disclosure does not limit the implementation of these services. Of course, the TPS server 1042 can also include other functions to provide more comprehensive and diverse services. In some embodiments, the TPS server 1042 is used to provide background services for the TPS client 1041, such as performing an adaptive radiotherapy planning optimization process.

[0058] The following section describes a real-time online tumor target area method provided in this disclosure, based on the implementation environment shown in Figure 1.

[0059] Figure 2 is a flowchart illustrating a real-time online tumor target region method according to this disclosure. In some embodiments, this real-time online tumor target region method is executed by an electronic device. For example, the electronic device can be the device shown in Figure 1 above. As shown in Figure 2, the real-time online tumor target region segmentation method provided by this disclosure, based on the image information and physiological signal information of the target object, utilizes a pre-trained tumor feature point prediction model to determine the location of tumor feature points in the real-time image of the target object, and, based on the location of the tumor feature points in the real-time image of the target object, segments the tumor target region in the real-time image of the target object, including the following steps:

[0060] S201: Acquire real-time images and physiological signal data of the target object, as well as the pre-planned images of the target object.

[0061] Real-time adaptive radiotherapy, which aims to eliminate the impact of intra-fractional changes in tumor morphology and location on radiotherapy, requires strong real-time image processing capabilities, including real-time image tumor target region segmentation.

[0062] In real-time tumor target segmentation, it is usually necessary to first acquire real-time images of the target object, that is, during radiotherapy, images of the target object are acquired in real time using image acquisition equipment.

[0063] During real-time tumor target segmentation, the poor image quality and blurred boundaries of the target object make it difficult to accurately identify key information. Therefore, this disclosure utilizes a physiological signal monitoring device to monitor the physiological signal data of the target object, guiding the accurate identification of key information within the image. This physiological signal data includes periodic or non-periodic motion signals such as respiratory signals, diaphragmatic reflex signals, and surface imaging signals. This physiological signal data can be acquired using devices including respiratory monitoring equipment and surface imaging equipment.

[0064] In this disclosure, the pre-planned image of the target object refers to the image corresponding to the treatment plan on which the current fractionated treatment is based, and the pre-planned image includes identified tumor feature points and tumor target area contours. Here, the identification of tumor feature points and tumor target area contours can be done manually or automatically by a machine.

[0065] In this disclosure, the pre-planned image can be a CBCT image, an MRI image (e.g., a 3D MRI image, a 4D MRI image), a CT image (e.g., a 4D CT image), or other modal images; the pre-planned image can include simulated positioning image data, fractionated treatment image data, etc.

[0066] In this disclosure, the target object can be a patient, an organ, or a tumor.

[0067] S202: Based on the real-time image of the target object, physiological signal data, and a pre-trained tumor feature point prediction model, determine the location of tumor feature points in the real-time image of the target object.

[0068] After obtaining real-time images and physiological signal data of the target object, it is necessary to determine the location of tumor feature points in the real-time images of the target object, and identify the tumor feature points in the real-time images of the target object based on the location of the tumor feature points.

[0069] In this disclosure, real-time images and physiological signal data of the target object are input into a pre-trained tumor feature point prediction model, and the location of tumor feature points in the real-time image of the target object is determined by the tumor feature point prediction model.

[0070] In this disclosure, the location of tumor feature points is used to characterize the location of the tumor target area, and the tumor feature points include:

[0071] (1) Anatomical landmarks associated with tumors, such as:

[0072] For lung tumors, the relevant anatomical landmarks may include: the apex of the lung (the uppermost point of the lung), the base of the lung (the lower edge of the lung, the area in contact with the diaphragm), the hilum of the lung (the location where the main bronchus and pulmonary artery enter the lung), and the bifurcation of the trachea (the location where the left and right main bronchi bifurcate).

[0073] For breast tumors, the relevant anatomical landmarks may include: the center point of the nipple, the edge of the upper outer quadrant of the breast, the edge of the lower inner quadrant of the breast, and the center point of the tumor.

[0074] For head and neck tumors, pharyngeal tumors, and upper esophageal tumors, the relevant anatomical landmarks may include: the angle of the mandible (the angle of the mandible), the tip of the mandible (the anterior tip of the mandible), the central point of the hyoid body (the center of the main body of the hyoid bone), the Adam's apple, the top of the cricoid cartilage (located below the thyroid cartilage), the upper border of the epiglottis, the posterior border of the soft palate, and the midpoint of the hard palate.

[0075] For prostate tumors, the relevant anatomical landmarks may include: the base of the prostate (where it connects to the bladder neck), the apex of the prostate (where it connects to the membranous part of the urethra), and the midpoint of the outer edge of the prostate (including: the left prostate, the right prostate, the anterior prostate, and the posterior prostate).

[0076] For cervical tumors, the relevant anatomical landmarks may include: the external os of the cervix (where the cervix connects to the vagina), the internal os of the cervix (where the cervix connects to the uterine body), the midpoint of the lateral margin of the cervix (including the midpoint of the left lateral margin of the cervix and the midpoint of the right lateral margin of the cervix), and the midpoint of the cervical wall (including the midpoint of the anterior cervical wall and the midpoint of the posterior cervical wall).

[0077] (2) Functional features related to tumors, such as: for head and neck tumors, pharyngeal tumors, and upper esophageal tumors, the functional features related to them may include: the base of the tongue, the cricopharyngeal muscle region at the esophageal inlet, the position of the vocal cords at the glottis level, etc.

[0078] (3) Tumor-related feature points that can directly reflect changes in tumor morphology, such as tumor edge points located in different directions of the tumor, tumor centroid (center of the tumor), etc.

[0079] (4) Tumor-related feature points that can indirectly reflect changes in tumor morphology, such as:

[0080] For lung tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the apex of the diaphragm, the midpoint of the clavicle, etc.

[0081] For breast tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the manubrium of the sternum (located in the upper part of the sternum), the medial end of the clavicle, etc.

[0082] For head and neck tumors, pharyngeal tumors, and upper esophageal tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the anterior tubercle of the first cervical vertebra and the spinous process of the seventh cervical vertebra.

[0083] For prostate tumors, tumor-related features that can indirectly reflect changes in tumor morphology may include: the apex of the seminal vesicle and the midpoint of the anterior rectal wall.

[0084] For cervical cancer, tumor-related feature points that can indirectly reflect changes in tumor morphology may include: the midpoint of the posterior wall of the bladder, the midpoint of the anterior wall of the rectum, etc. This disclosure achieves real-time online segmentation of the tumor target area based on the location of the above-mentioned tumor feature points.

[0085] S203: Based on the real-time image of the target object including tumor feature points and the pre-planned image of the target object, the tumor target area in the real-time image of the target object is segmented.

[0086] After determining the location and identifying the tumor feature points in the real-time image of the target object, the tumor target area in the real-time image of the target object is segmented based on the real-time image of the target object including the tumor feature points and the pre-planned image of the target object.

[0087] In some embodiments of this disclosure, tumor target region segmentation in the real-time image of the target object is achieved by performing tumor feature point-based registration between a real-time image of the target object and a pre-planned image of the target object, determining an offset matrix between the real-time image and the pre-planned image of the target object, and mapping the tumor target region contour identified in the pre-planned image of the target object to the real-time image of the target object based on the offset matrix.

[0088] To improve the efficiency of real-time tumor target region segmentation, a pre-trained image mapping relationship model is used to achieve registration based on tumor feature points and mapping of tumor target region contours between the real-time image of the target object and the pre-planned image of the target object.

[0089] In other embodiments of this disclosure, a pre-trained image mapping model is used to input a real-time image of the target object, including tumor feature points, and a pre-planned image of the target object. The image mapping model achieves registration based on tumor feature points and mapping of the tumor target region contour between the real-time image and the pre-planned image of the target object. Based on the mapped tumor target region contour, tumor target region segmentation in the real-time image of the target object is achieved. In this embodiment, the pre-trained image mapping model is used to indicate the image mapping relationship from the tumor target region contour in the pre-planned image to the tumor target region contour in the real-time image.

[0090] This disclosure proposes a real-time online tumor target region segmentation method. By using a pre-trained tumor feature point prediction model based on the image information and physiological signal information of the target object, the method determines the location of tumor feature points in the real-time image of the target object, and achieves real-time and accurate segmentation of the tumor target region in the real-time image of the target object based on the location of the tumor feature points in the real-time image of the target object.

[0091] This disclosure also provides a training method for a tumor feature point prediction model. Considering the diversity of tumor types and image modalities, to improve the generalization ability of the tumor feature point prediction model, enabling it to be applicable to multiple tumor types, effectively integrate multimodal data, and achieve higher real-time performance, this disclosure utilizes limited, different modal data (including image data and physiological signal data of different tumor types) to perform multi-task training on a deep convolutional neural network based on meta-learning theory, thereby obtaining the tumor feature point prediction model. Here, meta-learning is a learning method that learns how to learn; its goal is to improve the model's generalization ability on new tasks by learning learning strategies with limited sample data.

[0092] As shown in Figure 3, this disclosure provides a method for obtaining a tumor feature point prediction model by using different modal data of different types of tumors to perform multi-task training on a deep convolutional neural network based on meta-learning theory, including: S301: acquiring multimodal data;

[0093] Multimodal data refers to data related to multiple different tumor types, including multimodal image data and physiological signal data. For example, multimodal data may include: 4D CT image data of lung tumors, CBCT image data of lung tumors, physiological signal data related to lung tumors, 4D MRI image data of prostate tumors, and physiological signal data related to prostate tumors, etc.

[0094] In this disclosure, the multimodal images include identified tumor feature points, which may be manually annotated by clinical experts or machine-annotated. In this disclosure, the multimodal image data includes, but is not limited to: 4D CT images, 4D MRI images, CBCT images, 3D MRI images, dynamic MRI images, real-time KV projection images, and tomographic CBCT images.

[0095] In this disclosure, in order to identify and predict tumor feature points in certain unclear modal images, physiological signal data is introduced and combined with multimodal images to train a tumor feature point prediction model. The aim is to use physiological signal data as a guide and tumor feature point information in clear modal images to infer and predict tumor feature points in unclear modal images.

[0096] S302: Extract the features of each data point in the multimodal data to obtain single-modal image features and physiological signal features;

[0097] After acquiring the multimodal data, feature extraction is performed on each data point. Because multimodal data includes both image data from different modalities and physiological signal data, the extracted feature data includes both single-modal image features from different modalities and physiological signal features.

[0098] In this disclosure, when extracting features from image data of different modalities, a convolutional neural network (CNN) can be designed for each modality of image to perform single-modality feature extraction; when extracting features from physiological signal data, a fully connected layer or a recurrent neural network (RNN) can be used to extract physiological signal features.

[0099] S303: Fusion of single-modal image features and physiological signal features;

[0100] After extracting single-modal image features and physiological signal features, these extracted features are fused to obtain multimodal feature fusion data.

[0101] In this disclosure, multimodal feature fusion data can be obtained by stitching together the features of each modality (different single-modal image features and physiological signal features) and fusing the features of each modality based on an attention mechanism (calculating the attention weight of each modality feature and emphasizing important modalities).

[0102] S304: By using multimodal feature fusion data, a deep convolutional neural network is trained on a multi-task basis based on meta-learning theory to obtain a general tumor feature point prediction model.

[0103] After obtaining the multimodal feature fusion data, the tumor feature point prediction task for each specific tumor type is defined as a separate task, resulting in multiple tasks. For each task, a deep convolutional neural network is trained based on meta-learning theory using the multimodal feature fusion data to obtain a general tumor feature point prediction model. This general tumor feature point prediction model can predict tumor feature points for different modalities of images of different tumor types.

[0104] In this disclosure, the deep convolutional neural network can be any of a variety of neural network models such as U-Net, V-Net, and SAM.

[0105] This disclosure enables rapid and accurate prediction of tumor feature point locations in different imaging modalities of different tumor types by constructing a tumor feature point prediction model based on meta-learning theory and multimodal data fusion.

[0106] In some embodiments of this disclosure, in order to enable the above-mentioned general tumor feature point prediction model to quickly adapt to new tumor types and image modalities, the following step S305 may also be performed after step S304:

[0107] S305: When using a general tumor feature point prediction model to predict the location of tumor feature points, the general tumor feature point prediction model is optimized based on meta-learning theory to obtain the tumor feature point prediction model.

[0108] After obtaining a general tumor feature point prediction model, when using this model to predict tumor feature point locations, the parameters in the general tumor feature point prediction model are updated based on meta-learning theory to obtain a new tumor feature point prediction model (i.e., the idea of ​​optimizing while using). This tumor feature point prediction model can quickly adapt to new tumor types and image modalities, and has the ability to predict tumor feature point locations in different modal images of different types of tumors. As shown in Figure 4, this disclosure also provides a training method for an image mapping relationship model. This method uses multimodal pre-planned images and multimodal real-time images with identified tumor feature points and tumor target area contours to train a deep convolutional neural network to train an image mapping relationship model between the tumor target area contours in the multimodal pre-planned images and the tumor target area contours in the multimodal real-time images based on tumor feature points. The trained deep convolutional neural network is then determined as the image mapping relationship model. The method includes the following steps:

[0109] S401: Multimodal image data acquisition and processing;

[0110] We collected multimodal pre-planned images and real-time images from multiple patients with different tumor types, and accurately labeled the collected multimodal image data with tumor target areas, organs at risk, and tumor feature points to establish a high-quality training dataset.

[0111] In this disclosure, the precise annotation of tumor target areas, organs at risk, and tumor feature points in the aforementioned multimodal image data can be performed manually by clinical experts or by machines.

[0112] Different types of tumors include tumors in different locations, such as head and neck tumors, lung tumors, breast tumors, bladder tumors, rectal tumors, prostate tumors, and cervical tumors.

[0113] Multimodal image data includes pre-planned images (including simulated positioning image data and fractionated treatment image data) and real-time treatment image data (e.g., tomographic CBCT images, dynamic MRI images, KV projection images, etc.) of various modalities (e.g., CT images, 4D CT images, CBCT images, MRI images (including 3D MRI images, 4D MRI images, etc.).

[0114] S402: Image mapping relationship model training;

[0115] Using the high-quality training dataset established in step S401, a deep convolutional neural network (e.g., U-Net, V-Net, SAM, etc.) is trained to establish an image mapping relationship model between the tumor target region contour in the multimodal pre-planned image and the tumor target region contour in the multimodal real-time image, based on tumor feature points. Specifically, based on tumor feature points in the multimodal real-time image and the multimodal pre-planned image, an offset matrix is ​​established between the two images. Based on the relationship and offset matrix between the tumor target region contours in the multimodal pre-planned image and the multimodal real-time image, an image mapping relationship is established between the tumor target region contour in the multimodal pre-planned image and the tumor target region contour in the multimodal real-time image. The trained deep convolutional neural network is then used as the image mapping relationship model. Furthermore, to ensure the efficient operation of the tumor feature point prediction model and the image mapping relationship model established in this disclosure under real-time requirements, further optimizations such as pruning and quantization can be performed on the tumor feature point prediction model and the image mapping relationship model to reduce the number of model parameters and computational load, thus meeting real-time requirements.

[0116] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the real-time online tumor target segmentation method provided by the present disclosure.

[0117] According to embodiments of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause an electronic device to execute the real-time online tumor target segmentation method provided in the present disclosure.

[0118] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the real-time online tumor target segmentation method provided in this disclosure.

[0119] In some embodiments, the electronic device may be the real-time online tumor target segmentation device shown in FIG1 above. FIG5 shows a schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure. The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 500 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0120] As shown in Figure 5, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 can also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0121] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0122] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the real-time online tumor target segmentation method. For example, in some embodiments, the real-time online tumor target segmentation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the real-time online tumor target segmentation method described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform a real-time online tumor target segmentation method by any other suitable means (e.g., by means of firmware).

[0123] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0127] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0128] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0129] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A real-time online tumor target region segmentation method, characterized in that, include: Acquire real-time images and physiological signal data of the target object, as well as a pre-planned image of the target object, wherein the pre-planned image of the target object includes identified tumor feature points and tumor target area contours; Based on the real-time images and physiological signal data of the target object, as well as the pre-trained tumor feature point prediction model, the location of tumor feature points in the real-time images of the target object is determined; the location of the tumor feature points is used to characterize the location of the tumor target area, and the pre-trained tumor feature point prediction model can predict the location of tumor feature points in different modal images of different types of tumors. Based on the real-time image of the target object including tumor feature points and the pre-planned image of the target object, the tumor target area in the real-time image of the target object is segmented.

2. The method according to claim 1, characterized in that, The segmentation of the tumor target region in the real-time image of the target object, based on a real-time image of the target object including tumor feature points and a pre-planned image of the target object, includes: The real-time image of the target object, including tumor feature points, and the pre-planned image of the target object are input into a pre-trained image mapping model to achieve tumor target region segmentation in the real-time image of the target object; The pre-trained image mapping model is used to indicate the mapping relationship between the tumor target region contour in the pre-planned image of the target object and the tumor target region contour in the real-time image.

3. The method according to claim 1, characterized in that, By utilizing different modal data of different types of tumors, a deep convolutional neural network is trained using multi-task training based on meta-learning theory to obtain the tumor feature point prediction model. The different modal data includes image data and physiological signal data of different modalities.

4. The method according to claim 3, characterized in that, The method involves using different modalities of different types of tumors to train a deep convolutional neural network using multi-task training based on meta-learning theory, resulting in the tumor feature point prediction model, including: Acquire multimodal data, which is data related to multiple different tumor types. The multimodal data includes image data and physiological signal data of different modalities, and the images of different modalities include the identified tumor feature points. Features are extracted from each data point in the multimodal data to obtain single-modal image features and physiological signal features; The single-modal image features and physiological signal features are fused to obtain multimodal feature fusion data; By utilizing multimodal feature fusion data, a deep convolutional neural network is trained using multi-task training based on meta-learning theory to obtain a general tumor feature point prediction model. This general tumor feature point prediction model is then defined as the tumor feature point prediction model. This general tumor feature point prediction model can predict tumor feature points for different modal images of different types of tumors. One of the tasks in the multi-task set is the tumor feature point prediction task for a specific tumor type.

5. The method according to claim 4, characterized in that, After obtaining a general tumor feature point prediction model by utilizing multimodal feature fusion data to perform multi-task training on a deep convolutional neural network based on meta-learning theory, the method further includes: When using the general tumor feature point prediction model to predict the location of tumor feature points, the general tumor feature point prediction model is optimized based on meta-learning theory to obtain the tumor feature point prediction model.

6. The method according to claim 2, characterized in that, Using multimodal pre-planned images and real-time multimodal images with labeled tumor feature points and tumor target area contours, depth A convolutional neural network is used to train an image mapping relationship model between the tumor target region contour in a multimodal pre-planned image based on tumor feature points and the tumor target region contour in a multimodal real-time image. The trained deep convolutional neural network is then identified as the image mapping relationship model.

7. The method according to claim 1, characterized in that, The real-time images include at least one of the following: tomographic CBCT images, dynamic MRI images, and KV projection images.

8. The method according to claim 1, characterized in that, The planned images include at least one of CT images, CBCT images, and MRI images.

9. The method according to claim 1, characterized in that, The physiological signal data includes at least one of respiratory signals, diaphragmatic reflex signals, and surface imaging signals.

10. The method according to claim 1, characterized in that, The tumor feature points include at least one of the following: tumor-related anatomical landmarks, tumor-related functional feature points, and tumor-related feature points that can directly or indirectly reflect changes in tumor morphology.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

Images