A method and system for predicting tumor radiotherapy response based on multi-instance learning

Abstract

This application provides a method and system for predicting tumor radiotherapy response based on multi-instance learning. The method includes: constructing a radiotherapy image dataset; extracting and combining multi-dimensional features from the radiotherapy image dataset to obtain an embedded representation of the multi-dimensional features; the multi-dimensional features include two-dimensional convolutional features and three-dimensional convolutional features; constructing a Transformer-based multi-instance learning model; iteratively training the multi-instance learning model using the embedded representation of the multi-dimensional features as model input to obtain a trained multi-instance learning model; using the trained multi-instance learning model to predict the patient's mid-term radiotherapy response, and analyzing and evaluating the model prediction process using interpretable analysis, and evaluating the model prediction results for the patient's tumor region. The method provided in this application can predict the response after tumor radiotherapy, thereby intelligently assisting doctors in making precise radiotherapy decisions for patients and reducing the risk and harm to patient prognosis.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of medical imaging and deep learning, and more specifically, to a method and system for predicting tumor radiotherapy response based on multi-instance learning. Background Technology

[0002] Based on non-invasive or minimally invasive visualization of the internal structures of the human body, medical imaging technologies such as X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, and positron emission tomography (PET) have become important tools for the diagnosis and adjuvant treatment of modern oncology. Doctors use these imaging results to develop clinical treatment plans for cancer patients, aiming to achieve the goal of cancer treatment.

[0003] As one of the most important treatments for cancer, radiotherapy (RT) directly damages the DNA of cancer cells using high-energy rays. For patients with locally advanced non-small cell lung cancer, early assessment of radiotherapy efficacy using fluorodeoxyglucose (FDG) PET / CT during radiotherapy has proven to be of great significance. PET / CT, by quantifying changes in the tumor before and after treatment, can help tailor the optimal treatment strategy for each patient before morphological changes occur. The Standard Uptake Value (SUV) serves as a key prognostic indicator and provides valuable reference for measuring treatment effectiveness. However, during radiotherapy, radiation can also damage healthy tissues, leading to acute or long-term side effects; furthermore, traditional radiotherapy decisions rely heavily on physicians' subjective experience to set the radiation dose, lacking scientific rigor, indirectly affecting efficacy, and failing to guarantee a certain level of efficiency.

[0004] Therefore, developing a tumor radiotherapy response prediction method that combines accuracy and sophistication is an urgent technical problem to be solved, and it has guiding significance for the scientific formulation of tumor radiotherapy plans. Summary of the Invention

[0005] Based on the aforementioned medical technology needs, this application provides a method and system for predicting tumor radiotherapy response based on multi-instance learning. This method can predict the response after tumor radiotherapy, thereby intelligently assisting doctors in making precise radiotherapy decisions for patients and reducing prognostic risks and harms.

[0006] To achieve the above objectives and overcome the aforementioned technical deficiencies, in a first aspect, embodiments of this application provide a method for predicting tumor radiotherapy response based on multi-instance learning. This method includes the following steps: First, constructing a radiotherapy image dataset; the radiotherapy image dataset includes pre-radiotherapy image data, mid-radiotherapy image data, and radiotherapy dose setting data; then, extracting multi-dimensional features from the radiotherapy image dataset and combining and stitching them together to obtain an embedded representation of the multi-dimensional features; the multi-dimensional features include two-dimensional convolutional features and three-dimensional convolutional features; next, constructing a Transformer-based multi-instance learning model; then, using the embedded representation of the multi-dimensional features as model input, iteratively training the multi-instance learning model to obtain a trained multi-instance learning model; finally, using the trained multi-instance learning model to predict the patient's mid-radiotherapy response, and combining interpretable analysis to analyze and evaluate the model prediction process, and evaluating the prediction results for the patient's tumor region.

[0007] In some exemplary embodiments, constructing a radiotherapy image dataset includes: first, acquiring the patient's original tumor image data; then, preprocessing the patient's original tumor image data to construct a radiotherapy image dataset; wherein the preprocessing process includes: sequentially performing slice processing, standardization and normalization processing, region of interest localization, and multi-channel fusion processing on the original tumor image data of each patient.

[0008] In some exemplary embodiments, the patient's original tumor image data is preprocessed to construct a radiotherapy image dataset, including: First, the patient's pre-radiotherapy image data and radiotherapy dose setting data are segmented based on the z-axis to form slices; then, the size of each slice based on the z-axis is adjusted to a uniform H×W pixel size based on the largest tumor slice in the dataset, and the features in the slices are normalized; next, non-interest regions are excluded from the dataset by masking them to zero; and a multi-channel fusion strategy is combined, i.e., the pre-radiotherapy image is the first channel and the radiotherapy dose setting image is the second channel, to obtain the input image after fusion; wherein, the features in the slices include: the three-dimensional coordinates (x, y, z) of each pixel provided by the patient's pre-radiotherapy image data and the standard uptake value of the pixel before radiotherapy, and the three-dimensional coordinates and radiotherapy dose of each pixel provided by the radiotherapy dose setting data; the x and y values ​​of the slice features are normalized using the following formula:

[0009]

[0010] Among them, S min and S max These represent the minimum and maximum values ​​in each slice, respectively.

[0011] In some exemplary embodiments, extracting and combining multidimensional features from a radiotherapy image dataset includes: first, simultaneously extracting two-dimensional and three-dimensional convolutional features from the radiotherapy image dataset using a two-dimensional convolutional neural network and a three-dimensional convolutional neural network; then, using a combination method to concatenate the two-dimensional and three-dimensional convolutional features to obtain an embedded representation of the multidimensional features, which is used as input for subsequent network prediction; finally, the process of extracting multidimensional features includes: constructing a two-dimensional convolutional neural network architecture and a three-dimensional convolutional neural network architecture respectively by equipping multiple convolutional layers with activation functions; wherein, the two-dimensional convolutional... The input data of the neural network is in a two-dimensional format (B, N, C, H, W), where B represents the number of packages (i.e., the number of patients), N represents the number of slices per patient, C is the number of channels per slice, and H and W are the height and width of the slice, respectively. The input data of the three-dimensional convolutional neural network is in a three-dimensional format (B, C, D, H, W), where the depth D is equal to the number of slices N, and H and W are the height and width of the slice, respectively. A three-dimensional convolutional kernel is used to perform a three-dimensional convolution operation on the input three-dimensional data, moving in three dimensions to capture spatial relationships in the volumetric data and extract spatial features. The expression for the three-dimensional convolution operation is:

[0012]

[0013] in, Let (i,j,k) represent the element at position (i,j,k) in the input feature map of layer (l+1). b represents the weight at position (p, q, r) in the convolutional kernel of layer l. l These are the bias terms, where P, Q, and R are the sizes of the convolution kernel in the three dimensions, respectively.

[0014] In some exemplary embodiments, two-dimensional convolutional features are transformed into two-dimensional embedded representations through continuous convolutional pooling operations; three-dimensional convolutional features are transformed into two-dimensional embedded representations through fully connected layer compression; the data shape of the two-dimensional embedded representation is (B, N, M), where M is the embedding dimension; the data embedding dimension of the three-dimensional convolutional features is consistent with that of the two-dimensional convolutional features, which facilitates the splicing process.

[0015] In some exemplary embodiments, constructing a Transformer-based multi-instance learning model includes: constructing a Transformer-based multi-instance learning network architecture, the network architecture including a multi-head self-attention layer, a convolutional layer, and a fully connected layer; in the Transformer-based multi-instance learning model, a multi-head attention module combined with a feedforward network is used to compute and process the position representations of all elements in the processing sequence, wherein cls_token is added to the input feature sequence to enter the forward propagation process, and cls_token is specifically designed as the first element of the input sequence; used to capture and represent the global information of the entire input sequence.

[0016] In some exemplary embodiments, the deep learning training model is iteratively trained using the embedded representation of multidimensional features as input to the model. This includes: first, calculating the loss between the predicted result and the true label using a loss function; then, calculating the gradient by the optimizer and subsequently updating the model parameters in reverse; next, using leave-one-out cross-validation for model training and validation, monitoring the model's performance on the validation set, and adjusting the training process until performance no longer improves or a predetermined number of iterations is reached; the loss function used is the binary cross-entropy loss function, calculated as follows:

[0017]

[0018] Among them, y i It is the true label of sample i. is the model's predicted probability for sample i, and N is the total number of samples.

[0019] In some exemplary embodiments, during model training and validation, leave-one-out cross-validation selects one patient's data as the test set for each operation and repeats the selection multiple times, ensuring that each patient is tested once. During the training phase, two different categories of patients are selected as validation sets to ensure the model effectively distinguishes different types of radiotherapy responses, thus accurately selecting the best-performing model. The different categories of patients include those who respond to radiotherapy and those who do not. The patient classification is based on the rate of change of the average standard uptake value of all tumor slice pixels between the mid- and early-stage radiotherapy. The classification process includes: determining whether the average standard uptake value of all tumor slice pixels in the early-stage radiotherapy corresponds to the average standard uptake value of all tumor slice pixels in the mid-stage radiotherapy; if yes, the patient is classified as a responding patient (label 1); otherwise, the patient is classified as a non-responding patient (label 0). The correspondence is as follows:

[0020]

[0021] Wherein, ΔSUVmean represents the rate of change of the average standard uptake value of all tumor slice pixels between the mid- and early-stage radiotherapy; PreSUVmean represents the average standard uptake value of all tumor slice pixels in the early-stage radiotherapy; and MidSUVmean represents the average standard uptake value of all tumor slice pixels in the mid-stage radiotherapy.

[0022] In some exemplary embodiments, a trained multi-instance learning model is used to predict the mid-term response of patients to radiotherapy, and the model prediction process is analyzed and evaluated in conjunction with interpretability analysis, including: recording the attention weight scores generated by the attention mechanism of the Transformer, establishing interpretability analysis for the model's prediction process, and ensuring that the model provides certain medical interpretability for medical application scenarios.

[0023] Secondly, this application also provides a tumor radiotherapy response prediction system based on multi-instance learning. The system uses the tumor radiotherapy response prediction method based on multi-instance learning described in the above embodiments to predict tumor radiotherapy responses. It includes: a dataset module, a feature extraction module, a model building module, a model training module, and a model prediction analysis module connected in sequence. The dataset module is used to construct a radiotherapy image dataset, which includes pre-radiotherapy image data, mid-radiotherapy image data, and radiotherapy dose setting data. The feature extraction module extracts multi-dimensional features from the radiotherapy image dataset and combines and concatenates them to obtain an embedded representation of the multi-dimensional features. The multi-dimensional features include two-dimensional convolutional features and three-dimensional convolutional features. The model building module constructs a Transformer-based multi-instance learning model. The model training module iteratively trains the multi-instance learning model using the embedded representation of the multi-dimensional features as model input to obtain a trained multi-instance learning model. The model prediction analysis module uses the trained multi-instance learning model to predict the patient's mid-radiotherapy response and analyzes and evaluates the model prediction process using interpretable analysis, and evaluates the prediction results for the patient's tumor region.

[0024] Compared with the prior art, the technical solution provided in this application has at least the following advantages:

[0025] This application proposes a method and system for predicting tumor radiotherapy response based on multi-instance learning. The method includes the following steps: First, constructing a radiotherapy image dataset; the radiotherapy image dataset includes pre-radiotherapy image data, mid-radiotherapy image data, and radiotherapy dose setting data; then, extracting multi-dimensional features from the radiotherapy image dataset and combining and stitching them to obtain the embedded representation of the multi-dimensional features; the multi-dimensional features include two-dimensional convolutional features and three-dimensional convolutional features; next, constructing a Transformer-based multi-instance learning model; then, using the embedded representation of the multi-dimensional features as the model input, iteratively training the multi-instance learning model to obtain the trained multi-instance learning model; finally, using the trained multi-instance learning model to predict the patient's mid-radiotherapy response, and combining interpretable analysis to analyze and evaluate the model prediction process, and to evaluate the model prediction results for the patient's tumor region.

[0026] This application proposes a multi-instance learning-based method for predicting tumor radiotherapy responses. Considering the unique characteristics of weakly labeled scenarios in real-world medical settings, this method constructs a Transformer-based multi-instance learning model and introduces multi-head attention to optimize the prediction process, thereby predicting tumor responses to radiotherapy. Secondly, this application emphasizes the extraction and fusion of 2D and 3D image features. Based on the constructed feature engineering, it performs end-to-end prediction directly from PET / CT images and radiotherapy dose distribution maps, enhancing the model's multi-dimensional feature recognition capabilities and significantly improving the accuracy of tumor radiotherapy response identification and classification. Finally, this application provides interpretability analysis of the model's decision-making process through the distribution of attention weights, enhancing physicians' understanding and trust in the model's decisions and providing strong support for tumor treatment strategies. Furthermore, the model provided in this application, compared with other comparative algorithms on a full tumor dataset using evaluation metrics, verifies that the constructed tumor radiotherapy response prediction model balances effectiveness, accuracy, and interpretability, thus intelligently assisting physicians in making precise radiotherapy dose decisions for patients and supporting clinical radiotherapy. Attached Figure Description

[0027] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0028] Figure 1 This is a flowchart illustrating a tumor radiotherapy response prediction method based on multi-instance learning, provided as an embodiment of this application.

[0029] Figure 2 This is a flowchart of the data preprocessing process for a tumor radiotherapy response prediction model based on multi-instance learning, provided in one embodiment of this application.

[0030] Figure 3 This is a model construction framework diagram of a tumor radiotherapy response prediction model based on multi-instance learning provided in an embodiment of this application.

[0031] Figure 4 This is a schematic diagram of the structure of a tumor radiotherapy response prediction system based on multi-instance learning provided in an embodiment of this application. Detailed Implementation

[0032] As the background technology indicates, current tumor radiotherapy practice relies heavily on the physician's subjective judgment for dose allocation. However, the side effects of radiotherapy are influenced by various factors, including radiation dose, irradiation area, and the patient's individual health condition, making dose setting susceptible to individual differences and unforeseen circumstances. Therefore, the scientific validity, rationality, and effectiveness of existing technologies cannot be fully guaranteed.

[0033] In recent years, with the rapid development of artificial intelligence technologies such as deep learning, significant progress has been made in the field of computer-aided diagnosis, especially in image-guided analysis. By combining image processing technology with deep learning algorithms, the modern radiotherapy decision-making process has gradually moved towards precision, aiming to assist doctors in implementing more precise clinical radiotherapy plans and protocols, thereby improving patient efficacy and reducing the impact of treatment side effects.

[0034] Multiple Instance Learning (MIL) is a weakly supervised learning framework. In MIL, data exists in the form of "bags," each consisting of several instances. Labels are provided only at the bag level, and the labels of individual instances are unknown. The model's task is to infer instance-level information based on bag-level labels or to perform classification directly at the bag level. This approach allows the use of weakly labeled data and is suitable for scenarios where annotation is costly and fine-grained labels are difficult to obtain. In medical image analysis, fine-grained annotation of medical images requires specialized knowledge, is time-consuming and expensive, and the data is often characterized by scarce annotations. Furthermore, target regions in image slices are small and sparse; and the relationships between instances are complex—for example, different regions within organs or tissues may exhibit high heterogeneity, with different regions contributing differently to the classification task. Therefore, MIL, by combining bag-level label information with instance-level inference, demonstrates significant application potential in medical image analysis.

[0035] To address the existing problems and shortcomings, it is necessary to intelligently set radiotherapy doses by fully integrating medical resources. This application provides a method and model for predicting tumor radiotherapy response based on multi-instance learning. The method includes the following steps: First, constructing a radiotherapy image dataset; the radiotherapy image dataset includes pre-radiotherapy image data, mid-radiotherapy image data, and radiotherapy dose setting data; then, extracting and combining multi-dimensional features from the radiotherapy image dataset to obtain an embedded representation of the multi-dimensional features; the multi-dimensional features include two-dimensional convolutional features and three-dimensional convolutional features; next, constructing a Transformer-based multi-instance learning model; then, using the embedded representation of the multi-dimensional features as model input, iteratively training the multi-instance learning model to obtain a trained multi-instance learning model; finally, using the trained multi-instance learning model to predict the patient's mid-radiotherapy response, and combining interpretable analysis to analyze and evaluate the model prediction process, and evaluating the prediction results for the patient's tumor region. The tumor radiotherapy response prediction method and system based on multi-instance learning provided in this application have high accuracy in predicting whether or not a response will occur. By predicting whether or not a response will occur after tumor radiotherapy, the method can intelligently assist doctors in making precise radiotherapy decisions for patients, adjusting the radiotherapy dose for lung cancer patients, reducing the prognostic risks and harms of patients, and providing support for relevant radiotherapy clinical decisions.

[0036] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0037] See Figure 1 This application provides a method for predicting tumor radiotherapy response based on multi-instance learning, which includes the following steps:

[0038] Step S1: Construct a radiotherapy image dataset; the radiotherapy image dataset includes pre-radiotherapy image data (PETpre), mid-radiotherapy image data (PETmid), and radiotherapy dose setting data (Dose).

[0039] Step S2: Extract multidimensional features from the radiotherapy image dataset and combine and stitch them together to obtain the embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolutional features and three-dimensional convolutional features.

[0040] Step S3: Construct a multi-instance learning model based on Transformer.

[0041] Step S4: Using the embedded representation of multidimensional features as the model input, iteratively train the multi-instance learning model to obtain the trained multi-instance learning model.

[0042] Step S5: Use the trained multi-instance learning model to predict the patient's mid-term radiotherapy response, and combine interpretable analysis to analyze and evaluate the model prediction process, and evaluate the model prediction results for the patient's tumor area.

[0043] In some embodiments, constructing a radiotherapy image dataset in step S1 includes:

[0044] Step S101: Obtain the patient's original tumor imaging data.

[0045] Step S102: Preprocess the patient's original tumor image data to construct a radiotherapy image dataset.

[0046] The preprocessing process in step S102 includes: sequentially slicing the original tumor image data of each patient, standardizing and normalizing the data, locating the region of interest (ROI), and performing multi-channel fusion processing.

[0047] In some embodiments, step S102 involves preprocessing the patient's original tumor image data to construct a radiotherapy image dataset, including:

[0048] Step S1021: Perform z-axis segmentation processing on the patient's pre-radiotherapy imaging data and radiotherapy dose setting data to form slices.

[0049] Step S1022: Based on the largest tumor slice in the dataset, adjust the size of each slice along the z-axis to a uniform H×W pixel size, and normalize the features in the slices.

[0050] Step S1023: Remove non-interested regions from the dataset by masking them; and combine this with a multi-channel fusion strategy, i.e., pre-radiotherapy images as the first channel and radiotherapy dose setting images as the second channel, to obtain the input image after fusion.

[0051] In step S1022, the features in the slice include: the three-dimensional coordinates (x, y, z) and standard uptake value (SUV) of each pixel provided by the patient's pre-radiotherapy imaging data, and the three-dimensional coordinates and radiotherapy dose of each pixel provided by the radiotherapy dose setting data; the x and y values ​​of the slice feature are normalized using the following formula:

[0052]

[0053] Among them, S min and S max These represent the minimum and maximum values ​​in each slice, respectively; this numerical processing strategy helps to eliminate the influence of different scale factors.

[0054] Specifically, in step S1, the data is preprocessed and a structured radiotherapy image dataset is constructed, see [link to relevant documentation]. Figure 2 This application provides a data preprocessing flowchart for a tumor radiotherapy response prediction model based on multi-instance learning. The process includes: acquiring pre-radiotherapy image data (PETpre), mid-radiotherapy image data (PETmid), and radiotherapy dose setting data (Dose); firstly, performing continuous slicing processing on the image data of each patient based on the z-axis to obtain N cross-sectional slices; next, adopting a size standardization strategy to uniformly adjust the N obtained cross-sectional slices to the same size as the largest tumor slice in the dataset; furthermore, normalizing the x and y axis values ​​of each slice; finally, performing ROI processing and channel fusion to obtain the entire tumor input dataset. Non-ROI regions, i.e., those parts not belonging to the tumor volume, are zeroed out in the images through masking operations. A multi-channel fusion method is used, with the pre-radiotherapy PET image (PETpre) as the first channel and the dose image (Dose) as the second channel, which helps maintain the clinical relevance of the tumor region.

[0055] In some embodiments, step S2, which involves extracting and combining multidimensional features from the radiotherapy image dataset, includes:

[0056] Step S201: Simultaneously extract two-dimensional convolutional features and three-dimensional convolutional features of images from the radiotherapy image dataset using a two-dimensional convolutional neural network (2D CNN) and a three-dimensional convolutional neural network (3D CNN).

[0057] Step S202: Use a combination method to concatenate the two-dimensional convolutional features and the three-dimensional convolutional features to obtain the embedded representation of the multi-dimensional features, which is used as the input for subsequent network prediction.

[0058] Step S203, the process of extracting multidimensional features, includes: constructing a two-dimensional convolutional neural network architecture and a three-dimensional convolutional neural network architecture by equipping multiple convolutional layers with activation functions.

[0059] The input data of the two-dimensional convolutional neural network is in a two-dimensional format (B, N, C, H, W), where B represents the number of packages (i.e., the number of patients), N represents the number of slices per patient, C is the number of channels per slice, and H and W are the height and width of the slice, respectively. The input data of the three-dimensional convolutional neural network is in a three-dimensional format (B, C, D, H, W), where the depth D is equal to the number of slices N, and H and W are the height and width of the slice, respectively.

[0060] Step S2 mainly involves extracting features from the preprocessed multidimensional image data. First, a customized two-dimensional CNN architecture is used to extract two-dimensional convolutional features to obtain a two-dimensional embedding representation. Then, a three-dimensional CNN is used to extract spatial features through three-dimensional convolutional kernels and compress and transform the feature sequence. Finally, based on the comprehensive extraction of two-dimensional and three-dimensional convolutional features, the features are spliced ​​to complete the generation of the feature sequence embedding representation.

[0061] Specifically, in step S201, for two-dimensional feature extraction, the patient's slice input image (B,N,C,H,W) undergoes two convolutional layers (a convolutional layer with 5×5 kernel followed by ReLU activation and 2×2 max pooling) to obtain preliminary feature extraction, and then is processed by a fully connected layer to be transformed into a two-dimensional embedded representation (B,N,M).

[0062] In step S201, a 3D convolution kernel is used to perform a 3D convolution operation on the input 3D data, moving in three dimensions to capture spatial relationships in the volumetric data and extract spatial features; the expression for the 3D convolution operation is:

[0063]

[0064] in, Let (i,j,k) represent the element at position (i,j,k) in the input feature map of layer (l+1). b represents the weight at position (p, q, r) in the convolutional kernel of layer l. l These are the bias terms, where P, Q, and R are the sizes of the convolution kernel in the three dimensions, respectively.

[0065] In some embodiments, two-dimensional convolutional features are transformed into two-dimensional embedded representations through continuous convolutional pooling operations; three-dimensional convolutional features are transformed into two-dimensional embedded representations through fully connected layer compression; the data shape of the two-dimensional embedded representation is (B, N, M), where M is the embedding dimension; the data embedding dimension of the three-dimensional convolutional features is consistent with that of the two-dimensional convolutional features, which facilitates the splicing process.

[0066] It should be noted that the two-dimensional and three-dimensional embedding dimensions are consistent here, and the specific embedding dimension M can be optimized and selected according to the actual scenario and data characteristics.

[0067] The 3D CNN architecture consists of three 3×3×3 convolutional layers, each followed by a ReLU activation function and a 2×2×2 max pooling layer to introduce non-linear processing. An adaptive pooling layer is used as the final pooling layer to flexibly adjust the output size according to different input sizes and maintain output uniformity. After the convolutional pooling operation, the spatial features are compressed into a two-dimensional embedding representation with shape (B, N, M) through one or more fully connected layers.

[0068] In some embodiments, step S3, constructing a Transformer-based multi-instance learning model, includes: constructing a Transformer multi-instance learning network architecture, the network architecture including multi-head self-attention layers, convolutional layers, and fully connected layers. See also Figure 3 This application provides a model construction framework diagram for a tumor radiotherapy response prediction model based on multi-instance learning, including: introducing a multi-head attention mechanism, and including a stacked configuration of convolutional layers and fully connected layers, and initializing each network layer.

[0069] Specifically, in the Transformer-based multi-instance learning model, a multi-head attention module combined with a feedforward network is used to process the position representations of all elements in the processing sequence. Here, cls_token is added to the input feature sequence before entering the forward propagation process, and cls_token is specially designed as the first element of the input sequence to capture and represent the global information of the entire input sequence.

[0070] In some embodiments, step S4, which uses the embedded representation of multidimensional features as input to the model and iteratively trains the deep learning training model, includes:

[0071] Step S401: Calculate the loss between the predicted result and the true label using the loss function, and then use the Adam optimizer to calculate the gradient and update the model parameters in reverse.

[0072] Step S402: Use the Leave-One-Out Cross-Validation (LOOCV) method to train and validate the model, monitor the model's performance on the validation set, and adjust the training process until the performance no longer improves or the predetermined number of iterations is reached.

[0073] In step S401, the loss function used is the binary cross-entropy loss function, and the calculation formula is as follows:

[0074]

[0075] Among them, y i It is the true label of sample i. is the model's predicted probability for sample i, and N is the total number of samples.

[0076] In some embodiments, during model training and validation, LOOCV selects data from one patient as the test set for each operation and repeats the selection process multiple times, so that each patient is tested once. During the training phase, two different categories of patients (containing 0 and 1 labels) are selected as the validation set to ensure that the model can effectively distinguish different types of radiotherapy response and to accurately select the best-performing model.

[0077] The patients are categorized into two groups: those who respond to radiotherapy and those who do not. The classification is based on the rate of change in the average standardized uptake (SUP) of all tumor slice pixels between the mid- and early-stage radiotherapy periods. The classification process involves determining whether the SUP of all tumor slice pixels in the early-stage radiotherapy period corresponds to the SUP of all tumor slice pixels in the mid-stage radiotherapy period. If yes, the patient is classified as a responder (label 1); otherwise, the patient is classified as a non-responder (label 0). The correspondence is as follows:

[0078]

[0079] Wherein, ΔSUVmean represents the rate of change of the average standard uptake value of all tumor slice pixels between the mid- and early-stage radiotherapy; PreSUVmean represents the average standard uptake value of all tumor slice pixels in the early-stage radiotherapy; and MidSUVmean represents the average standard uptake value of a single pixel in the mid-stage radiotherapy of all tumor slice pixels.

[0080] In some embodiments, after data preprocessing, tumor radiotherapy images are used for feature extraction and prediction through the multidimensional feature Transformer multi-instance learning algorithm. The algorithm pseudocode is shown in Table 1, which illustrates the algorithm flow.

[0081] Table 1. Processing flow of the Transformer multi-instance learning algorithm for multi-dimensional features.

[0082]

[0083] In some embodiments, step S5 uses the trained multi-instance learning model to predict the patient's mid-term response to radiotherapy, and combines interpretability analysis to analyze and evaluate the model prediction process, including: recording the attention weight scores generated by the Transformer's attention mechanism, establishing interpretability analysis for the model's prediction process, and ensuring that the model provides certain medical interpretability for medical application scenarios.

[0084] Specifically, in step S5, the trained multi-instance learning model is used to predict the patient's mid-term response to radiotherapy, and the prediction process is analyzed and evaluated using interpretable analysis. The model is evaluated based on its predictions of the patient's tumor region, including:

[0085] Step S501: Based on the test set, input the data into the trained classifier to predict the results and obtain all test results.

[0086] Step S502: Summarize all test results and use them to calculate statistical indicators for model evaluation, such as accuracy and area under the receiver operating characteristic curve.

[0087] Step S503: Combine attention weights to construct interpretable analysis and analyze and evaluate the model prediction process.

[0088] Specifically, during the model prediction phase, the `cls_token` passes through a fully connected layer, outputting the predicted probability and predicted class. Attention weights generated in the Transformer network provide attention-based interpretive analysis of the model's decision-making process. Detailed analysis of different weights identifies the model's specific focus when processing 3D and 2D data. For example, does the model rely on overall volume information for prediction, or does it tend to rely more on certain key local slices? Especially when processing multi-slice scan data, the attention mechanism can indicate which slices contain the key information needed for diagnosis, allowing physicians to focus on these important images, thereby optimizing the diagnostic process and reducing the analysis of less important slices. Simultaneously, the model is evaluated based on prediction results on a tumor test set and output statistical indicators.

[0089] See Figure 4This application also provides a tumor radiotherapy response prediction system based on multi-instance learning. The system uses the tumor radiotherapy response prediction method based on multi-instance learning described in the above embodiments to predict tumor radiotherapy responses. It includes: a dataset module 101, a feature extraction module 102, a model building module 103, a model training module 104, and a model prediction and analysis module 105 connected sequentially. The dataset module 101 is used to construct a radiotherapy image dataset. The radiotherapy image dataset includes pre-radiotherapy image data (PETpre), mid-radiotherapy image data (PETmid), and radiotherapy dose setting data (Dose). The feature extraction module 102 is used to extract data from... Multidimensional features are extracted from the radiotherapy image dataset and combined and stitched together to obtain the embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolutional features and three-dimensional convolutional features; the model building module 103 is used to build a Transformer-based multi-instance learning model; the model training module 104 is used to iteratively train the multi-instance learning model with the embedded representation of the multidimensional features as the model input to obtain the trained multi-instance learning model; the model prediction and analysis module 105 is used to predict the patient's mid-term radiotherapy response using the trained multi-instance learning model, and to analyze and evaluate the model prediction process in conjunction with interpretable analysis, and to evaluate the model prediction results for the patient's tumor area.

[0090] Specifically, the dataset module 101 includes an image data preprocessing unit, which preprocesses and optimizes the dataset to construct a structured radiotherapy image dataset. Based on the three-dimensional coordinates and corresponding SUV of each pixel in the provided FDG ET / CT image data, and the three-dimensional position of the pixels recorded in the Dose radiotherapy dose image and the received radiotherapy dose, slices are made based on the z-axis, and the obtained N cross-sectional slices are standardized in size and normalized. In order to obtain a structured tumor radiotherapy input dataset, ROI localization processing and channel fusion operations are then performed, and the fused data is the input image dataset.

[0091] The feature extraction module 102 mainly extracts multidimensional convolutional features from multidimensional image data for subsequent prediction and analysis. It adopts a combination method for feature extraction, extracting two-dimensional and three-dimensional convolutional features simultaneously through traditional two-dimensional CNN and 3D CNN, and concatenating them into a feature embedding sequence representation to obtain a specific embedding representation.

[0092] The model building module 103 includes multi-head self-attention layers, convolutional layers, fully connected layers, etc.; based on the Transformer multi-instance learning network architecture, after entering the prediction stage, a special design is made to add cls_token to the input sequence, and the final category prediction is output through forward propagation; the parameters of all network layers are initialized to prepare the model for training.

[0093] The model training module 104 is used to calculate gradients and update model parameters based on the designed binary cross-entropy loss function and backpropagation; during iterative training, it monitors the performance of the model on the validation set.

[0094] The model prediction analysis module 105 is used to predict the model classification results based on the optimized model parameters and independent test sets; based on the attention score generated in the Transformer network during the model prediction process, it evaluates the degree to which the model considers the relative importance of different input features during prediction and can highlight key input features, thereby constructing the interpretability of the model.

[0095] The following describes in detail the tumor radiotherapy response prediction method and model based on multi-instance learning provided in this application with reference to a specific embodiment.

[0096] First, the tumor radiotherapy response prediction model based on multi-instance learning proposed in this application uses PyCharm as the development environment and is implemented on a computing platform equipped with an Intel Core i9 13900K processor and NVIDIA GeForce RTX 3090 dual graphics cards. The runtime environment is Ubuntu system, running Python 3.8 and the PyTorch framework.

[0097] Furthermore, it should be noted that the embodiments of this application analyze and evaluate the radiotherapy response to locally advanced non-small cell lung tumors.

[0098] First, the image data preprocessing module performs z-axis-based slicing, slice size standardization and normalization, region of interest (ROI) processing, and channel fusion on the raw radiotherapy image data to obtain a full tumor axial slice dataset. Each slice is 80×80 pixels in size, with a width and height of 80 pixels, to ensure input consistency.

[0099] Then, feature extraction is performed on the preprocessed tumor dataset. In the two-dimensional feature extraction, the patient slice (2, 80, 80) passes through a set of convolutional network layers. The first feature extraction layer consists of two convolutional layers with 5×5 kernels, each followed by ReLU activation and 2×2 max pooling. The second feature extraction layer processes the final pooled feature map output from the first layer through a fully connected layer, flattening it and transforming it into a 128-dimensional feature embedding. Assuming the i-th patient has N... i If the slice is slicing, then its two-dimensional feature embedding is (N i ,128); In three-dimensional feature extraction, the slice image of the i-th patient (2,N) i(80,80) First, a set of convolutional network layers is used for preliminary feature extraction; the first feature extraction layer consists of three 3×3×3 3D convolutional layers, each followed by ReLU activation and 2×2×2 max pooling. Considering that there may be local extreme signals and abnormal features in the entire tumor region, which have an important impact on the identification and evaluation of the entire tumor region, the second feature extraction layer selects adaptive max pooling as the last pooling layer, thereby highlighting the maximum value of the corresponding region and more accurately assessing the risk and predicting the treatment effect of the tumor subregion; assuming that the i-th patient has N i If a slice is given, its three-dimensional feature embedding is in the form of (Ni, 128).

[0100] Next, based on the Transformer-based multi-instance learning network architecture, a multi-instance learning training model is constructed through a model building module and a model training module. The sequence representation of the embedded features is used as the input to the main architecture of the multi-instance learning network. To prevent overfitting, four heads are configured in the Transformer, and dropout regularization is used in the Transformer structure, introducing a dropout ratio of 0.1. The model is trained and parameters are tuned through the training module. The LOOCV strategy is used for the model training and validation steps. By grouping patient data, it is ensured that samples from different patients are used as separate test sets for evaluation in each experiment. The remaining patients are divided into training sets during the training phase, and two patient samples (including 0 and 1 labels) are selected from each training set as validation sets, representing those with and without significant radiotherapy response, respectively. It should be noted that to prevent data leakage, it is necessary to ensure that tumor sample data from the same patient does not appear in the training set, validation set, and test set simultaneously. The optimal multi-instance learning model is determined based on the lowest loss function value obtained from the validation set.

[0101] Finally, based on the test set, the effectiveness of the multi-instance learning model was evaluated using the model prediction analysis module. Evaluation metrics included accuracy and the area under the receiver operating characteristic curve (AUC). Referring to Table 2, the model achieved an AUC of 0.814 and an accuracy of 0.696 on the entire tumor test set, outperforming the classic ABMIL (Attention-based Multiple Instance Learning) multi-instance learning model (AUC = 0.637; Accuracy = 0.652). This demonstrates the model's superior predictive performance on patient tumor samples, more effectively predicting and distinguishing pixels indicating "presence or absence of response." Therefore, considering all evaluation metrics, the model demonstrates better predictive performance for patient radiotherapy responses.

[0102] Table 2. Evaluation indicators of patient tumor samples

[0103]

[0104] Furthermore, after the trained multi-instance learning model predicts the response to tumor radiotherapy, this embodiment of the application performs interpretable analysis on the obtained attention weights based on the prediction results and process given by the multi-instance learning model. The attention weights for the first seven slices of each patient are calculated (see Table 3), where a larger weight value indicates that the slice is more important for prediction. The attention weights for two-dimensional and three-dimensional features are calculated separately (see Table 4), thereby revealing whether each patient prioritizes the two-dimensional features of the slices or the overall three-dimensional features in the prediction.

[0105] Table 3. Attention weight values ​​for patient tumor slide samples

[0106]

[0107]

[0108] Table 4. Two-dimensional and three-dimensional attention weights and results for patient tumor samples.

[0109]

[0110]

[0111] Based on this, the tumor radiotherapy response prediction method and system based on multi-instance learning provided in this application uses the patient's pre-radiotherapy FDG PET / CT images and dose images as data input, and combines feature stitching technology to extract two-dimensional and three-dimensional image features to form a multi-dimensional combined feature embedding. Finally, it uses a Transformer-based multi-instance learning network architecture combined with a multi-head attention mechanism to output classification prediction results, which helps to comprehensively capture key features and generate attention weights based on the prediction process in weakly labeled scenarios, thereby constructing interpretable analysis and effectively evaluating the effect of tumor radiotherapy models. The prediction model provided in this application has high accuracy in predicting whether a response will occur. By predicting whether a response will occur after tumor radiotherapy, it can intelligently assist doctors in adjusting the radiotherapy dose for lung cancer patients, providing support for relevant radiotherapy clinical decisions.

[0112] Based on the above technical solutions, this application proposes a method and system for predicting tumor radiotherapy response based on multi-instance learning. The method includes the following steps: First, constructing a radiotherapy image dataset; the radiotherapy image dataset includes pre-radiotherapy image data, mid-radiotherapy image data, and radiotherapy dose setting data; then, extracting multi-dimensional features from the radiotherapy image dataset and combining and stitching them to obtain the embedded representation of the multi-dimensional features; the multi-dimensional features include two-dimensional convolutional features and three-dimensional convolutional features; next, constructing a Transformer-based multi-instance learning model; then, using the embedded representation of the multi-dimensional features as the model input, iteratively training the multi-instance learning model to obtain the trained multi-instance learning model; finally, using the trained multi-instance learning model to predict the patient's mid-radiotherapy response, and combining interpretable analysis to analyze and evaluate the model prediction process, and to evaluate the prediction results for the patient's tumor region.

[0113] This application proposes a multi-instance learning-based method for predicting tumor radiotherapy responses. Considering the unique characteristics of weakly labeled scenarios in real-world medical settings, this method constructs a Transformer-based multi-instance learning model and introduces multi-head attention to optimize the prediction process, thereby predicting tumor responses to radiotherapy. Secondly, this application emphasizes the extraction and fusion of 2D and 3D image features. Based on the constructed feature engineering, it performs end-to-end prediction directly from PET / CT images and radiotherapy dose distribution maps, enhancing the model's multi-dimensional feature recognition capabilities and significantly improving the accuracy of tumor radiotherapy response identification and classification. Finally, this application provides interpretability analysis of the model's decision-making process through the distribution of attention weights, enhancing physicians' understanding and trust in the model's decisions and providing strong support for tumor treatment strategies. Furthermore, the model provided in this application, compared with other comparative algorithms on a full tumor dataset using evaluation metrics, verifies that the constructed tumor radiotherapy response prediction model balances effectiveness, accuracy, and interpretability, thus intelligently assisting physicians in making precise radiotherapy dose decisions for patients and supporting clinical radiotherapy.

[0114] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A method for predicting tumor radiotherapy response based on multi-instance learning, characterized in that, Includes the following steps: Construct a radiotherapy image dataset; the radiotherapy image dataset includes pre-radiotherapy image data, mid-radiotherapy image data, and radiotherapy dose setting data for patients; Multidimensional features are extracted from the radiotherapy image dataset and combined and stitched together to obtain the embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolutional features and three-dimensional convolutional features; Construct a multi-instance learning model based on Transformer; Using the embedded representation of multidimensional features as model input, the multi-instance learning model is iteratively trained to obtain the trained multi-instance learning model; The trained multi-instance learning model was used to predict the mid-term response of patients to radiotherapy, and the prediction process of the model was analyzed and evaluated in combination with interpretable analysis. The prediction results of the patient's tumor area were evaluated by the model. Extracting and combining multidimensional features from the radiotherapy image dataset includes: Two-dimensional convolutional neural networks and three-dimensional convolutional neural networks are used to simultaneously extract two-dimensional convolutional features and three-dimensional convolutional features from the radiotherapy image dataset. A combination method is used to concatenate two-dimensional and three-dimensional convolutional features to obtain an embedded representation of multi-dimensional features, which is used as input for subsequent network prediction. The process of extracting multidimensional features includes: constructing two-dimensional convolutional neural network architectures and three-dimensional convolutional neural network architectures by equipping multiple convolutional layers with activation functions; The input data shape of the two-dimensional convolutional neural network is as follows: The format is two-dimensional, where B represents the number of packages, i.e., the number of patients, N represents the number of slices per patient, C is the number of channels per slice, and H and W are the height and width of the slice, respectively. The input data shape of a 3D convolutional neural network is The data is in a 3D format, where the depth D is equal to the number of slices N, and H and W are the height and width of the slices, respectively. A 3D convolution kernel is used to perform a 3D convolution operation on the input 3D data, moving in three dimensions to capture spatial relationships in the volumetric data and extract spatial features. The expression for the 3D convolution operation is: in, Indicates the first l The position in the input feature map of layer +1 is... elements, Indicates the first l The position in the convolution kernel of the layer is The weight, It is a bias term. These represent the size of the convolutional kernel in three dimensions; Constructing a Transformer-based multi-instance learning model, including: A multi-instance learning network architecture for Transformer is constructed, which includes a multi-head self-attention layer, a convolutional layer, and a fully connected layer. In the Transformer-based multi-instance learning model, a multi-head attention module combined with a feedforward network is used to compute and process the positional representations of all elements in the processing sequence. Here, cls_token is added to the input feature sequence before entering the forward propagation process, and cls_token is specially designed as the first element of the input sequence to capture and represent the global information of the entire input sequence. The trained multi-instance learning model was used to predict patients' mid-term response to radiotherapy, and the model's prediction process was analyzed and evaluated using interpretable analytics, including: Record the attention weight scores generated by the Transformer's attention mechanism to establish interpretability analysis for the model's prediction process, ensuring that the model provides certain medical interpretability for medical applications.

2. The tumor radiotherapy response prediction method based on multi-instance learning according to claim 1, characterized in that, Construct a radiotherapy image dataset, including: Obtain the patient's original tumor imaging data; Preprocess the patient's original tumor image data to construct a radiotherapy image dataset; The preprocessing process includes: sequentially slicing the original tumor image data of each patient, standardizing and normalizing the data, locating the region of interest, and performing multi-channel fusion processing.

3. The tumor radiotherapy response prediction method based on multi-instance learning according to claim 2, characterized in that, The patient's raw tumor imaging data was preprocessed to construct a radiotherapy imaging dataset, including: The patient's pre-radiotherapy imaging data and the radiotherapy dose setting data are processed based on z-axis segmentation to form slices; based on Each slice size along the axis is adjusted to a uniform size based on the largest tumor slice in the dataset. Pixel size, and normalize the features in the slice; By masking and zeroing the non-interest regions, they are excluded from the dataset; and combined with a multi-channel fusion strategy, i.e., the pre-radiotherapy image is the first channel and the radiotherapy dose setting image is the second channel, the input image after fusion is obtained. The features in the slice include: the three-dimensional coordinates of each pixel provided by the patient's pre-radiotherapy imaging data. The standard uptake values ​​of pixels before radiotherapy and the three-dimensional coordinates and radiotherapy dose of each pixel provided by the radiotherapy dose setting data; Features of slices x , y Value normalization is performed using the following formula: in, and These represent the minimum and maximum values ​​in each slice, respectively.

4. The tumor radiotherapy response prediction method based on multi-instance learning according to claim 1, characterized in that, The two-dimensional convolutional features are transformed into two-dimensional embedded representations through continuous convolutional pooling operations; the three-dimensional convolutional features are compressed and transformed into two-dimensional embedded representations through fully connected layers. The data shape represented by two-dimensional embedding is ,in M For the dimension of embedding; The data embedding dimension of the three-dimensional convolutional features is consistent with that of the two-dimensional convolutional features, which facilitates the splicing process.

5. The tumor radiotherapy response prediction method based on multi-instance learning according to claim 1, characterized in that, Using the embedded representation of multidimensional features as input to the model, the multi-instance learning model is iteratively trained, including: The loss between the predicted result and the true label is calculated by the loss function, and then the gradient is calculated by the optimizer to update the model parameters in reverse. The leave-one-out cross-validation method is used for model training and validation. The performance of the model on the validation set is monitored, and the training process is adjusted until the performance no longer improves or the predetermined number of iterations is reached. The loss function used is the binary cross-entropy loss function, and the calculation formula is as follows: in, It is a sample i The true label, It is the model on the sample i The predicted probability, N It represents the total number of samples.

6. The tumor radiotherapy response prediction method based on multi-instance learning according to claim 5, characterized in that, During model training and validation Leave-one-out cross-validation selects one patient's data as the test set for each operation and repeats the selection multiple times, so that each patient is tested once. During the training phase, two different categories of patients are selected as the validation set to ensure that the model can effectively distinguish different types of radiotherapy response and to accurately select the best-performing model. The different categories of patients include those who respond to radiotherapy and those who do not. The patient categories were determined based on the rate of change of the average standard uptake value of all tumor slice pixels between the mid- and early-stage radiotherapy. The classification process includes: determining whether the average standard uptake value of all tumor slice pixels in the pre-radiotherapy period and the average standard uptake value of all tumor slice pixels in the mid-radiotherapy period satisfy the correspondence; if so, the patient is classified as a reactive class and labeled as label 1. If not, classify the patient as unresponsive and label it 0; The correspondence is as follows: in, This represents the rate of change in the average standard uptake value of all tumor slice pixels between the mid- and early-stage radiotherapy for the patient. This represents the average standard uptake value of all tumor slice pixels before radiotherapy. MidSUVmean This represents the average standard uptake value of all tumor slice pixels during the mid-stage of radiotherapy.

7. A tumor radiotherapy response prediction system based on multi-instance learning, comprising predicting tumor radiotherapy response using the tumor radiotherapy response prediction method based on multi-instance learning as described in any one of claims 1 to 6, characterized in that, include: The dataset module, feature extraction module, model building module, model training module, and model prediction and analysis module are connected sequentially; among them, The dataset module is used to construct a radiotherapy image dataset; the radiotherapy image dataset includes pre-radiotherapy image data, mid-radiotherapy image data, and radiotherapy dose setting data for patients. The feature extraction module is used to extract multidimensional features from the radiotherapy image dataset and combine and stitch them together to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolutional features and three-dimensional convolutional features; The model building module is used to build a Transformer-based multi-instance learning model; The model training module is used to iteratively train the multi-instance learning model with the embedded representation of multi-dimensional features as model input, so as to obtain the trained multi-instance learning model. The model prediction and analysis module is used to predict the mid-term response of patients to radiotherapy using a trained multi-instance learning model, and to analyze and evaluate the model prediction process in conjunction with interpretable analysis, and to evaluate the model prediction results for the patient's tumor area.

Images