Tumor radiotherapy reaction prediction method and system based on multi-instance learning

Abstract

The embodiment of the invention provides a tumor radiotherapy reaction prediction method and system based on multi-instance learning. The method comprises the following steps: constructing a radiotherapy image data set; extracting multi-dimensional features from the radiotherapy image data set and combining and splicing the multi-dimensional features to obtain embedded representation of the multi-dimensional features; the multi-dimensional features comprise two-dimensional convolution features and three-dimensional convolution features; a multi-instance learning model based on a Transform is constructed; taking the embedded representation of the multi-dimensional features as model input, and performing iterative training on the multi-instance learning model to obtain a trained multi-instance learning model; and predicting the radiotherapy metaphase reaction of the patient by using the trained multi-instance learning model, analyzing and evaluating the model prediction process in combination with interpretable analysis, and performing model evaluation on the prediction result of the tumor area of the patient. According to the method provided by the invention, the reaction condition after tumor radiotherapy can be predicted, so that a doctor can be intelligently assisted to carry out precise radiotherapy decision on a patient, and the prognosis risk and harm of the patient are reduced.

Description

Technical Field

[0001] The present application relates to the interdisciplinary field of medical imaging and deep learning, and more specifically, to a method and system for predicting tumor radiotherapy response based on multi-instance learning. Background Art

[0002] Based on the non-invasive or minimally invasive visualization of the internal structure of the human body, medical imaging technologies such as X-ray imaging, computerized tomography (CT), magnetic resonance imaging (MRI), ultrasound (Ultrasonic), and positron emission tomography (PET) have become important tools for modern oncology diagnosis and adjuvant treatment. Doctors use these imaging results to develop clinical diagnosis and treatment plans for cancer patients in order to achieve the goal of cancer treatment.

[0003] As one of the most important treatments for cancer, radiotherapy (RT) directly damages the DNA of cancer cells by using high-energy rays. For patients with locally advanced non-small cell lung cancer, the use of Fluoro Deoxy Glucose (FDG) PET / CT for early radiotherapy effect evaluation during radiotherapy has been shown to be of great significance. By quantifying the changes in tumors before and after treatment, PET / CT can assist in customizing the best individual treatment strategy for patients before morphological changes. Among them, the Standard Uptake Value (SUV), as a key prognostic evaluation indicator, also has a reference support significance in measuring the treatment effect. However, during the patient's radiotherapy, radiation may also cause damage to healthy tissues, leading to acute or long-term side effects; and traditional radiotherapy decisions mainly rely on the physician's subjective experience to set the radiotherapy dose, which lacks scientificity, indirectly affects the efficacy and cannot guarantee a certain efficiency.

[0004] Therefore, developing a method for predicting tumor radiotherapy response that is both accurate and advanced is a technical problem that needs to be solved urgently, and it has guiding significance for the scientific formulation of tumor radiotherapy plans. Summary of the invention

[0005] Based on the above-mentioned medical technology needs, the embodiments of the present application provide a tumor radiotherapy response prediction method and system based on multi-instance learning, which can predict the response of tumors after radiotherapy, and then intelligently assist doctors to make accurate radiotherapy decisions for patients, thereby reducing the risks and hazards of patient prognosis.

[0006] To achieve the above-mentioned objectives and solve the above-mentioned technical defects, in the first aspect, an embodiment of the present application provides a method for predicting tumor radiotherapy response based on multi-instance learning, the method comprising the following steps: first, constructing a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data, mid-radiotherapy image data and radiotherapy dose setting data; then, extracting multidimensional features from the radiotherapy image data set and combining and splicing them to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolution features and three-dimensional convolution features; next, constructing a Transformer-based multi-instance learning model; then, using the embedded representation of the multi-dimensional features as the model input, iteratively training the multi-instance learning model to obtain a trained multi-instance learning model; finally, using the trained multi-instance learning model to predict the patient's mid-term radiotherapy response, and combining interpretable analysis to analyze and evaluate the model prediction process, and performing model evaluation on the prediction results of the patient's tumor area.

[0007] In some exemplary embodiments, constructing a radiotherapy image data set includes: first, acquiring the patient's original tumor image data; then, preprocessing the patient's original tumor image data to construct the radiotherapy image data set; wherein the preprocessing process includes: sequentially performing slicing processing, standardization and normalization processing, region of interest positioning, and multi-channel fusion processing on each patient's original tumor image data.

[0008] In some exemplary embodiments, the original tumor image data of the patient is preprocessed to construct a radiotherapy image data set, including: first, the patient's pre-radiotherapy image data and radiotherapy dose setting data are segmented based on the z-axis to form slices; then, the size of each slice based on the z-axis is adjusted to a uniform H×W pixel size based on the largest tumor slice in the data set, and the features in the slice are normalized; next, the non-interested regions are excluded from the data set by performing a mask operation to zero; and a multi-channel fusion strategy is combined, that is, the pre-radiotherapy image is the first channel and the radiotherapy dose setting image is the second channel to obtain an input image after the fusion channel; wherein the features in the slice include: the three-dimensional coordinates (x, y, z) of each pixel provided by the patient's pre-radiotherapy image data and the standard uptake value of the pixel before radiotherapy, and the three-dimensional coordinates and radiotherapy dose of each pixel provided by the radiotherapy dose setting data; the feature x and y values ​​of the slice are normalized, and the formula is:

[0009]

[0010] Among them, S min and S max Represent the minimum and maximum values ​​in each slice, respectively.

[0011] In some exemplary embodiments, multidimensional features are extracted from a radiotherapy image data set and combined and spliced, including: first, through a two-dimensional convolutional neural network and a three-dimensional convolutional neural network, the two-dimensional convolutional features and the three-dimensional convolutional features of the image are simultaneously extracted from the radiotherapy image data set; then, the two-dimensional convolutional features and the three-dimensional convolutional features are spliced ​​by a combination method to obtain an embedded representation of the multidimensional features for input of subsequent network prediction; finally, the process of extracting the multidimensional features includes: through multiple convolutional layers equipped with activation functions, respectively constructing a two-dimensional convolutional neural network architecture and a three-dimensional convolutional neural network architecture; wherein the two-dimensional convolutional neural network architecture is used to extract the multidimensional features of the radiotherapy image data set; The input data shape of the neural network is in the two-dimensional format of (B, N, C, H, W), where B represents the number of packages, that is, the number of patients, N represents the number of slices of a single patient, C is the number of channels of the slice, and H and W are the height and width of the slice, respectively. The input data shape of the three-dimensional convolutional neural network is in the three-dimensional format of (B, C, D, H, W), where the depth D is equal to the number of slices N, and H and W are the height and width of the slice, respectively. The three-dimensional convolution kernel is used to perform a three-dimensional convolution operation on the input three-dimensional data, which moves in three dimensions and can capture the spatial relationship in the volume data and extract spatial features. The expression of the three-dimensional convolution operation is:

[0012]

[0013] in, represents the element at position (i, j, k) in the input feature map of layer l+1, represents the weight of the convolution kernel at position (p, q, r) in the lth layer, b l is the bias term, P, Q, and R are the sizes of the convolution kernel in three dimensions.

[0014] In some exemplary embodiments, two-dimensional convolutional features are converted into two-dimensional embedded representations through continuous convolutional pooling operations; three-dimensional convolutional features are converted into two-dimensional embedded representations through full-connected layer compression; the data shape of the two-dimensional embedded representation is (B, N, M), where M is the embedding dimension; the data embedding dimensions of the three-dimensional convolutional features are consistent with those of the two-dimensional convolutional features to facilitate splicing processing.

[0015] In some exemplary embodiments, a Transformer-based multi-instance learning model is constructed, including: constructing a Transformer-based multi-instance learning network architecture, the network architecture including a multi-head self-attention layer, a convolutional layer, and a fully connected layer; in the Transformer-based multi-instance learning model, a multi-head attention module is used in combination with a feedforward network to perform computational processing on the position representation of all elements of the processing sequence, wherein cls_token is added to the input feature sequence to enter the forward propagation process, and cls_token is specially designed as the first element of the input sequence; it is used to capture and represent the global information of the entire input sequence.

[0016] In some exemplary embodiments, the deep learning training model is iteratively trained with the embedded representation of multi-dimensional features as the input of the model, including: first, the loss between the prediction result and the true label is calculated by the loss function, and then the gradient is calculated by the optimizer, and then the model parameters are updated in reverse; then, the leave-one-out cross-validation method is used to train and verify the model, monitor the performance of the model on the validation set, and adjust the training process until the performance no longer improves or reaches a predetermined number of iterations; the loss function uses a binary cross entropy loss function, and the calculation formula is:

[0017]

[0018] Among them, y i is the true label of sample i, is the model's predicted probability for sample i, and N is the total number of samples.

[0019] In some exemplary embodiments, during the model training and verification process, the leave-one-out cross-validation method selects the data of one patient as the test set in each operation, and repeats the selection operation multiple times so that each patient is tested once; in the training stage, two different categories of patients are selected as the verification set to ensure that the model effectively distinguishes different types of radiotherapy responses, so as to accurately select the model with the best performance; wherein different categories of patients include radiotherapy responsiveness and radiotherapy non-responsiveness; the patient categories are classified according to the change rate of the average standard uptake value of all tumor slice pixels between the middle and early stages of radiotherapy of the patient; the classification process includes: judging whether the average standard uptake value of all tumor slice pixels in the early stage of radiotherapy and the average standard uptake value of all tumor slice pixels in the middle stage of radiotherapy meet the corresponding relationship; if so, the patient is classified as a responsive class, recorded as label 1; if not, the patient is classified as a non-responsive class, recorded as label 0; the corresponding relationship is:

[0020]

[0021] Among them, ΔSUVmean represents the rate of change of the average standard uptake value of all tumor slice pixels between the middle and early stages of radiotherapy; PreSUVmean represents the average standard uptake value of all tumor slice pixels in the early stage of radiotherapy, and MidSUVmean represents the average standard uptake value of all tumor slice pixels in the middle stage of radiotherapy.

[0022] In some exemplary embodiments, a trained multi-instance learning model is used to predict a patient's mid-term response to radiotherapy, and the model prediction process is analyzed and evaluated in combination with interpretable analysis, including: recording the attention weight scores generated by the Transformer's attention mechanism, establishing interpretable analysis for the model's prediction process, and ensuring that the model provides a certain medical interpretability basis in medical application scenarios.

[0023] In the second aspect, the embodiment of the present application also provides a tumor radiotherapy response prediction system based on multi-instance learning, which uses the tumor radiotherapy response prediction method based on multi-instance learning described in the above embodiment to predict tumor radiotherapy response, including: a data set module, a feature extraction module, a model construction module, a model training module and a model prediction analysis module connected in sequence; wherein the data set module is used to construct a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data, mid-radiotherapy image data and radiotherapy dose setting data; the feature extraction module is used to extract multi-dimensional features from the radiotherapy image data set and combine and splice them to obtain an embedded representation of the multi-dimensional features; the multi-dimensional features include two-dimensional convolution features and three-dimensional convolution features; the model construction module is used to construct a Transformer-based multi-instance learning model; the model training module is used to iteratively train the multi-instance learning model with the embedded representation of the multi-dimensional features as the model input to obtain a trained multi-instance learning model; the model prediction analysis module is used to use the trained multi-instance learning model to predict the patient's mid-term radiotherapy response, and analyze and evaluate the model prediction process in combination with interpretable analysis, and perform model evaluation on the prediction results of the patient's tumor area.

[0024] Compared with the prior art, the technical solution provided by the embodiment of the present application has at least the following advantages:

[0025] The present application proposes a method and system for predicting tumor radiotherapy response based on multi-instance learning, the method comprising the following steps: first, constructing a radiotherapy image data set; the radiotherapy image data set comprises patient pre-radiotherapy image data, mid-radiotherapy image data and radiotherapy dose setting data; then, extracting multidimensional features from the radiotherapy image data set and combining and splicing them to obtain an embedded representation of the multidimensional features; the multidimensional features comprise two-dimensional convolutional features and three-dimensional convolutional features; next, constructing a Transformer-based multi-instance learning model; then, using the embedded representation of the multidimensional features as the model input, iteratively training the multi-instance learning model to obtain a trained multi-instance learning model; finally, using the trained multi-instance learning model to predict the patient's mid-term radiotherapy response, and analyzing and evaluating the model prediction process in combination with interpretable analysis, and performing model evaluation on the prediction results of the patient's tumor area.

[0026] The method for predicting tumor radiotherapy response based on multi-instance learning proposed in this application takes into account the particularity of actual medical weak label scenarios. This application constructs a multi-instance learning model based on Transformer and introduces a multi-head attention mechanism to optimize the prediction process, so as to predict the tumor's response to radiotherapy. Secondly, this application particularly emphasizes the extraction and fusion of two-dimensional and three-dimensional image features, and directly performs end-to-end prediction from PET / CT images and radiotherapy dose distribution maps based on the constructed feature engineering, thereby enhancing the multi-dimensional feature recognition ability of the model and significantly improving the recognition and classification accuracy of tumor radiotherapy response. Finally, this application provides an interpretable analysis of the model decision process through the distribution of attention weights, which can enhance doctors' understanding and trust in model decisions and provide strong support for tumor treatment strategies. In addition, the model provided in this application can be compared with other comparison algorithms on the whole tumor data set through evaluation indicators, which can verify that the constructed tumor radiotherapy response prediction model takes into account effectiveness, accuracy and interpretability, so that it can intelligently assist doctors in making accurate radiotherapy dose decisions for patients and provide support for clinical radiotherapy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0028] Figure 1 A flowchart of a method for predicting tumor radiotherapy response based on multi-instance learning is provided in accordance with an embodiment of the present application.

[0029] Figure 2 A data preprocessing flowchart of a tumor radiotherapy response prediction model based on multi-instance learning provided in one embodiment of the present application.

[0030] Figure 3 A diagram of the model construction framework in a tumor radiotherapy response prediction model based on multi-instance learning provided in one embodiment of the present application.

[0031] Figure 4 A schematic diagram of the structure of a tumor radiotherapy response prediction system based on multi-instance learning provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] As can be seen from the background technology, in the current practice of tumor radiotherapy, the distribution of doses depends largely on the subjective judgment of doctors. The side effects of radiotherapy are affected by many factors, including radiation dose, irradiation area and individual health status of patients, which makes the dose setting susceptible to individual differences and accidental factors. Therefore, the scientificity, rationality and effectiveness of the existing technology cannot be fully guaranteed.

[0033] In recent years, with the rapid development of artificial intelligence technologies such as deep learning, significant progress has been made in the field of computer-aided diagnosis, especially in the field of image-guided analysis. By combining image processing technology with deep learning algorithms, the modern radiotherapy decision-making process has gradually developed towards precision, aiming to improve patient efficacy and reduce the impact of treatment side effects by assisting doctors in implementing more accurate clinical radiotherapy plans and program formulation.

[0034] Multiple Instance Learning (MIL) is a weakly supervised learning framework. In MIL, data exists in the form of "bags", each bag consists of several instances, and the labels are only provided at the bag level, and the labels of individual instances are unknown. The task of the model is to infer instance-level information based on bag-level labels or to complete classification directly at the bag level. This approach allows the use of weakly labeled data and is suitable for scenarios where the labeling cost is high and fine-grained labels are not easy to obtain. In medical image analysis, fine-grained labeling of medical images requires professional knowledge, is time-consuming and expensive, and the data is usually scarcely labeled; at the same time, the target area in the image slice is small and sparse; and the relationship between instances is complex. For example, different regions in an organ or tissue may be highly heterogeneous, and different regions contribute differently to the classification task. Therefore, MIL shows great application potential in medical image analysis by combining bag-level label information with instance-level inference.

[0035] In order to solve the existing problems and deficiencies, it is necessary to fully combine medical resources to intelligently set the radiotherapy dose. The embodiment of the present application provides a tumor radiotherapy response prediction method and model based on multi-instance learning, which includes the following steps: first, construct a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data, mid-radiotherapy image data and radiotherapy dose setting data; then, extract multidimensional features from the radiotherapy image data set and combine and splice them to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolutional features and three-dimensional convolutional features; next, construct a Transformer-based multi-instance learning model; then, use the embedded representation of the multidimensional features as the model input, iteratively train the multi-instance learning model, and obtain a trained multi-instance learning model; finally, use the trained multi-instance learning model to predict the patient's mid-term radiotherapy response, and analyze and evaluate the model prediction process in combination with interpretable analysis, and perform model evaluation on the prediction results of the patient's tumor area. The present application provides a tumor radiotherapy response prediction method and system based on multi-instance learning. The prediction method has a high accuracy in classifying whether a tumor will respond or not. By predicting whether a tumor will respond after radiotherapy, it can intelligently assist doctors in making precise radiotherapy decisions for patients, adjust the radiotherapy dose for lung cancer patients, reduce the risk and harm of patient prognosis, and provide support for related radiotherapy clinical decisions.

[0036] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

[0037] See also Figure 1 The present application embodiment provides a method for predicting tumor radiotherapy response based on multi-instance learning, the method comprising the following steps:

[0038] Step S1, constructing a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data (PETpre), mid-radiotherapy image data (PETmid) and radiotherapy dose setting data (Dose).

[0039] Step S2: extract multidimensional features from the radiotherapy image data set and combine and concatenate them to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolutional features and three-dimensional convolutional features.

[0040] Step S3: construct a Transformer-based multi-instance learning model.

[0041] Step S4: using the embedded representation of the multi-dimensional features as the model input, iteratively training the multi-instance learning model to obtain a trained multi-instance learning model.

[0042] Step S5: Use the trained multi-instance learning model to predict the patient's mid-term response to radiotherapy, analyze and evaluate the model prediction process in combination with interpretable analysis, and perform model evaluation on the prediction results of the patient's tumor area.

[0043] In some embodiments, constructing a radiotherapy image dataset in step S1 includes:

[0044] Step S101: Acquire original tumor image data of the patient.

[0045] Step S102: pre-process the original tumor image data of the patient to construct a radiotherapy image data set.

[0046] The preprocessing process in step S102 includes: performing slicing processing, standardization and normalization processing, region of interest (ROI) positioning and multi-channel fusion processing on the original tumor image data of each patient in sequence.

[0047] In some embodiments, the original tumor image data of the patient is preprocessed in step S102 to construct a radiotherapy image data set, including:

[0048] Step S1021: Perform z-axis segmentation processing on the patient's pre-radiotherapy image data and radiotherapy dose setting data to form slices.

[0049] Step S1022: The size of each slice based on the z-axis is adjusted to a uniform H×W pixel size based on the largest tumor slice in the data set, and the features in the slice are normalized.

[0050] Step S1023: Masking the non-interested regions to zero and exclude them from the data set; combining with a multi-channel fusion strategy, i.e., the pre-radiotherapy image is the first channel and the radiotherapy dose setting image is the second channel, to obtain an input image after the fusion channels.

[0051] The features in the slice in step S1022 include: the three-dimensional coordinates (x, y, z) of each pixel provided by the patient's pre-radiotherapy image data and the standard uptake value (SUV) of the pixel before radiotherapy, and the three-dimensional coordinates of each pixel and the radiotherapy dose provided by the radiotherapy dose setting data; the feature x and y values ​​of the slice are normalized, and the formula is:

[0052]

[0053] Among them, S min and S max Represent the minimum and maximum values ​​in each slice respectively; this numerical processing strategy helps to eliminate the impact of different scale factors.

[0054] Specifically, in step S1, the data is preprocessed and a structured radiotherapy image dataset is constructed. Figure 2 The embodiment of the present application provides a data preprocessing flow chart of a tumor radiotherapy response prediction model based on multi-instance learning, including: after obtaining pre-radiotherapy image data (PETpre), mid-radiotherapy image data (PETmid), and radiotherapy dose setting data (Dose), the image data of each patient is firstly processed based on the z-axis continuous slicing to obtain N cross-sectional slices; then, a size standardization strategy is adopted to uniformly adjust the N obtained cross-sectional slices to the same size as the largest tumor slice in the data set. In addition, the x and y axis values ​​of each slice are normalized. Finally, ROI processing and channel fusion are performed to obtain the entire tumor input data set. Non-ROI areas, that is, those parts that do not belong to the tumor volume, are set to zero in the image through masking operations, and a multi-channel fusion method is adopted to use the pre-radiotherapy PET image (PETpre) as the first channel and the dose (Dose) image as the second channel, which is conducive to maintaining the clinical relevance of the tumor area.

[0055] In some embodiments, step S2 extracts multi-dimensional features from the radiotherapy image data set and combines and splices them, including:

[0056] Step S201: extract two-dimensional convolutional features and three-dimensional convolutional features of images from a radiotherapy image dataset simultaneously through a two-dimensional convolutional neural network (2D CNN) and a three-dimensional convolutional neural network (3D CNN).

[0057] Step S202: Use a combination method to concatenate the two-dimensional convolutional features and the three-dimensional convolutional features to obtain an embedded representation of multi-dimensional features for input to subsequent network predictions.

[0058] Step S203, the process of extracting multi-dimensional features, includes: equipping a plurality of convolutional layers with activation functions, and respectively constructing a two-dimensional convolutional neural network architecture and a three-dimensional convolutional neural network architecture.

[0059] Among them, the input data shape of the two-dimensional convolutional neural network is in the two-dimensional format of (B, N, C, H, W), where B represents the number of packages, that is, the number of patients, N represents the number of slices of a single patient, C is the number of channels of the slice, and H and W are the height and width of the slice, respectively; the input data shape of the three-dimensional convolutional neural network is in the three-dimensional format of (B, C, D, H, W), where the depth D is equal to the number of slices N, and H and W are the height and width of the slice, respectively.

[0060] Step S2 is mainly to comprehensively extract features from the preprocessed multidimensional image data. First, through a customized two-dimensional CNN architecture, two-dimensional convolution features are extracted to obtain a two-dimensional embedded representation; then, a three-dimensional CNN is used to extract spatial features through a three-dimensional convolution kernel, and the feature sequence is compressed and transformed; finally, based on the comprehensive extraction of two-dimensional and three-dimensional convolution features, splicing processing is performed to complete the generation of feature sequence embedded representation.

[0061] Specifically, in step S201, for two-dimensional feature extraction, the patient's slice input image (B, N, C, H, W) is subjected to continuous convolution operations of two convolutional layers (a convolutional layer with a 5×5 convolution kernel followed by ReLU activation and a 2×2 maximum pooling) to obtain preliminary feature extraction, and then processed by a fully connected layer operation to be converted into a two-dimensional embedding representation (B, N, M).

[0062] In step S201, a three-dimensional convolution operation is performed on the input three-dimensional data using a three-dimensional convolution kernel, which moves in three dimensions, so as to capture the spatial relationship in the volume data and extract the spatial features; the expression of the three-dimensional convolution operation is:

[0063]

[0064] in, represents the element at position (i, j, k) in the input feature map of layer l+1, represents the weight of the convolution kernel at position (p, q, r) in the lth layer, b l is the bias term, P, Q, and R are the sizes of the convolution kernel in three dimensions.

[0065] In some embodiments, two-dimensional convolutional features are converted into two-dimensional embedded representations through continuous convolutional pooling operations; three-dimensional convolutional features are converted into two-dimensional embedded representations through full-connected layer compression; the data shape of the two-dimensional embedded representation is (B, N, M), where M is the embedding dimension; the data embedding dimensions of the three-dimensional convolutional features are consistent with those of the two-dimensional convolutional features, which facilitates splicing processing.

[0066] It should be noted that here, the two-dimensional and three-dimensional embedding dimensions are consistent, and the specific embedding dimension M can be optimized and selected according to the actual scenario and data characteristics.

[0067] The 3D CNN architecture consists of three 3×3×3 3D convolutional layers, each of which is equipped with a ReLU activation function and a 2×2×2 maximum pooling to introduce nonlinear processing. An adaptive pooling layer is used as the final pooling layer to flexibly adjust the output size according to the input size and keep the output uniform. After the convolution and pooling operation, the spatial features are compressed and converted into a two-dimensional embedding representation through one or more fully connected layers with a shape of (B, N, M).

[0068] In some embodiments, constructing a Transformer-based multi-instance learning model in step S3 includes: constructing a Transformer multi-instance learning network architecture, the network architecture including a multi-head self-attention layer, a convolutional layer, and a fully connected layer. Figure 3 , an embodiment of the present application provides a model construction framework diagram in a tumor radiotherapy response prediction model based on multi-instance learning, including: introducing a multi-head attention mechanism, including a stacking setting of convolutional layers and fully connected layers, and initializing each network layer.

[0069] Specifically, in the Transformer-based multi-instance learning model, a multi-head attention module is used in combination with a feedforward network to calculate and process the positional representations of all elements of the processing sequence, wherein cls_token is added to the input feature sequence to enter the forward propagation process, and cls_token is specially designed as the first element of the input sequence; it is used to capture and represent the global information of the entire input sequence.

[0070] In some embodiments, in step S4, the deep learning training model is iteratively trained using the embedded representation of the multi-dimensional features as the input of the model, including:

[0071] Step S401: Calculate the loss between the predicted result and the true label through the loss function, then use the Adam optimizer to calculate the gradient, and then update the model parameters in reverse.

[0072] Step S402: Use the Leave-One-Out Cross-Validation (LOOCV) method to train and validate the model, monitor the performance of the model on the validation set, and adjust the training process until the performance no longer improves or reaches a predetermined number of iterations.

[0073] The loss function in step S401 adopts a binary cross entropy loss function, and the calculation formula is:

[0074]

[0075] Among them, y i is the true label of sample i, is the model's predicted probability for sample i, and N is the total number of samples.

[0076] In some embodiments, during the model training and verification process, LOOCV selects the data of one patient as the test set through each operation, and repeats the selection operation multiple times so that each patient is tested once; in the training stage, two different categories of patients (including 0 and 1 labels) are selected as the verification set to ensure that the model effectively distinguishes different types of radiotherapy reactions, so as to accurately select the model with the best performance.

[0077] Among them, different categories of patients include those who respond to radiotherapy and those who do not respond to radiotherapy; the classification of patients is obtained by classifying according to the change rate of the average standard uptake value of all tumor slice pixels between the middle and early stages of radiotherapy; the classification process includes: judging whether the average standard uptake value of all tumor slice pixels in the early stage of radiotherapy and the average standard uptake value of all tumor slice pixels in the middle stage of radiotherapy meet the corresponding relationship; if so, the patient is classified as a responder and recorded as label 1; if not, the patient is classified as a non-responder and recorded as label 0; the corresponding relationship is:

[0078]

[0079] Among them, ΔSUVmean represents the rate of change of the average standard uptake value of all tumor slice pixels between the middle and early stages of radiotherapy; PreSUVmean represents the average standard uptake value of all tumor slice pixels in the early stage of radiotherapy, and MidSUVmean represents the average standard uptake value of all tumor slice pixels in the middle stage of radiotherapy for a single pixel.

[0080] In some embodiments, after data preprocessing, tumor radiotherapy images are subjected to feature extraction and prediction using a multi-dimensional feature Transformer multi-instance learning algorithm. The algorithm pseudo code is shown in Table 1, which shows the algorithm flow.

[0081] Table 1 Multi-dimensional feature Transformer multi-instance learning algorithm processing flow chart

[0082]

[0083] In some embodiments, step S5 uses the trained multi-instance learning model to predict the patient's mid-term response to radiotherapy, and analyzes and evaluates the model prediction process in combination with interpretable analysis, including: recording the attention weight scores generated by the Transformer's attention mechanism, establishing interpretable analysis for the model's prediction process, and ensuring that the model provides a certain medical explanatory basis in medical application scenarios.

[0084] Specifically, in step S5, the trained multi-instance learning model is used to predict the patient's mid-term response to radiotherapy, and the prediction process is analyzed and evaluated in combination with interpretable analysis. The model is evaluated based on the prediction results of the patient's tumor area, including:

[0085] Step S501: Based on the test set, input into the trained classifier prediction to obtain all test results;

[0086] Step S502: Summarize all test results to calculate statistical indicators of model evaluation, such as accuracy, area under the receiver operating characteristic curve, etc.;

[0087] Step S503: construct an interpretable analysis in combination with the attention weights to analyze and evaluate the model prediction process.

[0088] Specifically, in the model prediction stage, cls_token passes through a fully connected layer to output the predicted probability and predicted category; based on the attention weights generated in the Transformer network, it can provide attention-based explanatory analysis for the model's decision-making process. By carefully analyzing different weights, it is possible to identify the specific focus of the model when processing three-dimensional and two-dimensional data. For example, does the model rely on overall volume information for prediction, or is it more inclined to rely on certain key local slices. Especially when processing multi-slice scan data, the attention mechanism can indicate which slices contain the key information required for diagnosis, allowing doctors to focus on these important images, thereby optimizing the diagnosis process and reducing the analysis of less important slices. At the same time, the model is evaluated based on the prediction results on the tumor test set and the output statistical indicators.

[0089] See also Figure 4The embodiment of the present application also provides a tumor radiotherapy response prediction system based on multi-instance learning, which uses the tumor radiotherapy response prediction method based on multi-instance learning described in the above embodiment to predict tumor radiotherapy response, including: a data set module 101, a feature extraction module 102, a model construction module 103, a model training module 104 and a model prediction analysis module 105 connected in sequence; wherein the data set module 101 is used to construct a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data (PETpre), mid-radiotherapy image data (PETmid) and radiotherapy dose setting data (Dose); the feature extraction module 102 is used to extract the data from the pre-radiotherapy image data (PETpre), the mid-radiotherapy image data (PETmid) and the ... Multidimensional features are extracted from the radiotherapy image data set and combined and spliced ​​to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolutional features and three-dimensional convolutional features; the model construction module 103 is used to build a multi-instance learning model based on Transformer; the model training module 104 is used to iteratively train the multi-instance learning model with the embedded representation of the multi-dimensional features as the model input to obtain a trained multi-instance learning model; the model prediction analysis module 105 is used to use the trained multi-instance learning model to predict the patient's mid-term response to radiotherapy, and analyze and evaluate the model prediction process in combination with interpretable analysis, and perform model evaluation on the prediction results of the patient's tumor area.

[0090] Specifically, the data set module 101 includes an image data preprocessing unit, which preprocesses, optimizes and constructs a structured radiotherapy image data set. According to the three-dimensional coordinates of each pixel in the provided FDG ET / CT image data and the corresponding SUV, as well as the three-dimensional position of the pixel recorded in the Dose radiotherapy dose image and the radiotherapy dose received, slices are sliced ​​based on the z-axis, and the N cross-sectional slices obtained are unified in size and normalized; in order to obtain a structured tumor radiotherapy input data set, ROI positioning processing operations and channel fusion operations are then performed, and the input image data set is obtained after fusion.

[0091] The feature extraction module 102 is mainly used to comprehensively extract multi-dimensional convolution features from multi-dimensional image data for subsequent prediction analysis; a combination method is used for feature extraction, and two-dimensional and three-dimensional convolution features are simultaneously extracted through traditional two-dimensional CNN and 3D CNN, and the concatenation is converted into a feature embedding sequence representation to obtain a specific embedding representation.

[0092] The model building module 103 includes a multi-head self-attention layer, a convolutional layer, a fully connected layer, etc.; based on the Transformer multi-instance learning network architecture, after entering the prediction stage, a specially designed cls_token is added to the input sequence, and the final category prediction is output through forward propagation prediction; the parameters of all network layers are initialized to prepare the model for training.

[0093] The model training module 104 is used to implement gradient calculation and model parameter update according to the designed binary cross entropy loss function and back propagation; during the iterative training process, the performance of the model on the validation set is monitored.

[0094] The model prediction analysis module 105 is used to predict the model classification results based on the optimized model parameters and an independent test set; based on the attention score generated in the Transformer network during the model prediction process, the relative importance of different input features considered by the model during prediction is evaluated, and key input features can be highlighted, thereby constructing the interpretability of the model.

[0095] The following is a detailed description of the tumor radiotherapy response prediction method and model based on multi-instance learning provided by the present application with respect to a specific embodiment.

[0096] First, the tumor radiotherapy response prediction model based on multi-instance learning proposed in the embodiment of the present application uses Pycharm as the development environment and is carried out on a computing platform equipped with an Intel Core i9 13900K processor and NVIDIA GeForce RTX 3090 dual graphics cards. The operating environment is the Ubuntu system, equipped with Python 3.8 and Pytorch framework.

[0097] In addition, it should be noted that the embodiments of the present application analyze and evaluate the radiotherapy response of locally advanced non-small cell lung tumors.

[0098] First, the image data preprocessing module performs slice processing based on the z-axis, standardization and normalization of slice size, processing of the region of interest (ROI), and channel fusion on the original radiotherapy image data to obtain the axial slice data set of the whole tumor. The size of each slice is 80×80 pixels, 80 pixels wide and 80 pixels high, so as to ensure the consistency of the input.

[0099] Then, feature extraction is performed on the preprocessed tumor dataset. In the two-dimensional feature extraction, the patient slice (2, 80, 80) passes through a set of convolutional network layers. The first feature extraction layer includes two convolutional layers with 5×5 convolution kernels, each followed by ReLU activation and 2×2 maximum pooling; the second feature extraction layer processes the final pooled feature map output by the first layer through a fully connected layer, flattens it and converts it into a 128-dimensional feature embedding; assuming that the i-th patient has N i Slice, then its two-dimensional feature embedding is (N i ,128); In 3D feature extraction, the slice image of the i-th patient (2,N i,80,80) First, a set of convolutional network layers are used for preliminary feature extraction; the first feature extraction layer consists of three 3×3×3 3D convolutional layers, each of which is followed by ReLU activation and 2×2×2 maximum pooling. Considering that there may be local extreme signals and abnormal features in the entire tumor internal area, which has an important influence on the identification and evaluation of the entire tumor area, the second feature extraction layer selects adaptive maximum pooling as the last pooling layer, thereby highlighting the maximum value of the corresponding area and more accurately performing risk assessment and treatment effect prediction on tumor sub-regions; assuming that the i-th patient has N i slices, its 3D feature embedding is in the form of (Ni,128).

[0100] Next, based on the Transformer multi-instance learning network architecture, a multi-instance learning training model is constructed through the model construction module and the model training module. The sequence representation of the feature embedding is used as the input of the main structure of the multi-instance learning network. In order to prevent overfitting, 4 heads are configured in the Transformer, and the dropout regularization technology is used in the Transformer structure, and a dropout ratio of 0.1 is introduced. The model training and parameter tuning are carried out through the training module, and the LOOCV strategy is used for the training and verification steps of the model. By grouping the patient data, it is ensured that each experiment uses samples of different patients as a test set for evaluation. The remaining patients are divided into training sets during the training stage, and two patient samples (including 0 and 1 labels) are selected from them as verification sets, which represent responses to radiotherapy and no obvious responses, respectively. It should be noted that in order to prevent data leakage, it is necessary to ensure that the tumor sample data of the same patient cannot appear in the training set, verification set and test set at the same time. And the optimal multi-instance learning model is determined based on the lowest loss function value obtained from the verification set.

[0101] Finally, based on the test set, the effectiveness of the multiple instance learning model was evaluated through the model prediction analysis module. The evaluation indicators include accuracy (Accuracy) and the area under the receiver operating characteristic curve (AUC). Referring to Table 2, the model can reach an AUC of 0.814 on the entire tumor test set; the Accuracy can reach 0.696, which is better than the classic ABMIL (Attentionbased Multiple Instance Learning) multiple instance learning model (AUC = 0.637; Accuracy = 0.652). It can be seen that the model has an advantage in predicting performance in patient tumor samples, and can more effectively predict and distinguish pixels with or without response. Therefore, based on the comprehensive evaluation indicators, it can be seen that the model has a better predictive effect on the prediction of patient radiotherapy response.

[0102] Table 2 Evaluation index results of patient tumor samples

[0103]

[0104] In addition, after the trained multi-instance learning model predicts whether the tumor responds to radiotherapy, based on the prediction results and process given by the multi-instance learning model, the embodiment of the present application performs an interpretable analysis on the obtained attention weights. The attention weights of the first seven slices of each patient are calculated, see Table 3, where the larger the weight value, the more important the slice is for the prediction. The attention weights of the two-dimensional and three-dimensional features are calculated separately, see Table 4, from which it can be obtained whether each patient attaches more importance to the two-dimensional features of the slice or the overall three-dimensional features in the prediction.

[0105] Table 3 Results of attention weight values ​​of patient tumor slice samples

[0106]

[0107]

[0108] Table 4 2D and 3D attention weights and results of patient tumor samples

[0109]

[0110]

[0111] Based on this, the method and system for predicting tumor radiotherapy response based on multi-instance learning provided in this application uses the patient's pre-radiotherapy FDG PET / CT images and Dose dose images as data input, and extracts two-dimensional and three-dimensional image features in combination with feature splicing technology to form a multi-dimensional combined feature embedding; finally, a multi-instance learning network architecture based on Transformer is used, combined with a multi-head attention mechanism to output classification prediction results, which helps to fully capture key features and generate attention weights based on the prediction process in weak label scenarios, thereby constructing an interpretable analysis and effectively evaluating the effect of the tumor radiotherapy model. The prediction model provided in this application has a high accuracy in classification prediction of whether or not to respond. By predicting whether the tumor responds after radiotherapy, it can intelligently assist doctors in adjusting the radiotherapy dose of lung cancer patients and provide support for related radiotherapy clinical decisions.

[0112] Based on the above technical scheme, the present application proposes a method and system for predicting tumor radiotherapy response based on multi-instance learning, and the method includes the following steps: first, construct a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data, mid-radiotherapy image data and radiotherapy dose setting data; then, extract multidimensional features from the radiotherapy image data set and combine and splice them to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolution features and three-dimensional convolution features; next, construct a Transformer-based multi-instance learning model; then, use the embedded representation of the multi-dimensional features as the model input, iteratively train the multi-instance learning model, and obtain a trained multi-instance learning model; finally, use the trained multi-instance learning model to predict the patient's mid-term radiotherapy response, and analyze and evaluate the model prediction process in combination with interpretable analysis, and perform model evaluation on the prediction results of the patient's tumor area.

[0113] The method for predicting tumor radiotherapy response based on multi-instance learning proposed in this application takes into account the particularity of actual medical weak label scenarios. This application constructs a multi-instance learning model based on Transformer and introduces a multi-head attention mechanism to optimize the prediction process, so as to predict the tumor's response to radiotherapy. Secondly, this application particularly emphasizes the extraction and fusion of two-dimensional and three-dimensional image features, and directly performs end-to-end prediction from PET / CT images and radiotherapy dose distribution maps based on the constructed feature engineering, thereby enhancing the multi-dimensional feature recognition ability of the model and significantly improving the recognition and classification accuracy of tumor radiotherapy response. Finally, this application provides an interpretable analysis of the model decision process through the distribution of attention weights, which can enhance doctors' understanding and trust in model decisions and provide strong support for tumor treatment strategies. In addition, the model provided in this application can be compared with other comparison algorithms on the whole tumor data set through evaluation indicators, which can verify that the constructed tumor radiotherapy response prediction model takes into account effectiveness, accuracy and interpretability, so that it can intelligently assist doctors in making accurate radiotherapy dose decisions for patients and provide support for clinical radiotherapy.

[0114] Those skilled in the art can understand that the above-mentioned embodiments are specific examples for implementing the present application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application, so the scope of protection of the present application shall be based on the scope defined in the claims.

Claims

1. A method for predicting tumor radiotherapy response based on multi-instance learning, characterized in that: The following steps are involved: Constructing a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data, mid-radiotherapy image data and radiotherapy dose setting data; Extracting multidimensional features from the radiotherapy image data set and combining and splicing them to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolution features and three-dimensional convolution features; Build a Transformer-based multi-instance learning model; Iteratively training the multi-instance learning model using the embedded representation of the multi-dimensional features as a model input to obtain a trained multi-instance learning model; The trained multi-instance learning model is used to predict the patient's mid-term response to radiotherapy, and the model prediction process is analyzed and evaluated in combination with interpretable analysis, and the model is evaluated on the prediction results of the patient's tumor area.

2. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 1, characterized in that: Construct a radiotherapy image dataset, including: Obtain the patient's original tumor imaging data; Preprocess the original tumor image data of the patient to construct a radiotherapy image dataset; The preprocessing process includes: slicing, standardizing, normalizing, locating the region of interest, and multi-channel fusion processing of the original tumor image data of each patient in sequence.

3. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 2, characterized in that: Preprocess the original tumor image data of the patient to construct a radiotherapy image dataset, including: Performing z-axis segmentation processing on the patient's pre-radiotherapy image data and the radiotherapy dose setting data to form slices; The size of each slice based on the z-axis is adjusted to a uniform H×W pixel size based on the largest tumor slice in the dataset, and the features in the slice are normalized; The non-interested areas are masked and set to zero to exclude them from the data set. The multi-channel fusion strategy is combined, that is, the pre-radiotherapy image is the first channel and the radiotherapy dose setting image is the second channel to obtain the input image after the fusion channel. The features in the slice include: the three-dimensional coordinates (x, y, z) of each pixel provided by the pre-radiotherapy image data of the patient and the standard uptake value of the pixel before radiotherapy, and the three-dimensional coordinates of each pixel and the radiotherapy dose provided by the radiotherapy dose setting data; The characteristic x and y values ​​of the slice are normalized, and the formula is: Among them, S min and S max Represent the minimum and maximum values ​​in each slice, respectively.

4. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 1, characterized in that: Extracting multidimensional features from the radiotherapy image dataset and combining and splicing them, including: By using a two-dimensional convolutional neural network and a three-dimensional convolutional neural network, two-dimensional convolutional features and three-dimensional convolutional features of the image are simultaneously extracted from the radiotherapy image dataset; A combination method is used to concatenate the two-dimensional convolutional features and the three-dimensional convolutional features to obtain an embedded representation of multi-dimensional features for input into subsequent network predictions. The process of extracting multi-dimensional features includes: constructing a two-dimensional convolutional neural network architecture and a three-dimensional convolutional neural network architecture through multiple convolutional layers equipped with activation functions; Among them, the input data shape of the two-dimensional convolutional neural network is in the two-dimensional format of (B, N, C, H, W), B represents the number of packages, that is, the number of patients, N represents the number of slices of a single patient, C is the number of channels of the slice, and H and W are the height and width of the slice respectively; The input data shape of the 3D convolutional neural network is in the 3D format of (B, C, D, H, W), where the depth D is equal to the number of slices N, and H and W are the height and width of the slices respectively; a 3D convolution operation is performed on the input 3D data using a 3D convolution kernel, which moves in three dimensions, and can capture the spatial relationship in the volume data and extract spatial features; the expression of the 3D convolution operation is: in, represents the element at position (i, j, k) in the input feature map of layer l+1, represents the weight of the convolution kernel at position (p, q, r) in the lth layer, b l is the bias term, P, Q, and R are the sizes of the convolution kernel in three dimensions.

5. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 4, characterized in that: The two-dimensional convolutional features are converted into two-dimensional embedding representations through continuous convolutional pooling operations; the three-dimensional convolutional features are converted into two-dimensional embedding representations through full connection layer compression; The data shape of the two-dimensional embedding representation is (B, N, M), where M is the dimension of the embedding; The data embedding dimension of the three-dimensional convolution feature is consistent with that of the two-dimensional convolution feature, which facilitates splicing processing.

6. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 1, characterized in that: Build a Transformer-based multi-instance learning model, including: Construct a Transformer multi-instance learning network architecture, which includes a multi-head self-attention layer, a convolutional layer, and a fully connected layer; In the Transformer-based multi-instance learning model, a multi-head attention module is used in combination with a feedforward network to calculate and process the positional representation of all elements of the processing sequence, wherein cls_token is added to the input feature sequence to enter the forward propagation process, and cls_token is specially designed as the first element of the input sequence; it is used to capture and represent the global information of the entire input sequence.

7. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 1, characterized in that: The deep learning training model is iteratively trained using the embedded representation of the multi-dimensional features as the input of the model, including: The loss function is used to calculate the loss between the predicted result and the true label, and then the optimizer calculates the gradient to update the model parameters in reverse. Use the leave-one-out cross-validation method to train and validate the model, monitor the performance of the model on the validation set, and adjust the training process until the performance no longer improves or reaches the predetermined number of iterations; The loss function adopts the binary cross entropy loss function, and the calculation formula is: Among them, y i is the true label of sample i, is the model's predicted probability for sample i, and N is the total number of samples.

8. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 7, characterized in that: During model training and validation, Leave-one-out cross-validation selects one patient's data as the test set each time, and repeats the selection operation multiple times so that each patient is tested once. In the training phase, two different categories of patients are selected as the validation set to ensure that the model can effectively distinguish different types of radiotherapy responses, so as to accurately select the best performing model. The different categories of patients include those who respond to radiotherapy and those who do not respond to radiotherapy. The patient categories were classified based on the change rate of the average standard uptake value of all tumor slice pixels between the middle and early stages of radiotherapy; The classification process includes: determining whether the average standard uptake value of all tumor slice pixels in the early stage of radiotherapy and the average standard uptake value of all tumor slice pixels in the middle stage of radiotherapy meet the corresponding relationship; if so, the patient is classified as a responder and recorded as label 1; If not, the patient is classified as non-responsive and recorded as label 0; The corresponding relationship is: Among them, ΔSUVmean represents the change rate of the average standard uptake value of all tumor slice pixels between the middle and early stages of radiotherapy; PreSUVmean represents the average standard uptake value of all tumor slice pixels in the early stage of radiotherapy. MidSUVmean represents the average standard uptake value of all tumor slice pixels during the middle period of radiotherapy.

9. The method for predicting tumor radiotherapy response based on multi-instance learning according to claim 1, characterized in that: The trained multi-instance learning model is used to predict the patient's mid-term response to radiotherapy, and the model prediction process is analyzed and evaluated in combination with interpretable analysis, including: Record the attention weight scores generated by the Transformer's attention mechanism, establish an interpretable analysis for the model's prediction process, and ensure that the model provides a certain medical interpretability basis in medical application scenarios.

10. A tumor radiotherapy response prediction system based on multiple instance learning, which uses the tumor radiotherapy response prediction method based on multiple instance learning as described in any one of claims 1 to 9 to predict tumor radiotherapy response, characterized in that: include: The data set module, feature extraction module, model construction module, model training module and model prediction analysis module are connected in sequence; among them, The data set module is used to construct a radiotherapy image data set; the radiotherapy image data set includes the patient's pre-radiotherapy image data, mid-radiotherapy image data and radiotherapy dose setting data; The feature extraction module is used to extract multidimensional features from the radiotherapy image data set and combine and splice them to obtain an embedded representation of the multidimensional features; the multidimensional features include two-dimensional convolution features and three-dimensional convolution features; The model building module is used to build a Transformer-based multi-instance learning model; The model training module is used to iteratively train the multi-instance learning model using the embedded representation of the multi-dimensional features as the model input to obtain a trained multi-instance learning model; The model prediction analysis module is used to use the trained multi-instance learning model to predict the patient's mid-term response to radiotherapy, and analyze and evaluate the model prediction process in combination with interpretable analysis, and perform model evaluation on the prediction results of the patient's tumor area.

Images