Methods, systems, and software products for predicting tumor radiotherapy response based on 3D imaging and residual networks

Abstract

This invention discloses a method, system, and program product for predicting tumor radiotherapy response based on three-dimensional images and residual networks. The prediction method includes steps such as image acquisition and slicing, region of interest processing and image fusion, three-dimensional feature extraction, construction of a multi-gated hybrid expert model, model training, prediction of radiotherapy response regression values, and model performance evaluation. First, the patient's three-dimensional PETpre and three-dimensional Dose images are preprocessed. Second, the preprocessed images are input into a 3D CNN network with shared hard parameters to extract three-dimensional spatial features. Finally, the extracted three-dimensional features are used as independent variables X, and the average value of the radiation vehicle (SUV) and its rate of change are used as dependent variables Y, respectively, and input into the trained multi-gated hybrid expert model for prediction. This invention combines ResNet and 3D CNN technologies, utilizing a multi-task learning strategy to achieve collaborative prediction of the average value of the SUV and its rate of change, enabling more effective use of limited data and improving prediction accuracy and model generalization ability.

Description

Technical Field

[0001] This invention relates to the field of medical image processing and artificial intelligence, and more specifically, to a method, system, and program product for predicting tumor radiotherapy response based on three-dimensional images and residual networks. Background Technology

[0002] PET / CT is a fusion imaging technology that integrates the functional information provided by positron emission tomography (PET) and the anatomical information provided by computed tomography (CT), enabling more precise localization, assessment, and monitoring of various diseases, especially malignant tumors. With the development of precision radiotherapy for tumors, the application of PET / CT in radiotherapy planning is becoming increasingly widespread. Among these applications, PET / CT-based prediction of tumor radiotherapy response is of great significance in cancer treatment. Accurate prediction of tumor response to radiotherapy not only helps doctors develop optimal treatment plans but also contributes to the development of personalized medicine. The Standard Uptake Value (SUV) is an important indicator for quantitatively assessing tumor metabolic activity. By measuring the SUV value of a tumor region, areas of high metabolic activity can be effectively identified, which is directly significant for determining tumor invasiveness and treatment response. Therefore, by analyzing the dynamic changes in SUV values, the treatment outcome for patients can be accurately predicted.

[0003] With the continuous development of deep learning technology, its application in medical image analysis has gradually expanded from early classification tasks such as disease identification and diagnosis to more complex regression tasks, including evaluating treatment effectiveness and quantifying changes in lesion areas. In the field of tumor radiotherapy, accurately predicting patients' responses to treatment is crucial for developing personalized treatment plans.

[0004] Current research on small-sample medical data often employs strategies such as generative adversarial networks (GANs) to synthesize data samples and transfer learning techniques to leverage knowledge learned from other large-sample tasks. While these methods improve the efficiency of small-sample data processing to some extent, they still suffer from insufficient generalization ability to specific tasks. Due to the complexity of radiotherapy responses, single-task prediction models often struggle to accurately predict treatment efficacy indicators across different dimensions. Multi-task learning (MTL) offers a solution to this specific situation by simultaneously learning multiple related tasks within the same model, thereby improving the learning efficiency and prediction accuracy of each task. In predicting tumor radiotherapy responses, MTL can handle multiple related prediction tasks simultaneously. This approach not only improves the model's efficiency in utilizing data but also helps to better understand the intrinsic relationships between different radiotherapy responses, even with limited sample sizes. However, current research on predicting tumor radiotherapy responses rarely utilizes MTL to handle related regression problems, and no prediction model based on MTL simultaneously predicts the average SUV and its rate of change.

[0005] The fundamental assumption behind Multi-Task Learning (MTL) is that different tasks can share useful information that might be unavailable if learned in individual tasks. Sharing mechanisms in MTL can generally be categorized into hard parameter sharing and soft parameter sharing. In hard parameter sharing, multiple tasks share the weights of some neural network layers, such as sharing the first few convolutional layers, while subsequent specific layers are designed independently for each task. In soft parameter sharing, each task has an independent model, but additional regularization terms are introduced to ensure similarity of model parameters. Within the soft parameter sharing framework, the Multi-gate Mixture-of-Experts (MMoE) model provides an advanced framework for optimizing multi-task learning performance. The core design of the MMoE model includes multiple expert networks and multiple gating units. Each expert network is responsible for capturing a unique set of features from the input data, while the gating units dynamically adjust the contributions of each expert to each task. Furthermore, each task processes the fused information through an independent regression tower structure to produce the final output, ensuring specialized processing of tasks and independence of outputs.

[0006] Therefore, developing a tumor radiotherapy response prediction method based on three-dimensional images and residual networks, and achieving dual-task joint prediction of the mean SUV and its rate of change, is a technical problem that urgently needs to be solved. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a method, system, and program product for predicting tumor radiotherapy response based on 3D imaging and residual networks. The method first fuses a 2D PETpre image and a 2D Dose image before radiotherapy. Then, it uses a 3D CNN network with hard parameter sharing to extract features from the fused image. Next, the extracted features are input into a multi-gated hybrid expert model, which simultaneously predicts the average value and rate of change of SUV (Radio-Temperature Response) through a multi-task learning framework. This addresses the problems of existing models' inability to achieve joint prediction of two tasks, insufficient learning efficiency, and inadequate prediction accuracy.

[0008] To achieve the above objectives, in a first aspect, the present invention provides a method for predicting tumor radiotherapy response based on three-dimensional imaging and residual networks, characterized by comprising the following steps:

[0009] Step S1, Image Acquisition and Slicing: Acquire three-dimensional PETpre and three-dimensional Dose images of the patient before radiotherapy, and perform continuous slicing processing on the three-dimensional PETpre and three-dimensional Dose images based on the Z-axis; standardize and normalize the slice size to obtain two-dimensional PETpre and two-dimensional Dose images;

[0010] Step S2, Region of Interest Processing and Image Fusion: The two-dimensional PETpre image and the two-dimensional Dose image are divided into regions of interest and the tumor region data is retained. The pixel values ​​of non-tumor volume regions are set to zero through masking operations. The PETpre image and Dose image after the region of interest processing are fused to obtain the fused image dataset as the entire tumor input dataset.

[0011] Step S3, 3D Feature Extraction: Feature extraction is performed on the fused image using a 3D CNN network with hard parameter sharing;

[0012] Step S4: Construct a multi-gated hybrid expert model (3D ResMMoE model): Using ResNet as the expert network, construct N parallel expert networks, each of which independently learns different feature representations of the input data; design an adaptive gating mechanism, calculate the weight coefficients of each expert network through the softmax function, and realize the dynamic fusion of the expert network outputs; construct an independent regression tower structure for each prediction task, process the fused feature information, and generate the prediction output of the corresponding task;

[0013] Step S5, Model Training: The multi-gated hybrid expert model is trained using gradient descent, with mean squared error as the loss function to measure the difference between the predicted and actual values; the model is iteratively optimized until the performance no longer improves or the predetermined number of iterations is reached, and the optimal model parameters are obtained and saved.

[0014] Step S6: Predict the average value and rate of change of tumor standard uptake during radiotherapy: Input the fused image into the 3D CNN network to extract three-dimensional spatial features; use the extracted three-dimensional features as independent variable X, and the average value and rate of change of tumor standard uptake as dependent variable Y, respectively, and input them into the trained multi-gated hybrid expert model to construct a dual-task regression problem; the last layer in the regression tower structure is set as a fully connected layer with an output dimension of 1. Using two independent regression tower structures, the average value and rate of change of tumor standard uptake can be predicted respectively.

[0015] Step S7, Model Performance Evaluation: The performance of the multi-gated hybrid expert model is evaluated using the leave-one-out cross-validation strategy.

[0016] The proposed prediction method combines a hard-parameter-sharing 3D CNN network to extract 3D features, which not only improves the efficiency of parameter utilization but also enhances the model's generalization ability across different tasks. Simultaneously, by combining a multi-gated hybrid expert model within a soft-parameter-sharing framework, the multi-task learning paradigm is applied to the prediction of small sample data. This not only enables collaborative prediction of multiple regression predictions, improving computational efficiency, but also leverages the correlation and shared information features of multiple tasks to enhance the model's prediction performance and generalization ability.

[0017] Further, in step S1, the size of each slice is adjusted to the same size (H, W) as the largest tumor slice in the dataset; in the normalization process, the x and y axis values ​​of each slice are adjusted to a range between 0 and 1, using the following formula: ,in, and These represent the minimum and maximum values ​​in each slice, respectively.

[0018] Furthermore, in step S2, the image fusion employs a multi-channel fusion method.

[0019] Further, in step S3, the structure of the 3D CNN network is as follows: the convolutional kernels of the first, second, and third convolutional layers are all 3D convolutional kernels of size 3*3*3, the padding parameter is set to 1, and the convolution stride is set to 2; each convolutional layer is followed by a ReLU activation function, the calculation formula of which is... Each convolutional layer is followed by a max pooling operation with a kernel size of 2*2*2 and a stride of 2. A dropout layer is set after the third pooling layer with a dropout rate of 0.3. An adaptive max pooling layer is set after the dropout layer to adjust the feature map to a fixed size (1*10*10). Finally, the three-dimensional feature map is converted into a one-dimensional feature vector with a feature dimension of 6400 through a flattening operation, which serves as the input for the next stage.

[0020] Furthermore, the multi-gated hybrid expert model in step S4 includes two parallel expert networks, each consisting of two cascaded residual blocks and a fully connected output layer. The two residual blocks have the same structure, with each residual block structured as follows: First, the input features are mapped to a 32-dimensional hidden space through the first fully connected layer, then a non-linear transformation is introduced using the ReLU activation function, followed by regularization using a Dropout layer with a dropout rate of 0.4. Then, the features are mapped back to the original input dimension through the second fully connected layer. Finally, the original input is added to the transformed features to achieve a residual connection, and the output of the residual block is obtained through the ReLU activation function. The expert network then maps the features processed by the residual blocks to the target output dimension through a fully connected layer, completing the final feature transformation.

[0021] Furthermore, the regression tower in step S4 consists of two fully connected layers; the first fully connected layer maps the input features to a 10-dimensional hidden space and introduces a nonlinear transformation through the ReLU activation function, while using a Dropout layer with a dropout rate of 0.4 for regularization; the second fully connected layer maps the features to a 1-dimensional output; each regression task is configured with an independent tower structure, which is used to predict the average value of tumor standard uptake and its rate of change.

[0022] Further, in step S5, the formula for calculating the loss function is: ,in, It is a sample The true value, It is the model on the sample The predicted value, It represents the total number of samples.

[0023] Furthermore, in step S7, a leave-one-out cross-validation strategy is used to evaluate the model: each time, one patient is selected from the dataset as the test set, and the remaining patients are used as the training set, and this process is repeated until all patients have been evaluated as the test set once. In each round of training, in order to accurately select the best-performing model, two patients are specifically selected as the validation set, one of whom responds to radiotherapy (high rate of change in SUV value) and the other of whom does not respond significantly to radiotherapy (low rate of change in SUV value). The performance evaluation index of the model is the root mean square error (RMSE). By calculating the overall average RMSE and the RMSE of individual patients, the overall predictive performance and individual predictive accuracy of the model are comprehensively evaluated.

[0024] Secondly, the present invention provides a tumor radiotherapy response prediction system based on three-dimensional imaging and residual networks, comprising:

[0025] The image preprocessing module is used to perform continuous slicing processing based on the Z-axis on the acquired 3D PETpre and 3D Dose images, and to standardize and normalize the slice size to obtain 2D PETpre and 2D Dose images; then, the 2D PETpre and 2D Dose images are divided into regions of interest and the tumor region data is retained; the PETpre and Dose images after region of interest processing are fused to obtain the fused image dataset as the entire tumor input dataset;

[0026] The feature extraction module uses a 3D CNN network with shared hard parameters to extract features from the fused image dataset;

[0027] The model building module uses ResNet as the expert network to construct N parallel expert networks, each of which independently learns different feature representations of the input data. An adaptive gating mechanism is designed to calculate the weight coefficients of each expert network through the softmax function, thereby achieving dynamic fusion of the expert network outputs. An independent regression tower structure is constructed for each prediction task to process the fused feature information and generate the prediction output for the corresponding task.

[0028] The model training module is used to train the multi-gated hybrid expert model, obtain and save the optimal model parameters;

[0029] The prediction module includes a trained multi-gated hybrid expert model and a 3D CNN network for feature extraction. It is used to input the fused image into the 3D CNN network to extract three-dimensional features, and then input the extracted features into the trained multi-gated hybrid expert model to generate prediction results.

[0030] The model evaluation module is used to evaluate and test the best model saved in the model training module and calculate its performance metrics on the test set.

[0031] Finally, the present invention provides a computer program product that, when run on a computer, causes the computer to perform the tumor radiotherapy response prediction method based on three-dimensional images and residual networks as described above.

[0032] Compared with the prior art, the present invention has the following technical effects:

[0033] (1) This invention employs a 3D CNN network with hard parameter sharing to extract three-dimensional features from images. Compared with traditional methods, it can more comprehensively extract and utilize spatial information and deep features in image data, thereby providing richer and more accurate feature representations. At the same time, hard parameter sharing not only improves the efficiency of parameter utilization, but also enhances the model's generalization ability on different tasks;

[0034] (2) This invention applies the multi-task learning paradigm to the prediction of small sample data, which not only enables the collaborative prediction of multiple regression prediction values ​​and improves computational efficiency, but also improves the prediction performance and generalization ability of the model by utilizing the correlation and shared information of multiple tasks.

[0035] (3) The present invention constructs a tumor radiotherapy response prediction method and system based on three-dimensional images and residual networks. By combining ResNet and 3D CNN technology, the multi-task learning strategy is used to achieve collaborative prediction of the average value of SUV and its rate of change, which can make more effective use of limited data and improve the accuracy of prediction and the generalization ability of the model. Attached Figure Description

[0036] The invention, its features and advantages will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0037] Figure 1 This is a flowchart of the steps in a tumor radiotherapy response prediction method according to an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the prediction model in a tumor radiotherapy response prediction method according to an embodiment of the present invention;

[0039] Figure 3 This is a flowchart of tumor image data preprocessing in one embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of a tumor radiotherapy response prediction system in one embodiment of the present invention;

[0041] Figure 5 This is a line graph showing the RMSE value of the average change rate of various SUV models in one embodiment of the present invention;

[0042] Figure 6 This is a line graph showing the average RMSE value of each SUV model in one embodiment of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but these are not intended to limit the scope of the invention.

[0044] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that well-known algorithms and models are not shown in detail to avoid obscuring the gist of the invention; and the multi-channel fusion methods, 3D CNN networks, ResNet networks, gradient descent, and other techniques involved in the following effect examples are readily available prior art.

[0045] Furthermore, the execution order of actions, steps, etc. in the apparatus and methods shown in the claims, specification, and drawings can be implemented in any order, unless a specific order is explicitly specified, and as long as the output of the preceding processing is not used in the subsequent processing.

[0046] Example

[0047] See Figure 1 and Figure 2 This embodiment provides a method for predicting tumor radiotherapy response based on three-dimensional images and residual networks, including the following steps:

[0048] Step S1, Image Acquisition and Slicing: See Figure 3 The procedure involves acquiring three-dimensional PETpre and three-dimensional Dose images of the patient before radiotherapy, and then performing continuous slicing processing on the three-dimensional PETpre and three-dimensional Dose images based on the Z-axis. The slice sizes are then standardized and normalized to obtain two-dimensional PETpre and two-dimensional Dose images. As an example, the size of each slice is uniformly 80 pixels * 80 pixels.

[0049] Normalization refers to normalizing the x and y axis values ​​of each slice. This normalization process is achieved by adjusting the value range to between 0 and 1, and the formula is as follows: ,in, and These represent the minimum and maximum values ​​in each slice, respectively.

[0050] Step S2, Region of Interest Processing and Image Fusion: See Figure 3 The two-dimensional PETpre and Dose images are divided into regions of interest (ROIs) while retaining tumor region data. A masking operation is used to set the pixel values ​​of non-tumor volume regions to zero. The PETpre and Dose images, processed with the ROIs, are then fused to obtain a fused image dataset, which serves as the entire tumor input dataset. Preferably, the image fusion employs a multi-channel fusion method. See [example description missing] for further details. Figure 3 The pre-radiotherapy PET image (PETpre) is used as the first input channel, and the dose image is used as the second input channel for processing.

[0051] Step S3, 3D Feature Extraction: The fused image is subjected to feature extraction using a 3D CNN network with hard parameter sharing.

[0052] As a preferred technical solution, in step S3, the structure of the 3D CNN network is as follows: the convolutional kernels of the first, second, and third convolutional layers are all three-dimensional convolutional kernels of size 3*3*3, the padding parameter is set to 1, and the convolution stride is set to 2; each convolutional layer is followed by a ReLU activation function, the calculation formula of which is... Each convolutional layer is followed by a max pooling operation with a kernel size of 2*2*2 and a stride of 2. A dropout layer is set after the third pooling layer with a dropout rate of 0.3. An adaptive max pooling layer is set after the dropout layer to adjust the feature map to a fixed size (1*10*10). Finally, the three-dimensional feature map is converted into a one-dimensional feature vector with a feature dimension of 6400 through a flattening operation, which serves as the input for the next stage.

[0053] Step S4: Construct a multi-gated hybrid expert model: Using the ResNet network as the expert network, construct N parallel expert networks, each of which independently learns different feature representations of the input data; design an adaptive gating mechanism, calculate the weight coefficients of each expert network through the softmax function, and realize the dynamic fusion of the expert network outputs; construct an independent regression tower structure for each prediction task, process the fused feature information, and generate the prediction output of the corresponding task;

[0054] As a preferred technical solution, the multi-gated hybrid expert model in step S4 includes two parallel expert networks. Each expert network consists of two cascaded residual blocks and a fully connected output layer. The two residual blocks have the same structure. The specific structure of each residual block is as follows: First, the input features are mapped to a 32-dimensional hidden space through the first fully connected layer, a nonlinear transformation is introduced through the ReLU activation function, and regularization is performed using a Dropout layer with a dropout rate of 0.4. Then, the features are mapped back to the original input dimension through the second fully connected layer. Finally, the original input and the transformed features are added to achieve residual connection, and the output of the residual block is obtained through the ReLU activation function. Finally, the expert network maps the features processed by the residual blocks to the target output dimension through the fully connected layer, completing the final transformation of the features.

[0055] Step S5, Model Training: The multi-gated hybrid expert model is trained using gradient descent, with mean squared error as the loss function to measure the difference between the predicted and actual values. Iterative optimization is performed until performance no longer improves or a predetermined number of iterations is reached, and the optimal model parameters are obtained and saved. In step S5, the loss function is calculated using the following formula: ,in, It is a sample The true value, It is the model on the sample The predicted value, It represents the total number of samples.

[0056] As an example, during the model training phase, the model training parameters were optimized to 300 epochs, starting with a learning rate of 0.001, and the weights were adjusted using the Adam optimizer with a weight decay of 10⁻⁴. To further avoid overfitting and optimize the training process, an early stopping strategy was implemented, with a patience value of 100, meaning that training would stop if the performance on the validation set did not improve within 100 consecutive epochs.

[0057] Step S6: Predict the average value and rate of change of tumor standard uptake values ​​during mid-radiotherapy: Input the fused image into the 3D CNN network to extract three-dimensional spatial features; use the extracted three-dimensional features as independent variables X, and the average value and rate of change of tumor standard uptake values ​​SUV as dependent variables Y, respectively, and input them into the trained multi-gated hybrid expert model to construct a dual-task regression problem; the last layer in the regression tower structure is set as a fully connected layer with an output dimension of 1. Using two independent regression tower structures, the average value and rate of change of tumor standard uptake values ​​can be predicted respectively.

[0058] Step S7: Performance evaluation of the prediction model: The model performance is evaluated using the leave-one-out cross-validation strategy.

[0059] The leave-one-out cross-validation strategy is implemented as follows: Each time, one patient is selected from the dataset as the test set, and the remaining patients are used as the training set. This process is iterated until all patients have been evaluated once as part of the test set. During each training round, to accurately select the best-performing model, two patients are specifically selected as the validation set: one patient responds to radiotherapy (high rate of change in SUV value), and the other patient does not respond significantly to radiotherapy (low rate of change in SUV value). The model performance is evaluated using the root mean square error (RMSE). By calculating both the overall average RMSE and the RMSE of individual patients, the overall predictive performance and individual predictive accuracy of the model are comprehensively evaluated.

[0060] See Figure 4 This embodiment employs a tumor radiotherapy response prediction system based on multi-3D imaging and residual networks to implement the above prediction method. The prediction system includes:

[0061] The image preprocessing module is used to perform continuous slicing processing based on the Z-axis on the acquired 3D PETpre and 3D Dose images, and to standardize and normalize the slice size to obtain 2D PETpre and 2D Dose images; then, the 2D PETpre and 2D Dose images are divided into regions of interest and the tumor region data is retained; the PETpre and Dose images after region of interest processing are fused to obtain the fused image dataset as the entire tumor input dataset;

[0062] The feature extraction module uses a 3D CNN network with shared hard parameters to extract features from the fused image dataset;

[0063] The model building module uses ResNet as the expert network to construct two parallel expert networks, each of which independently learns different feature representations of the input data. An adaptive gating mechanism is designed to calculate the weight coefficients of each expert network through the softmax function, thereby achieving dynamic fusion of the expert network outputs. An independent regression tower structure is constructed for each prediction task to process the fused feature information and generate the prediction output for the corresponding task.

[0064] The model training module is used to train the multi-gated hybrid expert model, obtain and save the optimal model parameters;

[0065] The prediction module includes a trained multi-gated hybrid expert model and a 3D CNN network for feature extraction. It is used to input the fused image into the 3D CNN network to extract three-dimensional features, and then input the extracted features into the trained multi-gated hybrid expert model to generate prediction results.

[0066] The model evaluation module is used to evaluate and test the best model saved in the model training module and calculate its performance metrics on the test set.

[0067] The prediction system preprocesses tumor imaging data and radiotherapy dose data using the aforementioned prediction method, and then predicts the average SUV and its rate of change. RMSE is selected as the model evaluation index. By calculating the overall average RMSE and the RMSE of individual patients, the overall predictive performance and individual predictive accuracy of the model are comprehensively evaluated.

[0068] Leave-one-out cross-validation was employed to evaluate model performance. The validation process was as follows: in each round of validation, one patient was selected from the dataset as the test set, and the remaining patients were used as the training set. This process was iterated until all patients had been evaluated once as part of the test set. During each training round, to accurately select the best-performing model, two patients were specifically selected as the validation set: one patient responded to radiotherapy (high rate of change in SUV value), and the other patient did not respond significantly to radiotherapy (low rate of change in SUV value). RMSE was used as the model evaluation metric, comprehensively evaluating model performance from both the overall average RMSE and the RMSE of individual patients to verify the model's predictive accuracy and effectiveness in practical applications. Classic deep learning models and intermediate models (including 3D MMoE, 3D CNN, and 3D CNN with hard sharing mechanism) were compared and analyzed. The difference between 3D MMoE and the 3D ResMMoE model is that 3D MMoE uses a basic fully connected neural network for its expert network, containing two fully connected layers and one activation layer. All control models use the same training parameter configuration as the model of this invention, including hyperparameters such as training epochs and learning rate. Figure 5 , Figure 6 Table 1 shows the RMSE values ​​of the 3D ResMMoE model and the control model of this invention in predicting the rate of change of SUV average value and the SUV average value.

[0069] Table 1. Rate of Change of SUV Average Value and RMSE Value of SUV Average Value for Each Model

[0070] RMSE 3D ResMMoE 3D MMoE 3D CNN Hard parameter sharing SUV average change rate 0.196 0.223 0.223 0.257 SUV average 1.599 1.653 1.730 1.675

[0071] As shown in Table 1, the 3D ResMMoE model proposed in this invention significantly outperforms the other three models in predicting the average change rate and average value of SUVs. At the same time, the results of 3D MMoE are also better than 3D CNN, which demonstrates the advantages of the MMoE multi-task learning framework.

[0072] The aforementioned method for predicting tumor radiotherapy response based on 3D imaging and residual networks can be embodied in the form of a computer program product or a software functional unit. If this method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, the essence of this technical solution, or the part that contributes to the prior art, or a portion of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic system (which may be a personal computer, server, or network system, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0073] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0074] In summary, this invention provides a method, system, and program product for predicting tumor radiotherapy response based on 3D images and residual networks. The prediction method includes steps such as image acquisition and slicing, region of interest processing and image fusion, 3D feature extraction, construction of a multi-gated hybrid expert model, model training, prediction of radiotherapy response regression values, and model performance evaluation. First, the patient's 3D PETpre and 3D Dose images are preprocessed. Second, the preprocessed images are input into a 3D CNN network with shared hard parameters to extract 3D spatial features. Finally, the extracted 3D features are used as independent variables X, and the average value of the SUV and its rate of change are used as dependent variables Y, respectively, and input into the trained multi-gated hybrid expert model for prediction. This invention, by combining ResNet and 3D CNN technologies and utilizing a multi-task learning strategy, achieves collaborative prediction of the average value of the SUV and its rate of change, enabling more effective use of limited data and improving prediction accuracy and model generalization ability.

[0075] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0076] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above. Systems and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for predicting tumor radiotherapy response based on three-dimensional imaging and residual networks, characterized in that, Includes the following steps: Step S1, Image Acquisition and Slicing: Acquire three-dimensional PETpre and three-dimensional Dose images of the patient before radiotherapy, and perform continuous slicing processing on the three-dimensional PETpre and three-dimensional Dose images based on the Z-axis; standardize and normalize the slice size to obtain two-dimensional PETpre and two-dimensional Dose images; Step S2, Region of Interest Processing and Image Fusion: The two-dimensional PETpre image and the two-dimensional Dose image are divided into regions of interest and the tumor region data is retained. The pixel values ​​of non-tumor volume regions are set to zero through masking operations. The PETpre image and Dose image processed from the region of interest are fused together to obtain the fused image dataset, which is used as the entire tumor input dataset. Step S3, 3D Feature Extraction: Feature extraction is performed on the fused image using a 3D CNN network with hard parameter sharing. The structure of the 3D CNN network is as follows: the convolution kernels of the first, second, and third convolutional layers are all 3D convolution kernels of size 3*3*3, with a padding parameter set to 1 and a convolution stride set to 2; each convolutional layer is followed by a ReLU activation function, the calculation formula of which is... Each convolutional layer is followed by a max pooling operation with a kernel size of 2*2*2 and a stride of 2. A dropout layer with a dropout rate of 0.3 is set after the third pooling layer. An adaptive max pooling layer is then set after the dropout layer to adjust the feature map to a fixed size of 1*10*10. Finally, the three-dimensional feature map is converted into a one-dimensional feature vector with a feature dimension of 6400 through a flattening operation, which serves as the input for the next stage. Step S4: Construct a multi-gated hybrid expert model: Using ResNet as the expert network, construct two parallel expert networks, each of which independently learns different feature representations of the input data; design an adaptive gating mechanism, calculate the weight coefficients of each expert network through the softmax function, and realize the dynamic fusion of the expert network outputs; construct an independent regression tower structure for each prediction task, process the fused feature information, and generate the prediction output of the corresponding task; Each expert network consists of two cascaded residual blocks and a fully connected output layer. The two residual blocks have the same structure. The specific structure of each residual block is as follows: First, the input features are mapped to a 32-dimensional hidden space through the first fully connected layer, a non-linear transformation is introduced using the ReLU activation function, and regularization is performed using a Dropout layer with a dropout rate of 0.

4. Then, the features are mapped back to the original input dimension through the second fully connected layer. Finally, the original input and the transformed features are added to achieve a residual connection, and the output of the residual block is obtained through the ReLU activation function. The expert network finally maps the features processed by the residual blocks to the target output dimension through the fully connected layer, completing the final feature transformation. The regression tower consists of two fully connected layers; the first fully connected layer maps the input features to a 10-dimensional hidden space and introduces a non-linear transformation through the ReLU activation function, while using a Dropout layer with a dropout rate of 0.4 for regularization; the second fully connected layer maps the features to a 1-dimensional output; each regression task is configured with an independent tower structure, which is used to predict the average value of tumor standard uptake and its rate of change. Step S5, Model Training: The multi-gated hybrid expert model is trained using gradient descent, with mean squared error as the loss function to measure the difference between the predicted and actual values; the model is iteratively optimized until the performance no longer improves or the predetermined number of iterations is reached, and the optimal model parameters are obtained and saved. Step S6: Predict the average value and rate of change of the tumor standard uptake value during the mid-term of radiotherapy: Input the fused image into the 3D CNN network to extract the three-dimensional spatial features; The extracted three-dimensional features are used as independent variables X, and the average value of tumor standard uptake and its rate of change are used as dependent variables Y. These are then fed into a trained multi-gated hybrid expert model to construct a dual-task regression problem. The last layer in the regression tower structure is set as a fully connected layer with an output dimension of 1. By using two independent regression tower structures, the average value of tumor standard uptake and its rate of change can be predicted respectively. Step S7, Model Performance Evaluation: The performance of the multi-gated hybrid expert model is evaluated using the leave-one-out cross-validation strategy.

2. The method for predicting tumor radiotherapy response based on three-dimensional imaging and residual networks according to claim 1, characterized in that, In step S1, the size of each slice is adjusted to be the same as the largest tumor slice in the dataset; in the normalization process, the x and y axis values ​​of each slice are adjusted to be between 0 and 1, and the operation formula is as follows: ,in, and These represent the minimum and maximum values ​​in each slice, respectively.

3. The method for predicting tumor radiotherapy response based on three-dimensional imaging and residual networks according to claim 1, characterized in that, In step S2, the image fusion adopts a multi-channel fusion method.

4. The method for predicting tumor radiotherapy response based on three-dimensional imaging and residual networks according to claim 1, characterized in that, In step S5, the loss function is calculated using the following formula: ,in, It is a sample The true value, It is the model on the sample The predicted value, It represents the total number of samples.

5. The method for predicting tumor radiotherapy response based on three-dimensional imaging and residual networks according to claim 1, characterized in that, In step S7, a leave-one-out cross-validation strategy is used to evaluate the model: each time, one patient is selected from the dataset as the test set, and the remaining patients are used as the training set, and this process is repeated until all patients have been evaluated as the test set once. In each round of training, in order to accurately select the best-performing model, two patients are specifically selected as the validation set: one patient responds to radiotherapy and has a high rate of change in their SUV value, and the other patient does not respond significantly to radiotherapy and has a low rate of change in their SUV value. The root mean square error (RMSE) is used to evaluate the model's performance. By calculating the overall average RMSE and the RMSE of individual patients, the overall predictive performance and individual predictive accuracy of the model are comprehensively evaluated.

6. A tumor radiotherapy response prediction system based on three-dimensional imaging and residual networks, characterized in that, include: The image preprocessing module is used to perform continuous slicing processing based on the Z-axis on the acquired 3D PETpre and 3D Dose images, and to standardize and normalize the slice size to obtain 2D PETpre and 2D Dose images; then, the 2D PETpre and 2D Dose images are divided into regions of interest and the tumor region data is retained; the PETpre and Dose images after region of interest processing are fused to obtain the fused image dataset as the entire tumor input dataset; The feature extraction module employs a 3D CNN network with shared hard parameters to extract features from the fused image dataset. The structure of the 3D CNN network is as follows: the first, second, and third convolutional layers all use 3*3*3 three-dimensional convolutional kernels with a padding parameter set to 1 and a stride set to 2. Each convolutional layer is followed by a ReLU activation function, the calculation formula of which is... Each convolutional layer is followed by a max pooling operation with a kernel size of 2*2*2 and a stride of 2. A dropout layer with a dropout rate of 0.3 is set after the third pooling layer. An adaptive max pooling layer is then set after the dropout layer to adjust the feature map to a fixed size of 1*10*10. Finally, the three-dimensional feature map is converted into a one-dimensional feature vector with a feature dimension of 6400 through a flattening operation, which serves as the input for the next stage. A multi-gated hybrid expert model construction module uses ResNet as the expert network, constructing two parallel expert networks, each independently learning different feature representations of the input data. An adaptive gating mechanism is designed, using a softmax function to calculate the weight coefficients of each expert network, achieving dynamic fusion of the expert network outputs. An independent regression tower structure is constructed for each prediction task to process the fused feature information and generate the corresponding prediction output. Each expert network consists of two cascaded residual blocks and a fully connected output layer. The two residual blocks have the same structure. The specific structure of each residual block is as follows: first, the input features are mapped to a 32-dimensional hidden space through the first fully connected layer; then, a nonlinear transformation is introduced through the ReLU activation function, using a dropout rate of 0.

4. The dropout layer performs regularization; then, a second fully connected layer maps the features back to the original input dimension; finally, the original input and the transformed features are added to form a residual connection, and the output of the residual block is obtained through the ReLU activation function; the expert network finally maps the features processed by the residual block to the target output dimension through a fully connected layer, completing the final transformation of the features; the regression tower consists of two fully connected layers; the first fully connected layer maps the input features to a 10-dimensional hidden space and introduces a non-linear transformation through the ReLU activation function, while using a dropout layer with a dropout rate of 0.4 for regularization; the second fully connected layer maps the features to a 1-dimensional output; each regression task is configured with an independent tower structure, used to predict the average value of tumor standard uptake and its rate of change respectively; The model training module is used to train the multi-gated hybrid expert model, obtain and save the optimal model parameters; The prediction module includes a trained multi-gated hybrid expert model and a 3D CNN network for feature extraction. It is used to input the fused image into the 3D CNN network to extract three-dimensional features, and then input the extracted features into the trained multi-gated hybrid expert model to generate prediction results. The model evaluation module is used to evaluate and test the best model saved in the model training module and calculate its performance metrics on the test set.

7. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the tumor radiotherapy response prediction method based on three-dimensional images and residual networks as described in any one of claims 1 to 5.

Images