Tumor radiotherapy reaction prediction method and model based on dual-path network

Abstract

The invention provides a tumor radiotherapy reaction prediction method and model based on a dual-path network, and the method comprises the following steps: firstly, preprocessing source data, and dividing the preprocessed source data into a training set, a verification set and a test set; then, based on the dual-path network, constructing a deep learning model; training a deep learning model based on the training set, and performing model parameter adjustment on the deep learning model based on the verification set to obtain optimized model parameters and an optimized deep learning model; based on the test set and the optimized model parameters, performing performance evaluation on the optimized deep learning model to obtain a trained dual-path network model; and finally, inputting a to-be-predicted tumor data set into the trained dual-path network model, and predicting the radiotherapy reaction of the tumor. According to the method provided by the invention, medical interpretability and relatively high prediction precision are considered, doctors can be intelligently assisted to set the radiotherapy dosage, and harm of radiotherapy to patients is reduced.

Description

Technical Field

[0001] The present application relates to the interdisciplinary field of medical deep learning, and in particular to a method and model for predicting tumor radiotherapy response based on a dual-pathway network. Background Art

[0002] In recent years, with the advancement of tumor treatment technology, tumor radiotherapy has gradually developed in the direction of precision and personalization. In order to better predict the tumor's response to radiotherapy and provide patients with the most appropriate treatment plan, the role of imaging technology in tumor diagnosis and treatment has become more and more significant. Through multimodal medical imaging, such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), etc., doctors can obtain the biological information and spatial distribution characteristics of the tumor, thereby assisting in efficacy evaluation and individualized treatment plan design. However, due to the heterogeneity and complexity of tumors, how to effectively analyze imaging data is still a key challenge in the precision of radiotherapy.

[0003] Artificial intelligence, especially deep learning, has made significant progress in various fields in recent years, and has also brought new opportunities for accurate prediction in the field of tumor radiotherapy. Deep learning models are good at automatically extracting high-level features from large amounts of complex data, making them widely used in medical image analysis. Especially in radiomics, deep learning can effectively analyze the potential high-order features in patient images, extract information related to tumor biological heterogeneity and treatment response, and thus predict treatment outcomes. At present, common deep learning models include convolutional neural networks (CNN), recurrent neural networks (RNN), variational autoencoders (VAE), etc. These models are used in medical image analysis for tasks such as tumor segmentation, image registration, and tumor progression prediction.

[0004] Although there have been some studies on the accurate prediction of tumor radiotherapy response, there is still much room for improvement in effective models and medical interpretability. Therefore, developing a tumor radiotherapy prediction method based on deep learning that can accurately predict tumor changes before and after radiotherapy and take into account interpretability has become a technical problem that needs to be overcome. Summary of the invention

[0005] The embodiment of the present application provides a tumor radiotherapy response prediction method and model based on a dual-path network, which can predict the response of the tumor before and after radiotherapy, thereby serving as a reference for physicians when formulating radiotherapy doses for patients, reducing the harm of radiotherapy to the patient's body, and improving the efficiency and quality of physicians' work.

[0006] To solve the above technical problems, in the first aspect, the present invention provides a method for predicting tumor radiotherapy response based on a dual-pathway network, the method comprising the following steps: first, preprocessing the source data, and dividing the preprocessed source data into a training set, a validation set, and a test set; then, based on the dual-pathway network, constructing a deep learning model; next, training the deep learning model based on the training set, adjusting the model parameters of the deep learning model based on the validation set, and obtaining optimized model parameters and an optimized deep learning model; then, based on the test set and the optimized model parameters, performing a performance evaluation on the optimized deep learning model to obtain a trained dual-pathway network model; finally, inputting the tumor data set to be predicted into the trained dual-pathway network model to predict the radiotherapy response of the tumor.

[0007] In some exemplary embodiments, the source data is preprocessed, including the following steps: grayscale image processing, RGB channel fusion and data batch stacking are performed on the source data in sequence; wherein, the grayscale image processing includes: obtaining the standard uptake value (SUV) of the tumor pre-radiotherapy image from the patient PET / CT image source data, recorded as prePET, the SUV value of the mid-radiotherapy image is recorded as midPET, and the radiotherapy dose is recorded as Dose, and then the three sets of original data are sliced ​​along the z-axis during the PET / CT scan and converted into a two-dimensional matrix; RGB channel fusion includes: RGB channel fusion of prePET and Dose slices, that is, prePET is used as R channel data, and Dose is used as G channel data to generate an RGB image, recorded as prePET; midPET is used as the grayscale image color channel to generate a grayscale image, recorded as midPET; data batch stacking includes: stacking the prepared prePET and midPET slice sets into several batches according to the Batchsize set by deep learning, in preparation for the subsequent input model training, and Batchsize is the number of samples contained in each batch.

[0008] In some exemplary embodiments, the preprocessed source data is divided into a training set, a validation set, and a test set, including: according to the principle of leave-one-out cross-validation, first, the test set of each round of cross-validation is assigned to a test patient to ensure that all patients and their data sets can be used as test sets in a certain round of cross-validation to achieve full coverage; second, according to the degree of their response to radiotherapy, the patients are divided into three categories: high response, medium response, and low response, and the same number of patients and their data sets are selected from each category to form a validation set of the model; finally, the remaining unselected patients and their data sets are used as training sets for the model.

[0009] In some exemplary embodiments, a deep learning model is constructed based on a dual-path network, including the following steps: constructing a network main body architecture of the dual-path network; the network main body architecture of the dual-path network includes multiple Dual Path Block modules and two classifier layers; the network main body of the dual-path network obtains an output value through forward propagation, and after the output is adaptively pooled, it is further processed using a convolution classifier and a fully connected layer to obtain a final predicted feature map, and the feature map is converted to calculate the loss value and related prediction evaluation indicators; based on the network main body architecture of the dual-path network, the training set and the test set are converted into structured image data suitable for the dual-path network input, and the structured image data is input into the network main body architecture of the dual-path network for training, and the optimal model is achieved by adjusting the model parameters, and finally a deep learning model is obtained according to the lowest loss value obtained by the input verification set; before the structured image data is input into the network main body architecture of the dual-path network for training, it also includes: converting the structured image data into a tensor, and preprocessing and format conversion of the tensor as the input of the network main body architecture of the dual-path network.

[0010] In some exemplary embodiments, after constructing the main network architecture of the dual path network, and before converting the training set and the test set into structured image data suitable for the dual path network input based on the main network architecture of the dual path network, and inputting the structured image data into the main network architecture of the dual path network for training, it also includes: optimizing the main network architecture of the dual path network to obtain a shallow dual path network model; wherein the optimization processing includes: reducing the number of Dual Path Block modules in the dual path network, constructing a shallow network structure, and reducing the number of channels in each module to reduce the dimension of the feature map; using a smaller initial convolution kernel to reduce the number of parameters; and reducing the width expansion factor bw factor to reduce the width of the network.

[0011] In some exemplary embodiments, the main network architecture of the dual-path network also includes a specially designed network output layer; the specially designed network output layer is achieved by customizing the output layer; the customization includes: adding a 1×1 convolution layer at the end of the network to perform feature mapping on each pixel point, thereby outputting a feature map with the same spatial resolution as the input image to ensure the accuracy of pixel-level prediction; subsequently, the convolution features are converted into a specified number of output channels through a fully connected layer to obtain a specific prediction value for each pixel, and an adaptive pooling layer is introduced to adapt the output size to input images of different sizes; finally, the output result is converted into a target shape through a reshape operation to meet the task requirements of pixel-level prediction. This process ensures that the output result has a spatial resolution consistent with the input image to achieve accurate pixel-level prediction.

[0012] In some exemplary embodiments, the deep learning model is trained based on the training set, and the model parameters of the deep learning model are adjusted based on the validation set to obtain optimized model parameters and the optimized deep learning model, including: obtaining the pixel-level prediction output of the model through forward propagation, and calculating the loss function, the loss function is expressed in mean square error, and its calculation formula is:

[0013]

[0014] Among them, MSE is the mean square error, m represents the number of pixels in the first dimension, n represents the number of pixels in the second dimension, and midPET i，j is the true value, is the model prediction value, and its calculation result is used for back propagation to update the model parameters;

[0015] After each training, the model performance was evaluated based on the validation set, and the goodness of fit of the model was measured by calculating the coefficient of determination. The coefficient of determination was calculated using R 2 It is expressed as follows:

[0016]

[0017] Among them, m represents the number of pixels in the first dimension, n represents the number of pixels in the second dimension, and midPET i，j is the true value of the data, To obtain the model prediction value, the model hyperparameters are adjusted by evaluating on the validation set. The hyperparameters include learning rate and regularization term. Finally, the optimized model parameters and deep learning model are obtained, thereby improving the prediction accuracy of tumor radiotherapy response.

[0018] In some exemplary embodiments, the performance of the optimized deep learning model is evaluated based on the test set and the optimized model parameters to obtain a trained dual-path network model, including: using the same evaluation indicators as those used in training and verification, the evaluation indicators include mean square error and determination coefficient, and completing a test after each epoch is completed and the model parameters are updated. The test will not affect the gradient of the model.

[0019] In some exemplary embodiments, a tumor data set to be predicted is input into a trained dual-pathway network model to predict the radiotherapy response of the tumor, including: preprocessing the tumor data set to be predicted, the preprocessing including grayscale image processing, RGB channel fusion, and data batch stacking operations, so that the dimension of the preprocessed tumor data set to be tested is the same as the dimension of the tumor data sets in the training set, the validation set, and the test set, to obtain the preprocessed tumor data set; the preprocessed tumor data set is input into the trained dual-pathway network model to predict the radiotherapy response of the tumor.

[0020] In the second aspect, an embodiment of the present application also provides a tumor radiotherapy response prediction model based on a dual-pathway network, comprising a data processing module, a model construction module, a model training module, a model evaluation module and a prediction module connected in sequence; the data processing module is used to preprocess the source data, and divide the preprocessed source data into a training set, a validation set and a test set; the model construction module is used to construct a deep learning model based on the dual-pathway network; the model training module is used to train the deep learning model based on the training set, and adjust the model parameters of the deep learning model based on the validation set to obtain optimized model parameters and an optimized deep learning model; the model evaluation module is used to perform performance evaluation on the optimized deep learning model based on the test set and the optimized model parameters to obtain a trained dual-pathway network model; the prediction module is used to input the tumor data set to be predicted into the trained dual-pathway network model to predict the radiotherapy response of the tumor.

[0021] The technology provided by the embodiments of the present application has at least the following advantages:

[0022] The embodiment of the present application provides a method and model for predicting tumor radiotherapy response based on a dual path network, the method comprising the following steps: first, preprocessing the source data, and dividing the preprocessed source data into a training set, a validation set, and a test set; then, based on the dual path network, building a deep learning model; next, training the deep learning model based on the training set, adjusting the model parameters of the deep learning model based on the validation set, and obtaining optimized model parameters and an optimized deep learning model; then, based on the test set and the optimized model parameters, evaluating the performance of the optimized deep learning model to obtain a trained dual path network model; finally, inputting the tumor data set to be predicted into the trained dual path network model to predict the radiotherapy response of the tumor. The present application proposes a method and model for predicting tumor radiotherapy response based on a dual path network. First, by constructing a network architecture based on dual path networks (DPN), introducing a shallow optimized DPN structure, and combining a customized network output layer (Pixel-wise Output Layer), the adaptability and accuracy of the network in tumor image analysis are enhanced. Secondly, this application integrates multimodal medical imaging data, combines data preprocessing and batch stacking operations, and uses the leave-one-out cross-validation principle to reasonably divide the training set, validation set, and test set, thereby effectively improving the generalization ability and robustness of the model. Finally, by using the mean square error (MSE) and the coefficient of determination R 2 The indicators evaluate the performance of the model and verify the effectiveness, accuracy and tuning ability of the proposed model. The model provided in this application can accurately predict the response of tumors after radiotherapy, thereby intelligently assisting physicians in adjusting radiotherapy doses, formulating personalized and precise radiotherapy plans for patients, reducing unnecessary radiotherapy side effects, and improving physicians' work efficiency and diagnosis and treatment quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0024] Figure 1 A schematic flow chart of a method for predicting tumor radiotherapy response based on a dual-path network provided in one embodiment of the present application.

[0025] Figure 2 A schematic diagram of a data preprocessing flow of a method for predicting tumor radiotherapy response based on a dual-path network provided in one embodiment of the present application.

[0026] Figure 3 A network architecture diagram of a dual-path network provided in one embodiment of the present application.

[0027] Figure 4 A schematic diagram of the structure of a tumor radiotherapy response prediction model based on a dual-path network provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] As can be seen from the background technology, the current existing tumor radiotherapy settings still mainly rely on the clinical experience of doctors. Radiotherapy often produces a certain amount of side effects on patients, and has strong individual differences and contingency. Physician experience alone cannot guarantee high accuracy, high reliability and low side effects. Therefore, the precise setting of tumor radiotherapy doses is of great significance for achieving personalized and precise radiotherapy.

[0029] At present, the dose setting in tumor radiotherapy mainly depends on the experience of the physician, so the individual differences and contingency are large, and high accuracy, low reliability and low side effects cannot be guaranteed continuously. Dual path network (DPN) is a dual-channel neural network in the field of deep learning. It is a parallel network of residual neural network (ResNet) and dense connection network (DenseNet). It presents a new connection path topology inside. The equivalence of ResNet and DenseNet is revealed through the framework of high-order recurrent neural network (HORNN), and it is pointed out that ResNet supports the ability to reuse features and DenseNet supports the ability to explore new features. Therefore, DPN is actually a new dual-path network structure. It is a family of convolutional neural networks containing residual similar paths and densely connected similar paths (residual networks are essentially densely connected networks with shared connections). Through the parallel connection of the two, combined with their respective advantages, the performance is optimized. Therefore, DPN, as a dual-path network, can efficiently and accurately process image features. Therefore, this application takes DPN as the main starting point and does a lot of customization work for the accurate prediction before and after tumor radiotherapy.

[0030] In view of the shortcomings of the prior art, the embodiment of the present application provides a method and model for predicting tumor radiotherapy response based on a dual-path network. The method includes the following steps: first, preprocessing the source data, and dividing the preprocessed source data into a training set, a validation set, and a test set; then, based on the dual-path network, building a deep learning model; next, training the deep learning model based on the training set, adjusting the model parameters of the deep learning model based on the validation set, and obtaining optimized model parameters and an optimized deep learning model; then, based on the test set and the optimized model parameters, evaluating the performance of the optimized deep learning model, and obtaining a trained dual-path network model; finally, inputting the tumor data set to be predicted into the trained dual-path network model to predict the radiotherapy response of the tumor. The purpose of the present application is to provide a method and model for predicting tumor radiotherapy response based on a dual-path network. The method and model can accurately predict the response of the tumor after radiotherapy at the pixel level, thereby assisting radiotherapy physicians in adjusting the radiotherapy dose, reducing the harm of radiotherapy side effects to patients, and improving the work efficiency of physicians.

[0031] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

[0032] See also Figure 1 The present application embodiment provides a method for predicting tumor radiotherapy response based on a dual-path network, the method comprising the following steps:

[0033] Step S101: preprocess the source data, and divide the preprocessed source data into a training set, a validation set, and a test set.

[0034] Step S102: construct a deep learning model based on the dual-path network.

[0035] Step S103: train the deep learning model based on the training set, and adjust the model parameters of the deep learning model based on the validation set to obtain optimized model parameters and an optimized deep learning model.

[0036] Step S104: Based on the test set and the optimized model parameters, the performance of the optimized deep learning model is evaluated to obtain a trained dual-path network model.

[0037] Step S105: input the tumor data set to be predicted into the trained dual-path network model to predict the radiotherapy response of the tumor.

[0038] DPN is a dual-channel neural network in the field of deep learning. It is a parallel network of residual neural network (ResNet) and densely connected network (DenseNet). A new connection path topology is presented inside it. The equivalence of ResNet and DenseNet is revealed through the framework of high-order recurrent neural network (HORNN), and the ability of ResNet to support feature reuse and the ability of DenseNet to support the exploration of new features are pointed out. Therefore, DPN is actually a new dual-path network structure. It is a family of convolutional neural networks containing residual similarity paths and densely connected similarity paths (residual networks are essentially densely connected networks with shared connections). Through the parallel connection of the two, combined with their respective advantages, the performance is optimized. Therefore, DPN, as a dual-path network, can process image features efficiently and accurately. Therefore, the present application proposes a tumor radiotherapy response prediction method and model based on a dual-path network. The method and model can accurately predict the response of tumors after radiotherapy at the pixel level, thereby assisting radiotherapists in adjusting radiotherapy doses, reducing the harm of radiotherapy side effects to patients, and improving the work efficiency of physicians.

[0039] Specifically, in step S101, the source data (the original tumor data set of the patient) is preprocessed. The image SUV value of the tumor in the pre-radiotherapy period is extracted from the patient's PET / CT image source data, which is recorded as prePET; the image SUV value in the middle of the radiotherapy period is extracted, which is recorded as midPET; and the radiotherapy dose data is extracted, which is marked as Dose. Subsequently, the three sets of original data are sliced ​​along the z-axis during the PET / CT scan and converted into a two-dimensional matrix. Next, the prePET and Dose slices are fused into an RGB image, where prePET is used as the R channel data and Dose is used as the G channel data. The generated RGB image is named prePET; at the same time, the midPET slice is generated into a grayscale image, named midPET. Then, the processed prePET and midPET slice sets are stacked according to the batch size to form several batches, and based on the leave-one-out cross-validation strategy, one patient is assigned as a test subject in each round of validation to ensure that all patients become test patients in a certain round. Next, the patients are divided into three categories: high response, medium response, and low response according to the strength of their response to radiotherapy, and the same proportion of patients and their data are selected from each category as the validation set. Finally, the remaining patients and their data were used to construct the training set.

[0040] The standard uptake value mentioned in the above steps is a parameter used in PET imaging to quantify the distribution of radiotracers in the body, which represents the radioactivity concentration in the region of interest (ROI) per unit body weight. SUV is normalized to facilitate standardized comparison of results under different individuals and scanning conditions, and is widely used in tumor diagnosis, efficacy monitoring and disease staging. Its calculation process takes into account ROI radioactivity concentration, injection dose and body weight, and is affected by multiple factors such as patient metabolism and scanning conditions. It is an important indicator to assist clinical decision-making.

[0041] In some embodiments, the number of patients in the training set, validation set, and test set and the size of the data set should be balanced and coordinated. Generally, when the data set is small, for leave-one-out cross-validation, the ratio of training set: validation set: test set is about 21:3:1. If the data set is large, it is more reasonable to set this ratio to 6:2:2. This is done to prevent insufficient training set data from resulting in poor model effect or too much training set data from resulting in poor generalization ability of the results.

[0042] After dividing the training set, validation set, and test set, a deep learning model is constructed based on the DPN network architecture; then the deep learning model is trained based on the training set, and the model parameters of the deep learning model are adjusted based on the results of the validation set to achieve the optimal model.

[0043] The loss function used in the model training process is the mean square error (MSE) loss function. The pixel-level prediction output of the model is obtained through the forward propagation of the deep learning model, and the MSE is calculated. The calculation formula is:

[0044]

[0045] Among them, m represents the number of pixels in the first dimension, n represents the number of pixels in the second dimension, and midPET i，j is the true value of the data, is the model prediction value, and its calculation result is used for back propagation to update the model parameters; the model parameters are adjusted according to the MSE value after the model training converges, so that the MSE loss value after convergence during model verification is as small as possible, and finally the model parameters with the smallest MSE loss value after convergence are selected and imported into the deep learning model, and then the model prediction performance is tested using the test set.

[0046] In some embodiments, in step S101, source data preprocessing, data batch stacking, and division into a training set, a validation set, and a test set are performed, including:

[0047] Step S1011, perform grayscale processing, RGB channel fusion and other operations on the source data.

[0048] Step S1012: batch stack the prepared source data.

[0049] Step S1013, divide the set into training set, validation set and test set.

[0050] In some embodiments, in step S1011, grayscale image processing, RGB channel fusion and other operations are performed on the source data, including: grayscale image processing, RGB channel fusion and other operations are performed on the source data, that is, the standardized uptake value (SUV) of the tumor pre-radiotherapy image is obtained from the patient's PET / CT image source data, recorded as prePET, the SUV value of the mid-radiotherapy image is recorded as midPET, and the radiotherapy dose is recorded as Dose, and then the three sets of original data are sliced ​​along the z-axis during the PET / CT scan and converted into a two-dimensional matrix, and then the prePET and Dose slices are fused with RGB channels, that is, prePET is used as R channel data, and Dose is used as G channel data to generate an RGB image, recorded as prePET; midPET is used as the grayscale image color channel to generate a grayscale image, recorded as midPET.

[0051] In some embodiments, in step S1012, data are batch stacked, including stacking the prepared prePET and midPET slice sets into several batches according to the Batchsize set for deep learning, to prepare for subsequent input model training, where Batchsize is the number of samples contained in each batch.

[0052] In some embodiments, in step S1013, the training set, validation set and test set are divided, including: based on the Leave-One-Out-Cross-Validation principle, first, a test patient is assigned to each round of cross-validation, so that all patients and their data sets are used as test patients in a certain round of cross-validation to complete the traversal; secondly, the patients are classified into high response, medium response and low response according to the degree of their response to radiotherapy, and the same number of patients and their data sets are selected from each type of patients as the validation set of the model; finally, all the remaining patients and their data sets in the data set are used as the training set of the model.

[0053] In some embodiments, in step S102, a deep learning model is constructed based on the DPN network architecture, including:

[0054] Step S1021, constructing the main network architecture of DPN;

[0055] Step S1022: Based on the DPN network architecture constructed in step S1021, the training set and the test set are converted into structured image data suitable for DPN input, and the image data is input into the DPN network main architecture for training, and the optimization is achieved by adjusting the model parameters, and finally a deep learning model is obtained according to the minimum loss value obtained by inputting the verification set;

[0056] The main architecture of the DPN network includes multiple DualPath Block modules and two classifier layers; the main network of the DPN obtains the output value through forward propagation, and after the output is adaptively pooled, it is further processed by the convolution classifier and the fully connected layer to obtain the final predicted feature map, and the feature map is converted to calculate the loss value and related prediction evaluation indicators; before the image data is input into the main architecture of the DPN network for training, it also includes: converting the image data into a tensor, and performing necessary preprocessing and format conversion on the tensor as the input of the main architecture of the DPN network.

[0057] Specifically, the present application constructs a pre-architecture of the DPN network, based on which the training set and the test set are converted into structured image data suitable for DPN input, and then input into the main architecture of the DPN network for training. During the training process, the model parameters are adjusted to achieve optimal performance, and finally the deep learning model is determined based on the lowest loss value obtained by the validation set input.

[0058] In some embodiments, step S105 of inputting the tumor dataset to be predicted into the trained deep learning model includes:

[0059] Step S1051: pre-process the tumor data set to be predicted to obtain a processed tumor data set.

[0060] Step S1052: input the preprocessed tumor data set into the trained deep learning model to predict the tumor radiotherapy response.

[0061] In some embodiments, in step S1051, the tumor dataset to be predicted is preprocessed, including: grayscale image processing, RGB channel fusion, data batch stacking and other operations on the tumor dataset to be predicted, so that the dimension of the preprocessed tumor dataset to be tested is the same as the dimension of the tumor dataset in the training set, the validation set and the test set.

[0062] Specifically, the preprocessing process of the tumor dataset to be predicted is consistent with the preprocessing steps of the training set, validation set, and test set, including grayscale image processing, RGB channel fusion, and data batch stacking. The dimensions of the preprocessed tumor dataset to be tested should be consistent with the dimensions of the tumor datasets in the training set, validation set, and test set. The preprocessed data is then input into the trained deep learning model to generate prediction results, thereby determining the response of the tumor after radiotherapy.

[0063] See also Figure 2 The embodiment of the present application provides a data preprocessing flow diagram of a tumor radiotherapy response prediction method based on a dual-path network, including the following steps: first, extract relevant features from the patient PET / CT image source data, including the tumor's pre-radiotherapy image SUV value (denoted as prePET), mid-radiotherapy image SUV value (denoted as midPET) and radiotherapy dose (denoted as Dose), and slice these data along the z-axis of the PET / CT scan and convert them into a two-dimensional matrix; then, perform specific fusion processing on the slices, wherein prePET is used as R channel data and Dose is used as G channel data to generate an RGB image and denoted as prePET, and midPET is used as a grayscale image color channel to generate a grayscale image and denoted as midPET; then, stack the processed prePET and midPET slice sets into several batches according to the batchsize required by deep learning (such as batchsize=4) to meet the input layer requirements of the DPN network.

[0064] Specifically, tumor data is extracted from tumor imaging data, and source data preprocessing, data batch stacking and other operations are performed to finally obtain a data set suitable for input into a deep learning model.

[0065] In some embodiments, dividing the training set, the validation set, and the test set further includes: allocating the training set, the validation set, and the test set according to a leave-one-out cross-validation principle and a data set size.

[0066] Specifically, for the division of the data set, first, a test patient is assigned to each round of cross-validation, so that all patients and their data sets are used as test patients in a certain round of cross-validation to complete the traversal; secondly, the patients are classified into high response, medium response, and low response according to the degree of their response to radiotherapy, and the same number of patients and their data sets are selected from each category of patients as the validation set of the model; finally, all the remaining patients and their data sets in the data set are used as the training set of the model.

[0067] See also Figure 3, this application example provides a network architecture diagram of a dual path network (DPN). The main network architecture of DPN includes multiple Dual Path Block modules and two classifier layers; the network body of DPN obtains the output value through forward propagation, and after the output is adaptively pooled, it is further processed by the convolution classifier and the fully connected layer to obtain the final predicted feature map, and the feature map is converted to calculate the loss value and related prediction evaluation indicators; the network architecture of DPN includes converting the training set and the test set into structured image data suitable for DPN input, and inputting the image data into the main network architecture of DPN for training, achieving the optimal by adjusting the model parameters, and finally obtaining the deep learning model according to the lowest loss value obtained by the input verification set.

[0068] After constructing the main network architecture of the dual-path network, and before converting the training set and the test set into structured image data suitable for the dual-path network input based on the main network architecture of the dual-path network, and inputting the structured image data into the main network architecture of the dual-path network for training, it also includes: optimizing the main network architecture of the dual-path network to obtain a shallow dual-path network model.

[0069] Specifically, in the DPN network architecture, a relatively shallow custom DPN network model is used to alleviate overfitting, including: reducing the number of Dual Path Block modules in the DPN network, building a relatively shallow network structure, and reducing the number of channels in each module to reduce the dimension of the feature map; using a smaller initial convolution kernel size to reduce the number of parameters; and reducing the width expansion factor (bw factor) to reduce the width of the network. This method also uses grouped convolution to reduce parameters and calculations, further reducing the complexity of the model and alleviating overfitting.

[0070] The DPN structure proposed in this application is an innovative dual-path network structure with low model complexity, reduced risk of overfitting, and an innovative network output layer that can perform pixel-level prediction tasks on tumor images.

[0071] In some embodiments, the network architecture of DPN also includes: using a specially designed network output layer, namely the Pixel-wise Output Layer, that is, a customized output layer, adding a 1×1 convolution layer at the end of the network to perform feature mapping on each pixel point, thereby outputting a feature map with the same spatial resolution as the input image to ensure the accuracy of pixel-level prediction. Subsequently, the convolutional features are converted into a specified number of output channels through a fully connected layer to obtain a specific prediction value for each pixel, and an adaptive pooling layer is introduced to adapt the output size to input images of different sizes. Finally, the output result is converted to the target shape through a reshape operation to meet the task requirements of pixel-level prediction. This process ensures that the output result has a spatial resolution consistent with the input image to achieve accurate pixel-level prediction.

[0072] See also Figure 4 The embodiment of the present application also provides a tumor radiotherapy response prediction model based on a dual-path network, comprising: a data processing module 101, a model construction module 102, a model training module 103, a model evaluation module 104, a prediction module 105 and a data processing unit in the prediction module 105, which are connected in sequence; wherein the data processing module 101 is used to preprocess the source data and divide it into a training set, a validation set and a test set; the model construction module 102 is used to build a deep learning model based on the DPN network architecture; the model training module 103 is used to train the deep learning model based on the training set, and adjust the model parameters of the deep learning model based on the validation set to obtain optimized model parameters and an optimized deep learning model; the model evaluation module 104 is used to perform performance evaluation on the optimized deep learning model based on the test set and the optimized model parameters to obtain a trained DPN model; the prediction module 105 and its internal sub-module data processing unit are used to preprocess the tumor data set to be predicted and input it into the above-mentioned trained DPN model to predict the radiotherapy response of the tumor.

[0073] Based on the above technical scheme, an embodiment of the present application provides a method and model for predicting tumor radiotherapy response based on a dual-pathway network, and the method includes the following steps: first, preprocessing the source data, and dividing the preprocessed source data into a training set, a validation set, and a test set; then, based on the dual-pathway network, building a deep learning model; next, training the deep learning model based on the training set, adjusting the model parameters of the deep learning model based on the validation set, and obtaining optimized model parameters and an optimized deep learning model; then, based on the test set and the optimized model parameters, performing a performance evaluation on the optimized deep learning model to obtain a trained dual-pathway network model; finally, inputting the tumor data set to be predicted into the trained dual-pathway network model to predict the radiotherapy response of the tumor.

[0074] The embodiment of the present application proposes a method for predicting tumor radiotherapy response based on a dual-path network. First, by constructing a network architecture based on DPN, a shallowly optimized DPN structure is introduced, combined with a customized Pixel-wise OutputLayer, to enhance the adaptability and accuracy of the network in tumor image analysis. Secondly, the present application uses the principle of leave-one-out cross-validation to reasonably divide the training set, validation set, and test set through the fusion of multimodal medical imaging data, combined with data preprocessing and batch stacking operations, so as to effectively improve the generalization ability and robustness of the model. Finally, the performance of the model is evaluated by using the MSE and determination coefficient R2 indicators to verify the effectiveness, accuracy, and tuning ability of the proposed model. The model provided in this application can accurately predict the response of tumors after radiotherapy, so as to intelligently assist physicians in adjusting radiotherapy doses, formulate personalized and precise radiotherapy plans for patients, reduce unnecessary radiotherapy side effects, and improve the work efficiency and quality of diagnosis and treatment of physicians.

[0075] Those skilled in the art can understand that the above-mentioned embodiments are specific examples for implementing the present application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application, so the scope of protection of the present application shall be based on the scope defined in the claims.

Claims

1. A method for predicting tumor radiotherapy response based on a dual-path network, characterized in that: The following steps are involved: Preprocess the source data and divide the preprocessed source data into a training set, a validation set, and a test set; Build a deep learning model based on a dual-path network; Training the deep learning model based on the training set, and adjusting model parameters of the deep learning model based on the validation set to obtain optimized model parameters and an optimized deep learning model; Based on the test set and the optimized model parameters, a performance evaluation is performed on the optimized deep learning model to obtain a trained dual-path network model; The tumor data set to be predicted is input into the trained dual-pathway network model to predict the radiotherapy response of the tumor.

2. The method for predicting tumor radiotherapy response based on a dual-path network according to claim 1, characterized in that: Preprocess the source data, including the following steps: The source data is sequentially subjected to grayscale image processing, RGB channel fusion and data batch stacking operations; wherein the grayscale image processing includes: obtaining the standard uptake value SUV of the tumor pre-radiotherapy image from the patient PET / CT image source data, that is, the pre-radiotherapy image SUV value is recorded as prePET, the mid-radiotherapy image SUV value is recorded as midPET, and the radiotherapy dose is recorded as Dose, and then the three groups of original data are sliced ​​along the z-axis during the PET / CT scan and converted into a two-dimensional matrix; The RGB channel fusion includes: performing RGB channel fusion on prePET and Dose slices, that is, using prePET as R channel data and Dose as G channel data to generate an RGB image, which is recorded as prePET; using midPET as a grayscale image color channel to generate a grayscale image, which is recorded as midPET; The data batch stacking includes: stacking the prepared prePET and midPET slice sets into a plurality of batches according to the Batchsize set for deep learning, so as to prepare for the subsequent input model training, and the Batchsize is the number of samples contained in each batch.

3. The method for predicting tumor radiotherapy response based on dual-path network according to claim 1, characterized in that: Divide the preprocessed source data into training set, validation set and test set, including: According to the principle of leave-one-out cross-validation, first, the test set of each round of cross-validation is assigned to a test patient to ensure that all patients and their data sets can be used as test sets in a round of cross-validation to achieve full coverage; second, according to the degree of their response to radiotherapy, the patients are divided into three categories: high response, medium response, and low response, and the same number of patients and their data sets are selected from each category to form the validation set of the model; finally, the remaining unselected patients and their data sets are used as the training set of the model.

4. The method for predicting tumor radiotherapy response based on dual-path network according to claim 1, characterized in that: Based on the dual-path network, a deep learning model is constructed, which includes the following steps: Construct the main network architecture of the dual-path network; the main network architecture of the dual-path network includes multiple Dual PathBlock modules and two classifier layers; the main network of the dual-path network obtains the output value through forward propagation, and after the output is adaptively pooled, it is further processed by the convolution classifier and the fully connected layer to obtain the final predicted feature map, and the feature map is converted to calculate the loss value and related prediction evaluation indicators; Based on the main network architecture of the dual-path network, the training set and the test set are converted into structured image data suitable for the dual-path network input, and the structured image data is input into the main network architecture of the dual-path network for training. The model parameters are adjusted to achieve the optimal value, and finally the deep learning model is obtained according to the lowest loss value obtained by the input verification set. Before inputting structured image data into the main network architecture of the dual-path network for training, it also includes: The structured image data is converted into a tensor, and the tensor is preprocessed and format converted as the input of the main network architecture of the dual-path network.

5. The method for predicting tumor radiotherapy response based on dual-path network according to claim 4, characterized in that: After constructing the main network architecture of the dual-path network, and before converting the training set and the test set into structured image data suitable for the dual-path network input based on the main network architecture of the dual-path network, and inputting the structured image data into the main network architecture of the dual-path network for training, the method further includes: optimizing the main network architecture of the dual-path network to obtain a shallow dual-path network model; wherein, The optimization process includes: reducing the number of Dual Path Block modules in the dual path network, building a shallow network structure, and reducing the number of channels in each module to reduce the dimension of the feature map; using a smaller initial convolution kernel to reduce the number of parameters; and reducing the width expansion factor bw_factor to reduce the width of the network.

6. The method for predicting tumor radiotherapy response based on dual-path network according to claim 4, characterized in that: The main network architecture of the dual-path network also includes a specially designed network output layer; The specially designed network output layer is achieved by customizing the output layer; the customization includes: adding a 1×1 convolution layer at the end of the network to perform feature mapping on each pixel point, thereby outputting a feature map with the same spatial resolution as the input image to ensure the accuracy of pixel-level prediction; then, converting the convolution features into a specified number of output channels through a fully connected layer to obtain a specific prediction value for each pixel, and introducing an adaptive pooling layer to adapt the output size to input images of different sizes; finally, converting the output result into a target shape through a reshape operation to meet the task requirements of pixel-level prediction. This process ensures that the output result has a spatial resolution consistent with the input image to achieve accurate pixel-level prediction.

7. The method for predicting tumor radiotherapy response based on dual-path network according to claim 1, characterized in that: The deep learning model is trained based on the training set, and model parameters of the deep learning model are adjusted based on the validation set to obtain optimized model parameters and an optimized deep learning model, including: The pixel-level prediction output of the model is obtained through forward propagation, and the loss function is calculated. The loss function is expressed in mean square error, and its calculation formula is: Among them, MSE is the mean square error, m represents the number of pixels in the first dimension, n represents the number of pixels in the second dimension, and midPET i，j is the true value, is the model prediction value, and its calculation result is used for back propagation to update the model parameters; After each training, the model performance was evaluated based on the validation set, and the goodness of fit of the model was measured by calculating the coefficient of determination. The coefficient of determination was calculated using R 2 It means that the formula is: Among them, m represents the number of pixels in the first dimension, n represents the number of pixels in the second dimension, and midPET i，j is the true value of the data, For the model prediction value, the hyperparameters of the model are adjusted by evaluation on the validation set, and the hyperparameters include learning rate and regularization term, and finally the optimized model parameters and deep learning model are obtained, thereby improving the prediction accuracy of tumor radiotherapy response.

8. The method for predicting tumor radiotherapy response based on dual-path network according to claim 1, characterized in that: Based on the test set and the optimized model parameters, the performance of the optimized deep learning model is evaluated to obtain a trained dual-path network model, including: The same evaluation indicators as those used in training and verification are used, including mean square error and coefficient of determination. A test is completed after each epoch is completed and the model parameters are updated. This test will not affect the gradient of the model.

9. The method for predicting tumor radiotherapy response based on dual-path network according to claim 1, characterized in that: Inputting the tumor data set to be predicted into the trained dual-pathway network model to predict the radiotherapy response of the tumor, including: Preprocessing the tumor data set to be predicted, wherein the preprocessing includes grayscale image processing, RGB channel fusion, and data batch stacking operations, so that the dimension of the preprocessed tumor data set to be tested is the same as the dimension of the tumor data sets in the training set, the validation set, and the test set, to obtain the preprocessed tumor data set; The preprocessed tumor data set is input into the trained dual-pathway network model to predict the radiotherapy response of the tumor.

10. A tumor radiotherapy response prediction model based on a dual-path network, characterized in that: It includes a data processing module, a model building module, a model training module, a model evaluation module and a prediction module which are connected in sequence; The data processing module is used to preprocess the source data and divide the preprocessed source data into a training set, a verification set and a test set; The model building module is used to build a deep learning model based on a dual-path network; The model training module is used to train the deep learning model according to the training set, and adjust the model parameters of the deep learning model based on the validation set to obtain optimized model parameters and an optimized deep learning model; The model evaluation module is used to perform performance evaluation on the optimized deep learning model according to the test set and the optimized model parameters to obtain a trained dual-path network model; The prediction module is used to input the tumor data set to be predicted into the trained dual-pathway network model to predict the radiotherapy response of the tumor.

Images