Pretreatment hypopharyngeal tumor prognosis prediction method, device and medium

Abstract

The invention relates to the technical field of prognosis prediction of hypopharyngeal tumors, in particular to a method and device for prognosis prediction of hypopharyngeal tumors before treatment and a medium. According to the method, the device and the medium, a first model is constructed by an MRI image and is converted into a second model through transfer learning. The first model and / or the second model can be used for analyzing an MRI image and an immunomics image of a hypopharyngeal tumor so as to predict prognosis of a patient after chemical treatment and radiotherapy are combined, and a good prediction effect can be provided.

Description

Technical Field

[0001] The present application relates to the technical field of hypopharyngeal tumor prognosis prediction, and specifically to a method, device and medium for predicting the prognosis of hypopharyngeal tumors before treatment. Background Art

[0002] The main pathological type of hypopharyngeal tumors is squamous cell carcinoma, accounting for about 95%. Although hypopharyngeal squamous cell carcinoma accounts for 3-5% of head and neck squamous cell carcinomas, its incidence is lower than that of other head and neck sites, but because the hypopharynx is located deep and hidden, the early symptoms of hypopharyngeal tumors are not specific. Therefore, it is difficult to diagnose hypopharyngeal tumors early, and patients are mostly in the late stage when they seek medical treatment, and the prognosis is worse than other head and neck squamous cell carcinomas. The disease has poor biological behavior, low degree of cell differentiation, and almost all are moderately or poorly differentiated squamous cell carcinomas. It is common to metastasize to the neck early, often infiltrating the larynx, oropharynx, cervical esophagus, and parapharyngeal space. In addition, some patients with hypopharyngeal tumors have esophageal cancer. The 5-year survival rate is usually less than 50%, which is one of the head and neck malignant tumors with the worst prognosis. It can be seen that the early diagnosis and treatment of hypopharyngeal tumors and the prognosis judgment have become a difficult problem in clinical practice. Summary of the invention

[0003] Existing technologies have successfully applied MRI image omics features to predict distant metastasis of head and neck squamous cell carcinoma, distinguish early and late head and neck squamous cell carcinoma before surgery, and judge the efficacy and prognosis. However, there are no strict standard references in lesion segmentation, feature calculation, screening, statistical analysis, and predictive modeling, which makes the research results of hypopharyngeal tumors based on MRI image omics features vary to varying degrees between different centers or even in the same center, which may affect its predictive ability.

[0004] In view of this, the present application proposes a method, device and medium for predicting the prognosis of hypopharyngeal tumors before treatment, which uses MRI images to construct a first model and converts it into a second model through transfer learning. The first model and / or the second model can analyze the MRI images and immunogenomic images of hypopharyngeal tumors, and then predict the prognosis of patients after combined chemotherapy and radiotherapy, which can provide good prediction results.

[0005] According to one aspect of the present application, a method for predicting the prognosis of hypopharyngeal tumors before treatment is disclosed. The method includes constructing a first model, inputting a multi-sequence MRI image of the hypopharyngeal tumor before treatment into the first model, and outputting a first nomogram according to the first model. The first model characterizes the relationship between the preoperative multi-sequence MRI images of hypopharyngeal cancer patients with hypopharyngeal tumors and treatment response events. The first nomogram characterizes the probability of a first adverse treatment response event corresponding to a feature matrix associated with the currently input image data.

[0006] According to one aspect of the present application, a device for predicting the prognosis of hypopharyngeal tumors before treatment is disclosed. The prediction device includes an input unit, a feature extraction unit, a model unit, a training unit, a verification unit and an output unit. The input unit is used to input an MRI image and / or an immunogenomic image of a hypopharyngeal tumor before treatment. The feature extraction unit is used to extract features from the MRI image and / or the immunogenomic image. The model unit is used to store logistic regression algorithms, k-nearest neighbor algorithms, decision tree algorithms, naive Bayes algorithms, support vector algorithms, self-help aggregation learning algorithms and transfer learning algorithms, and is also used to construct a first model and a second model. The training unit is used to train the first model and / or the second model. The verification unit is used to verify the trained first model and / or the trained second model. The output unit is used to convert and output a first nomogram based on the calculation results of the verified first model, or to convert and output a second nomogram based on the calculation results of the verified second model.

[0007] According to one aspect of the present application, a computer-readable storage medium is disclosed, on which a pre-treatment hypopharyngeal tumor prognosis prediction program is stored. When the pre-treatment hypopharyngeal tumor prognosis prediction program is executed by a processor, the steps of the pre-treatment hypopharyngeal tumor prognosis prediction method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a schematic diagram of the structure of a device for predicting the prognosis of hypopharyngeal tumors before treatment provided in an embodiment.

[0009] Figure 2 The present invention is a schematic flow chart of a method for predicting the prognosis of hypopharyngeal tumors before treatment provided in an embodiment.

[0010] Figure 3 This is a flow chart of step S10 provided in the embodiment.

[0011] Figure 4 This is a flow chart of step S20 provided in an embodiment.

[0012] Figure 5 The present invention is a schematic flow chart of a method for predicting the prognosis of hypopharyngeal tumors before treatment provided in an embodiment.

[0013] Figure 6 This is a flow chart of step S40 provided in the embodiment.

[0014] Figure 7 This is a flow chart of step S50 provided in the embodiment. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0016] Pre-treatment hypopharyngeal tumor prognosis prediction device

[0017] Figure 1 A system architecture diagram of a pre-treatment hypopharyngeal cancer prognosis prediction device 100 (hereinafter referred to as prediction device 100) of an embodiment of the present application is provided, wherein the prediction device 100 is used to analyze the clinical data and image data of the hypopharyngeal cancer patient before treatment, and then predict whether the patient will have a hypopharyngeal cancer treatment event after treatment. Hypopharyngeal cancer treatment events include various adverse manifestations during the treatment process, such as reduced survival, recurrence, metastasis, death, etc.

[0018] like Figure 1 As shown, the prediction device 100 may include: an input unit 11, a feature extraction unit 12, a model unit 13, a training unit 14, a verification unit 15 and an output unit 16.

[0019] In some embodiments, the input unit 11, feature extraction unit 12, model unit 13, training unit 14, verification unit 15 and output unit 16 are related medical devices, which can be integrated into the same medical device or divided into multiple devices that are interconnected and communicated to form a medical system for use.

[0020] In some embodiments, when the prediction device 100 obtains the patient's hypopharyngeal tumor MRI multi-sequence images and corresponding treatment response events before treatment through the input unit 11, a first data set is formed. The corresponding treatment response event is the treatment result corresponding to the MRI multi-sequence image of each clinical sample, such as tumor elimination, tumor inhibition, tumor growth, tumor metastasis, other adverse manifestations, patient survival statistics, etc. The training unit 14 obtains the first training set from the first data set, and the verification unit 15 obtains the first verification set from the first data set. The feature extraction unit 12 extracts MRI image features from the first training set. The model unit 13 establishes a logistic regression model, a k-nearest neighbor model, a decision tree model, a naive Bayes model, and a support vector model according to the MRI feature matrix. The training unit 14 obtains five initialization weight vectors according to the logistic regression model, the k-nearest neighbor model, the decision tree model, the naive Bayes model, and the support vector model. The model unit 13 performs normalization according to the five initialization weight vectors to establish a self-help set model. The training unit 14 uses the first training set to perform ensemble learning on the self-help ensemble model, calculates the loss function, and updates the weights according to the mean using the gradient descent method until the self-help ensemble model converges to obtain the trained first model. The verification unit 15 uses the first verification set to verify the trained first model to obtain the verified first model. The verification includes at least one of k-fold cross-validation, receiver operating characteristic curve (ROC), area under the curve (AUC), sensitivity, specificity, decision curve analysis (DCA), and clinical impact curve (CIC) verification. When the prediction device 100 obtains a multi-sequence MRI image of a hypopharyngeal tumor before treatment through the input unit 11, the model unit 13 processes the multi-sequence MRI image using its verified first model, and the output unit 16 converts the processing result according to the verified first model into a first nomogram and outputs it. The first nomogram represents the probability of a first adverse treatment reaction event corresponding to a feature matrix related to the image data currently input by the prediction device 100.

[0021] In some embodiments, when the prediction device 100 obtains the immunogenomic staining map of the hypopharyngeal tumor before treatment and the corresponding treatment response event through the input unit 11 and forms a second data set. Wherein, the corresponding treatment response event is the treatment result corresponding to the immunogenomic staining map of each clinical sample, such as tumor elimination, tumor inhibition, tumor growth, tumor metastasis, other adverse manifestations, patient survival statistics, etc. The training unit 14 obtains the second training set from the second data set, and the verification unit 15 obtains the second verification set from the second data set. The feature extraction unit 12 extracts immunogenomic image features from the second training set. The training unit 14 uses the immunogenomic image features in the second training set to transfer learning to the verified first model stored in the model unit 13, and converts the verified first model into a trained second model. The verification unit 15 verifies the trained second model using the second verification set to obtain a verified second model. The verification includes at least one of k-fold cross validation, receiver operating characteristic curve (ROC), area under the curve (AUC), sensitivity, specificity, decision curve analysis (DCA), and clinical impact curve (CIC) verification. When the prediction device 100 obtains an MRI multi-sequence image and / or an immunogenomic image of a hypopharyngeal tumor before treatment, the model unit 13 processes the MRI multi-sequence image and / or the immunogenomic image using its verified second model, and the output unit 16 converts the processing result according to the verified second model into a second nomogram and outputs it. The second nomogram represents the probability of a second adverse treatment reaction event corresponding to a feature matrix related to the image data currently input by the prediction device 100.

[0022] In other words, by inputting the image data of the hypopharyngeal tumor of the patient before receiving chemotherapy and radiotherapy into the prediction device 100, the prediction device 100 can predict the adverse treatment reaction events of the hypopharyngeal cancer of the patient after receiving chemotherapy and radiotherapy, wherein the adverse treatment reaction events of the hypopharyngeal cancer can be, for example, prognostic events such as the possibility of recurrence or metastasis of the hypopharyngeal tumor, but are not limited thereto. These image data include MRI images and / or immunological images. Therefore, the prediction device 100 can predict the prognosis of the patient after receiving treatment to assist the user (e.g., a physician) in determining whether the treatment method needs to be adjusted.

[0023] The details of each element will be described below.

[0024] The prediction device 100 may be an image processing device, which may be implemented by any device having a microprocessor, such as a desktop computer, a notebook computer, a smart mobile device, a server or a cloud host and the like. In an embodiment, the prediction device 100 may have a network communication function for transmitting data through a network, wherein the network communication may be a wired network or a wireless network, so the prediction device 100 may also obtain image data through the network. In an embodiment, the prediction device 100 may be implemented by executing a program instruction set in a microprocessor, such as a program instruction set having a plurality of instructions, and the program instruction set may enable the processor to perform a special operation, thereby enabling the processor to perform the functions of the input unit 11, the feature extraction unit 12, the model unit 13, the training unit 14, the verification unit 15 and the output unit 16, but is not limited thereto. For example, in another embodiment, the program instruction set may also be implemented by different computer software. In some embodiments, the program instruction set may be stored in a non-transitory computer-readable medium, such as a memory, but is not limited thereto.

[0025] In some embodiments, the input unit 11 is a physical connection port for obtaining data from the outside. For example, when the prediction device 100 is implemented by a computer, the input unit 11 can be a USB interface on the computer, various transmission line connectors, etc., but it is not limited. In addition, the input unit 11 can also be integrated with a wireless communication chip, so that data can be received by wireless transmission. The input unit 11 can be a functional module, which can be implemented by an instruction code. For example, when the microprocessor in the prediction device 100 executes the instruction code, the instruction code can enable the microprocessor to perform the function of the input unit 11.

[0026] In some embodiments, the feature extraction unit 12 may extract MRI image features from MRI image data, or extract immunological image features from immunological staining images, and input these features into the training unit 14 for machine learning. Among them, the MRI image features may be, for example, conventional features, gradient features, and depth features of the MRI image, but are not limited thereto. Among them, conventional features include tumor volume, grayscale statistical features, texture features, and high-order wavelet features. Gradient features include pixel gradient features. Depth features are generalized features extracted using convolutional neural networks. Among them, the immunological image includes an immunological staining image of GLUT-1, an immunological staining image of HIF-1α, or an immunological staining image of GLUT-1. Immunological image features include wavelength features in immunological staining images and chromaticity at special wavelengths. For example, in the immunological staining image, green is a labeled antibody located in the cytoplasm; red is labeled as tubulin. The wavelength feature is 577-492nm or 780-620nm. The chromaticity characteristics are chromaticity values ​​at wavelengths of 577 to 492 nm or chromaticity values ​​at wavelengths of 780 to 620 nm, or chromaticity regions at wavelengths of 577 to 492 nm or chromaticity regions at wavelengths of 780 to 620 nm.

[0027] In some embodiments, the model unit 13 stores common machine learning algorithms, such as logistic regression, k-nearest neighbor, decision tree, naive Bayes, support vector, and can construct logistic regression model, k-nearest neighbor model, decision tree model, naive Bayes model and support vector model respectively according to the features provided by the feature extraction unit 12. The model unit 13 also stores common convolutional neural network algorithms (such as VGG16 neural network) so that the feature extraction unit can extract generalized features from MRI images. The model unit 13 also stores a self-help aggregation learning algorithm constructed according to the logistic regression model, k-nearest neighbor model, decision tree model, naive Bayes model and support vector model, a trained first model and a verified first model. The model unit 13 also has a transfer learning algorithm so that the training unit 14 uses the immunogenomic image features to perform transfer learning on the verified first model, and stores the trained second model, as well as the second model verified by the verification unit 15.

[0028] In some embodiments, the model unit 13 further performs calculations using the input image data and transmits the calculation results to the output unit 16 .

[0029] In some embodiments, the training unit 14 obtains a first training set from the first data set. The training unit 14 also obtains five initialization weight vectors according to the logistic regression model, the k-nearest neighbor model, the decision tree model, the naive Bayes model and the support vector model. The training unit 14 uses the first training set to perform ensemble learning on the self-service set model, calculates the loss function, and updates the weights according to the mean using the gradient descent method until the self-service set model converges to obtain the trained first model.

[0030] In some embodiments, the training unit 14 obtains a second training set from the second data set. The training unit 14 also uses the immunogenomic image features in the second training set to perform transfer learning on the verified first model, and converts the verified first model into a trained second model.

[0031] In some embodiments, the verification unit 15 obtains a first verification set from the first data set. The verification unit 15 also verifies the trained first model using the first verification set to obtain a verified first model.

[0032] In some embodiments, the verification unit 15 obtains a second verification set from the second data set. The verification unit 15 also verifies the trained second model using the second verification set to obtain a verified second model.

[0033] In some embodiments, the output unit 16 converts the calculation result according to the verified first model into a first nomogram and outputs the first nomogram. The first nomogram represents the probability of the first adverse treatment reaction event corresponding to the feature matrix related to the image data currently input by the prediction device 100.

[0034] In some embodiments, the output unit 16 converts the calculation result according to the verified second model into a second nomogram and outputs the second nomogram. The second nomogram represents the second adverse treatment response event probability corresponding to the feature matrix related to the image data currently input by the prediction device 100.

[0035] Methods for predicting the prognosis of hypopharyngeal tumors before treatment

[0036] Figure 2 is a flowchart of the basic steps of the method for predicting the prognosis of hypopharyngeal tumors before treatment in some embodiments, which is used to illustrate the steps of the prediction device 100 using the patient image data to predict the treatment characteristic time, and please also refer to Figure 1 .

[0037] like Figure 2As shown, the method for predicting the prognosis of hypopharyngeal tumors before treatment includes (S10) constructing a first model, (S20) inputting MRI multi-sequence images of hypopharyngeal tumors before treatment into the first model, and (S30) outputting a first nomogram according to the first model.

[0038] After step S10 is executed, a first model is constructed and exists in the model unit 13. The first model characterizes the relationship between preoperative MRI multi-sequence images of hypopharyngeal cancer patients and treatment response events.

[0039] When step S20 is executed, the first model in the model unit 13 of the prediction device 100 obtains the MRI multi-series images of the hypopharyngeal tumor before treatment.

[0040] After step S30 is executed, the prediction device 100 calculates a first nomogram, which represents the probability of a first adverse treatment reaction event corresponding to a feature matrix related to the image data currently input by the prediction device 100.

[0041] Therefore, the prediction method can predict the prognosis of the patient after receiving treatment, so as to assist the user (eg, a physician) in determining whether the treatment method needs to be adjusted.

[0042] In some embodiments, Figure 3 As shown, step S10 also includes:

[0043] S101: Obtain the patient's hypopharyngeal tumor MRI multi-sequence images before treatment and the corresponding treatment response events to form a first data set. The corresponding treatment response events are the treatment results corresponding to the MRI multi-sequence images of each clinical sample, such as tumor elimination, tumor inhibition, tumor growth, tumor metastasis, other adverse manifestations, patient survival statistics, etc.

[0044] S102: Obtain a first training set from the first data set, and obtain a first validation set from the first data set.

[0045] S103: extracting MRI image features from the first training set and forming an MRI feature matrix, and establishing a logistic regression model, a k-nearest neighbor model, a decision tree model, a naive Bayes model and a support vector model according to the MRI feature matrix.

[0046] S104: obtaining five initialization weight vectors according to the logistic regression model, the k-nearest neighbor model, the decision tree model, the naive Bayes model and the support vector model respectively, performing naturalization according to the five initialization weight vectors, and establishing a self-help set model.

[0047] S105: Perform ensemble learning on the self-service set model using the first training set, calculate the loss function, and update the weights using the gradient descent method according to the mean until the self-service set model converges to obtain a trained first model.

[0048] S106: Utilize the first validation set to validate the trained first model to obtain a validated first model.

[0049] In step S103 provided in some embodiments, the MRI image features may be, for example, conventional features, gradient features, and depth features of the MRI image, but are not limited thereto. Conventional features include tumor volume, grayscale statistical features, texture features, and high-order wavelet features. Gradient features include pixel gradient features. Depth features are generalized features extracted using convolutional neural networks. The MRI image feature matrix may be a matrix composed of at least one of conventional features, gradient features, and depth features.

[0050] In step S103 provided in some embodiments, the grayscale statistical feature describes the distribution of voxel intensity in the tumor area based on the feature of the intensity histogram, and extracts six intensity-based features from the two-dimensional tumor area, including variance, skewness, kurtosis, mean, energy, and entropy. The texture feature is a grayscale co-occurrence matrix, grayscale run matrix, grayscale region size matrix, and local grayscale difference matrix based on the heterogeneity of the tumor area, which are used to describe the characteristics of the texture distribution in the tumor. The characteristics based on the grayscale co-occurrence matrix characterize the texture of the image by calculating the grayscale relationship between adjacent voxels in the tumor area. The high-order wavelet feature is a feature extracted after performing a three-dimensional wavelet transform on the WI-weighted image, the WI-weighted image, and the proton density-weighted image. The wavelet transform includes wavelet deconstruction and reconstruction. The high-order wavelet feature includes the high-frequency part and the low-frequency part of each area of ​​the original image in the X direction, the Y direction, and the Z direction. For the eight wavelet transformed images obtained from the original image, the same feature extraction operation as in the previous section is performed to obtain the texture feature based on the wavelet transform.

[0051] In step S103 provided in some embodiments, since conventional features ignore the changes in substructures with slight differences in the image, these substructures may be different in morphology but have the same grayscale intensity. Therefore, conventional texture features may not be able to capture the local changes in the direction of each pixel in the local neighborhood. In order to fully explore the local difference information in the image, the present application also extracts the pixel gradients in the T1WI weighted image, T2WI weighted image and proton density weighted image of the MRI image to obtain pixel extraction features. Specifically, it includes obtaining the pixel gradients of the T1WI weighted image, the pixel gradients of the T2WI weighted image and the pixel gradients of the proton density weighted image, calculating the local main gradient directions of these pixels, obtaining a discretized gradient image, and quantifying the gradient features in the image.

[0052] In step S103 provided in some embodiments, a VGG16 neural network is used to perform deep learning on T1WI weighted images, T2WI weighted images, and proton density weighted images, the learning target is radiosensitivity, and the sixteenth layer (fully connected layer, 1×1000) is used as the output layer, and the output layer is a matrix composed of generalized features of radiosensitivity. A non-parametric test (Mann-Whitney Utest) is used to screen out features with significant differences between groups, and a P value less than 0.05 is set to indicate significance.

[0053] In step S106 provided in some embodiments, the validation includes at least one of k-fold cross validation, receiver operating characteristic curve (ROC), area under the curve (AUC), sensitivity, specificity and decision curve analysis (DCA), and clinical impact curve (CIC) validation.

[0054] In some embodiments, Figure 4 As shown, step S20 also includes: S201: extracting MRI image features from the MRI multi-series images; and S202: inputting the MRI image features into the verified first model.

[0055] like Figure 5 As shown, the method for predicting the prognosis of hypopharyngeal tumors before treatment also includes (S40) constructing a second model, (S50) inputting MRI multi-sequence images and / or immunogenomic images of hypopharyngeal tumors before treatment into the second model, and (S60) outputting a second nomogram based on the second model.

[0056] After step S40 is executed, a second model is constructed and exists in the model unit 13. The second model characterizes the relationship between preoperative MRI multi-sequence images and / or immunomics images of hypopharyngeal cancer patients and treatment response events.

[0057] After step S50 is executed, the second model in the model unit 13 of the prediction device 100 obtains the MRI multi-sequence images and / or immunogenomic images of the hypopharyngeal tumor before treatment.

[0058] After step S60 is executed, the prediction device 100 calculates a second nomogram, which represents the second adverse treatment reaction event probability corresponding to the feature matrix related to the MRI multi-sequence image and / or immunogenomics image currently input by the prediction device 100.

[0059] Therefore, the prediction method can predict the prognosis of the patient after receiving treatment, so as to assist the user (eg, a physician) in determining whether the treatment method needs to be adjusted.

[0060] In some embodiments, Figure 6 As shown, step S40 also includes:

[0061] S401: Obtain the immunogenomic staining map of the hypopharyngeal tumor before treatment and the corresponding treatment response events to form a second data set. The corresponding treatment response events are the treatment results corresponding to the immunogenomic staining map of each clinical sample, such as tumor elimination, tumor inhibition, tumor growth, tumor metastasis, other adverse manifestations, patient survival statistics, etc.

[0062] S402: Obtain a second training set from the second data set, and obtain a second validation set from the second data set.

[0063] S403: extracting immunogenomic image features from the second training set, performing transfer learning on the verified first model using the immunogenomic image features in the second training set, and converting the verified first model into the trained second model.

[0064] S404: Utilize a second validation set to validate the trained second model to obtain a validated second model.

[0065] In step S401 provided in some embodiments, the immunohistochemical staining image includes an immunohistochemical staining image of GLUT-1 and / or an immunohistochemical staining image of HIF-1α.

[0066] In step S403 provided in some embodiments, the immunogenomic image features include wavelength features in the immunogenomic staining image and chromaticity at a specific wavelength. For example, in the immunogenomic staining image, green is a labeled antibody located in the cytoplasm; red is a marker for tubulin. The wavelength feature is 577-492nm or 780-620nm. The chromaticity feature is a chromaticity value at a wavelength of 577-492nm or a chromaticity value at 780-620nm, or a chromaticity region of 577-492nm or a chromaticity region of 780-620nm.

[0067] In step S404 provided in some embodiments, the validation includes at least one of k-fold cross validation, receiver operating characteristic curve (ROC), area under the curve (AUC), sensitivity, specificity and decision curve analysis (DCA), and clinical impact curve (CIC) validation.

[0068] In some embodiments, Figure 7 As shown, step S50 also includes: S501: extracting MRI image features from the MRI multi-series images and / or extracting immunomics image features from the immunomics images; and S502: inputting the MRI image features and / or immunomics image features into the verified second model.

[0069] In some embodiments, the training unit 12 may use transfer learning technology to adjust some parameters of the first model, thereby converting the first model into a second model. Regarding the second model, the architecture of the second model includes an external perception convolution layer, an internal perception convolution layer, a global average pooling layer, and a loss function layer. Since the second model is formed by adjusting the first model, its feature path, the internal parameters of the second external perception convolution layer, the internal parameters of the second internal perception convolution layer, the internal parameters of the global average pooling layer, and the internal parameters of the loss function layer may be different from those of the first model. In an embodiment, the feature path of the second model may be different from the feature path of the first model, but this is not a limitation.

[0070] In this step, the feature path of the first model, the items of image features to be analyzed, and the internal parameters of each component are adjusted in combination with the added immunogenomic image features to complete the training and obtain a trained second model.

[0071] In some embodiments, the second model uses the characteristic path of the first model, the items used as the analyzed image features, and the internal parameters of each component for adjustment, but the first nomogram is adjusted. The first nomogram is adjusted by inputting the second model to obtain the probability of occurrence of the segmented image predicting the adverse treatment response event, and adjusting the first nomogram by obtaining the best prediction accuracy.

[0072] A flowchart of the process of establishing the second model of the present application, the training unit 12 trains the global average pooling layer and the loss function layer through a large amount of image data in the second training set (for example, including MRI multi-sequence images and immunomics images), and adjusts the first nomogram to form a second nomogram through a large amount of image data of hypopharyngeal tumors used for training (for example, including MRI multi-sequence images and immunomics images). Among them, a new feature path is established by training the global average pooling layer. In addition, the internal parameters of the loss function layer are adjusted by training to form a second nomogram. The parameters that can be adjusted in the global average pooling layer are parameters that integrate image features into segmented images to predict the probability of adverse treatment reaction events. The parameters that can be adjusted in the loss function layer are the calculation formulas represented by them.

[0073] In some embodiments, the external perception convolution layer and the internal perception convolution layer may each perform a normalization operation. The normalization operation here may be, for example, but not limited to, batch normalization. The normalization operation may normalize the data of the convolution operation results of each multi-layer perception convolution layer, thereby speeding up the convergence of subsequent data processing and making the training process more stable. In addition, in some embodiments, each multi-layer perception convolution layer may each include a pooling layer to perform a pooling operation. The pooling operation here may be, for example, maximum pooling. The role of the pooling layer is to reduce the size of the feature map obtained by the multi-layer perception convolution layer and to concentrate the features in the reduced feature map. In a broad sense, the role of the pooling layer can be regarded as extracting important features from the feature map, so that important features can be emphasized. In some embodiments, the maximum pooling layer can also be changed to an average pooling layer architecture.

[0074] In some embodiments, the external perception convolution layer, the internal perception convolution layer 1 and the global average pooling layer may each include an activation function. The activation function can be used to adjust the output of the external perception convolution layer, the internal perception convolution layer or the global average pooling layer, so that the output result produces a nonlinear effect, thereby improving the predictive ability of the training model. The activation function can be a saturated activation function or an unsaturated activation function. When the activation function is a saturated activation function, the activation function can adopt a tanh, sigmoid and other architectures, and when the activation function is an unsaturated activation function, the activation function 24 can adopt a linear rectification function (ReLU) or its variation architecture (such as ELU, LeakyReLU, PReLU, RReLU or other variation architectures). In a preferred embodiment, the activation function 24 of the multi-layer perception convolution layer 152 adopts the ReLU architecture, and the activation function 24 of the global average pooling layer 154 adopts an architecture other than ReLU.

[0075] In some embodiments, the loss function layer 156 may be: 1-((2*sensitivity*PositivePredictiveValue) / (sensitivity+PositivePredictiveValue)), where sensitivity is the proportion of all images where adverse treatment reaction events occur that are accurately predicted to occur as adverse treatment reaction events, and PositivePredictiveValue is the proportion of segmented images where adverse treatment reaction events are actually occurring among the segmented images predicted to occur as adverse treatment reaction events.

[0076] In some embodiments, the loss function layer 156 of the second model is adjusted to: sqrt(power(1-specificity,n)+power(1-sensitivity,n)), where specificity is the proportion of all segmented images predicted as having no adverse treatment response events that are accurately predicted as not occurring, sqrt is an operation function for extracting square roots, and power is a power operation function. In this embodiment, n is set to 2.

[0077] Therefore, the hypopharyngeal tumor prediction device and method of the present application can expand a small amount of image data through the input unit, without having to input a large amount of image data at the beginning. In addition, the second model of the present application can directly use the first model that has been trained, or adjust some parameters of the cancer prognosis prediction model to obtain it, without having to spend a lot of time to establish a new prediction model.

[0078] Computer readable storage medium

[0079] In addition, an embodiment of the present application also proposes a computer-readable storage medium, on which a pre-treatment hypopharyngeal tumor prognosis prediction program is stored. When the pre-treatment hypopharyngeal tumor prognosis prediction program is executed by a processor, the steps of the pre-treatment hypopharyngeal tumor prognosis prediction method as described above are implemented.

[0080] The expanded contents of the specific implementation methods of the pre-treatment hypopharyngeal tumor prognosis prediction device and computer-readable storage medium of the present application are basically the same as the embodiments of the pre-treatment hypopharyngeal tumor prognosis prediction method described above, and will not be elaborated here.

[0081] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0082] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0084] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

Claims

1. A method for predicting the prognosis of hypopharyngeal tumors before treatment, the method comprising: Build the first model; The first model characterizes the relationship between preoperative MRI multi-sequence images of hypopharyngeal cancer patients and treatment response events; Inputting multiple MRI sequence images of hypopharyngeal tumors before treatment into the first model; and A first nomogram is output according to the first model, wherein the first nomogram represents a first adverse treatment reaction event probability corresponding to a feature matrix associated with currently input image data.

2. According to the method of claim 1, the step of constructing the first model comprises: Obtaining multiple MRI sequence images of hypopharyngeal tumors of patients before treatment and corresponding treatment response events to form a first data set; wherein the corresponding treatment response events are treatment results corresponding to the multiple MRI sequence images of each clinical sample; Acquire a first training set from the first data set, and acquire a first validation set from the first data set; Extracting MRI image features from the first training set and forming an MRI feature matrix, and establishing a logistic regression model, a k-nearest neighbor model, a decision tree model, a naive Bayes model, and a support vector model according to the MRI feature matrix; Five initialization weight vectors are obtained according to a logistic regression model, a k-nearest neighbor model, a decision tree model, a naive Bayes model, and a support vector model, respectively, and a bootstrap ensemble model is established by performing naturalization according to the five initialization weight vectors; Using the first training set to perform ensemble learning on the self-service set model, calculating the loss function, and updating the weights according to the mean using the gradient descent method until the self-service set model converges to obtain a trained first model; The trained first model is verified using a first verification set to obtain a verified first model.

3. According to the method of claim 2, the MRI image features include conventional features, gradient features and depth features; Optionally, conventional features include tumor volume, grayscale statistical features, texture features, and high-order wavelet features; Optionally, the gradient feature comprises a pixel gradient feature; Optionally, the deep features are generalized features extracted using a convolutional neural network; Optionally, the MRI image feature matrix may be a matrix composed of at least one of conventional features, gradient features and depth features. 4 . The method according to claim 3 , wherein the pixel gradient features include pixel gradients obtained from a T1WI weighted image, a T2WI weighted image, and a proton density weighted image.

5. According to the method of claim 3, a VGG16 neural network is used to perform deep learning on T1WI weighted images, T2WI weighted images and proton density weighted images, the learning target is radiation sensitivity, and the fifteenth layer is used as the output layer to output the generalized features.

6. The method according to claim 1, comprising: constructing a second model, wherein the second model characterizes the relationship between preoperative MRI multi-sequence images and / or immunomics images of hypopharyngeal cancer patients and treatment response events; Inputting the MRI multi-sequence images and / or immunomics images of the hypopharyngeal tumor before treatment into the second model; and A second nomogram is output according to the second model; the second nomogram represents the probability of a second adverse treatment reaction event corresponding to a feature matrix associated with the currently input MRI multi-sequence image and / or immunomics image.

7. The method according to claim 6, wherein the step of constructing the second model comprises: Obtaining the immunogenomic staining map of hypopharyngeal tumors and corresponding treatment response events of the patients before treatment and forming a second data set; wherein the corresponding treatment response events are the treatment results corresponding to the immunogenomic staining map of each clinical sample; Acquire a second training set from the second data set, and acquire a second validation set from the second data set; Extracting immunogenomic image features from the second training set, performing transfer learning on the verified first model using the immunogenomic image features in the second training set, and converting the verified first model into the trained second model; The trained second model is verified using a second verification set to obtain a verified second model.

8. The method according to claim 7, wherein the immunogenomic staining pattern comprises an immunogenomic staining pattern of GLUT-1, an immunogenomic staining pattern of HIF-1α, or an immunogenomic staining pattern of GLUT-1; Optionally, the immunohistochemistry image features include wavelength features in the immunohistochemistry staining image and chromaticity at a specific wavelength.

9. A device for predicting the prognosis of hypopharyngeal tumors before treatment, comprising: An input unit, used for inputting a hypopharyngeal tumor MRI image and / or an immunogenomics image before treatment; A feature extraction unit, used for extracting features from the MRI image and / or the immunogenomics image; A model unit, used to store logistic regression algorithm, k-nearest neighbor algorithm, decision tree algorithm, naive Bayes algorithm, support vector algorithm, self-help aggregation learning algorithm and transfer learning algorithm, and also used to construct the first model and the second model; A training unit, used for training the first model and / or the second model; A verification unit, used to verify the trained first model and / or the trained second model; as well as An output unit is used to convert the calculation result of the verified first model into a first collinear diagram and output it, or to convert the calculation result of the verified second model into a second collinear diagram and output it.

10. A computer-readable storage medium having a pre-treatment hypopharyngeal tumor prognosis prediction program stored thereon, wherein the pre-treatment hypopharyngeal tumor prognosis prediction program, when executed by a processor, implements the steps of the pre-treatment hypopharyngeal tumor prognosis prediction method as described in any one of claims 1 to 8.

Images