Survival prediction method based on tumor pathological image and application thereof

Abstract

The invention discloses a survival prediction method based on a tumor pathological image, and the method comprises the steps: obtaining a key region of the tumor pathological image through deep learning and an attention mechanism, and obtaining a cell nucleus boundary feature, a cell nucleus texture feature and / or a cell nucleus image structure feature based on the key region of the tumor pathological image; and survival prediction is carried out through the trained machine learning model. According to the survival prediction method, the accuracy of survival prediction of tumor patients is improved, and meanwhile, the defect that a traditional deep learning method is poor in interpretability is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of pathological image analysis, and in particular to a survival prediction method based on tumor pathological images and its application. Background Art

[0002] In the modern medical field, pathological image analysis is one of the important means of cancer diagnosis and prognosis assessment. With the development of digital pathology technology, pathological images can be digitized with high resolution, which provides the possibility for the application of computer-aided diagnosis (CAD) and deep learning technology. In particular, the application of image processing and machine learning methods in automated pathological image analysis has made significant progress.

[0003] Currently, the main technologies used in pathology image analysis include traditional feature engineering methods and deep learning-based approaches. Traditional feature engineering methods often rely on handcrafted features (such as image texture, cell morphology, and cellular spatial structure) to represent multidimensional attributes with clear physical meaning. These methods often require complex image characterization processes, and due to the complex properties of tumor pathology images (such as the diversity of tumor tissue microecology), these handcrafted models often fail to achieve optimal results.

[0004] In recent years, deep learning methods have also been widely used in the analysis of pathological images. Deep learning methods can automatically extract and analyze features and make corresponding predictions.

[0005] Traditional deep learning methods represented by convolutional neural networks have shown good application scenarios in the fields of tumor tissue detection, risk classification, and prognosis prediction. At the same time, some of the latest deep learning methods such as Transformer and graph neural networks can also achieve good pathological image analysis results because they can better utilize the contextual information in the image for comprehensive prediction.

[0006] However, due to the black box nature of deep learning that is difficult to solve, and the process of using deep learning methods to process pathological images and obtain decision information is unknown, while medical testing often emphasizes the clear interpretability of causal relationships, the black box nature of deep learning undoubtedly limits its application in the field of medical image analysis.

[0007] Therefore, there is a need for a pathology image analysis and processing method that can not only accurately analyze pathology images, but also link the analysis results with pathology images to enhance interpretability. Summary of the Invention

[0008] The purpose of the present invention is to overcome the above-mentioned deficiencies in the prior art and to provide a survival prediction method based on tumor pathology images and its application.

[0009] The first object of the present invention is to provide a survival prediction method based on tumor pathology images.

[0010] The second object of the present invention is to provide a survival prediction model based on tumor pathology images.

[0011] The third object of the present invention is to provide an application of the above-mentioned survival prediction model in the preparation of a product for predicting the survival of tumor patients.

[0012] In order to achieve the above object, the present invention is implemented through the following scheme: A survival prediction method based on tumor pathology images comprises the following steps: S1. Pathology image data processing: S11. Constructing a tissue classification model; the tissue classification model is capable of identifying different tissues in pathological images; S12. Converting the tumor pathology image of the sample to be tested into a grayscale image, and obtaining the foreground area of ​​the grayscale image as the tissue area; S13. Dividing the tissue region into non-overlapping image blocks to be tested, and recording the coordinates of each image block to be tested in the tumor pathology image of the sample to be tested; S14. Input the image block to be tested into the tissue classification model to obtain the tissue classification result and feature vector of the image block to be tested; record the image block to be tested that is identified as a tumor by the tissue classification result and its eight neighboring image blocks as qualified image blocks, and record the number of qualified image blocks; Based on the feature vector of the image block to be tested and the number of qualified image blocks, an M×N feature matrix X is generated for the tumor pathology image of the sample to be tested, where N is the dimension of the feature vector and M is the number of qualified image blocks; S15. Based on the coordinates of the qualified image blocks in the tumor pathology image, use the k-nearest neighbor algorithm to construct an adjacency matrix A of the tumor pathology image of the test sample; wherein the feature matrix X and the adjacency matrix A are the graph structure data of the test sample; S2. Input the graph structure data of the sample to be tested into the trained graph convolutional neural network model to obtain the risk score of the sample to be tested.

[0013] Preferably, in step S11, a tissue classification model is constructed based on the Swin Transformer V2 model.

[0014] Preferably, the tumor pathology image in step S12 is a whole-slice image of tumor tissue.

[0015] Preferably, the tumor pathology image in step S12 is a tumor pathology image obtained by 64-fold sampling.

[0016] Preferably, in step S12, the Otsu threshold algorithm (OTSU) is used to obtain the foreground area in the grayscale image.

[0017] Preferably, the physical size of the image block to be measured in step S13 is suitable for the tissue classification model in step S11.

[0018] Preferably, in step S14, the image block to be tested obtained in step S13 is first subjected to a staining normalization process and then input into the tissue classification model obtained in step S11.

[0019] More preferably, Macehko staining normalization is performed.

[0020] Preferably, in step S15 , a k-nearest neighbor algorithm (k=4) is used to construct an adjacency matrix A of the tumor pathology image of the sample to be tested.

[0021] Preferably, the training process of the trained graph convolutional neural network model in step S2 is specifically as follows: Each sample in the training set 1 is processed according to step S1 to obtain the feature matrix X and adjacency matrix A of each sample in the training set 1; the samples in the training set 1 are all samples with known survival results; The graph convolutional neural network model is trained using the feature matrix X and adjacency matrix A of each sample in training set 1 as input and the risk score of each sample in training set 1 as the predicted value to obtain a trained graph convolutional neural network model. A larger risk score indicates a worse survival outcome for the sample. Among them, the input dimension and output dimension of each convolutional layer in the graph convolutional neural network model are both O. The feature matrix X of the sample is converted to the hidden dimension O through the fully connected layer of the graph neural network model, and the ReLU activation function and Dropout function are used to obtain the initialization feature H0 of the sample; Input the sample's initialization feature H0 and the sample's adjacency matrix A into the first graph convolution layer, and obtain the output H1 of the first graph convolution layer according to Formula I. Then, obtain the output H of the Lth graph convolution layer according to Formula II. L , L=(2,3,...L);the obtained H1、....、H L According to formula III, we can obtain feature H; Formula I: ; Formula II: ; Formula III: ; The feature H is input into a fully connected layer of the graph convolutional neural network model, and nonlinear changes are performed using the ReLU activation function and the Dropout function to obtain the feature H Ф ; H Ф According to formula IV, the attention weight matrix M is obtained, and the attention weight matrix M is used to calculate H according to formula V. Ф Perform weighted aggregation and process it through a fully connected layer and ReLU activation function to obtain the feature H ρ ; Formula IV: ; Formula V: ; Discretize the survival time of each sample into t (t = 4) non-overlapping time intervals, and combine the H ρ Calculate the conditional probability of death (hazards) within t time periods, and then calculate t survival probabilities S according to formula VII t ; Formula VI: ; Formula VII: ; The survival probability S t The overall risk of an individual is calculated according to Formula VIII; Formula VIII: .

[0022] More preferably, the hyperparameters of the graph convolutional neural network model are set as follows: using a Cox proportional loss function based on cross entropy, using an Adam optimizer, a learning rate of 8e-6, and a weight decay of 1e-5.

[0023] More preferably, the survival outcome includes survival status, time of death and last follow-up time.

[0024] Preferably, before obtaining the risk score of the sample to be tested in step S2, the following steps are further included: S3. Input the graph structure data of the test sample into the trained graph convolutional neural network model to obtain the attention weight matrix of the test sample; based on the attention weight matrix of the test sample, obtain the attention score of each qualified image block in the test sample, sort the attention scores in descending order, and select the top 10% to 100% of the image blocks as the analysis image blocks; S4. Tissue morphology feature extraction: S41. Use the trained cell nucleus segmentation and classification model to perform cell nucleus segmentation and classification on the analysis image block to obtain the outline and category of the cell nucleus in the analysis image block; S42. Based on the analysis of the outline and category of the cell nucleus in the image block, obtain the cell nucleus boundary features, cell nucleus texture features, cell nucleus map structure features and / or cell nucleus direction features of the sample to be tested; S5. Inputting the nuclear boundary features, nuclear texture features, and / or nuclear map structural features of the sample to be tested into the trained machine learning model to obtain a risk score for the sample to be tested; The trained machine learning model is a machine learning model trained using a training set, and the training set is tumor pathology images with known survival results.

[0025] Preferably, in step S3, image blocks in the top 70% of the attention score area are selected as analysis image blocks.

[0026] Preferably, the cell nucleus segmentation and classification model in step S41 is a StarDist model.

[0027] Preferably, the cell nucleus boundary features in step S42 include cell nucleus curvature features, cell nucleus size and cell nucleus shape features; the cell nucleus texture features include grayscale co-occurrence matrix features, image moment features, grayscale value statistical features, intensity contrast features and chromatin features; the cell nucleus map structure features include distribution features and proportion features; the cell nucleus direction features are the COrE features of the cell nucleus.

[0028] More preferably, the cell nucleus curvature characteristics include: the mean value and standard deviation of the curvature, the number of times the curvature changes its sign along the boundary, the maximum value, mean value, median, standard deviation, sum and number of positive and negative curvatures, the number, average peak value and average width of the main peaks of positive and negative curvatures, the ratio of the positive and negative curvature parts to the total perimeter of the boundary, and the ratio of the part where the curvature changes its sign to the total perimeter of the boundary; the mean, median, standard deviation, maximum value, minimum value, difference between the maximum value and the minimum value, kurtosis and skewness of the cell nucleus curvature characteristics are counted respectively.

[0029] The cell nuclear size and cell nuclear shape characteristics include: the minimum, maximum, median, mean, mode, 25th percentile, 75th percentile, standard deviation of the distance from the center of mass to the cell boundary, the maximum Feret diameter, the minimum and maximum caliper distances measured in all rotations, the ratio of the minimum caliper distance to the maximum caliper distance, the first component of the center of mass, the second component of the center of mass, the area, the perimeter, the area of ​​the bounding box, the area of ​​the convex hull, the equivalent diameter, the major axis length and the minor axis length of the best fitting ellipse, the eccentricity, the direction, the compactness, the ratio of the major axis to the minor axis of the best fitting ellipse, the roundness, the ratio of the area to the bounding box area, the Fourier descriptor and the Zernike moment; the mean, median, standard deviation, maximum value, minimum value, the difference between the maximum value and the minimum value, the kurtosis and the skewness of the cell nuclear size and cell nuclear shape characteristics are respectively counted.

[0030] More preferably, the gray-level co-occurrence matrix features include: contrast, dissimilarity, homogeneity, angular second moment, energy and correlation of the gray-level co-occurrence matrix when the pixel distance is 1, 5, and 20; and the mean, median, standard deviation, maximum value, minimum value, difference between the maximum and minimum values, kurtosis and skewness of the cell nucleus gray-level co-occurrence matrix are respectively counted.

[0031] More preferably, the image moment features include weighted centroid, weighted moment, normalized weighted moment, central weighted moment, Hu invariant weighted moment, moment, normalized moment, central moment and Hu invariant moment; and the mean, median, standard deviation, maximum value, minimum value, difference between maximum value and minimum value, kurtosis and skewness of the cell nucleus image moment features are counted respectively.

[0032] The grayscale value statistical features include: the minimum value, maximum value, median, mean value, mode, 25th percentile, 75th percentile, standard deviation, kurtosis, skewness and entropy of the cell nucleus grayscale value; the mean value, median value, standard deviation, maximum value, minimum value, difference between the maximum value and the minimum value, kurtosis and skewness of the cell nucleus grayscale value statistical features are counted respectively.

[0033] The intensity contrast features include: the ratio of the grayscale intensity of the 80th percentile to the grayscale intensity of the 20th percentile, and the ratio of the number of pixels with grayscale intensity ≥ the 80th percentile to the number of pixels with grayscale intensity ≤ the 20th percentile; the mean, median, standard deviation, maximum value, minimum value, difference between the maximum and minimum values, kurtosis and skewness of the cell nucleus intensity contrast features are respectively counted.

[0034] The chromatin features include: the ratio of the area occupied by heterochromatin to that of euchromatin, the ratio of the area occupied by heterochromatin to the total area of ​​the cell nucleus, the ratio of the total intensity of heterochromatin to the total intensity of euchromatin, and the ratio of the total intensity of heterochromatin to the total intensity within the region; the average value of the image intensity within the cell nucleus region plus the standard deviation of the image intensity is used as the threshold, the heterochromatin is the pixel with intensity ≥ the threshold, and the euchromatin is the pixel with intensity < the threshold; the mean, median, standard deviation, maximum value, minimum value, difference between the maximum and minimum values, kurtosis and skewness of the cell nucleus chromatin features are respectively counted.

[0035] More preferably, the Delaunay triangle is constructed with epithelial cells, inflammatory cells and connective tissue cells respectively, with the cell nucleus centroid as the node; the inflammatory cells include plasma cells, lymphocytes, neutrophils and eosinophils; The edge types are defined based on the cell types of the two nodes of each edge in Delaunay's triangle, including epithelial-inflammation, epithelial-connective tissue, and inflammation-connective tissue. The triangle types are defined based on the three nodes of Delaunay's triangle, including epithelial-epithelial-epithelial, inflammation-inflammation-inflammation, connective tissue-connective tissue-connective tissue, epithelial-epithelial-inflammation, epithelial-epithelial-connective tissue, inflammation-inflammation-connective tissue, inflammation-inflammation-epithelial, connective tissue-connective tissue-epithelial, connective tissue-connective tissue-inflammation, and epithelial-inflammation-connective tissue. Distribution characteristics are obtained based on Delaunay triangulation, including: node degree, triangle side length, triangle area, degree of various cell types and other types and their ratios, node degree of various cell types, side length of various edge types, and triangle area of ​​various triangle types; the mean, median, standard deviation, maximum value, minimum value, difference between maximum and minimum value, kurtosis and skewness of the distribution characteristics are statistically analyzed respectively.

[0036] More preferably, the proportion features include: the proportion of various cell types in all cells, the proportion of various types of edges in all edges, the proportion of the total length of various types of edges in the total length of all edges, the proportion of various types of triangles in all triangles, and the proportion of the total area of ​​various types of triangles in the total area of ​​all triangles.

[0037] Preferably, in step S42, based on the outline and category of the cell nucleus in the analysis image block obtained in step S41, the kurtosis of the maximum caliper distance of the epithelial cells, the skewness of the maximum caliper distance of the epithelial cells, the kurtosis of the maximum value of the distance from the center of mass of the epithelial cells to the cell boundary, the skewness of the maximum value of the distance from the center of mass of the epithelial cells to the cell boundary, the kurtosis of the 75th percentile of the maximum value of the distance from the center of mass of the epithelial cells to the cell boundary, the kurtosis of the maximum Feret diameter of the epithelial cells, the skewness of the maximum Feret diameter of the epithelial cells, the kurtosis of the second component of the center of mass of the epithelial cells, the kurtosis of the long axis length of the epithelial cells, and the Zernike moment of the epithelial cells are obtained. 19skewness, kurtosis of the real part 1 of the epithelial cell Fourier descriptor, standard deviation of the 25th percentile of the epithelial cell grayscale, median of the epithelial cell grayscale median, standard deviation of the epithelial cell grayscale median, kurtosis of the epithelial cell grayscale median, kurtosis of the 75th percentile of the epithelial cell grayscale, kurtosis of the epithelial cell grayscale maximum, skewness of the epithelial cell grayscale maximum, kurtosis of the epithelial cell grayscale mean, mean of the epithelial cell grayscale mode, median of the epithelial cell grayscale mode, standard deviation of the epithelial cell grayscale mode, median of the epithelial cell grayscale distribution kurtosis, standard deviation of the epithelial cell grayscale distribution kurtosis, mean of the epithelial cell grayscale distribution skewness, total intensity of epithelial cell heterochromatin and euchromatin The kurtosis of the ratio of the total intensity of the epithelial cells is 1, the kurtosis of the gray-level co-occurrence matrix contrast when the epithelial cell pixel distance is 5, the standard deviation of the gray-level co-occurrence matrix contrast when the epithelial cell pixel distance is 20, the kurtosis of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 1, the skewness of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 1, the kurtosis of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 5, the standard deviation of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 20, the kurtosis of the first component of the epithelial cell weighted centroid, and the epithelial cell weighted moment. 0,0 The mean value of epithelial cell weighted moment 0,0 The standard deviation of epithelial cell weighted moment 1,2 The median and epithelial cell weighted moments 2,0 The mean value of epithelial cell weighted moment 2,2 The median and epithelial cell weighted moments 2,2 Kurtosis, epithelial cell weighted moment 2,3 Kurtosis, epithelial cell weighted moment 2,3 Skewness, epithelial cell weighted moment 3,2 Kurtosis, epithelial cell weighted moment 3,2 Skewness, epithelial cell weighted moment 3,3 The median and epithelial cell weighted moments 3,3 Kurtosis, epithelial cell weighted moment 3,3 Skewness, epithelial cell center weighted moment 0,0 The mean of epithelial cell center weighted moment 0,0 median, epithelial cell center weighted moment 2,0 The mean of epithelial cell center weighted moment 2,0 median, epithelial cell center weighted moment 3,1 The maximum value of epithelial cell center weighted moment 3,1 Skewness, epithelial cell center weighted moment 3,3 The maximum value of epithelial cell center weighted moment 3,3 Skewness, epithelial cell moment2,2 Kurtosis, epithelial cell moment 2,3 Skewness, epithelial cell moment 3,0 The median of epithelial cell moment 3,2 Kurtosis, epithelial cell moment 3,2 Skewness, epithelial cell moment 3,3 Kurtosis, epithelial cell moment 3,3 Skewness, epithelial cell central moment 3,1 Skewness, epithelial cell central moment 3,2 median, epithelial cell center moment 3,3 skewness, skewness of Hu moment 6 of epithelial cells, skewness of minimum caliper distance of inflammatory cells, skewness of maximum caliper distance of inflammatory cells, skewness of maximum value of distance from centroid to cell border of inflammatory cells, kurtosis of median of maximum value of distance from centroid to cell border of inflammatory cells, skewness of median of maximum value of distance from centroid to cell border of inflammatory cells, skewness of mean of maximum value of distance from centroid to cell border of inflammatory cells, skewness of 25th percentile of maximum value of distance from centroid to cell border of inflammatory cells, skewness of maximum Feret's diameter of inflammatory cells, skewness of 0th component of centroid of inflammatory cells, skewness of 1st component of centroid of inflammatory cells, kurtosis of area of ​​inflammatory cells, range of perimeter of inflammatory cells, skewness of convex hull area of ​​inflammatory cells, skewness of equivalent diameter of inflammatory cells, kurtosis of major axis length of inflammatory cells, skewness of major axis length of inflammatory cells, curvature of inflammatory cells along border The skewness of the number of sign changes, the skewness of the maximum negative curvature of inflammatory cells, the range of the length of the negative curvature part of inflammatory cells, the maximum length of the negative curvature part of inflammatory cells, the range of the width of the main peak of positive curvature of inflammatory cells, the maximum width of the main peak of positive curvature of inflammatory cells, the range of the number of main peaks of negative curvature of inflammatory cells, the maximum number of main peaks of negative curvature of inflammatory cells, the skewness of the real part 1 of the Fourier descriptor of inflammatory cells, the maximum value of the imaginary part 4 of the Fourier descriptor of inflammatory cells, the imaginary part The range of 5, the minimum value of the imaginary part 5 of the Fourier descriptor of inflammatory cells, the kurtosis of the imaginary part 5 of the Fourier descriptor of inflammatory cells, the range of the real part 6 of the Fourier descriptor of inflammatory cells, the minimum value of the real part 6 of the Fourier descriptor of inflammatory cells, the maximum value of the imaginary part 6 of the Fourier descriptor of inflammatory cells, the standard deviation of the 25th percentile value of the grayscale value of inflammatory cells, the minimum value of the skewness of the grayscale value of inflammatory cells, the skewness of the 0th component of the weighted centroid of inflammatory cells, the kurtosis of the 1st component of the weighted centroid of inflammatory cells, the weighted moment of inflammatory cells 0,0 The maximum value of inflammatory cell weighted moment 0,0 Kurtosis, inflammatory cell weighted moment 0,0 Skewness, inflammatory cell weighted moment 0,1 Skewness, inflammatory cell weighted moment 0,2 Skewness, inflammatory cell weighted moment0,3 Kurtosis, inflammatory cell weighted moment 1,0 Skewness, inflammatory cell weighted moment 1,3 The range of the inflammatory cell weighted moment 1,3 The maximum value of inflammatory cell weighted moment 1,3 Kurtosis, inflammatory cell weighted moment 1,3 Skewness, inflammatory cell weighted moment 2,0 The range of the inflammatory cell weighted moment 2,1 Kurtosis, inflammatory cell weighted moment 2,2 Kurtosis, inflammatory cell weighted moment 2,2 Skewness, inflammatory cell weighted moment 2,3 Skewness, inflammatory cell weighted moment 3,0 Kurtosis, inflammatory cell weighted moment 3,0 Skewness, inflammatory cell weighted moment 3,1 Kurtosis, inflammatory cell weighted moment 3,2 Kurtosis, inflammatory cell weighted moment 3,2 Skewness, inflammatory cell weighted moment 3,3 Skewness, normalized weighted moment of inflammatory cells 0,2 Kurtosis, normalized weighted moment of inflammatory cells 0,2 Skewness, inflammatory cell center weighted moment 0,0 Kurtosis, inflammatory cell center weighted moment 0,0 Skewness, inflammatory cell center weighted moment 0,1 Skewness, inflammatory cell center weighted moment 0,2 Skewness, inflammatory cell center weighted moment 0,3 The minimum value of the inflammatory cell center weighted moment 0,3 Kurtosis, inflammatory cell center weighted moment 1,0 Kurtosis, inflammatory cell center weighted moment 2,1 The maximum value of the inflammatory cell center weighted moment 2,2 Kurtosis, inflammatory cell center weighted moment 2,2 Skewness, inflammatory cell center weighted moment 3,3 The kurtosis of inflammatory cells, the kurtosis of inflammatory cells Hu moment 1, the skewness of inflammatory cells Hu moment 1, 0,1 Skewness, inflammatory cell moment 0,2 The extreme poor, inflammatory cell matrix 0,2 The maximum value of inflammatory cell moment 0,3 Kurtosis, inflammatory cell moment 1,3 The extreme poor, inflammatory cell matrix 1,3 The maximum value of inflammatory cell moment 2,1 Skewness, inflammatory cell moment 2,2 Skewness, normalized moment of inflammatory cells 0,2 The maximum value of inflammatory cell normalized moment 0,2Kurtosis, normalized moment of inflammatory cells 0,2 Skewness, central moment of inflammatory cells 0,0 Skewness, central moment of inflammatory cells 0,1 Kurtosis, central moment of inflammatory cells 0,2 The extreme difference, inflammatory cell center moment 0,2 The maximum value of the inflammatory cell center moment 0,2 Kurtosis, central moment of inflammatory cells 0,3 The minimum value of the inflammatory cell center moment 0,3 The maximum value of the inflammatory cell center moment 2,0 Kurtosis, central moment of inflammatory cells 2,1 The maximum value of the inflammatory cell center moment 2,2 Kurtosis, central moment of inflammatory cells 2,2 Skewness, central moment of inflammatory cells 2,3 The maximum value of the inflammatory cell center moment 2,3 Kurtosis, central moment of inflammatory cells 2,3 Skewness, central moment of inflammatory cells 3,0 The maximum value of the inflammatory cell center moment 3,2 Kurtosis, central moment of inflammatory cells 3,3 The kurtosis of the connective tissue cells was analyzed, including the range of the mode of the distance from the centroid to the cell boundary of the connective tissue cells, the maximum of the mode of the distance from the centroid to the cell boundary of the connective tissue cells, the range of the width of the main peak of negative curvature of the connective tissue cells, the maximum of the width of the main peak of negative curvature of the connective tissue cells, the skewness of the minimum gray value of the connective tissue cells, the skewness of the 25th percentile gray value of the connective tissue cells, the skewness of the median gray value of the connective tissue cells, the skewness of the 75th percentile gray value of the connective tissue cells, and the skewness of the maximum gray value of the connective tissue cells. The difference between the maximum and minimum values, the standard deviation of the connective tissue cell gray value kurtosis, the range of the connective tissue cell gray value kurtosis, the maximum value of the connective tissue cell gray value kurtosis, the range of the connective tissue cell gray value skewness, the maximum value of the connective tissue cell gray value skewness, the minimum value of the gray level co-occurrence matrix contrast when the connective tissue cell pixel distance is 5, the kurtosis of the connective tissue cell weighted Hu moment 2, the kurtosis of the connective tissue cell weighted Hu moment 3, the skewness of the connective tissue cell weighted Hu moment 3, the connective tissue cell central moment 0,1 Skewness, connective tissue cell central moment 0,3the maximum value of the epithelial cell node degree count in the graph structure, the minimum value of the inflammation-inflammation edge length in the graph structure, the kurtosis of the inflammation-inflammation edge length in the graph structure, the skewness of the inflammation-inflammation edge length in the graph structure, the minimum value of the epithelial-epithelial-inflammation triangle area in the graph structure, the kurtosis of the connective tissue-connective tissue-connective tissue triangle area in the graph structure, the skewness of the connective tissue-connective tissue-connective tissue triangle area in the graph structure, the standard deviation of the epithelial-connective tissue-connective tissue triangle area in the graph structure, the kurtosis of the epithelial-inflammation-inflammation triangle area in the graph structure, the skewness of the epithelial-inflammation-inflammation triangle area in the graph structure, and the standard deviation of the epithelial-epithelial-connective tissue triangle area in the graph structure.

[0038] Preferably, the machine learning model in step S5 is a random survival forest model.

[0039] Preferably, the survival results in step S5 include survival status, death time and last follow-up time.

[0040] A survival prediction model based on tumor pathology images, comprising a data acquisition module, a data processing module, a data analysis module and a result output module; The data acquisition module is used to acquire tumor pathology images of tumor patients; The data processing module takes the tumor pathology image of the tumor patient as input and processes it according to step S1 of the above-mentioned survival prediction method to obtain graph structure data of the tumor pathology image; The data analysis module takes the graph structure data of the tumor pathology image as input and processes it according to any of the above-mentioned survival prediction methods to obtain a risk score for the tumor patient; The result output module is used to output the risk score of the tumor patient.

[0041] Preferably, if the risk score is greater than 5.1141, the predicted survival result of the tumor patient is high risk; if the risk score is ≤5.1141, the predicted survival result of the tumor patient is low risk.

[0042] Among them, high-risk cancer patients need strict attention and regular monitoring during treatment.

[0043] The present invention also seeks to protect the use of the above-mentioned survival prediction model in the preparation of a product for predicting the survival of tumor patients.

[0044] The present invention first uses deep learning methods and attention mechanisms to obtain candidate key areas, then performs refined quantitative tissue morphological feature extraction on pathological images in the candidate key areas, and builds a machine learning model based on this to perform survival prediction. Compared with conventional hand-crafted feature modeling methods and original deep learning methods, the survival prediction method shown in the present invention has improved prediction performance and is explainable.

[0045] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a survival prediction method based on tumor pathology images. This method first uses deep learning and an attention mechanism to identify key regions of the tumor pathology images. Based on these key regions, it then extracts nuclear boundary features, nuclear texture features, and / or nuclear map structural features. Survival prediction is then performed using a trained machine learning model. This method improves the accuracy of survival predictions for cancer patients while overcoming the poor interpretability of traditional deep learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the tissue classification model in Example 1; Figure 2 Schematic diagram of the classification results of the tissue classification model in Example 1; Figure 3 This is the whole section image of the cancer tissue in Example 2; Figure 4 Schematic diagram of a binary image of a grayscale image after OTSU processing in Example 2; Figure 5 Schematic diagram of the structure of the graph convolutional neural network model in Example 2; Figure 6 This is a graph showing the consistency index results when combining the final image features of each grouped image block in test set 1 in Example 3; Figure 7 Schematic diagram of the process of a survival prediction method based on tumor pathology images in Example 4. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below with reference to the accompanying drawings and specific examples. The examples are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. The experimental methods used in the following examples are conventional methods unless otherwise specified; the materials and reagents used are commercially available reagents and materials unless otherwise specified.

[0048] Example 1 Construction and verification of colorectal cancer tissue classification model based on Swin Transformer V2 1. Experimental Methods The schematic diagram of the organizational classification model is as follows Figure 1 shown.

[0049] 1. Sample set division This example uses the NCT-CRC-HE-100K and CRC-VAL-HE-7K datasets for training, validation, and testing of the tissue classification model; both datasets have been standardized by Macehko staining.

[0050] The NCT-CRC-HE-100K dataset contains 100,000 224-pixel × 224-pixel (112-μm × 112-μm) image patches derived from hematoxylin and eosin (H&E)-stained tissue sections of 86 colorectal cancer patients. Each image patch is annotated with the following nine tissue types: adipose tissue (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus (MUC), smooth muscle (MUS), normal colon mucosa (NORM), cancer-associated stroma (STR), and colorectal adenocarcinoma epithelium (TUM).

[0051] CRC-VAL-HE-7K dataset: Contains 7180 224 pixel × 224 pixel (112 μm × 112 μm) image blocks from different patients than the NCT-CRC-HE-100K dataset, and similarly labels each image block with the nine tissue types mentioned above.

[0052] Data partitioning: NCT-CRC-HE-100K is divided into training and validation sets with a ratio of 7:3, and CRC-VAL-HE-7K is used as the test set.

[0053] 2. Training of the Organization Classification Model Swin Transformer V2 was used as the deep learning model for colorectal cancer tissue classification. The label smoothed cross entropy loss function was used during training, and the label smoothing degree was set to 0.1. The AdamW optimizer was selected, the batch size was 64, the initial learning rate was 0.001, and the weight decay coefficient was 0.05.

[0054] The training was conducted for 8 rounds. The first two rounds used the LinearLR scheduler with a starting factor of 0.001; the last six rounds used the cosine annealing learning rate scheduler with eta_min set to 1e-5.

[0055] The image blocks in the training set were input into the model, and the classification results of the nine tissue types for each image block in the training set were output. During the training process, the parameters were adjusted to optimize the model performance, and the validation set was used to evaluate the classification accuracy after each round to obtain the trained tissue classification model.

[0056] 3. Testing of the Organizational Classification Model After training is completed, the image blocks of the test set (CRC-VAL-HE-7K dataset) are used as input, and the tissue classification model outputs the tissue classification results of the image blocks in the test set. Based on the tissue classification results of the test set by the tissue classification model, the classification accuracy of the test set is calculated.

[0057] 2. Experimental Results After 8 rounds of training, the classification accuracy of the model on the validation set reached 99.3932%, and the classification accuracy on the test set was 95.8914%, indicating that the tissue classification model can accurately classify different tissues in the image blocks. The classification results of the tissue classification model are shown in the figure below. Figure 2 shown.

[0058] Example 2 A graph convolutional neural network model for colorectal cancer survival prediction 1. Experimental Methods 1. Acquisition of image data and clinical data: The whole-slice images (WSIs) of cancer tissues stained with H&E were obtained from The Cancer Genome Atlas (TCGA)-COAD (colon adenocarcinoma) and TCGA-READ (rectal adenocarcinoma) datasets. Figure 3 As shown; WSIs whose source patients' tumor stages are in T2 or T3 stages are selected to form a WSI set to be used.

[0059] 2. Acquisition of graph structure data: Step 2.1: downsample each WSI in the unused WSI set obtained in step 1 by a factor of 64 and convert it into a grayscale image; Step 2.2: Use the Otsu threshold algorithm (OTSU) to process the grayscale image obtained in step 2.1 to obtain a binary image that can separate the tissue area from the background area, where white represents the tissue area and black represents the remaining area; the binary image of the grayscale image processed by OTSU is shown in the figure below. Figure 4 As shown; Remove the background area in the binary image to obtain the pathological image of the tumor to be tested; Step 2.3: Divide the tissue area in the pathological image to be tested into image blocks of 112 μm × 112 μm (224 pixels × 224 pixels), and save the coordinates of each image block; Step 2.4: After performing Macehko staining normalization processing on each image block obtained in step S2.3, input the tissue classification model in Example 1 to extract the N-dimensional feature vector and obtain the tissue category of the image block; Image blocks with tissue category TUM and their eight neighborhoods (upper, lower, left, right, and four diagonal lines) are considered qualified image blocks. Tumor pathology images with less than 2,000 or more than 20,000 qualified image blocks are filtered out. For each tumor pathology image, an M×N feature matrix X is generated, where M is the number of qualified image blocks and N is the dimension of the feature vector. Step 2.5: Based on the coordinates of the qualified image blocks obtained in step S2.4, use the k-nearest neighbor algorithm (k=4) to construct the adjacency matrix A of the tumor pathology image to be tested; the feature matrix X of the tumor pathology image to be tested and the adjacency matrix A together constitute the graph structure data of the tumor pathology image to be tested.

[0060] 3. Dataset division: The graph structure data of the tumor pathology image to be tested obtained in step 2 is divided into training set 1 and test set 1 in a ratio of 4:1.

[0061] 4. Graph Convolutional Neural Network Model Architecture: In the graph convolutional neural network model, the input dimension and output dimension of each convolutional layer are both O (O=256), the hidden layer dimension O is 256, the convolutional layer is GENconv, and the output dimension of the last fully connected layer is 4. The structural diagram of the graph convolutional neural network model is as follows Figure 5 As shown; The graph convolutional neural network converts the feature matrix X in the graph structure data of the tumor pathology image to be tested into a hidden dimension O through a fully connected layer, and uses the ReLU activation function and Dropout function to obtain the initialization feature H0 of the tumor pathology image to be tested; Input the initialization feature H0 and the adjacency matrix A of the tumor pathology image to be tested into the first layer of graph convolution layer, and obtain the output H1 of the first layer of graph convolution layer according to formula I, and then obtain the output H of the Lth layer of graph convolution layer according to formula II. L , L=(2,3,...L);the obtained H1,...,H L According to formula III, characteristic H is obtained; Formula I: ; Formula II: ; Formula III: ; The feature H is input into a fully connected layer of the graph convolutional neural network model, and nonlinear changes are performed using the ReLU activation function and the Dropout function to obtain the feature H Ф ; H ФAccording to formula IV, the attention weight matrix M is obtained, where the attention weight matrix is ​​a two-column matrix, one column is WSI and the other column is the attention score corresponding to WSI; and according to formula V, the attention weight matrix M is used to calculate H Ф Perform weighted aggregation and process it through a fully connected layer and ReLU activation function to obtain the feature H ρ ; Formula IV: ; Formula V: ; Discretize the survival time of each sample into t (t = 4) non-overlapping time intervals, and combine the H ρ Calculate the conditional probability of death (hazards) within t time periods, and then calculate t survival probabilities S according to formula VII t ; Formula VI: ; Formula VII: ; The survival probability S t The overall risk of an individual is calculated according to Formula VIII; Formula VIII: .

[0062] 5. Training of graph convolutional neural network model: Use training set 1 from step 3 as input to the graph convolutional neural network model shown in step 4. Train the graph convolutional neural network using 4-fold cross-validation, and determine the optimal hyperparameters based on the mean cross-validation performance. The final model's hyperparameters are: a Cox proportional loss function based on cross-entropy, an Adam optimizer, a learning rate of 8e-6, and a weight decay of 1e-5.

[0063] The first 10 training rounds use the LinearLR scheduler with a starting factor of 0.001; the next 10 rounds use the cosine annealing learning rate scheduler to train the graph convolutional neural network model according to the optimal hyperparameters to obtain the trained graph convolutional neural network model.

[0064] 6. Testing of graph convolutional neural network model: Use test set 1 to test and evaluate the trained graph convolutional neural network model, and calculate the consistency index of the graph convolutional neural network model on test set 1.

[0065] 2. Experimental Results When the trained graph convolutional neural network model was used to evaluate the samples in test set 1, the consistency index was 0.5983, indicating that the trained graph convolutional neural network model can achieve survival prediction for colorectal cancer patients.

[0066] Example 3 Construction of a random survival forest model for colorectal cancer patient survival prediction and threshold determination 1. Experimental Methods 1. Dataset This embodiment uses the training set 1 and the test set 1 in step 3 of embodiment 2 as data sets.

[0067] 2. Cell nucleus segmentation and classification The StarDist model, which performed well in the Conic Challenge, was used to segment and classify the cell nuclei in training set 1 and test set 1, and the outlines and categories of the cell nuclei in training set 1 and test set 1 were obtained.

[0068] 3. Attention score grouping According to the steps in Example 2, the graph convolutional neural network model described in Example 2 is used to perform survival prediction for each WSI in training set 1 and test set 1, and the attention weight matrix of each WSI in training set 1 and test set 1 is obtained to extract the attention score of each WSI.

[0069] Based on the attention scores of each WSI in training set 1 and test set 1, image blocks with the top 10%, top 20%, top 30%, top 40%, top 50%, top 60%, top 70%, top 80%, top 90%, and 100% areas of attention scores in training set 1 and test set 1, respectively, are selected as image blocks to be analyzed, and 10 groups are formed for training set 1 and test set 1, respectively.

[0070] 4. Feature extraction Based on the outlines and categories of the cell nuclei in the training set 1 and the test set 1 obtained in step 2, the cell nucleus boundary features, cell nucleus texture features, cell nucleus map structure features and cell nucleus direction features are extracted from each grouped image block of the training set 1 and the test set 1 obtained in step 3, respectively, to obtain the image features of each grouped image block. The four features are as follows: (1) Nuclear boundary characteristics include curvature characteristics, nuclear size and nuclear shape characteristics; Curvature characteristics: mean value and standard deviation of curvature, number of times the curvature changes sign along the boundary, maximum value, mean value, median value, standard deviation, sum, and number of positive and negative curvatures, number, average peak value, and average width of main peaks of positive and negative curvatures; ratio of the positive and negative curvature parts to the total perimeter of the boundary, ratio of the part where the curvature changes sign to the total perimeter of the boundary; Nuclear size and shape features: minimum, maximum, median, mean, mode, 25th percentile, 75th percentile, standard deviation of the distance from the centroid to the cell border, maximum Feret diameter, minimum and maximum caliper distances measured in all rotations, ratio of minimum to maximum caliper distance, first component of centroid, second component of centroid, area, perimeter, bounding box area, convex hull area, equivalent diameter, major and minor axis lengths of the best-fit ellipse, eccentricity, orientation, compactness, ratio of the major to minor axis of the best-fit ellipse, circularity, ratio of area to bounding box area, Fourier descriptor, Zernike moments.

[0071] The mean, median, standard deviation, maximum value, minimum value, difference between maximum and minimum value, kurtosis and skewness of the cell nucleus boundary characteristics were calculated respectively.

[0072] (2) Cell nucleus texture features include gray-level co-occurrence matrix features, image moment features, gray value statistical features, intensity contrast features, and chromatin features; Gray-level co-occurrence matrix features: contrast, dissimilarity, homogeneity, angular second moment, energy, and correlation of the gray-level co-occurrence matrix at pixel distances of 1, 5, and 20; Image moment features: weighted centroid, weighted moment, normalized weighted moment, center weighted moment, Hu invariant weighted moment, moment, normalized moment, center moment and Hu invariant moment; Gray value statistical features: minimum, maximum, median, mean, mode, 25th percentile, 75th percentile, standard deviation, kurtosis, skewness, and entropy of the cell nucleus gray value; Intensity contrast features: the ratio of the grayscale intensity of the 80th percentile to the grayscale intensity of the 20th percentile, and the ratio of the number of pixels with grayscale intensity greater than or equal to the 80th percentile to the number of pixels with grayscale intensity less than or equal to the 20th percentile; Chromatin features include the ratio of the area occupied by heterochromatin (pixels with grayscale intensity greater than or equal to the hc threshold) to that of euchromatin (pixels with intensity less than the hc threshold), the ratio of the area occupied by heterochromatin to the total area of ​​the nucleus, the ratio of the total intensity of heterochromatin to the total intensity of euchromatin, and the ratio of the total intensity of heterochromatin to the total intensity within the region. The mean image intensity within the nuclear region plus the standard deviation of the image intensity is used as the threshold. Heterochromatin pixels are defined as pixels with intensity ≥ the threshold, while euchromatin pixels are defined as pixels with intensity < the threshold.

[0073] The mean, median, standard deviation, maximum value, minimum value, difference between maximum and minimum value, kurtosis and skewness of the cell nucleus texture features were counted respectively.

[0074] (3) Graph structural features include distribution features and scale features; With the nucleus centroid as the node, the Delaunay triangle was constructed for epithelial cells, inflammatory cells (plasma cells, lymphocytes, neutrophils, eosinophils), and connective tissue cells respectively; The edge types are defined based on the two nodes of each edge in the Delaunay triangle, including epithelium-inflammation, epithelium-connective tissue, and inflammation-connective tissue; the triangle types are defined based on the three nodes of the Delaunay triangle, including epithelium-epithelium-epithelium, inflammation-inflammation-inflammation, connective tissue-connective tissue-connective tissue, epithelium-epithelial-inflammation, epithelium-epithelial-connective tissue, inflammation-inflammation-connective tissue, inflammation-inflammation-epithelium, connective tissue-connective tissue-epithelium, connective tissue-connective tissue-inflammation, and epithelium-inflammation-connective tissue.

[0075] Distribution characteristics and proportional characteristics were obtained based on Delaunay triangulation. The distribution characteristics included: node degree, triangle side length, triangle area, degree of various cell types and other types and their ratios, node degree of various cell types, side length of various edge types, and triangle area of ​​various triangle types. The mean, median, standard deviation, maximum value, minimum value, difference between maximum and minimum value, kurtosis, and skewness of each distribution characteristic were calculated respectively.

[0076] Proportional features include: the proportion of various cell types in all cells, the proportion of various types of edges in all edges, the proportion of the total length of various types of edges in the total length of all edges, the proportion of various types of triangles in all triangles, and the proportion of the total area of ​​various types of triangles in the total area of ​​all triangles.

[0077] (4) Cell nucleus direction feature: To calculate the direction of each cell nucleus, the COrE feature (cell nucleus direction feature) is calculated.

[0078] 5. Feature Screening Based on the median survival time of each sample in training set 1 and test set 1, the samples in each grouped image block in training set 1 and test set 1 were divided into long-term survival samples (survival time ≥ median) and short-term survival samples (survival time < median). The P value of the image features of each grouped image block in the long-term survival group and the short-term survival group in each grouped image block sample was calculated using t-test. Image features with a P value less than 0.05 were retained as preliminary screening image features. Then, the maximum variance minimum redundancy (mRMR) method was used to select 200 final image features from the preliminary screening image features, resulting in 200 final image features for each grouped image block. The 200 final image features include: Kurtosis of the maximum caliper distance of epithelial cells, skewness of the maximum caliper distance of epithelial cells, kurtosis of the maximum distance from the centroid to the cell boundary of epithelial cells, skewness of the maximum distance from the centroid to the cell boundary of epithelial cells, kurtosis of the 75th percentile of the maximum distance from the centroid to the cell boundary of epithelial cells, kurtosis of the maximum Feret diameter of epithelial cells, skewness of the maximum Feret diameter of epithelial cells, kurtosis of the second component of the centroid of epithelial cells, kurtosis of the long axis length of epithelial cells, Zernike moment of epithelial cells 19 skewness, kurtosis of the real part 1 of the epithelial cell Fourier descriptor, standard deviation of the 25th percentile of the epithelial cell grayscale, median of the epithelial cell grayscale median, standard deviation of the epithelial cell grayscale median, kurtosis of the epithelial cell grayscale median, kurtosis of the 75th percentile of the epithelial cell grayscale, kurtosis of the epithelial cell grayscale maximum, skewness of the epithelial cell grayscale maximum, kurtosis of the epithelial cell grayscale mean, mean of the epithelial cell grayscale mode, median of the epithelial cell grayscale mode, standard deviation of the epithelial cell grayscale mode, median of the epithelial cell grayscale distribution kurtosis, standard deviation of the epithelial cell grayscale distribution kurtosis, mean of the epithelial cell grayscale distribution skewness, total intensity of epithelial cell heterochromatin and euchromatin The kurtosis of the ratio of the total intensity of the epithelial cells is 1, the kurtosis of the gray-level co-occurrence matrix contrast when the epithelial cell pixel distance is 5, the standard deviation of the gray-level co-occurrence matrix contrast when the epithelial cell pixel distance is 20, the kurtosis of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 1, the skewness of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 1, the kurtosis of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 5, the standard deviation of the gray-level co-occurrence matrix dissimilarity when the epithelial cell pixel distance is 20, the kurtosis of the first component of the epithelial cell weighted centroid, and the epithelial cell weighted moment. 0,0 The mean value of epithelial cell weighted moment 0,0 The standard deviation of epithelial cell weighted moment 1,2 The median and epithelial cell weighted moments 2,0 The mean value of epithelial cell weighted moment 2,2 The median and epithelial cell weighted moments 2,2 Kurtosis, epithelial cell weighted moment 2,3 Kurtosis, epithelial cell weighted moment 2,3 Skewness, epithelial cell weighted moment 3,2 Kurtosis, epithelial cell weighted moment 3,2 Skewness, epithelial cell weighted moment 3,3 The median and epithelial cell weighted moments 3,3 Kurtosis, epithelial cell weighted moment 3,3 Skewness, epithelial cell center weighted moment 0,0 The mean of epithelial cell center weighted moment0,0 median, epithelial cell center weighted moment 2,0 The mean of epithelial cell center weighted moment 2,0 median, epithelial cell center weighted moment 3,1 The maximum value of epithelial cell center weighted moment 3,1 Skewness, epithelial cell center weighted moment 3,3 The maximum value of epithelial cell center weighted moment 3,3 Skewness, epithelial cell moment 2,2 Kurtosis, epithelial cell moment 2,3 Skewness, epithelial cell moment 3,0 The median of epithelial cell moment 3,2 Kurtosis, epithelial cell moment 3,2 Skewness, epithelial cell moment 3,3 Kurtosis, epithelial cell moment 3,3 Skewness, epithelial cell central moment 3,1 Skewness, epithelial cell central moment 3,2 median, epithelial cell center moment 3,3skewness, skewness of Hu moment 6 of epithelial cells, skewness of minimum caliper distance of inflammatory cells, skewness of maximum caliper distance of inflammatory cells, skewness of maximum value of distance from centroid to cell border of inflammatory cells, kurtosis of median of maximum value of distance from centroid to cell border of inflammatory cells, skewness of median of maximum value of distance from centroid to cell border of inflammatory cells, skewness of mean of maximum value of distance from centroid to cell border of inflammatory cells, skewness of 25th percentile of maximum value of distance from centroid to cell border of inflammatory cells, skewness of maximum Feret's diameter of inflammatory cells, skewness of 0th component of centroid of inflammatory cells, skewness of 1st component of centroid of inflammatory cells, kurtosis of area of ​​inflammatory cells, range of perimeter of inflammatory cells, skewness of convex hull area of ​​inflammatory cells, skewness of equivalent diameter of inflammatory cells, kurtosis of major axis length of inflammatory cells, skewness of major axis length of inflammatory cells, curvature of inflammatory cells along border The skewness of the number of sign changes, the skewness of the maximum negative curvature of inflammatory cells, the range of the length of the negative curvature part of inflammatory cells, the maximum length of the negative curvature part of inflammatory cells, the range of the width of the main peak of positive curvature of inflammatory cells, the maximum width of the main peak of positive curvature of inflammatory cells, the range of the number of main peaks of negative curvature of inflammatory cells, the maximum number of main peaks of negative curvature of inflammatory cells, the skewness of the real part 1 of the Fourier descriptor of inflammatory cells, the maximum value of the imaginary part 4 of the Fourier descriptor of inflammatory cells, the imaginary part The range of 5, the minimum value of the imaginary part 5 of the Fourier descriptor of inflammatory cells, the kurtosis of the imaginary part 5 of the Fourier descriptor of inflammatory cells, the range of the real part 6 of the Fourier descriptor of inflammatory cells, the minimum value of the real part 6 of the Fourier descriptor of inflammatory cells, the maximum value of the imaginary part 6 of the Fourier descriptor of inflammatory cells, the standard deviation of the 25th percentile value of the grayscale value of inflammatory cells, the minimum value of the skewness of the grayscale value of inflammatory cells, the skewness of the 0th component of the weighted centroid of inflammatory cells, the kurtosis of the 1st component of the weighted centroid of inflammatory cells, the weighted moment of inflammatory cells 0,0 The maximum value of inflammatory cell weighted moment 0,0 Kurtosis, inflammatory cell weighted moment 0,0 Skewness, inflammatory cell weighted moment 0,1 Skewness, inflammatory cell weighted moment 0,2 Skewness, inflammatory cell weighted moment 0,3 Kurtosis, inflammatory cell weighted moment 1,0 Skewness, inflammatory cell weighted moment 1,3 The range of the inflammatory cell weighted moment 1,3 The maximum value of inflammatory cell weighted moment 1,3 Kurtosis, inflammatory cell weighted moment 1,3 Skewness, inflammatory cell weighted moment 2,0 The range of the inflammatory cell weighted moment 2,1 Kurtosis, inflammatory cell weighted moment 2,2 Kurtosis, inflammatory cell weighted moment2,2 Skewness, inflammatory cell weighted moment 2,3 Skewness, inflammatory cell weighted moment 3,0 Kurtosis, inflammatory cell weighted moment 3,0 Skewness, inflammatory cell weighted moment 3,1 Kurtosis, inflammatory cell weighted moment 3,2 Kurtosis, inflammatory cell weighted moment 3,2 Skewness, inflammatory cell weighted moment 3,3 Skewness, normalized weighted moment of inflammatory cells 0,2 Kurtosis, normalized weighted moment of inflammatory cells 0,2 Skewness, inflammatory cell center weighted moment 0,0 Kurtosis, inflammatory cell center weighted moment 0,0 Skewness, inflammatory cell center weighted moment 0,1 Skewness, inflammatory cell center weighted moment 0,2 Skewness, inflammatory cell center weighted moment 0,3 The minimum value of the inflammatory cell center weighted moment 0,3 Kurtosis, inflammatory cell center weighted moment 1,0 Kurtosis, inflammatory cell center weighted moment 2,1 The maximum value of the inflammatory cell center weighted moment 2,2 Kurtosis, inflammatory cell center weighted moment 2,2 Skewness, inflammatory cell center weighted moment 3,3 The kurtosis of inflammatory cells, the kurtosis of inflammatory cells Hu moment 1, the skewness of inflammatory cells Hu moment 1, 0,1 Skewness, inflammatory cell moment 0,2 The extreme poor, inflammatory cell matrix 0,2 The maximum value of inflammatory cell moment 0,3 Kurtosis, inflammatory cell moment 1,3 The extreme poor, inflammatory cell matrix 1,3 The maximum value of inflammatory cell moment 2,1 Skewness, inflammatory cell moment 2,2 Skewness, normalized moment of inflammatory cells 0,2 The maximum value of inflammatory cell normalized moment 0,2 Kurtosis, normalized moment of inflammatory cells 0,2 Skewness, central moment of inflammatory cells 0,0 Skewness, central moment of inflammatory cells 0,1 Kurtosis, central moment of inflammatory cells 0,2 The extreme difference, inflammatory cell center moment 0,2 The maximum value of the inflammatory cell center moment 0,2 Kurtosis, central moment of inflammatory cells 0,3 The minimum value of the inflammatory cell center moment 0,3 The maximum value of the inflammatory cell center moment 2,0Kurtosis, central moment of inflammatory cells 2,1 The maximum value of the inflammatory cell center moment 2,2 Kurtosis, central moment of inflammatory cells 2,2 Skewness, central moment of inflammatory cells 2,3 The maximum value of the inflammatory cell center moment 2,3 Kurtosis, central moment of inflammatory cells 2,3 Skewness, central moment of inflammatory cells 3,0 The maximum value of the inflammatory cell center moment 3,2 Kurtosis, central moment of inflammatory cells 3,3 The kurtosis of the connective tissue cells was analyzed, including the range of the mode of the distance from the centroid to the cell boundary of the connective tissue cells, the maximum of the mode of the distance from the centroid to the cell boundary of the connective tissue cells, the range of the width of the main peak of negative curvature of the connective tissue cells, the maximum of the width of the main peak of negative curvature of the connective tissue cells, the skewness of the minimum gray value of the connective tissue cells, the skewness of the 25th percentile gray value of the connective tissue cells, the skewness of the median gray value of the connective tissue cells, the skewness of the 75th percentile gray value of the connective tissue cells, and the skewness of the maximum gray value of the connective tissue cells. The difference between the maximum and minimum values, the standard deviation of the connective tissue cell gray value kurtosis, the range of the connective tissue cell gray value kurtosis, the maximum value of the connective tissue cell gray value kurtosis, the range of the connective tissue cell gray value skewness, the maximum value of the connective tissue cell gray value skewness, the minimum value of the gray level co-occurrence matrix contrast when the connective tissue cell pixel distance is 5, the kurtosis of the connective tissue cell weighted Hu moment 2, the kurtosis of the connective tissue cell weighted Hu moment 3, the skewness of the connective tissue cell weighted Hu moment 3, the connective tissue cell central moment 0,1 Skewness, connective tissue cell central moment 0,3 the maximum value of the epithelial cell node degree count in the graph structure, the minimum value of the inflammation-inflammation edge length in the graph structure, the kurtosis of the inflammation-inflammation edge length in the graph structure, the skewness of the inflammation-inflammation edge length in the graph structure, the minimum value of the epithelial-epithelial-inflammation triangle area in the graph structure, the kurtosis of the connective tissue-connective tissue-connective tissue triangle area in the graph structure, the skewness of the connective tissue-connective tissue-connective tissue triangle area in the graph structure, the standard deviation of the epithelial-connective tissue-connective tissue triangle area in the graph structure, the kurtosis of the epithelial-inflammation-inflammation triangle area in the graph structure, the skewness of the epithelial-inflammation-inflammation triangle area in the graph structure, and the standard deviation of the epithelial-epithelial-connective tissue triangle area in the graph structure.

[0079] 6. Training of Random Survival Forest The 200 final image features of each grouped image block in training set 1 were used as input features. The hyperparameters of the random survival forest model were tuned by random search. 100 random searches were performed in the parameter space, and three-fold cross-validation was used for each search. The consistency index (C-Index) between the risk score output by the model and the actual survival status of the samples in training set 1 was calculated. The parameter configuration with the highest consistency index was selected as the optimal hyperparameter.

[0080] The parameter space includes: the number of trees is between 100 and 1000, the maximum depth of the tree is between 10 and 100, the minimum number of samples for splitting a node is between 2 and 20, and the minimum number of samples for a leaf node is between 1 and 10.

[0081] Based on the determined optimal hyperparameters, the random survival forest model was trained using the final image features of each grouped image block in training set 1 as input and the risk scores of the samples in training set 1 as prediction values. This resulted in a random survival forest model for predicting survival of colorectal cancer patients. The median risk score when training using samples in training set 1 was also recorded.

[0082] 7. Model Performance Evaluation The consistency index (C-Index) of the random survival forest model for colorectal cancer patient survival prediction was tested by combining the final image features of each grouped image block in test set 1.

[0083] 2. Experimental Results The consistency index results when combining the final image features of each grouped image block in test set 1 are shown in the figure below: Figure 6 As shown in the figure, the results show that when the final cell features of the image blocks in the top 70% of the attention score area are used as input, the random survival forest model performs best, with a C-Index of 0.7; in contrast, when the final features of the image blocks larger or smaller than 70% are used, the consistency index of the model decreases significantly.

[0084] The results show that the best prediction effect can be achieved by using the final cell features of the image blocks in the top 70% of the attention score as the input of the random survival forest model.

[0085] Example 4 A survival prediction method based on tumor pathology images A flowchart of a survival prediction method based on tumor pathology images is shown in Figure 7 As shown, the specific steps include: The tumor pathology image of the sample to be tested is subjected to the steps shown in Example 2 to obtain the attention weight matrix of the tumor pathology image of the sample to be tested, the attention score is extracted based on the attention weight matrix, and the image blocks with the top 70% of the attention scores are selected.

[0086] The StarDist model was used to segment and classify the cell nuclei in the top 70% of the image blocks with the highest attention scores, and the final cell features of the top 70% of the image blocks with the highest attention scores were obtained. The features were input into the random survival forest model for predicting the survival of colorectal cancer patients shown in Example 3 to obtain the risk score of the sample to be tested.

[0087] The median risk score of 5.1141 when training with training set 1 in Example 3 is used as the threshold. If the risk score of the test sample is greater than the threshold, it is high risk. If the risk score of the test sample is less than or equal to the threshold, it is low risk. High-risk samples have a worse survival prognosis than low-risk samples and require strict attention and regular monitoring during treatment.

[0088] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art will readily appreciate that other variations or modifications may be made based on the above descriptions and concepts. It is not necessary and impossible to provide an exhaustive list of all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A survival prediction method based on tumor pathology images, characterized in that: The following steps are involved: S1. Pathology image data processing: S11. Build a tissue classification model, wherein the tissue classification model can identify different tissues in the pathological image; S12. Converting the tumor pathology image of the sample to be tested into a grayscale image, and obtaining the foreground area of ​​the grayscale image as the tissue area; S13. Dividing the tissue region into non-overlapping image blocks to be tested, and recording the coordinates of each image block to be tested in the tumor pathology image of the sample to be tested; S14. Input the image block to be tested into the tissue classification model constructed in S11 to obtain the tissue classification result and feature vector of the image block to be tested; record the image block to be tested that is identified as a tumor by the tissue classification result and its eight neighboring image blocks as qualified image blocks, and record the number of qualified image blocks; Based on the feature vector of the image block to be tested and the number of qualified image blocks, an M×N feature matrix X is generated for the tumor pathology image of the sample to be tested, where N is the dimension of the feature vector and M is the number of qualified image blocks; S15. Based on the coordinates of the qualified image blocks in the tumor pathology image, use the k-nearest neighbor algorithm to construct an adjacency matrix A of the tumor pathology image of the sample to be tested; Among them, the feature matrix X and the adjacency matrix A are the graph structure data of the sample to be tested; S2. Input the graph structure data of the sample to be tested into the trained graph convolutional neural network model to obtain the risk score of the sample to be tested.

2. The survival prediction method according to claim 1, characterized in that In step S12, the Otsu threshold algorithm is used to obtain the foreground area in the grayscale image.

3. The survival prediction method according to claim 1, characterized in that In step S14 , the image block to be tested obtained in step S13 is first stained and normalized, and then input into the tissue classification model obtained in step S11 .

4. The survival prediction method according to claim 1, characterized in that The k-nearest neighbor algorithm in step S15 is a k-nearest neighbor algorithm with k=4.

5. The survival prediction method according to claim 1, characterized in that Before obtaining the risk score of the sample to be tested in step S2, the following steps are also included: S3. Input the graph structure data of the test sample into the trained graph convolutional neural network model to obtain the attention weight matrix of the test sample; based on the attention weight matrix of the test sample, obtain the attention score of each qualified image block in the test sample, sort the attention scores in descending order, and select the top 10% to 100% of the image blocks as the analysis image blocks; S4. Tissue morphology feature extraction: S41. Use the trained cell nucleus segmentation and classification model to perform cell nucleus segmentation and classification on the analysis image block to obtain the outline and category of the cell nucleus in the analysis image block; S42. Based on the analysis of the outline and category of the cell nucleus in the image block, obtain the cell nucleus boundary features, cell nucleus texture features, cell nucleus map structure features and / or cell nucleus direction features of the sample to be tested; S5. Inputting the nuclear boundary features, nuclear texture features, and / or nuclear map structural features of the sample to be tested into the trained machine learning model to obtain a risk score for the sample to be tested; The trained machine learning model is a machine learning model trained using a training set, and the training set is tumor pathology images with known survival results.

6. The survival prediction method according to claim 5, characterized in that: In step S3, the image blocks in the top 70% of the attention score area are selected as analysis image blocks.

7. The survival prediction method according to claim 5, characterized in that: The cell nucleus boundary features in step S42 include cell nucleus curvature features, cell nucleus size and cell nucleus shape features; the cell nucleus texture features include gray level co-occurrence matrix features, gray value statistical features, intensity contrast features and chromatin features; the cell nucleus map structural features include distribution features and proportion features; The cell nucleus direction feature is the COrE feature of the cell nucleus.

8. The survival prediction method according to claim 7, characterized in that: The nucleus curvature characteristics include: the mean value and standard deviation of the curvature, the number of times the curvature changes sign along the boundary, the maximum value, mean value, median value, standard deviation, sum, and number of positive and negative curvatures, the number, mean peak value, and mean width of positive and negative curvature peaks, the ratio of the positive and negative curvature parts to the total perimeter of the boundary, and the ratio of the part where the curvature changes sign to the total perimeter of the boundary; The cell nuclear size and cell nuclear shape features include: the minimum, maximum, median, mean, mode, 25th percentile, 75th percentile, standard deviation, maximum Feret diameter of the distance from the center of mass to the cell boundary, the longest distance between any two points on the boundary, the minimum and maximum caliper distances measured in all rotations, the ratio of the minimum caliper distance to the maximum caliper distance, area, perimeter, bounding box area, convex hull area, equivalent diameter, major axis length and minor axis length of the best-fit ellipse, eccentricity, orientation, compactness, ratio of the major axis to the minor axis of the best-fit ellipse, roundness, ratio of area to bounding box area, Fourier descriptor, and Zernike moment.

9. The survival prediction method according to claim 5, characterized in that: The machine learning model in step S5 is a random survival forest model.

10. A survival prediction model based on tumor pathology images, characterized in that: It includes data acquisition module, data processing module, data analysis module and result output module; The data acquisition module is used to acquire tumor pathology images of tumor patients; The data processing module takes the tumor pathology image of the tumor patient as input and processes it according to step S1 of the survival prediction method according to claim 1 to obtain graph structure data of the tumor pathology image; The data analysis module takes the graph structure data of the tumor pathology image as input and processes it according to the survival prediction method according to any one of claims 1 to 9 to obtain a risk score for the tumor patient; The result output module is used to output the risk score of the tumor patient.

Images