Pathological image survival analysis method based on hierarchical graph convolution and attention

Abstract

The application discloses a pathological image survival analysis method based on hierarchical graph convolution and attention. The method collects a full field digital image and carries out staining pretreatment. The cell nucleus is segmented from the pretreated pathological image and feature extraction is carried out, and then a cell graph is constructed. Meanwhile, the pretreated pathological image is cut and features are extracted, and then a slice graph is constructed. A multi-level cell graph and slice graph feature fusion model is constructed, including a cell graph convolution module, a slice graph convolution module and a gated attention reading module based on the cell graph, the cell graph features are extracted, fused into the slice graph, and the slice graph features are further extracted. Finally, the pathological image level representation is used for survival prediction by cox regression. By extracting different scale pathological image information and considering the spatial connectivity, the multi-level topological structure of the whole field pathological image is explored, the efficiency and granularity are balanced, and the prognostic analysis of the whole field digital pathological image is carried out from low level to high level.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to an artificial intelligence-based method for processing histopathological images, specifically a survival analysis method for pathological images based on hierarchical graph convolution and attention. Background Technology

[0002] In the medical field, digitized histopathological images contain a wealth of tissue and cell morphology information, making them the most effective basis for cancer diagnosis. Clinicians can not only use pathological images for cancer diagnosis, classification, and treatment, but also for prognostic analysis. With the development of artificial intelligence technology, especially the emergence of deep neural networks, machines can assist doctors in analyzing pathological images. Cancer prognosis is a challenging task. The goal is to predict the time of death or the occurrence of specific diseases such as cardiac arrest through the processing of histopathological images. Accurate survival prediction can not only assist doctors in developing treatment plans but also improve patients' quality of life.

[0003] However, with the increasing resolution of medical image acquisition equipment, limitations such as image size, data availability, and machine hardware necessitate cropping a full-view digital image into multiple small patches during image analysis, leading to the loss of some global information. To address this issue, graph convolutional neural network (GCN)-based methods model WSI as graph-structured data, combining global context and local connectivity with parameter efficiency. Existing techniques propose two methods: patch-based and cell-based. However, patch-based methods ignore the topology and microenvironment of cells, resulting in some information loss and poor model prediction accuracy, failing to meet the requirements of general pathological image prognostic analysis. Cell-based methods are too inefficient when modeling WSI with a large number of cells. Therefore, there is an urgent need for a pathological image representation method that integrates multi-level information and achieves a balance between efficiency and granularity. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a survival analysis method for pathological images based on hierarchical graph convolution and attention, which fuses cell images and slice images to solve the problem of lacking multi-scale fusion information in existing deep learning-based digital pathological image survival prediction methods.

[0005] The survival analysis method for pathological images based on hierarchical graph convolution and attention includes the following steps:

[0006] Step 1: Collect full-view digital pathology images and preprocess them, save the corresponding survival time, and construct a training set.

[0007] Step 2: Segment cell nuclei from the preprocessed pathological image and extract features to construct a cell graph:

[0008] s2.1. Use the pre-trained Hover-Net segmentation model to accurately depict the boundary of the cell nucleus, extract the morphological features of the cell nucleus to represent the complex tissues and overall micro-tissues of the cell, and use the centroid of the segmented cell nucleus as the spatial coordinates of the nodes in the cell graph.

[0009] s2.2 Since many cell nuclei with similar characteristics are distributed in some dense regions, in order to reduce redundancy, the farthest point sampling (FPS) method is used to select the cell nuclei to be retained. Compared with random sampling, farthest point sampling effectively alleviates the problem of sampling nuclei in sparse regions.

[0010] s2.3. The sampled cell nuclei are used as nodes in the cell graph, and the potential interactions between two cell nuclei are used as edges in the cell graph. Adjacent cell nuclei are found according to the set Euclidean distance threshold and connected.

[0011] Step 3: Divide the preprocessed pathological images into sections and construct patch-graphs:

[0012] s3.1. Use a sliding window to segment the preprocessed pathological image into K non-overlapping slices of the same size, and save the position coordinates (x,y) of the slices in the original pathological image.

[0013] s3.2. Input the slices obtained from s3.1 into the pre-trained ResNet-50 model for feature extraction.

[0014] s3.3. Use the slice features as nodes of the slice map, and associate each slice with its surrounding slices according to the saved position coordinates to construct the slice map.

[0015] Step 4: Construct a multi-level cell map and slice map feature fusion model, including a cell map convolutional module (cell-GNN), a slice map convolutional module (patch-GNN), and a gated attention-based readout module:

[0016] s4.1. The cell graph constructed in step 2 is fed into the cell graph convolution module. The feature representations of the neighborhood-aware fusion are extracted through the graph convolution layer, and then the multi-level embedding features of each node in the cell graph are aggregated through a bidirectional LSTM. The graph convolution layer selected is GENConv.

[0017]

[0018]

[0019]

[0020] Where, φ (l) This is the message constructor for the l-th layer, used to calculate the characteristics of the current node. Features of its neighboring nodes Messages between ρ (l) This is the message aggregation function for the l-th layer, used to aggregate all messages delivered to the current node v. (l) It is the update function of the l-th layer, using aggregated messages. Update the existing node features of the current node v. l = 1, 2, ..., L, and N(v) represents the set of neighboring nodes of the current node. ε is a positive constant that maintains numerical stability, and β is the inverse temperature hyperparameter in softmax.

[0021] Subsequently, the output features of each layer of the graph convolutional layer are sequentially input into the bidirectional LSTM as information with sequential temporal correlation, for each node v in each layer. Obtain its forward and backward hidden embeddings respectively. Then and The importance scores are obtained by concatenating the components and applying a linear mapping function. Finally, the feature representation of node v is obtained.

[0022] s4.2 Based on the relative spatial distribution relationship between cell nuclei and slices, a binary allocation matrix A is used. CG→PG By capturing the interlayer topology, the cell map is combined with the slice map. For the v-th cell nucleus and the w-th slice, the value of the binary assignment matrix is:

[0023]

[0024] The method for fusing cell image features and slice image features is as follows:

[0025]

[0026] H PG h represents the feature matrix corresponding to the slice image. CG M(w) represents the feature vector corresponding to the cell diagram, where M(w) = {v∈V} CG |A CG→PG (v,w)=1} represents the set of cell nuclei belonging to the w-th slice. CONCAT represents the splicing operation.

[0027] s4.3, Features of the slice image after fusing s4.2 The features h of each slice graph node w are obtained by sequentially inputting the graph into a convolutional layer and a bidirectional LSTM. wThen, through a gated attention layer, the attention score 'a' of each node is obtained. w :

[0028]

[0029] Where T denotes matrix transpose, θ, U, and V are trainable parameters, ⊙ denotes element-wise multiplication, sigm() represents the sigmoid activation function, and tanh() represents the tanh activation function. The pathological map-level features are ultimately represented as...

[0030] Step 5: Use pathological map-level features z, which integrates cellular and slice-level features, to perform survival prediction:

[0031] s5.1. The feature vector z is regressed through a fully connected layer to construct a Cox proportional hazards regression model. Survival analysis is performed on the pathological image, and the prediction result h(t,z) is output. The hazard regression model is as follows:

[0032] h(t,z)=h0(t)e λz

[0033] Where λ is a trainable parameter, h0(t) represents the baseline risk function, e represents the natural constant, and t represents the predicted survival time.

[0034] s5.2. Minimize the negative log-likelihood function to train the model parameters, setting the loss function as:

[0035]

[0036] Among them, z i Let t be the feature value of the i-th pathological image. i For the corresponding survival time, the binary variable c i Used to determine whether the i-th observation event is right-censored, h(t) i ,z i S(t) is the predicted risk function. i ,z i ) is the survival function, D train This represents the training set.

[0037] s5.3 Input the preprocessed pathological images into the trained model, fuse cell-level and slice-level features, and analyze the survival results.

[0038] The present invention has the following beneficial effects:

[0039] This module generates cell maps and slice maps for the same pathological image, and uses a graph neural network to better represent the topological structure of the histopathological image. It considers the spatial connectivity between cell maps and slice maps, explores the multi-level relationships within the entire field of view of the pathological image, fuses information at different scales, and realizes the interaction between example information and important morphological features of cell types, learning coarse-grained and fine-grained structural relationships in the microenvironment. This module makes the feature vectors learned by the network more robust, outperforming current research methods that only use single-level features. Furthermore, it can be generalized to survival prediction and classification tasks for various types of pathological images, demonstrating universality. Attached Figure Description

[0040] Figure 1 The flowchart shows a survival analysis method for pathological images based on hierarchical graph convolution and attention.

[0041] Figure 2 This is a schematic diagram of the multi-level cell map and slice map feature fusion model structure in the embodiment;

[0042] Figure 3 This is a schematic diagram of the cell graph convolution module structure in the embodiment. Detailed Implementation

[0043] The present invention will be further explained below with reference to the accompanying drawings;

[0044] like Figure 1 As shown, a survival analysis method for pathological images based on hierarchical graph convolution and attention is proposed. The dataset is collected and divided into five equal parts for 5-fold cross-validation. During the model training phase, the Adam optimization strategy is adopted, with a default learning rate of 2×10⁻⁶. -4 The weight decays to 1×10 -5 The specific steps are as follows:

[0045] Step 1: Collect 515 full-view digital images of lung adenocarcinoma (LUAD) from the TCGA (Cancer Genome Atlas) public dataset and preprocess them. Perform color normalization on all pathological images to eliminate color differences in pathological images caused by different staining agent ratios, staining and scanning factors. Then, use OpenCV to automatically segment the tissue regions of all full-view digital pathological images and filter out blank backgrounds and natural cavities.

[0046] Step 2: Segment cell nuclei from the preprocessed pathological image and extract features to construct a cell graph:

[0047] s2.1. Use the pre-trained Hover-Net to locate cell nuclei, and extract 15-dimensional cell nucleus morphology data based on the output binary mask of each cell nucleus, including cell nucleus size, eccentricity, orientation and major axis length, to obtain the 15-dimensional feature vector representation of each cell nucleus.

[0048] s2.2 Select the cell nuclei to be retained by farthest point sampling (FPS) and set the sampling ratio a = 0.35.

[0049] s2.3. Using the sampled cell nuclei as nodes in the cell graph and the potential interactions between two cell nuclei as edges in the cell graph, adjacent cell nuclei are found and connected according to a set Euclidean distance threshold to construct the cell graph G. CG ={V CG E CG H CG}, where V CG E represents a node in a cell diagram. CG H represents the edges connecting cell nodes. CG This represents the feature matrix of cell nodes.

[0050] Step 3: Divide the preprocessed pathological images into sections and construct patch-graphs:

[0051] s3.1. Use a sliding window to segment the preprocessed pathological image into K non-overlapping slices of size 512×512, and save the position coordinates (x,y) of the slices in the original pathological image.

[0052] s3.2. Input the slices obtained from s3.1 into the pre-trained ResNet-50 model for feature extraction. Perform spatial average pooling on the output of the third residual block of the ResNet-50 model to extract 1024-dimensional feature vectors, and then pack them into the slice feature matrix H. PG .

[0053] s3.3. Using the slice features as nodes in the slice map, each slice is associated with its surrounding slices based on its saved position coordinates (x, y), thus constructing the slice map G. PG ={V PG E PG H PG}, where V PG Represents a node in a slice graph, and E represents the edge between slices. PG .

[0054] Step 4: Construct a multi-level cell map and slice map feature fusion model, such as... Figure 2As shown, it includes a cell-GNN convolutional module, a patch-GNN convolutional module, and a gated attention-based readout module:

[0055] s4.1. The cell graph constructed in step 2 is fed into the cell graph convolution module. The feature representations of the neighborhood-aware fusion are extracted through the graph convolution layer, and then the multi-level embedding features of each node in the cell graph are aggregated through a bidirectional LSTM. The graph convolution layer selected is GENConv.

[0056]

[0057]

[0058]

[0059] Where, φ (l) This is the message constructor for the l-th layer, used to calculate the characteristics of the current node. Features of its neighboring nodes Messages between ρ (l) This is the message aggregation function for the l-th layer, used to aggregate all messages delivered to the current node v. (l) It is the update function of the l-th layer, using aggregated messages. Update the existing node characteristics of the current node v. l = 1, 2, ..., L, where N(v) represents the set of neighboring nodes of the current node. ε is a positive constant to maintain numerical stability, set to 10. -7 β is the inverse temperature hyperparameter in softmax, which is set to 1.

[0060] Subsequently, the output features of each layer of the graph convolutional layer are sequentially input into the bidirectional LSTM as information with sequential temporal correlation, for each node v in each layer. Obtain its forward and backward hidden embeddings respectively. Then and The importance scores are obtained by concatenating the components and applying a linear mapping function. Finally, the feature representation of node v is obtained.

[0061] s4.2 Based on the relative spatial distribution relationship between cell nuclei and slices, a binary allocation matrix A is used. CG→PG By capturing the interlayer topology, the cell map is combined with the slice map. For the v-th cell nucleus and the w-th slice, the value of the binary assignment matrix is:

[0062]

[0063] The method for fusing cell image features and slice image features is as follows:

[0064]

[0065] H PG h represents the feature matrix corresponding to the slice image. CG M(w) represents the feature vector corresponding to the cell diagram, where M(w) = {v∈V} CG |A CG→PG (v,w)=1} represents the set of cell nuclei belonging to the w-th slice. CONCAT represents the splicing operation.

[0066] s4.3, Features of the slice image after fusing s4.2 The features h of each slice graph node w are obtained by sequentially inputting the graph into a convolutional layer and a bidirectional LSTM. w Then, through a gated attention layer, the attention score 'a' of each node is obtained. w :

[0067]

[0068] Where T denotes matrix transpose, θ, U, and V are trainable parameters, ⊙ denotes element-wise multiplication, sigm() represents the sigmoid activation function, and tanh() represents the tanh activation function. The pathological map-level features are ultimately represented as...

[0069] Step 5: Use pathological map-level features z, which integrates cellular and slice-level features, to perform survival prediction:

[0070] s5.1. The feature vector z is regressed through a fully connected layer to construct a Cox proportional hazards regression model. Survival analysis is performed on the pathological image, and the prediction result h(t,z) is output. The hazard regression model is as follows:

[0071] h(t,z)=h0(t)e λz

[0072] Where λ is a trainable parameter, h0(t) represents the baseline risk function, e represents the natural constant, and t represents the predicted survival time.

[0073] s5.2. Minimize the negative log-likelihood function to train the model parameters, setting the loss function as:

[0074]

[0075] Among them, z i Let t be the feature value of the i-th pathological image. i For the corresponding survival time, the binary variable c i Used to determine whether the i-th observation event is right-censored, h(t)i ,z i S(t) is the predicted risk function. i ,z i ) is the survival function, D train This represents the training set.

[0076] s5.3 Use c-index as the evaluation metric for the model, and determine the model prediction results by comparing the predicted survival probability with the observed event time ranking.

Claims

1. A survival analysis method for pathological images based on hierarchical graph convolution and attention, characterized by: Specifically, the following steps are included: Step 1: Collect full-view digital pathology images and preprocess them, save the corresponding survival times, and construct a training set; Step 2: Segment cell nuclei from the preprocessed pathological images and extract their features to construct a cell map; Step 3: Divide the preprocessed pathological image into K non-overlapping slices of the same size, extract the slice image features, and retain the position coordinates of each slice in the original pathological image. Construct a slice map based on the positional relationship of the slices in the original pathological image. Step 4: Input the cell graph constructed in Step 2 into the graph convolutional layer, input the output features of each layer into the bidirectional LSTM in sequence, obtain the forward and backward hidden embeddings of each node in the l-th layer, concatenate them and obtain the importance score through the linear mapping function to obtain the feature representation of the node; Step 5: Based on the relative spatial distribution relationship between cell nuclei and slices, the cell map features and the corresponding slice map features are concatenated and fused. Then, the fused slice map features are sequentially input into the graph convolutional layer and the bidirectional LSTM to obtain the feature representation h of each slice map node w. w Then, through a gated attention layer, the attention score of each node is obtained. : Where T denotes matrix transpose, θ, U, and V are trainable parameters, ⊙ denotes element-wise multiplication, sigm() denotes the sigmoid activation function, and tanh() denotes the tanh activation function; the resulting pathological map-level features are ultimately represented as... ; Step 6: Regress the feature vector z through a fully connected layer to construct a Cox proportional hazards regression model, perform survival analysis on the pathological image, and output the prediction result h(t,z). The hazard regression model is as follows: Where λ is a trainable parameter, h0(t) represents the baseline risk function, e represents the natural constant, and t represents the predicted survival time; the trainable parameters of the risk regression model and the network are optimized using training set data; the preprocessed pathological images are input into the optimized model, cell-level and slice-level features are fused, and the survival results are analyzed.

2. The pathological image survival analysis method based on hierarchical graph convolution and attention as described in claim 1, characterized in that: In step one, the collected pathological images are color normalized to eliminate color differences in pathological images caused by different staining agent ratios, staining and scanning factors. Then, OpenCV is used to automatically segment the tissue regions of all full-view digital pathological images, filtering out blank backgrounds and natural cavities.

3. The pathological image survival analysis method based on hierarchical graph convolution and attention as described in claim 1, characterized in that: The pre-trained Hover-Net was used to segment cell nuclei from pathological images, and the morphological data of the cell nuclei were used as their feature vector representation.

4. The pathological image survival analysis method based on hierarchical graph convolution and attention as described in claim 1 or 3, characterized in that: The extracted cell nuclei were sampled at their furthest point, with a sampling ratio of... The sampled cell nuclei are used as nodes in the cell graph, and the potential interactions between two cell nuclei are used as edges in the cell graph. Adjacent cell nuclei are found based on a set Euclidean distance threshold to construct the cell graph. ,in Nodes representing the cell diagram, The edges represent the connections between cell nodes. This represents the feature matrix of cell nodes.

5. The pathological image survival analysis method based on hierarchical graph convolution and attention as described in claim 1, characterized in that: The segmented slices are sequentially input into a pre-trained ResNet-50 model for feature extraction. Spatial average pooling is performed on the output of the third residual block of the ResNet-50 model to obtain a 1024-dimensional feature vector, which serves as the slice image feature. Each slice is then used as a node in the slice map, and each slice is associated with its surrounding slices based on its saved position coordinates (x, y) to construct the slice map. ,in Represents a slice graph node. Indicates the edges between slices. This represents the feature matrix of a slice.

6. The pathological image survival analysis method based on hierarchical graph convolution and attention as described in claim 1, characterized in that: The graph convolutional layer is GENConv: in, This is the message constructor for the l-th layer, used to calculate the characteristics of the current node. Features of its neighboring nodes messages between ; This is the message aggregation function for the l-th layer, used to aggregate all messages passed to the current node v; It is the update function of the l-th layer, using aggregated messages. Update the existing node characteristics of the current node v; l=1,2,…L, N(v) represents the set of neighboring nodes of the current node; It is a positive constant that maintains numerical stability, and is set as . , This is the inverse temperature hyperparameter in softmax, set to 1; Bidirectional LSTM uses the output features of each layer of the graph convolutional layer as input data, and for each node v, the features at each layer... Obtain its forward and backward hidden embeddings respectively. , Then The importance scores are obtained by concatenating the components and applying a linear mapping function. Finally, the feature representation of node v is obtained. .

7. The pathological image survival analysis method based on hierarchical graph convolution and attention as described in claim 1, characterized in that: We performed 5x cross-validation using the data collected in step one. During the parameter optimization phase, we adopted the Adam optimization strategy with a default learning rate of [missing information]. The weight decays to .

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 3, 5 to 7.

Images