Pathological section curative effect prediction method based on graph convolutional network

Abstract

The invention discloses a pathological section curative effect prediction method based on a graph convolutional network, and the method comprises the following steps: obtaining a pathological section image, and carrying out the preprocessing of a pathological section; identifying and segmenting cells in the pathological section image to generate a segmentation mask of each cell; classifying the segmentation mask of each cell, identifying the type of each cell, and then extracting the feature corresponding to each cell; defining nodes and edges of the graph according to the cells and the features, then obtaining graph data of the graph as input of a graph convolutional network, and outputting a curative effect result of the pathological section; the graph data includes a node feature matrix and an adjacency matrix. The pathological treatment effect is modeled based on the graph convolution network of the cell granularity, and the spatial relationship and context information between pathological section images can be effectively captured by constructing the adjacent matrix and executing the graph convolution operation, so that the defects in the pathological feature fusion process are made up.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing and relates to a method for predicting the therapeutic effect of pathological sections based on a graph convolutional network. Background Art

[0002] In the field of medical pathology, pathological feature fusion technology has received widespread attention in recent years. With the rapid development of medical pathology and pathology, combining pathological features with pathological features has become an important method to improve the accuracy of disease diagnosis and prediction. Precision medicine represented by immunotherapy has shown great potential in the pan-cancer field, but the existing models are still insufficient in analyzing the tumor immune microenvironment based on pathological data to predict the efficacy of immunotherapy, which has become an important bottleneck restricting clinical applications.

[0003] Traditionally, pathology research usually divides tissue sections into small patches (Patch) and then uses the multi-instance learning (MIL) method for feature fusion. Although MIL has achieved some success in pathology image analysis, it has some obvious limitations when dealing with patch fusion, especially in considering the spatial relationship between patches:

[0004] 1. Ignoring spatial context information: In the MIL framework, each patch is treated as an independent entity, and the spatial relationship between them is ignored. This means that even if adjacent patches may contain shared important biological information, MIL cannot utilize this spatial context to improve the accuracy of prediction. This defect directly leads to the following problems in pathological image analysis:

[0005] (1) Incomplete analysis of the immune microenvironment: Key features of the tumor immune microenvironment, such as the pattern of immune cell infiltration and the characteristic distribution of tumor boundary areas, usually rely on spatial information across patches for comprehensive analysis. Existing methods fail to capture these spatial contexts, resulting in limited depth and accuracy of immune microenvironment analysis.

[0006] (2) The prediction results are less robust: the lack of spatial information makes the model unstable when processing pathological images with high heterogeneity, and the prediction results are easily affected by the features of a single patch, thereby reducing the credibility of the prediction of the efficacy of immunotherapy.

[0007] 2. Fixed spatial relationship assumption: In some MIL methods, although spatial information can be indirectly considered by designing specific pooling strategies, this usually requires making fixed assumptions about the spatial relationship between patches. This assumption may not be applicable to all types of pathological images, limiting the versatility and flexibility of the model, and further bringing the following limitations:

[0008] (1) Limited versatility: Pathological images may exhibit different structural characteristics under different disease types, tissue locations, or staining conditions. Fixed spatial relationship assumptions cannot adapt to these changes, causing the model to show significant performance degradation when applied across scenarios.

[0009] (2) Insufficient expressive power of spatial features: Fixed assumptions limit the model’s ability to capture complex spatial relationships, such as the nonlinear interactions between tumor cells and immune cells. As a result, the model exhibits low explanatory power when predicting complex biological phenomena (such as tumor immune escape mechanisms).

[0010] In summary, a new method is urgently needed to realize pathological image analysis and make up for the above defects. Summary of the invention

[0011] The first objective of the present invention is to address the deficiencies in the prior art and to provide a method for predicting the therapeutic efficacy of pathological sections based on a graph convolutional network. By constructing an adjacency matrix and performing graph convolution operations, the spatial relationship and contextual information between patches can be effectively captured, thereby making up for the deficiencies in the pathological feature fusion process.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions: Step S1, obtaining a pathological slice image and preprocessing the pathological slice; Step S2: using the Mask R-CNN model to identify and segment cells in the pathological slice image, and generate a segmentation mask for each cell; Step S3: Use the DenseNet121 model to classify the segmentation mask of each cell and identify its cell type ; At the same time, the DenseNet121 model is used to extract the features corresponding to each cell ; Step S4: define each cell as a node of the graph; for any two nodes of the same type, calculate the Euclidean distance between their features; define a threshold value, and , if the Euclidean distance between its features is less than the threshold, then and There is an edge between them, denoted by ; Traverse all nodes and build a graph ;in Represents a collection of nodes, represents a set of edges; Step S5: Get image The graph data is used as the input of the graph convolutional network (GCN), and the graph convolutional network (GCN) predicts and outputs the efficacy results of the pathological slices; the graph data includes a node feature matrix and an adjacency matrix.

[0013] The second object of the present invention is to provide a pathological section efficacy prediction system based on graph convolutional network for implementing the above method, comprising the following modules: An image preprocessing module, used to obtain and preprocess pathological slice images; The cell recognition and classification module is used to perform cell recognition and classification on the preprocessed pathological slice images to obtain the type and characteristics of each cell; A graph construction module, which constructs a graph including a node feature matrix and an adjacency matrix according to the type and features of each cell; The efficacy prediction module uses graph convolutional networks to model and train graph data and output efficacy prediction results.

[0014] A third object of the present invention is to provide an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above method.

[0015] A fourth object of the present invention is to provide a machine-readable storage medium storing machine-executable instructions, which, when called and executed by a processor, prompt the processor to implement the above method.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses cells as nodes of the graph and uses graph convolutional networks (GCN) to model at the cell granularity, fully considering the spatial contextual relationship between cells, and can effectively extract the deep learning features of cells. The present invention separates cells from pathological images through instance segmentation. Through the edge and point relationship of the graph structure, the present invention not only enhances the expression of cell characteristics, but also improves the understanding of the relationship between cells, and finally achieves accurate modeling of pathological efficacy, which can identify cell types and make accurate prognosis predictions. The method of the present invention combines the advantages of deep learning and graph convolutional networks, can better capture the complex spatial and type characteristics between cells in pathological images, and has high accuracy and interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with embodiments and drawings.

[0019] like Figure 1 As shown in the figure, a method for predicting the therapeutic effect of pathological sections based on a graph convolutional network is provided in the present invention, comprising the following steps: Before applying the present invention, the pathological slice (Whole Slide Image, WSI) needs to be cut into small pieces (patches), and then the Macenko method is used for staining standardization to eliminate the color deviation caused by different scanning equipment and staining techniques to ensure the consistency of image color.

[0020] (1) Cell recognition based on Mask RCNN This paper selects a target detection and segmentation model based on Mask R-CNN (Region-based Convolutional Neural Networks), which can simultaneously identify objects in an image and generate corresponding pixel-level segmentation masks. The specific process is as follows:

[0021] a) Model initialization 1. Backbone network selection: ResNet-101 was selected as the feature extractor of Mask R-CNN for its strong representation ability to capture complex pathological features.

[0022] 2. Parameter initialization: The present invention uses ImageNet pre-trained weights to initialize the backbone network of the model, which can accelerate convergence and improve the initial performance of the model.

[0023] 3. RPN (Region Proposal Network): Mask R-CNN comes with an RPN module for generating candidate regions. The present invention adjusts the anchor size and aspect ratio in RPN to adapt to the specific cell morphology in pathological images.

[0024] b) Hyperparameter selection During the training process of Mask R-CNN, the following hyperparameters are adjusted according to the actual situation: 1. Learning rate: The initial learning rate is set to 0.001, and a learning rate scheduling strategy with cosine annealing is used to balance training speed and convergence.

[0025] 2. Batch size: The initial batch size is set to 2 and adjusted accordingly depending on the GPU memory capacity.

[0026] 3. Optimizer: Adopt Adam optimizer, which is suitable for processing sparse data and has strong adaptability.

[0027] 4. Positive-negative sample ratio: In the RPN stage, the present invention sets the ratio of positive and negative samples to 1:3 to ensure more accurate candidate region generation for cell regions.

[0028] c) Model training 1. Multi-GPU training: In order to accelerate model training, the present invention will use multi-GPU parallel computing. If the amount of data is large, the present invention can consider using mixed precision training to reduce memory usage.

[0029] 2. Loss function: The loss function consists of classification loss, bounding box regression loss, and mask segmentation loss. These losses will be balanced in a weighted manner, and the specific weight parameters will be adjusted through experiments.

[0030] 3. Model training strategy: Use the 5-fold cross validation method to ensure the robustness and generalization of the model. At the same time, add an early stopping mechanism during the training process to avoid overfitting.

[0031] (2) Densenet121 cell type identification When using the Densenet121 model for cell type identification, the following operations were performed to improve the performance and accuracy of the model: a) Model initialization: The Densenet121 pre-trained on ImageNet is used as the basic model, which inherits the feature extraction capabilities of large-scale natural image datasets. This helps the model to converge and optimize quickly on pathological image data.

[0032] During the initialization stage of the model weights, a transfer learning strategy that matches the pathological data will be adopted to smoothly transition to the learning process of specific tasks to ensure the stability and adaptability of the model's initial state.

[0033] b) Loss function and optimizer: The present invention uses the cross entropy loss function for the multi-classification task of cell types to measure the difference between the predicted results and the actual class labels. The cross entropy loss function can effectively handle the problem of unbalanced data distribution.

[0034] The Adam optimizer is used. This optimizer introduces a weight decay mechanism based on Adam to further improve the generalization ability of the model and can effectively control the oscillation phenomenon in weight updating, thereby improving the stability of training.

[0035] c) Learning rate adjustment strategy: In order to dynamically adjust the learning rate, the cosine annealing learning rate strategy is used. The initial learning rate is set to 0.001, and the learning rate gradually decreases as the training progresses to avoid the model falling into the local optimum.

[0036] The formula for how the learning rate changes with the training cycle is: in, Set to 0, Set to 0.001, is the current training cycle number, is the total training cycle. This strategy can smoothly adjust the learning rate and improve the fine-tuning accuracy of the model at the end of training.

[0037] (3) Cell types and cell characteristics Through the previous Mask RCNN and DenseNet121 models, the present invention has identified the cell type for any patch. , At the same time, for each cell, we can use the penultimate layer of the DenseNet121 model to extract its deep learning features, which are recorded here as , where i is all the cells in the current patch that have been recognized by the MaskRCNN model.

[0038] (4) GCN graph structure construction In the graph definition of the present invention, each pathological cell is regarded as a node in the graph. The connection relationship between nodes is represented by edges. Specifically, the present invention can formally define the graph construction process as follows:

[0039] Node & edge definitions: set up A node set representing pathological cells, where is the total number of pathological cells, Indicates nodes for all cells in a pathological patch.

[0040] Edges between pathological cells: For any two cell nodes with the same type and , the present invention calculates the Euclidean distance between its features; defines a threshold (usually a hypermodel parameter, generally greater than 0.6 or 0.7), for any and , if the Euclidean distance between their features is less than the threshold, there is an edge between them, recorded as .

[0041] The graph shows: Based on the above definitions of nodes and edges, the present invention can construct a graph ,in is a collection of nodes, It is an edge set, an edge set This includes the edges between all pathological cells and between the imaging sub-regions and pathological cells.

[0042] (5) GCN model predicts prognosis Figure data preparation: According to the above graph construction method, prepare graph data, including node feature matrix (in is the total number of nodes, is the feature dimension) and the adjacency matrix . Adjacency Matrix Represents the connection relationship between nodes, that is: if the node and nodes If there is an edge between ,otherwise .

[0043] Model initialization: Initialize the parameters of the GCN model, including the weight matrices of each graph convolutional layer. These parameters can be initialized randomly or using pre-trained weights.

[0044] Loss calculation and optimization: Based on the output of the GCN model and the true label, the loss function is calculated. The loss function can be cross entropy loss for classification tasks or mean square error loss for regression tasks.

[0045] The gradient of the loss function with respect to the model parameters is calculated using the back-propagation algorithm, and an optimizer such as Adam is used to update the model parameters to minimize the loss.

[0046] Learning Rate: At the same time, in order to better generalize, the present invention carefully sets the learning rate. The present invention adopts the cosine decay learning rate algorithm. The learning rate of the present invention is set as follows:

[0047] in, , , Represent the minimum learning rate, maximum learning rate and number of iterations respectively.

[0048] The above embodiments are the basic principles, main features and advantages of the present invention. Moreover, the present invention only illustrates the technical aspects and does not limit the protection scope of the present invention. Some improvements made according to the above embodiments are within the protection scope of the present invention.

Claims

1. A method for predicting the therapeutic effect of pathological sections based on graph convolutional networks, characterized in that: The method comprises the following steps: Step S1, obtaining a pathological slice image and preprocessing the pathological slice; Step S2, identifying and segmenting cells in the pathological slice image, and generating a segmentation mask for each cell; Step S3, classify the segmentation mask of each cell, identify the type of each cell, and then extract the features corresponding to each cell; Step S4: define each cell as a node of the graph; for any two nodes of the same type, calculate the Euclidean distance between their features, and then determine whether there is an edge between the nodes based on the Euclidean distance; traverse all nodes to construct a graph ;in Represents a collection of nodes, represents a set of edges; Step S5: Get image The graph data is used as the input of the graph convolutional network GCN, and the graph convolutional network GCN predicts and outputs the therapeutic effect results of the pathological slices; the graph data includes a node feature matrix and an adjacency matrix.

2. The method for predicting the therapeutic effect of pathological sections based on graph convolutional networks according to claim 1, characterized in that: The preprocessing in step S1 includes: dividing the pathological section image into small blocks, and then using Macenko staining to perform standardization processing on the small blocks.

3. The method for predicting the therapeutic effect of pathological sections based on graph convolutional networks according to claim 1, characterized in that: The Mask R-CNN model is used in step S2.

4. The method for predicting the therapeutic effect of pathological sections based on graph convolutional networks according to claim 1, characterized in that: The DenseNet121 model is used in step S3.

5. The method for predicting the therapeutic effect of pathological sections based on graph convolutional networks according to claim 1, characterized in that: In step S4: set up A node set representing pathological cells, where is the total number of pathological cells, Indicates Nodes for all cells in a pathological patch; For any two cell nodes of the same type and , calculate the Euclidean distance between its features; define a threshold, for any and , if the Euclidean distance between its features is less than the threshold, then and There is an edge between them, denoted by .

6. The method for predicting the therapeutic effect of pathological sections based on graph convolutional networks according to claim 1, characterized in that: The graph data in step S5 includes a node feature matrix and the adjacency matrix ,in is the total number of nodes, is the characteristic dimension; Adjacency Matrix Represents the connection relationship between nodes, that is: and nodes If there is an edge between ,otherwise .

7. A pathological section efficacy prediction system based on graph convolutional network for implementing the method according to any one of claims 1 to 6, comprising the following modules: An image preprocessing module, used to obtain and preprocess pathological slice images; The cell recognition and classification module is used to perform cell recognition and classification on the preprocessed pathological slice images to obtain the type and characteristics of each cell; A graph construction module, which constructs a graph including a node feature matrix and an adjacency matrix according to the type and features of each cell; The efficacy prediction module uses graph convolutional networks to model and train graph data and output efficacy prediction results.

8. An electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method according to any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method described in any one of claims 1-6.

Images