A pathological image processing method, model and device

Abstract

The application discloses a pathological image processing method, constructs a cell graph in a received pathological image, the cell graph contains cell structure information; inputs the cell graph into a trained pathological image processing model to obtain a fusion feature H O for predicting expression of a gene marker. The pathological image processing model fuses a graph convolutional neural network (GNN) model and a convolutional neural network (CNN) model, the GNN model is used for extracting a low-dimensional feature H G from the cell graph, the CNN model is used for extracting a low-dimensional feature H I from the pathological image, the features H G and H I are spliced after a fusion layer to obtain a fusion feature H O .

Description

Technical Field

[0001] This invention belongs to the fields of biology and pathology, and specifically relates to a method, model, and device for processing pathological images. Background Technology

[0002] In biology and pathology, structural information in cell morphology, such as cell aggregation and intercellular distance, reflects the expression of gene markers (e.g., microsatellite instability in colorectal cancer). However, there is little discussion on how to utilize this characteristic for gene marker prediction. Summary of the Invention

[0003] One embodiment of the present invention is a pathological image processing method, which constructs a cell map in a received pathological image, the cell map containing cell structure information;

[0004] The cell map is input into a trained pathological image processing model to obtain fusion features HO for predicting gene marker expression.

[0005] This invention explicitly extracts cellular structure information from pathological images in a graph-based manner, and utilizes a graph convolutional neural network to extract geometric information features from the graph. These features are then fused with the features extracted from the pathological images by the convolutional neural network to predict the expression of gene markers.

[0006] This invention proposes a fusion model of graph convolution and convolutional neural networks for predicting gene markers in pathological images. Significant improvements were achieved on three pathological datasets: colorectal cancer pathological image prediction of microsatellite instability, gastric cancer prediction of microsatellite instability (MSI), and gastric cancer prediction of programmed cell death-ligand 1 (PD-L1) expression. The classification accuracy was significantly improved compared to pathological image processing models using only convolutional neural networks. Attached Figure Description

[0007] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0008] Figure 1 A schematic diagram of a pathological image processing model according to one embodiment of the present invention. Detailed Implementation

[0009] The problem this invention aims to solve is how to extract cellular structural information from histopathological images using graph convolutional neural networks to assist convolutional neural networks in predicting gene markers.

[0010] According to one or more embodiments, a pathological image processing method includes two stages: the first stage extracts structured graph information from the pathological image, and the second stage provides a graph and image fusion model (i.e., a fusion model of graph convolution and convolutional neural networks).

[0011] The first stage involves constructing a cell map from pathological images: a pre-trained nuclear segmentation neural network (such as UNet, TransUNet, etc.) is used to segment nuclear regions from the tissue pathology images. Each segmented nuclear region constitutes a node in the graph, denoted as v. j , where the subscript j is a number for the cell nucleus.

[0012] Next, the weights of the edges between graph nodes are calculated based on the distance between the centroids of the cell nucleus regions. Graph node v j and v i The weight w of the edge between ij The calculation formula is as follows:

[0013]

[0014] Wherein d(v i v j ) represents the cell nucleus v j and v i The distance between the centers of gravity, d c This represents the critical distance, determining the maximum effective range of cell distance. It's a hyperparameter determined by the magnification of the pathological image. The distance is expressed in pixels and may need to be converted based on the image magnification. For example, when the actual physical distance is 50 micrometers, the conversion depends on the magnification. At 20x magnification (0.5 micrometers / pixel), dc is taken as 40 pixels.

[0015] When the distance between the cell centroids is less than d c The weight is Otherwise, the weight is 0, meaning there is no edge connecting these two nodes. Each graph node v j The node features are determined by predefined radios features of this cell nucleus region, depending on the needs of downstream tasks to adjust the radios features. At this stage, the constructed graph characterizes the morphological geometry between cells in the pathological image, serving as input to the graph convolutional neural network in the next stage.

[0016] In the second stage, a fusion model of Graph Convolutional Neural Networks (GNN) and Convolutional Neural Networks (CNN) is implemented. First, a... GNN model(The model structure is not limited, such as but not limited to GCN, GIN) to extract a low-dimensional feature H that characterizes the geometry of the pathological image from the cell map constructed in the first stage. G Here, the subscript G indicates that the feature describes the geometric information of the pathological image. That is, the GNN model takes the cell image as input and obtains a one-dimensional feature output H after graph convolution operation. G At the same time, train a CNN model (The model structure is not limited, such as but not limited to DenseNet and ResNet) used to extract low-dimensional features H at the image level from pathological images. I The subscript I indicates that this feature is portrayal Image layer information That is, the CNN model takes pathological images as input, and obtains a one-dimensional feature output H after convolution operation. I .

[0017] Because H G and H I The cell images are extracted from pathological images, and the pathological images themselves are extracted using GNN and CNN respectively, thus depicting different features and possessing complementarity. Therefore, the two low-dimensional features are first concatenated, and then a fusion layer is used to further refine the concatenated features. Integrating predictions for downstream gene expression tasks The fusion layer incorporates two strategies: a learnable MLP layer or a learnable Transformer layer. H is recorded. G and H I The fusion feature obtained after splicing is H O The subscript 'O' indicates that this feature is a fused feature. The overall model structure is as follows: Figure 1 As shown.

[0018] MLP-based fusion layer model: The MLP fusion strategy employs iterations of multiple MLP units MLPBlock to obtain the fusion feature H. O Expression:

[0019] H O =MLPBlock(...(MLPBlock(H C (2)

[0020] Among them, H C For H G and H I The characteristics after linking, i.e., H C =concat[H I H G ], concat is a join operation.

[0021] The expression for MLPBlock is as follows:

[0022] MLPBlock(H)=Dropout(ReLU(Linear(H))) (3)

[0023] Here, Linear is a learnable linear layer, ReLU is the ReLU activation function, and Dropout is the Dropout regularization layer.

[0024] Transformer-based fusion layer model: The fusion strategy based on Transformer is expressed as follows:

[0025] H O =Pooling(TransBlock(...(TransBlock(H C (4)

[0026] Among them, H C For H G and H I The characteristics after linking, i.e., H C =concat[H I H G ], concat is a join operation.

[0027] Pooling refers to the pooling layer, and TransBlock is the basic unit of transformer, expressed as follows:

[0028] TransBlock(H)=ResidualPreNorm(MLPBlock, ResidualPreNorm(MHSA,H)) (5)

[0029] MHSA is a multi-headed self-attention mechanism layer, and ResidualPreNorm is a residual pre-layerNorm layer. MLPBlock is defined as described above.

[0030] Experimental comparison results of the present invention.

[0031] This experiment utilized a subset of the publicly available TCGA pathological image dataset to construct microsatellite instability (negative vs. positive) predictions for colorectal cancer pathological images, gastric cancer microsatellite instability (negative vs. positive) predictions, and privately collected gastric cancer data to predict PDL1 expression. The classification accuracy of this invention is more than 6.0% higher than that achieved using only convolutional neural networks (including ResNet, DenseNet, and MobileNetV3) in terms of AUC.

[0032] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, or they may be electrical, mechanical, or other forms of connection.

[0033] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0034] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of processing a pathological image, characterized by, The method comprises the steps of: constructing a cell graph in the received pathological image, the cell graph containing cell structure information; inputting the cell image into a trained pathological image processing model to obtain a fusion feature for predicting expression of a gene marker; The method for constructing the cell graph comprises: The cell nucleus region is segmented from the pathological image by using a pre-trained cell nucleus segmentation model, and each segmented cell nucleus region constitutes a node of the cell graph, denoted as wherein subscript j is a number of a cell nucleus in a pathological image. The weights of edges between graph nodes are calculated based on the distance between the centroids of the cell nucleus regions. For any two nodes in the graph... and Weight of the edges between The calculation formula (1) is as follows: （1） wherein, represents the nucleus of the cell and the 2-dimensional Euclidean distance between the centers of gravity, represents the critical distance, which determines the maximum cell distance of action, and is determined by the specific data set and the magnification of the pathological image, When the distance between the centers of the cell nuclei is less than , the weight is ; otherwise, the weight is 0, i.e. there is no edge between these two nodes. The pathological image processing model fuses a graph convolutional neural network GNN model and a convolutional neural network CNN model, The GNN model is used to extract low-dimensional features from the cell map , a CNN model is used to extract low-dimensional features at the image level from the pathology images , Features and After splicing through a fusion layer, the fusion features are .

2. The method of pathological image processing according to claim 1, characterized in that, The fusion layer is based on an MLP fusion strategy, adopts iteration of multiple MLP units MLPBlock, and obtains fusion features expression of the fusion protein = MLPBlock(...(MLPBlock( ))) (2) wherein is and the concatenated features, i.e. =concat[ , ], concat being a concatenation operation, The expression of MLPBlock is as follows, MLPBlock(H) = Dropout(ReLU(Linear(H))) (3) where Linear is a learnable linear layer, ReLU is a ReLU activation function, and Dropout is a Dropout regularization layer.

3. The method of pathological image processing according to claim 1, characterized in that, The fusion layer is based on a Transformer fusion strategy, and the expression of the Transformer fusion strategy is as follows, = Pooling(TransBlock(…(TransBlock( )))) (4) wherein, is and linked features, i.e. = concat[ , ], concat is a concatenation operation, Pooling is a pooling layer, and TransBlock is a basic unit of a transformer, which is expressed as follows: TransBlock(H) = ResidualPreNorm(MLPBlock, ResidualPreNorm(MHSA, H)) (5) MHSA is a multi-headed self-attention mechanism layer, and ResidualPreNorm is a residual pre-Layer Norm layer, that is, the Layer Norm layer is placed in advance.

4. A pathological image processing model, characterized in that, The model inputs a cell map, obtains fusion features as for predicting expression of gene markers; The cell graph is constructed in the received pathological image, and the cell graph contains cell structure information; the method for constructing the cell graph comprises: The cell nucleus region is segmented from the pathological image by using a pre-trained cell nucleus segmentation model, and each segmented cell nucleus region constitutes a node of the cell graph, denoted as wherein subscript j is a number of a cell nucleus in a pathological image. The weights of edges between graph nodes are calculated based on the distance between the centroids of the cell nucleus regions. For any two nodes in the graph... and Weight of the edges between The calculation formula (1) is as follows: （1） where, represents the nucleus of the cell and the 2-dimensional Euclidean distance between the centers of gravity, represents the critical distance, which determines the maximum cell distance of action, and is determined by the specific data set and the magnification of the pathological image, When the distance between the centers of the cell nuclei is less than , the weight is ; otherwise the weight is 0, i.e. there is no edge linking the two nodes. The pathological image processing model fuses a graph convolutional neural network GNN model and a convolutional neural network CNN model, The GNN model is used to extract low-dimensional features from the cell map , a CNN model is used to extract low-dimensional features at the image level from the pathology images , Features and After splicing through a fusion layer, the fusion features are .

5. A pathological image processing apparatus characterized by comprising: The device comprises a memory; and A processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method of any one of claims 1 to 3.

6. A storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1 to 3.

Images