Pathological image analysis method based on effective metric learning and graph transformer

Abstract

The application discloses a kind of based on effective metric learning and pathological image analysis method of graph transformer, first, the pre-acquired existing pathological image data data is pretreated;Second, a certain quantity of image blocks are collected and input into VGG-16 network using effective metric learning loss function for training, the model obtained by training is used as feature extractor;The features of image block are extracted, and the adjacency matrix of graph is constructed;Then, the deep network based on graph transformer is constructed;Finally, the gradient descent method of back propagation is used to train the model, and finally the pathological image analysis result is output.The deep feature extractor improved by metric learning compared with the traditional method of directly using the feature extractor of ImageNet to extract the features of pathological images can cover the comprehensive features of pathological images, which can not only increase the inter-class difference but also increase the intra-class difference, so better results can be achieved.

Description

Technical Field

[0001] This invention belongs to the fields of digital image analysis, pathology, and machine learning, and specifically relates to a pathological image analysis method based on effective metric learning and graph transformer. Background Technology

[0002] As a highly heterogeneous disease, cancer development is not only due to the unlimited proliferation of tumor cells, but also intrinsically linked to the degree of differentiation and maturation of tumor cells and morphological changes. Pathological images, as important cellular microscopic images, provide crucial information for cancer diagnosis, grading, and prognostic analysis, and are therefore widely used by pathologists to predict clinical outcomes. With the continuous development of whole-slide pathological image digitization technology, more and more researchers are beginning to use computers to analyze pathological images of different types of cancer (such as lung cancer, colon cancer, breast cancer, and brain cancer). Compared with traditional manual slide reading methods, computer-aided pathological image analysis methods can significantly improve the efficiency, accuracy, and consistency of diagnosis and prognostic analysis.

[0003] In recent years, with the continuous development of deep learning technology in the medical field, many researchers have begun to use deep learning methods for quantitative analysis of pathological images. Most of these methods use models pre-trained on ImageNet to extract morphological features from pathological images. For example, Kausar et al. used a pre-trained feature extractor to extract features to detect small-sized mitotic figures. However, pre-trained deep feature extractors do not always achieve good results in pathological tasks because there is an inherent difference between pathological images and natural images; the texture patterns present in natural data may not be applicable to pathological images. How to overcome the gap between pre-trained feature extractors and natural and pathological images has become an important problem that urgently needs to be solved. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a pathological image analysis method based on effective metric learning and graph transformer, which can extract and process the features of pathological images using pathological image data.

[0005] Technical solution: This invention provides a pathological image analysis method based on effective metric learning and graph transformer, comprising the following steps:

[0006] (1) Preprocess the existing pathological image data that has been acquired in advance;

[0007] (2) From the image data obtained in step (1), a certain number of image patches are collected and input into the VGG-16 network with an effective metric learning loss function for training. The trained model is used as a feature extractor. The feature extractor is used to extract the features of the image patches and the adjacency matrix of the graph is constructed using these features.

[0008] (3) Construct a deep network based on graph transformer. The input of this deep neural network is the adjacency matrix generated in step (2), and the loss function is the cross-entropy loss function.

[0009] (4) Based on the loss function in step (3), the gradient descent method of backpropagation is used. With each gradient descent, the loss of the training set will become smaller and smaller. When the loss of the training set reaches the set value or the number of iterations exceeds the preset threshold, the training is completed. The model is applied to the test set to output the pathological image analysis results.

[0010] Furthermore, the implementation process of step (1) is as follows:

[0011] Considering that pathological images contain a large number of non-cellular areas, the background area is first filtered out based on the values ​​of each pixel in the R, G, and B channels. The filtering rule is that if the pixel value in any channel is greater than 200, or if 90% of the pixels in the 200*200 neighborhood of the pixel are background areas, then the pixel is filtered out. The sample is then augmented by rotating and flipping the acquired image blocks to increase the sample size.

[0012] Furthermore, the image block size described in step (2) is 512*512.

[0013] Furthermore, the implementation process of step (2) is as follows:

[0014] Image patches are fed into an efficient metric network based on am-softmax for training. After N rounds of training, a deep feature extractor based on efficient metric learning is obtained. The loss function used in efficient metric learning is as follows:

[0015]

[0016] Where s is the scaling factor, W is the weight of the last fully connected layer, and y i For the corresponding full slice tag;

[0017] For each image patch, find the k image patches with the closest feature Euclidean distance to it and construct edges; the position of each image patch is encoded, and the specific encoding method is as follows:

[0018]

[0019] Where, d model It is the dimension of encoding, which adds the positional encoding of the image patch to the image patch features, and then feeds it into the graph transformer model.

[0020] Furthermore, the graph transformer-based deep network described in step (3) consists of a multi-head attention mechanism, a normalization layer, and a forward propagation layer; the basic component of the multi-head attention mechanism module is the attention module, which is as follows:

[0021]

[0022] Where Q is the consultation vector, K is the key-value vector, and V is the value vector; multi-head attention enables the model to jointly attend to information from different representation subspaces at different locations, as shown below:

[0023] MultiHead(Q,K,V)=Concat(head1,head2,...,head h W O

[0024] in,

[0025] The output of the Transformer is fed into multiple layers of graph convolutions, where each layer is represented as:

[0026] H (l+1) =f(H (l) A (l) )

[0027] x′x=σ(A (l) H (l) W (l) )

[0028] Among them, H l It is the input of layer 1, A (l) This is the corresponding adjacency matrix; before feeding it into the graph convolutional neural network, the adjacency matrix is ​​modified as follows:

[0029]

[0030] in, yes The block diagonal matrix; the output of the entire graph transformer is represented as:

[0031]

[0032] After applying GCN to pool the features of each patch, a two-layer fully connected neural network and the softimax function are used to predict the labels.

[0033] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: 1. It adopts a deep neural network and applies it to the gene mutation prediction task based on pathological images. Compared with the traditional method of directly using the feature extractor of ImageNet to extract features of pathological images, the deep feature extractor improved by metric learning can cover the comprehensive features of pathological images, which can not only increase the inter-class differences but also increase the intra-class differences, thus achieving better results; 2. The data is preprocessed by rotating and flipping the original image, which increases the data sample size. Attached Figure Description

[0034] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0035] The present invention will now be described in further detail with reference to the accompanying drawings.

[0036] This invention provides a pathological image analysis method based on effective metric learning and graph transformer, such as... Figure 1 As shown, it includes the following steps:

[0037] Step 1: Preprocess the existing pathological image data that has been acquired in advance.

[0038] Since the original pathological images have a small sample size, data augmentation is first performed by rotating (90°, 180°, 270°) and flipping the original images to increase the sample size. Considering that there are many non-cellular areas (white areas) in the pathological images, background areas are first filtered out based on the values ​​of each pixel in the R, G, and B channels. The filtering rule is that if the pixel value in any channel is greater than 200, or if 90% of the pixels in the 200*200 neighborhood of the pixel are background areas, then the pixel is filtered out. In addition, considering the color differences between different tissue pathological sections, histogram matching is used to achieve staining normalization.

[0039] Step 2: From the image data obtained in Step 1, collect a certain number of image patches and input them into a VGG-16 network that uses an effective metric learning loss function for training. Use the trained model as a feature extractor. Use the feature extractor to extract features from the image patches and use these features to construct the adjacency matrix of the graph.

[0040] From the image data obtained in step 1, a certain number of image blocks of size 512*512 are collected and the image blocks in the training set are sent to the metric learning module. The feature extractor is improved by using the am-softmax loss function.

[0041] In medicine, a single full-slice digital pathology image often contains tens of millions of pixels, making it impossible to directly train on images of this scale using existing deep neural network-based methods. Without increasing the cost of expert labeling, a feasible approach is to randomly select smaller image patches from the full-slice pathology image to train the network. Directly using a deep feature extractor pre-trained on ImageNet might degrade model performance; therefore, all full slices are divided into training, validation, and test sets. A certain number of image patches are extracted from each full slice. This project assumes that the KRAS gene mutation markers in the image patches are consistent with the KRAS gene mutation markers in the full-slice pathology image. The image patches from the training set are then fed into an am-softmax-based efficient metric network for training. After 100 rounds of training, a deep feature extractor based on efficient metric learning is obtained. The loss function used in efficient metric learning is:

[0042]

[0043] Where s is the scaling factor, W is the weight of the last fully connected layer, and y i This is the corresponding full-slice tag.

[0044] After obtaining a feature extractor based on effective metrics, this feature extractor is used to extract features from image patches belonging to the same full slice. For each full slice, an adjacency matrix is ​​constructed using the nearest neighbor algorithm. Specifically, for each image patch, edges are constructed by finding the k image patches whose features are closest to it via Euclidean distance. For each image patch, its position is encoded in this project, as follows:

[0045]

[0046] Where, d model This refers to the encoding dimension. This project adds the positional encoding of image patches to the image patch features, and then feeds the result into the graph transformer model.

[0047] Step 3: Construct a graph transformer-based deep network. The input to this deep neural network is the adjacency matrix generated in Step 2, and the loss function is the cross-entropy loss function.

[0048] like Figure 1 As shown, the graph transformer model consists of a multi-head attention mechanism, a normalization layer, and a forward propagation layer. The basic component of the multi-head attention mechanism module is the attention module. The attention module can be represented as:

[0049]

[0050] Where Q is the consultation vector, K is the key-value vector, and V is the value vector. Multi-head attention enables the model to jointly attend to information from different representation subspaces at different locations. This can be represented as:

[0051] MultiHead(Q,K,V)=Concat(head1,head2,...,head h W O

[0052] in,

[0053] Before feeding the samples into the graph transformer, this project sums the positional and feature encodings of each image patch. The output of the transformer is then fed into multiple layers of graph convolutions, where each layer can be represented as:

[0054] H (l+1) =f(H (l) A (l) )

[0055] x′x=σ(A (l) H (l) W (l) )

[0056] Among them, H l It is the input of the l-th layer, A (l) This is the corresponding adjacency matrix. Before feeding it into the convolutional neural network shown in the figure, this project made the following changes to the adjacency matrix:

[0057]

[0058] in, yes The block diagonal matrix. Therefore, the output of the entire graph transformer can be represented as:

[0059]

[0060] After applying GCN to pool the features of each patch, a two-layer fully connected neural network and a softmax function are used to predict the labels.

[0061] Step 4: Use gradient descent with backpropagation. With each gradient descent, the loss of the training set will become smaller and smaller. Training is complete when the loss of the training set reaches the set value or the number of iterations exceeds the preset threshold. Apply the model to the test set to obtain the KRAS gene mutation results of the test set.

Claims

1. A pathological image analysis method based on effective metric learning and graph transformer, characterized in that, Includes the following steps: (1) Preprocess the existing pathological image data that has been acquired in advance; (2) From the image data obtained in step (1), a certain number of image patches are collected and input into the VGG-16 network with an effective metric learning loss function for training. The trained model is used as a feature extractor. The feature extractor is used to extract the features of the image patches and the adjacency matrix of the graph is constructed using these features. (3) Construct a deep network based on graph transformer. The input of this deep neural network is the adjacency matrix generated in step (2), and the loss function is the cross-entropy loss function. (4) Based on the loss function in step (3), the gradient descent method of backpropagation is used. With each gradient descent, the loss of the training set will become smaller and smaller. When the loss of the training set reaches the set value or the number of iterations exceeds the preset threshold, the training is completed. The model is applied to the test set and the pathological image analysis results are output. The implementation process of step (2) is as follows: Image patches are fed into an efficient metric network based on am-softmax for training. After N rounds of training, a deep feature extractor based on efficient metric learning is obtained. The loss function used in efficient metric learning is as follows: in, Scaling factor These are the weights of the last fully connected layer. For the corresponding full slice tag; For each image patch, find the K image patches with the closest feature Euclidean distance to it and construct an edge; the position of each image patch is encoded, and the specific encoding method is as follows: ； in, It is the dimension of encoding, which adds the positional encoding of the image patch to the image patch features, and then feeds it into the graph transformer model.

2. The pathological image analysis method based on effective metric learning and graph transformer according to claim 1, characterized in that, The implementation process of step (1) is as follows: Considering that pathological images contain a large number of non-cellular areas, the background area is first filtered out based on the values ​​of each pixel in the R, G, and B channels. The filtering rule is that if the pixel value in any channel is greater than 200, or if 90% of the pixels in the 200*200 neighborhood of the pixel are background areas, then the pixel is filtered out. The sample is augmented by rotating and flipping the acquired image patches to increase the sample size.

3. The pathological image analysis method based on effective metric learning and graph transformer according to claim 1, characterized in that, The image block size mentioned in step (2) is 512*512.

4. The pathological image analysis method based on effective metric learning and graph transformer according to claim 1, characterized in that, The deep network based on graph transformer described in step (3) consists of a multi-head attention mechanism, a normalization layer, and a forward propagation layer; the basic component of the multi-head attention mechanism module is the attention module, which is as follows: ； in, It is a consultation vector. It is a key-value vector. The value vector; multi-head attention enables the model to jointly attend to information from different representation subspaces at different locations, represented as: ； in, ; The output of the Transformer is fed into multiple layers of graph convolutions, where each layer is represented as: ； ； in, It is the input of the l-th layer. This is the corresponding adjacency matrix; before feeding it into the graph convolutional neural network, the adjacency matrix is ​​modified as follows: ； in, yes The block diagonal matrix; the output of the entire graph transformer is represented as: ； After applying GCN to pool the features of each patch, a two-layer fully connected neural network and a softmax function are used to predict the labels.

Images