A two-stage full-view digital pathology slice classification method based on multi-instance learning

Abstract

This invention discloses a two-stage full-view digital pathology slide classification method based on multi-instance learning, comprising the following steps: Step 1, acquiring full-view digital pathology slides; Step 2, cropping, scaling, and filtering the images; Step 3, training the embedding space of the pathology slides using a combination of multi-instance learning and supervised metric learning; Step 4, using self-attention to detect and classify the pathology slides. In the first stage, multi-instance learning provides pseudo-labels and constructs...<anchor,positive,negative> The algorithm trains an encoder using data. In the second stage, the encoder from the first stage is directly used to extract features from the pathological slides, and a self-attention network is used to perform the final classification of the slides. This proposed two-stage, full-view digital pathological slide classification method based on multi-instance learning can classify entire pathological slides for diagnosis, providing valuable reference for pathologists.

Description

Technical Field

[0001] This method belongs to the field of computer vision and is mainly used for the classification of whole-field-of-view digital pathology slides (WSIs). A two-stage classification method based on multi-instance learning assigns pseudo-labels to sub-slides of WSIs, and a supervised metric learning model is used to train the embedding vectors, aiming to make the embedding vectors of slides of the same class as close as possible, and the embedding vectors of slides of different classes as far apart as possible. The pseudo-label method essentially provides prior information to the model; supervised metric learning training requires significantly less data, hardware (GPU capacity), and training cycles compared to unsupervised contrastive learning. Background Technology

[0002] Pathologists use microscopes to observe the structure of human cells and tissues and the characteristics of lesions to make pathological diagnoses. Digital pathology diagnostic technology is an organic combination of computer technology and traditional optical technology. The resulting full-view digital pathology slides have ultra-high resolution and ultra-wide viewing angle; they are not only easy to store, but can also be transmitted over networks, and can even be directly annotated, laying a solid foundation for future artificial intelligence.

[0003] However, traditional AI algorithms cannot handle pathological slides with hundreds of millions of pixels: one approach is to replace the entire slide by cropping the central region of the image; another approach is to massively compress the entire digital pathological slide image. The former reduces the model's receptive field, and the latter reduces the slide resolution. Therefore, both methods reduce the accuracy of the model's diagnosis.

[0004] Multi-instance learning (MIM) can divide a full-view digital pathology slide image (called a bag) into several sub-slices (called instances). By training on instance-level slices, the results are then aggregated into a bag-level classification problem. Based on different aggregation approaches, MIM methods can be divided into two main categories: The first is "hard aggregation," which uses the bag's label as a pseudo-label for the instance, training the instance-level slice classifier. The bag is considered negative if all instances are negative, otherwise positive. The second is "soft aggregation," which obtains the embedding vectors of instance-level slices through an instance-level slice encoder. The embedding vectors of the bag are obtained by pooling the embedding vectors of instances within the bag, and then the bag is directly classified. Experiments show that the second approach is more effective, especially after introducing an attention mechanism. It can not only classify WSIs (Wide Slice Indicators) but also identify lesion regions based on attention scores. Currently, unsupervised contrastive learning is a common method for training instance-level slice encoders; however, unsupervised contrastive learning has stringent requirements on hardware (GPU capacity).

[0005] Based on the current situation, this paper proposes a two-stage full-view digital pathology slide classification method based on multi-instance learning. It combines two convergence approaches of multi-instance learning: the first stage uses pseudo-labels to supervise the training of the instance-level encoder; the second stage uses an attention mechanism to perform "soft convergence" of instances, directly training a bag-level classifier. A common residual network is used for feature extraction. The entire slide is cropped from left to right and top to bottom, and several sub-slices are selected. The label of the entire slide is used as the pseudo-label of the sub-slices, and the residual network is trained with new sample data. Finally, the embedding vectors of the sub-slices are used as input to the attention network to train the classification of the entire pathology slide.

[0006] This invention integrates the above ideas to make judgments on full-view digital pathological slide images. The research found that this invention, based on two-stage multi-instance learning, is innovative and original. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a classification method for two-stage full-view digital pathological slide images based on multi-instance learning. This classification method can effectively save the cost of sample annotation and training while ensuring accuracy.

[0008] The multi-instance learning model proposed in this invention (see attached specification for model structure diagram) Figure 1 First, the original full-view digital pathology slides (referred to as packages) need to be cropped from left to right and top to bottom into images of 1150×1150 pixels (referred to as instances) to ensure that the samples have uniform size. Second, to facilitate faster training and inference, the images are compressed to 224×224 pixels. Finally, images with a background ratio of more than 70% are discarded.

[0009] The preprocessed data serves as input for the first stage of the model. This stage primarily handles instance-level slice embedding learning, mapping instance-level slices to their corresponding vector representations. The goal is to ensure that embedding vectors of similar slices are as close as possible in the embedding space, while embedding vectors of different slices are as far apart as possible. A popular recent method is unsupervised contrastive learning, which augments slices to ensure they are as similar as possible to augmented slices and as dissimilar as possible to all other slices. However, this method has a long training cycle and stringent hardware requirements (GPU capacity). This invention trains the encoder using supervised metric learning through pseudo-labels.

[0010] A residual network F(*) is used as the base encoder. The entire full-view digital pathology slide is referred to as a bag, defined as B, and a 224×224 sample cut from this full-view digital pathology slide is referred to as an instance, defined as x. i The residual network is used to analyze packets (B={x1,…,x...)n For each instance in}), a prediction is made to obtain the probability value p of each instance being abnormal. i Extract the maximum predicted value p = max(p1,...,p...). n For each instance x, the true label of its parent packet is used as the pseudo label y. Therefore, a batch of instance-level training samples X = {{x1, y1}, ..., {x i ,y i},…,{x k ,y k}}, where y i = {0, 1}, where k represents the total number of instances extracted.

[0011] To learn the embedding space of instances, a triplet dataset is first constructed, and then the triplet loss function is trained. Let x... i Treat it as an anchor, i.e., write it as From x i To extract a sample from a dataset with the same labels, write it as... From x i Extracting a sample from a dataset with different labels, denoted as Then there is a ternary dataset The ternary loss function Lp is:

[0012]

[0013] The margin is a hyperparameter.

[0014] Furthermore, to ensure the accuracy of the pseudo-labels provided for instances, a multilayer perceptron is needed after the encoder to perform binary classification training on the instance-level data. Here, the cross-entropy loss function Lce is used:

[0015]

[0016] in

[0017]

[0018] σ(*) is the sigmoid function.

[0019] W represents the fully connected parameters, and b represents the bias of the fully connected layer.

[0020] F(*) outputs a 1*512 dimensional vector.

[0021] The final loss function is the sum of the cross-entropy loss function and the ternary loss function:

[0022] Loss = Lce + Lp

[0023] Therefore, the first stage is essentially a multi-task problem. On one hand, it uses pseudo-labels to construct a triplet dataset to learn the embedding space of slices; on the other hand, it uses pseudo-labels to directly perform binary classification to improve the accuracy of pseudo-labels.

[0024] After the first stage of training is completed, simply use the encoder to obtain the embedding vectors of all instances in the bag, retain these vectors, and use them as input for the second stage.

[0025] In the second stage, the embedding vectors from the first stage are used as input, and a self-attention mechanism is used as the network model to perform binary classification training on the entire slice.

[0026] Assume package B has n instances, B = {x1, ..., xn} n}, here x i For the i-th instance of package B, first use a residual network to obtain a 512-dimensional vector for each instance.

[0027] e i =F(x) i )

[0028] Combine the embeddings of n instances of package B:

[0029] z0 = [e1, ...,e n ]

[0030] The n*512 dimension z0 is passed through a multi-head attention network MultiHeads:

[0031] z′0 = MultiHeads(z0)

[0032] The principle of multi-head attention networks is to divide instance e into multiple attention networks. i q is obtained after a fully connected layer. i ,k i ,v i Let and represent the query, key, and value of the i-th instance, respectively. Then, the query vector is multiplied by the keys of all other instances to obtain the attention score. Finally, the weighted sum of the attention score and the value is used as the value for instance e. i Vector update. The mathematical formula is as follows:

[0033] q i =W1*e i +b1,Q=concat[q i ,…,q n ]

[0034] k i =W2*e i +b2,K=concat[k i ,…,kn ]

[0035] v i =W3*e i +b3,V=concat[v i ,…,v n ]

[0036]

[0037] Here d is q i The dimension of "multi-head" refers to the technique of dividing the embedding vector into multiple smaller vectors, performing the attention network above on each separately, and finally merging them together.

[0038] Using a multi-head attention network ensures that each instance pays attention to all other instances in the same bag, providing a global perspective. In contrast, traditional convolutional neural networks only have a local receptive field.

[0039] Sum the residuals obtained above z'0 with the original input z0, and then perform layer regularization (LN):

[0040] z′1=LN(z′0+z0)

[0041] Then it passes through a fully connected MLP network, followed by a second residual summation and layer regularization:

[0042] z1=LN(MLP(z′1)+z′1)

[0043] Finally, after passing through the mean pooling layer and the activation function σ(*), the final result of this package is obtained:

[0044] y = σ(W*meanPooling(z1) + b)

[0045] Here, the range of y is [0,1], representing the probability of an anomaly in the entire slice. σ(*) is the sigmoid activation function.

[0046] Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall model structure of the present invention. Detailed Implementation

[0048] The model proposed in this invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0049] A two-stage full-view digital pathology slide classification method based on multi-instance learning, comprising the following steps:

[0050] Step 1: Acquire full-field digital pathology slides;

[0051] Step two: Crop, scale, and filter the image;

[0052] First, the full-view digital pathology slides are cropped into 1150×1150 pixel samples from left to right and top to bottom to ensure that the samples have a uniform size. Second, to facilitate training, the samples are compressed to a size of 224×224. Finally, samples with a background ratio of more than 70% are discarded.

[0053] Step 3: Train the embedding space of pathological slides using a combination of multi-instance learning and supervised metric learning;

[0054] Based on the fundamental concepts of multi-instance learning, the entire full-view digital pathology slide is referred to as a package, defined as B, and a 224×224 sample cut from this full-view digital pathology slide is referred to as an instance, defined as x. i Using a residual network F(*) as the base encoder, the packets (B={x1,…,x) are processed through the residual network. n For each instance in}), a prediction is made to obtain the probability value p of each instance being abnormal. i Extract the maximum predicted value p = max(p1,...,p...). n For each instance x, the true label of its parent packet is used as the pseudo label y. Therefore, a batch of instance-level training samples X = {{x1, y1}, ..., {x i ,y i},…,{x k ,y k}}, where y i = {0, 1}, where k represents the total number of instances extracted.

[0055] Based on this batch of instance-level training samples, instances can be classified into two categories, using the cross-entropy loss function Lce:

[0056]

[0057] in

[0058]

[0059] σ(*) is the sigmoid function.

[0060] W represents the fully connected parameters, and b represents the bias of the fully connected layer.

[0061] F(*) outputs a 1*512 dimensional vector.

[0062] On the other hand, a deep metric learning model can be trained, allowing the model to learn the embedding space of instances. Here, the triplet loss function is used:

[0063]

[0064] x i Treat it as an anchor, i.e., write it as From x i To extract a sample from a dataset with the same labels, write it as... From x i Extracting a sample from a dataset with different labels, denoted as Finally, margin is an adjustable hyperparameter.

[0065] The final loss function is the sum of the cross-entropy loss function and the ternary loss function.

[0066] Loss = Lce + Lp

[0067] Step four: Use self-attention to examine and classify the pathological slides.

[0068] Each packet is used as input to the self-attention network, and the output is the result for that packet. A packet has n instances, B = {x1, ..., xn}. n}, B is the definition of a package, x i For the i-th instance of this package, first use a residual network to obtain a 512-dimensional vector e for each instance. i :

[0069] e i =F(x) i )

[0070] The encoded packet first passes through a multi-head attention network (MultiHeads).

[0071] z0 = [e1, ...,e n ]

[0072] z′0 = MultiHeads(z0)

[0073] The principle of multi-head self-attention networks is to process instance vectors e... i q is obtained after a fully connected layer. i k i v i Let and represent the query, key, and value of the i-th instance, respectively. Then, the query vector is multiplied by the key vectors of all other instances to obtain the attention score. Finally, the weighted sum of the attention score and the value vector is used as the instance vector e. i Updated. The mathematical formula is as follows:

[0074] q i =W1*e i +b1,Q=concat[q i ,…,q n ]

[0075] k i =W2*e i +b2,K=concat[k i ,…,k n ]

[0076] v i =W3*e i +b3,V=concat[v i ,…,v n ]

[0077]

[0078] Where concat[*] is the vector concatenation operation; d is the value of q. i The dimensions are: W1, W2, W3 are the parameters of the three fully connected layers; b1, b2, b3 are the biases of the three fully connected layers.

[0079] Then, after passing through layers of regularized LN, fully connected MLP, and mean pooling, the probability value y of the abnormal result of packet B is finally obtained:

[0080] z′1=LN(z′0+z0)

[0081] z1=LN(MLP(z′1)+z′1)

[0082] y = σ(W*meanPooling(z1) + b)

[0083] Finally, based on the y-value and the set threshold, it can be determined whether the slice is abnormal.

Claims

1.A two-stage whole slide digital pathology slide classification method based on multiple instance learning, characterized in that: Comprising the following steps, Step one, collect the whole field digital pathology section image; Step two, crop, scale, and filter the whole field digital pathology section image obtained in step one; Step three, train the whole field digital pathology section processed in step two using a combination of multi-instance learning and supervised metric learning, complete the first stage training of the model; Step four, add a self-attention network after the model trained in step three to realize the detection and classification of case images, complete the second stage training of the model; The implementation process of step three is as follows: The whole field digital pathology section is called a package, defined as B, and the 224*224 size sample cut from the whole field digital pathology section is called an instance, defined as x i , let the package B contain n instances, then the package B is written as B={x1,…,x n}; using the residual network F(*) as the base encoder, the i-th instance in the package B is predicted by the residual network, and the probability value p i of the instance developing lesions is obtained; the maximum probability value p=max(p1,...,p i ,...,p n ) of the n instances is extracted, and the corresponding instance x is obtained, and the real label of the parent package is taken as the pseudo label y of the instance x; thus, a batch of instance level training samples X={{x1,y1},…,{x i ,y i},…,{x k ,y k}} is obtained, wherein x i represents the i-th extracted instance, y i is the label corresponding to the instance, and k represents the total number of extracted instances; According to this batch of instance-level training samples, on the one hand, the instances are classified into two categories, and here the cross-entropy loss function Lce is used: where k represents the total number of instances, x i represents the i-th extracted instance, y i is the label corresponding to the instance; is the model output value, is a sigmoid function, W is a parameter of full connection, b is a bias of full connection; F(*) is an encoder function, and a 1*512-dimensional vector is output. On the other hand, the deep metric learning model is trained, so that the model learns the embedding space of the instance, and here the triadic loss function Lp is used: x i denote the i-th instance of the extracted, consider it as anchor, i.e. write from x i denote a sample from the data set of the same label, write from x i denote a sample from the data set of the different label, write Finally, margin is a tunable hyper-parameter; The final loss function is the sum of the cross-entropy loss function and the triadic loss function Loss: Loss = Lce + Lp. 2.The two-stage whole slide digital pathology slide classification method based on multiple-instance learning according to claim 1, wherein: The implementation process of step two is as follows, First, the whole field digital pathology section is cropped into 1150x1150 pixel samples in the order from left to right and from top to bottom, ensuring that the samples have uniform size; second, in order to facilitate training, the samples are compressed to 224x224 size; finally, samples with more than 70% background are discarded. 3.The two-stage whole slide digital pathology slide classification method based on multiple-instance learning according to claim 2, characterized in that: The implementation process of step four is as follows: Each bag is taken as input to a self-attention network, and the output is the result of the bag; a bag has n instances, B = x1,..., xn n}, B is the definition of a bag, xi i is the i-th instance of the bag, a 512-dimensional vector e i is first obtained using a residual network for each instance e i = F(x i ) The encoded package first passes through the multi-head attention network MultiHeads: z0=[e1,...,e n ] z'0 = MultiHeads(z0) Here z0 is the aggregation of n instance vectors, and z'0 is the output of the multi-head attention network, both of which have a dimension of n*512; In multi-head self-attention networks, the instance vectors e i Through a fully connected layer, we get q i , k i , v i , which represent the query, key, and value of the i-th instance, respectively. Then the query vector is dot producted with the key vectors of all other instances as attention scores, and finally the weighted sum of the value vectors with the attention scores as the instance vector e i The update of the parameters; the formula is as follows: q i = W1* e i + b1, Q = concat[q i ,..., q n ] k i = W2* e i + b2, k = concat[k i ,..., k n ] v i = W3* e i + b3, v = concat[v i ,…, v n ] where concat[*] is a vector concatenation operation; d is the dimension of q i W1, W2, W3 are three fully connected layer parameters respectively; b1, b2, b3 are three fully connected layer biases respectively. Then it passes through the layer normalization LN, the fully connected network MLP, and the average pooling layer meanPooling in turn, and finally gets the probability value y of the package B abnormality: z'1 = LN(z'0 + z0) z1 = LN(MLP(z1) + z1) y = σ(W*meanPooling(z1) + b) Here W and b are the parameters and bias of the fully connected layer respectively; σ(*) is the sigmoid function; Finally, according to the value of y and the set threshold, it is judged whether an abnormality occurs.

Images