A lesion category perception-based digital pathology image representation learning method

Abstract

This invention discloses a representation learning method for digital pathology images based on lesion category awareness, comprising the following steps: S1. Obtaining a dataset; S2. Pre-training based on weakly supervised contrastive learning; S3. Obtaining a feature extractor for digital pathology images after queue updating. Addressing the shortcomings of existing frameworks, this invention proposes a representation learning method for digital pathology images. This method does not rely on expert annotation, requiring only slice-level classification labels. It utilizes annotation information with minimal acquisition cost to learn robust and discriminative image representations, and its labeling performance in downstream tasks outperforms existing state-of-the-art methods.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically to a representation learning method for digital pathological images based on lesion category perception. Background Technology

[0002] Cancer poses a significant threat to human health, and in the medical field, the diagnosis of digital pathology images is a crucial step in the cancer diagnosis process. With the rapid development of computer and microscopic imaging technologies, the acquisition of digital pathology images has become convenient and fast, making automated computer analysis algorithms suitable for whole-slice images a research hotspot in recent years. Research based on whole-slice image (WSI) analysis is gaining popularity in computational pathology; however, due to hardware resource limitations, WSI images are difficult to directly input into deep neural networks (DNNs) for training. A typical solution is to generate WSI-level representations by aggregating local tissue representations before inference.

[0003] In terms of algorithms, most current algorithms rely on fine-grained annotation for supervised learning. However, digital pathology images, due to their much higher resolution than natural scene images and their complex structure and diverse types, make annotation extremely arduous, and large-scale annotation difficult to achieve. To address this contradiction, transfer learning is a widely accepted method for learning local tissue representations. For example, existing technologies disclose techniques that directly extract tissue representations using models pre-trained on ImageNet and apply these representations to downstream whole-slice classification tasks. However, there is a "domain difference" between natural images and histopathological images. Some weakly supervised learning methods tend to transform the feature domain through multiple instance learning (MIL). For example, using MIL, a CNN is used to predict the positive probability of instances within each data packet, and samples with high confidence are selected based on the positive probability for training. However, the upper limit of such methods is limited by the model's performance outside the training data domain.

[0004] Based on this, self-supervised learning methods have gradually been introduced for learning the representation of pathological tissues. One existing approach, SimTriplet, utilizes a self-supervised contrastive learning method to learn the representation of pathological tissues. It also reconsiders the applicability of contrastive learning methods to pathological images, reconstructing a positive sample construction strategy by sampling positive samples from neighboring tissue regions, thus strengthening the spatial correlation of local tissues.

[0005] Most existing technical solutions follow the supervised learning paradigm, and their performance depends on large-scale, fine-grained annotation. Some weakly supervised learning methods do not focus on the quality of representations, are mostly designed for downstream tasks, and their data packet modeling methods are only applicable to binary classification tasks, with very little research on multi-class classification tasks. Although self-supervised learning methods have achieved great success in natural image analysis tasks, due to the differences between the patterns of pathological images and natural scene images, existing technical solutions still have considerable room for improvement.

[0006] Therefore, how to provide a representation learning method for digital pathology images based on lesion category awareness that no longer relies on large-scale, fine-annotation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a representation learning method for digital pathological images based on lesion category perception. This method does not rely on fine annotation by experts, but only requires slice-level classification labels. It learns robust and discriminative image representations by using annotation information with minimal acquisition cost, and the labels outperform existing state-of-the-art methods in downstream tasks.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A representation learning method for digital pathology images based on lesion category awareness includes the following steps:

[0010] S1. Obtain the dataset: Cropped images x from the input digital pathology images form a dataset D, where x∈D. Pseudo-sample labels are assigned to each x; the assignment rule is: for a full slice S, the slice category is... For any sample taken from S, the label

[0011] S2. Pre-training based on weakly supervised contrastive learning:

[0012] S21. After data augmentation of x, two different augmented views v are obtained. q and v k ;

[0013] S22. v q and v k The inputs are fed into the query branch and the key branch of the contrastive learning network, respectively, and the encoders in the query branch and the key branch generate the corresponding representation z. q =f q (v q ) and z k =f k (v kThe representation is mapped to the space where contrastive loss is applied through projectors in the query branch and the key branch, respectively, to obtain q = P. q (z q ) and k + =P k (z k ), where k + q represents a positive sample, and q represents a negative sample.

[0014] S23. The representation z of the negative sample q =f q (v q The lesion storage queue M is used to update the weight θ of each query branch. q and the weight θ of the key branch k ;

[0015] S3. Update the representation of the negative samples in the queue M to obtain a feature extractor for digital pathology images: use the average KL divergence to determine the set of samples to be updated, specifically as follows:

[0016]

[0017]

[0018] Where B represents the set of representations contained in a training batch, U represents the set of representations in a training batch that needs to be updated, and D... KL (·) is the KL divergence function, and P(x) represents the k to be updated. + The similarity distribution between the representations in the queue and the expected distribution, Q(x), where i is the number of samples of each class in M.

[0019] Preferably, the encoder includes a convolutional neural network, and the projector includes a multilayer perceptron, wherein the multilayer perceptron sequentially includes a first linear layer, a ReLU layer, and a second linear layer.

[0020] Preferably, the lesion storage queue M is represented as:

[0021] M={k i0 ,k i1 ,...,k iN |i=0,1,...,K}

[0022] Where N is the storage capacity of each slice category, and K represents the total number of slice categories.

[0023] Preferably, the weight θ of the query branch is updated. q The method is as follows:

[0024] The weight parameters of the query branch are updated by minimizing the InfoNCE loss:

[0025]

[0026] in, The set of categories that need to be used as negative samples for comparison is defined as follows: τ is the temperature coefficient, k yi This represents the i-th representation of category y in queue M.

[0027] Preferably, the weight θ of the key branch is updated. k The method is as follows:

[0028] The query branch is updated using the following formula via an exponential moving average mechanism:

[0029] Updated using an exponential moving average mechanism:

[0030] θ k =mθ k +(1-m)θ q m∈[0,1)

[0031] Where m is the momentum update parameter.

[0032] Preferably, P(x) and Q(x) are expressed as:

[0033]

[0034]

[0035] Where K is the total number of slice categories, and 1(·) function when The value is 1 if it is true, and 0 otherwise.

[0036] As can be seen from the above technical solution, compared with the prior art, this invention discloses a representation learning method for digital pathological images based on lesion category awareness. It utilizes only slice-level annotation information as weak supervision for model training, greatly alleviating the difficulty of large-scale annotation of digital pathological slices. The model does not rely on the fine annotations of pathology experts, thus allowing for rapid updates and iterations on large-scale data. Simultaneously, it alleviates, to some extent, the problems of heavy dependence on the training environment and poor generalization ability caused by strong supervision information in supervised learning models. This invention alleviates the contradiction between the difficulty of large-scale annotation of digital pathological slices and the dependence of model accuracy on supervised information. Furthermore, it proposes improvements to existing pathological image representation learning methods by embedding semantic information into the image representation. This advantage means that this invention can obtain robust and discriminative image representations using low-cost slice-level annotation information and is suitable for pre-training on large-scale data. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a representation learning method for digital pathological images based on lesion category awareness, provided by the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] This invention discloses a representation learning method for digital pathology images based on lesion category awareness, comprising the following steps:

[0041] S1. Obtain the dataset: Cropped images x from the input digital pathology images form a dataset D, where x∈D. Pseudo-sample labels are assigned to each x; the assignment rule is: for a full slice S, the slice category is... For any sample taken from S, the label

[0042] S2. Pre-training based on weakly supervised contrastive learning:

[0043] S21. After data augmentation of x, two different augmented views v are obtained. q and v k ;

[0044] S22. v q and v k The inputs are fed into the query branch and the critical branch of the contrastive learning network, respectively, and the encoders within the query branch and the critical branch generate the corresponding representation z. q =f q (v q ) and z k =f k (v k ), and then map the representation to the space where the contrastive loss is applied through projectors in the query branch and the key branch respectively, to obtain q = Pq (z q ) and k + =P k (z k ), where k + q represents a positive sample, and q represents a negative sample.

[0045] S23. Represent the negative samples z q =f q (v q The lesion storage queue M is used to update the weight θ of each query branch. q and the weight θ of the key branch k ;

[0046] S3. Update the representation of negative samples in queue M to obtain a feature extractor for digital pathology images: Based on the assumption that the feature distance between samples of the same class is less than the feature distance between samples of different classes, this invention uses the average Kullback-Leibler (KL) divergence to determine the sample set to be updated, as follows:

[0047]

[0048]

[0049] Where B represents the set of representations contained in a training batch, U represents the set of representations in a training batch that needs to be updated, and D... KL (·) is the KL divergence function, and P(x) represents the k to be updated. + The similarity distribution between the representations in the queue and the expected distribution, Q(x), where i is the number of samples of each class in M.

[0050] It should be noted that:

[0051] For S1, in this instance, the sliding window method can be used to crop digital pathology slides into 256×256 image blocks.

[0052] For S2, the contrastive learning method aims to extract intra-class similarities and inter-class differences from image content. For example... Figure 1 As shown, the contrastive learning network architecture used in this invention is a Siamese network consisting of two branches: a query branch and a key branch. The query branch and the key branch have the same structure. The query branch is encoded by an encoder f. q and projector p q They are composed of a set of trainable weights θ q Confirmed; the key branch belongs to the encoder f k and projector p k They consist of a set of non-trainable weights θ k definition.

[0053] For S3, to ensure that the representations in each category's storage queue represent the corresponding class during training, each queue needs to be selectively updated. This invention proposes a novel queue update strategy (QRS), such as... Figure 1 As shown in (b). The strategy of this invention aims to refine the queues so that each queue consists of the most representative representations of the corresponding class. This invention uses Kullback-Leibler (KL) divergence to find potential noisy samples.

[0054] To further implement the above technical solution, the encoder includes a convolutional neural network, and the projector includes a multilayer perceptron, which sequentially includes a first linear layer, a ReLU layer, and a second linear layer.

[0055] To further implement the above technical solution, the lesion storage queue M is represented as follows:

[0056] M={k i0 ,k i1 ,...,k iN |i=0,1,...,K}

[0057] Where N is the storage capacity of each slice category, and K represents the total number of slice categories.

[0058] To further implement the above technical solution, the weight θ of the query branch is updated. q The method is as follows:

[0059] The weight parameters of the query branch are updated by minimizing the InfoNCE loss:

[0060]

[0061] in, The set of categories that need to be used as negative samples for comparison is defined as follows: τ is the temperature coefficient, k yi This represents the i-th representation of category y in queue M.

[0062] To further implement the above technical solution, the weights θ of the key branches are updated. k The method is as follows:

[0063] The query branch is updated using the following formula via an exponential moving average mechanism:

[0064] Updated using an exponential moving average mechanism:

[0065] θ k =mθ k +(1-m)θ q m∈[0,1)

[0066] Where m is the momentum update parameter.

[0067] To further implement the above technical solution, P(x) and Q(x) are expressed as:

[0068]

[0069]

[0070] Where K is the total number of slice categories, and 1(·) function when The value is 1 if it is true, and 0 otherwise.

[0071] This method may also include: S4. Downstream task migration

[0072] According to S1 to S3 above, a feature extractor for digital pathology images can be trained to extract features from digital pathology images.

[0073] The feature extractor can be fine-tuned using a small amount of labeled information to obtain a new model with better feature representation for a specific task; or it can be directly transferred to a specific downstream task to perform slice-level classification to obtain coarse diagnostic information of the slices, or transferred to a segmentation task to obtain coarse segmentation information of the lesion area for doctors to refer to.

[0074] The invention will be further explained below with reference to specific experiments:

[0075] In this embodiment, the method of the present invention was validated on an endometrial cancer pathology dataset (containing 698 slides) and an EGFR dataset (containing 754 slides). TransMIL was used as the whole-slide classification framework to compare the representations learned by the method of the present invention with those of other representation learning methods. The comparison results are shown in Tables 1 and 2:

[0076] Table 1 Comparison of average classification accuracy of endometrial cancer pathology datasets

[0077]

[0078] Table 2 Comparison of average classification accuracy of EFGR pathology dataset

[0079]

[0080] Experimental results show that the present invention introduces slice-level labels as supervision information and embeds semantic information into image representation, which effectively improves the performance of the learned representation on downstream tasks. Compared with other advanced transfer learning and self-supervised learning representation learning methods, the method of the present invention achieves the best performance on downstream tasks.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0082] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lesion class-aware based digital pathology image representation learning method, characterized in that, The method comprises the following steps: S1. Obtain a data set: crop the input digital pathology image to obtain a cropped image x, form a data set D, wherein x e D, and assign a pseudo sample label to each x; the assignment rule is: for a whole section S, the section category is For any sample sampled from S, the label S2. Pre-training based on weakly supervised contrastive learning: S21. Obtain two different augmented views v after data augmentation on x q and v k ; S22. v q and v k are input into the query branch and the key branch of the contrastive learning network respectively, and through the encoders in the query branch and the key branch, representations z q q (v q ) and z k k (v k ) are respectively generated, and through the projectors in the query branch and the key branch, the representations are mapped to the space where the contrastive loss is applied, obtaining q = P q (z q ) and k + = P k (z k ), where k + is the positive sample and q is the negative sample;​​ S23. Characterize z of the negative sample q = f q (v q ) is stored in the lesion storage queue M, and the weight θ of the query branch and the weight θ of the key branch are updated respectively; q ; k ; S3. Updating the representation of the negative samples in the queue M, and obtaining a feature extractor for digital pathology images after updating: using the average KL divergence to determine the sample set to be updated, in the following manner: wherein, represents a set of representations contained in a training batch, represents a set of representations in a training batch that need to be updated, D KL (·) is the KL divergence function, denotes the k + distribution of similarity between the representation and the representations in the queue, denotes the expected distribution, i is the number of samples in each class in M. 2.The lesion class-aware digital pathology image representation learning method of claim 1, wherein, The encoder comprises a convolutional neural network, and the projector comprises a multi-layer perceptron, which comprises a first linear layer, a ReLU layer and a second linear layer in sequence. 3.The lesion class-aware digital pathology image representation learning method of claim 1, wherein, The lesion storage queue M is represented as: M = {k i0 ,k i1 ,…,k iN |i = 0, 1, …, K} Wherein, N is the storage capacity of each slice category, and K represents the total number of slice categories. 4.The lesion class-aware digital pathology image representation learning method of claim 1, wherein, updating the weight θ of the query branch q The method is as follows: The weight parameters of the query branch are updated by minimizing the InfoNCE loss: wherein, represents a set of classes that need to be contrasted as negative samples, defined as τ is the temperature coefficient, k yi represents the i-th representation of the y-th class in the queue M. 5.The lesion class-aware digital pathology image representation learning method of claim 1, wherein, updating weights θ of the key branches k The method is: The weight parameters are updated by an exponential moving average mechanism: θ k = m θ k + (1 - m) θ q , m ∈ [0, 1) Wherein, m is a momentum update parameter. 6.The lesion class-aware digital pathology image representation learning method of claim 1, wherein, and is represented by: where K is the total number of slice categories, k yi represents the i-th representation of the y-th category in the queue M, the indicator function 1(·) takes the value 1 when and 0 otherwise.

Images