A pathological whole section feature learning method based on self-supervised learning

Abstract

The application relates to the technical field of digital image processing, and relates to a pathological whole slice feature learning method based on self-supervised learning; a whole slice image is acquired, and initial image blocks of the whole slice are generated through preprocessing; after whole slice features are constructed and randomly masked, image feature blocks that are not masked are screened out and input into an encoder; an encoding module performs attention weighting calculation according to spatial position information of the image blocks, and generates an encoded result after weighting; masked image block features are initialized, and are combined according to position information and the encoded result; the combined image block features are input into a decoder to obtain a decoding result; a reconstruction loss is calculated according to the initial image blocks and the decoding result, and model pre-training is performed; the application uses a large amount of pathological whole slice image data without label information to perform feature learning and mining, learns feature representation with strong generalization ability and rich semantic expression, and can have high performance in specific downstream tasks such as whole slice classification.

Description

Technical Field

[0001] This invention relates to the field of digital image processing technology, and more specifically to a method for learning the features of whole pathological slides based on self-supervised learning. Background Technology

[0002] Currently, due to hardware resource limitations, whole-slice image analysis struggles to directly process gigapixel whole slices within an end-to-end framework. The Hierarchical Image Pyramid Transformer (HIPT) approach explores and proposes a new challenge: whole-slice-level self-supervised learning. This method leverages the inherent hierarchical structure of whole slices to construct a multi-level self-supervised learning framework for learning high-resolution image representations. This approach enables multi-instance learning-based frameworks to utilize a large number of unlabeled whole slices, further improving the accuracy and robustness of tumor feature recognition.

[0003] HIPT is a hierarchical learning framework based on a greedy training strategy. Biases and errors generated at each level of the representation model are accumulated in the final decision model. Furthermore, the VisionTransformer (ViT) backbone used in HIPT was originally designed for fixed-size natural scene images with consistent positional information, but whole-slice histopathological sections are scale-variable and isotropic. The positional embedding strategy of ViT introduces ambiguity into structural modeling. To mitigate this problem, the Kernel Attention Transformer (KAT) establishes hierarchical masks based on local anchors, thus preserving multi-scale relative distance information during training. However, these masks are manually defined, which is not trainable and lacks directional information. Currently, the embedding strategies for describing the structure of whole slices are incomplete.

[0004] Therefore, how to fully learn the spatial information of whole-slice images to improve model performance is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method for learning features of whole pathological slides based on self-supervised learning, which can learn feature representations with strong generalization ability and rich semantic expression, and can perform high performance in specific downstream tasks such as whole slide classification.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A self-supervised learning-based method for learning features from whole pathological slides includes the following steps:

[0008] A full-slice image is acquired, and initial image patches of the full slice are generated through preprocessing. Full-slice features are constructed from the initial image patches and randomly masked. The unmasked image patch features are then selected and input into the encoder. The encoder consists of multiple encoding modules connected in series. Each encoding module performs attention weighting calculation based on the spatial location information of the image patches and generates a weighted encoding result. The masked image patch features are initialized and combined with the encoding result based on their location information. The combined image patch features are input into the decoder to obtain the decoding result. The reconstruction loss is calculated based on the initial image patches and the decoding result, and the model is pre-trained.

[0009] Preferably, the steps further include:

[0010] After pre-training, parameter fine-tuning is performed: the parameters of the pre-trained encoder framework network are extracted and used as the full-slice feature extractor; the full-slice image is input, and the features extracted by the encoder and the full-slice labels are input into the classifier, and the network is trained using the cross-entropy loss function.

[0011] Preferably, the step of constructing full-slice features for the initial image patch includes:

[0012] The anchor point features of the whole slice image blocks are identified using the DINO framework, and spatial location information is added to each initial image block to obtain the relative distance matrix and angle matrix between each initial image block and the anchor point features.

[0013] Preferably, the encoding module includes a first normalization layer, a cross-attention layer, a second normalization layer, and a multilayer perceptron connected in sequence;

[0014] The cross-attention layer performs attention calculations based on the relative distance matrix and angle matrix output by the first normalization layer:

[0015]

[0016]

[0017] Among them, K (n) X is the anchor feature output by the nth encoding module. (n) For the first

[0018] n encoded output image patch features, and ) all represent the features in the nth encoding module that are changed from d e dimensional mapping to d k A learnable mapping function of dimension d e It is the input dimension, d k It is the output dimension, d k It can be greater than, less than or equal to d eThis achieves feature upscaling, downscaling, and feature transformation in the same dimensionality space, respectively. (n) Let P be the relative distance matrix of the image blocks in the nth encoding module. (n) The angle information matrix of the image patch relative to the anchor point in the nth encoding module and σ represents the mapping of the relative distance matrix and the relative angle matrix into learnable embedded features, respectively, and σ is the softmax activation function.

[0019] Preferably, in the feature processing of multiple cascaded coding modules, kernel redirection is performed before inputting into the next coding module. Each anchor feature is sorted according to the attention score of each direction, and the direction with the highest score is selected as the new main direction, thereby updating the direction information of all image blocks.

[0020] Preferably, the kernel redirection step includes:

[0021] Obtain the angle matrix and the attention matrix of the anchor point features; weight the angle matrix using the attention matrix of the anchor point features, calculate and sort the attention scores of each angle, select the highest score as the main direction for angle update, and obtain the updated angle; subtract the angle matrix from the updated angle to obtain the new angle matrix.

[0022] A pathological whole-slice feature extraction network model based on self-supervised learning includes an input layer, an encoder, a decoder, and an output layer. The input layer receives whole-slice images and generates image patches. The image patches are sequentially encoded and decoded by the encoder and the decoder, and then the whole-slice features are output through the output layer. The encoder is composed of multiple encoding modules connected in series. The encoding modules perform attention weighting calculation based on the spatial location information of the image patches and generate weighted encoding results.

[0023] Preferably, the encoding module includes a first normalization layer, a cross-attention layer, a second normalization layer, and a multilayer perceptron connected in sequence.

[0024] Preferably, the decoder is composed of multiple decoding modules connected in series, and the network structure of the decoding modules is the same as that of the encoding modules.

[0025] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a pathological whole-slice feature learning method based on self-supervised learning. It can utilize a large amount of unlabeled pathological whole-slice image data for feature learning and mining, and learn feature representations with strong generalization ability and rich semantic expression, which can achieve high performance in specific downstream tasks such as whole-slice classification. It proposes a kernel cross-attention mechanism based on spatial information perception, and introduces relative distance and relative orientation embedding information based on the anchor point mechanism. Combined with a dynamic orientation update strategy, it not only saves the resource overhead in the computation process, but also adapts to the isotropic characteristics of pathological whole slices, eliminates ambiguity in the semantic space, and is more conducive to expressing complete semantic information. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 The attached figure is a schematic diagram of a self-supervised learning-based method for learning features of whole pathological slides provided by the present invention.

[0028] Figure 2 The attached figure is a schematic diagram of the pathological whole-slice feature learning algorithm provided by the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1

[0031] like Figure 1 and Figure 2 This invention discloses a method for learning features of whole pathological slides based on self-supervised learning, comprising the following steps:

[0032] S1: Obtain the full-slice image and generate initial image patches of the full slice through preprocessing. During preprocessing, spatial location information of the anchor points is obtained through location clustering.

[0033] S2: Full slice image feature encoding: After constructing full slice features for the initial image block and randomly masking them, the unmasked image feature blocks are selected and input into the encoder; the encoder is composed of multiple encoding modules connected in series. The encoding modules perform attention weighting calculation based on the spatial location information of the image blocks and generate the weighted encoding result.

[0034] Specifically, the input is a full-slice image patch, with each patch being 256×256 pixels. Using the DINO framework, this patch is mapped to a 3-dimensional feature vector. Spatially uniform anchor feature vectors are added to the full slice, and spatial location information is added to all image patches. A portion of the image patch features and their corresponding location information are randomly removed. The remaining image patch features, their corresponding location information, and anchor feature vectors are then input into the full-slice feature encoder.

[0035] S3: Initialize the masked image patch features, randomly initialize a feature vector at each masked position; combine the feature vector with the encoding result based on the position information; input the combined image patch features into the decoder to obtain the decoding result.

[0036] S4: Calculate the reconstruction loss based on the initial image patch and the decoding result, and perform model pre-training.

[0037] To further implement the above scheme, S5 is also included: After pre-training, parameter fine-tuning is performed: extract the parameters of the pre-trained encoder framework network as the full-slice feature extractor; input the full-slice image, input the features extracted by the encoder and the full-slice labels into the classifier, and train the network through the cross-entropy loss function.

[0038] To further implement the above scheme, the encoding module consists of a first normalization layer, a cross-attention layer, a second normalization layer, and a multilayer sensing mechanism connected in sequence; the cross-attention layer performs attention calculation based on the relative distance matrix and angle matrix output by the first normalization layer.

[0039] The data streams for attention calculation of image patch features and anchor point features are represented as follows:

[0040]

[0041]

[0042] Among them, K (n) X is the anchor feature output by the nth encoding module. (n) For the first

[0043] n encoded output image patch features, and ) all represent the features in the nth encoding module that are changed from d e dimensional mapping to dk A learnable mapping function of dimension D (n) Let P be the relative distance matrix of the image blocks in the nth encoding module. (n) The angle information matrix of the image patch relative to the anchor point in the nth encoding module and σ represents the mapping of the relative distance matrix and the relative angle matrix into learnable embedded features, respectively, and σ is the softmax activation function.

[0044] To further implement the above scheme, during the feature processing of multiple cascaded coding modules, kernel redirection is performed before inputting into the next coding module. Each anchor feature is sorted according to the attention score of each direction, and the direction with the highest score is selected as the new main direction, thereby updating the direction information of all image blocks.

[0045] Specifically, the kernel redirection steps include: obtaining the angle matrix and the attention matrix of the anchor features; weighting the angle matrix using the attention matrix of the anchor features, calculating and sorting the attention scores of each angle, selecting the highest score as the main direction for angle update, and obtaining the updated angle; and subtracting the angle matrix from the updated angle to obtain the new angle matrix.

[0046] Example 2

[0047] Based on the same inventive concept, this invention provides a pathological whole-slice feature extraction network model based on self-supervised learning, including an input layer, an encoder, a decoder, and an output layer. The input layer receives the whole-slice image and generates image patches. The image patches are sequentially encoded and decoded by the encoder and decoder, and then the whole-slice features are output through the output layer. The encoder consists of multiple encoding modules connected in series. The encoding modules perform attention weighting calculations based on the spatial location information of the image patches and generate weighted encoding results. The encoding module includes a first normalization layer, a cross-attention layer, a second normalization layer, and a multilayer perceptron connected in series. The decoder consists of multiple decoding modules connected in series, and the decoding modules have the same network structure as the encoding modules.

[0048] This invention fully utilizes unlabeled data for pre-training to learn the semantic representation of the image itself. The method designs a cross-attention module based on spatial orientation awareness of the entire slice, which can fully capture image features at varying scales. Simultaneously, it combines an anchor point principal direction update strategy to adaptively fit the isotropic nature of the entire slice. This method also proposes a pre-training task for reconstructing the features of the entire slice, fully mining the potential semantic features contained within the entire slice itself without using any labeled information, thereby assisting the model in learning features that more effectively solve specific downstream application tasks. Compared to existing weakly supervised methods, this patent's method can fully utilize a large amount of unlabeled data to mine semantic information, achieving better performance with only a small amount of labeled data for fine-tuning. Compared to existing self-supervised learning methods, this patent's method considers the spatial location information and isotropic characteristics of the entire slice, proposing a position-aware cross-attention module and an anchor point principal direction update module, achieving superior performance. The performance evaluation criteria include the area under the receiver operating feature curve (AUC) and accuracy (ACC).

[0049] The details are shown in Table 1.

[0050] Table 1. Classification results of whole slices

[0051]

[0052] This method employs self-supervised learning, requiring only a small amount of labeled information to achieve high-precision classification. Transformer-based Multiple Instance Learning (TransMIL) and Kernel Attention Transformer (KAT) are weakly supervised learning methods, while Masked Autoencoders (MAE) and Hierarchical Image Pyramid Transformer (HIPT) are self-supervised learning methods. The proposed method was compared with existing methods on two datasets, achieving the best results under training conditions of 35% and 100% labeled data, respectively.

[0053] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0054] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for learning features of whole pathological slides based on self-supervised learning, characterized in that, Includes the following steps: Obtain the full slice image and generate initial image patches of the full slice through preprocessing; After constructing full-slice features for the initial image patch and randomly masking it, the unmasked image feature patches are selected and input into the encoder. The encoder consists of multiple encoding modules connected in series. The encoding modules perform attention weighting calculation based on the spatial location information of the image patch and generate the weighted encoding result. The step of constructing full-slice features for the initial image patch includes: The anchor point features of the whole slice image patch were identified using the DINO framework, and spatial position information was added to each initial image patch to obtain the relative distance matrix and angle matrix between each initial image patch and the anchor point features. The encoding module includes a first normalization layer, a cross-attention layer, a second normalization layer, and a multilayer perceptron connected in sequence. The cross-attention layer performs attention calculations based on the relative distance matrix and angle matrix output by the first normalization layer: in, For the first Anchor point features output by each encoding module For the first Image patch features output by each encoding module , and All are represented as the first Learnable mapping functions in each encoding module For the first The relative distance matrix of image patches in each encoding module No. The angle information matrix of image blocks relative to anchor points in each encoding module and These are the learnable embedding features corresponding to the relative distance matrix and the relative angle matrix, respectively. It is the softmax activation function; In the feature processing of multiple cascaded coding modules, kernel redirection is performed before inputting into the next coding module. Each anchor feature is sorted according to the attention score of each direction, and the direction with the highest score is selected as the new main direction, thereby updating the direction information of all image blocks. The kernel redirection includes the following steps: Obtain the attention matrix for the angle matrix and anchor point features; The angle matrix is ​​weighted by the attention matrix of the anchor point features, the attention scores of each angle are calculated and sorted, and the highest score is selected as the main direction for angle update to obtain the updated angle. Subtract the angle matrix from the updated angle to obtain the new angle matrix; Initialize the masked image patch features and combine them with the encoding result based on the position information; input the combined image patch features into the decoder to obtain the decoding result; The reconstruction loss is calculated based on the initial image patch and the decoding result, and the model is pre-trained.

2. The method for learning features of whole pathological slides based on self-supervised learning according to claim 1, characterized in that, The steps also include: After pre-training, parameter fine-tuning is performed: the parameters of the pre-trained encoder framework network are extracted and used as the full-slice feature extractor; the full-slice image is input, and the features extracted by the encoder and the full-slice labels are input into the classifier, and the network is trained using the cross-entropy loss function.

3. A pathological whole-slice feature extraction network model based on self-supervised learning, characterized in that, The pathological whole-slice feature learning method according to any one of claims 1 or 2 is adopted, including an input layer, an encoder, a decoder and an output layer; the input layer receives the whole-slice image and generates image blocks; the image blocks are sequentially encoded and decoded by the encoder and the decoder and then the whole-slice features are output through the output layer. The encoder consists of multiple encoding modules connected in series. The encoding modules perform attention weighting calculations based on the spatial location information of the image blocks and generate weighted encoding results.

4. The pathological whole-slice feature extraction network model based on self-supervised learning according to claim 3, characterized in that, The encoding module includes a first normalization layer, a cross-attention layer, a second normalization layer, and a multilayer perceptron connected in sequence.

5. The pathological whole-slice feature extraction network model based on self-supervised learning according to claim 3, characterized in that, The decoder is composed of multiple decoding modules connected in series, and the network structure of the decoding modules is the same as that of the encoding modules.

Images