Full-view digital slide image classification method integrating multi-scale feature context

Abstract

The present invention discloses a full-view digital slice image classification method integrating multi-scale feature context. In view of the problems that conventional classification methods are poor in accuracy due to the large size, multi-level resolution, and difficulty in feature extraction of full-view digital images, the present invention combines contrastive learning, multi-scale features, and self-attention mechanism to obtain a full-view digital image classification model with better effect. First, the collected full-view digital images are segmented at different resolutions to obtain slices with uniform size and low information content. Secondly, a feature extraction network with feature decoupling capability is trained for the slices obtained at different resolutions using contrastive learning to perform feature extraction. The obtained multi-scale features are further integrated, and the feature map is reconstructed as the input of the network. Finally, a network with a self-attention mechanism is used to obtain the contextual information of the input data, thereby improving the global representation capability of the features and making the prediction results more accurate.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and medical data analysis and mining, and in particular to a full-view digital slice image classification method based on fusion of multi-scale feature contexts. Background Art

[0002] In recent years, with the development of deep learning (DL) technology, various modal image classification models have been proposed one after another. Convolutional neural networks and recurrent neural networks have also been widely introduced into the field of medical image analysis, and have achieved good results in conventional medical images such as CT and MRI. With the development of digital imaging equipment, full-view digital slice technology has begun to replace traditional glass slices and become the mainstream choice for obtaining tissue pathology images. Digital pathology slices have the following characteristics: large size, high information content, and multi-level resolution. Although these characteristics provide a detailed and clear representation of the tissue environment, they also bring many inconveniences to clinicians in reading the films, and also cause certain limitations for the application of deep learning methods in full-view digital pathology slices. For example, doctors have low efficiency in reading films, poor consistency in reading films, and large size and multi-level resolution make it difficult to input neural networks for end-to-end training like natural images. Therefore, full-view digital slices are usually divided into patch images tiles of specified pixel size. One full-view digital slice image can usually obtain thousands or even tens of thousands of patch image tiles. However, the labels of full-view digital slice images often only have a single clinical diagnosis label, and there is no pixel-by-pixel ROI delineation label. Therefore, the problem of using patch images to classify full-view digital slide images is usually converted into a weakly supervised problem based on multi-instance learning. Although multi-instance learning can classify full-view digital slide images, it usually ignores the relationship between different patch images and does not utilize multi-level resolution features. Therefore, there is a certain amount of information redundancy in the training process, feature fusion is difficult, and the model is difficult to converge; the classification accuracy is not ideal; and the classification results lack interpretability. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a full-view digital slice image classification method integrating multi-scale feature contexts, so as to obtain a classification model for the full-view digital slice image.

[0004] The technical solution of the present invention is: a full-view digital slice image classification method integrating multi-scale feature context, comprising:

[0005] S1. Collect a batch of original whole slide images (WSI);

[0006] S2, performing sliding window segmentation processing on the original full-view digital slice image WSI collected by S1 at different magnifications, and performing normalization processing at the same time, and determining whether to save the patch image tile according to the set threshold, and finally obtaining multiple groups of patch image tiles corresponding to each full-view digital slice image WSI;

[0007] S3. For the multiple groups of patch image tiles with different resolutions obtained in S2, the corresponding patch feature extraction models are trained using a self-supervised contrastive learning method. The extraction model has good feature decoupling ability for patch image tiles of different categories.

[0008] S4, the patch image tiles preprocessed by S2 are grouped according to resolution, and are respectively input into the feature extraction model of the corresponding resolution obtained in S3 to extract features of all patch image tiles and store them.

[0009] S5. Use a pyramid feature fusion method to fuse the multiple features at different resolutions extracted by S4 to obtain feature data after the patch image tile features at different resolutions are fused.

[0010] S6, dividing the tile feature data obtained in S5 into data sets based on the full-view digital slice image. The full-view digital slice image WSI is randomly divided and the division results are saved. The training set D trn Used for training and validation set D of classification model val For verification of the model effect, the training set and validation set contain all patch images corresponding to each full-field digital slice image WSI.

[0011] S7, in the data loading stage, the patch image tile in S6 is restored according to its position in the full-view digital slice image WSI to obtain a new image representation WSI′, and the WSI′ is subjected to center cropping and supplementation operations and size normalization.

[0012] S8, the image obtained in S7 is represented by WSI′ according to D trn and D val The weight of the context information of the input WSI′ at the tile level of the patch image is obtained through the self-attention mechanism. This weight will affect the prediction process and classification results of the model, thereby obtaining the final output. trn For training the model, D val Validate the performance of the model in each batch and save it in D val The parameters that achieve the best performance are used as the final parameters of the model.

[0013] In S2, each full-view digital slice image WSI is subjected to a sliding window segmentation process, including:

[0014] S2.1, obtaining the image sizes w and h of the original full-field digital slice image WSI at different magnifications; wherein w and h are the width and length of the image at the specified magnification, respectively;

[0015] S2.2. Calculate the number of window sliding times for the image size at the specified magnification obtained:

[0016]

[0017] In formula (1.1), win w Indicates the window width, N w The number of times the window slides horizontally, and is rounded down;

[0018]

[0019] In formula (1.2), win h Indicates the window length, N h The number of vertical sliding times of the window, rounded down;

[0020] S2.3. For each window calculated in S2.2, determine whether the tissue in the window reaches the threshold for saving the patch image tile during sliding segmentation. If it reaches the threshold, normalize it and save it, otherwise discard it.

[0021] The patch feature extraction model construction method in S3 includes:

[0022] S3.1. For the contrastive learning described in S3, the SimCLR framework is used for its implementation. The SimCLR framework first performs data augmentation on the input patch image tile, and then inputs the original image and augmented data in the same batch as input data into the Encoder of the SimCLR framework for encoding, and obtains the feature representation h of the original image and augmented data. i ,h j , and then use two Projection Heads to i ,h j Continue with feature representation and get z i , z j .

[0023] z i =g(h i )=W (2) σ(W (1) h i ) (1.3)

[0024] In formula (1.4), h i is the feature representation of the Encoder output, W (1) , W (2) There are two learnable parameter layers, σ is the nonlinear activation function ReLU, g() is the Projection Head, z i is the final feature representation of the original image.

[0025] S3.2. For the final feature representation z described in S3.1 i , z j , using the maximum similarity loss function

[0026]

[0027] In formula (1.5), z i is the final feature representation of the original image, z j is the feature representation of the enhanced image, z k is the feature representation of other irrelevant images, τ is an adjustable parameter used to scale the input, N represents the batch size; i, j, k represent the i-th image in batch N, the enhanced image j obtained by image i, and other irrelevant images k in the batch, respectively.

[0028] S3.3. The specific implementation of the data enhancement method described in S3.1 is to use the transforms.ColorJitter() method in the torchvision deep learning package in the pytorch deep learning framework [import the torchvision package in the deep learning framework PyTorch, and use the transforms.ColorJitter() method in the package] to randomly change the brightness, contrast, saturation and hue of the image; the transforms.RandomGrayscale() method grayscales the image according to the specified probability; the transforms.Resize() method scales the image to the size required by the network input, the GaussianBlur() method Gaussian blurs the image, and the transforms.RandomHorizontalFlip() method randomly crops the image to the size required by the network input.

[0029] S3.4. For the Encoder described in S3.1, use the EfficientNet V2 Small neural network with the classifier part removed, use the value of the loss function in S3.2 as the evaluation indicator of the model, select the parameter with the lowest loss on the validation set and save it, and load the parameter as the pre-trained weight in S4, and use the weight to extract the features of the patch image tile.

[0030] The features of the patch image tiles at different resolutions obtained in S4 are represented by vectors, and the size of the vector is specified as needed and is recorded as dim.

[0031] The specific method for fusing all patch image tiles at different resolutions in S5 is as follows: query all corresponding patch image tiles at high magnifications according to the position number of the low-magnification patch image tile, obtain the weight through inner product operation, multiply the low-magnification patch image tile by the weight, and add the result to the corresponding high-magnification patch image to obtain a new feature representation new_tile.

[0032] new_tile=(<low_tile,high_tile> ×low_tile)+high_tile (1.5) In formula (1.5), low_tile represents the patch image tile at low resolution, high_tile represents the patch image tile at high resolution, and new_tile is still represented by a vector, and its size is the same as that of high_tile.

[0033] The new image representation WSI′ is obtained in S7, and the center cropping, supplementation, and size normalization operations are performed, including:

[0034] S7.1. For each patch image new_tile obtained after feature fusion of each WSI, read the numbers one by one to obtain its maximum row number max_r and column number max_c in the full-view digital slice image at 20x magnification. Generate an empty tensor WSI' according to the maximum row and column numbers, the shape of which is (max_r, max_c, dim), and fill new_tile in the corresponding position of WSI' according to the number.

[0035] S7.2. For the size normalization in S7, the specific implementation method is: the new feature representation WSI′ obtained in S7.1 is centrally cropped according to a fixed size. According to the selected size, the WSI′ that is less than the size is padded at the periphery, and the padded value is 0. The padded size is the difference between the actual size and the selected size, so as to obtain the final feature representation WSI′.

[0036] The range of high and low magnifications is not the same for different data, but all zoom levels can be obtained by using the level_demensions method in the openslide package, with the median of the zoom level as the boundary. Generally speaking, in the present invention, when the magnification is less than or equal to 10 times, it is considered to be a low magnification; when it is greater than 10 times, it is considered to be a high magnification.

[0037] The beneficial effects of the present invention are as follows: Taking into account the problems that conventional prediction methods use a single resolution and there is no correlation between patch image tiles, resulting in low model accuracy and poor generalization, the present invention first adopts a self-supervised contrast learning method to train a patch image tile feature extraction network for the full-view digital slice image WSI without expert-annotated ROI area, and performs feature decoupling on patch image tiles containing different tissues. The model is then applied at different magnifications to obtain differentiated tile-level features. By introducing a pyramid feature fusion method and combining the idea of ​​feature map reconstruction, the obtained results are made more interpretable. Using a self-attention mechanism, the contextual information between different tiles is fused to predict the final result, so as to reduce the complexity of the problem and improve the model's robustness, interpretability and prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of the present invention;

[0039] Figure 2 This is the confusion matrix diagram of the classification results of the ablation experiment for contrastive learning;

[0040] Figure 3 This is the confusion matrix diagram of the classification results of the ablation experiment for feature fusion;

[0041] Figure 4 It is a confusion matrix diagram of the classification results of the ablation experiment for context;

[0042] Figure 5 It is a confusion matrix diagram of the classification results of the present invention. DETAILED DESCRIPTION

[0043] The invention is further described below in conjunction with the accompanying drawings and embodiments, but the content of the invention is not limited to the scope of the embodiments.

[0044] Example 1

[0045] A context-related full-view digital image multi-scale slice classification modeling method, comprising:

[0046] S1. Collect a batch of original whole slide images (WSI);

[0047] S2, performing sliding window segmentation processing on the original full-view digital slice image WSI collected by S1 at 5 times and 20 times resolution, and performing normalization processing at the same time, and determining whether to save the patch image tile according to the set threshold, and finally obtaining two groups of patch image tiles corresponding to each full-view digital slice image WSI;

[0048] S3: For the two sets of patch image tiles with different resolutions obtained in S2, the corresponding patch feature extraction models are trained using the self-supervised contrastive learning method. This model has good feature decoupling ability for patch image tiles of different categories.

[0049] S4, the patch image tiles preprocessed by S2 are grouped according to resolution, and are respectively input into the feature extraction model of the corresponding resolution obtained in S3 to extract features of all patch image tiles and store them.

[0050] S5. The feature at 5 times the resolution extracted by S4 and the multiple features at 20 times the resolution corresponding to the feature are fused using a pyramid feature fusion method to obtain feature data after the patch image tile features at different resolutions are fused.

[0051] S6, dividing the tile feature data obtained in S5 into data sets based on the full-view digital slice image. The full-view digital slice image WSI is randomly divided and the division results are saved. The training set D trn Used for training and validation set D of classification model val For verification of the model effect, the training set and validation set contain all patch images corresponding to each full-field digital slice image WSI.

[0052] S7, in the data loading stage, the patch image tile in S6 is restored according to its position in the full-view digital slice image WSI to obtain a new image representation WSI′, and the WSI′ is subjected to center cropping and supplementation operations and size normalization.

[0053] S8, the image obtained in S7 is represented by WSI′ according to D trn and D val The weight of the context information of the input WSI′ at the tile level of the patch image is obtained through the self-attention mechanism. This weight will affect the prediction process and classification results of the model, thereby obtaining the final output. trn For training the model, D val Validate the performance of the model in each batch and save it in D val The parameters that achieve the best performance are used as the final parameters of the model.

[0054] Optionally, the step of performing sliding window segmentation processing on each full-view digital slice image WSI in S2 includes:

[0055] S2.1. Use the openslide-python open source package to process the full-view digital slice images. The openslide-python package is a Python interface to the openslide C language library, which can process full-view digital slices. Use the DeepZoomGenerator in the package to generate the object of the full-view digital slice image, call the level_demensions method, and pass in the specified zoom level in the parameter to obtain the image size at the corresponding resolution. The maximum resolution of the full-view digital slice used in this example is 40×, so the corresponding levels at 20× magnification and 5× magnification are L all -1,L all -3. The obtained image size is returned as w and h, which are the width and length of the image at the specified magnification respectively;

[0056] S2.2. Calculate the number of window sliding times for the image size at the specified magnification obtained:

[0057]

[0058] In formula (1.1), win w Indicates the window width, N w The number of times the window slides horizontally, and is rounded down;

[0059]

[0060] In formula (1.2), win h Indicates the window length, N h The number of vertical sliding times of the window, rounded down;

[0061] S2.3. For each window calculated in S2.2, determine whether the organization in the window reaches the threshold for saving the patch image tile during sliding segmentation. If it reaches the threshold, normalize it and save it as a png format image, otherwise discard it.

[0062] Optionally, the patch feature extraction model construction process in S3 is as follows:

[0063] S3.1. For the contrastive learning described in S3, the SimCLR framework is used for its implementation. First, the collected full-view digital slice images are divided into a training set D according to an 8:2 ratio. contra-train and validation set D contra-valThe divided data set is input into the SimCLR framework, which first performs data augmentation on the input patch image tile, and then takes the original image and augmented data in the same batch as input data and inputs them into the Encoder of the SimCLR framework for encoding to obtain the feature representation h of the original image and augmented data. i ,h j , and then use two Projection Heads to h i ,h j Continue with feature representation and get z i , z j .

[0064] z i =g(h i )=W (2) σ(W (1) h i ) (1.3)

[0065] In formula (1.3), h i is the feature representation of the Encoder output, W (1) , W (2) There are two learnable parameter layers, σ is the nonlinear activation function ReLU, g() is the Projection Head, z i is the final feature representation of the original image;

[0066] S3.2. For the final feature representation z described in S3.1 i , z j , using the maximum similarity loss function

[0067]

[0068] In formula (1.4), z i is the final feature representation of the original image, z j is the feature representation of the enhanced image, z k is the feature representation of other unrelated images, τ is an adjustable parameter used to scale the input, and N represents the batch size.

[0069] S3.3. The specific implementation of the data augmentation method described in S3.1 is to use the transforms.ColorJitter() method in the torchvision deep learning package in the pytorch deep learning framework to randomly change the brightness, contrast, saturation and hue of the image; the transforms.RandomGrayscale() method randomly converts the image to grayscale with a probability of 0.2; the transforms.Resize() method scales the image to 224×224 size, the GaussianBlur() method performs Gaussian blur on the image, and the transforms.RandomHorizontalFlip() method randomly crops the image to 224×224 size.

[0070] S3.4. For the Encoder described in S3.1, use the EfficientNet V2 Small neural network with the classifier part removed, use the value of the loss function in S3.2 as the evaluation indicator of the model, select the parameter with the lowest loss on the validation set and save it, and load the parameter as the pre-trained weight in S4, and use the weight to extract the features of the patch image tile.

[0071] Optionally, the features of the patch image tiles at different resolutions obtained in S4 are represented by vectors, and the sizes of the vectors are all 1×1280.

[0072] Optionally, the specific method of fusing all patch image tiles at different resolutions in S5 is: query all patch image tiles at 20 times the resolution corresponding to the position number of the patch image tile at 5 times the resolution, obtain the weight by inner product operation, multiply the patch image tile at 5 times the resolution by the weight, and add the result to the 20 times patch image to obtain a new feature representation new_tile.

[0073] new_tile=(<tile_5,tile_20> ×tile_5)+tile_20 (1.5)

[0074] In formula (1.5), tile_5 represents the patch image tile at 5 times the resolution, and tile_20 represents the patch image tile at 20 times the resolution. new_tile is still represented by a vector, and its size is the same as that of tile_20. All operations are completed using the numpy package of Python.

[0075] Optionally, the obtaining of the new image representation WSI′ and performing center cropping, supplementation, and size normalization operations in S7 includes:

[0076] S7.1. For each patch image new_tile obtained after feature fusion of each WSI, read the numbers one by one, split the file name using str.split(), and obtain the maximum row number max_r and column number max_c of the full-view digital slice image at 20x magnification. Generate an empty tensor WSI′ based on the maximum row and column numbers, the shape of which is (max_r, max_c, 1280), and fill new_tile in the corresponding position of WSI′ according to the number.

[0077] S7.2. For the size normalization in S7, the specific implementation method is: the new feature representation WSI′ obtained in S7.1 is centrally cropped according to a fixed size. The selected size is 48×48, and the WSI′ that is less than this size is padded at the periphery, and the padded value is 0. The padded size is the difference between the actual size and the selected size, so as to obtain the final feature representation WSI′.

[0078] Optionally, the model structure with the self-attention mechanism described in S8 refers to Vision TransformerBase, and is changed on the input channel to match WSI′.

[0079] Traditional multi-instance-based learning methods can generally be divided into two categories. One is to select only instances with higher scores for training in each round. The disadvantage of this method is that the data involved in each training is too small, so a large amount of training data is required to obtain better results. The other is to consider each instance as an independent and identically distributed individual. Although this method greatly increases the data involved in training compared to the first method, it ignores the correlation between different instances and does not take into account the global spatial information of the instance itself. Although the self-attention mechanism can calculate the correlation between all instances, it was first proposed in natural language processing and cannot be applied to the field of computer vision. The Vision Transformer proposed a good solution to this problem. The Vision Transformer mainly consists of three parts, namely Patch+PositionEmbedding, Transformer Encoder, and MLP Head. Among them, the input image is first divided by Linear Projection of Flattened Patches, and becomes an embedded vector token after projection, which is convenient for adding position encoding. Secondly, through the Transformer Encoder module, the multi-head self-attention mechanism is used to calculate the context information weights between different tokens. Finally, a simple MLP is used to calculate the features containing contextual weight information to obtain the final classification result.

[0080]

[0081] Formula (1.6) is the overall process of Vision Transformer. class is a category token, is the Nth patch cut from the input image P, E is the linear layer, E pos represents the position code, z 0 Represents the original input of the current network; z′ in b 1 is the output of the multi-head self-attention mechanism in the lth transformer block; in formula c, z 1 is the output of the multilayer perceptron in the lth transformer block. In formula d, y is the class token in the output of the last transformer block. prediction results.

[0082]

[0083] In formula (1.7), formula (a) Attention(Q, K, V) is the attention query, Q, K, V query matrix, key matrix, value matrix, is the scaling value; Formula (b) is the multi-head attention mechanism, head h represents the output of the hth head, W O is the transformation matrix; Formula (c) is a supplementary explanation of each head in Formula (b), where Q, K, V have the same meaning as in Formula (a), and W i Q , W i K , W i V are the corresponding transformation matrices respectively.

[0084] During the training process, the loss function uses the classic cross entropy loss function of the classification model; the optimizer selects the AdamW optimizer, which is simple to implement and efficient in calculation; in the verification part, the model with the highest classification accuracy is saved.

[0085] The classification accuracy (ACC):

[0086]

[0087] In formula (1.8), TP represents the number of samples predicted as positive; TN represents the number of samples predicted as negative; FP represents the number of samples predicted as positive; and FN represents the number of samples predicted as negative.

[0088] Recall: The ratio of correctly classified positive samples to all positive samples.

[0089]

[0090] Specificity (SPE): The ratio of correctly classified negative samples to all negative samples.

[0091]

[0092] Example 2: Figure 1 As shown, an optional specific method of the present invention is described in detail below.

[0093] Step 1: Collect whole-field digital pathology slide images (Whole Slide Image, WSI) of gastric cancer patients stained with HE in the Cancer Hospital. The format is svs format, and the size of each image ranges from 2GB to 4GB. The data labels are clinical diagnosis labels, and there are no pixel-by-pixel ROI delineation labels.

[0094] Step 2: For the WSI in svs format, the Python openslide package was used to perform sliding window segmentation on the original WSI of each patient. The window size was 512x 512, and the tissue threshold within the window was 15%. The process was performed at 5× and 20× resolutions, and preprocessing such as staining normalization was performed on the areas that met the requirements. Finally, two groups of tiles corresponding to each WSI were obtained, and each tile at 5× resolution corresponded to 0 to 16 tiles at 20× resolution.

[0095] The DeepZoomGenerator method of the openslide package in Python is used to obtain the total number of pyramid layers of the original full-view digital slice image WSI and the size of the target slice layer, and then the number of window sliding times is calculated according to the set window size to segment the full-view digital slice image WSI of the selected target slice layer; the window involved in the calculation of the number of window sliding times w 、win h Both are set to 512;

[0096] Step 3: Use the self-supervised contrastive learning framework SimCLR to train two tile feature extraction networks for the tile images at the two resolutions collected in the second step, and save the weight file with the lowest loss on the validation set as the model for extracting tile features in the fourth step.

[0097] (1) First, all tiles are divided into training set D according to the ratio of 8:2 based on the full-view digital slices. contra_train , validation set D contra_val

[0098] (2) Use the transforms.ColorJitter() method in the torchvision deep learning package in the pytorch deep learning framework to randomly change the brightness, contrast, saturation and hue of the image, the transforms.RandomGrayscale() method to randomly convert the image to grayscale with a probability of 0.2, the transforms.Resize() method to scale the image to 224×224, the GaussianBlur() method to perform Gaussian blur on the image, and the transforms.RandomHorizontalFlip() method to randomly crop the image to 224×224; for the validation set D val Only transforms.Resize is performed to unify the image size to 224x224.

[0099] (3) EfficientNet V2 Small without the classifier is selected as the encoder of SimCLR. EfficientNetV2 is the latest version of the EfficientNet series, which improves many problems existing in the V1 version. It adopts a progressive learning method, which dynamically adjusts the regularization method according to the size of the training image. It has faster training speed, fewer parameters, and higher model accuracy. In addition, the neural network structure search NAS technology is used to search for the best combination of MBConv and Fused-MBConv to solve the problem of slow use of deep separable convolution in shallow layers.

[0100] (4) Use the EfficientNet V2 Small weights for 20× and 5× resolutions obtained in (3) to extract features from the patch image tiles at 20× and 5× resolutions, and save the obtained features separately. Among them, EfficientNet V2 Small only uses the feature extractor part, and the size of the extracted patch image feature vector is 4×1280. This example reduces the dimension of the feature vector and uses the adaptive average pooling layer provided by pytorch for pooling. The size of the feature vector after pooling is 1×1280, and the feature vector of this size is saved in the form of a dictionary. The key of the dictionary is the name of the saved patch image, and the value is the value of the 1×1280 feature vector extracted from the patch image corresponding to the name.

[0101] Step 4: Read all feature vectors obtained in step 3 (4) and perform feature fusion. Record the features at different resolutions belonging to the same full-view digital slice image, and find the corresponding feature vector at 20× by naming the feature vector at 5× magnification. Perform inner product on the two sets of feature vectors and calculate the similarity weight. Then multiply the similarity weight with the 5× feature vector, and add the result to the 20× feature vector to obtain the feature vector after the fusion of features at different resolutions, and save the vector. The specific formula is shown in formula (1.1) in Example 2,

[0102] new_tile=(<tile_5,tile-20> ×tile_5)+tile-20 (1.1)

[0103] Among them, new_tile is the new feature after feature fusion, tile_5 and tile_20 are the feature vectors saved in the fourth step at 5× resolution and 20× resolution respectively.

[0104] All the saved patch image feature vectors are divided into datasets according to the full-view digital slice image WSI name, and 70% is used as the training set D for downstream tasks. train , 30% as the validation set D for downstream tasks val .

[0105] Step 5: In the data loading Dataset stage, all feature vectors after feature fusion are reconstructed into feature maps with the full-view digital slice image WSI as the unit to achieve the effect of simulating natural image input. The specific operation is to customize the data loading class My_Dataset to inherit the torch.utils.data.dataset class of pytorch to realize the data loading function. For each patch image new_tile obtained after feature fusion of each WSI, read them one by one, and use the str.split() method to obtain its maximum row number max_r and column number max_c under the full-view digital slice image with 20 times magnification. Generate an empty tensor WSI′ according to the maximum row and column numbers. The shape of the tensor is (max_r, max_c, 1280), and fill new_tile in the corresponding position of WSI′ according to the number. This tensor will be input into the downstream module as a new feature map to obtain the classification results.

[0106] Step 6: In the model training and validation phase, the data obtained in step 5 are input into the corresponding models according to the differences between the training set and the validation set to obtain the model prediction output. Then use the validation set D val , verify the performance of the model in each batch of training, and select valThe best performing parameters are used as the parameters of the classification model, and the confusion matrix of the model classification results is drawn, and the ACC, SEN, and SPE of the model are calculated.

[0107] The model with the best accuracy on the validation set is saved to obtain a full-view digital slice image classification model that integrates multi-scale feature context. The main hyperparameters of the experiment are set to epoch=500, the optimizer selects the AdamW optimizer, the learning rate size is 2e-4, the weight decay is 1e-4, and the automatic adjustment of the learning rate strategy is not used. train For example, the flow of data in the Vision Transformer Base model is first divided by Linear Projection of Flattened Patches, where the size of the divided patches is 16×16. After projection, it becomes an embedded vector token that is convenient for adding position encoding, and adding a learnable class token for subsequent modules to perform classification probability prediction. Secondly, through the Transformer Encoder module, the multi-head self-attention mechanism is used to calculate the context information weights between different tokens. The implementation formula of the multi-head self-attention mechanism is shown in Equation 1.19. Finally, through a simple multi-layer perceptron, the features containing context weight information are calculated and predicted based on the class token to obtain the final classification prediction result.

[0108] The above method is applicable to the construction of a full-view digital slice image classification method based on the fusion of multi-scale feature context. This example verifies the effectiveness of the present invention by combining HE-stained full-view digital pathological slice images of gastric cancer patients in a cancer hospital. The batch of data contains a total of 123 WSI original images, of which 85 are used as training sets and 38 are used as validation sets. The formats are all in svs format, and the size of each image ranges from 2GB to 4GB.

[0109] The implementation case of the present invention adopts accuracy ACC, sensitivity SEN, and specificity SPE as evaluation indicators of model prediction performance. The calculation formulas of the three indicators have been given in formulas (1.8) to (1.10) of Example 1. The values ​​of the three indicators are all between 0 and 1. The closer to 1, the better the performance.

[0110] Table 1 Comparison of gastric cancer recurrence prediction performance evaluation in ablation experiments of this method

[0111]

[0112] According to Table 1, it can be seen that the classification method proposed by the present invention is better than the method after the ablation experiment itself. The full-view digital slice image classification method based on the fusion of multi-scale feature context proposed by the present invention has better performance, and each module plays a role in improving the accuracy. Not only does it decouple the features of patch images with small feature differences through contrast learning, which greatly reduces the complexity of training; it also takes into account the multi-scale features of full-view digital slice images, imitates the diagnosis process of pathologists, and integrates multi-scale features to accelerate the convergence speed of the model and improve the accuracy of the model; and reconstructs the feature map, combines the self-attention mechanism to obtain the contextual information between patch image tiles, and improves the model robustness, interpretability and prediction accuracy. The present invention fully reflects its own advantages.

[0113] like Figure 2-Figure 5 As shown, the classification results obtained by the method of the present invention and its comparative method are shown. From the classification results, it can be seen that the prediction accuracy of the present invention is higher than that of other models.

[0114] The above case is used to illustrate the effectiveness and feasibility of the present invention. The focus of the present invention is to use contrastive learning method to decouple features and reconstruct feature maps after fusing multi-scale features. However, the implementation method of contrastive learning, the encoder encoder is not limited to the fixed network structure in the implementation case. The self-attention mechanism is then used to establish the relationship between context information.

[0115] The idea of ​​introducing the self-attention mechanism is to solve the problem of poor model performance, lack of interpretability, and poor generalization performance caused by conventional multi-instance learning without paying attention to contextual information, rather than targeting a specific case, and not limiting the present invention. Any modification or improvement of the present invention that is not made within the spirit and scope of protection of the claims will be limited by the scope of protection of the present invention.

Claims

1. A full-view digital slide image classification method integrating multi-scale feature context, characterized in that: include: S1. Collect a batch of original full-field digital slice images WSI; S2, performing sliding window segmentation processing on the original full-view digital slice image WSI collected by S1 at different magnifications, performing normalization processing at the same time, and determining whether to save the patch image tile according to the set threshold, and finally obtaining multiple groups of patch image tiles corresponding to each full-view digital slice image WSI; S3, for the multiple groups of patch image tiles with different resolutions obtained in S2, respectively, a corresponding patch feature extraction model is trained using a self-supervised contrastive learning method, wherein the extraction model has good feature decoupling capability for patch image tiles of different categories; S4, grouping the patch image tiles preprocessed by S2 according to the resolution, and inputting them into the feature extraction model of the corresponding resolution obtained by S3, extracting the features of all the patch image tiles, and storing them; S5, using a pyramid feature fusion method to fuse the multiple features at different resolutions extracted by S4, and obtain feature data after the patch image tile features at different resolutions are fused; The specific method for fusing all patch image tiles at different resolutions in S5 is as follows: query all patch image tiles at high magnifications corresponding to the low magnification patch image tile according to its position number, obtain the weight by inner product operation, multiply the low magnification patch image tile by the weight, and add the result to the corresponding high magnification patch image to obtain a new feature representation new_tile; new_tile=(<low_tile,high_tile>×low_tile)+high_tile (1.5) In formula (1.5), low_tile represents the patch image tile at low resolution, high_tile represents the patch image tile at high resolution, and new_tile is still represented by a vector, and its size is the same as that of high_tile; S6, taking the full-view digital slice image as a unit, dividing the tile feature data obtained in S5 into a data set; randomly dividing the full-view digital slice image WSI, and saving the division results, and its training set D trn Used for training and validation set D of classification model val For verification of the model effect, the training set and validation set contain all patch images corresponding to each full-view digital slice image WSI; S7, in the data loading stage, the patch image tile in S6 is restored according to its position in the full-view digital slice image WSI to obtain a new image representation WSI', and the WSI' is subjected to center cropping and supplementation operations and size normalization; For each patch image new_tile obtained after feature fusion of each WSI, read the numbers one by one to obtain its maximum row number max_r and column number max_c in the high-magnification full-view digital slice image; generate an empty tensor WSI' according to the maximum row and column numbers, the shape of which is (max_r, max_c, dim), and fill new_tile in the corresponding position of WSI' according to the number; dim is the size of the vector; S8, the image obtained in S7 is represented by WSI' according to D trn and D val The weight of the context information of the input WSI' at the tile level of the patch image is obtained through the self-attention mechanism; this weight will affect the prediction process and classification results of the model, thereby obtaining the final output; where D trn For training the model, D val Validate the performance of the model in each batch and save it in D val The parameters that achieve the best performance are used as the final parameters of the model.

2. The full-view digital slide image classification method based on fusion of multi-scale feature context according to claim 1 is characterized in that: In S2, each full-view digital slice image WSI is subjected to a sliding window segmentation process, including the following steps: S2.1, obtaining the image sizes w and h of the original full-field digital slice image WSI at different magnifications; wherein w and h are the width and length of the image at the specified magnification, respectively; S2.

2. Calculate the number of window sliding times for the image size at the specified magnification obtained: In formula (1.1), win w Indicates the window width, N w The number of times the window slides horizontally, and is rounded down; In formula (1.2), win h Indicates the window length, N h The number of vertical sliding times of the window, rounded down; S2.

3. For each window calculated in S2.2, determine whether the tissue in the window reaches the threshold for saving the patch image tile during sliding segmentation. If it reaches the threshold, normalize it and save it, otherwise discard it.

3. The full-view digital slide image classification method based on fusion of multi-scale feature context according to claim 1 is characterized in that: The patch feature extraction model construction method in S3 includes the following steps: S3.

1. For the contrastive learning described in S3, the implementation method uses the SimCLR framework; the SimCLR framework first performs data augmentation on the input patch image tile, and then inputs the original image and augmented data in the same batch as input data into the Encoder of the SimCLR framework for encoding, and obtains the feature representation h of the original image and augmented data. i ,h j , and then use two ProjectionHeads to h i ,h j Continue with feature representation and get z i ,z j ; z i =g(h i )=W (2) σ(W (1) h i ) (1.3) In formula (1.3), h i is the feature representation of the Encoder output, W (1) , W (2) There are two learnable parameter layers, σ is the nonlinear activation function ReLU, g() is the Projection Head, z i is the final feature representation of the original image; S3.

2. For the final feature representation z described in S3.1 i ,z j , using the maximum similarity loss function In formula (1.5), z i is the final feature representation of the original image, z j is the feature representation of the enhanced image, z k is the feature representation of other irrelevant images, τ is an adjustable parameter used to scale the input, N represents the batch size; i, j, k represent the i-th image in batch N, the enhanced image j obtained by image i, and other irrelevant images k in the batch respectively; S3.

3. The specific implementation of the data enhancement method described in S3.1 is as follows: use the transforms.ColorJitter() method in the torchvision deep learning package in the pytorch deep learning framework to randomly change the brightness, contrast, saturation and hue of the image; the transforms.RandomGrayscale() method grayscales the image according to the specified probability; the transforms.Resize() method scales the image to the size required for network input; the GaussianBlur() method Gaussian blurs the image; the transforms.RandomHorizontalFlip() method randomly crops the image to the size required for network input; S3.

4. For the Encoder described in S3.1, use the EfficientNetV2 Small neural network with the classifier part removed, use the value of the loss function in S3.2 as the evaluation index of the model, select the parameter with the lowest loss on the validation set and save it, and load the parameter as the pre-trained weight in S4, and use the weight to extract the features of the patch image tile.

4. The full-view digital slide image classification method based on fusion of multi-scale feature context according to claim 1, characterized in that: The features of the patch image tiles at different resolutions obtained in S4 are represented by vectors, and the size of the vector is specified as needed and is recorded as dim.

5. The full-view digital slide image classification method based on fusion of multi-scale feature context according to claim 1, characterized in that: The steps of obtaining a new image representation WSI' in S7, performing center cropping, supplementation, and size normalization operations include: For the size normalization in S7, the specific implementation method is: the new feature representation WSI' is centrally cropped according to a fixed size; based on the selected size, the WSI' that is less than the size is filled on the periphery, the filling value is 0, and the filling size is the difference between the actual size and the selected size, so as to obtain the final feature representation WSI'.

6. The full-view digital slide image classification method based on fusion of multi-scale feature context according to claim 1, characterized in that: Accuracy ACC, sensitivity SEN, and specificity SPE are used as evaluation indicators of model prediction performance. The values ​​of the three indicators are all between 0 and 1. The closer to 1, the better the performance. The classification accuracy ACC: In formula (1.6), TP represents the number of samples predicted as positive; TN represents the number of samples predicted as negative; FP represents the number of samples predicted as positive; FN represents the number of samples predicted as negative. Recall rate: the ratio of correctly classified positive samples to all positive samples; Specificity SPE: The proportion of correctly classified negative samples to all negative samples;

Images