A medical image classification method based on dynamic sparse contrastive learning

Abstract

The application belongs to the field of pathological image classification, and particularly relates to a medical image classification method based on dynamic sparse contrast learning, which comprises the following steps: obtaining a medical image to be classified, and pre-processing the medical image; inputting the pre-processed image into a dynamic sparse multi-instance learning framework to obtain the weight of the dynamic sparse multi-instance learning framework under different scales; loading the weight under different scales into a cross-scale contrast learning backbone network, learning the 1024-dimensional features, and obtaining features of different scales; inputting the features of different scales into a cross-scale contrast learning framework to obtain fusion features; and inputting the fusion features into a classifier for classification; the dynamic sparse layer in the model is used to adaptively filter and mine key instances from all instances in the whole slide image, so as to help the model make scientific and accurate decisions quickly, and thus the generalization and classification ability of the model is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of pathological image classification, and in particular relates to a medical image classification method based on dynamic sparse contrast learning. Background Art

[0002] The multi-instance learning method currently used most in pathology image classification is to aggregate the features extracted from thousands of small slices split from a single full-slice image into full-slice-level features by combining various efficient attention mechanisms with multi-instance learning for downstream classification tasks.

[0003] Previous multi-instance learning methods can effectively capture the correlation between multiple instance-level slice features through global self-attention or local attention mechanisms. However, in many challenging whole-slide pathology images, the information that truly determines the label of each slice typically accounts for only a tiny fraction (<10%) of the total information. This means that the image information necessary for clinical diagnosis is too small compared to the information contained in the whole-slide image. Specifically, challenging slices often contain very small tumor regions. Furthermore, the effective area varies from slice to slice. WSI diagnosis often only requires determining the small region where the tumor is located. This discrepancy between the essential information required for clinical diagnosis and the vast amount of information contained in WSI not only significantly increases the workload and time consumption of professional pathologists, but also affects the consistency and accuracy of diagnoses due to subjective and fatigued observations. Similarly, this imbalance in the number of instances in the bag can lead to biased models that overactivate or confuse the true determinants of the target.

[0004] At the same time, in previous studies, there have been many works exploring how to combine multi-scale information to simulate the experience and expertise of human pathologists. However, the multi-scale feature fusion task is still plagued by the following problems that remain to be solved: 1. Semantic gap: Features extracted at different scales contain semantic information from different scales, which may have inconsistent or incompatible parts; this may cause the model to capture confusing semantic representations when fusing multi-scale features. 2. Spatial alignment problem: Multi-scale features from different scales may not be completely aligned in space, so interpolation or padding operations may be required when performing feature fusion; these operations may introduce additional noise or inconsistent information. 3. Difficulty in weight allocation: In the multi-scale feature fusion task, it is often complex to determine the reasonable weights of features from different scales; due to the different contributions of different scales, the determination of weights may become complicated; if the weights are improperly allocated, features at some scales may be ignored or overly dominate the fusion and final results. Summary of the Invention

[0005] To solve the problems in the prior art, the application provides a medical image classification method based on dynamic sparse contrast learning, which comprises the following steps: acquiring a medical image to be classified, and preprocessing the medical image; inputting the preprocessed image into a classification model based on a dynamic sparse multi-instance learning framework to obtain a classification result; and visualizing the classified medical image.

[0006] The classification model based on the dynamic sparse multi-instance learning framework is trained, and the training comprises the following steps:

[0007] S1, acquiring a full slice image dataset, preprocessing the images in the full slice image dataset, and obtaining 1024-dimensional features;

[0008] S2, dividing all the 1024-dimensional features into a training set and a test set;

[0009] S3, inputting the features in the training set into a dynamic sparse multi-instance learning framework DSTMIL for training, and obtaining the weights of the dynamic sparse multi-instance learning framework under different scales;

[0010] S4, loading the weights of the dynamic sparse multi-instance learning framework under different scales into a cross-scale contrast learning backbone network, learning the 1024-dimensional features, and obtaining features under different scales;

[0011] S5, inputting the features under different scales into a cross-scale contrast learning framework for training, fusing the features under different scales through a weight distribution module, and obtaining final fused features;

[0012] S6, inputting the final fused features into a classifier for classification;

[0013] S7, calculating a loss function of the model according to the classification result, adjusting the model parameters, and completing the training of the model when the loss function converges;

[0014] S8, inputting the data in the test set into the trained model, and evaluating the model by using an evaluation index.

[0015] The application has the following beneficial effects:

[0016] The application can help the model to make scientific and accurate decisions quickly by using the dynamic sparse layer in the model to adaptively filter and mine the key instances from all instances in the whole full slice image, thereby enhancing the generalization and classification ability of the model. The network of the application can observe all tissue regions from a global perspective, thereby benefiting from global information to gradually discover and select key instances, to exclude a large amount of redundant information, and to ensure that the performance of the model serves the target. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is the overall flow chart of the present invention;

[0018] Figure 2 This is a model structure diagram of the present invention;

[0019] Figure 3 This is the attention visualization result diagram of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] A dynamic sparse multi-instance contrastive learning framework is provided. In this multi-instance framework, a dynamic token sparsification layer is designed to be distributed in the transformer-based multi-instance learning framework. By combining global information and local low-dimensional information, key instances are gradually and adaptively selected in the distributed layer, so that the final package-level representation benefits from those more representative instance-level samples that can be mined in the package while benefiting from the global overall information. Similarly, in order to address the semantic gap problem of the fusion between different scale features under multi-scale, the present invention adopts a cross-scale contrastive learning training framework to better obtain the common information that contributes to the final decision from different resolutions in multi-scale. Compared with previous contrastive learning, the present invention performs contrastive supervised training on the package-level representation of pathological images under a multi-scale weak supervision framework to overcome the semantic gap brought about by combining training at different scales, so that information at different scales can be effectively combined in the weak supervision learning framework. In addition, an end-stage iterative weight distribution module is also designed in the present invention to help adjust the weight distribution.

[0022] By adaptively and dynamically sparsely screening out key instances that are truly important and effective for the classification target, the prediction accuracy of the model is improved while the generalization ability of the model is further improved.

[0023] Dynamic Sparse Token Multi-Instance Learning: This approach combines a designed dynamic sparsification module with a mainstream multi-head self-attention mechanism. In the dynamic sparsification layer, we aggregate the local and global information of all instances to obtain a representation that determines the probability of each instance being discarded or kept. We also use Gumbel-Softmax reparameterized sampling to sample these probabilities. Finally, we reformulate the self-attention update formula based on the attention mask matrix.

[0024] Cross-scale contrastive learning framework: Slice features from different scales within the same full-slice image are passed through a dynamic sparse token multi-instance learning network to obtain a full-slice image-level representation. This representation is then fed into a linear layer to obtain an identity-mapped representation, which serves as the positive contrast sample. Based on their full-slice-level labels, k samples are randomly selected from the memory bank of the opposite class as negative contrast samples. A contrastive learning loss is then applied to perform cross-scale contrastive learning.

[0025] Iterative Feature Allocation Module: When fusion is performed at multiple scales, the key is to determine the appropriate weights for features at different scales. The initial integration ratio of input features and the resulting fusion weights will influence the final weight distribution. Therefore, an iterative weight allocation module is implemented at the end of the framework. This module adaptively averages the weights of the initial integrated features and performs multiple rounds of weight allocation to obtain the weight allocation scheme and final integrated features that best meet the optimization objectives.

[0026] A medical image classification method based on dynamic sparse contrast learning, such as Figure 1 As shown, the method includes: obtaining a medical image to be classified and preprocessing the medical image; inputting the preprocessed image into a trained classification model based on a dynamic sparse multi-instance learning framework to obtain a classification result; and visualizing the classified medical image machine.

[0027] Training a classification model based on the dynamic sparse multiple instance learning framework includes:

[0028] S1. Obtain a full-slice image dataset and preprocess the images in the full-slice image dataset to obtain 1024-dimensional features;

[0029] S2, divide all 1024-dimensional features into training set and test set;

[0030] S3. Input the features in the training set into the dynamic sparse multi-instance learning framework DSTMIL for training to obtain the weights of the dynamic sparse multi-instance learning framework at different scales;

[0031] S4. Load the weights of the dynamic sparse multi-instance learning framework at different scales into the backbone network of cross-scale contrastive learning, and learn the 1024-dimensional features to obtain features at different scales;

[0032] S5. Input the features of different scales into the cross-scale contrastive learning framework for training, and fuse the features of different scales through the weight distribution module to obtain the final fused features;

[0033] S6. Input the final fusion features into the classifier for classification;

[0034] S7. Calculate the loss function of the model based on the classification results, adjust the model parameters, and complete the model training when the loss function converges;

[0035] S8. Input the data in the test set into the trained model and evaluate the model using evaluation indicators.

[0036] In this embodiment, preprocessing of medical images includes: using the OUST method to remove irrelevant background areas of the medical images; cropping the active area into non-overlapping patches of 256*256 size and magnifications of 20× and 40×; performing tensorization and normalization on the non-overlapping patches; and inputting the normalized patches into a frozen-weight ResNet50 network to obtain 1024-dimensional features. All 1024-dimensional features are divided into a training set, a validation set, and a test set in a ratio of 7:2:1. The training set is used to train the model, the validation set is used to verify the trained model, and the test set is used to select the optimal trained model.

[0037] The medical images used are from CAMELYON16, a large-scale public breast cancer pathology image dataset released in 2016. This dataset is a large-scale dataset provided by a challenge for tumor pathology images. This dataset is used to evaluate the performance of computer-aided diagnosis (CAD) systems, particularly for the automatic detection and localization of tumor regions in breast cancer pathology images. These images are digitized and scanned from pathological tissue sections of breast cancer patients. The images are high-resolution and contain detailed structural information of cells and tissues. The OSTU method is used to remove the background, and the foreground active area is cropped into small, non-overlapping blocks of uniform size. These blocks are then fed into a ResNet50 network loaded with pre-frozen weights for feature extraction.

[0038] The detailed construction and process of dynamic sparse multi-instance learning are as follows: First, non-overlapping patches are obtained from WSI, and the resnet50 network with frozen weights pre-trained on image-1K is used to extract its feature embedding X = {x1, x2, ..., x N}, as input to the DSTMIL framework. This framework mainly consists of multiple self-attention heads and a sparse decision layer. By independently determining the positions of the attention and decision layers, it gradually obtains more decisive key instances and their class tokens in a hierarchical manner. Finally, the class token is fed into a fully connected network.

[0039] The sparsity decision layer (S-Layer) dynamically generates a corresponding binary mask M∈{1,0,…,1}^(N+1) for the token queue input into the framework. Based on this mask, it determines which tokens will be retained or discarded in the next round. This mask is initialized to all 1s, i.e., M={1,1,…,1}^(N+1). The mask corresponding to the class token is fixed to 1 during training and is never discarded. In the decision layer, the probability π of each token being retained or discarded is determined by aggregating the local information L and global information G of all tokens.

[0040]

[0041] I=concate(L,G)+CLS k-1 ∈R N×C

[0042] π=Softmax(MLP(I))∈R N×2 ,

[0043] Among them, MLP refers to the linear transformation of dimension reduction, and CLS is the class token of the previous layer.

[0044] In order to make the binary decision mask M obtained by π sampling differentiable, the Gumbel-Softmax reparameterization sampling technique is applied to obtain

[0045]

[0046] Where ⊙ represents the Hadamard product operator.

[0047] Dynamic sparse multi-instance learning framework DSTMIL, from each WSI non-overlapping split small block embedding vector X = {x1, x2, ..., x N} and its corresponding unknown patch-level label y={y1,y2,…,y N},At the same time, the bag-level label Y∈{0,1} of WSI is put into the standard MIL paradigm, that is:

[0048]

[0049] Under the weak supervision framework of WSI, a pooling function p is designed to extract bag-level embedding representations from all embedding instances:

[0050]

[0051] where a is the attention weight, and s is the linear projection function. In this framework, a sparsification decision layer is introduced to filter tokens to gradually excavate key instances with the largest information content. At the same time, in order to make the decided to be discarded token not participate in the next contribution, the attention mask matrix Q strategy is generated:

[0052]

[0053] On the basis of the attention mask matrix Q, the attention update formula is:

[0054]

[0055] Where A represents the attention matrix generated after the attention mask strategy. It can avoid the influence of the attention matrix on the token decided to be discarded in the training, so that the attention matrix is only related to the token decided to be retained in the current layer.

[0056] The core of the cross-scale contrastive learning framework CSCL is to compare the bag-level representations from different categories while constraining the differences between representations from different scales. This makes the overall optimization goal increase the similarity between embeddings from different scales and embeddings from the same category, and far away from the difference between embeddings from different categories, so as to overcome the semantic gap between semantic information in different scales. Therefore, we set two memory banks for the contrastive learning framework to store bag-level representations of different categories. Representations from different scales but belonging to the same category are stored in the same memory bank, and the overall framework is as shown in Figure 2 b. In the cross-scale contrastive learning framework based on weak supervision, the contrastive loss function CL is used to constrain the bag-level representations from different resolutions, so that the optimization of the two focuses on the organizational areas that truly benefit from those with higher contribution importance:

[0057] For positive samples, define the bag-level representations z h and z l from different scales, and the similarity between them is the cosine similarity:

[0058]

[0059] For negative samples randomly selected from the memory bank N, the similarity between negative samples is:

[0060]

[0061] Finally, the contrastive loss Contrastive Loss (CL) of the positive sample pair is:

[0062]

[0063] Meanwhile, there is also a cross-entropy loss function CEL of the package level prediction and the package level label obtained after the iterable feature assignment module. In summary, the overall loss function in the end-to-end framework is defined as:

[0064] L = CEL + aCL

[0065] Wherein, τ in the above formula is a temperature coefficient, and a is a hyperparameter for controlling the degree of CL. In this paper, τ is generally set to 5, and a is set to 0.8.

[0066] The visualization of the classified medical images shows that the model performance based on the dynamic sparse token multi-instance learning and cross-scale contrast learning framework has better performance than the previous method, and the AUC, ACC, PRECISION, F1, SENSITIVITY, SPECIFICITY and other indicators are improved. As shown in the following table: Figure 3 The attention heat map of the model can effectively identify the large area of tumors of Marco-metastasis and the few tumor cells or micro metastases of Micro-metastasis.

[0067] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and do not limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A medical image classification method based on dynamic sparse contrast learning, characterized in that: include: Obtaining medical images to be classified and preprocessing the medical images; The preprocessed image is input into the trained classification model based on the dynamic sparse multiple instance learning framework to obtain the classification result; Visualize the classified medical images; Training a classification model based on the dynamic sparse multiple instance learning framework includes: S1. Obtain a full-slice image dataset and preprocess the images in the full-slice image dataset to obtain 1024-dimensional features; S2, divide all 1024-dimensional features into training set and test set; S3. Input the features in the training set into the dynamic sparse multi-instance learning framework DSTMIL for training to obtain the weights of the dynamic sparse multi-instance learning framework at different scales; specifically, obtain non-overlapping patches from the full slice image, use the image-1K pre-trained resnet50 network with frozen weights, and extract its feature embedding X = {x1, x2, ..., x N The feature embedding is input into the DSTMIL framework, which consists of multiple self-attention heads and a sparsified decision layer. By independently determining the positions of the attention and decision layers, the framework hierarchically and gradually obtains more decisive key instances and their class tokens. The class tokens are then fed into a fully connected network to obtain the weights of the dynamic sparse multi-instance learning framework at different scales. S4. Loading the weights of the dynamic sparse multi-instance learning framework at different scales into the backbone network of cross-scale contrastive learning, and learning the 1024-dimensional features to obtain features at different scales. Specifically, the weights obtained at different scales are loaded parameter by parameter into the corresponding weight parameter position of the cross-scale contrastive learning backbone network according to the corresponding relationship of the network layer to achieve weight transfer. S5. Input the features of different scales into the cross-scale contrastive learning framework for training, and fuse the features of different scales through the weight distribution module to obtain the final fused features; Inputting features of different scales into the cross-scale contrastive learning framework for training includes: the contrastive learning framework sets up two memory banks to store packet-level representations of different categories respectively, and representations of the same category at different scales are stored in the same memory bank; the contrastive learning framework uses the contrast loss function CL to constrain the packet-level representations at different resolutions, that is, for positive samples, the packet-level representations z at different scales are defined. h and z l , calculate the cosine similarity between the two, and calculate the cosine similarity for the negative sample N randomly selected from the memory bank; calculate the contrast loss and cross entropy loss of the positive sample pair based on the two cosine similarities, and generate the overall loss function in the end-to-end framework based on the contrast loss and cross entropy loss; S6. Input the final fusion features into the classifier for classification; S7. Calculate the loss function of the model based on the classification results, adjust the model parameters, and complete the model training when the loss function converges; S8. Input the data in the test set into the trained model and evaluate the model using evaluation indicators.

2. The medical image classification method based on dynamic sparse contrast learning according to claim 1, characterized in that: The preprocessing of medical images includes: using the OUST method to remove irrelevant background areas of medical images; cropping the effective area into non-overlapping blocks of 256*256 size and magnification of 20× and 40×; tensorizing and normalizing the non-overlapping blocks; and inputting the normalized blocks into the resnet50 network with frozen weights to obtain 1024-dimensional features.

3. The medical image classification method based on dynamic sparse contrast learning according to claim 2, characterized in that: The OUST method is used to remove irrelevant background areas of medical images, including: converting the medical image into a grayscale image; setting a feature attribute threshold, comparing the feature attribute value of each pixel in the grayscale image with the feature attribute threshold, and treating the pixels whose feature attribute value is less than the feature attribute threshold as irrelevant background areas; and removing all irrelevant background areas to obtain a binary image.

4. The medical image classification method based on dynamic sparse contrast learning according to claim 1, characterized in that: The sparsification decision layer S-Layer is the token queue input into the framework. It dynamically obtains a corresponding binary mask M∈{1,0,…,1}^(N+1) through the sparsification decision layer, and retains or discards the token according to the mask. At the same time, the mask is initialized to all 1s, that is, M={1, 1,…,1}^(N+1), where the mask corresponding to the class token is fixed to 1 during training; in the decision layer, the probability π of each token being discarded or saved is obtained by aggregating the embedding obtained by local information L and global information G of all tokens.

5. The medical image classification method based on dynamic sparse contrast learning according to claim 4, characterized in that: The probability π of each token being discarded or saved is: I=concate(L,G)+CLS k-1 ∈R N×C π=Softmax(MLP(I))∈R N×2 , Among them, MLP refers to the linear transformation of reduced dimension, CLS is the class token of the previous layer, N is the number of instances in a full slice image, C is the dimension of instance features, and M is the number of instances in a full slice image. i is the binary decision mask value obtained in the previous layer, I is the feature value of ClassToken in the previous layer model by concatenating global information features, local information features and the previous layer, and concate is the feature concatenation operation.

6. The medical image classification method based on dynamic sparse contrast learning according to claim 1, characterized in that: The contrast loss is: For positive samples, define the packet-level representation z at different scales h and z l , the similarity between the two is cosine similarity: For N negative samples randomly selected from the memory bank, the similarity between negative samples is: Among them, p s is the cosine similarity of the positive sample, z h is the packet-level feature at high resolution, z l is the packet-level feature at low resolution, τ is the temperature coefficient, N is the negative sample feature set, n s is the cosine similarity between negative samples.

7. The medical image classification method based on dynamic sparse contrast learning according to claim 1, characterized in that: The loss function of the model is: L=CEL+αCL Among them, CEL is the cross entropy loss, α is the hyperparameter that controls the degree of CL, and CL is the contrast loss.

Images