Pathological image region-level similarity retrieval method based on optimal transmission

Abstract

The invention relates to a pathological image region-level similarity retrieval method based on optimal transmission. The pathological image region-level similarity retrieval method comprises the following steps of 1, data preprocessing, wherein WSI slicing and image block extraction and data set division are carried out; 2, multi-scale feature extraction: extracting features by using a pre-training model; step 3, feature processing: using variance-based feature selection and UMAP and PCA methods to carry out dimension reduction processing; 4, similarity calculation: calculating the similarity by using the Euclidean distance and the optimal transmission distance; and 5, retrieval and evaluation: calculating the similarity between the query image and the image in the candidate database, ranking and evaluating the accuracy by using Top-K retrieval results, and comparing the retrieval performances of different feature processing modes and similarity measurement methods. According to the method, fine feature extraction is carried out at the micro tile level, similarity measurement is carried out based on the optimal transmission theory, feature selection and a UMAP dimension reduction method are combined, and the precision and reliability of pathological image region-level similarity retrieval are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing and artificial intelligence, and specifically relates to a deep learning feature extraction, dimensionality reduction and similarity retrieval method for pathological images. Background Art

[0002] Existing pathological image similarity retrieval methods mainly include: feature extraction methods based on traditional image processing technology, methods using convolutional neural networks (CNN) for feature extraction, deep learning models based on attention mechanisms, and common similarity measurement methods such as Euclidean distance and cosine similarity. Although existing technologies have made progress in pathological image retrieval, they still have the following major shortcomings:

[0003] Feature extraction is not sophisticated enough: It fails to fully utilize the multi-scale information and cellular-level morphological features of pathological images. Insufficient heterogeneity processing: The cells and tissues in pathological images are highly heterogeneous. Existing methods fail to effectively distinguish these heterogeneous features during feature extraction, which may lead to deviations in retrieval results. Limitations of similarity measurement methods: Commonly used similarity measurement methods such as Euclidean distance or cosine distance may not fully capture the complexity of the distribution of pathological image features. Insufficient interpretability: It is difficult to provide an intuitive similarity explanation.

[0004] Chinese patent document CN114168781A discloses a pathological image retrieval method, wherein the method includes: obtaining a pathological image to be queried; generating a hash code of the pathological image to be queried; using the hash code to search in a preset database to obtain a query result of the pathological image to be queried, wherein the database contains a correspondence between the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the pathological image data. The embodiment of the present application not only realizes the full-size pathological image feature extraction that takes into account both local and overall information, but also trains the feature extraction parameters under unsupervised conditions to achieve a better feature representation, and the hash code optimizes the retrieval matching mode, greatly improving the retrieval performance.

[0005] Chinese patent document CN118260442A discloses a pathological image product quantization retrieval method based on self-supervised learning, which relates to the field of medical image processing. The present invention is to solve the problem that the model for extracting image features in PQ retrieval cannot extract key information of pathological images. The pathological image product quantization retrieval method based on self-supervised learning described in the present invention inputs the retrieved pathological image into the pathological image SPQ retrieval model to obtain the image feature vector of the retrieved pathological image; inputs all retrieval images in the retrieval library into the pathological image SPQ retrieval model to obtain the product quantization code of the retrieval image; calculates the Euclidean distance between the image feature vector of the retrieved pathological image and the product quantization code of each retrieval image respectively; sorts all Euclidean distances from small to large, and extracts the top 1% of the Euclidean distances as the retrieval Euclidean distance; and uses the retrieval image corresponding to the retrieval Euclidean distance as the retrieval result of the retrieved pathological image.

[0006] However, there is no report on the region-level similarity retrieval method for pathological images based on optimal transmission. Summary of the invention

[0007] The present invention aims to overcome the deficiencies in the prior art and provide a pathological image region-level similarity retrieval method based on optimal transmission theory. The method improves retrieval accuracy and reliability through multi-scale feature extraction, nonlinear dimensionality reduction and efficient similarity measurement, while taking into account the heterogeneity and complexity of pathological images, providing a more accurate similar case retrieval tool for pathological diagnosis and research.

[0008] The present invention provides a pathological image region level similarity retrieval method based on optimal transmission, and the pathological image region level similarity retrieval method comprises the following steps:

[0009] Step 1: Data preprocessing: WSI slicing and tile extraction, data set division;

[0010] Step 2: Multi-scale feature extraction: extract features using a pre-trained model;

[0011] Step 3: Feature processing: Use variance-based feature selection and UMAP and PCA methods for dimensionality reduction;

[0012] Step 4: Similarity calculation: use Euclidean distance and optimal transmission distance to calculate similarity;

[0013] Step 5: Retrieval and evaluation: Calculate the similarity between the query image and the images in the candidate database, use Top-K retrieval result ranking to evaluate the accuracy, and compare the retrieval performance of different feature processing methods and similarity measurement methods.

[0014] As a preferred example, the step 4 includes:

[0015] Similarity measure based on Euclidean distance:

[0016] For two tiles t i and t j The class tokens are c i and c j , the dimensions are all dim, and the Euclidean distance calculation formula is:

[0017]

[0018] Among them, c i，k and c j，k Respectively represent the feature values ​​of the class token in the kth dimension. i and t j The patch tokens are averaged to get M_avg i and M_avg j The dimensions are all dim and the Euclidean distance is also used to calculate the similarity.

[0019] As another preferred example, the step 4 includes: similarity measurement based on OT distance:

[0020] For two tiles t i and t j The patch tokens feature matrices are M i and M j , the dimensions are all 256x dim, and the OT distance is calculated by the following cost matrix calculation and OT distance calculation.

[0021] As another preferred example, the cost matrix calculation: The cost matrix C is based on M i and M j The cost between the midpoint pairs is calculated. i The kth point in M j The lth point in the kl It can be defined as the Euclidean distance between their feature vectors: C kl =||M i，k -M j，l || 2 .

[0022] As another preferred example, the OT distance calculation: using the cost matrix and the regularization parameter, the OT distance is calculated by the Sinkhorn-Knopp algorithm, the goal is to find a transportation matrix π that minimizes the regularized transportation cost

[0023]

[0024] where U(r, c) represents the set of transport matrices with row and column sums equal to r and c respectively, chosen to be uniformly distributed. The regularization parameter ∈ is set to 0.05.

[0025] As another preferred example, in step 3, the first 128 most representative feature dimensions are selected by comparing the order of feature variances. The standard deviation of each feature is calculated:

[0026]

[0027] where σ i is the standard deviation of the ith feature, x ij is the i-th eigenvalue of the j-th sample, is the mean of the ith feature, and N is the number of samples.

[0028] As another preferred example, the vector σ is sorted in descending order to obtain a sorted index set, and the first 128 indexes are selected to represent the 128 feature dimensions with the largest variance, and the feature dimensions selected based on variance in the class token and the patch token are obtained respectively.

[0029] As another preferred example, in step 3:

[0030] The dimension reduction described is as follows: the UMAP algorithm is used to perform nonlinear dimension reduction on the selected class token and patch token respectively, the parameter is set to n_neighbors=50, the feature is reduced to 16 dimensions, and the dimension reduction model is trained and saved.

[0031] As another preferred example, in step 3, the PCA model is also trained down to 16 dimensions and saved using the same hyperparameter settings.

[0032] The advantages of the present invention are: the present invention provides a pathological image region-level similarity retrieval method based on optimal transmission, which has significant advantages and positive effects compared with the prior art. Specifically as follows:

[0033] Retrieval performance at the regional level: Compared with the traditional Euclidean distance measurement method, the OT distance measurement method adopted in the present invention has significantly improved the retrieval performance at the pathological region level.

[0034] Feature selection effect: Through the variance-based feature selection method, the feature representation is further optimized, and the retrieval performance is improved in all indicators.

[0035] Dimensionality reduction effect: The UMAP dimensionality reduction method is used to reduce the dimensionality of high-dimensional features, retaining important feature information and further improving the retrieval performance. The retrieval performance after UMAP dimensionality reduction is better than that of the cases without dimensionality reduction and PCA dimensionality reduction in all indicators.

[0036] Functional differences:

[0037] Complex structure processing: The present invention fully considers the complexity of the internal structure of the pathological image, and can more accurately capture the fine-grained features in the pathological image through feature extraction and dimensionality reduction processing at the micro-block level.

[0038] Efficient similarity measurement: The OT distance method is used for similarity measurement. Compared with the traditional Euclidean distance and cosine distance, it is more suitable for large-scale parallel computing acceleration of high-dimensional data, improving computing efficiency and accuracy.

[0039] In summary, the present invention effectively improves the accuracy and reliability of region-level similarity retrieval of pathological images by performing fine feature extraction at the micro-tile level, measuring similarity based on optimal transfer theory, and combining feature selection and UMAP dimensionality reduction method, which is significantly better than the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 : Flow chart of the present invention.

[0041] Figure 2 : The statistical chart of the regional-level retrieval comparison experiment results is a histogram of MVAcc@5 performance comparison using different feature processing methods and distance measurement methods. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0043] Example 1

[0044] Please see Figure 1 :A pathological image region-level similarity retrieval method based on optimal transmission, which mainly includes the following steps

[0045] Step 1:

[0046] Data preparation:

[0047] According to the tissue mask composed of different categories of regions annotated by pathologists on the whole slide image (WSI), the effective tissue area of ​​the WSI was determined, and non-overlapping blocks were cut in the WSI at 10 times magnification with 256x256 pixels as the unit. Each pixel represents the actual tissue area of ​​1μm x 1μm, ensuring sufficient resolution to capture the key morphological features of cell tissue.

[0048] Dataset division:

[0049] Cut the tiles in each annotated category and divide the training set and test set according to the WSI, with each accounting for 50%. All tiles from the WSI that are divided into the training set will be assigned to the training set, and vice versa.

[0050] Step 2: Feature extraction:

[0051] The high-dimensional features of the input pathology images are extracted using the pathology image pre-trained deep learning model UNI based on the Vision Transformer (ViT) model architecture. The model architecture of UNI is ViT-Large, and the patch_size of ViT is set to 16, and the extracted feature dimension is set to 1024. The 256x256 pixel tiles obtained by cutting are subdivided into 256 16x16 pixel micro-tiles. The size of each micro-tile is approximately the size of a single cell. The spatial information interaction between micro-tiles is obtained through the self-attention mechanism of ViT to realize cell-level feature extraction. Each tile obtains a class token with a dimension of 1024 and a patch tokens feature matrix with a dimension of 256x1024 (256: the number of micro-tiles in the tile; 1024: feature dimension).

[0052] Use UNI to extract class tokens and patch tokens of all tiles in the training set and test set for subsequent region-level similarity retrieval.

[0053] Step 3: Feature selection and dimensionality reduction:

[0054] First, 1,000 images are randomly sampled from the training set (1,000 class tokens and 256,000 patch tokens are extracted from UNI. In order to reduce computational complexity and data redundancy, 8,000 patch tokens are randomly sampled again) to determine the selection of feature dimensions and train the dimensionality reduction model.

[0055] Feature Selection:

[0056] By comparing the order of feature variance, the first 128 most representative feature dimensions are selected. The standard deviation of each feature is calculated:

[0057]

[0058] where σ i is the standard deviation of the ith feature, x ij is the i-th eigenvalue of the j-th sample, is the mean of the i-th feature, and N is the number of samples. The sorted index set is obtained by sorting the vector σ in descending order, and the first 128 indexes are selected, indicating the 128 feature dimensions with the largest variance.

[0059] The feature dimensions based on variance selection in class token and patch token are obtained respectively.

[0060] Dimensionality reduction:

[0061] The UMAP (Uniform Manifold Approximation and Projection) algorithm is used to perform nonlinear dimensionality reduction on the selected class token and patch token, and the parameter is set to n_neighbors = 50, the feature is reduced to 16 dimensions, and the dimensionality reduction model is trained and saved. Using the same hyperparameter settings, the PCA (Principal components analysis) model is also trained and reduced to 16 dimensions and saved.

[0062] The selected dimension and dimensionality reduction model are used to infer all the tiles used for the training set and the test set, respectively, to obtain the UNI extracted class token (1024) and patch tokens (256x1024), the feature selected class token (128) and patch tokens (256x128), and the reduced class token (16) and patch tokens (256x16) of each tile.

[0063] Step 4: Similarity retrieval

[0064] Similarity measure based on Euclidean distance:

[0065] For two tiles t i and t j The class tokens are c i and c j , the dimensions are all dim, and the Euclidean distance calculation formula is:

[0066]

[0067] Among them, c i，k and c j，k Respectively represent the feature values ​​of the class token in the kth dimension. i and t j The patch tokens are averaged to get M_avg i and M_avg j The dimensions are all dim and the Euclidean distance is also used to calculate the similarity.

[0068] Similarity measurement based on OT distance:

[0069] For two tiles t i and t j The patch tokens feature matrices are M i and M j , the dimensions are 256x dim, and the OT distance is calculated by the following steps:

[0070] 1. Cost matrix calculation. The cost matrix C is based on M i and M j The cost between the midpoint pairs is calculated. i The kth point in M j The lth point in the kl It can be defined as the Euclidean distance between their eigenvectors:

[0071] C kl =||M i，k -M j，l || 2

[0072] 2. OT distance calculation: Using the cost matrix and regularization parameter, the OT distance is calculated by the Sinkhorn-Knopp algorithm. The goal is to find a transportation matrix π that minimizes the regularized transportation cost:

[0073]

[0074] where U(r, c) represents the set of transport matrices with row and column sums equal to r and c respectively, chosen to be uniformly distributed. The regularization parameter ∈ is set to 0.05.

[0075] Similarity search:

[0076] Each image in the test set is used as a query image and compared with each image in the training set. The training set is used as a candidate database (keys) and no supervised learning is performed. The class labels are only used for evaluation. Different similarity metrics use different feature types. Euclidean distance uses class tokens, and OT distance uses patch-tokens.

[0077] Before distance measurement, the key database is centered by subtracting the Euclidean centroid of the key database and normalizing each key to unit length. For each query image query, the same translation and normalization steps are performed first, and then the distance metric between the query tile and each key is calculated. The smaller the distance, the higher the similarity. The retrieval results are sorted by similarity score and the performance is evaluated using the category label.

[0078] Consider the retrieval accuracy, Acc@K, where K∈{1, 3, 5}. If the correctly labeled tile is in the top K results, the retrieval is successful. In addition, MVAcc@5 is used, based on the majority vote of the top 5 retrieval results as the retrieval label. If there are multiple categories that receive the highest votes, the category label of the tile that is most similar to the query is selected. See Figure 2

[0079] The invention discloses a region-level similarity retrieval method applied to pathological images, which includes: a retrieval strategy based on a feature extraction and dimension reduction method of pathological blocks and a similarity measurement method based on OT distance.

[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the method of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A pathological image region-level similarity retrieval method based on optimal transmission, characterized by: The pathological image region level similarity retrieval method comprises the following steps: Step 1: Data preprocessing: WSI slicing and tile extraction, data set division; Step 2: Multi-scale feature extraction: extract features using a pre-trained model; Step 3: Feature processing: Use variance-based feature selection and UMAP and PCA methods for dimensionality reduction; Step 4: Similarity calculation: use Euclidean distance and optimal transmission distance to calculate similarity; Step 5: Retrieval and evaluation: Calculate the similarity between the query image and the images in the candidate database, use Top-K retrieval result ranking to evaluate the accuracy, and compare the retrieval performance of different feature processing methods and similarity measurement methods.

2. The pathological image region-level similarity retrieval method according to claim 1, characterized in that: The step 4 includes: Similarity measure based on Euclidean distance: For two tiles t i and t j The class tokens are c i and c j , the dimensions are all dim, and the Euclidean distance calculation formula is: Among them, c i，k and c j，k Respectively represent the feature values ​​of the class token in the kth dimension. i and t j The patchtokens are averaged to get M_avg i and M_avg j The dimensions are all dim and the Euclidean distance is also used to calculate the similarity.

3. The pathological image region-level similarity retrieval method according to claim 1, characterized in that: The step 4 comprises: Similarity measurement based on OT distance: For two tiles t i and t j The patch tokens feature matrices are M i and M j , the dimensions are all 256xdim, and the OT distance is calculated by the following cost matrix calculation and OT distance calculation.

4. The pathological image region-level similarity retrieval method according to claim 3, characterized in that: The cost matrix calculation: The cost matrix C is based on M i and M j The cost between the midpoint pairs is calculated. i The kth point in M j The lth point in the kl It can be defined as the Euclidean distance between their feature vectors: C kl =||M i，k -M j，l || 2 .

5. The pathological image region-level similarity retrieval method according to claim 4, characterized in that: The OT distance calculation: The OT distance is calculated by the Sinkhorn-Knopp algorithm using the cost matrix and regularization parameter. The goal is to find a transportation matrix π that minimizes the regularized transportation cost. where U(r, c) represents the set of transport matrices with row and column sums equal to r and c respectively, chosen to be uniformly distributed. The regularization parameter ∈ is set to 0.

05.

6. The pathological image region-level similarity retrieval method according to claim 1, characterized in that: In step 3, the first 128 most representative feature dimensions are selected by comparing the order of feature variances. The standard deviation of each feature is calculated: where σ i is the standard deviation of the ith feature, x ij is the i-th eigenvalue of the j-th sample, is the mean of the ith feature, and N is the number of samples.

7. The pathological image region-level similarity retrieval method according to claim 6, characterized in that: According to the descending sorting of the vector σ, the sorted index set is obtained, and the first 128 indexes are selected, indicating the 128 feature dimensions with the largest variance, and the feature dimensions selected based on variance in the class token and patch token are obtained respectively.

8. The pathological image region-level similarity retrieval method according to claim 1, characterized in that: In step 3: The dimension reduction described is as follows: the UMAP algorithm is used to perform nonlinear dimension reduction on the selected class token and patch token respectively, the parameter is set to n_neighbors=50, the feature is reduced to 16 dimensions, and the dimension reduction model is trained and saved.

9. The pathological image region-level similarity retrieval method according to claim 8, characterized in that: In step 3, the PCA model is also trained down to 16 dimensions and saved using the same hyperparameter settings.

Images