Pathological image classification method and imaging method based on deep learning
By constructing a pathological image classification model based on the Vision Transformer network, the shortcomings of existing pathological image classification technologies are addressed, achieving highly reliable and accurate classification of multiple types of pathological images and improving the model's versatility.
Patent Information
- Application Number
- CN202411248679.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-06
AI Technical Summary
Existing deep learning-based pathological image classification schemes have shortcomings in representing the entire pathological image, cannot be applied to the classification of multiple types of pathological images simultaneously, and rely on the personal skills of professionals, resulting in poor reliability and accuracy.
A pathological image classification model based on the Vision Transformer network is constructed, including a feature extraction module, a vector generation module, and a classification module. Through feature extraction, clustering, residual accumulation, and principal component analysis, combined with the Transformer network and classifiers such as support vector machines or Adaboost, efficient classification of pathological images is achieved.
It improves the reliability and accuracy of pathological image classification, enhances the model's versatility, and makes it applicable to the classification of multiple types of pathological images.
Smart Images

Figure CN118781436B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical image processing, and specifically relates to a pathological image classification method and an imaging method based on deep learning. Background Art
[0002] Pathological images play an irreplaceable role in both clinical and basic medical research. Therefore, accurate classification of pathological images is of great significance for both clinical and basic medical research.
[0003] At present, the traditional pathological image classification scheme often adopts the manual classification scheme, that is, researchers or staff with professional medical backgrounds classify pathological images manually based on their own professional ability and technical background through reading pathological images. However, this manual pathological image classification scheme is not only extremely dependent on the personal ability of professionals, but also has relatively poor reliability and accuracy.
[0004] In recent years, with the rapid development of deep learning, researchers have proposed various types of classification models for various pathological images. For example, Coudray et al. successfully classified pathological images of lung squamous cell carcinoma and lung adenocarcinoma using the Inception-V3 network. Adnan et al. also learned the relationship between image blocks through color feature sampling and graph neural network, and achieved good results in lung cancer pathological image classification. However, the current pathological image classification scheme based on deep learning can only select some small blocks of images to aggregate to obtain the final pathological image representation. This type of scheme has defects in the representation of the entire pathological image; moreover, the current pathological image classification scheme cannot be applied to the classification of multiple types of pathological images at the same time, and its versatility is poor. Summary of the invention
[0005] One of the purposes of the present invention is to provide a pathological image classification method based on deep learning with high reliability, good accuracy and good versatility.
[0006] A second object of the present invention is to provide an imaging method that includes the pathological image classification method based on deep learning.
[0007] The pathological image classification method based on deep learning provided by the present invention comprises the following steps:
[0008] S1. Obtain existing pathological images;
[0009] S2. Preprocessing the pathological image obtained in step S1 to construct a training data set;
[0010] S3. Build an initial model for pathological image classification based on the Vision Transformer network;
[0011] The constructed model includes a feature extraction module, a vector generation module, and a classification module;
[0012] A feature extraction module is constructed based on the Vision Transformer network. The feature extraction module is used to extract features from the input pathological image and output a feature vector for each image block.
[0013] A vector generation module is constructed based on clustering, residual accumulation and principal component analysis. The vector generation module is used to reduce the dimension of the feature vector of each image block to improve the processing speed and accuracy of the model.
[0014] A classification module is constructed based on a deep learning model; the classification module is used to complete the classification of pathological images according to the vectors generated by the vector generation module;
[0015] S4. Using the training data set constructed in step S2, the initial pathological image classification model constructed in step S3 is trained to obtain a pathological image classification model;
[0016] S5. Using the pathological image classification model obtained in step S4, actual pathological image classification is performed.
[0017] The existing pathological image described in step S1 specifically includes the following contents:
[0018] The existing pathological images include several types of pathological images of the same type of disease, and the type corresponding to each pathological image is marked;
[0019] or
[0020] The existing pathological images include pathological images containing lesions and pathological images without lesions and normal for any disease, and the category corresponding to each pathological image is marked; the categories include the category containing lesions and the category without lesions and normal.
[0021] The preprocessing described in step S2 specifically includes the following steps:
[0022] Center cropping: center cropping the pathological image obtained in step S1 to ensure that the width and height of the image reach the set size;
[0023] Image segmentation: Use a non-overlapping grid to divide the center-cropped image into several image blocks; each block is used as a basic unit;
[0024] Reading and filtering of image blocks: extract each image block; for each image block, analyze the pixel value of the total red channel in the image block, count the number of pixels whose pixel values are less than the set value, identify the image blocks whose pixel numbers are less than the set value as background blocks and filter them, completing the reading and filtering of image blocks.
[0025] The feature extraction module is constructed based on the Vision Transformer network, and specifically includes the following steps:
[0026] Rearranging the image blocks obtained in step S2;
[0027] Decompose each image block into several patches according to the set requirements, and add a set marker at the beginning of each patch sequence; the purpose of setting the marker is to help the model capture the global information of the sequence;
[0028] The patch with the set mark is input into the pre-trained ViT_256 model: The ViT_256 model encodes each patch. During the encoding process, each patch is encoded through several layers of Transformer networks; each layer of Transformer network encodes the context information of each element in the sequence of patches based on the self-attention mechanism and feedforward neural network; the processing process of each layer of Transformer network is expressed as and ,in is the layer normalization operation, x is the input feature, is the multi-head attention mechanism processing function, is a fully connected network processing function; after being encoded by several layers of Transformer networks, the output of the set mark is considered to represent the global information of the entire input image block; the set mark set of each image block is aggregated to obtain the global features of the image block;
[0029] The global features of the obtained image block are reshaped into a three-dimensional array, and the reshaped three-dimensional array is transposed to obtain the feature map of each image block.
[0030] The vector generation module is constructed based on clustering, residual accumulation and principal component analysis, and specifically includes the following steps:
[0031] The feature vector of each image block output by the feature extraction module is clustered using the K-means clustering algorithm to obtain the codebook:
[0032] The following formula is used as the loss function of the K-means clustering algorithm: Where J is the loss function value; N is the total number of data points; is the i-th data point; is the jth cluster center; is the Euclidean distance;
[0033] Randomly select K data points as initial cluster centers;
[0034] For each data point, find the nearest cluster center and assign the data point to the corresponding cluster;
[0035] For each cluster, the cluster center is recalculated using the following formula: In the formula is the kth cluster center; is the number of data points in the kth cluster; is the i-th data point;
[0036] Repeat the above formula to iteratively update the cluster center until the set requirements are met; finally, K cluster centers are obtained; at this time, each cluster center is used as a typical feature of the image data, and all cluster centers constitute a codebook;
[0037] Based on the obtained codebook and residual accumulation, the residual vector of each cluster center is calculated:
[0038] Compare each feature vector with the cluster center in the codebook and determine the cluster center to which each feature vector is closest;
[0039] For each eigenvector, the residual between the eigenvector and the closest cluster center is calculated using the following formula: : In the formula is the feature vector The closest cluster center to the corresponding The distance between
[0040] For each cluster center, the residual vector of the cluster center is calculated using the following formula: : In the formula Represents the feature vector Assigned to cluster center k;
[0041] Based on principal component analysis, the residual vector is reduced in dimension:
[0042] Normalize the residual vector of each cluster center;
[0043] Connect the normalized residual vectors to obtain the global residual vector;
[0044] The global residual vector is normalized again to obtain a normalized global residual vector;
[0045] The normalized global residual vector is subjected to principal component analysis to obtain the output of the final vector generation module.
[0046] The classification module is constructed based on the deep learning model, and specifically includes the following steps:
[0047] Use Transformer network, SVM support vector machine network or Adaboost and other classifiers to build a classification module.
[0048] When the Transformer network is used to construct a classification module, the processing of the classification module includes the following steps:
[0049] The constructed classification module includes input layer, intermediate layer, encoding layer and classification layer;
[0050] The input layer, the middle layer, the encoding layer, and the classification layer are connected in series;
[0051] Input layer: The input layer is used to receive the output of the vector generation module;
[0052] Middle layer: The middle layer includes the first fully connected layer, the first normalized layer, the second fully connected layer and the second normalized layer connected in series; the first fully connected layer is used to map the input features to a space of set dimensions, and the first normalized layer is used to annotate the output of the first fully connected layer; the second fully connected layer is used to compress the input features to a space of set dimensions; the second normalized layer is used to standardize the output of the second fully connected layer; the processing process of the first fully connected layer is expressed as , is the output of the first fully connected layer, is the input of the first fully connected layer, is the weight matrix of the first fully connected layer, is the bias vector of the first fully connected layer; the processing process of the second fully connected layer is expressed as , is the output of the second fully connected layer, is the input of the second fully connected layer, is the weight matrix of the second fully connected layer, is the bias vector of the second fully connected layer;
[0053] Encoding layer: The encoding layer is based on a multi-head self-attention mechanism and a feedforward network to capture the dependencies between various positions in the input features. The processing of the encoding layer is expressed as: In the formula is the output of the l+1th layer; is the output of the lth layer; is the output of the l+1th layer through the feedforward network;
[0054] Classification layer: The classification layer uses a fully connected layer to convert the output of the encoding layer into a category prediction result; the processing of the fully connected layer is expressed as , y is the output of the fully connected layer, is the input of the fully connected layer, is the weight matrix of the fully connected layer, is the bias vector of the fully connected layer.
[0055] The training described in step S4 specifically includes the following steps:
[0056] The following loss function is used: Where L is the loss function value; C is the total number of categories; is the true category label of the target; The probability that the target predicted by the model is category i;
[0057] The Adam optimizer is used to train the model:
[0058] During training, the following formula is used to adjust the learning rate of each parameter in the model: In the formula is the model parameter value at the t+1th time; is the model parameter value of the tth time; Bias-corrected second-order moment estimate of the gradient; is the bias-corrected first-order moment estimate of the gradient; is the learning rate and , is the initial learning rate, is the attenuation factor, s is the step size, is the floor function.
[0059] The present invention also provides an imaging method including the pathological image classification method based on deep learning, further comprising the following steps:
[0060] S6. The pathological image classification result obtained in step S5 is marked and re-imaged on the actual pathological image, and finally a pathological image with the pathological image classification result is output.
[0061] The pathological image classification method and imaging method based on deep learning provided by the present invention construct an initial model for pathological image classification based on the VisionTransformer network, and use the model for training and actual pathological image classification; therefore, the present invention can not only realize the classification of pathological images, but also has higher reliability, better accuracy and better versatility. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 The figure is a schematic diagram of the method flow of the classification method of the present invention.
[0063] Figure 2 The figure is a schematic diagram of the method flow of the imaging method of the present invention. DETAILED DESCRIPTION
[0064] like Figure 1 The method flow chart of the classification method of the present invention is shown as follows: The pathological image classification method based on deep learning provided by the present invention comprises the following steps:
[0065] S1. Obtain existing pathological images; specifically including the following:
[0066] The existing pathological images include several types of pathological images of the same type of disease, and the type corresponding to each pathological image is marked; for example, for the type of cancer lung cancer, the pathological images may include medical records of vocal cord squamous cell carcinoma, pathological images of lung adenocarcinoma, pathological images of head and neck squamous cell carcinoma, pathological images of colorectal cancer, pathological images of lung adenocarcinoma, and pathological images of lung squamous cell carcinoma;
[0067] or
[0068] The existing pathological images include pathological images containing lesions and pathological images without lesions and normal for any disease, and the category corresponding to each pathological image is marked; the categories include categories containing lesions and categories without lesions and normal; for example, for a single cancer (such as lung adenocarcinoma), the pathological images at this time may include pathological images containing lung adenocarcinoma lesions and normal pathological images without lesions;
[0069] S2. Preprocessing the pathological image obtained in step S1 to construct a training data set; specifically comprising the following steps:
[0070] Center cropping: The pathological image obtained in step S1 is center cropped to ensure that the width and height of the image reach the set size; in specific implementation, the pathological image is center cropped to ensure that the image width (W) and height (H) can be defined by the patch size defined in the subsequent model (such as ). The image can be evenly divided into small patches to reduce edge noise while focusing on key information in the center of the image. If the width and height are not integer multiples of the patch size, make necessary adjustments to ensure that the sizes match, such as cropping to the nearest integer multiple size so that and ,in is the adjusted width, is the adjusted height, mod is the remainder operation;
[0071] Image segmentation: Use a non-overlapping grid to divide the center-cropped image into several image blocks; each block is used as a basic unit; in specific implementation, the cropped image will be further subdivided into The block is used as the basic unit of model processing. and the adjusted height The non-overlapping grid division is applied on the image to ensure that the image blocks can evenly cover the entire image;
[0072] Reading and filtering of image blocks: extract each image block; for each image block, analyze the pixel value of the total red channel in the image block, count the number of pixels with pixel values less than the set value, identify the image block with pixel numbers less than the set value as the background block and filter it, and complete the reading and filtering of the image block; in specific implementation, an advanced image processing library can be used to accurately extract each image block from a large-size slide; the extracted image blocks are initially filtered to exclude background or non-critical areas, and blocks containing a large amount of background are identified by analyzing the pixel value of the red channel (the number of pixel values less than 220); this filtering operation is further optimized to ensure that the image block contains enough valid information;
[0073] The above preprocessing operations ensure that no matter what the size of the original image is, image patches of uniform specifications can be obtained, and information retention and noise reduction are optimized to a certain extent; these unified image patches can provide efficient and homogenized input for model training and subsequent image analysis;
[0074] S3. Build an initial model for pathological image classification based on the Vision Transformer network;
[0075] The constructed model includes a feature extraction module, a vector generation module and a classification module; the feature extraction module is used to extract features from the input pathological image and output the feature vector of each image block; the vector generation module is used to reduce the dimension of the feature vector of each image block to improve the processing speed and accuracy of the model; the classification module is used to complete the classification of the pathological image according to the vector generated by the vector generation module;
[0076] The specific steps include:
[0077] Based on the Vision Transformer network, a feature extraction module is constructed; specifically, the following steps are included:
[0078] The image blocks obtained in step S2 are rearranged; in specific implementation, the rearrange function of the einops library can be used to implement it, which is expressed as In the formula is the rearranged image block; is the original input image data; b is the batch size; c is the number of channels; p is the number of image blocks; l is the depth of the image block; w is the width of the image block; h is the height of the image block; p1 is the number of image blocks expanded along the width direction; p2 is the number of image blocks expanded along the height direction;
[0079] Decompose each image block into several patches according to the set requirements, and add a set marker at the beginning of each patch sequence; the purpose of the set marker is to help the model capture the global information of the sequence; in specific implementation, add a set marker [CLS] at the beginning of each patch sequence;
[0080] The patch with the set mark is input into the pre-trained ViT_256 model: The ViT_256 model encodes each patch. During the encoding process, each patch is encoded through several layers of Transformer networks; each layer of Transformer network encodes the context information of each element in the sequence of patches based on the self-attention mechanism and feedforward neural network; the processing process of each layer of Transformer network is expressed as and ,in is the layer normalization operation, x is the input feature, is a multi-head attention mechanism processing function (used to capture information from different subspaces in parallel), is a fully connected network processing function; after being encoded by several layers of Transformer networks, the output of the set mark is considered to represent the global information of the entire input image block; the set mark set of each image block is aggregated to obtain the global features of the image block;
[0081] The global features of the obtained image blocks are reshaped into a three-dimensional array (to maintain the spatial relationship of the original image blocks in the image), and the reshaped three-dimensional array is transposed to obtain the feature map of each image block;
[0082] Based on clustering, residual accumulation and principal component analysis, a vector generation module is constructed; specifically, the following steps are included:
[0083] The feature vector of each image block output by the feature extraction module is clustered using the K-means clustering algorithm to obtain the codebook:
[0084] The following formula is used as the loss function of the K-means clustering algorithm: Where J is the loss function value; N is the total number of data points; is the i-th data point; is the jth cluster center; is the Euclidean distance;
[0085] When clustering, K data points are randomly selected as the initial cluster centers;
[0086] For each data point, find the nearest cluster center and assign the data point to the corresponding cluster;
[0087] For each cluster, the cluster center is recalculated using the following formula: In the formula is the kth cluster center; is the number of data points in the kth cluster; is the i-th data point;
[0088] Repeat the above formula to iteratively update the cluster center until the set requirements are met (for example, the update of the cluster center is very small or the preset number of iterations is reached); finally, K cluster centers are obtained; at this time, each cluster center is used as a typical feature of the image data, and all cluster centers constitute a codebook;
[0089] Based on the obtained codebook and residual accumulation, the residual vector of each cluster center is calculated:
[0090] Compare each feature vector with the cluster center in the codebook and determine the cluster center to which each feature vector is closest;
[0091] For each eigenvector, the residual between the eigenvector and the closest cluster center is calculated using the following formula: : In the formula is the feature vector The closest cluster center to the corresponding The distance between
[0092] For each cluster center, the residual vector of the cluster center is calculated using the following formula: : In the formula Represents the feature vector Assigned to cluster center k;
[0093] Based on principal component analysis, the residual vector is reduced in dimension:
[0094] The residual vector of each cluster center obtained by the above steps is Different cluster centers may have different amplitudes due to different numbers of features assigned to them. To eliminate the amplitude differences, the residual vector of each cluster center is normalized to ensure that the length of each vector is uniform;
[0095] Connect the normalized residual vectors to obtain the global residual vector;
[0096] The global residual vector is normalized again to obtain a normalized global residual vector to optimize its performance and stability;
[0097] In order to further reduce the dimension of the vector and improve the retrieval efficiency, the normalized global residual vector is subjected to principal component analysis to obtain the output of the final vector generation module. In specific implementation, the goal of principal component analysis PCA is to find a transformation matrix to map the original vector to a low-dimensional subspace so that the projected vector retains as much data variance as possible. The process is expressed as Where Y is the vector after dimensionality reduction; X is the vector before dimensionality reduction; V is the PCA transformation matrix; by selecting a suitable PCA transformation matrix V, redundant information and noise in the data can be removed, and the most important features can be extracted to achieve dimensionality reduction of the vector, thereby improving retrieval efficiency;
[0098] The vector generated by the vector generation module is a powerful image representation that can capture and encode key local feature information in the image, reduce the dimension of the data while retaining key information, and significantly improve processing speed and accuracy;
[0099] Based on the deep learning model, a classification module is constructed; the specific steps include:
[0100] Use classifiers such as Transformer network, SVM support vector machine network or Adaboost network to build a classification module;
[0101] When the Transformer network is used to construct a classification module, the processing of the classification module includes the following steps:
[0102] The constructed classification module includes input layer, intermediate layer, encoding layer and classification layer;
[0103] The input layer, the middle layer, the encoding layer, and the classification layer are connected in series;
[0104] Input layer: The input layer is used to receive the output of the vector generation module;
[0105] Middle layer: The middle layer includes the first fully connected layer, the first normalized layer, the second fully connected layer and the second normalized layer connected in series; the first fully connected layer is used to map the input features to a space of set dimensions, and the first normalized layer is used to annotate the output of the first fully connected layer; the second fully connected layer is used to compress the input features to a space of set dimensions; the second normalized layer is used to standardize the output of the second fully connected layer; the processing process of the first fully connected layer is expressed as , is the output of the first fully connected layer, is the input of the first fully connected layer, is the weight matrix of the first fully connected layer, is the bias vector of the first fully connected layer; the processing process of the second fully connected layer is expressed as , is the output of the second fully connected layer, is the input of the second fully connected layer, is the weight matrix of the second fully connected layer, is the bias vector of the second fully connected layer;
[0106] Encoding layer: The encoding layer is based on a multi-head self-attention mechanism and a feedforward network to capture the dependencies between various positions in the input features. The processing of the encoding layer is expressed as: In the formula is the output of the l+1th layer; is the output of the lth layer; is the output of the l+1th layer through the feedforward network;
[0107] Classification layer: The classification layer uses a fully connected layer to convert the output of the encoding layer into a category prediction result; the processing of the fully connected layer is expressed as , y is the output of the fully connected layer, is the input of the fully connected layer, is the weight matrix of the fully connected layer, is the bias vector of the fully connected layer;
[0108] After the Transformer network is used to build the classification module, the remaining two methods can call the modules in the scikit-learn machine learning block for training;
[0109] S4. Using the training data set constructed in step S2, the initial pathological image classification model constructed in step S3 is trained to obtain a pathological image classification model;
[0110] In specific implementation, the training comprises the following steps:
[0111] The following loss function is used: Where L is the loss function value; C is the total number of categories; is the true category label of the target; The probability that the target predicted by the model is category i;
[0112] The Adam optimizer is used to train the model:
[0113] During training, the following formula is used to adjust the learning rate of each parameter in the model: In the formula is the model parameter value at the t+1th time; is the model parameter value of the tth time; Bias-corrected second-order moment estimate of the gradient; is the bias-corrected first-order moment estimate of the gradient; is the learning rate and , is the initial learning rate, is the attenuation factor, s is the step size, is the floor rounding function;
[0114] S5. Using the pathological image classification model obtained in step S4, actual pathological image classification is performed.
[0115] The advantages of the method of the present invention are described below in conjunction with embodiments:
[0116] In the public CPTAC dataset, three types of cancer, LUAD (lung adenocarcinoma), CCRCC (renal clear cell carcinoma) and PDA (pancreatic ductal adenocarcinoma), were tested. The ACC indicator is the accuracy indicator; the AUC indicator is the area under the ROC curve; both indicators are used to reflect the classification ability of the model.
[0117] The method of the present invention is compared with the existing solution, and the comparison results are shown in Table 1:
[0118] Table 1 Schematic diagram of the comparison results of species identification
[0119]
[0120] Then, different cancer subtypes are classified. In this embodiment, LUAD (lung adenocarcinoma) subtype and LSCC (lung squamous cell carcinoma) subtype under NLCSC (non-small cell lung cancer) are selected for subtype classification and comparative experiments. The results are shown in Table 2:
[0121] Table 2 Schematic diagram of subtype identification comparison results
[0122]
[0123] In Tables 1 and 2, the three schemes MIL, CLAM-MB and CLAM-SB are all proposed by Ming Y Lu, Drew FKWilliamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and FaisalMahmood et al. in the paper Data efficient and weakly supervised computationalpathology on whole slide images in 2020; the DS-MIL scheme is proposed by Bin Li, Yin Li, andKevin W Eliceiri in the paper Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastivelearning in 2021; the HIPT-N scheme is proposed by Richard J. Chen1, Chengkuan Chen et al. in the paper Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning in 2022;
[0124] It can be seen from Table 1 and Table 2 that the scheme of the present invention has obvious advantages in both species identification and subtype identification, which shows that the scheme of the present invention has higher reliability and better accuracy.
[0125] like Figure 2 The figure shows a schematic diagram of the method flow of the imaging method of the present invention: the imaging method disclosed in the present invention, which includes the pathological image classification method based on deep learning, comprises the following steps:
[0126] S1. Obtain existing pathological images;
[0127] S2. Preprocessing the pathological image obtained in step S1 to construct a training data set;
[0128] S3. Build an initial model for pathological image classification based on the Vision Transformer network;
[0129] S4. Using the training data set constructed in step S2, the initial pathological image classification model constructed in step S3 is trained to obtain a pathological image classification model;
[0130] S5. Using the pathological image classification model obtained in step S4, the actual pathological image classification is performed;
[0131] S6. The pathological image classification result obtained in step S5 is marked and re-imaged on the actual pathological image, and finally a pathological image with the pathological image classification result is output.
[0132] The imaging method disclosed in the present invention can be directly applied to existing machines and equipment for acquiring pathological images, or directly applied to a terminal (such as a computer); in specific applications, the existing scheme is used to acquire the actual pathological image, and then the acquired pathological image data is input into the corresponding machine equipment or terminal. At this time, the machine equipment or terminal can obtain the classification prediction result of the pathological image according to the imaging method disclosed in the present invention, and display the classification prediction result on the pathological image through different types of representation (such as color, text, etc.), and then perform secondary imaging and output; at this time, the output image is the pathological image with the classification prediction result of the pathological image, which greatly facilitates the subsequent work of clinical medical personnel and laboratory experimenters.
Claims
1. A pathological image classification method based on deep learning, characterized in that The steps include: S1. Obtain existing pathological images; S2. Preprocessing the pathological image obtained in step S1 to construct a training data set; S3. Build an initial model for pathological image classification based on the Vision Transformer network; The constructed model includes a feature extraction module, a vector generation module, and a classification module; A feature extraction module is constructed based on the Vision Transformer network. The feature extraction module is used to extract features from the input pathological image and output a feature vector for each image block. A vector generation module is constructed based on clustering, residual accumulation and principal component analysis. The vector generation module is used to reduce the dimension of the feature vector of each image block to improve the processing speed and accuracy of the model. The specific steps include: The feature vector of each image block output by the feature extraction module is clustered using the K-means clustering algorithm to obtain the codebook: The following formula is used as the loss function of the K-means clustering algorithm: Where J is the loss function value; N is the total number of data points; is the i-th data point; is the jth cluster center; is the Euclidean distance; Randomly select K data points as initial cluster centers; For each data point, find the nearest cluster center and assign the data point to the corresponding cluster; For each cluster, the cluster center is recalculated using the following formula: In the formula is the kth cluster center; is the number of data points in the kth cluster; is the i-th data point; Repeat the above formula to iteratively update the cluster center until the set requirements are met; finally, K cluster centers are obtained; at this time, each cluster center is used as a typical feature of the image data, and all cluster centers constitute a codebook; Based on the obtained codebook and residual accumulation, the residual vector of each cluster center is calculated: Compare each feature vector with the cluster center in the codebook and determine the cluster center to which each feature vector is closest; For each eigenvector, the residual between the eigenvector and the closest cluster center is calculated using the following formula: : In the formula is the feature vector The closest cluster center to the corresponding The distance between For each cluster center, the residual vector of the cluster center is calculated using the following formula: : In the formula Represents the feature vector Assigned to cluster center k; Based on principal component analysis, the residual vector is reduced in dimension: Normalize the residual vector of each cluster center; Connect the normalized residual vectors to obtain the global residual vector; The global residual vector is normalized again to obtain a normalized global residual vector; Perform principal component analysis on the normalized global residual vector to obtain the output of the final vector generation module; Build a classification module based on deep learning models; The classification module is used to complete the classification of pathological images according to the vectors generated by the vector generation module; S4. Using the training data set constructed in step S2, the initial pathological image classification model constructed in step S3 is trained to obtain a pathological image classification model; S5. Using the pathological image classification model obtained in step S4, actual pathological image classification is performed.
2. The pathological image classification method based on deep learning according to claim 1 is characterized in that The existing pathological image described in step S1 specifically includes the following contents: The existing pathological images include several types of pathological images of the same type of disease, and the type corresponding to each pathological image is marked; Alternatively, the existing pathological images include pathological images containing lesions and pathological images without lesions and normal for any disease, and the category corresponding to each pathological image is marked; the categories include categories containing lesions and categories without lesions and normal.
3. The pathological image classification method based on deep learning according to claim 2 is characterized in that The preprocessing described in step S2 specifically includes the following steps: Center cropping: center cropping the pathological image obtained in step S1 to ensure that the width and height of the image reach the set size; Image segmentation: Use a non-overlapping grid to divide the center-cropped image into several image blocks; each block is used as a basic unit; Reading and filtering of image blocks: extract each image block; for each image block, analyze the pixel value of the total red channel in the image block, count the number of pixels whose pixel values are less than the set value, identify the image blocks whose pixel numbers are less than the set value as background blocks and filter them, completing the reading and filtering of image blocks.
4. The pathological image classification method based on deep learning according to claim 3 is characterized in that The feature extraction module is constructed based on the Vision Transformer network, and specifically includes the following steps: Rearranging the image blocks obtained in step S2; Decompose each image block into several patches according to the set requirements, and add a set marker at the beginning of each patch sequence; The purpose of setting labels is to help the model capture the global information of the sequence; The patch with the set mark is input into the pre-trained ViT_256 model: The ViT_256 model encodes each patch. During the encoding process, each patch is encoded through several layers of Transformer networks; each layer of Transformer network encodes the context information of each element in the sequence of patches based on the self-attention mechanism and feedforward neural network; the processing process of each layer of Transformer network is expressed as and ,in is the layer normalization operation, x is the input feature, is the multi-head attention mechanism processing function, is a fully connected network processing function; after being encoded by several layers of Transformer networks, the output of the set mark is considered to represent the global information of the entire input image block; the set mark set of each image block is aggregated to obtain the global features of the image block; The global features of the obtained image blocks are reshaped into a three-dimensional array, and the reshaped three-dimensional array is transposed to obtain the feature map of each image block.
5. The pathological image classification method based on deep learning according to claim 4 is characterized in that The classification module is constructed based on the deep learning model, and specifically includes the following steps: The classification module is constructed using Transformer network, SVM support vector machine network or Adaboost network.
6. The pathological image classification method based on deep learning according to claim 5 is characterized in that When the Transformer network is used to construct a classification module, the processing of the classification module includes the following steps: The constructed classification module includes input layer, intermediate layer, encoding layer and classification layer; The input layer, the middle layer, the encoding layer and the classification layer are connected in series; Input layer: The input layer is used to receive the output of the vector generation module; Middle layer: The middle layer includes the first fully connected layer, the first normalized layer, the second fully connected layer and the second normalized layer connected in series; the first fully connected layer is used to map the input features to a space of set dimensions, and the first normalized layer is used to annotate the output of the first fully connected layer; The second fully connected layer is used to compress the input features into a space of set dimensions; the second normalization layer is used to standardize the output of the second fully connected layer; the processing process of the first fully connected layer is expressed as , is the output of the first fully connected layer, is the input of the first fully connected layer, is the weight matrix of the first fully connected layer, is the bias vector of the first fully connected layer; the processing process of the second fully connected layer is expressed as , is the output of the second fully connected layer, is the input of the second fully connected layer, is the weight matrix of the second fully connected layer, is the bias vector of the second fully connected layer; Encoding layer: The encoding layer is based on a multi-head self-attention mechanism and a feedforward network to capture the dependencies between various positions in the input features. The processing of the encoding layer is expressed as: In the formula is the output of the l+1th layer; is the output of the lth layer; is the output of the l+1th layer through the feedforward network; Classification layer: The classification layer uses a fully connected layer to convert the output of the encoding layer into a category prediction result; the processing process of the fully connected layer is expressed as , y is the output of the fully connected layer, is the input of the fully connected layer, is the weight matrix of the fully connected layer, is the bias vector of the fully connected layer.
7. The pathological image classification method based on deep learning according to claim 6 is characterized in that The training described in step S4 specifically includes the following steps: The following loss function is used: Where L is the loss function value; C is the total number of categories; is the true category label of the target; The probability that the target predicted by the model is category i; The Adam optimizer is used to train the model: During training, the following formula is used to adjust the learning rate of each parameter in the model: In the formula is the model parameter value at the t+1th time; is the model parameter value of the tth time; Bias-corrected second-order moment estimate of the gradient; is the bias-corrected first-order moment estimate of the gradient; is the learning rate and , is the initial learning rate, is the attenuation factor, s is the step size, is the floor function.
8. An imaging method comprising the pathological image classification method based on deep learning according to any one of claims 1 to 7, characterized in that The following steps are also included: S6. The pathological image classification result obtained in step S5 is marked and re-imaged on the actual pathological image, and finally a pathological image with the pathological image classification result is output.
