Histopathology image classification method based on color deconvolution and self-attention model
Through the combination of color deconvolution and self-attention model, the problems of low efficiency and low accuracy of histopathological image classification in the prior art are solved, and high-accuracy breast cancer tissue cell image classification is achieved, which enhances the model Robustness and classification effects.
Patent Information
- Application Number
- CN202111411631.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-25
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-11-25
AI Technical Summary
The prior art has cumbersome steps, low efficiency and low accuracy in histopathological image classification, making it difficult to quickly and accurately perform automated analysis of breast cancer tissue cell images.
The color deconvolution technology is used to preprocess tissue cell images, and the self-attention model is used to classify it. Combined with offline and online image color deconvolution methods, fine-tuning training is performed through the pre-trained swin-transformer model to improve image classification. Accuracy.
It significantly improves the accuracy of histopathological image classification, enhances the robustness of the model to the staining differences of histopathological image, prevents overfitting, and achieves good image classification effects.
Smart Images

Figure CN114066874B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical tissue cell image processing, and specifically relates to a tissue cell image classification method combining color deconvolution with a self-attention model. Background Art
[0002] According to data released by the World Health Organization's International Agency for Research on Cancer, in 2020, there were approximately 19.3 million new cancer cases and approximately 10 million deaths worldwide. Among them, the number of new breast cancer cases reached 2.26 million, accounting for 11.7% of the world's new cancer cases, exceeding the 2.2 million new cases of lung cancer, becoming the world's number one cancer. In the diagnosis of breast cancer, examination of breast tissue sections by pathologists remains the gold standard for clinical diagnosis. The development of digital pathology has led to an explosive growth in the number of digital tissue cell images. With the improvement of people's health awareness and the gradual popularization of breast cancer diagnosis, there is an urgent need to find an automated method for rapid analysis of tissue pathology images.
[0003] In tissue pathology, the most commonly used stains are hematoxylin and eosin. Hematoxylin binds to nucleic acids, staining the cell nucleus dark blue or purple, while eosin adheres to proteins in the tissue, staining the cytoplasm and extracellular matrix pink. Through the color deconvolution operation, the RGB image of tissue cells stained with hematoxylin-eosin can be color deconvolved, and the contribution of each stain can be calculated based on the absorbance of the specific stain, thereby achieving the effect of staining and separating tissue staining.
[0004] In the past, the classification of tissue pathology images often involved manual extraction of image features or the use of machine learning methods. However, these methods have the characteristics of cumbersome steps, low efficiency, and low accuracy, making them difficult to use in the actual classification process of tissue cell images. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides a histopathology image classification method based on color deconvolution and self-attention model, which improves the accuracy of histopathology image classification by performing a deconvolution operation on tissue cell images and then classifying the deconvolved images using a self-attention model.
[0006] The histopathological image classification method based on color deconvolution and self-attention model of the present invention adopts the following scheme: it includes adopting an offline image color deconvolution method or an online image color deconvolution method;
[0007] The steps of offline image color deconvolution method are as follows:
[0008] S100, obtaining a dataset of histopathological images stained with hematoxylin-eosin, wherein the histopathological images in the dataset are standard RGB three-channel color images, and the pixels thereof constitute a matrix H t ;
[0009] S200, the matrix H t The pixel value of 0 is set to 1×10 -7 , and then standardize each pixel value to form a new matrix H t ';
[0010] S300, the new matrix H t 'Multiply with the color deconvolution standard matrix to get the matrix D t , matrix D t That is the pixel matrix H t The matrix after color deconvolution;
[0011] S400, matrix D t Normalize and get the matrix D t ', matrix D t 'All elements are in the interval [0,1];
[0012] S500, the matrix D t 'Multiply each element by 255 and keep the integer to get the HED color space image;
[0013] S600, dividing the HED color space image into a training set, a validation set, and a test set according to a certain ratio, which are used for subsequent model training, validation, and evaluation, respectively, and scaling all images; performing online data enhancement on the training set images;
[0014] S700, normalizing and standardizing the enhanced image;
[0015] S800, using the swin-transformer model pre-trained on the ImageNet dataset, modify the last fully connected layer to a single MLP (Multilayer Perception) layer, where the number of output neurons in the MLP layer is 2, and send the training set images normalized by S700 to the model for fine-tuning training;
[0016] S900, during the training process, the validation set is used for verification every fixed number of iterations, and the model with the highest classification accuracy on the validation set is selected during the iteration process, and then the test set is used for verification to obtain the final model classification accuracy.
[0017] The steps of online image color deconvolution method are as follows:
[0018] T100, obtain a standard RGB three-channel histopathology image dataset stained with hematoxylin-eosin;
[0019] T200, divide the images in the data set into training set, validation set, and test set according to a certain ratio, which are used for subsequent model training, validation, and evaluation, respectively. All images are scaled to images of size w×w, and then online data enhancement operations are performed on the training set images;
[0020] T300, performing the same normalization operation as step S400 and the same standardization operation as step S700 on the enhanced image;
[0021] T400, using the swin-transformer self-attention model pre-trained on the ImageNet dataset, modifying the last fully connected layer, and adding an input negation operation to the model head, that is, for each input value x, output -x. Then add another convolution layer, denoted as conv1, where the convolution kernel size is 1×1, the number of input channels and the number of output channels are both 3, and no bias parameter is used. The modified model is denoted as the de-swt model;
[0022] T500, using the swin-transformer model pre-trained on the ImageNet dataset, the last fully connected layer of the model is changed to a single MLP layer, the number of output neurons in the MLP layer is 2, and an input negation operation is added to the head of the model, that is, for each input value x, -x is output, and then a convolution layer is added, denoted as conv1, where the convolution kernel size is 1×1, the number of input channels and the number of output channels are both 3, and no bias parameter is used. The modified model is denoted as the de-swt model;
[0023] T600, split the color deconvolution standard matrix into three 3×1 matrices by column and load them into the weight parameters of the conv1 layer of the de-swt model;
[0024] T700, input the standardized training set in step T300 into the de-swt model, and use a specific learning rate lr for fine-tuning training. In order to prevent the gradient of the de-swt model from disappearing when back-propagating to the conv1 layer, the learning rate of the conv1 layer is set to p×lr, where p is the learning rate magnification factor. p is used to increase the learning rate of the model in the conv1 layer, so that the parameters of the conv1 layer change with the iteration of the training set;
[0025] T800, during the training process of step T600, the model is verified using the validation set data at a certain interval of iterations, the model with better effect on the validation set is selected, and the parameters of the model conv1 layer are extracted and combined into a new image color deconvolution matrix by column, denoted as N;
[0026] T900, load each column value of the color deconvolution matrix N into the conv1 layer parameters of the de-swt model, and reload the swin-transformer parameters pre-trained on the ImageNet dataset into the remaining corresponding layers of the de-swt model, set p to 1, so that the learning rate in the conv1 layer is reset to lr;
[0027] T1000, use the training set to retrain the de-swt model obtained in step T800, and use the validation set data to validate the model at certain intervals of iterations. Finally, select the model with the highest accuracy on the validation set, and use the test set to test the model classification results, so as to obtain the final model classification accuracy.
[0028] Further, in step S200, the pixel value of the pixel point with a pixel value of 0 in the data set is set to 1×10-7, and then the pixel value of the i-th row and j-th column of the k-th channel of Ht is calculated. calculate:
[0029]
[0030] Among them, k represents the three channels of R, G, and B, and then the standardized pixel value is obtained. And form a new matrix Ht'.
[0031] Further, in step S300, the matrix Ht' is multiplied by the color deconvolution standard matrix M, where M is:
[0032]
[0033] Assume m k,c (1≤k,c≤3) is the element in the kth row and cth column of the matrix M, (c=1,2,3, represents the three channels of the image; 1≤i≤wt,1≤j≤ht) is the pixel point in the i-th row and j-th column of the c channel in the pixel matrix Dt, then:
[0034]
[0035] The matrix Dt is constructed, and Dt is the result of color deconvolution of the original image pixel matrix Ht. The three color channels of Dt are recorded as H, E, and D, representing the staining information of hematoxylin, eosin, and diaminobenzidine, respectively. The obtained matrix Dt is the HED color space matrix.
[0036] Furthermore, in step S400, the matrix Dt is normalized to obtain Dt':
[0037]
[0038] Where (Dt)max and (Dt)min represent the maximum and minimum values of the elements in the matrix Dt, respectively, so that
[0039] Further, in step S700, the enhanced image is normalized in the same way as in step S400, and then the mean (μ1, μ2, μ3) = (0.458, 0.456, 0.406) and variance (σ1, σ2, σ3) = (0.229, 0.224, 0.225) of the three-channel pixels of the ImageNet dataset are selected.
[0040] By formula:
[0041]
[0042] Normalize each image.
[0043] Further, in steps S600 and T200, the training set is numerically enhanced, and the data enhancement includes random center rotation, horizontal flipping, vertical flipping, scaling, and random changes in image brightness, contrast, saturation, and hue; wherein the blank portion of the edge after the center of the image is rotated is filled with 0 pixels, and the image brightness, contrast, and saturation are randomly changed to 80% to 120%, 90% to 110%, and 90% to 110% of the original image, respectively, and the hue change range is -0.1 to 0.1.
[0044] Furthermore, in step T500, the deconvolution matrix is split into the following three 3×1 column vectors by column, denoted as z1, z2, and z3:
[0045]
[0046] And these three column vectors are loaded as the initial weight parameters of the conv1 layer of the de-swt model. In the calculation process of the de-swt model, each z1, z2, and z3 perform convolution operations on the input data respectively to obtain the i-th channel data after deconvolution, and finally obtain the three-channel information of H, E, and D respectively.
[0047] Furthermore, in step T700, the model with better effect on the verification set is selected as the model whose classification accuracy on the verification set is lower than the highest accuracy obtained in the iterative process.
[0048] Furthermore, the offline image color deconvolution method uses an attention mechanism in the swin-transformer model in the online image color deconvolution method.
[0049] Compared with the prior art, the present invention has the following beneficial effects:
[0050] 1. In step S200, the pixel value of 0 is set to 1×10 -7 , to prevent the logarithm of 0 pixels from getting negative infinitesimal, or it can be replaced with other smaller numbers such as: 1×10 -6 , and will not affect subsequent classification;
[0051] 2. In step S500, the matrix D t Each normalized element in ' is multiplied by 255 and the integer is retained to obtain the HED color space image, and the information of each channel of H, E, and D can be extracted to observe the results of dyeing and separation of each dye, which can be used to test the dyeing and separation effect;
[0052] 3. Only when the image is scaled in steps S600 and T200 will the height and width of the image be changed, and the scaled size will be suitable for the input of the subsequent model, while the deconvolution and image enhancement operations do not change the size of the image;
[0053] 4. Perform numerical enhancement on the training set, which changes the brightness, contrast, saturation, and hue of the training set images, so that the model can be robust to the staining differences of histopathological images and not be susceptible to classification errors caused by differences in tissue staining conditions. No data enhancement is performed on the validation set and the test set in order to enable the model to reflect its true classification effect by relying on the classification accuracy on the validation set and the test set.
[0054] 5. In step T800, the model with better effect on the validation set is selected to prevent the conv1 layer parameters from overfitting the validation set data;
[0055] 6. The swin-transformer model uses the attention mechanism. The color deconvolution operation is combined with the self-attention mechanism to achieve good image classification results.
[0056] 7. S700 and T900 use model parameters pre-trained on the ImageNet dataset, so in steps S600 and T300, the standard values and variance values of the pixels in the ImageNet dataset are used to standardize the dataset, which enables the model to achieve better classification results. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] The accompanying drawings described herein are used to provide a further understanding of the present application, in which:
[0058] Figure 1 Flow charts of two implementation methods of the present invention;
[0059] Figure 2 Detailed flow chart of two implementation methods of the present invention;
[0060] Figure 3 These are examples of benign and malignant samples in the BreakHis dataset;
[0061] Figure 4 This is an example of color deconvolution and channel separation for the BreakHis dataset;
[0062] Figure 5 Enhanced samples for the image data of the BreakHis dataset training set;
[0063] Figure 6 This is the curve of the classification accuracy change of the training set and the validation set during the training process of the offline image color deconvolution model.
[0064] Figure 7 This is the curve of the classification accuracy of the training set and the validation set during the training process of the online image color deconvolution model step T800;
[0065] Figure 8 This is the curve of the classification accuracy change of the training set and the validation set during the training process of the online image color deconvolution model step T1000. DETAILED DESCRIPTION
[0066] See also Figure 1-Figure 2 As shown in one of the embodiments, the embodiment includes two implementation methods: offline image color deconvolution or online image color deconvolution.
[0067] Offline image color deconvolution implementation method:
[0068] See also Figure 3As shown, S100, obtain a tissue pathology image dataset stained with hematoxylin-eosin. This embodiment selects the BreakHis breast cancer tissue pathology image dataset, which is 7909 images collected from 82 patients, including 24 benign patients, from which 2480 benign images were collected; 58 malignant patients, from which 5429 malignant images were collected, benign image and malignant image samples. There are four magnifications in the dataset, 40 times, 100 times, 200 times and 400 times. The image size is 700×460, which is scaled to an image of 224×224. The pixels of each image constitute a matrix H of size 224×224×3. t , where 3 corresponds to the R, G, and B channels of the image;
[0069] S200, (k=1,2,3, represents the R,G,B channels of the image; 1≤i,j≤224) is H t The elements in The value is within the standard RGB image pixel value range, that is, when When using 1×10 -7 Replace, so that Then calculate:
[0070]
[0071] Get the standardized value in And form the matrix H t ';
[0072] S300, the matrix H t 'Multiply with the color deconvolution standard matrix M, where M is:
[0073]
[0074] Assume m k,c (1≤k,c≤3) is the element in the kth row and cth column of the matrix M, (c=1,2,3, representing the three channels of the image; 1≤i,j≤224) is the pixel matrix D t The pixel point in the i-th row and j-th column of the c channel is:
[0075]
[0076] The matrix D t , D t That is the original image pixel matrix H t The result after color deconvolution, where Dt The three color channels are denoted as H, E, and D, representing the staining information of hematoxylin, eosin, and diaminobenzidine (DAB), respectively. The resulting matrix D t That is the HED color space matrix;
[0077] S400, use the formula:
[0078]
[0079] For the matrix D t Normalize to get D t ', where (D t ) max ,(D t ) min Denote the matrix D t The maximum and minimum values of the elements in , so that 0≤(y c i,j )'≤1;
[0080] See also Figure 4 As shown, S500, the matrix D obtained after step S400 is t 'Multiply each element in by 255 and keep the integer, then you can get the HED color space image after color deconvolution, and then D t 'The H, E, and D channels of the image are extracted separately to obtain the image after staining separation. The effect of staining separation can be verified by observing these three channel images, image color deconvolution and HED channel separation results;
[0081] S600. Divide the HED color space image obtained in step S500 into a training set, a validation set, and a test set according to a ratio of approximately 7:1.5:1.5, and use them for subsequent training, validation, and evaluation of the model, respectively. All images of each patient are divided into only one of the training set, validation set, and test set, and it is always ensured that images belonging to the same patient do not appear in any two data sets among the training set, validation set, and test set. Finally, 5769 images of 59 patients are used as training sets, 1063 images of 11 patients are used as validation sets, and 1077 images of the remaining 12 patients are used as test sets;
[0082] See also Figure 5As shown, S700 performs online random data enhancement on the training set data, sets the probability of horizontal flipping and vertical flipping of the image to 0.5, and randomly rotates the center by 30°, wherein the blank part of the edge after the image center is rotated is supplemented with 0 pixels, and the image brightness, contrast, and saturation are randomly changed to 80% to 120%, 90% to 110%, and 90% to 110% of the original image, and the hue change range is -0.1 to 0.1.
[0083] S800, perform the same normalization operation as step S400 on the enhanced image, and then select the mean (μ1, μ2, μ3) = (0.458, 0.456, 0.406) and variance (σ1, σ2, σ3) = (0.229, 0.224, 0.225) of the three-channel pixels of the ImageNet dataset. By formula:
[0084]
[0085] Normalize each image;
[0086] S900, using the swin-transformer model pre-trained on the ImageNet dataset, the last fully connected layer was modified to a single MLP (Multilayer Perception) layer, where the number of output neurons in the MLP layer was 2, and the normalized training set images were sent to the model for fine-tuning training. During the training process, the image batch size was set to 32, all training sets were trained for 20 epochs, and the model was trained for a total of 10 iterations. times. The initial learning rate of the model is set to 1×10 -5 , the learning rate is reduced to 0.5 times every 5 epochs, the loss value is calculated using the cross entropy loss function, and the Adam optimizer is used for optimization;
[0087] See also Figure 6 As shown in Figure 1, S1000, during the model training process, the validation set was used to validate the trained model every 72 iterations, and finally 50 classification accuracies of the validation set were obtained. The highest accuracy on the validation set was 93.59% ( Figure 6 The model with the highest classification accuracy in the validation set is selected and the model is verified with the test set. The final classification accuracy of the test set is 93.51%. The classification results of the test set are shown in Table 1 below.
[0088] Table 1: Classification results of swin-transformer model
[0089]
[0090] Online image color deconvolution implementation method:
[0091] See also Figure 3 As shown, T100 is the same as step S100 of the offline image color deconvolution solution embodiment, obtaining the BreakHis dataset image, scaling the image to a size of 224×224, and dividing it into a training set, a test set, and a validation set;
[0092] T200, performing corresponding online data enhancement on the training set images of the BreakHis dataset according to step S600 in the offline image color deconvolution scheme;
[0093] T300, through the formula:
[0094]
[0095] For each image D in BreakHis t Pixels x in k i,j Normalized, where (D t ) max ,(D t ) min Denote the matrix D t The maximum and minimum values of the elements in . Then through the formula:
[0096]
[0097] Right D t The pixels in are standardized, where (μ1, μ2, μ3) = (0.458, 0.456, 0.406), (σ1, σ2, σ3) = (0.229, 0.224, 0.225);
[0098] T400, using the swin-transformer model pre-trained on the ImageNet dataset, the last connection layer of the model is changed to a single MLP layer, the number of output neurons of the MLP layer is 2, and a negation operation is added to the head of the model, that is, for any input x, the output is -x. Then add another convolution layer, denoted as conv1, where the convolution kernel size is 1×1, the number of input channels and the number of output channels are both 3, and no bias parameter is used. The modified model is denoted as the de-swt (deconvolution swintransformer) model;
[0099] T500, split the deconvolution matrix into the following three 3×1 column vectors by column:
[0100]
[0101] And load these three column vectors as the initial weight parameters of the conv1 layer of the de-swt model;
[0102] T600, input the standardized training set in step T300 into the de-swt model, and set the initial learning rate lr of the model to 1×10 -5 The learning rate magnification factor p of the conv1 layer model is set to 10, that is, the initial learning rate of the conv1 layer is 1×10 -4 , and the learning rate of the remaining layers is 1×10 -5 During the training process, the image batch size was set to 32, all training sets were trained for 20 epochs, and the model was trained for a total of 100 iterations. The learning rate lr is reduced to 0.5 times every 5 epochs, and the Adam optimizer is used for optimization;
[0103] See also Figure 7 As shown in Figure 1, during the model training process, the trained model was verified with the validation set every 72 iterations, and finally 50 validation set classification accuracy rates were obtained. The highest validation set accuracy rate was 94.52%, which was the 14th validation set test result. The second highest validation set accuracy rate was 93.59%, which was obtained in the 15th validation set test result ( Figure 7 The model with a classification accuracy of 93.59% in the validation set is selected, and the conv1 layer parameters of the model are extracted as follows:
[0104]
[0105] Combining the vectors by columns gives us the new color deconvolution matrix M':
[0106]
[0107] T800, reload the swin-transformer model part in de-swt with the parameters trained on the ImageNet dataset, where the conv1 layer learning rate magnification p is set to 1, so that the learning rates of all layers of the model are consistent. Then retrain the model, where the image batch size is set to 32, all training sets are trained for 20 epochs, and the model is trained for a total of 100 iterations. times. The initial learning rate of the model is set to 1×10 -5 , the learning rate is reduced to 0.5 times every 5 epochs, the cross entropy loss function is used, and the Adam optimizer is used for optimization;
[0108] See Figure 8As shown in Figure 1, in step T900, during the process of retraining the model in step T800, the trained model is verified with the validation set every 72 iterations, and finally 50 classification accuracy rates of the validation set are obtained. The model with the highest classification accuracy of the validation set is selected ( Figure 8 The highest accuracy rate on the validation set is 95.36%, and the model is verified with the test set, and the classification accuracy rate of the test set is 94.17%. The classification results are shown in Table 2 below.
[0109] Table 2: Classification results of de-swt model
[0110]
[0111] The above is the best embodiment of the present invention. The results of the embodiment show that combining color deconvolution with the self-attention model can achieve good image classification results. Both the offline image color deconvolution and the online image color deconvolution schemes use the swin-transformer model because the model uses the attention mechanism. The models that can be used here are not limited to the swin-transformer model.
[0112] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A histopathological image classification method based on color deconvolution and self-attention model, characterized by: It includes adopting an offline image color deconvolution method or an online image color deconvolution method; The steps of offline image color deconvolution method are as follows: S100, obtaining a dataset of histopathological images stained with hematoxylin-eosin, wherein the histopathological images in the dataset are standard RGB three-channel color images, and the pixels thereof constitute a matrix H t ; S200, the matrix H t The pixel value of 0 is set to 1×10 -7 , and then standardize each pixel value to form a new matrix H t '; S300, the new matrix H t 'Multiply with the color deconvolution standard matrix to get the matrix D t , matrix D t That is the pixel matrix H t The matrix after color deconvolution; the color deconvolution standard matrix M is: Assume m k,c is the element in the kth row and cth column of the matrix M, and 1≤k,c≤3; is the pixel point in the i-th row and j-th column of the c-th channel in the pixel matrix Dt; where c = 1, 2, 3 represents the three channels of the image; 1 ≤ i ≤ w t ,1≤j≤h t ,but: in, represents the pixel value of the i-th row and j-th column of the k-th channel of Ht, The matrix Dt is formed. Dt is the result of color deconvolution of the original image pixel matrix Ht. The three color channels of Dt are recorded as H, E, and D, which represent the staining information of hematoxylin, eosin, and diaminobenzidine, respectively. The obtained matrix Dt is the HED color space matrix. S400, matrix D t Normalize and get the matrix D t ', matrix D t 'All elements are in the interval [0,1]; S500, the matrix D t 'Multiply each element by 255 and keep the integer to get the HED color space image; S600, dividing the HED color space image into a training set, a validation set, and a test set according to a certain ratio, which are used for subsequent model training, validation, and evaluation, respectively, and scaling all images; performing online data enhancement on the training set images; S700, normalizing and standardizing the enhanced image; S800, using the swin-transformer model pre-trained on the ImageNet dataset, modifying the last fully connected layer to a single MLP layer, where the number of output neurons in the MLP layer is 2, and sending the training set images normalized by S700 to the model for fine-tuning training; S900, during the training process, the validation set is used for validation at fixed iteration times, and the model with the highest classification accuracy on the validation set is selected during the iteration process, and then the test set is used for validation, so as to obtain the final model classification accuracy; Or the steps of online image color deconvolution method are as follows: T100, obtain a standard RGB three-channel histopathology image dataset stained with hematoxylin-eosin; T200, divide the images in the data set into training set, validation set, and test set according to a certain ratio, which are used for subsequent model training, validation, and evaluation, respectively. All images are scaled to images of size w×w, and then online data enhancement operations are performed on the training set images; T300, normalize and standardize the enhanced image; T400, using the swin-transformer self-attention model pre-trained on the ImageNet dataset, modifying the last fully connected layer, and adding an input negation operation to the model head, that is, for each input value x, output -x; then add another convolution layer, denoted as conv1, where the convolution kernel size is 1×1, the number of input channels and the number of output channels are both 3, and no bias parameter is used. The modified model is denoted as the de-swt model; T500, split the color deconvolution standard matrix M into three 3×1 matrices by column, and load them into the weight parameters of the conv1 layer of the de-swt model; the color deconvolution standard matrix M is: T600, input the standardized training set in step T300 into the de-swt model, and use a specific learning rate lr for fine-tuning training. In order to prevent the gradient of the de-swt model from disappearing when back-propagating to the conv1 layer, the learning rate of the conv1 layer is set to p×lr, where p is the learning rate magnification factor. p is used to increase the learning rate of the model in the conv1 layer, so that the parameters of the conv1 layer change with the iteration of the training set; T700, during the training process of step T500, the model is verified using the validation set data at a certain interval of iterations, the model with better effect on the validation set is selected, and the parameters of the model conv1 layer are extracted, and combined into a new image color deconvolution matrix by column, recorded as N; T800, load each column value of the color deconvolution matrix N into the conv1 layer parameters of the de-swt model, and reload the swin-transformer parameters pre-trained on the ImageNet dataset into the remaining corresponding layers of the de-swt model, set p to 1, so that the learning rate in the conv1 layer is reset to lr; T900, use the training set to retrain the de-swt model obtained in step T700, and use the validation set data to validate the model at certain intervals of iterations. Finally, select the model with the highest accuracy on the validation set, and use the test set to test the model classification results, so as to obtain the final model classification accuracy.
2. The histopathological image classification method according to claim 1, characterized in that: In step S200, the pixel value of the pixel point with a pixel value of 0 in the data set is set to 1×10 -7 , then the pixel value of the i-th row and j-th column of the k-th channel of Ht calculate: Among them, k represents the three channels of R, G, and B, and then the standardized pixel value is obtained. And form a new matrix Ht'.
3. The method for histopathological image classification based on color deconvolution and self-attention model according to claim 1, characterized in that: In step S600 or T200, the training set is numerically enhanced, and the data enhancement includes random center rotation, horizontal flipping, vertical flipping, scaling, and random changes in image brightness, contrast, saturation, and hue; wherein the blank portion of the edge after the center of the image is rotated is supplemented with 0 pixels, and the image brightness, contrast, and saturation are randomly changed to 80% to 120%, 90% to 110%, and 90% to 110% of the original image, respectively, and the hue change range is -0.1 to 0.
1.
4. The method for histopathological image classification based on color deconvolution and self-attention model according to claim 1, characterized in that: In step T500, the color deconvolution standard matrix M is split into the following three 3×1 column vectors by column, denoted as z1, z2, and z3: And these three column vectors are loaded as the initial weight parameters of the conv1 layer of the de-swt model. In the calculation process of the de-swt model, each z1, z2, and z3 perform convolution operations on the input data respectively to obtain the i-th channel data after deconvolution, and finally obtain the three-channel information of H, E, and D respectively.
5. The method for histopathological image classification based on color deconvolution and self-attention model according to claim 1, characterized in that: In step T600, the learning rate of the conv1 layer is amplified by 10 times, and training is performed in step T700 to obtain the parameters of the new conv1 layer. Then, the layer parameters are migrated to the conv1 layer of the model in step T800, and the learning rate amplification factor of the conv1 layer is set to 1. Then, training is performed to obtain the final classification result.
6. The method for histopathological image classification based on color deconvolution and self-attention model according to claim 1, characterized in that: In step T700, the model with better effect on the verification set is selected to select the model whose classification accuracy on the verification set is lower than the highest accuracy obtained in the iterative process.
7. The method for histopathological image classification based on color deconvolution and self-attention model according to claim 1, characterized in that: The offline image color deconvolution method or the online image color deconvolution method uses an attention mechanism in a swin-transformer model.
