Breast pathology whole-slice classification method based on multi-feature fusion progressive discrimination

Abstract

The present invention discloses a breast pathology whole-slice classification method based on multi-feature fusion progressive discrimination. Considering the spatial correlation between the pathology image block and the surrounding image blocks, a variety of feature fusion methods are used to aggregate the features of the pathology image block and the surrounding image blocks to obtain a global descriptor. From the global descriptor, representative features and features of the heat map are selected for connection to obtain representative and explanatory whole-slice features. At the same time, a multi-feature fusion progressive discrimination method is used to perform the final recognition of the whole-slice image. The method can avoid the generation of outliers and eliminate the negative impact on the final whole-slice property judgment. The connection of the above two types of features can improve the overall representation ability of the whole-slice feature descriptor, so that the trained recognition model has higher accuracy. At the same time, the progressive discrimination method reduces the generation of false negatives in the classification process, making it more suitable for clinical diagnosis and improving the diagnosis quality and efficiency of doctors.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging diagnosis technology and graphics and image processing software in the field of electronic information, and in particular to a breast pathology full-slice classification method based on multi-feature fusion progressive discrimination. Background Art

[0002] The diagnosis of breast cancer requires microscopic image evaluation to detect tumors at an early stage and to design treatment methods based on the pathological diagnosis results.

[0003] With the development of microphotography and whole-slice scanning technology, pathological sections can be saved in the form of digital images, making it possible to apply computer vision methods to the field of pathology. [1] Whole Slide Images (WSIs) are digital slides with full fields of view that are generated by scanning and splicing traditional glass slides using a fully automatic microscope scanning system. They contain rich and detailed information from tissue to cell levels. The emergence of digital pathology has effectively saved the manpower and time costs of the hospital's pathology department, improved the quality and efficiency of pathology diagnosis, and alleviated the development difficulties of the pathology department to a certain extent.

[0004] In recent years, many scholars have studied the recognition of pathological images. Early methods for processing breast tissue images mostly used traditional machine learning algorithms. This type of algorithm is based on artificially designed features for classification. Methods based on artificial features and machine learning have certain limitations. Since cells of different categories have great similarities, feature extraction is extremely difficult, and it is impossible to design high-level abstract features. At present, with the widespread application of deep learning methods based on convolutional neural networks, research on computer-aided diagnosis technology for digital slices of histopathology has made significant progress.

[0005] The block sampling method is a method of full-slice classification based on deep learning. This method first cuts the full-slice image into multiple small-sized blocks, then inputs these small-sized blocks into the convolutional neural network to extract features, and then aggregates these features as the features of the full slice, and finally uses the classifier to classify the full slice features to obtain the final classification result. For example, Wang [2] et al. extracted 256×256 pixel blocks from sentinel lymph node slices of breast cancer using a sliding window method, and sent them into the trained GoogLeNet model. They combined the malignancy probabilities of all blocks to obtain a heat map of canceration in the whole slice, extracted 28 features in the heat map, including tumor area and number of tumor areas, and finally sent them into a random forest for classification of the whole slice. [3]et al. proposed a new EM-based method to automatically identify small patches with high discriminability for training CNN models, and train a decision fusion model to aggregate the image patch predictions given by CNN.

[0006] The above methods all directly use small-sized image patches as the input of the CNN model without considering the spatial correlation between these image patches. Generally speaking, the tumor area of ​​the whole slice is larger than the cut image patch, so that the image patches with high tumor probability are relatively clustered in space. When a patch is located in the tumor area, its adjacent patches are also likely to be marked as tumors because they are located in adjacent areas. In order to explicitly model the spatial correlation between adjacent image patches, Kong [4] et al. proposed a spatial network using a long short-term memory (LSTM) network to capture spatial correlation based on image patch features extracted from a CNN classifier. To detect cancer metastasis in whole-slice images, Li et al. [5] et al. proposed a neural conditional random field (NCRF) deep learning framework, which considers the spatial correlation between adjacent image blocks by integrating a CNN model and a fully connected conditional random field (CRF). However, the above two methods use a two-stage method to obtain spatial correlation. The CNN stage cannot obtain the spatial correlation between image blocks, resulting in the separation of the two stages and failure to achieve end-to-end calculation.

[0007] After obtaining the cancer property judgment of many small images, in order to obtain the cancer type of the entire image, it is necessary to summarize the properties of the small images to obtain the full slice feature. At present, the full slice feature extraction methods are mainly divided into two categories: one is the method based on the tumor area heat map [2] , which focuses on the overall morphological characteristics of the predicted tumor area of ​​the whole slice image and the characteristics of the entire tissue area, but the features of the heat map are manually designed and extracted, which has the limitations of low efficiency and insufficient accuracy; the other type is a method based on CNN feature aggregation [6] , it extracts the N-dimensional feature vector of each image block from the CNN model, and aggregates the features of the discriminative image blocks into the features of the full slice image. However, the features finally obtained are high-level abstract features extracted after multiple convolution operations, which are not very interpretable. Considering that the above two full slice feature extraction methods have their own advantages and disadvantages, the present invention takes the advantages of both by feature connection, splicing the features based on the heat map and the features based on the CNN, so that the features of the full slice image are more representative and at the same time increase the interpretability.

[0008] In the process of pathological image diagnosis, the tolerance for false negatives and false positives is different in clinic. Generally, people are more sensitive to false negatives and believe that false negatives are much more serious than false positives because false negatives can lead to misjudgment and missed diagnosis, miss the best treatment opportunity, and bring greater risks to patients.

[0009] Most existing full-slice classification methods judge one of the obtained features, such as Wang et al. [7] Among the full slice features obtained by multiple feature fusion methods, the one with the highest classification accuracy is selected as the final classification feature. However, the results given by relying solely on any one feature have a high false negative ratio, and the details of the image that each feature focuses on are different. Therefore, using these features one by one to judge the WSI images that were judged negative last time can minimize the false negative ratio. Through literature research, it is found that there are currently no related reports on this idea at home and abroad. Summary of the invention

[0010] In order to overcome the defects of the prior art, the present invention adopts a multi-feature fusion progressive discrimination method to progressively judge multiple features to reduce or avoid the generation of false negatives in the process of pathological image recognition.

[0011] 1) In response to the above-mentioned proposal of using small-sized image blocks as input of the CNN model without considering the spatial correlation between these image blocks, the present invention inputs multiple pathological image blocks as a whole into the CNN model, filters out image blocks of different types, and effectively avoids the emergence of abnormal outliers.

[0012] 2) In view of the limitations and poor interpretability of the full-slice features ultimately used for classification, the present invention aggregates the global descriptor and the heat map features to make the full-slice features more representative, detects the cancerous areas while classifying the full-slice images, and makes the recognition results more interpretable.

[0013] 3) In the process of pathological image diagnosis, false negatives will cause greater harm than false positives and single feature judgment is prone to cause false negatives. The present invention adopts a multi-feature progressive discrimination method, which uses each feature to judge the image judged as negative one by one, so as to reduce the occurrence of false negatives in the process of pathological image recognition.

[0014] The technical solution of the breast pathology full-slice classification method based on multi-feature fusion progressive discrimination of the present invention comprises the following steps:

[0015] S1: Use the Ostu method to remove blank areas in the whole slice, extract the tissue area in the pathological image, and use the random point method in the tissue area to prepare the training set and test set for CNN model training;

[0016] S2: Select the improved ResNet34 network as the image block recognition model and train it;

[0017] S3: Prepare a dataset for whole-slice image classification and input it into the trained image block recognition model. Extract the feature vector and cancer probability of the image block from the CNN model, perform feature aggregation on each block to form a block descriptor, and save the coordinates and cancer probability of each block to obtain a tumor area heat map.

[0018] S4: Use three feature fusion methods (average, maximum, weighted) to aggregate the block descriptors of the pathological image to obtain three types of class descriptors respectively, and aggregate all class descriptors to obtain the final global descriptor;

[0019] S5: Use manually designed features to extract tumor region heat map features;

[0020] S6: Select representative features from the global descriptor and connect them with the features of the heat map to obtain the feature descriptor of the final full-slice image;

[0021] S7: Use a multi-feature progressive decision method to make a final judgment on the three obtained full-slice feature descriptors.

[0022] Step S1 specifically includes the following steps:

[0023] S1.1: Manually mark the cancerous areas in breast pathology images; professional pathologists are usually invited to participate in manual marking;

[0024] S1.2: Use the Ostu segmentation method to remove blank areas in breast pathology images, extract tissue areas, and improve calculation efficiency;

[0025] S1.3: In a random point-selecting manner, image blocks of 244×244 pixels were extracted from the tumor area of ​​the breast pathology full sections marked as cancerous as the positive data set, and image blocks of 244×244 pixels were extracted from the normal tissue areas of the breast pathology full sections marked as cancerous and the breast pathology full sections marked as non-cancerous as the negative data set;

[0026] S1.4: Use the Reinhard color transfer method to normalize the colors of all image patches, and further divide the dataset into training and test sets.

[0027] Step S2 uses an improved ResNet34 network as the prediction model for the image block. The model is based on ResNet34. In order to make the network adapt to inputs of different sizes, the last fully connected layer is replaced with two fully convolutional layers. At the same time, the padding operations of all convolutional layers in ResNet34 are deleted to avoid boundary effects during prediction. Then, the training set is used to train ResNet34. Specifically, the following steps are included:

[0028] S2.1: Use data augmentation methods such as flipping, rotation, and cropping to increase the diversity of the training set;

[0029] S2.2: Set the learning rate, weight and other parameters in the model;

[0030] S2.3: Process the input data through the improved network structure, calculate the loss value, and back-propagate the gradient to the parameters of the neural network;

[0031] S2.4: Update the weight values ​​in the network according to the update rules.

[0032] In a specific embodiment, S3 specifically includes:

[0033] S3.1: Create a dataset for full-slice classification, extract 896×896 pixel blocks in the tissue area of ​​the pathological image using a sliding window method, and use Reinhard to normalize the color of the image blocks. We call the 896×896 pixel image block a Block, which can be regarded as a large grid composed of n×n image blocks. In the present invention, n=4;

[0034] S3.2: Directly input the Block into the trained convolutional neural network to predict the cancerous nature of the image block. During the prediction process, for each image block, a 512-dimensional feature f is output from the penultimate convolutional layer of the improved ResNet34, and its cancer probability p is output in the last convolutional layer. Then, the image block features in the Block are aggregated through three feature fusion methods to obtain the Block feature descriptor B m , where m∈[1,M], M is the number of feature fusion methods;

[0035] S3.3: Obtain the probability value of the Block. For each Block, take the average value of the cancer probability of each image block contained in it as the cancer probability of the entire Block (this is a malignant tumor probability value between 0 and 1, representing the probability of the Block suffering from a malignant tumor).

[0036] S3.4: Based on the cancer probability values ​​of all blocks, a heat map of the tumor area in the whole slice image is drawn, and the location of the malignant tumor is highlighted.

[0037] In a specific embodiment, the three feature fusion methods described in S3.2 include:

[0038] 1) Weighted: Consider the features of all image blocks in the same block, and assume that the contribution of each image block feature to the block feature is proportional to its cancer probability. The calculation formula is as follows:

[0039]

[0040] 2) Take the maximum value: The feature of the image block with the highest probability is taken as the feature of the block. The calculation formula is as follows:

[0041]

[0042] 3) Take the average value: Take the average of the features of all image blocks in the Block as the feature of the Block. The calculation formula is as follows:

[0043]

[0044] Where B represents the Block descriptor, n is the number of image blocks in each row (column) of the Block, and p i,j , f i,j are the cancer probability and characteristics of the image block in the i-th row and j-th column in the Block.

[0045] Step S4 specifically includes the following steps:

[0046] S4.1: Whether the Block has discriminative power is determined by judging whether the probability of the Block exceeds a certain threshold α. The present invention selects the threshold α=0.3.

[0047] S4.2: In the full slice image, the features B of all discriminative blocks are m Aggregate to form class feature C m .

[0048] The class characteristics are:

[0049]

[0050] Among them B k,m is B in step S3 m The obtained block feature descriptor, N represents the number of discriminant blocks.

[0051] Step S5 specifically includes the following steps:

[0052] After obtaining the malignant tumor heat map of the whole slice tissue pathology image, the geometric and morphological feature information of the malignant tumor area is extracted from the heat map, and the information is combined to form the feature H of the heat map;

[0053] The features of the extracted heat map include:

[0054] Such as the number of cancer cell areas, the maximum, mean and variance of the area and perimeter of the malignant tumor area, the percentage and density value of the malignant tumor area in the entire tissue area, the size of the longest axis in the largest malignant tumor area, the size of the eccentricity of the malignant tumor area, and other 31 morphological and geometric characteristics.

[0055] The characteristics of the extracted heat map are shown in Table 1.

[0056] Table 1 Heat map features

[0057]

[0058]

[0059] Step S6 specifically includes the following steps:

[0060] S6.1: Use feature selection to eliminate the class descriptor C obtained in S4 m The irrelevant or redundant features in the class feature subset D are obtained m , thereby reducing the number of features, improving model accuracy, and reducing running time;

[0061] S6.2: Subset D of class features after feature selection m Connect with the feature H of the heat map extracted by S5 to form the final feature G of the whole slice image m .

[0062] The feature selection refers to selecting features based on the feature differences between positive samples and negative samples. Let pos and neg represent the cancerous images and normal images in the training set respectively, N pos and N neg Represents the number of cancerous images and normal images, v i,k represents the k-th dimension feature of the i-th image, then the difference diff of the k-th feature k for:

[0063]

[0064] Then sort all feature differences from largest to smallest, and select the top 100 features with the largest feature differences to obtain the class feature subset D m .

[0065] Step S7 specifically includes the following steps:

[0066] S7.1: Extract the features of the three full-slice images of the training set (G1, G2, G3), input the three feature sets into the random forest respectively, train the random forest, and obtain the decision makers (RF1, RF2, RF3);

[0067] S7.2: Input the feature G1 of the whole slice image to the random forest decision maker RF1 to determine the type of the whole slice, that is, the two types of cancer and non-cancer. m If the type of the whole slice image is determined to be cancerous, then the step ends and the final determined type of the whole slice is cancerous; otherwise, step S7.3 is executed;

[0068] S7.3: Input feature G2 into random forest decision maker RF2 to determine the type of the whole slice. If RF2 determines that the type of the whole slice image is cancerous, then end the step and output the final determined type of the whole slice as cancerous; otherwise, execute step S7.4;

[0069] S7.4: Input feature G3 into the random forest decision maker RF3 to determine the type of the whole slice. If RF3 determines that the type of the whole slice image is cancer, then the final determination type of the whole slice is output as cancer. If the random forest decision maker RF3 determines that the whole slice image is cancer-free, then the final determination type of the whole slice is output as cancer-free. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of the breast pathology full-slice classification method based on multi-feature fusion progressive discrimination of the present invention;

[0071] Figure 2 The present invention is a flowchart of progressive decision-making in the breast pathology full-slice classification method based on multi-feature fusion progressive discrimination. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] like Figure 1 As shown, the technical solution of the breast pathology full-slice classification method based on multi-feature fusion progressive discrimination of the present invention includes the following steps:

[0074] S1: Use the Ostu method to remove blank areas in the whole slice, extract the tissue area in the pathological image, and use the random point method in the tissue area to prepare the training set and test set for CNN model training;

[0075] S2: Select the improved ResNet34 network as the image block recognition model, and use the training set of the CNN model to train the network parameters using the stochastic gradient descent algorithm;

[0076] S3: The whole slice image is cut into blocks in a sliding window manner and input into the trained image block recognition model, the feature vector and cancer probability of the image block are extracted from the recognition model, the features of each block are aggregated to form a block descriptor, and the coordinates and cancer probability of each block are saved to obtain the tumor area heat map;

[0077] S4: The block descriptors of the pathological image are aggregated using the three feature fusion methods of averaging, maximizing, and weighting to obtain three types of class descriptors respectively, and all class descriptors are aggregated to obtain the final global descriptor;

[0078] S5: 31 geometric and morphological features of the malignant tumor area are extracted from the heat map and combined to form the features of the heat map;

[0079] S6: Select representative features from the global descriptor and connect them with the features of the heat map to obtain the final feature descriptors of the three full-slice images; S7: Use a multi-feature progressive decision method to make a final judgment on the three obtained full-slice feature descriptors.

[0080] Step S1 specifically includes the following steps:

[0081] S1.1: Invite professional pathologists to manually mark the cancerous areas in breast pathology images;

[0082] S1.2: Use the Ostu segmentation method to remove blank areas in breast pathology images, extract tissue areas, and improve calculation efficiency;

[0083] S1.3: In a random point-selecting manner, image blocks of 244×244 pixels were extracted from the tumor area of ​​the breast pathology full sections marked as cancerous as the positive data set, and image blocks of 244×244 pixels were extracted from the normal tissue areas of the breast pathology full sections marked as cancerous and the breast pathology full sections marked as non-cancerous as the negative data set;

[0084] S1.4: Use the Reinhard color transfer method to normalize the colors of all image patches, and further divide the dataset into training and test sets.

[0085] Step S2 uses an improved ResNet34 network as the prediction model for the image block. The model is based on ResNet34. In order to make the network adapt to inputs of different sizes, the last fully connected layer is replaced with two fully convolutional layers. At the same time, the padding operations of all convolutional layers in ResNet34 are deleted to avoid boundary effects during prediction. Then, the training set is used to train ResNet34. Specifically, the following steps are included:

[0086] S2.1: Use data augmentation methods such as flipping, rotation, and cropping to increase the diversity of the training set;

[0087] S2.2: Set the learning rate, weight and other parameters in the model;

[0088] S2.3: Process the input data through the improved network structure, calculate the loss value, and back-propagate the gradient to the parameters of the neural network;

[0089] S2.4: Update the weight values ​​in the network according to the update rules.

[0090] In a specific embodiment, S3 specifically includes:

[0091] S3.1: Create a dataset for full-slice classification, extract 896×896 pixel blocks in the tissue area of ​​the pathological image using a sliding window method, and use Reinhard to normalize the color of the image blocks. We call the 896×896 pixel image block a Block, which can be regarded as a large grid composed of n×n image blocks. In the present invention, n=4;

[0092] S3.2: Directly input the Block into the trained convolutional neural network to predict the cancerous nature of the image block. During the prediction process, for each image block, a 512-dimensional feature f is output from the penultimate convolutional layer of the improved ResNet34, and its cancer probability p is output in the last convolutional layer. Then, the image block features in the Block are aggregated through three feature fusion methods to obtain the Block feature descriptor B m , where m∈[1,M], M is the number of feature fusion methods;

[0093] S3.3: Obtain the probability value of the Block. For each Block, take the average value of the cancer probability of each image block contained in it as the cancer probability of the entire Block (this is a malignant tumor probability value between 0 and 1, representing the probability of the Block suffering from a malignant tumor).

[0094] S3.4: Based on the cancer probability values ​​of all blocks, a heat map of the tumor area in the whole slice image is drawn, and the location of the malignant tumor is highlighted.

[0095] In a specific embodiment, the method of fusing the three features described in S3.2 includes:

[0096] 1) Weighted method: This method considers the features of all image blocks in the same block, and assumes that the contribution of each image block feature to the block feature is proportional to its cancer probability. The calculation formula is as follows:

[0097]

[0098] 2) Maximum method: This method uses the features of the image block with the highest probability as the features of the block. The calculation formula is as follows:

[0099]

[0100] 4) Average value method: This method takes the average of the features of all image blocks in the Block as the feature of the Block. The calculation formula is as follows:

[0101]

[0102] Where B represents the Block descriptor, n is the number of image blocks in each row (column) of the Block, and p i,j , f i,j are the cancer probability and characteristics of the image block in the i-th row and j-th column in the Block.

[0103] Step S4 specifically includes the following steps:

[0104] S4.1: Whether the Block has discriminative power is determined by judging whether the probability of the Block exceeds a certain threshold α. The present invention selects the threshold α=0.3.

[0105] S4.2: In the full slice image, the features B of all discriminative blocks are m Aggregate to form class feature C m .

[0106] In a specific embodiment, the class feature aggregation method includes:

[0107]

[0108] Among them B k,m is B in step S3 m The obtained block feature descriptor, N represents the number of discriminant blocks.

[0109] Step S5 specifically includes the following steps:

[0110] After obtaining the malignant tumor heat map of the whole slice tissue pathology image, the geometric and morphological feature information of the malignant tumor area is extracted from the heat map, and the information is combined to form the feature H of the heat map;

[0111] The characteristics of the extracted heat map are shown in Table 1:

[0112] Table 1 Heat map features

[0113]

[0114]

[0115] Step S6 specifically includes the following steps:

[0116] S6.1: Use feature selection to eliminate the class descriptor C obtained in S4 m The irrelevant or redundant features in the class feature subset D are obtained m , thereby reducing the number of features, improving model accuracy, and reducing running time;

[0117] S6.2: Subset D of class features after feature selection m Connect with the feature H of the heat map extracted by S5 to form the final feature G of the whole slice image m .

[0118] The feature selection refers to selecting features based on the feature differences between positive samples and negative samples. Let pos and neg represent the cancerous images and normal images in the training set respectively, N pos and N neg Represents the number of cancerous images and normal images, v i,k represents the k-th dimension feature of the i-th image, then the difference diff of the k-th feature k for:

[0119]

[0120] Then sort all feature differences from largest to smallest, and select the top 100 features with the largest feature differences to obtain the class feature subset D m .

[0121] like Figure 2 As shown, step S7 specifically includes the following steps:

[0122] S7.1: Extract the features of the three full-slice images of the training set (G1, G2, G3), input the three feature sets into the random forest respectively, train the random forest, and obtain the decision makers (RF1, RF2, RF3);

[0123] S7.2: Input the feature G1 of the whole slice image to the random forest decision maker RF1 to determine the type of the whole slice, that is, the two types of cancer and non-cancer. m If the type of the whole slice image is determined to be cancerous, then the step ends and the final determined type of the whole slice is cancerous; otherwise, step S7.3 is executed;

[0124] S7.3: Input feature G2 into random forest decision maker RF2 to determine the type of the whole slice. If RF2 determines that the type of the whole slice image is cancerous, then end the step and output the final determined type of the whole slice as cancerous; otherwise, execute step S7.4;

[0125] S7.4: Input feature G3 into the random forest decision maker RF3 to determine the type of the whole slice. If RF3 determines that the type of the whole slice image is cancer, then the final determination type of the whole slice is output as cancer. If the random forest decision maker RF3 determines that the whole slice image is cancer-free, then the final determination type of the whole slice is output as cancer-free.

[0126] Those skilled in the art will appreciate that the drawings are only provided to facilitate the understanding of the readers, and the modules or processes in the drawings are not necessarily required for implementing the present invention.

[0127] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

[0128] References

[0129] [1] Jin Xu, Wen Ke, Lü Guofeng, et al. A review of deep learning applications in histopathology[J]. Journal of Image and Graphics, 2020, 25(10): 1982-1993.

[0130] [2]Wang D,Khosla A,Gargeya R,et al.Deep Learning for IdentifyingMetastatic Breast Cancer[J].2016.

[0131] [3]Hou L,Samaras D,Kurc T M,et al.Patch-Based Convolutional NeuralNetwork for Whole Slide Tissue Image Classification[J].2016 IEEE Conferenceon Computer Vision and Pattern Recognition(CVPR),2016,2424-2433。

[0132] [4]Kong B,Xin W,Li Z,et al.Cancer Metastasis Detection via SpatiallyStructured Deep Network[C].International Conference on Information Processingin Medical Imaging.Springer,Cham,2017。

[0133] [5]Li Y,Ping W.Cancer Metastasis Detection With Neural ConditionalRandom Field[J].2018。

[0134] [6]Yan X,Jia Z,Wang L B,et al.Large scale tissue histopathology imageclassification,segmentation,and visualization via deep convolutionalactivation features[J].BMC Bioinformatics,2017,18(1):1-17。

[0135] [7]Wang X,Chen H,Gan C,et al.Weakly Supervised Deep Learning forWhole Slide Lung Cancer Image Analysis[J].IEEE Transactions on Cybernetics,2019,99:1-13。

Claims

1. A breast pathology full-slice classification method based on multi-feature fusion progressive discrimination, characterized in that: The following steps are involved: S1: Use the Ostu method to remove blank areas in the whole slice, extract the tissue area in the pathological image, and use random points in the tissue area to create training sets and test sets for CNN model training; S2: Select an improved ResNet34 network as the image block recognition model, and use the training set trained by the CNN model to train the network parameters using the stochastic gradient descent algorithm; use an improved ResNet34 network as the image block prediction model, which is based on ResNet34, replaces the last fully connected layer with two fully convolutional layers, and deletes the padding operations of all convolutional layers in ResNet34 to avoid boundary effects during prediction; then use the training set to train ResNet34; Step S2 specifically includes the following steps: S2.1: Use data augmentation methods including flipping, rotation, and cropping to increase the diversity of the training set; S2.2: Set the learning rate and weight in the model; S2.3: Process the input data through the improved ResNet34, calculate the loss value, and back-propagate the gradient to the parameters of the neural network; S2.4: Update the weight values ​​in the network according to the update rules; S3: Cut the whole slice image into blocks in a sliding window manner and input them into the trained image block recognition model, extract the feature vector and cancer probability of the image block from the recognition model, perform feature aggregation on each block to form a block descriptor, and save the coordinates and cancer probability of each block to obtain a tumor area heat map; specifically, the following steps are included: S3.1: Create a dataset for whole-slice classification. Use a sliding window method to extract image blocks of 896×896 pixels in the tissue area of ​​the pathological image, and use Reinhard to normalize the color of the image blocks. The image blocks of 896×896 pixels are called Blocks, and Blocks can be regarded as a large grid composed of n×n image blocks. S3.2: Directly input the Block into the trained convolutional neural network to predict the cancerous nature of the image block; during the prediction process, for each image block, a 512-dimensional feature f is output from the second-to-last convolutional layer of the improved ResNet34, and its cancer probability p is output in the last convolutional layer; then the image block features in the Block are aggregated by weighting, taking the maximum value, and taking the average value to obtain the Block feature descriptor B m , where m∈[1,M], M is the number of feature fusion methods; S3.3: Obtain the probability value of the Block: For each Block, take the average value of the cancer probability of each image block contained in it as the cancer probability of the entire Block; S3.4: Based on the cancer probability values ​​of all blocks, a heat map of the tumor area in the whole slice image is drawn, and the location of the malignant tumor is highlighted; 1) Weighted: Consider the features of all image blocks in the same block, and assume that the contribution of each image block feature to the block feature is proportional to its cancer probability. The calculation formula is as follows: 2) Take the maximum value: The feature of the image block with the highest probability is taken as the feature of the block. The calculation formula is as follows: 3) Take the average value: Take the average of the features of all image blocks in the Block as the feature of the Block. The calculation formula is as follows: Where B represents the Block descriptor, n is the number of image blocks in each row (column) of the Block, and p i,j , f i,j are the probability and characteristics of cancer in the image block in the i-th row and j-th column of the Block; S4: The block descriptors of the pathological image are aggregated by averaging, maximizing, and weighting to obtain three types of class descriptors, and all class descriptors are aggregated to obtain the final global descriptor; S5: 31 geometric and morphological features of the malignant tumor area are extracted from the heat map and combined to form the features of the heat map; S6: Select representative features from the global descriptor and connect them with the features of the heat map to obtain the final feature descriptors of the three full-slice images; S7: Use multi-feature progressive decision making to make a final judgment on the three obtained full-slice feature descriptors.

2. The breast pathology full-slice classification method based on multi-feature fusion progressive discrimination as claimed in claim 1, characterized in that: Step S1 specifically includes the following steps: S1.1: Manually mark cancerous areas in breast pathology images; S1.2: Use the Ostu segmentation method to remove blank areas in breast pathology images, extract tissue areas, and improve calculation efficiency; S1.3: In a random point-selecting manner, image blocks of 244×244 pixels were extracted from the tumor area of ​​the breast pathology full sections marked as cancerous as the positive data set, and image blocks of 244×244 pixels were extracted from the normal tissue areas of the breast pathology full sections marked as cancerous and the breast pathology full sections marked as non-cancerous as the negative data set; S1.4: Use the Reinhard color transfer method to normalize the colors of all image patches, and further divide the dataset into training and test sets.

3. The breast pathology full-slice classification method based on multi-feature fusion progressive discrimination as claimed in claim 1, characterized in that: Step S4 specifically includes the following steps: S4.1: Determine whether a block is discriminative by judging whether the probability of the block exceeds a certain threshold α; S4.2: In the full slice image, the features B of all discriminative blocks are m Aggregate to form class feature C m ; The class features are: Among them B k,m is B in step S3 m The obtained block feature descriptor, N represents the number of discriminant blocks.

4. The breast pathology full-slice classification method based on multi-feature fusion progressive discrimination as claimed in claim 1, characterized in that: Step S5 specifically includes the following steps: After obtaining the malignant tumor heat map of the whole slice tissue pathology image, the geometric and morphological feature information of the malignant tumor area is extracted from the heat map, and the information is combined to form the feature H of the heat map; The characteristics of the extracted heat map are shown in Table 1: Table 1 Heat map features 5. The breast pathology full-slice classification method based on multi-feature fusion progressive discrimination as claimed in claim 1, characterized in that: Step S6 specifically includes the following steps: S6.1: Use feature selection to eliminate the class descriptor C obtained in S4 m The irrelevant or redundant features in the class feature subset D are obtained m , thereby reducing the number of features, improving model accuracy, and reducing running time; S6.2: Subset D of class features after feature selection m Connect with the feature H of the heat map extracted by S5 to form the final feature G of the whole slice image m ; The feature selection refers to selecting features based on the feature differences between positive samples and negative samples; let pos and neg represent the number of cancerous images and normal images in the training set, respectively, and N pos and N neg Represents the number of cancerous images and normal images, v i,k represents the k-th dimension feature of the i-th image, then the difference diff of the k-th feature k for: Then sort all feature differences from largest to smallest, and select the top 100 features with the largest feature differences to obtain the class feature subset D m .

6. The breast pathology full-slice classification method based on multi-feature fusion progressive discrimination as claimed in claim 1, characterized in that: Step S7 specifically includes the following steps: S7.1: Extract the features of the three full-slice images of the training set (G1, G2, G3), input the three feature sets into the random forest respectively, train the random forest, and obtain the decision makers (RF1, RF2, RF3); S7.2: Input the feature G1 of the whole slice image to the random forest decision maker RF1 to determine the type of the whole slice, that is, the two types of cancer and non-cancer; if RF m If the type of the whole slice image is determined to be cancerous, then the step ends and the final determined type of the whole slice is cancerous; otherwise, step S7.3 is executed; S7.3: Input feature G2 into random forest decision maker RF2 to determine the type of the whole slice. If RF2 determines that the type of the whole slice image is cancerous, then end the step and output the final determined type of the whole slice as cancerous; otherwise, execute step S7.4; S7.4: Input feature G3 into the random forest decision maker RF3 to determine the type of the whole slice. If RF3 determines that the type of the whole slice image is cancer, then the final determination type of the whole slice is output as cancer. If the random forest decision maker RF3 determines that the whole slice image is cancer-free, then the final determination type of the whole slice is output as cancer-free.

7. The breast pathology full-slice classification method based on multi-feature fusion progressive discrimination as claimed in claim 4, characterized in that: n＝4。 8. The breast pathology full-slice classification method based on multi-feature fusion progressive discrimination as claimed in claim 5, characterized in that: Threshold α=0.3.

Images