A breast cancer whole-slice pathology image classification method based on second-order pooling hybrid supervised learning

Abstract

The present invention discloses a method for classifying breast cancer full-slice pathology images based on second-order pooling hybrid supervised learning, comprising: dividing the breast cancer full-slice pathology images into a training set and a test set, and dividing the images into example-level images; constructing a network structure based on second-order pooling hybrid supervised learning, the network structure including an example classifier, using images in the training set to perform self-supervisory training on the network structure, thereby performing feature encoding on the example feature encoder; using the test set images to test the trained network structure, and classifying the breast cancer full-slice pathology images. The method combines the self-supervision method with the weak supervision method to construct an example classifier. Second-order pooling is introduced in the example-level classifier to increase the focus on example-level features, capture the structure and pattern of the data, and improve the accuracy of the final breast cancer full-slice image classification through more sophisticated example-level analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a breast cancer full-slice pathological image classification method based on second-order pooling hybrid supervised learning. Background Art

[0002] Pathological diagnosis is a crucial part of the clinical diagnosis and treatment of breast cancer. Accurate pathological diagnosis is based on pathological sampling, pathological sections and pathological image analysis. The analysis of pathological sections and pathological images can provide accurate disease diagnosis information, including the specific type of tumor, tumor grading and staging, and molecular pathological information of the tumor, etc., providing a basis for the formulation of subsequent clinical diagnosis and treatment plans for patients and predicting the clinical prognosis of patients. However, the current pathological diagnosis time is long, the report is issued late, and in some cases, due to the subjective cognitive differences of different pathologists on pathological images, the consistency of pathological diagnosis under certain circumstances is poor. Therefore, the application of computer-aided diagnosis technology in pathological images has become particularly important.

[0003] In the analysis of whole-slice pathology images based on deep learning, researchers face two major challenges. The first is that whole-slice pathology images have a huge resolution, usually up to 100,000×100,000 pixels, which makes processing these images very complex and computationally intensive; second, whole-slice pathology images have limited label information and lack detailed information about the affected area at different magnifications. Currently, most weakly supervised works for whole-slice pathology image classification use the "multi-instance learning" method. In the multi-instance learning method, the entire whole-slice pathology image is regarded as a bag, which contains multiple examples, which are composed of local pathology images obtained by cutting from the whole-slice pathology image. If at least one example in the bag is positive for the disease, the entire bag is marked as positive; only when all examples in the bag are negative, it is judged as negative. Therefore, the classification results of pathology images are inaccurate. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention discloses a method for classifying breast cancer full-slice pathological images based on second-order pooling hybrid supervised learning, which specifically includes the following steps:

[0005] The breast cancer full-slice pathology images are divided into a training set and a test set, and Gaussian blur processing and anti-binary threshold processing are performed on the images in the training set and the test set to read the key areas, and the images are divided into example-level images;

[0006] Constructing a network structure based on second-order pooling hybrid supervised learning, the network structure includes an example classifier, and the example classifier is constructed based on an example feature encoder and a second-order visual perception module;

[0007] The network structure is trained by self-supervising using images in the training set, so as to perform feature encoding on the example feature encoder; the example feature encoder and the randomly initialized second-order visual perception module that have completed the self-supervised training are trained by a weakly supervised learning process, and the model weight values ​​are saved; in the learning process, example-level probability prediction is performed, and packet-level pseudo labels are assigned to examples with high probability to update the network parameters, and example-level probability prediction and packet-level pseudo label assignment are performed alternately;

[0008] The test set images were used to test the trained network structure and classify breast cancer whole-slice pathology images.

[0009] Furthermore, when constructing the example classifier: the self-supervised method is combined with the multi-instance learning method, and the example-level image is feature-encoded using the example feature encoder that has been pre-trained through self-supervision, wherein the example feature encoder is trained using the pathological image's own features. During the training process, the cross entropy loss function is optimized by maximizing the consistency between sub-regions of the same image, wherein the loss function is as follows:

[0010]

[0011] Where: P s and P t Represent the outputs of the student network and the teacher network in the self-supervised method, respectively. E represents -alogb. For a given image x, a set of different views M is generated, including two global views. and and multiple low-resolution local views x′.

[0012] Furthermore, the example feature encoder encodes each example-level image into N d-dimensional visual tokens and classification tokens, and connects all d-dimensional visual tokens and classification tokens to form a matrix, then {z ij ∈R d , j=0,1,...,N} represents all tokens. In the visual perception module, the classification tokens are divided into first-order moments. The multi-head convolution cross covariance is used to calculate the visual tokens to obtain the second-order moments. The first-order moments and second-order moments are predicted by examples through the fully connected layer, and the two are combined as the final example-level prediction results. The calculation process is as follows:

[0013] y pred =FC(z i0 )+FC(MHC 3 (z ij )),j∈[1,2,…,N]

[0014] Where: z i0 is the classification token, z ijFor visual tokens, MHC 3 is the cross covariance calculation of multi-head convolution, FC is the fully connected layer, y pred is the final prediction result.

[0015] Furthermore, when the example feature encoder that has completed self-supervised training and the randomly initialized second-order visual perception module are trained using a weakly supervised learning process: the example feature encoder is trained using a self-supervised method, features are extracted for the examples divided by each full-slice pathology image, and the predicted probability of the example is calculated using a randomly initialized second-order visual perception module. According to a set prediction probability threshold, the predicted probability is divided and labels are assigned to the examples according to the division results. Top-K examples are used and corresponding pseudo-labels at the full-slice pathology image level are assigned, thereby iteratively updating the parameters of the example classifier, wherein the predicted example probability assignment of pseudo-labels and parameter optimization are performed alternately.

[0016] Furthermore, in the second-order visual perception module, the example-level probability prediction is performed based on the first-order moment and the second-order moment: in the multi-head convolution covariance, the visual token is reduced by 1×1 convolution. Indicates that the feature is transformed along the feature dimension It is divided into h heads, SP represents the split operation, is an element Normalization to control K i The size of the multi-head cross covariance between two adjacent split features is calculated to calculate the second-order statistics, where ε is the GELU activation function and Concat is the concatenation operation. It is a 3×3 convolution operation with a step of 2, and finally a local representation aggregation LRA is performed. The calculation process is as follows:

[0017]

[0018] Due to the adoption of the above technical solution, the present invention provides a breast cancer full-slice pathology image classification method based on second-order pooling hybrid supervised learning. This method can effectively perform benign and malignant diagnosis of breast cancer full-slice pathology images by using the first-order moment and second-order moment of the feature through a hybrid supervision method. The algorithm proposed by the present invention is simple and efficient. In the face of the problems of super-resolution of full-slice pathology images and lack of fine-grained labels, it can still achieve efficient classification effects and can perform clinical auxiliary diagnosis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 Flow chart of the method of the present invention

[0021] Figure 2 A network structure diagram of the method disclosed in the present invention

[0022] Figure 3 The second-order visual module structure diagram of the method disclosed in the present invention DETAILED DESCRIPTION

[0023] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] like Figure 1 The method for classifying breast cancer whole-slice pathology images based on second-order pooling hybrid supervised learning is shown, and specifically includes the following steps:

[0027] S1: Process the breast cancer full-slice pathology images as the network input dataset in the following way:

[0028] S11: Divide the breast cancer whole-slice pathology images into training set and test set in a 2:1 ratio.

[0029] S12: Perform Gaussian blur on the image to smooth the image and remove small noise and irrelevant details. The formula is:

[0030]

[0031] S13: Through inverse binary threshold processing, the image is converted into a binary image to highlight the tissue area and filter out the important parts.

[0032] S14: The regions occupying more than 40% of the image and the related tissue information are retained. Multiple non-overlapping examples of size 512×512 are generated from these retained regions for subsequent training and analysis.

[0033] S2 constructs a network structure based on second-order pooling hybrid supervised learning, specifically in the following way:

[0034] S21: Construct an example classifier, which consists of two parts: an example feature encoder and a second-order visual perception module.

[0035] S22: In the example feature encoder part, the self-supervised pre-trained example feature encoder is used for feature encoding. The self-supervised method adopts a self-distillation method. The network as a whole consists of two branches: the teacher network and the student network. Both use the network structure of Vision Transformer-s, with network symbols and parameters respectively. and Given a fixed teacher network Then the student's network parameter θ s for:

[0036]

[0037] Where: H is -alogb, P s and P t represents the output of the student network and the teacher.

[0038] S23: The example encoder part is trained by self-supervision method using the characteristics of the example-level image. During the calculation process, the cross entropy loss is optimized by maximizing the consistency between sub-regions of the same image. The specific loss function is calculated as follows:

[0039]

[0040] Where: P s and P tRepresent the output of the student network and the teacher network in the self-supervised method, and E represents -alogb. For a given image x, a set of different views M is generated, including two global views and and several low-resolution local views x′.

[0041] S3: Construct a visual perception module to predict the benign and malignant nature of breast cancer, using the following method:

[0042] S31: In the example feature encoder, each example is encoded into N d-dimensional visual tokens and one classification token, and all tokens are concatenated to form a matrix, then {z ij ∈R d , j = 0, 1, ..., N} represents all tokens. The classification tokens are segmented into first-order moments in the visual perception module, while the visual tokens are segmented into first-order moments using multi-head convolution cross covariance (MHC 3 ) to effectively calculate the second-order moment. Through the fully connected layer, the first-order moment and the second-order moment are predicted separately, and the two are combined as the final example-level prediction result. The calculation process is as follows:

[0043] y pred =FC(z i0 )+FC(MHC 3 (z ij )),j∈[1,2,…,N]

[0044] Where: z i0 is the classification token, z ij For visual tokens, MHC 3 is the cross covariance calculation of multi-head convolution, FC is the fully connected layer, y pred is the final prediction result.

[0045] S32: During the training process, the features of the examples in the bag are first extracted using the feature extractor trained by the frozen self-supervised method. Next, the predicted probability of the example is calculated using the frozen randomly initialized second-order visual perception module.

[0046] S33: According to the set prediction probability threshold, the prediction probability is divided, and the full-slice pathology image level label is assigned to the example according to the division result, where the threshold is 0.5.

[0047] S34: During the training of the example classifier, the Top-K examples are used and the corresponding full-slice pathology image-level pseudo labels are assigned, so as to iteratively update the parameters of the example classifier, wherein the predicted example probability assignment of pseudo labels and parameter optimization are performed alternately.

[0048] S4: In the multi-head convolution covariance, for the visual Awareness TokenFirst, the dimensionality is reduced by 1×1 convolution. Then, along the feature dimension, the feature It is divided into h heads. SP represents the split operation. is an element Normalization to control K i The size of the multi-head cross covariance between two adjacent split features is calculated to calculate the second-order statistics, where ε is the GELU activation function and Concat is the concatenation operation. It is a 3×3 convolution operation with a step of 2, and finally a local representation aggregation LRA is performed. The calculation process is as follows:

[0049]

[0050] S5: Use the image data in the training set to iteratively train the network to update the network parameters, specifically in the following way:

[0051] S51: Set network hyperparameters, where the AdamW optimizer is used in the process of self-supervised training of the example encoder, and 35 rounds of training are trained on the examples of the Camelyon-16 dataset. The training strategy uses cosine annealing to dynamically adjust the learning rate and adopts automatic mixed precision. Finally, the training weights of the last round are saved. In the training phase of the example classifier, the Adam optimizer is used to perform 35 rounds of training on the examples divided by the Camelyon-16 dataset, and the initial learning rate is set to 1×e -4 .

[0052] S6 updates the network parameters in iterations, performs training and verification, and saves the optimal weights. The specific method is as follows:

[0053] S61: In each iteration, the network parameters are updated and training and validation are performed. The classification results of the whole slice image are generated by aggregating the prediction results of the examples. The weight of the classification effect of the whole slice pathology image of breast cancer is saved according to the best validation set accuracy.

[0054] The present invention provides a method for classifying breast cancer full-slice pathological images based on second-order pooling hybrid supervised learning. The method combines the self-supervised method with the weakly supervised method to construct an example classifier. Second-order pooling is introduced in the example-level classifier, and the information of the visual token is used as a supplement to the classification token, thereby increasing the focus on example-level features, capturing the structure and pattern of the data, and improving the accuracy of the final breast cancer full-slice image classification through more refined example-level analysis.

[0055] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A breast cancer full-slice pathology image classification method based on second-order pooling hybrid supervised learning, characterized in that include: The breast cancer full-slice pathology images are divided into a training set and a test set, and Gaussian blur processing and anti-binary threshold processing are performed on the images in the training set and the test set to read the key areas, and the images are divided into example-level images; Constructing a network structure based on second-order pooling hybrid supervised learning, the network structure comprising an example classifier, wherein the example classifier is constructed based on an example feature encoder and a second-order visual perception module; The network structure is trained by self-supervision using images in the training set, thereby encoding features of the example feature encoder; The weakly supervised learning process is used to train the self-supervised trained example feature encoder and the randomly initialized second-order visual perception module, and the model weight values ​​are saved; In the learning process, the example-level probability prediction is performed, and the packet-level pseudo-labels are assigned to the examples with high probability to update the network parameters, and the example-level probability prediction and the assignment of packet-level pseudo-labels are performed alternately; The trained network structure was tested using test set images, and breast cancer full-slice pathology images were classified; The example feature encoder encodes each example-level image into N d-dimensional visual tokens and classification tokens, and connects all d-dimensional visual tokens and classification tokens to form a matrix, then {z ij ∈R d , j=0,1,...,N} represents all tokens. In the visual perception module, the classification tokens are divided into first-order moments. The multi-head convolution cross covariance is used to calculate the visual tokens to obtain the second-order moments. The first-order moments and second-order moments are predicted by examples through the fully connected layer, and the two are combined as the final example-level prediction results. The calculation process is as follows: y pred =FC(from i0 )+FC(MHC 3 (from ij )),j∈[1,2,…,N] Where: z i0 is the classification token, z ij For visual tokens, MHC 3 is the cross covariance calculation of multi-head convolution, FC is the fully connected layer, y pred is the final prediction result.

2. According to claim 1, a method for classifying breast cancer whole-slice pathological images based on second-order pooling hybrid supervised learning is characterized by: When constructing the example classifier: the self-supervised method is combined with the multi-instance learning method, and the example-level image is feature-encoded using the example feature encoder that has been pre-trained by self-supervision. The example feature encoder is trained using the pathological image's own features. During the training process, the cross entropy loss function is optimized by maximizing the consistency between sub-regions of the same image. The loss function is as follows: Where: P s and P t They represent the outputs of the student network and the teacher network in the self-supervised method, respectively. E represents -a log b. For a given image x, a set of different views M is generated, including two global views. and and multiple low-resolution local views x′.

3. The method for classifying breast cancer whole-slice pathological images based on second-order pooling hybrid supervised learning according to claim 1, characterized in that: When the weakly supervised learning process is used to train the example feature encoder that has completed self-supervised training and the randomly initialized second-order visual perception module: the example feature encoder is trained using the self-supervised method, and features are extracted for the examples divided by each full-slice pathology image. The predicted probability of the example is calculated using the randomly initialized second-order visual perception module, and the predicted probability is divided according to the set prediction probability threshold, and labels are assigned to the examples according to the division results. The Top-K examples are used and the corresponding pseudo-labels at the full-slice pathology image level are assigned, so as to iteratively update the parameters of the example classifier, wherein the predicted example probability assignment pseudo-labels and parameter optimization are performed alternately.

4. The method for classifying breast cancer whole-slice pathological images based on second-order pooling hybrid supervised learning according to claim 3 is characterized in that: In the second-order visual perception module, the example-level probability prediction is performed based on the first-order moment and the second-order moment: in the use of multi-head convolution covariance, the visual token is reduced by 1×1 convolution. Indicates that the feature is transformed along the feature dimension It is divided into h heads, SP represents the split operation, and l2 is the l2 normalization of an element to control K i The size of the multi-head cross covariance between two adjacent split features is calculated to calculate the second-order statistics, where ε is the GELU activation function and Concat is the concatenation operation. It is a 3×3 convolution operation with a step of 2, and finally a local representation aggregation LRA is performed. The calculation process is as follows: K={K1;K2;…;K h-1 }

Images