Breast tissue pathology image classification method based on two-stage feature fusion network

Through two-stage feature fusion network and self-supervised contrast learning, the problems of overfitting and spatial correlation neglect in breast histopathological image classification are solved, the accuracy and reliability of image classification are improved, and more accurate auxiliary diagnostic suggestions are provided .

CN118968177APending Publication Date: 2024-11-15ZHEJIANG UNIV OF TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411129021.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing breast histopathological image classification method has problems of overfitting and unbalanced distribution of positive image blocks when processing full-field digital slices, resulting in low accuracy in model classification, and traditional methods ignore spatial correlations between images. This results in poor feature extraction effect.

Method used

Using a method based on a two-stage feature fusion network, combining self-supervised contrast learning and multi-scale feature fusion technology, the image is divided into image blocks at different magnifications through Otsu threshold algorithm and chunking technology, and the features are extracted using self-supervised contrast learning algorithm. , and enhance the generalization and feature representation capabilities of the model through attention mechanism and feature fusion module.

Benefits of technology

It improves the accuracy and reliability of breast histopathological image classification, enhances the model's ability to capture key pathological features, reduces the risk of misdiagnosis, and provides more accurate auxiliary diagnostic suggestions, shortens diagnosis time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968177A_ABST
    Figure CN118968177A_ABST
Patent Text Reader

Abstract

A mammary tissue pathology image classification method based on a two-stage feature fusion network comprises the following steps: generating an attention score for each image block by using an attention mechanism, then selecting front beta% significant instances with higher scores to form a new package, randomly dividing the new package into n groups, and taking each group as an independent sub-package; a feature selection strategy is adopted, the features of each sub-packet are selected and fused into packet level features, and finally classification of pathological images is achieved through a packet classifier; the two-stage feature fusion network uses self-supervised contrast learning to train multi-scale features so as to generate powerful image block characterization, and in order to obtain the features, the network adopts a multi-scale feature fusion technology, and image information under different scales can be effectively integrated so as to provide more comprehensive and finer organization feature description. According to the invention, the robustness and discrimination capability of the model are improved; and the challenge of low accuracy in the classification task of the mammary tissue pathology image is effectively handled.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of medical imaging diagnosis technology and the field of electronic information graphics and image processing technology, and relates to a breast tissue pathology image classification method. Background Art

[0002] Breast cancer is the leading cancer threat to women's health, accounting for about 31% of female cancers. It is also the leading cause of death in women aged 40-79 and men aged 60-79. Accurate diagnosis of breast cancer plays a key role in subsequent treatment and prognosis. Histopathological image analysis plays a vital role in modern medicine and is generally considered the gold standard for breast cancer diagnosis.

[0003] Traditional breast histopathology diagnosis is performed manually by examining pathological tissue sections using an optical microscope. However, these diagnostic methods are facing major challenges due to social issues such as environmental changes and the increasing aging of the population. With the development of whole-slide digital scanning technology, pathological sections can be saved in the form of whole-view digital sections (WSIs), which greatly improves the work efficiency of pathologists. However, the huge size of pathological sections (100,000×100,000 pixels) poses certain challenges to manual reading. Furthermore, differences in the professional level of different pathologists may lead to subjective bias in the analysis of the same section, increasing the risk of misdiagnosis. To address these problems, researchers have developed a computer-aided diagnosis system using computer vision and machine learning technology, opening up a new path for pathological diagnostic analysis. This approach can reduce the workload of pathologists and improve the efficiency and accuracy of pathological diagnosis.

[0004] With the rise of full-view digital slide technology, computer-aided analysis has been used in pathological image classification research. Early breast tissue pathology image classification methods mostly used traditional machine learning algorithms. This type of algorithm relies on manually designed features for classification, which limits their application scope to a certain extent. Due to the high similarity between pathological cells, feature extraction becomes extremely complex and it is difficult to design high-level abstract features. In recent years, classification methods based on deep learning have overcome the limitations of traditional methods that require manual design and extraction of features through their ability to autonomously learn features in the network, and have achieved remarkable results in computer-aided diagnosis research of full-view digital slides. However, these deep learning methods usually require a large amount of annotation information to obtain satisfactory results. At the same time, due to limited computer memory, it is difficult to directly process WSI. Therefore, researchers have developed many deep learning methods based on image patches. They divide each WSI into hundreds or even thousands of image patches, and then annotate the image patches and send them to model training. Obviously, these methods still require a large amount of precise annotation information. In the absence of pixel-level annotation, researchers at home and abroad have begun to use weakly supervised learning methods based on multiple instance learning (MIL) for WSI classification.

[0005] In recent years, with the development of the field of whole-view digital slide (WSI) analysis, weakly supervised learning frameworks, especially multiple instance learning (MIL) models, have become a hot topic in academic research. MIL models use coarse-grained slice-level labels for training, avoiding the need for fine-grained annotation of each instance. In MIL, each WSI is defined as a bag, and the image blocks divided from the WSI are regarded as instances in the bag. For a bag, if there is at least one positive instance in it, the bag is called a positive bag. On the contrary, if all the instances in it are negative, the bag is called a negative bag. MIL methods can be mainly divided into two categories: instance-based methods and embedding-based methods. The former trains an instance classifier to obtain the prediction of each instance, and then aggregates the predictions of each instance in the bag to obtain the bag prediction. The latter extracts the features of each instance in the bag through an encoder, aggregates the instance features into bag-level features through an aggregation operator, and then trains the bag classifier in a supervised manner. Studies have shown that instance-based methods are inferior to embedding-based methods in classification performance.

[0006] However, there are two major challenges in the current embedding-based MIL model. First, the number of slices used for training is limited, and the only learning information available is the slice-level labels, which makes the model vulnerable to overfitting and unable to learn rich feature representations. In addition, the distribution of image patches (instances) in positive WSI (packages) is extremely unbalanced, and the proportion of positive image patches is very small. Direct processing of such slices may cause misclassification. To address this challenge, existing studies have explored methods to divide full-slice images into multiple sub-packages to virtually increase the number of packages. For example, Yao et al. used the K-means clustering algorithm to cluster all image patches of the same patient into different phenotypes, and then regarded each phenotype as a sub-package. However, the clustering algorithm is based on coordinate calculations, which introduces additional coordinate information for the sub-packages. In addition, the clustering-based grouping method determines similar image patches by calculating the distance between embeddings, which incurs a lot of computational overhead. To avoid complex calculations, Zhang et al. adopted a random grouping strategy. This strategy randomly divides a parent package into several sub-packages. Similarly, Zhang et al. used the MAS grouping strategy to divide the embedding into a fixed number of sub-packages through a random grouping method and MSA (multi-head self-attention). However, due to their randomness, these methods may result in the failure to obtain at least one positive instance in the positive sub-package, resulting in a decrease in the model classification accuracy. Secondly, most current self-supervised methods only use image patches of a single scale to train the feature extraction network, ignoring the spatial correlation between full-slice images. Generally, in clinical practice, pathologists also use pathological images of different magnifications to comprehensively examine the lesion area. In order to model the spatial correlation between images of different magnifications, Shen et al. proposed a two-layer backbone network by combining convolutional neural networks (CNNs) and Transformers to fuse the spatial correlation of images of different magnifications. In addition, they also improved the self-supervised learning algorithm for training these features. This method fuses features from CNN and Transformer through a channel cascade operation. However, features from different networks may differ in spatial resolution and depth, which may cause feature alignment problems. In addition, the two-layer backbone network consumes a lot of computing resources and time. Summary of the invention

[0007] In order to overcome the shortcomings of the existing technology, the present invention provides a breast tissue pathology image classification method based on a two-stage feature fusion network, adopts a self-supervised contrastive learning framework to train the feature extraction network, and combines multi-scale feature fusion technology to integrate breast tissue pathology image information at different scales. The purpose of this is to make full use of the sensitivity of convolutional neural networks to local and global information in images, so as to extract information with important diagnostic significance in pathological images; in addition, a two-stage classification task based on attention score and feature fusion mechanism is designed and implemented. This design not only increases the number of packages that the model can process, but also further improves the effect of model training through the advantages of significant features.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A breast tissue pathology image classification method based on a two-stage feature fusion network comprises the following steps:

[0010] Step 1: Use the Otsu threshold algorithm to perform threshold segmentation on the breast tissue pathology image. Automatically select the initialization threshold based on the image to divide the image into foreground and background. Since only the tissue target is screened out and the background is excluded at this stage, this method can quickly find the threshold of the target;

[0011] Step 2: The breast tissue pathology images were processed at 20x and 10x magnifications respectively using a block technology to divide them into a series of image blocks;

[0012] Step 3: The image blocks obtained in step 2 are respectively smoothed using median filtering technology to reduce the noise and image blur that may be introduced in the process of pathological image preparation;

[0013] Step 4: Design a feature fusion module that combines the features of breast tissue pathology image blocks at the same location but different magnifications;

[0014] Step 5: Apply the self-supervised contrastive learning algorithm and fuse the multi-scale feature vectors through step 4 for training, aiming to extract more discriminative feature representations and enhance the generalization ability of the feature extractor. After the training reaches convergence, retain the best model weights for subsequent analysis tasks;

[0015] Step 6: Use the feature extractor pre-trained in step 5 to extract features of image patches at different magnifications, and then fuse these features through the feature fusion module in step 4;

[0016] Step 7: Apply the attention mechanism to the fused features, calculate the attention score of each instance, and select the top β% significant instances with the highest scores to form a new package. Then randomly divide these significant instances into n groups, each group as an independent sub-package;

[0017] Step 8: Select the instance features of each sub-package and concatenate them to obtain the final package-level features;

[0018] Step 9: Input the packet-level features obtained in step 8 into the packet classifier for discrimination, and obtain the final binary classification result of whether the breast tissue pathology image is cancerous or not.

[0019] Furthermore, in step 2, the pathological image is divided into pixel sizes suitable for mainstream convolutional neural network processing, and preparation is made for subsequent feature fusion at different magnifications. The process is as follows:

[0020] Step 2.1: Use the OpenSlide open source library to read breast tissue pathology image I;

[0021] Step 2.2: First, acquire images at 20x and 10x magnifications, then select a window of fixed size and continuously slide along the horizontal and vertical directions of image I with a fixed step size s smaller than the window size to acquire image blocks;

[0022] Step 2.3: Save the image block for subsequent processing.

[0023] Furthermore, in step 3, the noise signal in the image block is removed to improve the image quality, and the process is as follows:

[0024] Step 3.1: Use the OpenCV vision library to load the image patch and apply an observation window with a radius of r for local sampling;

[0025] Step 3.2: Sort the pixel values ​​in the current observation window to determine the median pixel value;

[0026] Step 3.3: The calculated pixel median value is used to replace the pixel at the center of the observation window to enhance the local contrast and clarity of the image;

[0027] Step 3.4: Repeat steps 3.2 to 3.3 to filter the entire image block.

[0028] Furthermore, in step 4, the spatial correlation between image blocks of different scales is considered, so that the model can capture detailed information of pathological images at different scales, thereby providing a more comprehensive description of tissue features. The process is as follows:

[0029] Step 4.1: For each image block with a magnification of 10 times, extract the feature vector of the image block, and simultaneously obtain the feature vector of the sub-image block at the same position at a magnification of 20 times;

[0030] Step 4.2: Concatenate the feature vector at 10 times the rate with each feature vector at 20 times the rate, and finally obtain 4 feature vectors.

[0031] In step 7, a package is divided into multiple sub-packages to expand the scale of training data and allow the model to pay more attention to those significant instances, thereby improving the model's sensitivity and recognition ability for pathological features. The process is as follows:

[0032] Step 7.1: Use the attention mechanism in AB-MIL to obtain the attention score of each instance;

[0033] Step 7.2: Sort all attention scores in descending order and select the top β% significant instances as new training samples.

[0034] Step 7.3: Randomly divide the obtained salient instances into n groups, and each group is regarded as an independent sub-bag.

[0035] In step 8, representative instance features are selected from the sub-packages to aggregate package-level features and used for subsequent package-level classification. The process is as follows:

[0036] Step 8.1: Apply the Softmax function to the last layer output of the multi-layer perceptron MLP to convert the original output into the cancer probability corresponding to each instance in the sub-packet;

[0037] Step 8.2: For each sub-package, first sort the cancer probability of each instance, then select the instance features with the highest and lowest probabilities, and concatenate these two feature vectors to obtain a new feature vector to represent this sub-package;

[0038] Step 8.3: The feature vectors representing each sub-packet processed in step 8.2 are concatenated again to obtain the final packet-level features.

[0039] In the steps 8.2 and 8.3, the splicing method is selected to be channel splicing or space splicing.

[0040] In step 2, the blocking technology uses random blocking, grid blocking or overlapping sampling blocking; the size of the divided image blocks is selected from 256×256, 512×512 or 768×768 pixels.

[0041] In step 3, the filtering method is selected from methods such as mean filtering or maximum filtering.

[0042] In step 4, the splicing is performed by channel splicing or space splicing.

[0043] In the step 4.1, a Resnet network or a VGG network is used to extract image block features.

[0044] In step 5, contrastive learning algorithms such as MoCov2 or SimCLR are selected as the self-supervised learning framework. These algorithms can effectively learn and extract highly robust feature representations without relying on manual annotation.

[0045] In step 9, a multi-layer perceptron or a convolutional neural network is selected as a packet classifier.

[0046] The present invention proposes a two-stage feature fusion network for benign and malignant discrimination of breast tissue pathology images. In the first stage, a package is randomly divided into multiple groups according to a grouping strategy, and each group forms an independent sub-package. In the second stage, a feature selection strategy is used to select the features of each sub-package and fuse them into package-level features, and then the classification of breast tissue pathology images is realized through a package classifier. In addition, in order to further improve the effect of the self-supervised learning algorithm in training feature extraction networks, the present invention designs a feature fusion module. The module is used to fuse features of breast tissue pathology images from different magnifications. Through this strategy, the self-supervised learning algorithm can more effectively capture and integrate multi-scale information of pathological images, enhancing the robustness and generalization ability of feature representation.

[0047] The beneficial effects of the present invention are mainly manifested in: the method uses a self-supervised contrastive learning algorithm to train multi-scale features, so that it can capture and learn highly discriminative features in breast tissue pathology images. Secondly, a two-stage feature fusion network is used to distinguish benign and malignant pathological images. It is worth noting that the present invention only focuses on those significant instances with higher attention scores, which usually contain key information in pathological images. In addition, allowing the model to pay more attention to significant instances can further enhance its ability to capture and understand key pathological features, thereby improving the diagnostic accuracy and reliability of the model. This innovative method of the present invention is expected to provide doctors with more accurate auxiliary diagnosis suggestions, while shortening the diagnosis time, thereby reducing the workload of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the multi-scale feature fusion method.

[0049] Figure 2 It is a two-stage feature fusion network flowchart. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings.

[0051] Reference Figure 1 and Figure 2 , a breast tissue pathology image classification method based on a two-stage feature fusion network, comprising the following steps:

[0052] Step 1: Use the Otsu threshold algorithm to perform threshold segmentation on the breast tissue pathology image. Automatically select the initialization threshold based on the image to divide the image into foreground and background. Since only the tissue target is screened out and the background is excluded at this stage, this method can quickly find the threshold of the target;

[0053] Step 2: The breast tissue pathology images were processed at 20x and 10x magnifications respectively using a block technology to divide them into a series of image blocks;

[0054] In step 2, the blocking technology uses random blocking, grid blocking or overlapping sampling blocking; the size of the divided image blocks is selected from 256×256, 512×512 or 768×768 pixels.

[0055] In step 2, the pathological image is divided into pixel sizes suitable for mainstream convolutional neural network processing, and preparation is made for subsequent feature fusion at different magnifications. The process is as follows:

[0056] Step 2.1: Use the OpenSlide open source library to read breast tissue pathology image I;

[0057] Step 2.2: First, acquire images at 20x and 10x magnifications, then select a window of fixed size and continuously slide along the horizontal and vertical directions of image I with a fixed step size s smaller than the window size to acquire image blocks;

[0058] Step 2.3: Save the image block for subsequent processing.

[0059] Step 3: The image blocks obtained in step 2 are respectively smoothed using median filtering technology to reduce the noise and image blur that may be introduced in the process of pathological image preparation;

[0060] In step 3, the filtering method is selected from methods such as mean filtering or maximum filtering.

[0061] In step 3, the noise signal in the image block is removed to improve the image quality. The process is as follows:

[0062] Step 3.1: Use the OpenCV vision library to load the image patch and apply an observation window with a radius of r for local sampling;

[0063] Step 3.2: Sort the pixel values ​​in the current observation window to determine the median pixel value;

[0064] Step 3.3: The calculated pixel median value is used to replace the pixel at the center of the observation window to enhance the local contrast and clarity of the image;

[0065] Step 3.4: Repeat steps 3.2 to 3.3 to filter the entire image block.

[0066] Step 4: Design a feature fusion module that combines the features of breast tissue pathology image blocks at the same location but different magnifications;

[0067] In step 4, the splicing is performed by channel splicing or space splicing.

[0068] In step 4, the spatial correlation between image blocks of different scales is considered, so that the model can capture detailed information of pathological images at different scales, thereby providing a more comprehensive description of tissue features. The process is as follows:

[0069] Step 4.1: For each image block with a 10x magnification, extract the feature vector of the image block, and simultaneously obtain the feature vector of the sub-image block at the same position at a 20x magnification; Resnet network or VGG network is used to extract the image block features;

[0070] Step 4.2: Concatenate the feature vector at 10 times the rate with each feature vector at 20 times the rate, and finally obtain 4 feature vectors.

[0071] Step 5: Apply the self-supervised contrastive learning algorithm and fuse the multi-scale feature vectors through step 4 for training, aiming to extract more discriminative feature representations and enhance the generalization ability of the feature extractor. After the training reaches convergence, retain the best model weights for subsequent analysis tasks;

[0072] In step 5, contrastive learning algorithms such as MoCov2 or SimCLR are selected as the self-supervised learning framework. These algorithms can effectively learn and extract highly robust feature representations without relying on manual annotation.

[0073] Step 6: Use the feature extractor pre-trained in step 5 to extract features of image patches at different magnifications, and then fuse these features through the feature fusion module in step 4;

[0074] Step 7: Apply the attention mechanism to the fused features, calculate the attention score of each instance, and select the top β% significant instances with the highest scores to form a new package. Then randomly divide these significant instances into n groups, each group as an independent sub-package;

[0075] In step 7, a package is divided into multiple sub-packages to expand the scale of training data and allow the model to pay more attention to those significant instances, thereby improving the model's sensitivity and recognition ability for pathological features. The process is as follows:

[0076] Step 7.1: Use the attention mechanism in AB-MIL to obtain the attention score of each instance;

[0077] Step 7.2: Sort all attention scores in descending order and select the top β% significant instances as new training samples.

[0078] Step 7.3: Randomly divide the obtained salient instances into n groups, and each group is regarded as an independent sub-bag.

[0079] Step 8: Select the instance features of each sub-package and concatenate them to obtain the final package-level features;

[0080] In step 8, representative instance features are selected from the sub-packages to aggregate package-level features and used for subsequent package-level classification. The process is as follows:

[0081] Step 8.1: Apply the Softmax function to the last layer output of the multi-layer perceptron MLP to convert the original output into the cancer probability corresponding to each instance in the sub-packet;

[0082] Step 8.2: For each sub-package, first sort the cancer probability of each instance, then select the instance features with the highest and lowest probabilities, and concatenate these two feature vectors to obtain a new feature vector to represent this sub-package;

[0083] Step 8.3: The feature vectors representing each sub-packet processed in step 8.2 are concatenated again to obtain the final packet-level features.

[0084] In the steps 8.2 and 8.3, the splicing method is selected to be channel splicing or space splicing.

[0085] Step 9: Input the packet-level features obtained in step 8 into the packet classifier for discrimination, and obtain the final binary classification result of whether the breast tissue pathology image is cancerous or not.

[0086] In step 9, a multi-layer perceptron or a convolutional neural network is selected as a packet classifier.

[0087] The technical solution of the present invention is used to integrate the block features of breast tissue pathology images at different magnifications. This strategy takes into account both the local and global information of the image, not only captures the key information in the diagnosis process, but also improves the accuracy and reliability of pathological image analysis. In addition, by training multi-scale features using a self-supervised contrastive learning algorithm, a more robust and discriminative image representation can be learned, improving the performance of the model in pathological image classification tasks. The present invention proposes a two-stage feature fusion network, which is intended to be used for benign and malignant judgment of full-slice images of breast tissue. The method adopts a grouping strategy based on attention scores to divide a package into n sub-packages, and then selects the features of each sub-package for fusion according to feature extraction rules, and finally obtains package-level features. Finally, the package-level features are used to determine whether the pathological image has become cancerous. This method uses significant instances in the package to divide the sub-packages and pays attention to the representative features in the sub-packages, which is relatively rare in existing studies.

[0088] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A breast tissue pathology image classification method based on a two-stage feature fusion network, characterized in that: The method comprises the following steps: Step 1: Use the Otsu threshold algorithm to perform threshold segmentation on the breast tissue pathology image, automatically select the initialization threshold based on the image, and divide the image into foreground and background; Step 2: The breast tissue pathology images were processed at 20x and 10x magnifications respectively using a block technology to divide them into a series of image blocks; Step 3: The image blocks obtained in step 2 are respectively smoothed using median filtering technology to reduce the noise and image blur that may be introduced in the process of pathological image preparation; Step 4: Design a feature fusion module that combines the features of breast tissue pathology image blocks at the same location but different magnifications; Step 5: Apply the self-supervised contrastive learning algorithm and fuse the multi-scale feature vectors through step 4 for training, aiming to extract more discriminative feature representations and enhance the generalization ability of the feature extractor; after the training reaches convergence, retain the best model weights for subsequent analysis tasks; Step 6: Use the feature extractor pre-trained in step 5 to extract features of image patches at different magnifications, and then fuse these features through the feature fusion module in step 4; Step 7: Apply the attention mechanism to the fused features, calculate the attention score of each instance, and select the top β% significant instances with the highest scores to form a new package, and then randomly divide these significant instances into n groups, each group as an independent sub-package; Step 8: Select the instance features of each sub-package and concatenate them to obtain the final package-level features; Step 9: Input the packet-level features obtained in step 8 into the packet classifier for discrimination, and obtain the final binary classification result of whether the breast tissue pathology image is cancerous or not.

2. The breast tissue pathology image classification method based on a two-stage feature fusion network as claimed in claim 1, characterized in that: In step 2, the pathological image is divided into pixel sizes suitable for mainstream convolutional neural network processing, and preparation is made for subsequent feature fusion at different magnifications. The process is as follows: Step 2.1: Use the OpenSlide open source library to read breast tissue pathology image I; Step 2.2: First, acquire images at 20x and 10x magnifications, then select a window of fixed size and continuously slide along the horizontal and vertical directions of image I with a fixed step size s smaller than the window size to acquire image blocks; Step 2.3: Save the image block for subsequent processing.

3. The breast tissue pathology image classification method based on a two-stage feature fusion network as described in claim 1 or 2, characterized in that: In step 3, the noise signal in the image block is removed to improve the image quality. The process is as follows: Step 3.1: Use the OpenCV vision library to load the image patch and apply an observation window with a radius of r for local sampling; Step 3.2: Sort the pixel values ​​in the current observation window to determine the median pixel value; Step 3.3: The calculated pixel median value is used to replace the pixel at the center of the observation window to enhance the local contrast and clarity of the image; Step 3.4: Repeat steps 3.2 to 3.3 to filter the entire image block.

4. The breast tissue pathology image classification method based on a two-stage feature fusion network as described in claim 1 or 2, characterized in that: In step 4, the spatial correlation between image blocks of different scales is considered, so that the model can capture detailed information of pathological images at different scales, thereby providing a more comprehensive description of tissue features. The process is as follows: Step 4.1: For each image block with a magnification of 10 times, extract the feature vector of the image block, and simultaneously obtain the feature vector of the sub-image block at the same position at a magnification of 20 times; Step 4.2: Concatenate the feature vector at 10 times the rate with each feature vector at 20 times the rate, and finally obtain 4 feature vectors.

5. The breast tissue pathology image classification method based on a dual-stage feature fusion network as described in claim 1 or 2, characterized in that: In step 7, a package is divided into multiple sub-packages to expand the scale of training data and allow the model to pay more attention to those significant instances, thereby improving the model's sensitivity and recognition ability for pathological features. The process is as follows: Step 7.1: Use the attention mechanism in AB-MIL to obtain the attention score of each instance; Step 7.2: Sort all attention scores in descending order and select the top β% significant instances as new training samples; Step 7.3: Randomly divide the obtained salient instances into n groups, and each group is regarded as an independent sub-bag.

6. The breast tissue pathology image classification method based on a dual-stage feature fusion network as described in claim 1 or 2, characterized in that: In step 8, representative instance features are selected from the sub-packages to aggregate package-level features and used for subsequent package-level classification. The process is as follows: Step 8.1: Apply the Softmax function to the last layer output of the multi-layer perceptron MLP to convert the original output into the cancer probability corresponding to each instance in the sub-packet; Step 8.2: For each sub-package, first sort the cancer probability of each instance, then select the instance features with the highest and lowest probabilities, and concatenate these two feature vectors to obtain a new feature vector to represent this sub-package; Step 8.3: The feature vectors representing each sub-packet processed in step 8.2 are concatenated again to obtain the final packet-level features.

7. The breast tissue pathology image classification method based on a dual-stage feature fusion network as claimed in claim 6, characterized in that: In steps 8.2 and 8.3, the stitching method is selected as channel stitching or space stitching.

8. The breast tissue pathology image classification method based on a dual-stage feature fusion network as described in claim 1 or 2, characterized in that: In step 2, the blocking technique is selected from random blocking, grid blocking or overlapping sampling blocking; the size of the divided image blocks is selected from 256×256, 512×512 or 768×768 pixels.

9. The breast tissue pathology image classification method based on a dual-stage feature fusion network as described in claim 1 or 2, characterized in that: In step 3, the filtering method is a mean filtering method or a maximum filtering method.

10. The breast tissue pathology image classification method based on a dual-stage feature fusion network according to claim 1 or 2, characterized in that: In the step 4, channel stitching or spatial stitching is selected for splicing; in the step 4.1, Resnet network or VGG network is selected for extracting image block features; in the step 5, MoCov2 or SimCLR contrast learning algorithm is selected as the self-supervised learning framework.

Citation Information

Patent Citations

  • Breast cancer full-size pathological image classification method based on attention multi-instance learning

    CN114998647A

  • Pathological image breast cancer molecular typing method and system based on self-supervised pre-training and multi-instance learning

    CN116884597A

  • Breast cancer full-slice image classification method combining self-supervision and weak supervision learning

    CN117237733A

  • Colorectal cancer digital pathological image differentiation method and system based on weakly supervised learning

    WO2022228349A1