Breast cancer full-slice image classification method based on example search and multi-example learning

Abstract

The invention provides a breast cancer full-slice image classification method based on example search and multi-example learning. The method comprises the following steps: firstly, extracting a foreground region from a full-slice image and segmenting the foreground region into structured image blocks; and then, extracting deep semantic features of the image blocks by using a feature extractor fusing CNN and Transform, and modeling a spatial relationship between the image blocks by introducing a position-coded MIL aggregator, thereby completing full-image-level feature fusion and classification prediction. After convergence of an MIL aggregator, a pseudo-label image block set is dynamically constructed based on the confidence of image blocks, an example classifier is introduced to train a feature extractor of shared parameters, iterative optimization of the feature extractor is realized, and the discrimination capability and classification performance of a model to a breast cancer lesion area are further improved. In a weak supervision pathological image classification task, the accuracy, robustness and expandability of breast cancer WSI classification are remarkably improved through structural design innovation and optimization strategy cooperation, and the method has important clinical application value and industrial transformation prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of breast cancer pathology image analysis, and in particular relates to an automatic classification method for breast cancer full-slice images based on example search and multi-example learning. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women, and its incidence continues to rise globally. According to the 2022 Cancer Statistics Report, breast cancer accounts for 23.8% of new cancer cases in women worldwide, with an incidence rate of 46.8 per 100,000 and a mortality rate of 12.7 per 100,000. It has become one of the leading causes of cancer death among women worldwide. Early breast cancer often presents as no obvious symptoms, often presenting as a painless lump or minor structural changes, making it difficult for patients to take it seriously. Approximately half of patients present to the clinic in the advanced or late stages of the disease, severely impacting their prognosis and survival rate. Therefore, improving the accuracy of early breast cancer diagnosis is crucial for improving cure rates and reducing mortality.

[0003] Currently, clinical breast cancer diagnosis relies primarily on pathological diagnosis, which involves morphological and immunohistochemical analysis of tissue sections obtained from the lesion, followed by interpretation by a pathologist. While this manual interpretation method remains the gold standard, it is highly dependent on the physician's clinical experience, is highly subjective, labor-intensive, and carries the risk of misdiagnosis and missed diagnosis. Furthermore, with the increasing demand for remote diagnosis and treatment, multi-center collaboration, and large-scale screening, traditional pathological diagnosis methods are no longer able to meet the practical needs of achieving both efficiency and accuracy.

[0004] The development of digital pathology technology has made it possible to automatically analyze pathology images. This technology uses high-resolution scanning equipment to convert pathology slides into whole slide images (WSI), laying the foundation for subsequent image calculation and modeling. In recent years, the development of computer vision and deep learning has promoted the rise of computational pathology (CPATH), making automatic diagnosis of breast cancer based on WSI a research hotspot. However, due to the large size of WSI (a single image file often reaches 1 to 3GB), its manual labeling is expensive. In practical applications, it often only has image-level labels and cannot obtain refined block-level annotations. This weak labeling situation poses a huge challenge to model training.

[0005] The Multiple Instance Learning (MIL) method, as a commonly used weakly supervised strategy, is widely used in WSI classification tasks. This method uses a feature extractor to extract features from multiple image blocks (instances) after WSI division and uses a MIL aggregator to aggregate the features to achieve image-level prediction. However, in the existing technology, MIL methods often use frozen pre-trained feature extractors, resulting in a lack of joint optimization between the extractor and aggregator, which limits the final classification performance. In addition, traditional MIL aggregators ignore the spatial structure and contextual relationships between image blocks, and cannot fully understand the distribution characteristics of breast tissue at the macro and micro levels, which affects the model's ability to perceive and discriminate malignant lesions.

[0006] To overcome the above problems, it is urgent to build an end-to-end trainable multi-instance learning framework with context modeling capabilities to fully mine the discriminative information in WSI images and improve the accuracy of breast cancer pathology section classification. Summary of the Invention

[0007] To address the decoupling issues of feature extractors and aggregators in existing MIL methods and the neglect of spatial position information between image blocks, this paper proposes an instance search-based multiple instance learning (IS-MIL) method for breast cancer full-slide image classification based on existing weakly supervised classification methods to improve the accuracy of breast cancer pathology slide classification.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A breast cancer whole-slice image classification method based on example search and multi-instance learning specifically includes the following steps:

[0010] Step 1: Preprocess the full-slice image, including foreground region extraction and image segmentation, to obtain a structured image block sequence;

[0011] Step 2: Input the image block into the feature extractor to extract deep semantic features and generate a sequence of image block feature vectors through global pooling;

[0012] Step 3: Input the image block feature vector sequence into the MIL aggregator based on the gated attention mechanism, introduce position encoding to calculate the attention weight, and perform weighted fusion to generate the global feature matrix of the full slice image, and complete the classification prediction and MIL aggregator parameter optimization;

[0013] Step 4: After the MIL aggregator converges, the confidence of each image block is calculated based on the image block feature vector obtained in step 2 and the corresponding attention weight and prediction value obtained in step 3; image blocks are selected according to the confidence sorting to construct a pseudo-label image block set, and input into the example classifier for training; the example classifier includes a feature extractor that shares parameters with step 2, and the parameters of the feature extractor and the multi-layer perceptron MLP are optimized through training, and the updated feature extractor parameters are synchronized with the feature extractor in step 2 to improve the quality of subsequent image block feature extraction; the above process can be iteratively executed to further improve the overall performance of the model.

[0014] Compared with existing methods, the breast cancer whole-slice image classification method based on example search and multi-example learning of the present invention has the following significant advantages and positive effects:

[0015] Enhanced semantic modeling capabilities for image patches: Existing methods generally use shallow or single-layer architectures like ResNet to extract image patch features, making it difficult to simultaneously capture fine-grained textures and inter-image context. This paper constructs a multi-scale feature extractor, CTransPath, that fuses CNN and Swin Transformer. This extractor can fully extract local pathological details and global spatial structural features of image patches, enhancing the semantic representation of image patches and providing richer high-dimensional feature support for subsequent breast cancer classification tasks.

[0016] Fully utilize the spatial structural information of image blocks: This invention introduces a position encoding mechanism into the MIL aggregator to fuse the semantic features of the image blocks with their spatial position information in the whole image, so that the aggregation weights have spatial perception capabilities, thereby enhancing the aggregator's ability to understand pathological tissue structures, realizing context-structure-aware whole-image classification prediction, and significantly improving the accuracy of whole-image feature representation and the model's ability to discriminate breast cancer areas.

[0017] Realize collaborative optimization of feature extractors and aggregators: In existing methods, feature extractors are often frozen and parameters cannot be adjusted dynamically according to the task, which limits the model's expressive ability. The present invention designs an example search mechanism to screen potential key lesion areas based on the category confidence of image blocks, constructs an example classifier, and guides the feature extractor to focus on representative high-discriminative areas. During the training process, while keeping the main path feature extractor frozen, the example classifier indirectly optimizes its parameters through pseudo-label feedback, thereby enhancing the feature expression ability. At the same time, this mechanism realizes the collaborative training between the feature extractor and the MIL aggregator, effectively improving the model's discriminative ability and robustness in the task of classifying full-slice images of breast cancer.

[0018] Dynamic screening of pseudo-labels improves training results: The present invention dynamically sorts image blocks according to their confidence levels, selects some image blocks with high and low confidence levels to construct pseudo-label sets, and thus achieves selective construction of pseudo-label sets. As the number of iterations increases, the number of pseudo-labeled image blocks gradually increases, effectively reducing the interference of inaccurate initial pseudo-labels on model training. Compared with the existing method of constructing pseudo-labels from all image blocks, the strategy of the present invention significantly improves the quality of pseudo-labels and the stability of training, enhances the discriminative ability and robustness of the model, and optimizes the automatic classification performance of full-slice breast cancer images.

[0019] Outperforming existing methods in multiple indicators: Experimental results show that the classification accuracy, AUC value and F1 score of the method of the present invention on the Camelyon16 breast cancer pathology slide dataset are significantly higher than those of existing mainstream methods. The highest AUC value is as high as 99.41%, which is better than the existing baseline method (such as 94.34% of the BCL model), verifying the advanced nature and effectiveness of this method.

[0020] In summary, in the weakly supervised pathology image classification task, the present invention significantly improves the accuracy, robustness and scalability of breast cancer WSI classification through structural design innovation and optimization strategy collaboration, and has important clinical application value and industrial transformation prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is an experimental flow chart of the present invention;

[0022] Figure 2 This is a flowchart of preprocessing of breast cancer whole-slice images provided in Embodiments 1, 2, and 3 of the present invention;

[0023] Figure 3 This is a schematic diagram of the data set and its divided sample size provided in embodiments 1, 2, and 3 of the present invention;

[0024] Figure 4 This is the attention heat map provided in Embodiments 1, 2, and 3 of the present invention. DETAILED DESCRIPTION

[0025] In accordance with the above technical solution, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the present invention is not limited to the following specific embodiments, and all equivalent changes made on the basis of the technical solution of this application fall within the scope of protection of the present invention.

[0026] This paper provides a breast cancer whole-slide image classification method based on example search and multiple example learning (IS-MIL), which is used to effectively improve breast cancer classification performance under weak supervision conditions with only WSI-level labels. The method mainly includes the following steps:

[0027] Step 1: Preprocess the full slice image, including foreground region extraction and image segmentation, to obtain a structured image block sequence as subsequent input.

[0028] Step 2: Input the image block into the feature extractor to extract deep semantic features and generate an image block feature vector sequence through global pooling.

[0029] Step 3: Input the image block feature vector sequence into the MIL aggregator based on the gated attention mechanism, introduce position encoding to calculate the attention weight, and perform weighted fusion to generate the global feature matrix of the full slice image, and complete the classification prediction and MIL aggregator parameter optimization;

[0030] Step 4: After the MIL aggregator converges, the confidence of each image block is calculated based on the image block feature vector obtained in step 2 and the corresponding attention weight and prediction value obtained in step 3; image blocks are selected according to the confidence sorting to construct a pseudo-label image block set, and input into the example classifier for training; the example classifier contains a feature extractor that shares parameters with step 2. The parameters of the feature extractor and the multi-layer perceptron MLP are optimized through training, and the updated feature extractor parameters are synchronized with the feature extractor in step 2 to improve the quality of feature extraction of subsequent image blocks; the above process can be iteratively executed to further improve the overall performance of the model.

[0031] The present invention also includes the following technical features:

[0032] Specifically, the step 1 includes:

[0033] Step 1.1: Take a set of full-slice image datasets with slice-level labels (in represents the i-th full slice image, Y i ∈{0, 1} represents the classification label corresponding to the image, 0 represents normal and 1 represents breast cancer) as input data.

[0034] Step 1.2: Segment the whole slice image to extract the foreground area.

[0035] Specifically, the step 1.2 is described in detail as follows:

[0036] Step 1.2.1: Perform color space conversion on the original whole-slice image, converting the RGB image into an HSV image to improve the distinction between pathological tissue structures.

[0037] Step 1.2.2: Apply a median filter to the saturation channel of the HSV image to smooth the image edges.

[0038] Step 1.2.3: Use a fixed threshold segmentation algorithm on the saturation channel to extract the foreground area (i.e., tissue area) and generate the corresponding binary mask image.

[0039] Step 1.2.4: Apply morphological closing operation to the mask image to eliminate small holes and connect adjacent areas;

[0040] Step 1.2.5: Filter the connected regions in the mask according to the area threshold to obtain a complete and accurate foreground region.

[0041] Step 1.3: Divide the foreground area into blocks.

[0042] Specifically, the step 1.3 is described in detail as follows:

[0043] Step 1.3.1: Set the fixed image block size to 224 × 224 pixels and use a sliding window strategy to crop the foreground area.

[0044] Step 1.3.2: Crop the foreground image with non-overlapping sliding steps to obtain a set of image blocks. Each image block Since the tissue area of ​​different WSIs is different, the number of image blocks N i It may be different in different samples.

[0045] Specifically, the step 2 includes:

[0046] Step 2.1: Input all image blocks obtained in step 1 into the feature extractor (CTransPath), which consists of a CNN module and a Swin Transformer module.

[0047] Step 2.2: Use the CNN module to input each image block Perform feature extraction and obtain a feature map of size h / 4×w / 4×96:

[0048]

[0049] in, Represents a convolution operator with a kernel size of k×k.

[0050] Step 2.3: Use the Swin Transformer module to further model the contextual relationship of the image block and enhance the global semantic information of the feature map, obtaining a deep feature map of size h / 32×w / 32×768:

[0051]

[0052] Specifically, the step 2.3 is described in detail as follows:

[0053] Step 2.3.1: Output the feature map of the CNN module Enter the Swin Transformer module. The Swin Transformer module mainly consists of two parts: the Patch Merging layer and the Swin Transformer Block.

[0054] Step 2.3.2, use the patch merging layer to merge the input feature map Downsampling is performed to reduce the resolution and adjust the number of channels, thereby forming a hierarchical design and reducing the amount of computation.

[0055] Step 2.3.3: Downsample the feature map after the patch merging layer Input to the Swin Transformer block, each Swin Transformer block consists of two layers.

[0056] In the first layer, window multi-head self-attention (W-MSA) is used to capture local information. The image is split into multiple local windows of size M×M, and a multi-head self-attention mechanism (W-MSA) is independently performed within each window to capture local context information. Specifically, the input features within each window are linearly transformed to generate query (Q), key (K), and value (V) vectors. The attention weights are then obtained through dot product and softmax normalization. The features are then fused with relative position bias to form a window-level representation.

[0057] To further enhance cross-window modeling capabilities, a shifted window attention mechanism (SW-MSA) is introduced in the second layer. This mechanism enables information interaction through window offsets, effectively enhancing global perception. Each layer module is updated through residual connections and multi-layer perceptrons (MLPs), ultimately outputting a deep feature map of size h / 32 × w / 32 × 768.

[0058] Step 2.4: Each image block obtained in step 2.3 Apply the global pooling operator F to the deep feature map global (·) Globally aggregate features to generate feature vectors

[0059]

[0060] in, The feature vectors of all image blocks in the same slice image are integrated and represented as a sequence, then the full slice image X i It can be expressed as:

[0061] Specifically, the step 3 includes:

[0062] Step 3.1: The whole slice image X obtained in step 2 i The feature vector sequence of all image blocks Input into the MIL aggregator built based on the gated attention mechanism.

[0063] Step 3.2: For the above full slice image X i The image block feature vector sequence By adding a position encoding vector, the MIL aggregator considers the spatial distribution information of image blocks in the entire image when aggregating image block features, thereby enhancing the model's ability to model the spatial relationship of image blocks.

[0064] Step 3.3: Add the full slice image X after position encoding i The image block feature vector is input into the attention module, the attention score of each image block is calculated through the fully connected layer, and the Softmax function is used for normalization to obtain the attention weight corresponding to each image block.

[0065] Step 3.4: According to the above attention weights, the full slice image X i The feature vector sequence of all image blocks Perform weighted summation to finally obtain the global feature matrix of the entire slice image:

[0066] Step 3.5: transform the above global feature matrix Input classification layer f CLS (·) Perform classification prediction and obtain prediction results Use the cross entropy loss function to calculate the true label Y i and the predicted results The error between them is used to optimize the training of the MIL aggregator until the MIL aggregator converges.

[0067] Specifically, step 4 includes:

[0068] Step 4.1: After the MIL aggregator converges, obtain the full-slice image X i Each image block The eigenvector of And through the classification layer f CLS (·) The corresponding eigenvector Mapping to each image block About Xi The predicted value of

[0069] Step 4.2: Extract each image block obtained in step 3.3 from the MIL aggregator The attention weight The attention weight The corresponding image block obtained in step 4.1 The predicted value of Multiply and calculate the confidence of each image block This confidence measures the image block Belongs to class Y i possibility.

[0070] Step 4.3: Based on the confidence of all the above image blocks For image blocks Sort in descending order to generate an ordered index set of image blocks Assume the current number of iterations is t, and select the top K confidence rankings according to the sorting results. t image patches and set their pseudo labels to the true labels Y of the full slice image i ∈{0, 1}, where 1 indicates breast cancer and 0 indicates no breast cancer. t The pseudo label of each image block is fixed to 0, and a set of pseudo label image blocks is obtained. t is a positive integer that changes dynamically with the number of iterations t, defined as K t = min(num, (t+1)·10), where num is a preset hyperparameter that controls the growth rate. As t increases, the number of image patches involved in pseudo-label construction gradually increases, thereby reducing the interference of inaccurate initial pseudo-labels on model training.

[0071] In step 4.4, the pseudo-labeled image patch set constructed in step 4.3 is fed into the example classifier for training. This example classifier consists of a feature extractor and a multi-layer perceptron (MLP). During training, the feature extractor and MLP parameters of the example classifier are optimized by calculating the loss between the pseudo-labels and the output of the example classifier. After training converges, the feature extractor parameters of the example classifier are shared with the feature extractor in step 2, achieving parameter sharing.

[0072] Embodiments 1, 2, and 3:

[0073] This embodiment discloses a method for classifying breast cancer whole-slice images based on example search and multi-example learning, which includes the following steps:

[0074] Step 1: Preprocess the full slice image, including foreground region extraction and image segmentation, to obtain a structured image block sequence as subsequent input.

[0075] Step 2: Input the image block into the feature extractor to extract deep semantic features and generate an image block feature vector sequence through global pooling.

[0076] Step 3: Input the image block feature vector sequence into the MIL aggregator based on the gated attention mechanism, introduce position encoding to calculate the attention weight, and perform weighted fusion to generate the global feature matrix of the full slice image, and complete the classification prediction and MIL aggregator parameter optimization;

[0077] Step 4: After the MIL aggregator converges, the confidence of each image block is calculated based on the image block feature vector obtained in step 2 and the corresponding attention weight and prediction value obtained in step 3; image blocks are dynamically selected based on the confidence sorting to construct a pseudo-label image block set, and input into the example classifier for training; the example classifier contains a feature extractor that shares parameters with step 2, and the parameters of the feature extractor and the multi-layer perceptron MLP are optimized through training, and the updated feature extractor parameters are synchronized with the feature extractor module in step 2 to improve the quality of subsequent image block feature extraction; the above process can be iteratively executed to further improve the overall performance of the model.

[0078] Step 1: Preprocess the full slice image, including foreground region extraction and image segmentation, to obtain a structured image block sequence as subsequent input.

[0079] In this step, the acquisition and annotation process of breast cancer whole-slice images (WSIs) is as follows: First, tissue samples obtained from surgical resection or needle biopsy are collected from breast cancer patients. After fixation in 10% neutral formalin, the tissue is dehydrated and transparentized, then embedded in paraffin and cooled to solidify. Subsequently, the paraffin-embedded blocks are cut into serial tissue sections with a thickness of 4-5 microns using a microtome and mounted on glass slides. After hematoxylin and eosin (H&E) staining, the sections are scanned at 40× magnification using a high-resolution digital pathology scanner (such as Hamamatsu or Leica), generating digital WSI images with a resolution of approximately 0.25 μm / pixel, commonly in the .tiff format. Finally, a professional pathologist diagnoses each WSI using a microscope or digital image. The presence of malignancy is determined based on histological features such as cellular atypia, mitotic figures, and ductal abnormalities. Slice-level labels are then assigned: 1 for the presence of breast cancer and 0 for the absence of breast cancer.

[0080] Step 1 specifically includes:

[0081] Step 1.1: Take a set of full-slice image datasets with slice-level labels (in Indicates the i-th WSI, Yi ∈{0, 1} represents the classification label corresponding to the image, 0 represents normal, and 1 represents breast cancer) is used as input data for the breast cancer image classification task of the present invention.

[0082] Step 1.2: Segment the whole slice image to extract the foreground area.

[0083] Specifically, the step 1.2 is described in detail as follows:

[0084] Step 1.2.1: Perform color space conversion on the original whole-slice image, converting the RGB image into an HSV image to improve the distinction between pathological tissue structures.

[0085] Step 1.2.2: Apply a median filter to the saturation channel of the HSV image to smooth the image edges.

[0086] Step 1.2.3: Use a fixed threshold segmentation algorithm on the saturation channel to extract the foreground area (i.e., tissue area) and generate the corresponding binary mask image.

[0087] Step 1.2.4: Apply morphological closing operation to the mask image to eliminate small holes and connect adjacent areas;

[0088] Step 1.2.5: Filter the connected regions in the mask according to the area threshold to obtain a complete and accurate foreground region.

[0089] Step 1.3: Divide the foreground area into blocks.

[0090] Specifically, the step 1.3 is described in detail as follows:

[0091] Step 1.3.1: Set the fixed image block size to 224 × 224 pixels and use a sliding window strategy to crop the foreground area.

[0092] Step 1.3.2: Crop the foreground image with non-overlapping sliding steps to obtain a set of image blocks. Each image block Since the tissue area of ​​different WSIs is different, the number of image blocks N i It may be different in different samples.

[0093] Step 2: Input the image block into the feature extractor to extract deep semantic features and generate an image block feature vector sequence through global pooling.

[0094] In Example 1, to further evaluate the effectiveness and advancement of the proposed method, a comparative experiment was conducted using two feature extractors: the ResNet-50 feature extractor, widely used in existing methods, and the CTransPath feature extractor proposed in this invention. In Examples 2 and 3, the CTransPath feature extractor proposed in this invention was used uniformly.

[0095] Step 2 specifically includes:

[0096] Step 2.1: Input all image blocks obtained in step 1 into the feature extractor (CTransPath). The feature extractor (CTransPath) consists of a CNN module and a Swin Transformer module.

[0097] Step 2.2: Use the CNN module to input each breast cancer pathology image block Perform feature extraction and obtain a feature map of size h / 4×w / 4×96:

[0098]

[0099] in, Represents a convolution operator with a kernel size of k×k.

[0100] Step 2.3: Use the Swin Transformer module to further model the contextual relationship of the image blocks and enhance the global semantic information of the above feature map, and obtain a deep feature map of size h / 32×w / 32×768:

[0101]

[0102] Specifically, the step 2.3 is described in detail as follows:

[0103] Step 2.3.1: Output the feature map of the CNN module Enter the Swin Transformer module. The Swin Transformer module mainly consists of two parts: the Patch Merging layer and the Swin Transformer Block.

[0104] Step 2.3.2, use the patch merging layer to merge the input feature map Downsampling is performed to reduce the resolution and adjust the number of channels, thereby forming a hierarchical design and reducing the amount of computation.

[0105] Step 2.3.3: Downsample the feature map after the patch merging layer Input to the Swin Transformer Block. Each Swin Transformer Block consists of two layers.

[0106] In the first layer, window multi-head self-attention (W-MSA) is used to capture local information. The image is split into multiple local windows of size M×M, and a multi-head self-attention mechanism (W-MSA) is independently performed within each window to capture local context information. Specifically, the input features within each window are linearly transformed to generate query (Q), key (K), and value (V) vectors. The attention weights are then obtained through dot product and softmax normalization. The features are then fused with relative position bias to form a window-level representation.

[0107] Assume that the input feature of each local window is represented as That is, a window of size M×M has C channels. The calculation process of window-based self-attention (W-SA) is as follows:

[0108] Linear projection: Q←W Q I, K←W K I, V←W V I

[0109] in, It is a trainable linear projection weight matrix that projects the input feature I into three subspaces: query, key, and value, with a dimension of d.

[0110] Windowed Self-Attention (W-SA):

[0111] Here, B is a relative or absolute position encoding matrix used to embed position information, and the Softmax function is used to normalize the attention weights. This operation captures the contextual relationships between pixels within the window. The interaction relationships are obtained by calculating the dot product between K and Q, and the weights are then scaled and projected into space V to obtain the W-SA feature embedding. The self-attention operation can be performed multiple times in parallel, and the results are concatenated to obtain the output features of the W-MSA.

[0112] To further enhance cross-window modeling capabilities, a shifted window attention mechanism (SW-MSA) is introduced in the second layer. This mechanism enables information interaction through window offsets, effectively enhancing global perception. Each layer module is updated through residual connections and multi-layer perceptrons (MLPs), ultimately outputting a deep feature map of size h / 32 × w / 32 × 768.

[0113] Assuming that the window size of the lth layer is M×M, then in the l+1th layer, all windows will be shifted by (M / 2, M / 2) pixels relative to the previous layer before SW-MSA calculation is performed.

[0114] The calculation process of the l-th layer and l+1-th layer Swin Transformer Block is as follows:

[0115]

[0116] Where: z (l) is the output embedding of layer l; LN(·) denotes layer normalization; W-MSA(·) and SW-MSA(·) denote windowed multi-head self-attention and shifted windowed multi-head self-attention operations, respectively; MLP(·) is a feedforward network consisting of two linear layers and GELU activations; all modules employ residual connections to facilitate deep network training. This alternating structure (W-MSA → SW-MSA) is the key mechanism by which the Swin Transformer builds global modeling capabilities, effectively overcoming the limitation of traditional windowed attention, which is limited to a local receptive field.

[0117] Step 2.4: Each image block obtained in step 2.3 Apply the global pooling operator F to the deep feature map global (·) Globally aggregate features to generate feature vectors

[0118]

[0119] in, The feature vectors of all image blocks in the same slice image are integrated and represented as a sequence, then the full slice image X i It can be expressed as:

[0120] Specifically, the step 3 includes:

[0121] Step 3.1: The whole slice image X obtained in step 2 i The feature vector sequence of all image blocks Input into the MIL aggregator built based on the gated attention mechanism.

[0122] Step 3.2: For the above full slice image X i The image block feature vector sequence By adding a position encoding vector, the MIL aggregator considers the spatial distribution information of image blocks in the entire image when aggregating image block features, thereby enhancing the model's ability to model the spatial relationship of image blocks.

[0123] Due to the original characteristics It does not contain spatial position information, so the spatial relationship between image blocks is explicitly modeled by adding position encoding, which is defined as follows:

[0124]

[0125] Where t∈{0, 1, ..., d-1} represents the dimension index of the position encoding vector, is the frequency factor, and d represents the feature dimension. The above design enables the model to simultaneously utilize the semantic features of image patches and their spatial position relationships, thereby improving the discrimination ability during multi-instance aggregation.

[0126] Step 3.3: Add the full slice image X after position encoding i The image block feature vector is input into the attention module, the attention score of each image block is calculated through the fully connected layer, and the Softmax function is used for normalization to obtain the attention weight corresponding to each image block.

[0127] The above weights The Softmax function is used for normalization, which is defined as follows:

[0128]

[0129] Among them, FC(·) represents the fully connected layer operation, For image blocks The position encoding vector of .

[0130] In the first and third embodiments, the method for adding the position encoding module PE proposed by the present invention is uniformly adopted.

[0131] In Example 2, in order to verify the effectiveness of the position encoding module adding method in the method proposed in the present invention, three comparative experiments are constructed to perform image block feature fusion: (1) retaining the position encoding module PE, (2) removing the position encoding module PE, and (3) removing the position encoding module PE and adding a new position encoding module PE1.

[0132] The position encoding module PE and the new position encoding module PE1 are added as follows:

[0133] Assume the input feature matrix is Represents the feature representation of N image patches from the whole slice image (WSI), and the position encoding matrix is

[0134] The calculation process of the new position encoding method PE1 is:

[0135] Z=softmax(FC(FC(X+B)))·(X+B)

[0136] Among them, FC(··) represents the fully connected layer operation, softmax(··) represents the standard normalized weight calculation, and the output Represents the weighted global feature representation. The core idea is to embed the position information directly into the image block feature matrix and jointly participate in modeling in the attention weighting process.

[0137] The calculation process of the position encoding method PE adopted by the method proposed in the present invention is as follows:

[0138] Z=softmax(FC(FC(X))+B)·X

[0139] This approach differs in that it introduces the positional encoding as a bias term into the weight calculation path, rather than adding it directly to the feature input. In this approach, the positional encoding B is used to adjust the attention score of each image patch without directly changing the feature input. This allows spatial information to be injected without interfering with semantic expression, enhancing the model's ability to model the positional information of image patches.

[0140] Step 3.4: Attention weights are used to analyze the full slice image X i The feature vector sequence of all image blocks Perform weighted summation to finally obtain the global feature matrix of the entire slice image:

[0141] By weighted summation, the global feature vector of the i-th image can be obtained: Each dimension represents the global response associated with category c and is calculated as follows:

[0142]

[0143] in, Represents the feature vector The value of the cth component in , Represents an image block The attention weight in the aggregation is used to reflect the image block’s effect on the predicted category Y i contribution.

[0144] Step 3.5: transform the above global feature matrix Input classification layer f CLS (·) Perform classification prediction and obtain prediction results Use the cross entropy loss function to calculate the true label Y i and the predicted results The error between them is used to optimize the training of the MIL aggregator until the MIL aggregator converges.

[0145] Specifically, step 4 includes:

[0146] Step 4.1: After the MIL aggregator converges, obtain the full-slice image X i Each image block The eigenvector of And through the classification layer f CLS (·) The corresponding eigenvector Mapping to each image block About X i The predicted value of

[0147] Step 4.2: Extract each image block calculated in step 3.3 from the MIL aggregator The attention weight The attention weight The corresponding image block obtained in step 4.1 The predicted value of Multiply and calculate the confidence of each image block This confidence measures the image block Belongs to class Y i possibility.

[0148] Step 4.3: Based on the confidence of all the above image blocks For image blocks Sort in descending order to generate an ordered index set of image blocks Assume the current number of iterations is t, and select the top K confidence rankings according to the sorting results. t image patches and set their pseudo labels to the true labels Y of the full slice image i ∈{0, 1}, where 1 indicates breast cancer and 0 indicates no breast cancer. t The pseudo label of each image block is fixed to 0, and a set of pseudo label image blocks is obtained. t is a positive integer that changes dynamically with the number of iterations t, defined as K t=min(num, (t+1)·10), which is a preset hyperparameter that controls the growth rate. As t increases, the number of image patches involved in pseudo-label construction gradually increases, thereby reducing the interference of inaccurate initial pseudo-labels on model training.

[0149] In the first and second embodiments, the number of image blocks in step 4.3 of the method proposed by the present invention is uniformly set as follows: K t =min(num, (t+1)·10), where num=40.

[0150] In Example 3, in order to further evaluate the performance of the multi-instance learning method based on example search proposed in the present invention under different settings of the number of image blocks, two example classifier training data scale adjustment strategies are designed.

[0151] The first strategy is to introduce normal image blocks of different proportions to enhance the model's ability to distinguish non-tumor areas. Considering that the tumor area usually occupies a large proportion in the whole slice image (WSI), the number of normal image blocks with the label "0" is appropriately increased when assigning pseudo labels. Specifically, let the number of breast cancer image blocks be K t =(t+1)×10(K in step 4.3 t =min(num, (t+1)·10) modified), on this basis, the number of normal image blocks is set to K t 1 times, 1.5 times, 2 times and 3 times of neg =1·K t 、num neg =1.5·K t 、num neg =2·K t 、num neg =3·K t .

[0152] The second strategy is to limit the number of image patches assigned pseudo labels to control the scale and quality of training samples. Specifically, by adjusting K in step 4.3 t = min(num, (t+1)·10) The control parameter num of the number of image blocks is used. Experiments are conducted under the conditions where num is set to 10, 20, 30, 40, and 50. The classification performance under different settings is compared.

[0153] In step 4.4, the set of pseudo-labeled image patches constructed in step 4.3 is fed into the example classifier for training. This example classifier consists of a feature extractor and a multi-layer perceptron (MLP). During training, the feature extractor and MLP parameters of the example classifier are optimized by calculating the loss between the pseudo-labels and the classifier output. After training converges, the feature extractor parameters of the example classifier are shared with the feature extractor in step 2, achieving parameter sharing.

[0154] In Example 2, in order to further evaluate the effectiveness of the example search module of the method proposed in the present invention, two comparative experiments were constructed: (1) retaining the example search module, that is, implementing steps 1-4; (2) removing the example search module, that is, only implementing steps 1-3. In Examples 1 and 3, the example search module proposed in the present invention was retained, that is, steps 1-4 were implemented.

[0155] In Examples 1, 2, and 3, the Camelyon16 dataset is used to verify the effectiveness of the method proposed in the present invention. Figure 1 The flowchart shown in the figure sequentially performs image block feature extraction, image block feature aggregation, and example search training to optimize feature extractor parameters. The image block obtained in step 1 is input into the feature extractor to generate an image block feature vector sequence; the image block feature vector sequence obtained in step 2 is input into the MIL aggregator to generate a global feature representation of the entire slice image, thereby completing the classification prediction and aggregator parameter optimization; after the MIL aggregator converges, the features, attention weights, and prediction values ​​of each image block in step 3 are extracted, the image block confidence is calculated, and a pseudo-label image block set is dynamically constructed based on the confidence ranking, and this set is input into the example classifier for training. The example classifier includes a feature extractor module that shares parameters with step 2. The parameters of this feature extractor and the MLP classifier are optimized through training, and the updated feature extractor parameters are synchronized with the feature extractor module described in step 2 to further improve the quality of image block feature extraction. The above process can be iteratively executed to continuously optimize the overall model performance and improve the classification effect of breast cancer pathology images. Finally, an independent test set was input into the trained model for evaluation to obtain the final classification results of the breast cancer whole-slice image classification task, thereby verifying the model's generalization ability and practical application effect.

[0156] Table 1 Classification performance of breast cancer full slide image classification methods on the Camelyon16 dataset

[0157]

[0158]

[0159] Note: ABMIL

[10] , CLAM-SB

[11] , CLAM-MB

[11] , TransMIL

[40] and DTFD-MIL

[13] are five classic multi-instance learning (MIL) methods, and DSMIL

[12] , NcIEMIL

[20] , PseMIX

[88] , PMIL

[18] , WSI-FT

[15] and BCL

[16] are six MIL-based whole-slice image classification methods. IS-MIL(CTransPath) is the method proposed in this paper, and IS-MIL(ResNet50) is the method after replacing the feature extractor of the method proposed in this paper with ResNet50 instead of CTransPath.

[0160] To verify the superiority of the proposed method in breast cancer whole-slice image classification, Table 1 shows the specific numerical values ​​of the classification performance evaluation indicators based on the proposed method in Example 1, as well as the specific numerical values ​​of the classification performance evaluation indicators of 11 existing whole-slice image classification methods using the same dataset as Example 1.

[0161] Evaluation metrics include accuracy, AUC value, and F1 score. Accuracy measures the proportion of whole-slice images correctly predicted by the method of the present invention; AUC value measures the discrimination ability of the method of the present invention, that is, the ability of the method of the present invention to correctly distinguish whole-slice images at different thresholds; F1 score is the harmonic mean of precision and recall. Precision measures the proportion of all whole-slice images predicted to have breast cancer by the method of the present invention that actually have breast cancer. Recall measures the proportion of all whole-slice images that actually have breast cancer that are correctly classified. A higher F1 score indicates that the model achieves a better balance between precision and recall.

[0162] Clinical interpretability. Breast cancer classification requires a balance between sensitivity and specificity, aiming to identify as many cancer cases as possible while avoiding misdiagnoses. Therefore, the primary goal of breast cancer full-slide image classification is to achieve a higher AUC value, with secondary goals being higher accuracy and F1 score.

[0163] In Example 1, the accuracy, AUC value and F1 score of the breast cancer whole-slice image classification method of the present invention are 97.67%, 99.41% and 96.84% respectively, which are significantly better than the existing eleven breast cancer whole-slice image classification methods. Among them, the accuracy rate is improved by 6.2% to 15.5% compared with other methods, the AUC value is improved by 5.07% to 18.56%, and the F1 score is improved by 7.13% to 15.85%. In particular, in terms of the AUC value indicator, the method proposed by the present invention reaches 99.41%, which is significantly improved compared with the best-performing BCL method (94.34%), reflecting the excellent stability and good generalization ability of the method proposed by the present invention in the task of breast cancer whole-slice image classification.

[0164] In addition, since the feature extractors of these methods all use ResNet50, in order to further evaluate the effectiveness of the method proposed in the present invention, after the feature extractor of the method proposed in the present invention was replaced from CTransPath to ResNet50, the accuracy, AUC value and F1 score of breast cancer full-slice image classification were 92.24%, 94.50% and 90.57% respectively, which are still significantly better than the existing eleven breast cancer full-slice image classification methods. Among them, the accuracy rate was improved by 0.77% to 10.07% compared with other methods, the AUC value was improved by 0.16% to 13.65%, and the F1 score was improved by 0.86% to 9.58%. The classification results verified the unique advantages of the method proposed in the present invention in the example search module and the aggregator module, indicating that the excellent performance of this method is not only due to the performance of the feature extractor itself, but also due to its efficient information aggregation method, which can effectively improve the classification ability in pathological image classification tasks.

[0165] In Example 1, the proposed method fully demonstrated its superiority in the classification of breast cancer whole-slice images. Its efficient information aggregation method and example search strategy can effectively improve the classification ability of the model and show strong robustness in the problem of class imbalance.

[0166] Table 2 Impact of key modules on the classification performance of the proposed method on the Camelyon16 dataset

[0167]

[0168] Note: w / o ISS: remove the example search module (without ISS), w / o PE: remove the position encoding module (without PE), w / o PE+w / PE1: remove the position encoding module (without PE), add a new position encoding module (without PE1), Full model: no module is removed (full model).

[0169] In order to verify the effectiveness of each key module in the method proposed in the present invention, Table 2 shows the specific values ​​of the classification performance evaluation indicators in Example 2 based on the method proposed in the present invention.

[0170] Evaluation metrics include accuracy, AUC value, and F1 score. Accuracy measures the proportion of whole-slice images correctly predicted by the method of the present invention; AUC value measures the discrimination ability of the method of the present invention, that is, the ability of the method of the present invention to correctly distinguish whole-slice images at different thresholds; F1 score is the harmonic mean of precision and recall. Precision measures the proportion of all whole-slice images predicted to have breast cancer by the method of the present invention that actually have breast cancer. Recall measures the proportion of all whole-slice images that actually have breast cancer that are correctly classified. A higher F1 score indicates that the model achieves a better balance between precision and recall.

[0171] Clinical interpretability. Breast cancer classification requires a balance between sensitivity and specificity, aiming to identify as many cancer cases as possible while avoiding misdiagnoses. Therefore, the primary goal of breast cancer full-slide image classification is to achieve a higher AUC value, with secondary goals being higher accuracy and F1 score.

[0172] In Example 2, when the key modules were not removed, the accuracy, AUC value, and F1 score of the proposed method for breast cancer full-slide image classification were 97.67%, 99.41%, and 96.84%, respectively. The classification results show that the proposed method provides a more comprehensive observation and evaluation from the perspective of the importance and effectiveness of the key modules. After removing the example search module, the accuracy, AUC value, and F1 score of the proposed method for breast cancer full-slide image classification decreased from 97.67%, 99.41%, and 96.84% to 85.27%, 89.00%, and 83.18%, respectively. After removing the position encoding module, the accuracy, AUC value, and F1 score of the proposed method for breast cancer full-slide image classification decreased from 97.67%, 99.41%, and 96.84% to 96.90%, 97.96%, and 95.83%, respectively. After removing the position encoding module (PE) and adding a new position encoding module (PE1), the accuracy, AUC value and F1 score of breast cancer full-slice image classification decreased from 97.67%, 99.41% and 96.84% to 93.02%, 96.96% and 91.09%, respectively.

[0173] This shows that the example search module and position encoding module of the breast cancer full-slice image classification method proposed in the present invention play a key role in improving the accuracy and stability of the breast cancer full-slice image classification model. The example search module enhances the robustness of example screening, and the position encoding module effectively models the spatial relationship between image blocks, verifying the effectiveness and necessity of the key designs in the proposed method, and further demonstrating the superiority of the position encoding method (PE) designed in the proposed method.

[0174] Table 3 Effect of setting the number of image blocks on the classification performance of the proposed method on the Camelyon16 dataset

[0175]

[0176] In order to further explore the impact of the number of image blocks set in the example search strategy of the method proposed in the present invention on the classification performance of the method proposed in the present invention, Table 3 shows the specific values ​​of the classification performance evaluation indicators based on the method proposed in the present invention in Example 3.

[0177] Evaluation metrics include accuracy, AUC value, and F1 score. Accuracy measures the proportion of whole-slice images correctly predicted by the method of the present invention; AUC value measures the discrimination ability of the method of the present invention, that is, the ability of the method of the present invention to correctly distinguish whole-slice images at different thresholds; F1 score is the harmonic mean of precision and recall. Precision measures the proportion of all whole-slice images predicted to have breast cancer by the method of the present invention that actually have breast cancer. Recall measures the proportion of all whole-slice images that actually have breast cancer that are correctly classified. A higher F1 score indicates that the model achieves a better balance between precision and recall.

[0178] Clinical interpretability. Breast cancer classification requires a balance between sensitivity and specificity, aiming to identify as many cancer cases as possible while avoiding misdiagnoses. Therefore, the primary goal of breast cancer full-slide image classification is to achieve a higher AUC value, with secondary goals being higher accuracy and F1 score.

[0179] In Example 3, when the method of the present invention is executed in the first embodiment and the hyperparameter num is set to 40, the accuracy, AUC value and F1 score of breast cancer full-slice image classification are 97.67%, 99.41% and 96.84% respectively. The classification results show that the method of the present invention provides a more comprehensive observation and evaluation from the perspective of data distribution of breast cancer full-slice images and performance improvement and computational efficiency. Implementation method 1, when the number of normal image blocks is set to twice the number of breast cancer image blocks (i.e., numn e When the hyperparameter g = 2·Kt), the AUC value for breast cancer full-slide image classification is higher than the AUC value when the number of normal image blocks is set to other multiples of the breast cancer image blocks. In implementation mode 2, when the hyperparameter num is set to 40, the accuracy, AUC value, and F1 score for breast cancer full-slide image classification are all higher than those when the hyperparameter num is set to other values. This demonstrates the efficiency of setting the number of image blocks in the example search strategy of the breast cancer full-slide image classification method proposed in this invention.

[0180] The preferred embodiments of the present disclosure are described in detail above. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0181] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0182] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A breast cancer full-slice image classification method based on example search and multi-instance learning, characterized by: The following steps are involved: Step 1: Preprocess the full-slice image, including foreground region extraction and image segmentation, to obtain a structured image block sequence; Step 2: Input the image block into the feature extractor to extract deep semantic features and generate a sequence of image block feature vectors through global pooling; Step 3: Input the image block feature vector sequence into the MIL aggregator based on the gated attention mechanism, introduce position encoding to calculate the attention weight, and perform weighted fusion to generate the global feature matrix of the full slice image, and complete the classification prediction and MIL aggregator parameter optimization; Step 4: After the MIL aggregator converges, the confidence of each image block is calculated based on the image block feature vector obtained in step 2 and the corresponding attention weight and prediction value obtained in step 3; image blocks are selected according to the confidence sorting to construct a pseudo-label image block set, and input into the example classifier for training; the example classifier includes a feature extractor that shares parameters with step 2, and the parameters of the feature extractor and the multi-layer perceptron MLP are optimized through training, and the updated feature extractor parameters are synchronized with the feature extractor in step 2 to improve the quality of subsequent image block feature extraction; the above process can be iteratively executed to further improve the overall performance of the model.

2. The breast cancer whole-slice image classification method based on example search and multi-instance learning according to claim 1, characterized in that: The step 1 comprises: Step 1.1: Take a set of full-slice image datasets with slice-level labels As input data, represents the i-th full slice image, Y i ∈{0, 1} represents the classification label corresponding to the image, 0 represents normal and 1 represents breast cancer; Step 1.2: Segment the whole slice image to extract the foreground area; Step 1.3: Divide the foreground area into blocks.

3. The breast cancer whole-slice image classification method based on example search and multi-example learning according to claim 2, characterized in that: The step 2 includes: Step 2.1: Input all image blocks obtained in step 1 into the feature extractor, which consists of a CNN module and a Swin Transformer module. Step 2.2: Use the CNN module to input each image block Perform feature extraction and obtain a feature map of size h / 4×w / 4×96: in, Represents a convolution operator with a convolution kernel size of k×k; Step 2.3: Use the Swin Transformer module to further model the contextual relationship of the image block and enhance the global semantic information of the feature map, obtaining a deep feature map of size h / 32×w / 32×768: Step 2.4: Each image block obtained in step 2.3 Apply the global pooling operator F to the deep feature map global (·) Globally aggregate features to generate feature vectors in, The feature vectors of all image blocks in the same slice image are integrated and represented as a sequence, then the full slice image X i It can be expressed as:

4. The breast cancer whole-slice image classification method based on example search and multi-example learning according to claim 1, characterized in that: The step 2.3 includes: Step 2.3.1: Output the feature map of the CNN module Input Swin Transformer module; Swin Transformer module consists of two parts: patch merging layer and Swin Transformer block; Step 2.3.2: Use the patch merging layer to merge the input feature map Perform downsampling; Step 2.3.3: Downsample the feature map after the patch merging layer Input to the Swin Transformer block, each Swin Transformer block consists of two layers; In the first layer, window multi-head self-attention is used to capture local information; the input feature map The network is divided into multiple local windows of size M×M, and a multi-head self-attention mechanism is independently executed in each window. Specifically, the input features in each window are linearly transformed to generate query, key, and value vectors respectively. The attention weights are then obtained through dot product and softmax normalization. The features are then fused with relative position bias to form a window-level representation. A shift window attention mechanism is introduced in the second layer. Each layer module is updated through residual connection and multi-layer perceptron, and finally outputs a deep feature map with a size of h / 32×w / 32×768.

5. The breast cancer whole-slice image classification method based on example search and multi-example learning according to claim 4, characterized in that: The step 3 comprises: Step 3.1: The whole slice image X obtained in step 2 i The feature vector sequence of all image blocks Input into the MIL aggregator built based on the gated attention mechanism; Step 3.2: For the above full slice image X i The image block feature vector sequence Add position encoding vector; Step 3.3: Add the full slice image X after position encoding i The image block feature vector is input into the attention module, the attention score of each image block is calculated through the fully connected layer, and the Softmax function is used for normalization to obtain the attention weight corresponding to each image block; Step 3.4: According to the above attention weights, the full slice image X i The feature vector sequence of all image blocks Perform weighted summation to finally obtain the global feature matrix of the entire slice image: Step 3.5: transform the above global feature matrix Input classification layer f CLS (·) Perform classification prediction and obtain prediction results Use the cross entropy loss function to calculate the true label Y i and the predicted results The error between them is used to optimize the training of the MIL aggregator until the MIL aggregator converges.

6. The breast cancer whole-slice image classification method based on example search and multi-example learning according to claim 5, characterized in that: Step 4 includes: Step 4.1: After the MIL aggregator converges, obtain the full-slice image X i Each image block The eigenvector of And through the classification layer f CLS (·) The corresponding eigenvector Mapping to each image block About X i The predicted value of Step 4.2: Extract each image block obtained in step 3.4 from the MIL aggregator The attention weight The attention weight The corresponding image block obtained in step 4.1 The predicted value of Multiply and calculate the confidence of each image block This confidence measures the image block Belongs to class Y i possibility; Step 4.3: Based on the confidence of all the above image blocks For image blocks Sort in descending order to generate an ordered index set of image blocks Assume the current number of iterations is t, and select the top K confidence rankings according to the sorting results. t image patches and set their pseudo labels to the true labels Y of the full slice image i ∈{0, 1}, where 1 indicates breast cancer and 0 indicates no breast cancer. t The pseudo label of each image block is fixed to 0, and a set of pseudo label image blocks is obtained; K t is a positive integer that changes dynamically with the number of iterations t, defined as K t =min(num, (t+1)·10), where num is a preset hyperparameter that controls the growth rate; Step 4.4: Input the set of pseudo-label image patches constructed in step 4.3 into the example classifier for training. The example classifier consists of a feature extractor and a multi-layer perceptron. During training, the feature extractor and MLP parameters of the example classifier are optimized by calculating the loss between the pseudo-label and the output of the example classifier. After training converges, the feature extractor parameters of the example classifier are shared with the feature extractor in step 2 to achieve parameter sharing.

Images