A breast cancer in situ and invasive carcinoma segmentation method based on multi-modal WSI
By combining H&E and CK5/6 stained images, the multimodal WSI segmentation method solves the problem of insufficient segmentation accuracy under single resolution and modality, and achieves efficient and accurate segmentation of breast cancer in situ and invasive carcinoma.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTH CHINA UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-01
AI Technical Summary
Existing WSI tumor segmentation methods suffer from insufficient segmentation accuracy, low computational efficiency, and unreliability in distinguishing between carcinoma in situ and invasive carcinoma due to their reliance on single-resolution and single-modality images.
A multimodal WSI-based segmentation method for breast cancer in situ and invasive carcinoma was adopted. By combining H&E staining images and CK5/6 staining images, the final segmentation results were generated through multi-resolution fusion and cross-modal feature fusion.
It improves the accuracy and efficiency of breast cancer segmentation, reduces computational redundancy, and enhances the accuracy of identifying the boundaries between carcinoma in situ and invasive carcinoma.
Smart Images
Figure CN121962598A_ABST
Abstract
Description
A Multimodal WSI-based method for segmenting breast cancer in situ and invasive carcinoma Technical Field
[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a method for segmenting breast cancer in situ and invasive carcinoma based on multimodal WSI. Background Technology
[0002] In the breast cancer diagnosis and treatment system, differentiating between carcinoma in situ and invasive carcinoma is a crucial aspect of pathological classification. With the rapid development of digital case technology, whole-slice images provide new technical support for breast cancer classification. Most existing whole-slice image (WSI) tumor segmentation methods typically process WSI images at a single resolution, which has significant limitations. Low-resolution images cannot fully capture the subtle features such as cell size and shape in invasive breast cancer, leading to decreased segmentation accuracy. While high-resolution images provide more detailed information, they are computationally intensive, and complex local textures can easily introduce artifacts, affecting the overall accuracy of segmentation.
[0003] To address this issue, some methods have attempted to combine information from different resolutions, mainly falling into two categories. One category performs segmentation at high resolution, supplemented by low-resolution images to provide global information and compensate for the limitations of a single scale. The other category employs a concentric magnification strategy to supplement detailed information in the target region at low resolution. However, this method magnifies the entire region, failing to distinguish key areas, introducing a large amount of redundant information, and increasing computational costs. Most WSI tumor segmentation tasks rely primarily on H&E-stained sections. However, H&E-stained images have limitations in distinguishing cells and tissue structures. In tumor segmentation tasks, determining whether invasion has occurred often requires observing the absence of myoepithelial cells. Since myoepithelial cells and smooth muscle cells exhibit similar characteristics, relying solely on H&E-stained images for judgment is prone to error, which necessitates physicians typically combining multiple stained sections for comprehensive analysis. Summary of the Invention
[0004] To address the aforementioned shortcomings in existing technologies, the present invention provides a method for segmenting in situ and invasive breast cancer based on multimodal WSI, which solves the problems of insufficient segmentation accuracy, low computational efficiency, and insufficient reliability in distinguishing between in situ and invasive cancer caused by existing WSI tumor segmentation technologies that rely on single-resolution and single-modal images for breast cancer analysis.
[0005] To achieve the above objectives, the technical solution adopted in this invention is as follows: a method for segmenting in situ and invasive breast cancer based on multimodal WSI, comprising the following steps: acquiring H&E stained images of breast cancer patients and spatially aligned CK5 / 6 stained whole-section images, and performing preprocessing after labeling in situ and invasive cancer to obtain H&E stained global region slices, H&E stained key region slices, CK5 / 6 stained global slices, and edge masks corresponding to CK5 / 6 stained global slices; inputting the H&E stained global region slices into the target branch of the H&E semantic segmentation module, and inputting the H&E stained key region slices into the detail branch of the H&E semantic segmentation module, respectively. After feature extraction, the data is input into the multi-resolution fusion module to obtain the fused H&E multi-resolution semantic features. CK5 / 6 stained global slices are input into the CK5 / 6 semantic segmentation module, and the CK5 / 6 images corresponding to the edge masks are input into the edge detection module. After feature extraction, these are input into the edge feature fusion module to obtain CK5 / 6 semantic edge multi-scale features. The fused H&E multi-resolution semantic features and CK5 / 6 semantic edge multi-scale features are input into the multi-scale cross-modal feature fusion module for processing to obtain the final fused feature map. Based on the final fused feature map, a segmentation result is generated, which is used to distinguish between breast cancer in situ and invasive cancer regions.
[0006] Further: The preprocessing of the H&E staining images specifically includes: acquiring H&E staining images of breast cancer patients and labeling the in situ carcinoma region and invasive carcinoma region to obtain labeled H&E staining images; normalizing the labeled H&E staining images and slicing them using a sliding window of size 512×512 and stride 128 at a spatial resolution of 1.00 MPP to obtain H&E staining global region slices; inputting the H&E staining global region slices into an attention-guided UNet network for semantic segmentation, outputting a heatmap, outputting a saliency heatmap corresponding to the H&E staining global region, and converting it to a grayscale image; based on the grayscale image, searching for the region with the largest response value using a sliding window of size 128×128, recording the spatial coordinates of the region in the H&E staining global region slice to obtain the region of interest; mapping the region of interest to the H&E staining image based on the spatial coordinates of the region of interest in the H&E staining global region slice, and slicing it at a spatial resolution of 0.25 MPP. Under MPP conditions, the corresponding high-resolution regions were extracted to obtain H&E staining key region sections.
[0007] The further beneficial effects mentioned above are as follows: This invention addresses the problems of high computational load and loss of detail in low-resolution image processing when processing high-resolution images. By using heatmap-based region localization, high-resolution sampling is performed on the key regions with the most diagnostic value. This allows for the preservation of global context while focusing on key detail features, achieving a balance between computational efficiency and segmentation accuracy.
[0008] Further: The preprocessing of the CK5 / 6 staining images specifically includes: acquiring CK5 / 6 staining images of breast cancer patients, and annotating the CK5 / 6 staining images according to the annotations of the in situ carcinoma region and invasive carcinoma region in the corresponding H&E staining images to obtain annotated CK5 / 6 staining images; normalizing the annotated CK5 / 6 staining images, and slicing them using a sliding window with a size of 512×512 and a step size of 128 under the condition of spatial resolution of 1.00 MPP to obtain initial CK5 / 6 staining global slices; aligning the initial CK5 / 6 staining global slices and the H&E staining global region slices one-to-one in spatial coordinates to obtain CK5 / 6 staining global slices; converting the CK5 / 6 staining global slices into grayscale images through image processing; and extracting the tumor edge region according to the grayscale image and the corresponding annotation information to obtain the edge mask corresponding to the CK5 / 6 staining global slices.
[0009] The further beneficial effects mentioned above are as follows: by aligning the global slices of CK5 / 6 staining with the global slices of H&E staining and performing edge extraction, the biological boundary information of CK5 / 6 staining is transformed into a structured auxiliary signal, which makes up for the ambiguity and uncertainty of H&E images in identifying the infiltration front and provides reliable and complementary semantic guidance for subsequent cross-modal fusion.
[0010] Further: obtaining the fused H&E multi-resolution semantic features specifically includes: extracting detailed information from the target branch of the H&E semantic segmentation module based on H&E-stained global region slices to obtain target multi-scale features; extracting high-resolution fine-grained information from the detailed branch of the H&E semantic segmentation module based on H&E-stained key region slices to obtain detailed multi-scale features; unifying the channel dimensions of the target multi-scale features and detailed multi-scale features using a multi-resolution fusion module, and fusing the detailed multi-scale features with the corresponding position of the target multi-scale features based on the spatial position of the key region corresponding to the detailed multi-scale features in the global region to obtain the fused H&E multi-resolution semantic features.
[0011] The further beneficial effects mentioned above are as follows: by unifying the alignment of channel dimensions, and corresponding and fusing the spatial positions of target multi-scale features and detail multi-scale features, multi-resolution fusion is achieved, which solves the problems of computational redundancy and feature interference caused by global magnification in traditional methods; this invention can fuse high-resolution detail features into the corresponding key positions in the global feature map, thereby reducing computational costs while enhancing the feature enhancement of minute details and improving the ability to distinguish between minute differences between carcinoma in situ and invasive carcinoma.
[0012] Furthermore: the resolution changes of the encoder and decoder for the target branch and the detail branch are respectively... and ,satisfy:
[0013]
[0014]
[0015] in, For the resolution change of the target branch, For the resolution changes of the detailed branches. The depth of downsampling in the target branch. The depth of downsampling in the detail branch. Enter the resolution of the patch in the target branch. Enter the resolution of the patch in the detail branch.
[0016] Furthermore, the expression for the fused H&E multi-resolution semantic features is as follows:
[0017]
[0018]
[0019] in, For the target branch, the target multi-scale features, The fused H&E multi-resolution semantic features The key region is the top-left y-coordinate of the rectangular region within the global region. This represents the bottom right y-coordinate of the rectangular region within the global region representing the critical area. The x-coordinate of the top-left corner of the rectangular region within the global region represents the critical area. The x-coordinate of the bottom right corner of the rectangular region within the global region represents the key area. For detailed multi-scale features, For channel attention mechanism, It is the Sigmoid activation function. It is a fully connected layer. It is the ReLU activation function. The average pooling is applied to the entire series.
[0020] Further: obtaining the CK5 / 6 semantic edge multi-scale features specifically includes: processing the CK5 / 6 semantic segmentation module based on the CK5 / 6 stained global slice to obtain semantic multi-scale features; learning semantic information and edge features through the edge detection module based on the edge mask to obtain edge multi-scale features; and performing cross-modal fusion based on the semantic multi-scale features and edge multi-scale features through the edge feature fusion module to obtain the CK5 / 6 semantic edge multi-scale features.
[0021] The further beneficial effects mentioned above are as follows: by fusing CK5 / 6 staining global slices and edge features across modalities, the generated CK5 / 6 semantic edge multi-scale features can represent semantic information and image boundaries, providing clear guidance for the segmentation of H&E staining images and improving the accuracy and precision of identifying the boundaries of carcinoma in situ and invasive carcinoma.
[0022] Furthermore, the expression for the CK5 / 6 semantic edge multi-scale feature is as follows:
[0023]
[0024]
[0025]
[0026] in, For the CK5 / 6 semantic segmentation branch module, the decoder in the first... The output feature map of the layer, For the CK5 / 6 semantic segmentation branch module, the decoder in the first... The output feature map of the layer, This is a bilinear upsampling operation. For the edge feature fusion module Features output by the layer encoder The final output features of the edge feature fusion module are... For depthwise separable convolution, The feature map after gating attention. This is the spliced feature map. It is the sigmoid activation function. For convolution operations, for, The enhanced semantic features of the CK5 / 6 semantic segmentation branch module after calibration using the channel attention mechanism. This refers to the enhanced edge features of the edge detection branch module after calibration using the channel attention mechanism.
[0027] Furthermore, the expression for the final fused feature map is as follows:
[0028]
[0029]
[0030]
[0031]
[0032] in, For the multi-scale cross-modal feature fusion module in the first The output feature map of the layer, For the multi-scale cross-modal feature fusion module in the first The output feature map of the layer, This is an intermediate spliced feature map. It is the sigmoid activation function. It is a fully connected layer. It is the ReLU activation function. For global average pooling, For splicing operations, The decoder for the H&E semantic segmentation module in the first... Feature map of the layer The decoder for the edge detection module in the first... Feature map of the layer For axial cross attention The output of the layer, The feature map is obtained by calculating the axial cross-attention in the horizontal direction. The feature map is calculated for the vertical axial cross attention. The softmax activation function is used. This is the projection weight matrix of the query vector in the horizontal direction. This is the projection weight matrix of the query vector in the vertical direction. This is the projection weight matrix of the horizontal key vector. This is the projection weight matrix of the vertical direction key vector. The projection weight matrix is the value vector in the horizontal direction. The projection weight matrix is the value vector in the vertical direction. For transpose operation, is the dimension of the key vector.
[0033] The beneficial effects of this invention are as follows: This invention solves the problems of single-resolution limitations, insufficient utilization of multimodal information, and computational redundancy in existing WSI segmentation methods. By fusing multimodal information from H&E staining and CK5 / 6 staining, and combining a 1.00MPP global field of view with a 0.25MPP key region multi-resolution strategy, key regions are identified using heatmaps. This preserves global contextual information while fully mining fine-grained features at high resolution, reducing noise interference from irrelevant regions and redundant computation. Simultaneously, through multi-resolution fusion, edge feature fusion, and multi-scale cross-modal feature fusion modules containing spatial attention and axial cross-attention, semantic information and edge features of images can be mined, reducing the error of distinguishing breast cancer solely based on H&E staining images, and improving the accuracy of tumor segmentation and edge accuracy. Attached Figure Description
[0034] Figure 1 is a schematic diagram of the segmentation method for in situ and invasive breast cancer based on multimodal WSI; Figure 2 is a schematic diagram of the segmentation model for in situ and invasive breast cancer based on multimodal WSI. Detailed Implementation
[0035] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0036] Example 1, as shown in Figure 1, illustrates a method for segmenting in situ and invasive breast cancer based on multimodal WSI. The method includes the following steps: acquiring H&E stained images of breast cancer patients and spatially aligned CK5 / 6 stained whole-section images; after labeling in situ and invasive cancer, preprocessing is performed to obtain H&E stained global region slices, H&E stained key region slices, CK5 / 6 stained global slices, and edge masks corresponding to the CK5 / 6 stained global slices; inputting the H&E stained global region slices into the target branch of the H&E semantic segmentation module, and inputting the H&E stained key region slices into the detail branch of the H&E semantic segmentation module, respectively, for feature extraction. The images are then input into a multi-resolution fusion module to obtain fused H&E multi-resolution semantic features. CK5 / 6 stained global slices are input into a CK5 / 6 semantic segmentation module, and the CK5 / 6 images corresponding to the edge masks are input into an edge detection module. After feature extraction, these are input into an edge feature fusion module to obtain CK5 / 6 semantic edge multi-scale features. The fused H&E multi-resolution semantic features and CK5 / 6 semantic edge multi-scale features are then input into a multi-scale cross-modal feature fusion module for processing to obtain a final fused feature map. Based on the final fused feature map, a segmentation result is generated, which is used to distinguish between breast cancer in situ and invasive cancer regions.
[0037] Figure 2 shows a schematic diagram of a segmentation model for in situ and invasive breast cancer based on multimodal WSI. It illustrates the structure of the segmentation model, which consists of an H&E semantic segmentation module (gray area, H&E Semantic Segmentation Branch), a CK5 / 6 edge detection module (blue area, CK5 / 6 Edge Detection Branch), and a multi-scale feature branch (green area). EFFM is used to fuse CK5 / 6 semantic segmentation features and edge segmentation features. EFFM is used to fuse features from the H&E semantic segmentation module and the CK5 / 6 edge detection module, ultimately outputting the H&E segmentation result. The left side of the H&E semantic segmentation module represents its target branch processing 10×magnification H&E stained slices (i.e., the 10×target branch in the figure) and its detail branch processing 40×magnification H&E stained slices (i.e., the 40×target branch in the figure). The left side of the CK5 / 6 edge detection module represents its edge detection branch and semantic segmentation branch, while the right side displays the output results of each branch.
[0038] To address the shortcomings of existing WSI tumor segmentation techniques, such as insufficient accuracy or computational redundancy in processing single-resolution images, lack of attention to detail during multi-resolution image fusion due to failure to differentiate key regions, and large segmentation errors due to reliance solely on H&E staining images, this invention acquires dual-modal staining images (H&E and CK5 / 6 staining) and preprocesses them to obtain global region slices, key region slices, and edge masks, enabling the application of multi-modal staining images and improving segmentation accuracy. The H&E semantic segmentation module of this invention generates global region slices and key region slices with different resolutions through a dual-branch design, and obtains the fused H&E multi-resolution semantic features through a multi-resolution fusion module, thus considering both global contextual information and ensuring local detail features. This invention extracts edge features through a CK5 / 6 semantic segmentation module, an edge detection module, and an edge feature fusion module. The system integrates and fuses the CK5 / 6 semantic edge multi-scale features with the H&E multi-resolution semantic features and the CK5 / 6 semantic edge multi-scale features through a multi-scale cross-modal feature fusion module containing spatial attention and axial cross attention. This process ultimately achieves accurate segmentation of carcinoma in situ and invasive carcinoma. The CK5 / 6 stained images assist the H&E stained images in tumor segmentation. The dual-modal images alleviate the problem of insufficient segmentation accuracy of single stained images, improve the robustness and reliability of segmentation, and facilitate more accurate differentiation between carcinoma in situ and invasive carcinoma regions of breast cancer, thereby improving the segmentation efficiency and the reliability of differentiation of breast cancer images.
[0039] In one embodiment of the present invention, most existing WSI tumor segmentation methods typically process WSI into single-resolution breast cancer images, which has significant limitations: low-resolution breast cancer images cannot fully capture subtle features such as cell size and shape in invasive breast cancer, leading to decreased segmentation accuracy; while high-resolution breast cancer images provide more detailed information, they are computationally intensive, and complex local textures can easily introduce artifacts, affecting the overall segmentation accuracy. Most WSI tumor segmentation tasks rely primarily on H&E-stained slices, but H&E-stained images have limitations in distinguishing between cells and tissue structures; in tumor segmentation tasks, determining whether invasion has occurred usually requires observing whether myoepithelial cells are missing. Since myoepithelial cells and smooth muscle cells exhibit similar characteristics, relying solely on H&E-stained images for judgment is prone to errors. Therefore, the present invention introduces CK5 / 6-stained images, which can specifically mark the edges of invasive cancer, providing clearer boundary information. This not only assists in the precise localization of breast cancer tumors but also significantly improves the segmentation accuracy on H&E-stained images. Therefore, this invention acquires H&E staining images and CK5 / 6 staining images of breast cancer patients, and performs preprocessing after labeling carcinoma in situ and invasive carcinoma to obtain H&E staining global region slices, H&E staining key region slices, CK5 / 6 staining global slices, and edge masks corresponding to CK5 / 6 staining global slices, respectively. In a specific embodiment of this invention, H&E staining images and CK5 / 6 staining images of breast cancer patients can be obtained from breast cancer datasets such as the BCSS public dataset.The preprocessing of H&E-stained images specifically includes: acquiring H&E-stained images of breast cancer patients and labeling the in situ carcinoma and invasive carcinoma regions to obtain labeled H&E-stained images; normalizing the labeled H&E-stained images and slicing them using a sliding window of size 512×512 and a stride of 128 at a spatial resolution of 1.00 MPP (10x magnification) to obtain H&E-stained global region slices; inputting the H&E-stained global region slices into an attention-guided UNet network for semantic segmentation, outputting a heatmap, outputting a saliency heatmap corresponding to the H&E-stained global region, and converting it to a grayscale image; based on the grayscale image, searching for the region with the largest response value using a sliding window of size 128×128, recording the spatial coordinates of the region in the H&E-stained global region slice to obtain the region of interest; mapping the region of interest to the H&E-stained image based on the spatial coordinates of the region of interest in the H&E-stained global region slice, and then processing the image at a spatial resolution of 0.25 MPP. Under MPP (40x magnification) conditions, the corresponding high-resolution regions are extracted to obtain H&E staining key region slices. Preprocessing of CK5 / 6 staining images specifically includes: acquiring CK5 / 6 staining images of breast cancer patients, and annotating the CK5 / 6 staining images based on the annotations of in situ carcinoma and invasive carcinoma regions in the corresponding H&E staining images to obtain annotated CK5 / 6 staining images; normalizing the annotated CK5 / 6 staining images, and slicing them using a sliding window of size 512×512 and a step size of 128 under a spatial resolution of 1.00 MPP to obtain initial CK5 / 6 staining global slices; aligning the initial CK5 / 6 staining global slices and the H&E staining global region slices one-to-one in spatial coordinates to obtain CK5 / 6 staining global slices; converting the CK5 / 6 staining global slices into grayscale images through image processing; and extracting tumor edge regions based on the grayscale images and their corresponding annotation information to obtain the edge mask corresponding to the CK5 / 6 staining global slices.
[0040] In one embodiment of the present invention, high-resolution images, i.e., H&E-stained global region slices, can preserve subtle features such as cell nuclear morphology and tissue structure for distinguishing between carcinoma in situ and invasive carcinoma, but suffer from high computational cost and slow processing efficiency. Low-resolution images, i.e., H&E-stained key region slices, can obtain results quickly, but lose some feature information, leading to a decrease in segmentation accuracy. Existing technologies also employ concentric magnification strategies, but this involves unnecessary magnification, introducing irrelevant noise, and reducing efficiency and segmentation accuracy. To address this, the present invention designs a dual-branch structure for the H&E semantic segmentation module, including a target branch and a detail branch, and combines them with a resolution fusion module to enhance the detail features in the global features. The present invention inputs the H&E-stained global region slices into the target of the H&E semantic segmentation module. The H&E semantic segmentation module takes slices of the H&E-stained key regions and inputs them into the detail branch. After feature extraction, these slices are input into the multi-resolution fusion module to obtain the fused H&E multi-resolution semantic features. Specifically, this includes: extracting detailed information from the target branch of the H&E semantic segmentation module based on slices of the global H&E-stained region to obtain target multi-scale features; extracting high-resolution fine-grained information from the detail branch of the H&E semantic segmentation module based on slices of the H&E-stained key regions to obtain detailed multi-scale features; and using the multi-resolution fusion module to unify the channel dimensions of the target multi-scale features and the detailed multi-scale features. Based on the spatial position of the key regions corresponding to the detailed multi-scale features in the global region, the detailed multi-scale features are fused to the corresponding positions of the target multi-scale features to obtain the fused H&E multi-resolution semantic features.
[0041] In the H&E semantic segmentation branch, this invention employs a multi-resolution input strategy to improve model performance. Specifically, firstly, H&E-colored global region slices magnified 10 times are trained using a VGG-UNet network to generate preliminary segmentation results. Subsequently, Grad-CAM is used to generate a saliency heatmap. , where each pixel value This indicates the importance of the corresponding region to the model's decision; to avoid the problem of traditional concentric magnification methods potentially amplifying irrelevant regions, a sliding window (128×128 pixels) is used to search for saliency peak regions. .
[0042] The fusion feature of the target branch is The fusion feature of the shape of the detailed branches is ,in Indicates the size of the feature map. This indicates the number of channels, all of which are three channels. It is a global average pooling operation. It is a fully connected layer, and sigmoid and ReLU are activation functions.
[0043] To fuse H&E features at different resolutions, this invention proposes a multi-resolution fusion module; since the target branch and detail branch reside at different decoder depths, the target multi-scale features of the target branch are... The detailed multi-scale features of the shape of the detailed branches are To unify the channel dimensions, the Squeeze-and-Excitation (SE Layer) module is used to adjust the number of channels. Considering that key regions are determined based on the resolution of the original input features, during fusion, due to... and At different scales, the coordinates of key areas need to be scaled accordingly to make them suitable. At the resolution, the scaled coordinates of the key region are Subsequently, regarding Perform cropping and compare with the reduced number of channels. Integration, to obtain Its expression is as follows:
[0044] Next, the fused features are smoothed:
[0045] Last Update Complete feature fusion:
[0046] Features after fusion Continuous upsampling is performed, with the decoder of the target branch progressively upsampling to generate a 10× resolution segmentation result. The detail branch continues upsampling to generate a 40× resolution segmentation result. .
[0047] To fully learn the semantic information of images at different resolutions, the target branch and detail branch are trained independently during the encoding phase; and to ensure that the target branch and detail branch are aligned at the same resolution, an appropriate fusion method needs to be selected; during the encoder downsampling process, the resolution changes as follows: ,in This indicates the downsampling depth of the encoder. This is the input patch resolution (unit: μm / px); during the decoding process, the resolution changes to... ,in Indicates encoder depth, This indicates the decoder depth. The resolution changes of the encoder and decoder for the target branch and detail branch are respectively... and To ensure that the target branch and the detail branch are fused at the same resolution, the feature maps of the two branches need to satisfy the following:
[0048]
[0049]
[0050] in, For the resolution change of the target branch, For the resolution changes of the detailed branches. The depth of downsampling in the target branch. The depth of downsampling in the detail branch. Enter the resolution of the patch in the target branch. Enter the resolution of the patch in the detail branch; It is the relative height of the target branch and the detail branch. When the target branch and the detail branch have the same resolution, and It should meet the following requirements:
[0051] In one embodiment of the present invention, CK5 / 6 stained images are added and combined with H&E staining to form a multimodal image. CK5 / 6 staining can specifically mark the edges of invasive cancer cells, providing clearer boundary information. This edge information not only assists in the precise localization of the tumor but also significantly improves the segmentation accuracy on the H&E stained image. Therefore, the present invention inputs the CK5 / 6 stained global slices into the CK5 / 6 semantic segmentation module and the edge mask into the edge detection module. After processing, they are input into the edge feature fusion module to obtain CK5 / 6 semantic edge multi-scale features. Specifically, this includes: processing the CK5 / 6 stained global slices through the CK5 / 6 semantic segmentation module to obtain semantic multi-scale features; learning semantic information and edge features through the edge detection module based on the edge mask to obtain edge multi-scale features; and performing cross-modal fusion based on the semantic multi-scale features and edge multi-scale features through the edge feature fusion module to obtain CK5 / 6 semantic edge multi-scale features.
[0052] In a specific embodiment of the present invention, to fully exploit the semantic information and edge characteristics of CK5 / 6 stained sections, the present invention proposes an edge feature fusion module, which includes a channel attention mechanism, a gated attention mechanism, and a depthwise separable convolution. First, the four encoded features of the semantic segmentation branch and the edge detection branch... and Important feature channels are selected separately using a channel attention mechanism to generate enhanced features. and Then the two feature maps and By concatenating the data along the channel dimension, a new feature representation is obtained. The concatenated features are then fed into an attention gate for fusion. The gating attention mechanism can dynamically adjust the spatial weights of the feature maps, focusing on key regions (such as the edge of carcinoma in situ) while suppressing background noise.
[0053] Finally, depthwise separable convolutions are used to generate the final fused features, which significantly reduces the number of parameters and computational complexity while maintaining the feature expressiveness.
[0054] The fusion module in the encoder's four fusion features These features are recursively fused layer by layer with the decoder across the CK5 / 6 semantic segmentation branch:
[0055] in This is a bilinear upsampling operation. Through multi-level fusion, the model can simultaneously utilize shallow detail information and deep semantic information, enhancing its edge detection capabilities. Finally, the semantic segmentation branch generates CK5 / 6 segmentation results. The edge detection branch generates the grayscale image CK5 / 6 segmentation result. .
[0056] In one embodiment of the present invention, the fused H&E multi-resolution semantic features and CK5 / 6 semantic edge multi-scale features are input into a multi-scale cross-modal feature fusion module for processing to obtain the final fused feature map. The multi-scale cross-modal feature fusion module includes a spatial attention mechanism, an axial cross-attention mechanism, and a feature splicing channel attention mechanism. The multi-scale cross-modal feature fusion module aims to integrate the semantic segmentation information of H&E staining slices with the edge detection information of CK5 / 6 staining slices. By guiding the H&E branch with the semantic-edge information of CK5 / 6 staining slices, a more accurate semantic segmentation result can be achieved. The present invention designs a multi-scale cross-modal feature fusion module with a hierarchical progressive structure, utilizing feature maps of four different scales from the H&E semantic segmentation module. and feature maps of four corresponding scales from the CK5 / 6 edge detection module. Through integration with the upper layer Iterative fusion is used to achieve layer-by-layer fusion of multi-scale features.
[0057] To effectively fuse the two sets of features mentioned above, the multi-scale feature fusion module of this invention includes a spatial attention mechanism, an axial cross attention mechanism, and a feature concatenation channel attention mechanism. First, the spatial attention mechanism is used to dynamically adjust the input feature map. and The attention-weighted feature map is obtained. and
[0058] Next, an axial cross-attention mechanism is used to capture the interaction relationships between different modal features, especially long-distance dependencies. This process includes attention calculations in both the horizontal and vertical directions.
[0059]
[0060]
[0061] Among them, the weighted features are obtained by calculating horizontal attention. and weighted features in the vertical direction Finally, the two are added together to obtain the axial cross-attention output. Finally, the feature maps processed by the spatial attention mechanism are... and Axial cross attention output and the fusion features obtained from the previous layer The features are concatenated and then processed through a channel attention mechanism to obtain the final fused feature map. :
[0062]
[0063] superscript Both represent layers; in the expression, the same layer is used. The concept is marked with a superscript The final text indicates It is the result of axial cross-attention output, and is a holistic concept.
[0064] The final expression for the fused feature map is as follows:
[0065]
[0066]
[0067]
[0068]
[0069] Among them, among them, For the multi-scale cross-modal feature fusion module in the first The output feature map of the layer, For the multi-scale cross-modal feature fusion module in the first The output feature map of the layer, This is an intermediate spliced feature map. It is the sigmoid activation function. It is a fully connected layer. It is the ReLU activation function. For global average pooling, For splicing operations, The decoder for the H&E semantic segmentation module in the first... Feature map of the layer The decoder for the edge detection module in the first... Feature map of the layer For axial cross attention The output of the layer, The feature map is obtained by calculating the axial cross-attention in the horizontal direction. The feature map is calculated for the vertical axial cross attention. The softmax activation function is used. This is the projection weight matrix of the query vector in the horizontal direction. This is the projection weight matrix of the query vector in the vertical direction. This is the projection weight matrix of the horizontal key vector. This is the projection weight matrix of the vertical direction key vector. The projection weight matrix is the value vector in the horizontal direction. The projection weight matrix is the value vector in the vertical direction. For transpose operation, is the dimension of the key vector.
[0070] In one embodiment of the present invention, a segmentation result is generated based on the final fused feature map, the segmentation result being used to distinguish between breast cancer in situ regions and invasive cancer regions.
[0071] The goal of the H&E semantic segmentation branch of this invention is to generate semantic segmentation results by fusing H&E image information from multiple resolutions. This branch consists of two encoder-decoder architecture branches: a target branch and a detail branch. (1) The target branch processes 10×magnification H&E stained slices, which can cover a large area while retaining sufficient detail information, thus balancing global vision and local features. (2) The detail branch processes 40×magnification H&E stained slices, providing high-resolution fine-grained information by magnifying key regions in the 10×magnification slices. By combining information from these two resolutions, the model can achieve an effective balance between global vision and local details, thereby improving segmentation performance.
[0072] The CK5 / 6 edge detection branch of this invention aims to learn semantic information and edge features from CK5 / 6 images aligned with H&E. Considering the characteristics of CK5 / 6, this branch also consists of two encoder-decoder architecture branches, including a semantic segmentation branch and an edge detection branch: (1) The semantic segmentation branch is used to segment CK5 / 6 stained slices, providing cross-modal semantic information for aligned H&E by learning histological features. (2) The edge detection branch aims to extract the edge structure features of CK5 / 6 slices converted to grayscale, providing complementary edge guidance information for H&E segmentation. Thus, complementary edge guidance and semantic information are provided for H&E segmentation.
[0073] The goal of the multi-scale fusion branch is to enhance the segmentation performance of H&E images by fusing multi-dimensional features from CK5 / 6 stained slices. By fusing multi-resolution semantic information obtained from the H&E semantic segmentation branch and edge features obtained from the CK5 / 6 edge detection branch, the model guides the acquisition of 10×magnification H&E slice segmentation results. The model ultimately outputs 10×magnification and 40×magnification H&E stained slice segmentation results, as well as semantic and edge segmentation results for CK5 / 6 stained slices.
[0074] The beneficial effects of this invention are as follows: This invention solves the problems of single-resolution limitations, insufficient utilization of multimodal information, and computational redundancy in existing WSI segmentation methods. By fusing multimodal information from H&E staining and CK5 / 6 staining, and combining a 1.00MPP global field of view with a 0.25MPP key region multi-resolution strategy, key regions are identified using heatmaps. This preserves global contextual information while fully mining fine-grained features at high resolution, reducing noise interference from irrelevant regions and redundant computation. Simultaneously, through multi-resolution fusion, edge feature fusion, and multi-scale cross-modal feature fusion modules containing spatial attention and axial cross-attention, semantic information and edge features of images can be mined, reducing the error of distinguishing breast cancer solely based on H&E staining images, and improving the accuracy of tumor segmentation and edge accuracy.
Claims
1. A method for segmenting breast cancer in situ and invasive carcinoma based on multimodal WSI, characterized in that, The process includes the following steps: acquiring H&E stained images of breast cancer patients and spatially aligned CK5 / 6 stained whole-section images, and performing preprocessing after labeling carcinoma in situ and invasive carcinoma to obtain H&E stained global region slices, H&E stained key region slices, CK5 / 6 stained global slices, and edge masks corresponding to the CK5 / 6 stained global slices; inputting the H&E stained global region slices into the target branch of the H&E semantic segmentation module, and inputting the H&E stained key region slices into the detail branch of the H&E semantic segmentation module, performing feature extraction on each, and then inputting them into the multi-resolution fusion module to obtain the fused H&E multi-resolution semantic features; inputting the CK5 / 6 stained global slices into the CK5 / 6 semantic segmentation module, and inputting the CK5 / 6 images corresponding to the edge masks into the edge detection module, performing feature extraction on each, and then inputting them into the edge feature fusion module to obtain CK5 / 6 semantic edge multi-scale features; The fused H&E multi-resolution semantic features and CK5 / 6 semantic edge multi-scale features are input into the multi-scale cross-modal feature fusion module for processing to obtain the final fused feature map. Based on the final fused feature map, a segmentation result is generated, which is used to distinguish between the in situ carcinoma region and the invasive carcinoma region of breast cancer.
2. The method for separating in situ and invasive breast cancer according to claim 1, characterized in that, The preprocessing of the H&E staining images specifically includes: acquiring H&E staining images of breast cancer patients and labeling the in situ carcinoma region and invasive carcinoma region to obtain labeled H&E staining images; normalizing the labeled H&E staining images and slicing them using a sliding window of size 512×512 and stride 128 at a spatial resolution of 1.00 MPP to obtain H&E staining global region slices; inputting the H&E staining global region slices into an attention-guided UNet network for semantic segmentation, outputting a heatmap, outputting a saliency heatmap corresponding to the H&E staining global region, and converting it to a grayscale image; based on the grayscale image, searching for the region with the largest response value using a sliding window of size 128×128, recording the spatial coordinates of the region in the H&E staining global region slice to obtain the region of interest; based on the spatial coordinates of the region of interest in the H&E staining global region slice, mapping the region of interest to the H&E staining image, and extracting the corresponding high-resolution region at a spatial resolution of 0.25 MPP to obtain H&E staining key region slices.
3. The method for separating in situ and invasive breast cancer according to claim 2, characterized in that, The preprocessing of the CK5 / 6 staining images specifically includes: acquiring CK5 / 6 staining images of breast cancer patients, and annotating the CK5 / 6 staining images according to the annotations of the in situ carcinoma region and invasive carcinoma region in the corresponding H&E staining images to obtain annotated CK5 / 6 staining images; normalizing the annotated CK5 / 6 staining images, and slicing them using a sliding window with a size of 512×512 and a step size of 128 under the condition of spatial resolution of 1.00 MPP to obtain initial CK5 / 6 staining global slices; aligning the initial CK5 / 6 staining global slices and the H&E staining global region slices one-to-one in spatial coordinates to obtain CK5 / 6 staining global slices; converting the CK5 / 6 staining global slices into grayscale images through image processing; and extracting the tumor edge region according to the grayscale image and the corresponding annotation information to obtain the edge mask corresponding to the CK5 / 6 staining global slices.
4. The method for separating in situ and invasive breast cancer according to claim 1, characterized in that, The process of obtaining the fused H&E multi-resolution semantic features specifically includes: extracting detailed information from the target branch of the H&E semantic segmentation module based on H&E-stained global region slices to obtain target multi-scale features; extracting high-resolution fine-grained information from the detailed branch of the H&E semantic segmentation module based on H&E-stained key region slices to obtain detailed multi-scale features; unifying the channel dimensions of the target multi-scale features and detailed multi-scale features using a multi-resolution fusion module; and fusing the detailed multi-scale features with the corresponding position of the target multi-scale features based on the spatial position of the key region corresponding to the detailed multi-scale features in the global region to obtain the fused H&E multi-resolution semantic features.
5. The method for separating breast cancer in situ from invasive carcinoma according to claim 4, characterized in that, The resolution changes of the encoder and decoder for the target branch and the detail branch are respectively... and ,satisfy: in, For the resolution change of the target branch, For the resolution changes of the detailed branches. The depth of downsampling in the target branch. The depth of downsampling in the detail branch. Enter the resolution of the patch in the target branch. Enter the resolution of the patch in the detail branch.
6. The method for separating in situ and invasive breast cancer according to claim 4, characterized in that, The expression for the fused H&E multi-resolution semantic features is as follows: in, For the target branch, the target multi-scale features, The fused H&E multi-resolution semantic features The key region is the top-left y-coordinate of the rectangular region within the global region. This represents the bottom right y-coordinate of the rectangular region within the global region representing the critical area. The x-coordinate of the top-left corner of the rectangular region within the global region represents the critical area. The x-coordinate of the bottom right corner of the rectangular region within the global region represents the key area. For detailed multi-scale features, For channel attention mechanism, It is the Sigmoid activation function. It is a fully connected layer. It is the ReLU activation function. This is global average pooling.
7. The method for separating in situ and invasive breast cancer according to claim 1, characterized in that, The process of obtaining CK5 / 6 semantic edge multi-scale features specifically includes: processing global slices stained with CK5 / 6 using the CK5 / 6 semantic segmentation module to obtain semantic multi-scale features; learning semantic information and edge features using the edge detection module based on the edge mask to obtain edge multi-scale features; and performing cross-modal fusion using the edge feature fusion module based on the semantic multi-scale features and edge multi-scale features to obtain CK5 / 6 semantic edge multi-scale features.
8. The method for separating in situ and invasive breast cancer according to claim 7, characterized in that, The expression for the CK5 / 6 semantic edge multi-scale feature is as follows: in, For the CK5 / 6 semantic segmentation branch module, the decoder in the first... The output feature map of the layer, For the CK5 / 6 semantic segmentation branch module, the decoder in the first... The output feature map of the layer, This is a bilinear upsampling operation. For the edge feature fusion module Features output by the layer encoder The final output features of the edge feature fusion module are... For depthwise separable convolution, The feature map after gating attention. This is the spliced feature map. It is the sigmoid activation function. For convolution operations, for, The enhanced semantic features of the CK5 / 6 semantic segmentation branch module after calibration using the channel attention mechanism. This refers to the enhanced edge features of the edge detection branch module after calibration using the channel attention mechanism.
9. The method for separating in situ and invasive breast cancer according to claim 1, characterized in that, The expression for the final fused feature map is as follows: in, For the multi-scale cross-modal feature fusion module in the first The output feature map of the layer, For the multi-scale cross-modal feature fusion module in the first The output feature map of the layer, This is an intermediate spliced feature map. It is the sigmoid activation function. It is a fully connected layer. It is the ReLU activation function. For global average pooling, For splicing operations, The decoder for the H&E semantic segmentation module in the first... Feature map of the layer The decoder for the edge detection module in the first... Feature map of the layer For axial cross attention The output of the layer, The feature map is obtained by calculating the axial cross-attention in the horizontal direction. The feature map is calculated for the vertical axial cross attention. The softmax activation function is used. This is the projection weight matrix of the query vector in the horizontal direction. This is the projection weight matrix of the query vector in the vertical direction. This is the projection weight matrix of the horizontal key vector. This is the projection weight matrix of the vertical direction key vector. The projection weight matrix is the value vector in the horizontal direction. The projection weight matrix is the value vector in the vertical direction. For transpose operation, is the dimension of the key vector.