Pathological section image segmentation method and device based on multi-modal supervised contrast learning

Abstract

This invention belongs to the interdisciplinary field of medical image analysis and deep learning, and discloses a method and apparatus for pathological slide image segmentation using multimodal supervised contrastive learning. The method generates biological prior feature maps through decoupling, extracts visual and semantic features by combining the master encoder and prior encoder, and achieves feature alignment using a cross-modal attention mechanism. Supervised contrastive learning is introduced to enhance intra-class compactness and inter-class separability. A multi-scale decoupled decoder is designed, with structural similarity loss and boundary-aware loss respectively supervising structural and detail branches. Finally, dynamic weighted fusion and pseudo-label calibration strategies are used to optimize segmentation performance, significantly improving the accuracy and robustness of pathological image segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing and computer vision, in particular to a pathological section image segmentation method and device based on multi-modal supervised contrast learning. BACKGROUND

[0002] In clinical pathological diagnosis and research, accurate cell, gland and tumor region segmentation of hematoxylin-eosin (H&E) stained digital pathology whole section images is a key foundation for realizing quantitative analysis and intelligent auxiliary diagnosis. However, current pathological image segmentation methods based on deep learning still face several outstanding challenges.

[0003] Chinese patent, publication number: CN119273664B, publication date: May 23, 2025, discloses a whole section pathological image segmentation and auxiliary annotation method. First, the whole section pathological image WSI is divided into a model training data set and a to-be-annotated data set, the cancer regions in the model training data set are annotated, and a pathological image segmentation data set and a model prediction data set are constructed; the U-Net segmentation model is trained using the pathological image segmentation data set; second, the trained U-Net segmentation model is used to predict the small image blocks in the model prediction data set, generate the segmentation results of each small image block, and splice the results into a whole pathological section; finally, the contours of the segmentation results are extracted, and the coordinate point values of the contours are written into an XML annotation file, and a pathologist can review the quality of the annotation on the pathological image viewing software ASAP.

[0004] The above technical solution has the following disadvantages:

[0005] 1. Limited model feature learning and discrimination ability: the standard U-Net architecture and complete supervised training are used, and the model learning completely depends on the manually annotated pixel-level labels. This method does not explicitly introduce or utilize the inherent histological prior knowledge of pathological images (such as cell morphology, tissue structure, etc.), nor does it use a special mechanism to enhance the feature consistency of similar samples and the feature discrimination of different samples, which may lead to insufficient feature discrimination of the model learned on complex and diverse pathological images, affecting the segmentation accuracy, especially in the case of limited annotation data or sample imbalance.

[0006] 2. Strong dependence on annotated data, and single way to improve model robustness and generalization: the model performance improvement is heavily dependent on a large amount of high-quality manual annotation data, which has high annotation cost. Its improvement mainly focuses on post-processing of segmentation (generating XML auxiliary annotation file) to reduce the subsequent annotation burden of doctors, but the model training process itself does not integrate learning mechanisms (such as contrast learning, self-training, etc.) aimed at reducing annotation dependence and enhancing feature robustness. This limits the generalization ability of the model in limited annotation or data distribution change scenarios. SUMMARY

[0007] In order to solve the problems of the prior art, the present application aims to provide a pathological section image segmentation method and device based on multi-modal supervised contrast learning, which can reduce the dependence on a large amount of fine annotation data, improve the learning and discrimination ability of the pathological image segmentation model for complex tissue characteristics, and improve the accuracy and robustness of the final segmentation result.

[0008] To achieve the above object, the present application adopts the following technical solution:

[0009] The present application provides a multi-modal supervised contrast learning pathological section image segmentation method, which comprises:

[0010] S101, acquiring a pathological section image;

[0011] S102, standardizing and preprocessing the pathological section image;

[0012] S103, cropping the preprocessed image into a plurality of image blocks, and generating a corresponding biological prior feature map from each image block;

[0013] S104, inputting the image blocks into a main encoder to extract visual features, and inputting the corresponding biological prior feature map into a prior encoder to extract semantic features;

[0014] S105, performing cross-modal fusion on the visual features and the semantic features, using an attention mechanism, taking the semantic features as keys and values and taking the visual features as queries, to complete feature alignment and enhancement;

[0015] S106, performing supervised contrast learning after the main encoder: generating a view pair by applying two kinds of data augmentation to the image blocks, and calculating a supervised contrast loss to reduce the feature distance of similar samples and increase the feature distance of different samples;

[0016] S107, inputting the cross-modal features into a multi-scale decoupling decoder, and supervising the structure decoding branch of the multi-scale decoupling decoder by a structural similarity loss to learn regional semantics, and supervising the detail decoding branch by a boundary perception loss to learn boundary details;

[0017] S108, dynamically weighting and fusing the output features of the structure decoding branch and the detail decoding branch to generate a preliminary segmentation prediction map;

[0018] S109, generating a high-confidence pseudo label based on the preliminary segmentation prediction map, combining the exponential moving average output of the historical parameters for calibration, and updating the network parameters of the main encoder, the prior encoder and the decoder using the calibrated pseudo label.

[0019] As a preferred technical solution, step S102 comprises:

[0020] Color normalization is performed using the Macenko method based on color deconvolution or a style transfer network based on deep learning to eliminate color deviations caused by staining and scanning, thus ensuring a consistent color distribution in the image. An adaptive threshold is set by calculating the color intensity, saturation, or texture features of the image to identify and remove background areas below the threshold, retaining only biological tissue areas. The output is a preprocessed image that has undergone color normalization and background removal.

[0021] As a preferred technical solution, step S103 involves cropping the preprocessed image to obtain multiple image patches and generating corresponding biological prior feature maps, including:

[0022] Adaptive mesh generation is performed based on tissue saliency detection. A smaller cropping step size is used in dense tissue regions and a larger step size is used in sparse regions to balance detail preservation and computational efficiency, resulting in multiple image patches. Each image patch is input into a learnable biological prior decoupling network. This network is trained under the joint constraints of a pixel-level self-supervised reconstruction task and a gradient-guided task connected to the main segmentation network through an encoder-decoder structure. This adaptively decouples biological prior feature maps that both preserve the original morphological semantics and enhance the information related to the segmentation of the target pathological structure.

[0023] As a preferred technical solution, step S104 involves inputting the image patch into the main encoder to extract visual features, and inputting the corresponding biological prior feature map into the prior encoder to extract semantic features, including:

[0024] A deep heterogeneous dual encoder is constructed, comprising a master encoder and a prior encoder. The master encoder employs a deep convolutional neural network pre-trained on a natural image dataset to extract visual features from image patches. The prior encoder shares some low-level weights with the master encoder and is used to extract semantic features from biological prior feature maps. A deformable convolutional cross-modal alignment module is introduced between the corresponding feature layers of the master encoder and the prior encoder. The deformable convolutional cross-modal alignment module uses the preliminary alignment result of the previous layer as a supervision signal. By calculating the spatial correlation between the two feature maps, it adaptively learns the offset field, guiding the prior encoder to progressively align with the visual features of the master encoder at the pixel-level spatial position when extracting semantic features.

[0025] As a preferred technical solution, step S105 involves cross-modal interaction and fusion of visual and semantic features, including:

[0026] Parallel cross-modal interactive fusion is performed at multiple scales: At each feature level, visual features are used as queries and semantic features as keys and values, and are input into a dynamic attention weight generation network. The generated network generates attention weights based on the local spatial correlation between visual and semantic features. The generated attention weights are used to weight and aggregate semantic features to obtain context-enhanced semantic information. Through a spatially and channel-adaptive gated fusion unit, the enhanced semantic information is fused with the original visual features element by element to adaptively determine the information retention ratio of each position and channel. Finally, through a feature pyramid network with skip connections, the fused features from each layer are integrated and upsampled to output cross-modal features that deeply fuse visual details and prior semantics.

[0027] As a preferred technical solution, step S106 involves performing supervised contrastive learning after the main encoder, including:

[0028] Two independent pathological perception data augmentations are applied to image patches to generate a pair of augmented views, with augmentation parameters dynamically adjusted based on tissue type. These augmented views are then input into the main encoder, and an additional feature projection head maps the high-dimensional visual features to a low-dimensional contrast learning space of a unit hypersphere. Supervised contrast loss is computed in the contrast learning space: this loss not only brings the features of the two augmented views from the same image patch closer together, but also utilizes pathological labels to bring together features of samples of the same category and push away features of samples of different categories. Simultaneously, an attention-weighted hard negative sample mining strategy is employed to impose a greater penalty on hard negative sample pairs that are similar in features to the anchor sample but of a different category. By minimizing the supervised contrast loss, the main encoder is driven to learn feature representations that are robust to staining differences and imaging artifacts and highly discriminative of different pathological structures.

[0029] As a preferred technical solution, step S107 involves inputting the cross-modal features into a multi-scale decoupled decoder for decoding and reconstruction, including:

[0030] A multi-scale decoupled decoder with a bidirectional information guidance mechanism is constructed, comprising a structural decoding branch and a detail decoding branch. The structural decoding branch takes deep cross-modal features as input, recovers resolution through upsampling and residual connections, and utilizes a region-aware attention module to focus on the overall morphology of the tissue region. Training of the structural decoding branch is supervised by a loss function that calculates a multi-scale structural similarity index to learn region-level semantic consistency. The detail decoding branch takes shallow cross-modal features as input, preserves high-resolution details through lightweight convolutions and skip connections, and integrates an edge enhancement module to strengthen high-frequency boundary information. Training of the detail decoding branch is conducted by combining weighted cross-multiplication... Entropy and boundary FocalLoss loss functions are used for supervision to learn pixel-level boundary accuracy. During the decoding process, a progressive feature interaction strategy is introduced: in multiple upsampling stages, the structural decoding branch outputs the learned regional semantic prior to the detail decoding branch through the attention weight map to guide the focusing of detail features; the detail decoding branch, in turn, feeds back the boundary confidence map it generates by applying the Sigmoid function to the structural decoding branch to correct the region prediction; this bidirectional interaction is achieved through a cross-branch attention module, where the output features of one branch are convolved and adjusted, and then fused into the corresponding features of another branch in the form of element-wise multiplication or addition.

[0031] As a preferred technical solution, step S108, dynamically weighting and fusing the output features of the structural decoding branch and the detail decoding branch to generate a preliminary segmentation prediction map, includes:

[0032] A content-aware, dynamically gated fusion network is constructed to receive feature maps output from the structural decoding branch and the detail decoding branch. A lightweight convolutional module analyzes the dual-branch feature maps to generate a pair of spatially adaptive fusion weight maps to quantify the degree of semantic consistency and boundary detail requirements at each location. Based on the fusion weight maps, the dual-branch feature maps are summed element-wise with weights, thereby assigning higher weights to structural branch features in tissue interior regions and higher weights to detail branch features in cell boundary and gland contour regions. The fused feature maps are input into a lightweight prediction head, which outputs the initial probability distribution of each pixel corresponding to different pathological tissue categories, i.e., the preliminary segmentation prediction map.

[0033] As a preferred technical solution, step S109, based on the preliminary segmentation prediction map, involves online pseudo-label refinement and self-training, including:

[0034] Pixel-level uncertainty estimation is performed on the preliminary segmentation prediction map, and a dynamic threshold is set to filter out high-confidence regions to generate initial pseudo-labels. The current initial pseudo-labels are compared with the predictions generated by the exponential moving average (EMA) model of historical network parameters. Noise is filtered out through consistency filtering and probability-weighted averaging to obtain calibrated smooth pseudo-labels. A course learning strategy is adopted to dynamically adjust the training objective: in the early stage of training, strong supervision loss based on labeled data is relied upon, and weak supervision loss based on calibrated pseudo-labels is gradually introduced as training progresses, and the mixed weights of strong supervision loss and weak supervision loss are adaptively adjusted. Finally, the strong supervision loss based on labeled data and the self-training loss of calibrated pseudo-labels are combined to form a hybrid training objective, and the network parameters of the master encoder, prior encoder and multi-scale decoupled decoder are jointly updated through backpropagation.

[0035] This application also provides a pathological slide image segmentation device based on multimodal supervised contrastive learning, the device comprising:

[0036] The data preprocessing module is used to acquire pathological slide images and perform standardized preprocessing, including color normalization and background removal.

[0037] The feature generation module is used to crop the preprocessed image into multiple image patches and decouple the corresponding biological prior feature map from each image patch.

[0038] The feature extraction and fusion module includes a main encoder and a prior encoder, which are used to extract visual features from image patches and semantic features from biological prior feature maps, respectively, and perform cross-modal fusion through an attention mechanism, where semantic features are used as keys and values ​​and visual features are used as queries to complete feature alignment and enhancement.

[0039] The supervised contrastive learning module is used to perform supervised contrastive learning after the main encoder. It generates view pairs by applying data augmentation to image patches and calculates supervised contrastive loss to bring the feature distance of similar samples closer and push the feature distance of dissimilar samples further apart.

[0040] The decoding and prediction module includes a multi-scale decoupled decoder, which inputs cross-modal features into the structural decoding branch and the detail decoding branch, respectively supervised by structural similarity loss and boundary-aware loss, to learn region semantics and boundary details, and dynamically weights and fuses the output features of the two branches to generate a preliminary segmentation prediction map;

[0041] The self-training optimization module is used to generate high-confidence pseudo-labels based on the initial segmentation prediction map, calibrate them by combining the exponential moving average output of historical parameters, and update the network parameters of the master encoder, prior encoder and multi-scale decoupled decoder using the calibrated pseudo-labels.

[0042] Compared with the prior art, the beneficial effects of this application are as follows:

[0043] 1. By introducing biological prior feature maps and constructing a dual-encoder architecture, a deep fusion of visual representation and histological semantic knowledge is achieved. In the feature extraction stage, the prior encoder explicitly models inherent pathological prior information such as cell morphology and tissue structure, while the main encoder focuses on pixel-level visual pattern learning. After alignment and enhancement through a cross-modal attention mechanism, the biological rationality and discriminative ability of feature expression are significantly improved, overcoming the problem of insufficient feature abstraction caused by the traditional U-Net's reliance on pixel labels. It can maintain stable performance, especially in scenarios with sparse annotations or class imbalance.

[0044] 2. The supervised contrastive learning module introduces a feature constraint mechanism at the output of the main encoder that is compact within classes and separate between classes. By applying diverse data augmentations to the same image patch to generate positive and negative sample pairs, and using supervised signals to guide the optimization of feature space distribution, the robustness of the model to pathological morphological variations is effectively enhanced, the dependence on large-scale finely labeled data is reduced, and the model's generalization ability on small samples and cross-center data is improved.

[0045] 3. The multi-scale decoupled decoder adopts a dual-branch design of structure and detail, with differentiated supervision by structural similarity loss and boundary awareness loss respectively. This enables the network to collaboratively learn the semantic consistency of regions and the geometric accuracy of boundaries, significantly improving the problems of blurred boundaries and structural breaks that are prone to occur in traditional single segmentation heads. It is especially suitable for pathological target segmentation tasks with complex topological structures such as glands and tumor nests.

[0046] 4. The dynamic weighted fusion mechanism adaptively adjusts the contribution of structural and detail branches based on the local prediction confidence, avoiding the suboptimal fusion problem caused by fixed weights, and further improving the overall coherence and detail fidelity of the segmentation results.

[0047] 5. A semi-supervised update strategy combining pseudo-label generation and exponential moving average calibration is adopted. The training signal is gradually expanded by the model’s own high-confidence prediction. The network parameters are continuously optimized without increasing the cost of manual annotation, forming a closed-loop self-enhancing training process. This not only alleviates the annotation bottleneck, but also enhances the long-term training stability and convergence quality of the model.

[0048] In summary, this application achieves synergistic innovation across four levels: feature modeling, discriminative learning, decoding structure, and training strategy. This significantly improves the accuracy, robustness, and practicality of pathological image segmentation, providing more reliable technical support for clinical auxiliary diagnosis. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the steps of the pathological slide image segmentation method based on multimodal supervised contrastive learning proposed in this application. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present application, the technical solutions in specific embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0051] like Figure 1 As shown, this application provides a method for pathological slide image segmentation based on multimodal supervised contrastive learning, including the following steps:

[0052] S101, Obtain pathological slide images.

[0053] Specifically, digitized pathological slide files are read from a data source. This is accomplished by calling the software development kit (SDK) that accompanies the digital slide scanner or by reading files in a standard format that conforms to specifications. For example, the `open_slide` function of the open-source library OpenSlide can be used to read a full slide image file with the .svs extension, or the corresponding pathological slide sequence can be retrieved and downloaded through the DICOM interface of the hospital's Picture Archiving and Communication System (PACS).

[0054] Pathological slide images are digital images formed under a microscope from thin sections of biological tissue that have undergone specific staining treatment. This application uses hematoxylin-eosin staining, also known as H&E staining. Its core characteristic is that the cell nucleus is stained blue-purple by hematoxylin, while the cytoplasm and extracellular matrix are stained pink to varying degrees by eosin, thus creating a sharp contrast that facilitates observation of the morphology and structure of cells and tissues.

[0055] The following are typical image examples of several common tissues under H&E staining:

[0056] Normal colon tissue: The image shows a smooth mucosa without villi, with a large number of vertically arranged crypts lined with numerous goblet cells in a vacuolated manner, and a small number of lymphocytes in the lamina propria.

[0057] Liver tissue: Hepatic lobule structure can be observed; hepatocytes have abundant, polygonal cytoplasm and round, centrally located nuclei. In a congested state, hepatic sinusoids are visible as dilated and filled with red blood cells.

[0058] Skin tissue: The epidermis, dermis, and subcutaneous tissue can be clearly distinguished. The epidermis consists of the basal layer, spinous layer, granular layer, and stratum corneum from the base to the surface.

[0059] Example of inflamed or diseased tissue:

[0060] Inflammation: In images of colitis, a large number of inflammatory cells such as neutrophils and lymphocytes can be seen infiltrating the mucosa and submucosa, and the crypt structures may be deformed or destroyed.

[0061] Tumors: In images of liver cancer tissue, cells exhibit atypia, characterized by varying cell size and shape, increased nucleus-cytoplasm ratio, darker nuclear staining, and pathological mitotic figures.

[0062] S102, perform standardized preprocessing on the pathological slide images to obtain preprocessed images. Standardized preprocessing includes color normalization and background removal.

[0063] S103, the preprocessed image is cropped to obtain multiple image blocks, and the corresponding biological prior feature map is generated from each image block.

[0064] S104, input the image patch into a master encoder to extract visual features, and input the biological prior feature map corresponding to the same image patch into a prior encoder to extract semantic features.

[0065] S105 involves cross-modal interaction and fusion of visual features extracted by the main encoder and semantic features extracted by the prior encoder to obtain fused cross-modal features. An attention mechanism is employed, using semantic features as keys and values, and visual features as queries, to achieve feature alignment and enhancement.

[0066] S106, Supervised contrastive learning is performed after the main encoder: Two independent data augmentation transformations are applied to the image patch to generate a pair of augmented views, and a supervised contrastive loss is computed based on the features of the pair of augmented views. The supervised contrastive loss is used to bring closer the feature distances of augmented views from the same image patch and to bring closer the feature distances of samples with the same pathology category label in the feature projection space, while pushing away the feature distances of samples from different pathology categories.

[0067] In step S107, cross-modal features are input into a multi-scale decoupled decoder for decoding and reconstruction. The multi-scale decoupled decoder includes a structural decoding branch and a detail decoding branch. The structural decoding branch processes deep cross-modal features and is supervised by a structural similarity loss function to learn semantic consistency at the region level. The detail decoding branch processes shallow cross-modal features and is supervised by a boundary-aware loss function to learn pixel-level boundary accuracy.

[0068] S108, the feature maps output by the dynamic weighted fusion structure decoding branch and the detail decoding branch, generate a preliminary segmentation prediction map.

[0069] S109, Online pseudo-label refinement and self-training based on the preliminary segmentation prediction map: Generate high-confidence pseudo-labels based on the currently generated preliminary segmentation prediction map, calibrate the high-confidence pseudo-labels by combining the exponential moving average output corresponding to the historical network parameters, and update the network parameters of the master encoder, prior encoder and multi-scale decoupled decoder using the calibrated pseudo-labels.

[0070] Furthermore, in step S102, the standardization preprocessing of the pathological slide images includes:

[0071] The Macenko method based on color deconvolution or a style transfer network based on deep learning is used to normalize the colors of pathological slide images to eliminate color deviations caused by differences in staining procedures and scanning equipment, ensuring that all images have a consistent color distribution. This step aims to address color differences arising from variations in hematoxylin and eosin concentrations, staining times, scanning equipment models, and lighting conditions between different hospitals and batches, ensuring that the color distribution of all input images is within the same standard space. This is the foundation for the stability of subsequent algorithms.

[0072] One method based on color deconvolution is the Macenko method. This method, based on the Lambert-Beer law, decouples the RGB optical density (OD) space of the H&E-stained image to a two-dimensional space spanned by the color vectors of hematoxylin (H) and eosin (E). First, standard H and E color vectors, i.e., the staining matrix, are statistically derived from a batch of representative images. For each image to be processed, its OD value is calculated, and deconvolution is performed using the standard staining matrix to separate the density maps of the H and E channels. Subsequently, these two density maps can be standardized, such as by matching the density statistics of a reference image. Finally, the standard staining matrix is ​​used to resynthesize the RGB image. This process is equivalent to stripping away the original staining and then re-staining according to the standard formula.

[0073] For example, after deconvolution and normalization, a purplish slice and a pinkish slice will show consistent, standard blue-purple staining in their cell nuclei and consistent, standard pink staining in their eosin in their cytoplasm, respectively.

[0074] One approach, based on deep learning style transfer, involves constructing an image-to-image translation network such as CycleGAN or U-Net, and training it on a large number of unpaired pathological images with varying source domain colors and standard target domain colors. The network learns to transfer the color style of any input image to a unified, ideal color style. This method does not rely on a fixed physical model, can learn more complex color mapping relationships, and has better correction capabilities for abnormal staining.

[0075] By calculating the color intensity, saturation, or texture features of the image and setting an adaptive threshold, regions below the adaptive threshold are identified as background and removed, retaining only regions of interest containing biological tissue. Digital pathology whole-slice images often contain a large number of tissueless white background areas; this step aims to automatically identify and remove these areas, retaining only regions of interest containing biological tissue, thereby reducing unnecessary calculations and avoiding background interference.

[0076] Specifically, the RGB image after color normalization is converted to a color space more suitable for color perception and analysis. Color intensity or value: directly corresponds to the V channel in HSV. The background is usually white or a very light color with a high V value, while tissue areas have relatively low V values ​​due to staining. Saturation: corresponds to the S channel in HSV. Unstained background areas have low color purity and very small S values; tissue areas have higher S values ​​due to staining. Texture features: Typically, the image is first converted to grayscale, and then the gray-level co-occurrence matrix is ​​used to calculate local texture indices. Background areas are smooth and homogeneous, with low contrast and high energy; while tissue areas, containing cells, glands, and other structures, have more complex textures, higher contrast, and lower energy.

[0077] Calculate a comprehensive feature value F for each pixel. For example, a simple and effective rule could be: when the saturation of a pixel is below a threshold th... s And the brightness is higher than the threshold th v At this point, it is initially classified as background. More complex methods can combine texture features such as contrast C.

[0078] Threshold th s ,th v ,th c Instead of using fixed values, an adaptive thresholding algorithm, such as Otsu's Method, is used to dynamically determine the threshold for each feature channel of each image. For example, using Otsu's Method on the S-channel histogram of the entire image, a threshold th is automatically calculated that optimally divides pixels into low-saturation background and high-saturation organization. s .

[0079] Furthermore, the initial background mask may contain noise or small holes, which usually require morphological operations for optimization: first, use opening operations (erosion followed by dilation) to remove small white noise points that are mistakenly identified as tiny tissue points in the background; then, use closing operations (dilation followed by erosion) to fill the small holes in the background area that are mistakenly identified as background tissue points, finally obtaining a clean and connected background area mask.

[0080] Output the preprocessed image after color normalization and background removal.

[0081] Furthermore, in step S103, the preprocessed image is cropped to obtain multiple image patches, and a corresponding biological prior feature map is generated from each image patch by decoupling:

[0082] The preprocessed image undergoes adaptive meshing based on tissue saliency detection. Smaller cropping strides are used in densely tissueed regions to preserve details, while larger strides are used in sparsely tissueed regions to improve computational efficiency, resulting in multiple image patches. The goal of this step is to efficiently and intelligently sample image patches for model training from a large full-slice image. Its core lies in dynamically adjusting the sampling density based on the importance of the tissue region.

[0083] Specifically, first, a tissue saliency probability map is computed by performing a low-resolution scan of the entire preprocessed image or by using a lightweight convolutional neural network (CNN) for fast inference. This can be achieved by calculating the texture complexity of local regions, such as the entropy of the gray-level co-occurrence matrix, color saturation in HSV space, or by using a pre-trained binary classification model for tissue vs. background. Regions with high probability values ​​represent dense tissue and complex structures, such as the tumor core area, while regions with low probability values ​​represent sparse tissue or background.

[0084] Based on the aforementioned saliency map, a non-uniform grid is used for partitioning. In dense regions with high tissue saliency values, a smaller cropping step size, such as 128 pixels, and potentially a smaller window size are set to achieve high-density sampling, ensuring the capture of subtle cellular atypia and complex glandular structures. In sparse regions or the stroma with low tissue saliency values, a larger step size, such as 256 or 512 pixels, is used to reduce redundant sampling and improve overall processing efficiency. Finally, based on this dynamic grid, a series of image patches of fixed size (e.g., 256 x 256 pixels) with varying spatial density are cropped from the original image.

[0085] Each image patch is fed into a learnable biological prior decoupling network. This network, trained end-to-end, adaptively decouples the most effective features for the downstream segmentation task from the original image, outputting a task-adaptive biological prior feature map. This step aims to automatically learn and extract the most advantageous auxiliary features for the final segmentation task from the original image patches, rather than relying on fixed algorithms such as color deconvolution.

[0086] The learnable biological prior decoupled network consists of an encoder and a decoder. The encoder comprises five convolutional layers with channel numbers of 32, 64, 128, 256, and 512 respectively. Each convolutional layer has a 3×3 kernel size and a stride of 2 for downsampling. The activation function is ReLU. The decoder employs a symmetrical structure, consisting of five deconvolutional layers or upsampling plus convolutional layers with channel numbers of 512, 256, 128, 64, and 32 respectively. Each convolutional layer has a 3×3 kernel size and a stride of 2 for upsampling. The activation function is ReLU. The last layer outputs with the same number of channels as the prior feature map, and the activation function is Sigmoid.

[0087] The training of the learnable biological prior decoupling network is constrained by an auxiliary pixel-level self-supervised reconstruction task and a gradient-guided task connected to the main segmentation network to ensure that the generated prior features retain both the original morphological semantics of the cell nucleus and glands and reinforce the information most relevant to the segmentation of the target pathological structure.

[0088] The learnable biological prior decoupling network employs a dual-constraint training mechanism, with a total loss function. Defined as:

[0089]

[0090] in, For pixel-level self-supervised reconstruction loss, This is the gradient-guided loss. and In this embodiment, the weighting ratio coefficient is used. The value is 0.6. The value is set to 0.4. This ratio has been experimentally verified to effectively enhance segmentation-related features while preserving the original morphological information.

[0091] Pixel-level self-supervised reconstruction task: This task requires the decoder to reconstruct the original image patch well from the encoded features. Its loss function is L1 loss, or mean absolute error, which calculates the average of the absolute differences between the predicted and true values. The formula is:

[0092]

[0093] in, Indicates the number of samples. Indicates the first The true value of each sample Indicates the first The predicted value is calculated for each sample. The penalty for error is linear. This means that one unit of error results in one unit of loss. Therefore, it is less sensitive to outliers, and the optimization process tends to yield a median estimate, which may cause the generated image to appear slightly blurry in some details, but the overall structure remains stable.

[0094] Gradient-guided task: The learnable biological prior decoupling network and the main segmentation network (i.e., the encoder and decoder mentioned in the claims) are jointly trained end-to-end. Specifically, the gradients generated by the main segmentation network during the segmentation task are backpropagated to the decoder and even the encoder of the prior decoupling network via a chain rule. This gradient signal directly indicates which feature information is most helpful in improving segmentation accuracy. Therefore, based on the reconstruction task, the decoupling network adaptively adjusts its parameters so that the generated prior feature maps can enhance the information that the main segmentation network is interested in, such as enhancing the boundaries of tumor regions and suppressing irrelevant stained background.

[0095] Furthermore, in step S104, inputting the image patch into a master encoder to extract visual features, and inputting the biological prior feature map corresponding to the same image patch into a prior encoder to extract semantic features, includes:

[0096] Construct a deep heterogeneous dual encoder, which consists of a master encoder and a prior encoder.

[0097] The main encoder uses a deep convolutional neural network, such as ResNet, DenseNet, or Vision Transformer, pre-trained on natural image datasets such as ImageNet as the feature extraction backbone to extract visual features with rich texture and spatial context from RGB format image patches.

[0098] The prior encoder employs a convolutional network that shares some of the underlying weights with the main encoder. The input to the prior encoder is a biological prior feature map generated by a learnable biological prior decoupling network. The prior encoder transfers general low-level feature extraction capabilities through the shared underlying weights, and then extracts semantic features that are highly correlated with pathological structures through independent subsequent layers.

[0099] The first one or two convolutional layers of the prior encoder completely share weights with the corresponding bottom layers of the main encoder. This is because the bottom convolutional kernels typically learn general low-level features such as edges and blobs, which are common to processing raw RGB images and abstract prior feature maps. Sharing weights enables capability transfer and significantly reduces the number of parameters. Starting from a certain layer, the prior encoder has its own independent convolutional layers. These layers specifically learn how to further refine high-level semantic features highly correlated with specific pathological structures such as glandular morphology and nuclear atypia from the general features extracted from the shared bottom layers.

[0100] A deformable convolutional cross-modal alignment module is introduced between the corresponding feature layers of the main encoder and the prior encoder. This allows the prior encoder to align the two visual features and semantic features at pixel-level spatial positions by referencing the spatial structure of the corresponding layer features of the main encoder when extracting semantic features. The deformable convolutional cross-modal alignment module is inserted between the corresponding feature layers of the main encoder and the prior encoder, for example, after the third convolutional block.

[0101] The deformable convolutional cross-modal alignment module works as follows: Using the output feature map of a certain layer of the main encoder, rich in precise spatial location information, as a reference, an additional lightweight convolutional layer predicts a two-dimensional offset field for each spatial location on the feature map of the corresponding layer of the prior encoder. The learned offset field is then used to perform a deformable convolution operation on the feature map of the prior encoder. This means that the sampling points of the convolutional kernel are no longer a regular grid, but actively point to their correct alignment positions based on the offset. This process allows semantic information in the prior feature map, such as a cell nucleus, to dynamically and non-rigidly deform based on the spatial context provided by the main encoder features, such as the actual boundary of the cell nucleus, thereby achieving pixel-level spatial alignment.

[0102] During training, an image patch and its corresponding prior feature map are input into the dual encoders. The information flow is as follows: During the extraction of semantic features, the prior encoder receives spatial guidance from the deformable alignment module of the main encoder at multiple levels, ensuring that the generated semantic features are spatially consistent with the visual features. Finally, the two encoders output a series of spatially preliminarily aligned, multi-scale visual and semantic features for subsequent deep fusion steps.

[0103] Furthermore, in step S105, the visual features extracted by the main encoder and the semantic features extracted by the prior encoder are subjected to cross-modal interaction and fusion to obtain the fused cross-modal features, including:

[0104] A multi-scale learnable cross-modal interaction and fusion mechanism is constructed, which is executed in parallel at different feature levels of the encoder network. This mechanism is implemented at different depths of the encoder network, such as conv2 of ResNet. x conv3 x conv4 x The output layer simultaneously extracts visual and semantic features. These features constitute multiple sets of feature pairs corresponding to different layers, where shallow features have high resolution and rich details, while deep features have strong semantic meaning and a large receptive field.

[0105] At each level, the visual features extracted by the master encoder are used as the query tensor, and the semantic features extracted by the prior encoder are used as the key and value tensors, respectively. These are input into a dynamic attention weight generation network. The dynamic attention weight generation network dynamically generates attention weights based on the correlation between visual and semantic features within local spatial regions. The generated attention weights are then used to weight and aggregate the semantic features to obtain context-enhanced semantic information.

[0106] Specifically, at each level, an improved cross-attention process is performed. The visual features of that level are used as the query, and the semantic features are used as both the key and value. The query and key are concatenated or summed along the channel dimension and then fed into a lightweight convolutional module, a dynamic attention weight generation network. This module learns the correlation between the two in the local space and outputs a dynamic attention weight map of the same size as the feature map. This approach is more efficient and focuses on the local context compared to computing global dot product attention. The generated dynamic attention weight map is used to perform a weighted summation of the semantic features (values) to obtain context-enhanced semantic information. At this point, the semantic information has been modulated and filtered by the local context of the visual features.

[0107] An adaptive gating fusion unit is used to fuse context-enhanced semantic information with the original visual features element by element. The adaptive gating fusion unit learns a spatial and channel-adaptive weight matrix to determine how much original visual information should be retained and how much supplementary semantic information should be introduced at each position and each feature channel.

[0108] Specifically, the original visual features and the context-enhanced semantic information obtained in the previous step are input into the adaptive gating fusion unit. The adaptive gating fusion unit analyzes both inputs through a sub-network incorporating channel attention and spatial attention mechanisms, generating a spatially and channel-adaptive weight matrix. The value range is from 0 to 1. According to the formula:

[0109]

[0110] Element-by-element fusion is performed. Among them, Indicates visual characteristics, This represents the enhanced semantic features. The fusion weight represents a feature... A tensor of the same dimension, where each element has a value between 0 and 1. The symbol represents element-wise multiplication, and 1 represents an AND operation. Tensors of the same dimension with all elements equal to 1. The model uses these tensors to autonomously decide whether to rely on the original visual details or the enhanced semantic priors at each location and each channel.

[0111] The feature maps, after being processed and fused from multiple levels, are integrated and upsampled through a feature pyramid network with skip connections, outputting cross-modal features that restore spatial resolution and deeply fuse visual details and prior semantics.

[0112] Specifically, the feature maps from all levels, after undergoing the dynamic attention-gated fusion process described above, are input into a Feature Pyramid Network (FPN). High-level low-resolution, semantically strong features are upsampled and element-wise added or concatenated with the next level's high-resolution, multi-detail features, combined with skip connections. This process proceeds top-down, level by level, ultimately outputting a cross-modal feature pyramid with restored spatial resolution and deep fusion of microscopic details to macroscopic semantics. The highest-resolution feature map is directly used for subsequent decoding and segmentation.

[0113] Furthermore, in step S106, performing supervised contrastive learning after the main encoder includes:

[0114] Two independent adaptive pathological-sensing data enhancement transformations are applied to image patches to generate a pair of enhanced views, with the combination of enhancement parameters dynamically adjusted based on real-time analysis of tissue types within the image patches.

[0115] Specifically, for each image patch, instead of applying a generic random enhancement, enhancement parameters are dynamically configured based on a rapid analysis of its content, such as a preliminary assessment of its tissue density and main structural categories. For example, for regions with dense cell nuclei, color jitter simulating staining differences is applied; for tissue structures such as glands, geometric transformations simulating slice deformation are applied. This generates two semantically consistent but visually distinct enhanced views of the same image patch, forming a high-quality positive sample pair.

[0116] The two augmented views are fed into the main encoder, and the extracted high-dimensional visual features are mapped into a low-dimensional contrastive learning feature space of a unit hypersphere through an additional feature projection head that shares some of the low-level parameters with the main encoder but has independent high-level parameters.

[0117] Specifically, the generated pair of enhanced views are input into the main encoder and the segmentation main network, respectively, sharing parameters. After the main encoder extracts the high-dimensional visual features, it feeds them into a separate feature projection head. This projection head is typically a small multilayer perceptron (MLP), whose lower layers may share some weights with the shallow layers of the main encoder to reuse low-level features, but its top layers are independent. Its core function is to map the high-dimensional features to a low-dimensional, normalized vector space, namely the unit hypersphere, which is specifically designed to optimize the contrastive loss.

[0118] In the contrastive learning feature space, supervised contrastive loss is calculated. The loss function of supervised contrastive loss not only requires the feature representations of two enhanced views from the same original image patch to be close to each other, but also requires the use of existing pixel-level or image-level pathological labels to bring the features of all samples belonging to the same pathological category in the same batch closer together, while pushing the features of samples from different categories further apart.

[0119] The supervised contrastive loss function, within a low-dimensional feature projection space specifically designed for contrastive learning, adjusts the distribution of sample feature vectors such that: features of positive sample pairs are brought closer together. This includes two views of the same image patch augmented with different data, as well as all other samples in the same training batch with the same pathological category label. Features of negative sample pairs are pushed apart. In other words, the distance between features of samples from different pathological categories should be as large as possible.

[0120] For example, the loss function for supervised contrastive loss is:

[0121]

[0122] in, It is the current sample anchor point sample. Anchor point sample The eigenvectors in the projection space. Anchor point sample The collection of all positive samples, including their enhanced view and other samples of the same class in the batch. It is a positive sample index. It is the feature vector of the positive sample. Anchor point sample It is a set of samples from all negative samples of different categories. It is a temperature hyperparameter used to control the discrimination of difficult samples. It is the size of the positive sample set. The dot product of vectors is used to measure similarity.

[0123] The loss function that minimizes the supervised contrast loss aims to drive the main encoder to learn a better feature representation: 1. High invariance to irrelevant changes such as staining differences and imaging artifacts: because different enhanced views of the same content are required to have similar features, the encoder is forced to ignore these irrelevant visual changes. 2. Strong discriminative power against different pathological tissue structures: because similar pathological samples are brought closer together and different samples are pushed apart, the encoder is forced to extract discriminative features that can clearly distinguish different pathological categories.

[0124] This application introduces an attention-weighted hard negative sample mining strategy. This means that in the above loss calculation, not all negative samples are treated equally. Instead, hard negative samples that are highly similar to the anchor sample features but belong to different categories are identified and given greater weight in the loss calculation. This guides the model to focus more on optimizing these most difficult-to-distinguish boundary cases and further sharpens the decision boundary of features.

[0125] A hard negative sample mining strategy based on attention weighting is introduced, which applies a larger penalty weight to hard negative sample pairs that are similar in features to the anchor sample but belong to a different category when calculating the loss. By minimizing the supervised contrast loss, the main encoder is driven to learn robust feature representations that are highly invariant to unrelated changes in staining differences and imaging artifacts, while also exhibiting strong discriminative power against different pathological tissue structures.

[0126] Specifically, the parameters of the master encoder and projector head are updated by minimizing the aforementioned supervised contrast loss through backpropagation. The direct goal of optimization is to ensure that the feature distribution in the projection space satisfies the following: compact intra-class clustering of similar samples and sparse inter-class dispersion, with samples of different classes far apart.

[0127] Furthermore, in step S107, inputting the cross-modal features into a multi-scale decoupled decoder for decoding and reconstruction includes:

[0128] Construct a multi-scale decoupling decoder with a bidirectional information guidance mechanism.

[0129] The structural decoding branch of the multi-scale decoupled decoder takes deep cross-modal features as input, restores spatial resolution through a series of upsampling layers and residual connections, and uses a region-aware attention module to focus on the overall morphology and semantic consistency of the organization region. The training of the structural decoding branch is supervised by structural similarity loss, which calculates the structural similarity index between the predicted segmentation map and the ground truth label at multiple scales to promote accurate segmentation of large-scale continuous regions.

[0130] The detail decoding branch takes shallow cross-modal features as input, preserves high-resolution spatial details through lightweight convolutional layers and skip connections, and integrates an edge enhancement module to explicitly enhance high-frequency information of cell boundaries and gland contours. The training of the detail decoding branch is supervised by boundary-aware loss, which combines distance-transform-based weighted cross-entropy and FocalLoss for boundary pixels to accurately learn pixel-level boundary accuracy.

[0131] Specifically, 1. Structure decoding branch:

[0132] Input and Processing: The system takes deep, low-resolution cross-modal features rich in high-level semantics as input. Spatial resolution is gradually restored through a series of upsampling layers such as transposed convolutions or bilinear interpolation and residual connections.

[0133] Region-aware attention module: An attention mechanism is embedded in the network, enabling this branch to focus on the overall morphology and semantic consistency of tissue regions, such as identifying the entire tumor region or glandular unit, ensuring the connectivity and integrity of the segmentation results.

[0134] Supervision: Supervision is provided using structural similarity loss. Structural similarity loss forces the branch to learn to generate segmentation maps that are highly consistent with the real-world structure at multiple scales by comparing the predicted map with the ground truth label's structural similarity index across multiple scales. By minimizing the structural similarity loss, the structural decoding branch can be driven to produce segmentation maps that are highly consistent with the real-world structure in terms of overall morphology, contour, and connectivity, thereby particularly optimizing segmentation performance for large-scale, continuous tissue regions such as entire tumor regions and glandular units.

[0135] The Structural Similarity Index (SSIM) is a metric used to measure the perceptual similarity between two images. It compares images across three dimensions: luminance, contrast, and structure. For two images... and At a certain scale, the formula for calculating SSIM is:

[0136]

[0137] in, and Represents two images or image blocks to be compared; It is a brightness comparison function, calculated based on the mean of local regions of two images; It is a contrast comparison function, calculated based on the standard deviation of local regions in two images; It is a structural comparison function, calculated based on the covariance and standard deviation of local regions in two images; , , These are parameters used to adjust the relative weights of the three components: brightness, contrast, and structure. In this application, they are simplified to... .

[0138] To capture structural information at different scales, the image is downsampled multiple times, SSIM is calculated at each scale, and finally the results at each scale are weighted and multiplied to obtain MS-SSIMMulti-ScaleSSIM.

[0139] In deep learning, structural similarity loss is defined as:

[0140]

[0141] in, The image represents the model's prediction. This represents the target image.

[0142] By minimizing this loss, i.e. maximizing the MS-SSIM value, the predicted map is structurally similar to the true label at multiple scales from coarse to fine.

[0143] 2. Detail Decoding Branch:

[0144] Input and Processing: The input consists of shallow, high-resolution but semantically weak cross-modal features. Lightweight convolutional layers and skip connections are used to maximize the preservation of detail information in the original image.

[0145] Edge enhancement module: Integrates specific modules for explicitly detecting and enhancing high-frequency details such as cell boundaries and gland contours to prevent boundary blurring.

[0146] Supervision signal: Supervision is performed using boundary-aware loss. The boundary-aware loss combines distance-transform-based weighted cross-entropy to give higher weights to pixels closer to the boundary and FocalLoss for boundary pixels to focus on hard-to-classify boundary pixels, thereby driving this branch to accurately learn pixel-level boundary orientation.

[0147] For example, the formula for boundary-aware loss is:

[0148]

[0149] in, and For weighting coefficients, the recommended value range is [value range missing]. , The specific value can be fine-tuned based on the boundary complexity of the dataset. In this embodiment, the value is... , .

[0150] Weighted cross-entropy based on distance transform:

[0151]

[0152] in, Total number of pixels Total number of categories; It is a pixel In category The real labels on it; It is the model predicts the pixel. Category The probability of; It is a pixel The weights are calculated from the distance transformation graph: ; For pixels The normalized Euclidean distance to the nearest true boundary. As the boundary attention intensity factor, in this embodiment The value is 10. The value is 5.

[0153] FocalLoss for boundary pixels:

[0154]

[0155] in, It is a pixel The true category; It is the model predicts the pixel. Belongs to its true category The probability of; This is the focus parameter, used to reduce the loss weight of easily classified samples. In this embodiment... The value is 2.0; For category The balancing weights are used to handle class imbalance problems.

[0156] During the decoding process of the structural decoding branch and the detail decoding branch, a progressive feature interaction and fusion strategy is introduced. That is, in multiple upsampling stages, the structural decoding branch outputs the region semantic prior learned by the structural decoding branch to the detail decoding branch as attention guidance, while the detail decoding branch feeds back the boundary confidence map to the structural decoding branch to correct the region prediction.

[0157] Specifically, the two branches are not completely independent during parallel upsampling; they interact at several intermediate stages, such as upsampling to 1 / 4 and 1 / 2 resolution. Structure-to-detail guidance: The structure decoding branch provides its learned region semantic priors—for example, a probability map of a tumor region—as an attention guide map to the detail decoding branch at the same stage. The detail branch uses this information to determine which regions should have more refined boundaries, avoiding over-optimization on irrelevant backgrounds. Detail-to-structure feedback: The detail decoding branch feeds back its generated boundary confidence map to the structure decoding branch at the same stage. The structure branch uses this information to correct any coarse or inflated region predictions, aligning the structural boundaries with more precise details.

[0158] The two branches ultimately output their respective feature maps, focusing on macroscopic structure and microscopic details, respectively. These two outputs will be fed into the subsequent S108 step adaptive gating fusion module for final fusion, generating a preliminary segmentation prediction map.

[0159] Furthermore, in step S108, the feature maps output by the dynamically weighted fusion structure decoding branch and the detail decoding branch are used to generate a preliminary segmentation prediction map, including:

[0160] A content-aware dynamic gating fusion network is constructed. The dynamic gating fusion network takes the feature maps output by the structure decoding branch and the feature maps output by the detail decoding branch as inputs. A lightweight convolutional module simultaneously analyzes the semantic confidence and boundary clarity of the feature maps of the two branches in spatial location, and generates a pair of spatially adaptive fusion weight maps accordingly.

[0161] Specifically, the Dynamic Gated Fusion Network is a lightweight decision module used to evaluate the value of each input feature at each spatial location. Input: Receives two feature maps. Derived from structural branches, rich in region-level semantics and Derived from detailed branches, rich in pixel-level boundary information.

[0162] The lightweight convolution module has the following structure: it contains two convolutional layers. The first layer has a 3×3 kernel size, 16 channels, and uses ReLU activation. The second layer has a 3×3 kernel size, 2 channels, and uses Sigmoid activation. This module will... and The input is concatenated along the channel dimension, and the output is a weight map of the two channels.

[0163] Weighted graph generation and quantization calculation methods:

[0164] Semantic confidence and boundary clarity The quantization calculation is as follows:

[0165]

[0166]

[0167] Here, Concat represents the channel concatenation operation. This indicates that the second convolutional layer is activated with Sigmoid. This indicates that the first convolutional layer is activated with ReLU. and These represent the 0th and 1st channels of the output tensor, respectively.

[0168] The specific calculation formula for generating the fused weighted graph is as follows:

[0169]

[0170]

[0171] in, It is set to a very small value, such as 1e-6, to prevent the denominator from being zero. and Satisfying constraints Element-wise addition. This means that at each pixel location, the contributions of the two branches are complementary and normalized.

[0172] Decision logic: This lightweight convolutional module is trained to identify: 1. Semantic confidence: in which regions, such as within an organization... 1. Semantic information is reliable and dominant; 2. Boundary clarity: in which areas such as cell and gland outlines... The boundary information is precise and critical.

[0173] Based on the fusion weight map, the feature maps of the two branches are summed element-wise with weights to obtain the fused features. :

[0174]

[0175] This allows the network to assign higher weights to structural branch features in regions with high semantic consistency requirements within the organization, and higher weights to detail branch features in detail-sensitive regions such as cell boundaries and gland contours, achieving adaptive fusion at the feature level. Within the organization, in regions with high semantic consistency requirements, the network learns to... Approaching 1, Approaching zero, thus trusting and preserving the coherent semantic information provided by structural branches. In detail-sensitive areas such as cell boundaries and gland contours, the network learns to make... Approaching 1, Approaching 0 allows for trust in and enhancement of the sharp boundary information provided by detailed branches.

[0176] The fused feature map is input into a lightweight prediction head, which consists of several convolutional layers: a small number of convolutional layers, such as one to three 3×3 convolutions, and a final 1×1 convolutional layer. This layer maps the high-dimensional features to an initial probability distribution corresponding to different pathological tissue categories for each pixel. The prediction head outputs a preliminary segmentation prediction map. These convolutional layers fine-tune and integrate the fused features, while the final 1×1 convolutional layer maps the high-dimensional feature channels to the same number of channels as the number of pathological tissue categories. Each channel corresponds to an unnormalized score (logits) for one category. The prediction head outputs an initial probability distribution for each pixel belonging to each category, implicitly calculated in the loss function, after passing through a Softmax function. This initial segmentation prediction map integrates the global structure and local details, forming the basis for subsequent refinement.

[0177] Furthermore, in step S109, the online pseudo-label refinement and self-training based on the preliminary segmentation prediction map includes:

[0178] A pseudo-label quality assessment and calibration mechanism based on uncertainty estimation is introduced. The uncertainty of the initial segmentation prediction map is estimated at the pixel level, the prediction confidence of each pixel is calculated, and a dynamic prediction confidence threshold is set according to the prediction confidence. Only high-confidence regions with uncertainty below the dynamic prediction confidence threshold are retained as initial pseudo-labels.

[0179] Specifically, methods such as Monte Carlo Dropout or multi-scale test-time augmentation are used to perform multiple forward propagations on the same image, resulting in several slightly different prediction maps. The uncertainty of the prediction for a pixel is quantified by calculating the variance or entropy of the class probability distribution of these predictions at each pixel. High variance or high entropy indicates that the model's prediction for that pixel is unstable and has low confidence.

[0180] The formula for calculating the dynamic threshold or the dynamic adjustment strategy is as follows:

[0181] set up This represents the set of uncertainties for all pixels in the current batch. for The mean, for The standard deviation. Dynamic threshold. Defined as:

[0182]

[0183] in, In this embodiment, the adjustment coefficient is... The value is set to 1.5. This formula ensures that the threshold dynamically fluctuates with the overall uncertainty level of the current batch of predictions: when the overall model prediction is uncertain, the threshold automatically increases to filter out a very small number of high-confidence pixels; when the model prediction is stable, the threshold decreases to utilize more pixels.

[0184] Only retain those with less uncertainty The pixel predictions are used as high-confidence pseudo-labels. These pixels are usually located within regions that the model is already confident about, and their pseudo-labels, i.e., the categories predicted by the model, are relatively reliable.

[0185] A historical consistency calibration module is constructed to compare the currently generated high-confidence pseudo-labels with a prediction generated by historical network parameters on the same image. Only regions with consistent predictions are retained, and the probability distribution of pseudo-labels in these regions is weighted and averaged to filter out temporary prediction noise and obtain smooth and stable calibrated pseudo-labels.

[0186] Specifically, an exponential moving average (EMA) model is maintained, whose parameters are the moving average of the current model's parameters. This EMA model represents a more stable and less noisy historical version during training. The same image is input into both the current model and the EMA model, resulting in two sets of predictions. The obtained high-confidence pseudo-labels are compared pixel-by-pixel with the EMA model's predictions. Only pixel regions whose predicted categories are consistent are retained. The probability distribution of the pseudo-labels in these regions is weighted and averaged from the soft labels of the EMA model to obtain the final calibrated pseudo-labels. This operation filters out temporary erroneous predictions caused by the randomness of the current model.

[0187] A course learning strategy is adopted to dynamically adjust the utilization intensity of calibrated pseudo-labels. In the early stage of training, strong supervision loss is relied on labeled data and high-confidence pseudo-labels. As training progresses, weak supervision loss on calibrated pseudo-labels is gradually introduced, and the weight of self-training loss in the total loss is adaptively adjusted to make use of unlabeled or weakly labeled data.

[0188] Specifically, a dynamic weighting function is designed to control the weight of the self-training loss calculated based on calibrated pseudo-labels in the total loss. In the early stages of training, this weight is set to 0 or very small, and the model primarily relies on labeled data for learning. As the training epochs increase, the model's capabilities improve, and the quality of the pseudo-labels increases, this weight gradually increases linearly or according to a predetermined schedule. This allows the model to learn robustly and progressively from its own generated, increasingly higher-quality pseudo-labels.

[0189] For example, a dynamic weighting function is as follows:

[0190]

[0191] in, This indicates the number of iterations in the current training round; This represents the total number of preset warm-up rounds, i.e., the number of rounds required for the weight to increase linearly from its initial value to its maximum value; This represents the initial weight, which is usually set to 0 or a small value such as 0.001; This represents the preset maximum value of the weight of the loss item, which is 0.5 in this embodiment.

[0192] For example, another dynamic weighting function is:

[0193]

[0194] in, This indicates the number of iterations in the current training round; It is the center point of the function, corresponding to the cycle in which the midpoint of the S-curve is located; Is it controlling growth or transitioning from 0 to... The parameter of the rate; This indicates the preset maximum value of the weight of the loss item.

[0195] The supervised loss calculated based on labeled data is combined with the self-training loss calculated based on calibrated pseudo-labels to form a hybrid training objective. All trainable network parameters of the master encoder, prior encoder and multi-scale decoupled decoder are jointly updated through the backpropagation algorithm.

[0196] Among these, supervised losses calculated based on labeled data include cross-entropy loss or Dice loss, commonly used in segmentation tasks. Self-training losses calculated based on calibrated pseudo-labels, and losses calculated for unlabeled or weakly labeled data, typically also use cross-entropy. The total loss formula is:

[0197]

[0198] in, These are the dynamically adjusted course learning weights in step 3. Minimize them using the backpropagation algorithm. At the same time, the parameters of all components, including the master encoder, prior encoder, and multi-scale decoupled decoder, are updated.

[0199] It should be noted that in actual training, the gradient update applied in step S109 may also include: the parameters of the learnable biological prior decoupling network from S103; the parameters of cross-modal interactive fusion modules such as dynamic attention weight generation networks and adaptive gating fusion units from S105; the parameters of the supervised contrastive learning head, i.e., the feature projection head, from S106; the parameters of the dynamic gating fusion network and the lightweight prediction head from S108; and any other trainable layers defined in the network for feature transformation, alignment, or prediction, such as deformable convolutional layers and specific attention modules.

[0200] This application also provides a pathological slide image segmentation device based on multimodal supervised contrastive learning, the device comprising:

[0201] The image acquisition and preprocessing module is used to acquire pathological slide images and perform standardized preprocessing on them to obtain preprocessed images. Standardized preprocessing includes color normalization and background removal.

[0202] The feature extraction module crops the preprocessed image into multiple image patches and decouples the corresponding biological prior feature map from each image patch. The image patches are input into a main encoder to extract visual features, and the corresponding biological prior feature map of the same image patch is input into a prior encoder to extract semantic features.

[0203] The feature fusion module performs cross-modal interaction and fusion of visual features extracted by the main encoder and semantic features extracted by the prior encoder to obtain fused cross-modal features. An attention mechanism is employed, using semantic features as keys and values, and visual features as queries, to achieve feature alignment and enhancement.

[0204] The contrastive learning module performs supervised contrastive learning after the main encoder: it applies two independent data augmentation transformations to image patches to generate a pair of augmented views, and computes a supervised contrastive loss based on the features of the pair of augmented views. The supervised contrastive loss is used to bring closer the feature distances of augmented views from the same image patch and to bring closer the feature distances of samples with the same pathology category label, while pushing away the feature distances of samples from different pathology categories, in the feature projection space.

[0205] The decoding and prediction module takes cross-modal features as input to a multi-scale decoupled decoder for decoding and reconstruction. The multi-scale decoupled decoder includes a structural decoding branch and a detail decoding branch. The structural decoding branch processes deep cross-modal features and is supervised by a structural similarity loss function to learn semantic consistency at the region level. The detail decoding branch processes shallow cross-modal features and is supervised by a boundary-aware loss function to learn pixel-level boundary accuracy. The feature maps output from the structural and detail decoding branches are dynamically weighted and fused to generate an initial segmentation prediction map.

[0206] The training optimization module is used for online pseudo-label refinement and self-training based on the preliminary segmentation prediction map: high-confidence pseudo-labels are generated based on the currently generated preliminary segmentation prediction map, and the high-confidence pseudo-labels are calibrated by combining the exponential moving average output corresponding to the historical network parameters. The network parameters of the master encoder, prior encoder and multi-scale decoupled decoder are updated using the calibrated pseudo-labels.

[0207] It should be noted that the terms "first," "second," and similar terms used in this application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, "one" or similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. "Multiple" or "several" indicates at least two. Unless otherwise stated, terms such as "front," "back," "left," "right," "lower," and / or "upper" are for illustrative purposes only and are not limited to a location or spatial orientation. Terms such as "include" or "contain" indicate that the elements or objects preceding "include" cover the elements or objects listed following "include" or their equivalents, and do not exclude other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0208] The singular forms "a," "say," and "this" used in this application specification and appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms used herein refer to and / or include any or all possible combinations of one or more associated listed items.

[0209] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A pathological slide image segmentation method based on multimodal supervised contrastive learning, characterized in that, include: S101, acquire pathological slide images; S102, Standardize the preprocessing of pathological slide images; S103, the preprocessed image is cropped into multiple image blocks, and the corresponding biological prior feature map is generated from each image block; S104, input the image patch into the main encoder to extract visual features, and input the corresponding biological prior feature map into the prior encoder to extract semantic features; S105 performs cross-modal fusion of visual and semantic features, employs an attention mechanism, uses semantic features as keys and values ​​and visual features as queries, to complete feature alignment and enhancement; S106, Supervised contrastive learning is performed after the main encoder: two data augmentations are applied to the image patch to generate view pairs, and supervised contrastive loss is calculated to bring similar samples closer together and push apart the feature distance of dissimilar samples. S107, cross-modal features are input into the multi-scale decoupled decoder. The structural decoding branch of the multi-scale decoupled decoder is supervised by structural similarity loss to learn region semantics, and the detail decoding branch is supervised by boundary awareness loss to learn boundary details. S108, the output features of the dynamic weighted fusion structure decoding branch and the detail decoding branch are used to generate a preliminary segmentation prediction map; S109 generates high-confidence pseudo-labels based on the preliminary segmentation prediction map, calibrates them by combining the exponential moving average output of historical parameters, and updates the network parameters of the master encoder, prior encoder, and decoder using the calibrated pseudo-labels.

2. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 1, characterized in that, Step S102, performing standardization preprocessing on the pathological slide images, including: Color normalization is performed using the Macenko method based on color deconvolution or a style transfer network based on deep learning to eliminate color deviations caused by staining and scanning, thereby ensuring a consistent color distribution in the image. An adaptive threshold is set by calculating the color intensity, saturation, or texture features of the image to identify and remove background areas below the threshold, retaining only biological tissue areas; Output a preprocessed image that has undergone color normalization and background removal.

3. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 1, characterized in that, Step S103 involves cropping the preprocessed image to obtain multiple image patches and generating corresponding biological prior feature maps, including: Adaptive mesh generation is performed based on tissue saliency detection. A smaller cropping step size is used in dense tissue regions and a larger step size is used in sparse regions to balance detail preservation and computational efficiency, thereby obtaining multiple image patches. Each image patch is input into a learnable biological prior decoupling network. This network is trained under the constraints of a pixel-level self-supervised reconstruction task and a gradient-guided task connected to the main segmentation network through an encoder-decoder structure. This adaptively decouples biological prior feature maps that preserve the original morphological semantics while enhancing the information related to the segmentation of the target pathological structure.

4. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 3, characterized in that, Step S104: Input the image patch into the main encoder to extract visual features, and input the corresponding biological prior feature map into the prior encoder to extract semantic features, including: Construct a deep heterogeneous dual encoder, including a master encoder and a prior encoder; The main encoder employs a deep convolutional neural network pre-trained on a natural image dataset to extract visual features from image patches. The prior encoder shares some of the underlying weights with the main encoder, and is used to extract semantic features from the biological prior feature map. A deformable convolutional cross-modal alignment module is introduced between the corresponding feature layers of the master encoder and the prior encoder. The deformable convolutional cross-modal alignment module uses the preliminary alignment result of the previous layer as a supervision signal. By calculating the spatial correlation between the two feature maps, it adaptively learns the offset field and guides the prior encoder to achieve progressive alignment with the visual features of the master encoder at the pixel level when extracting semantic features.

5. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 4, characterized in that, Step S105 involves cross-modal interaction and fusion of visual and semantic features, including: Parallel execution of cross-modal interaction fusion at multiple scales: At each feature level, visual features are used as queries and semantic features are used as keys and values, which are then input into a dynamic attention weight generation network. The generated network generates attention weights based on the local spatial correlation between visual and semantic features. The generated attention weights are used to weight and aggregate semantic features to obtain context-enhanced semantic information; Through a spatially and channel-adaptive gated fusion unit, the enhanced semantic information is fused with the original visual features element by element to adaptively determine the information retention ratio of each position and channel. Finally, a feature pyramid network with skip connections is used to integrate and upsample the fused features from each layer, outputting cross-modal features that deeply fuse visual details and prior semantics.

6. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 4, characterized in that, Step S106, performing supervised contrastive learning after the main encoder, including: Two independent pathological-sensory data enhancements are applied to image patches to generate a pair of enhanced views, with enhancement parameters dynamically adjusted based on tissue type; The two augmented views are fed into the main encoder, and the high-dimensional visual features are mapped to the low-dimensional contrastive learning space of the unit hypersphere through an additional feature projection head. Supervised contrastive loss is computed in the contrastive learning space: Supervised contrastive loss not only brings the features of two enhanced views from the same image patch closer together, but also uses pathological labels to bring the features of samples of the same class closer together and push the features of samples of different classes further apart; Meanwhile, an attention-weighted hard negative sample mining strategy is adopted to impose a greater penalty on hard negative sample pairs that are similar in features to anchor samples but different in category; By minimizing the supervised contrast loss, the main encoder is driven to learn feature representations that are robust to staining differences and imaging artifacts and have strong discriminative power for different pathological structures.

7. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 1, characterized in that, Step S107 involves inputting the cross-modal features into a multi-scale decoupled decoder for decoding and reconstruction, including: Construct a multi-scale decoupled decoder with a bidirectional information guidance mechanism, including a structural decoding branch and a detail decoding branch; The structural decoding branch takes deep cross-modal features as input, recovers resolution through upsampling and residual connections, and uses a region-aware attention module to focus on the overall morphology of the tissue region. The training of the structural decoding branch is supervised by a loss function that calculates a multi-scale structural similarity index to learn region-level semantic consistency. The detail decoding branch takes shallow cross-modal features as input, preserves high-resolution details through lightweight convolution and skip connections, and integrates an edge enhancement module to strengthen high-frequency boundary information. The training of the detail decoding branch is supervised by a loss function that combines weighted cross-entropy and boundary FocalLoss to learn pixel-level boundary accuracy. During the decoding process, a progressive feature interaction strategy is introduced: in multiple upsampling stages, the structural decoding branch outputs the learned regional semantic priors to the detail decoding branch through the attention weight map to guide the focusing of detail features; the detail decoding branch, in turn, feeds back the boundary confidence map it generates by applying the Sigmoid function to the structural decoding branch to correct the region prediction; this bidirectional interaction is achieved through a cross-branch attention module, in which the output features of one branch are adjusted by convolution and then fused into the corresponding features of another branch in the form of element-wise multiplication or addition.

8. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 7, characterized in that, Step S108, dynamically weighting and fusing the output features of the structural decoding branch and the detail decoding branch to generate a preliminary segmentation prediction map, includes: Construct a content-aware dynamic gating fusion network to receive the feature maps output by the structure decoding branch and the detail decoding branch; A lightweight convolutional module is used to analyze the dual-branch feature map and generate a pair of spatially adaptive fusion weight maps to quantify the degree of semantic consistency and boundary details required at each location. Based on the fusion weight map, the bi-branch feature map is summed element by element, thereby giving higher weight to structural branch features in the internal tissue region and higher weight to detailed branch features in the cell boundary and gland contour region. The fused feature map is input into a lightweight prediction head, which outputs the initial probability distribution of each pixel corresponding to different pathological tissue categories, i.e., the preliminary segmentation prediction map.

9. The pathological slide image segmentation method based on multimodal supervised contrastive learning according to claim 1, characterized in that, Step S109, based on the preliminary segmentation prediction map, performs online pseudo-label refinement and self-training, including: Pixel-level uncertainty estimation is performed on the preliminary segmentation prediction map, and a dynamic threshold is set to filter out high-confidence regions and generate initial pseudo-labels. The current initial pseudo-label is compared with the prediction generated by the exponential moving average (EMA) model of historical network parameters. Noise is filtered out by consistency filtering and probability weighted averaging to obtain the calibrated smooth pseudo-label. The training objective is dynamically adjusted using a course learning strategy: in the early stage of training, strong supervised loss based on labeled data is relied upon, and weak supervised loss based on calibrated pseudo-labels is gradually introduced as the training progresses, and the mixed weights of strong supervised loss and weak supervised loss are adaptively adjusted. Finally, a hybrid training objective is formed by combining the strongly supervised loss with labeled data and the self-training loss with calibrated pseudo-labels, and the network parameters of the master encoder, prior encoder and multi-scale decoupled decoder are jointly updated through backpropagation.

10. A pathological slide image segmentation device based on multimodal supervised contrastive learning, characterized in that, include: The data preprocessing module is used to acquire pathological slide images and perform standardized preprocessing, including color normalization and background removal. The feature generation module is used to crop the preprocessed image into multiple image patches and decouple the corresponding biological prior feature map from each image patch. The feature extraction and fusion module includes a main encoder and a prior encoder, which are used to extract visual features from image patches and semantic features from biological prior feature maps, respectively, and perform cross-modal fusion through an attention mechanism, where semantic features are used as keys and values ​​and visual features are used as queries to complete feature alignment and enhancement. The supervised contrastive learning module is used to perform supervised contrastive learning after the main encoder. It generates view pairs by applying data augmentation to image patches and calculates supervised contrastive loss to bring the feature distance of similar samples closer and push the feature distance of dissimilar samples further apart. The decoding and prediction module includes a multi-scale decoupled decoder, which inputs cross-modal features into the structural decoding branch and the detail decoding branch, respectively supervised by structural similarity loss and boundary-aware loss, to learn region semantics and boundary details, and dynamically weights and fuses the output features of the two branches to generate a preliminary segmentation prediction map; The self-training optimization module is used to generate high-confidence pseudo-labels based on the initial segmentation prediction map, calibrate them by combining the exponential moving average output of historical parameters, and update the network parameters of the master encoder, prior encoder and multi-scale decoupled decoder using the calibrated pseudo-labels.

Images