An innovative self-supervised based pathological image virtual staining method and system

Abstract

The present application belongs to the technical field of digital pathological image processing, and particularly relates to an innovative pathological image virtual staining method based on self-supervision. By introducing a task-independent self-supervised pre-training stage before the virtual staining task, the encoder can autonomously learn strong robust features which are highly sensitive to the structure of the tissue morphology and invariant to the staining style and registration error from a large number of unannotated or weakly annotated pathological images. Combined with the strong encoder of the self-supervised pre-training, a multi-scale decoding method is designed. This method can fully integrate the pathological features captured at different levels by the encoder and enhanced by pre-training, and finally generate images in multiple scales in cooperation, thereby significantly improving the structural fidelity, detail clarity and visual authenticity of the generated images.

Description

Technical Field

[0001] This invention belongs to, but is not limited to, the field of digital pathological image processing technology, and particularly relates to an innovative virtual staining method and system for pathological images based on self-supervision. Background Technology

[0002] Virtual staining technology aims to achieve the conversion between different pathological staining modalities through computational models, such as generating IHC staining images that reveal the expression of specific proteins from widely used but information-limited H&E staining. This technology can significantly reduce the time and economic costs of IHC staining and promote multimodal pathological analysis. With the development of deep learning, especially generative adversarial networks and contrastive learning, this field has made significant progress. However, existing methods still face serious challenges in addressing the core issue of how to learn robust and discriminative pathological semantic features from limited and imperfect supervised data. The following is an analysis of representative technical approaches and their limitations:

[0003] Generative models based on strictly pixel-paired supervision, represented by Pix2Pix and its variants (such as Pyramid Pix2Pix with Gaussian pyramid loss), rely on training with pixel-level precisely aligned H&E-IHC image pairs. Their core principle is learning a deterministic mapping from the source domain to the target domain. However, obtaining such perfectly paired data is extremely difficult in practice because sequentially slicing and staining the same tissue block introduces tissue deformation, tearing, and registration errors, leading to the so-called "inconsistent ground truth pair" problem. Directly using such data and applying pixel-level reconstruction losses (such as L1 and L2 losses) causes the model to learn noise patterns caused by alignment errors, severely impairing the pathological accuracy and structural fidelity of the generated images. The fundamental limitation of these methods lies in their over-reliance on unrealistically high-quality data and strong supervisory signals.

[0004] Unpaired image translation models based on cycle consistency, represented by CycleGAN and its derivatives, eliminate the dependence on paired data by introducing cycle consistency loss, requiring only independent image sets of two staining domains for training. This alleviates the pressure of data acquisition to some extent. However, the cycle consistency constraint is a macroscopic and relatively weak constraint, which can easily lead to structural distortions and loss of semantic information at the cellular and tissue levels in complex pathological image translation. The model may find "shortcuts" that satisfy the "cycle" but are pathologically incorrect, such as blurring key lesion areas or generating unreasonable staining patterns. The fundamental problem lies in the lack of explicit modeling and constraints on the inherent pathological structure of the tissue itself, and the lack of clear pathological semantic guidance in the learning process.

[0005] Recent improvements that incorporate contrastive learning and adaptive supervision:

[0006] To address the problem of inconsistent ground truth pairs, recent research has shifted towards combining contrastive learning with adaptive mechanisms. The ASP adaptive supervised contrastive loss method proposes an adaptive supervised PatchNCE loss. Its core principle is to leverage the robustness of contrastive learning to label noise, bringing generated image patches closer to (potentially inconsistent) real IHC image patches in the feature space, while distancing them from non-corresponding image patches. Furthermore, it dynamically reduces the contribution of inconsistent regions to the loss through adaptive weighting based on the similarity between generated and real patches. This represents an approach to mining relatively correct relationships from noisy supervision signals. Methods based on weak pathological consistency constraints: Another type of work utilizes the weak correspondences of consecutive slices to design multi-level weak pathological consistency constraints. For example, through contrastive learning or Jensen-Shannon divergence, it constrains the consistency of the pathological relationship distribution between generated and actual IHC images in the feature space, and introduces adaptive weights to address alignment differences at different levels. The PSPStain pathological semantic preservation learning method goes a step further, directly modeling molecular-level pathological semantics (such as protein expression levels). By calculating optical density maps and designing focal optical density maps, the consistency of protein expression distribution between generated and real images is explicitly constrained. Simultaneously, prototype consistency learning is introduced to enhance tumor semantic alignment across images.

[0007] While the aforementioned recent methods have made valuable explorations in utilizing imperfect supervised data and incorporating pathological semantics, they share a deeper bottleneck: the learning objectives of their core feature extractors (encoders) are either constrained by noisy pixel / block-level supervision or rely on complex handcrafted constraints designed for specific downstream tasks (such as protein expression consistency). Methods like ASP directly compare the encoder's feature learning with ground truth pairs that may contain significant mismatch information; the quality of the feature space heavily depends on the accuracy of the adaptive weight estimation, which can be unreliable early in training or in highly inconsistent regions. Pathological semantic constraint methods, while introducing valuable pathological priors, often impose additional objectives (such as optical density and pathological relation distribution) on top of the features already extracted by the model. The encoder itself is not specifically designed to extract these robust, staining-pattern-independent pathological features, leading to a gap between feature learning and high-level semantic constraints, potentially causing the optimization process to get trapped in local optima. The root cause is that existing methods fail to employ a task-agnostic, self-driven pre-training approach in the initial stages of model training. This allows the encoder to autonomously learn general pathological feature representations from massive, easily accessible unlabeled or weakly labeled pathological images. These representations are capable of decoupling tissue morphology and staining patterns, are sensitive to lesion regions, and are robust to registration errors. Without such a robust, pre-trained "feature foundation," subsequent translation models must learn feature extraction and domain transformation simultaneously from scratch or under noisy supervision, significantly increasing the learning difficulty and limiting the model's generalization ability and final performance.

[0008] Therefore, there is an urgent need in this field for a new method that can pre-learn high-quality and robust pathological image feature representations, so as to provide a stable, reliable and semantically rich feature foundation for virtual staining tasks, thereby fundamentally improving the performance, robustness and generalization ability of virtual staining models. Summary of the Invention

[0009] The purpose of this invention is to overcome the above-mentioned defects in existing virtual staining techniques for pathological images. In particular, it addresses the problems of existing methods being sensitive to noise supervision, having insufficient ability to preserve pathological semantics, and having weak model generalization due to the lack of robust and universal pathological feature representation. Therefore, this invention proposes an innovative virtual staining method based on self-supervised feature extraction.

[0010] This invention is implemented as follows: an innovative virtual staining method for pathological images based on self-supervision, which includes a first stage of self-supervised pre-training of the encoder and a second stage of adversarial fine-tuning of the overall model;

[0011] (1) Data preparation

[0012] The required data includes:

[0013] Virtual coloring training data: two sets of images that do not require pairing, namely the source domain image set and the target domain image set; these data come from public datasets such as BCI and MIST; the images need to be preprocessed uniformly, including cropping to a fixed size and normalizing pixel values ​​to the range of [-1, 1];

[0014] (2) Self-supervised pre-training of the encoder, specifically including:

[0015] The goal of this stage is to train a robust pathological image feature extractor;

[0016] Step 2.1: Construct the contrastive learning task; for each image x in the pre-training dataset, generate two different views x_q and x_k through random data augmentation; augmentation strategies include: random color jitter, random Gaussian blur, random cropping and scaling back to the original size, etc.

[0017] Step 2.2: Feature extraction; The enhanced image pair (x_q, x_k) is input into the encoder E to be trained to obtain the corresponding feature vectors z_q = E(x_q) and z_k = E(x_k); In the calculation, a small projection head is usually connected after the encoder to map the features to a lower-dimensional space that is more suitable for contrastive learning.

[0018] Step 2.3: Calculate the contrast loss; use normalized temperature-scale cross-entropy loss; for samples within a batch, consider z_q and its corresponding z_k as positive sample pairs, and features of z_q and other samples within the batch as negative sample pairs; the loss function ensures that the similarity of positive sample pairs in the feature space is much higher than that of negative sample pairs; the loss function is as follows:

[0019] ,

[0020] in, For cosine similarity, Here, N represents the temperature hyperparameter, and N represents the batch size.

[0021] Step 2.4: Model training; Update the parameters of encoder E and projector head using the Adam optimizer; Training continues for multiple rounds until the loss converges; After training, retain the parameters of encoder E and discard the projector head;

[0022] (3) Construction and initialization of the overall model

[0023] Step 3.1: Construct the enhanced generator, which consists of two parts:

[0024] The encoder part directly loads and freezes the ContrastiveEncoder pre-trained in the first stage; its structure includes: input layer → two downsampling convolutional layers → multiple residual blocks; this part is responsible for extracting deep semantic features of the input image.

[0025] Decoder section: Create a new MultiScaleDecoder; its structure includes: multiple upsampling stages → output layer; each upsampling stage is as follows: 3x3 convolutional layer → PixelShuffle operation → instance normalization → ReLU activation; finally, after reflection padding and 7x7 convolutional layer, the image is output using the Tanh activation function; the decoder is responsible for reconstructing the target domain image from the features extracted by the encoder.

[0026] Step 3.2: Construct the complete CycleGAN framework; instantiate two enhancement generators: H&E → IHC and IHC → H&E; Simultaneously, instantiate two PatchGAN discriminators: Used to determine the authenticity of IHC images. Used to determine the authenticity of H&E images;

[0027] (4) Adversarial fine-tuning of the virtual staining model

[0028] In this stage, unpaired H&E and IHC image sets are used to train the complete model built in step 3 end-to-end.

[0029] Step 4.1: Forward propagation; for a batch of training data:

[0030] Generate image: , ;

[0031] Loop reconstruction image: , ;

[0032] Identity mapping image: , ;

[0033] To maintain feature robustness: for and Random enhancements were performed separately to obtain... and and respectively through .encoder and .encoder extracts features;

[0034] Step 4.2: Loss Calculation; Total Loss It is the weighted sum of the following losses:

[0035] Combat loss ( ): Using least squares GAN loss encourages the generator to deceive the discriminator;

[0036] Cyclic consistency loss ( Using L1 loss, constraints and , and As close as possible;

[0037] Loss of identity ( (Optional): Use L1 loss, with constraints and , and Being as close as possible helps stabilize color mapping;

[0038] Contrast regularization loss ( ):calculate and Between features and The contrast loss between features is used as a regularization term; its purpose is to solidify the robust feature representations learned by the encoder and prevent them from degrading in the high-noise environment of adversarial training.

[0039] The total loss formula is: Among them, hyperparameters =10, =0.5, =0.1;

[0040] Step 4.3: Backpropagation and optimization; using an alternating optimization strategy:

[0041] Fixed discriminator and Update generator and Minimize the parameters ;

[0042] Fixed generator and Update the discriminator and The parameters are set to maximize its ability to distinguish between real and generated images;

[0043] The Adam optimizer (β1=0.5, β2=0.999) was used with an initial learning rate of 2e-4 and a linear decay strategy.

[0044] (5) Reasoning stage

[0045] In one specific embodiment, the HER2 staining task on the BCI dataset is used for validation;

[0046] Pre-training: All H&E images in the BCI dataset were used as pre-training data; the encoder E was a ContrastiveEncoder with 6 residual blocks; the Adam optimizer was used for 40 rounds of training with a batch size of 2 and an initial learning rate of 1e-3.

[0047] Fine-tuning: Load the pre-trained encoder and construct G_A and G_B; the discriminator is a 70x70 PatchGAN; train for 60 epochs on the unpaired HER2 data from BCI with a batch size of 1; other hyperparameters are as described above.

[0048] Furthermore, the core of this method lies in constructing an enhanced generator that integrates a self-supervised pre-trained encoder and a multi-scale decoder, and integrating it into a recurrent consistency generative adversarial network framework to achieve high-quality and robust conversion from H&E stained images to IHC stained images.

[0049] Furthermore, the self-supervised pre-trained encoder based on contrastive learning specifically includes:

[0050] The core innovation of this deep convolutional neural network structure lies in the introduction of a self-supervised pre-training stage that is decoupled from the downstream staining task. In this stage, the encoder is trained using only a large number of readily available pathological images that do not require paired annotations. The aim is to learn a general pathological feature representation that is highly sensitive to tissue morphology and structure while being invariant to changes in staining patterns.

[0051] Network Structure: This encoder uses ResNet as its basic architecture, consisting of an initial convolutional layer, several downsampling layers, and a series of residual blocks. Specifically, the input image first passes through a padding layer and a 7x7 convolutional layer to extract initial features. Subsequently, it is downsampled through two convolutional layers with a stride of 2, gradually expanding the receptive field and compressing the spatial dimension. After downsampling, multiple residual blocks are connected to learn deep, abstract feature representations. The final output of the encoder is a high-dimensional feature tensor.

[0052] Self-supervised pre-training task: Pre-training adopts a contrastive learning paradigm; for the input image Fm, two different augmented views Fm1 and Fm2 are generated through random data augmentation; these two views are fed into a weighted encoder E to obtain the corresponding feature representations. and The goal of pre-training is to bring features from different augmented views of the same original image closer together in the feature space, while simultaneously pushing features from different original images further apart. This goal is achieved by minimizing the Normalized Temperature Scale Cross-Entropy Loss (NT-Xent Loss).

[0053]

[0054] Where sim(·,·) represents the cosine similarity. This is a temperature hyperparameter; through this pre-training, the encoder learns to extract features that are robust to the essential structure of the tissue but insensitive to non-essential changes such as color and slight deformation.

[0055] Furthermore, the multi-scale decoder specifically includes:

[0056] The decoder is responsible for reconstructing the IHC staining image of the target domain from the abstract features extracted by the encoder; this improves the detail clarity and structural fidelity of the generated image.

[0057] Network Structure: The decoder employs an upsampling architecture, with its input being the feature tensor output by the pre-trained encoder. The core innovation lies in using pixel-recombined upsampling instead of traditional transposed convolution. Specifically, the decoder includes several upsampling stages. In each stage, a convolutional layer first expands the number of channels to four times the target number, then a PixelShuffle operation is applied for a 2x upsampling, effectively reducing checkerboard artifacts during the upsampling process. After upsampling, a normalization layer and activation function are applied sequentially. After multiple upsampling stages, the feature map is restored to the spatial dimensions of the original input image. Finally, a 7x7 convolutional layer and a Tanh activation function are used to output the final IHC image.

[0058] Multi-scale feature fusion: The multi-scale decoding mentioned above is essentially about the decoder starting from the deep, low-resolution but semantically rich features output by the encoder, and gradually recovering spatial details through progressive upsampling; the multi-level features captured by the encoder during downsampling are implicitly used to guide image generation at different scales through the upsampling path of the decoder, ensuring consistency from global structure to local details.

[0059] Furthermore, the enhancement generator is integrated into the overall training process, specifically including:

[0060] The pre-trained encoder and multi-scale decoder are combined to form an augmentation generator G; then, the augmentation generator is embedded into the standard CycleGAN framework for end-to-end fine-tuning training.

[0061] Overall framework: The overall model contains two enhancement generators. and and two discriminators (Discrimination of IHC domain) and (Discrimination of H&E domain); Training process does not require pixel-level paired images;

[0062] Loss function: The overall training objective is composed of the weighted sum of the following loss components:

[0063] (1) Adversarial loss: Makes the generated image indistinguishable in the target domain; ;

[0064] (2) Cyclic consistency loss: ensures that image transformation is reversible; ;

[0065] (3) Identity loss: Encourages the generator to keep the input unchanged when the input is already a target domain image, which helps to stabilize the color distribution;

[0066] (4) Self-supervised contrastive loss: During the fine-tuning stage, features are extracted from the input image and its augmented view and the contrastive loss is calculated as a regularization term to maintain the robust feature representation learned by the encoder during the pre-training stage and prevent feature degradation during adversarial training. ;

[0067] Training steps:

[0068] (1) Encoder pre-training stage: Collect a large number of unpaired pathological images; construct a contrastive learning task based on NT-Xent Loss, and pre-train the encoder network independently until the loss converges;

[0069] (2) Overall model initialization: Freeze or load the pre-trained encoder weights, combine them with the randomly initialized multi-scale decoder, and construct the boost generators G_A and G_B; initialize the discriminators D_A and D_B;

[0070] (3) Adversarial fine-tuning phase: using unpaired H&E and IHC image datasets; in each training iteration:

[0071] a. Forward propagation: , , , ;

[0072] b. Generate an enhanced view for the contrastive loss and extract features: , ;

[0073] c. Calculate the total generator loss L_G and backpropagate to update the parameters of G_A and G_B;

[0074] d. Calculate the discriminator losses L_D_A and L_D_B, and backpropagate to update the parameters of D_A and D_B;

[0075] (4) Inference stage: For the input H&E image, the virtual stained IHC image can be obtained by using the trained generator G_A for forward propagation.

[0076] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0077] 1. Significantly improves the visual quality and structural integrity of generated images.

[0078] This invention introduces pre-training with self-supervised contrastive learning, enabling the encoder to learn robust features that are sensitive to the intrinsic structure of tissues and invariant to staining patterns. Combined with a multi-scale decoder for progressively refined feature reconstruction, this method achieves breakthroughs in both the visual realism and quantitative metrics of the generated images. Figure 2 As shown, the IHC images generated by this invention (CycleGAN_MaM) outperform other comparative methods in terms of the clarity and coherence of nuclear morphology, chromatin distribution, and specific staining regions (brown staining regions), exhibiting the highest visual consistency with real IHC staining results. Quantitative evaluation (as shown in Table 1) further confirms this advantage: on the authoritative BCI dataset, the peak signal-to-noise ratio (PSNR) of this method reaches 28.5108, and the structural similarity index (SSIM) reaches 0.6747, both significantly surpassing all existing state-of-the-art methods (CUT, PyramidP2P, ASP, PSPStain, Li et al.). The higher PSNR and SSIM values ​​objectively demonstrate the superior ability of this method to reduce pixel-level errors and preserve image structural information.

[0079] Table 1 Quantitative Assessment Table

[0080] Method PSNR↑ SSIM↑ FID↓ CUT[1] 18.1246 0.4483 65.0 PyramidP2P[2] 19.9488 0.4647 80.1 ASP[3] 17.8651 0.4923 54.3 PSPStain[4] 18.6220 0.4498 Li[5] 19.132 0.499 50.1 CycleGAN_MaM 28.5108 0.6747 38.9802

[0081] 2. Significantly enhances the realism and domain adaptability of generated results.

[0082] The Frechet Inception Distance (FID) is an important indicator measuring the distance between the generated image distribution and the real image distribution. A lower FID value indicates better overall realism and intra-domain adaptability of the generated image. As shown in Table 1, the FID value of our method is only 38.9802, significantly lower than other comparative methods. This is attributed to the self-supervised pre-training enabling the encoder to learn a more general feature distribution within the pathological image domain, and the multi-scale decoder's ability to more faithfully decode these features into images conforming to the statistical characteristics of the target domain. A lower FID value means that the virtual IHC images generated by our method are visually closer to real IHC staining results, thus possessing higher clinical reference value.

[0083] 3. Effectively solves the problem of dependence on noisy supervision data, improving model robustness.

[0084] Existing advanced methods such as ASP and PSPStain require complex loss functions (e.g., adaptive weighting, pathological semantic constraints) to mitigate the impact of noisy supervision when dealing with inconsistent paired data. In contrast, the core advantage of this invention lies in its pre-implemented self-supervised feature learning stage. This stage utilizes a large amount of unlabeled data and, through contrastive learning, enables the encoder to master tissue structure features unaffected by specific staining patterns. As shown in Table 1, this method outperforms methods like ASP, which specifically address the "inconsistent ground truth pair" problem, without designing complex components specifically for it. This demonstrates that by acquiring robust low-level feature representations, the model fundamentally reduces its reliance on noisy pixel-level supervision, resulting in a more stable training process and greater tolerance to data defects.

[0085] 4. Achieving comprehensive leadership in key performance indicators validates the effectiveness of the technical solution.

[0086] Table 1's comprehensive comparison shows that this invention achieves superior performance in all three core metrics for evaluating virtual staining effectiveness from different dimensions: PSNR (pixel accuracy), SSIM (structural similarity), and FID (distribution realism). This comprehensive lead is not achieved through improvements to a single module, but rather stems from the systematic innovation of the synergistic effect of the "self-supervised pre-trained encoder" and the "multi-scale decoder." This strongly demonstrates that the technical solution of this invention—namely, first learning general pathological features through a self-supervised task, and then using multi-scale decoding for refined generation within an adversarial framework—is an effective way to significantly improve the performance of virtual staining tasks with unpaired data.

[0087] Experimental conditions and methods: The above beneficial effects are validated experimentally on the publicly available breast cancer immunohistochemistry challenge dataset. The division of the training and test sets is consistent with the comparative literature to ensure fairness in the comparison. All experiments were conducted in the same hardware environment (NVIDIA RTX 4080 GPU) and software framework (PyTorch). During evaluation, PSNR and SSIM were calculated based on paired images, while FID was calculated based on the depth feature distribution of all images in the test set. The training of this method follows the two-stage process described above: first, the encoder is self-supervised pre-trained for 40 rounds on 13580 unlabeled pathological images; then, the overall CycleGAN_MaM model is fine-tuned end-to-end for 60 rounds on unpaired images in the BCI dataset, with a batch size of 1 and the Adam optimizer used.

[0088] Secondly, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0089] (1) In the field of digital pathology and computational staining, a long-standing core contradiction is how to achieve high-fidelity and high-semantic-accuracy virtual staining without relying on perfectly matched data (which is difficult to obtain in practice). Existing technologies, in order to solve the problem of "inconsistent ground truth pairs," all follow the approach of "patching within a given model framework," for example:

[0090] Path A (Post-processing): Methods such as ASP design complex adaptive weighting mechanisms on the already noisy supervisory signal, attempting to "identify and ignore" erroneous parts during training. This is like trying to distinguish speech in a noisy environment, with limited and unstable results.

[0091] Path B (Increased Constraints): Methods such as PSPStain introduce additional semantic constraint modules that require prior knowledge or complex computations (such as optical density decomposition and prototype extraction) to attempt to "correct" the generated results from a higher level. This increases the system complexity and training difficulty.

[0092] None of the aforementioned approaches address the root cause of the problem—the vulnerability of the model's fundamental feature extraction capabilities to noisy data. This invention recognizes this fundamental issue and proposes a disruptive "pre-processing reinforcement" approach: before the model encounters noisy staining transformation tasks, a powerful feature extractor (encoder) sensitive to the essential pathological structure is pre-forged using massive amounts of readily available unlabeled data through self-supervised contrastive learning. This approach directly overcomes the fundamental technical challenge of "feature learning being constrained by noisy supervision," thus ensuring the robustness of subsequent generation tasks from the outset. This invention is the first to successfully and systematically introduce self-supervised learning into virtual staining tasks in a purposeful manner, demonstrating that the paradigm of "learning general features first, then learning specific transformations" can stably and significantly outperform all technical solutions that improve upon existing paradigms, solving a long-sought-after but unsuccessful goal in the industry.

[0093] (2) In the technical field to which this invention pertains, there exists a prevalent and deeply ingrained bias of “task-specific training.” This bias holds that for complex and sophisticated image-to-image translation tasks such as H&E to IHC virtual staining, all components of the model, especially the feature extractor, must and can only learn effective feature representations through end-to-end training on target task-specific data (even if noisy pairings) and loss functions (such as adversarial loss and recurrent loss). It is generally believed in the industry that “general” pre-training detached from specific staining tasks offers limited help in improving the final staining results, or that traditional transfer learning (such as pre-training on ImageNet) is ineffective due to the “domain gap” between natural and pathological images.

[0094] This invention effectively overcomes the aforementioned technical biases. It creatively proposes and verifies that within the domain of pathological images, there exists a more fundamental tissue structure feature independent of specific staining patterns. By designing a domain-specific, task-agnostic self-supervised pre-training task (contrastive learning), the encoder can effectively learn this feature. Subsequently, this encoder, possessing "pathological common sense," is connected to downstream staining tasks for fine-tuning. This two-stage paradigm of "domain-specific self-supervised pre-training + task-specific fine-tuning" successfully breaks the mindset of "end-to-end de novo training is necessary." Experimental data fully demonstrates that this paradigm is not only feasible but also delivers a significant performance leap (leading in PSNR, SSIM, and FID). Therefore, this invention overcomes the long-held biases held by those skilled in the art regarding virtual staining model training methods, opening up a new and more efficient technical route. Attached Figure Description

[0095] Figure 1 This is a flowchart of the main method framework provided in the embodiments of the present invention;

[0096] Figure 2 This is a visualization result diagram provided by an embodiment of the present invention;

[0097] Figure 3 This is a visualization of the feature clustering results of the self-supervised contrastive learning encoder provided in this embodiment of the invention;

[0098] Figure 4 This is a qualitative comparison diagram between the virtual staining generated image and the real image provided in the embodiments of the present invention;

[0099] Figure 5 This is a comparison chart of quantitative indicators of different virtual staining methods provided in the embodiments of the present invention;

[0100] Figure 6 This is a graph showing the loss change during the model training process provided in this embodiment of the invention. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0102] like Figure 1 As shown, this embodiment of the invention provides an innovative virtual staining method for pathological images based on self-supervision. The method includes a first stage of self-supervised pre-training of the encoder and a second stage of adversarial fine-tuning of the overall model.

[0103] (1) Data preparation

[0104] The required data includes:

[0105] Virtual coloring training data: two sets of images that do not require pairing, namely the source domain image set and the target domain image set; these data come from public datasets such as BCI and MIST; the images need to be preprocessed uniformly, including cropping to a fixed size and normalizing pixel values ​​to the range of [-1, 1];

[0106] (2) Self-supervised pre-training of the encoder, specifically including:

[0107] The goal of this stage is to train a robust pathological image feature extractor;

[0108] Step 2.1: Construct the contrastive learning task; for each image in the pre-training dataset Two different views are generated through random data augmentation. and Enhancement strategies include: random color dithering, random Gaussian blur, and random cropping followed by scaling back to the original size.

[0109] Step 2.2: Feature extraction; The enhanced image pairs ( , Input to the encoder to be trained In the process, the corresponding feature vectors are obtained respectively. and In computation, a small projection head is usually attached after the encoder to map the features to a lower-dimensional space that is more suitable for contrastive learning.

[0110] Step 2.3: Calculate the contrast loss; use normalized temperature-scale cross-entropy loss; for samples within a batch, Its corresponding Positive sample pairs are considered positive sample pairs, while features of other samples within the batch are considered negative sample pairs. The loss function ensures that the similarity between positive sample pairs and negative sample pairs in the feature space is much higher. The loss function is as follows:

[0111] ,

[0112] in, For cosine similarity, For temperature hyperparameters, Batch size;

[0113] Step 2.4: Model training; using the Adam optimizer to train the encoder. Update the parameters of the projector head; continue training for multiple rounds until the loss converges; after training, retain the encoder. The parameters are discarded;

[0114] (3) Construction and initialization of the overall model

[0115] Step 3.1: Build the enhanced generator, generator It consists of two parts:

[0116] The encoder part directly loads and freezes the ContrastiveEncoder pre-trained in the first stage; its structure includes: input layer → two downsampling convolutional layers → multiple residual blocks; this part is responsible for extracting deep semantic features of the input image.

[0117] Decoder section: Create a new MultiScaleDecoder; its structure includes: multiple upsampling stages → output layer; each upsampling stage is as follows: 3x3 convolutional layer → PixelShuffle operation → instance normalization → ReLU activation; finally, after reflection padding and 7x7 convolutional layer, the image is output using the Tanh activation function; the decoder is responsible for reconstructing the target domain image from the features extracted by the encoder.

[0118] Step 3.2: Construct the complete CycleGAN framework; instantiate two enhancement generators: H&E → IHC and IHC → H&E; Simultaneously, instantiate two PatchGAN discriminators: Used to determine the authenticity of IHC images. Used to determine the authenticity of H&E images;

[0119] (4) Adversarial fine-tuning of the virtual staining model

[0120] In this stage, unpaired H&E and IHC image sets are used to train the complete model built in step 3 end-to-end.

[0121] Step 4.1: Forward propagation; for a batch of training data:

[0122] Generate image: , ;

[0123] Loop reconstruction image: , ;

[0124] Identity mapping image: , ;

[0125] To maintain feature robustness: for and Random enhancements were performed separately to obtain... and and respectively through .encoder and .encoder extracts features;

[0126] Step 4.2: Loss Calculation; Total Loss It is the weighted sum of the following losses:

[0127] Combat loss ( ): Using least squares GAN loss encourages the generator to deceive the discriminator;

[0128] Cyclic consistency loss ( Using L1 loss, constraints and , and As close as possible;

[0129] Loss of identity ( (Optional): Use L1 loss, with constraints and , and Being as close as possible helps stabilize color mapping;

[0130] Contrast regularization loss ( ):calculate and Between features and The contrast loss between features is used as a regularization term; its purpose is to solidify the robust feature representations learned by the encoder and prevent them from degrading in the high-noise environment of adversarial training.

[0131] The total loss formula is: Among them, hyperparameters =10, =0.5, =0.1;

[0132] Step 4.3: Backpropagation and optimization; using an alternating optimization strategy:

[0133] Fixed discriminator and Update generator and Minimize the parameters ;

[0134] Fixed generator and Update the discriminator and The parameters are set to maximize its ability to distinguish between real and generated images;

[0135] Using the Adam optimizer ( =0.5, =0.999), with an initial learning rate set to 2e-4 and a linear decay strategy employed;

[0136] (5) Reasoning stage

[0137] In one specific embodiment, the HER2 staining task on the BCI dataset is used for validation;

[0138] Pre-training: All H&E images in the BCI dataset were used as pre-training data; the encoder E was a ContrastiveEncoder with 6 residual blocks; the Adam optimizer was used for 40 rounds of training with a batch size of 2 and an initial learning rate of 1e-3.

[0139] Fine-tuning: Load the pre-trained encoder and build and The discriminator is a 70x70 PatchGAN; it is trained for 60 epochs on the unpaired HER2 data from BCI with a batch size of 1; other hyperparameters are as described above.

[0140] Furthermore, the core of this method lies in constructing an enhanced generator that integrates a self-supervised pre-trained encoder and a multi-scale decoder, and integrating it into a recurrent consistency generative adversarial network framework to achieve high-quality and robust conversion from H&E stained images to IHC stained images.

[0141] Furthermore, the self-supervised pre-trained encoder based on contrastive learning specifically includes:

[0142] The core innovation of this deep convolutional neural network structure lies in the introduction of a self-supervised pre-training stage that is decoupled from the downstream staining task. In this stage, the encoder is trained using only a large number of readily available pathological images that do not require paired annotations. The aim is to learn a general pathological feature representation that is highly sensitive to tissue morphology and structure while being invariant to changes in staining patterns.

[0143] Network Structure: This encoder uses ResNet as its basic architecture, consisting of an initial convolutional layer, several downsampling layers, and a series of residual blocks. Specifically, the input image first passes through a padding layer and a 7x7 convolutional layer to extract initial features. Subsequently, it is downsampled through two convolutional layers with a stride of 2, gradually expanding the receptive field and compressing the spatial dimension. After downsampling, multiple residual blocks are connected to learn deep, abstract feature representations. The final output of the encoder is a high-dimensional feature tensor.

[0144] Self-supervised pre-training task: Pre-training adopts a contrastive learning paradigm; for the input image... Two different augmented views are generated through random data augmentation. and These two views are fed into a weighted encoder E to obtain the corresponding feature representations. and The goal of pre-training is to bring features from different augmented views of the same original image closer together in the feature space, while simultaneously pushing features from different original images further apart. This goal is achieved by minimizing the Normalized Temperature Scale Cross-Entropy Loss (NT-Xent Loss).

[0145] ,

[0146] in, For cosine similarity, This is a temperature hyperparameter; through this pre-training, the encoder learns to extract features that are robust to the essential structure of the tissue but insensitive to non-essential changes such as color and slight deformation.

[0147] Furthermore, the multi-scale decoder specifically includes:

[0148] The decoder is responsible for reconstructing the IHC staining image of the target domain from the abstract features extracted by the encoder; this improves the detail clarity and structural fidelity of the generated image.

[0149] Network Structure: The decoder employs an upsampling architecture, with its input being the feature tensor output by the pre-trained encoder. The core innovation lies in using pixel-recombined upsampling instead of traditional transposed convolution. Specifically, the decoder includes several upsampling stages. In each stage, a convolutional layer first expands the number of channels to four times the target number, then a PixelShuffle operation is applied for a 2x upsampling, effectively reducing checkerboard artifacts during the upsampling process. After upsampling, a normalization layer and activation function are applied sequentially. After multiple upsampling stages, the feature map is restored to the spatial dimensions of the original input image. Finally, a 7x7 convolutional layer and a Tanh activation function are used to output the final IHC image.

[0150] Multi-scale feature fusion: The multi-scale decoding mentioned above is essentially about the decoder starting from the deep, low-resolution but semantically rich features output by the encoder, and gradually recovering spatial details through progressive upsampling; the multi-level features captured by the encoder during downsampling are implicitly used to guide image generation at different scales through the upsampling path of the decoder, ensuring consistency from global structure to local details.

[0151] Furthermore, the enhancement generator is integrated into the overall training process, specifically including:

[0152] The pre-trained encoder described above is combined with the multi-scale decoder to form an augmented generator. Subsequently, the boosted generator is embedded into the standard CycleGAN framework for end-to-end fine-tuning training.

[0153] Overall framework: The overall model contains two enhancement generators. and and two discriminators (Discrimination of IHC domain) and (Discrimination of H&E domain); Training process does not require pixel-level paired images;

[0154] Loss function: The overall training objective is composed of the weighted sum of the following loss components:

[0155] (1) Adversarial loss: Makes the generated image indistinguishable in the target domain; ;

[0156] (2) Cyclic consistency loss: ensures that image transformation is reversible;

[0157] ,in Indicates L1 normal form;

[0158] (3) Identity loss: Encourages the generator to keep the input unchanged when the input is already a target domain image, which helps to stabilize the color distribution;

[0159] (4) Self-supervised contrastive loss: During the fine-tuning stage, features are extracted from the input image and its augmented view and the contrastive loss is calculated as a regularization term to maintain the robust feature representation learned by the encoder during the pre-training stage and prevent feature degradation during adversarial training.

[0160] ;

[0161] Training steps:

[0162] (4) Encoder pre-training stage: Collect a large number of unpaired pathological images; construct a contrastive learning task based on NT-Xent Loss, and pre-train the encoder network independently until the loss converges;

[0163] (5) Overall model initialization: Freeze or load the pre-trained encoder weights and combine them with the randomly initialized multi-scale decoder to build the augmentation generator. and Initialize the discriminator and ;

[0164] (6) Adversarial fine-tuning phase: using unpaired H&E and IHC image datasets; in each training iteration:

[0165] a. Forward propagation: , , , ;

[0166] b. Generate an enhanced view for the contrastive loss and extract features: , ;

[0167] c. Calculate the total generator loss And backpropagate updates and Parameters;

[0168] d. Calculate the discriminator loss and And backpropagate updates and Parameters;

[0169] (4) Inference stage: For the input H&E image, only the trained generator needs to be used. By performing forward propagation, a virtual stained IHC image can be obtained.

[0170] Example 1: A Self-Supervised Virtual Staining Workflow Based on Unpaired H&E and IHC Pathological Images

[0171] In this embodiment, a virtual staining method for pathological images is constructed. The data used includes H&E staining image sets and IHC staining image sets from publicly available pathological datasets. There is no pixel-level or slice-level pairing relationship between the two types of images. All input images are cropped to a uniform spatial size before entering the model, and the pixel values ​​are linearly normalized to the range of negative one to positive one. First, the encoder is pre-trained in a self-supervised manner using only the H&E image set. Multiple enhanced views are generated by randomly cropping, color perturbing, and blurring the same image, and the encoder is trained using feature similarity constraints to learn feature representations that are stable for tissue morphology and structure.

[0172] After completing self-supervised pre-training, the encoder is loaded into the generative model, forming the generator together with the newly initialized decoding structure. A discriminative structure is introduced to form a cyclic consistency adversarial training framework. During training, adversarial constraints, cyclic reconstruction constraints, and feature consistency regularization constraints are used to enable the model to achieve stable conversion of H&E to IHC staining styles under unpaired conditions. After training, simply inputting the H&E pathological image to be processed into the generator will output a virtual staining result with the target immunostaining style.

[0173] Example 2: Structure and Training Example of a Self-Supervised Comparative Learning Encoder

[0174] In this embodiment, the encoder used in the self-supervised pre-training stage employs a deep convolutional neural network structure. Initial low-level features are extracted at the input using large-size convolutional kernels. Subsequently, spatial dimensions are progressively compressed and the receptive field is expanded through multi-level downsampling. At deeper levels, abstract representations of tissue morphology are learned through residual structures. During training, at least two enhanced views are randomly generated for each pathological image, and these are input into a shared-weight encoder to obtain corresponding feature representation vectors.

[0175] By constructing a contrastive learning objective based on feature similarity, different enhanced views from the same original pathological image maintain high consistency in the feature space, while features from different images remain distinguishable. After multiple rounds of training, the encoder can stably extract general pathological features that are independent of staining method variations but highly sensitive to tissue structure, providing a robust feature foundation for subsequent virtual staining tasks.

[0176] Example 3: An Example of Reconstructing a Target-Stained Image Using a Multi-Scale Decoder

[0177] In this embodiment, the decoding part adopts a multi-scale, step-by-step reconstruction structure. Its input is a low-resolution, high-semantic feature tensor output by a self-supervised pre-trained encoder. The decoding process gradually restores the spatial resolution through multi-level upsampling. Each level of upsampling first maps the channel dimension and then expands the spatial size, thereby achieving continuous reconstruction of features from coarse to fine.

[0178] This multi-scale decoding method allows the model to maintain both the coherence of the overall tissue structure and the clarity of local details when generating target stained images, effectively reducing artifacts and structural misalignment. The final output image maintains a high degree of consistency with the real target stained image in terms of spatial resolution, tissue morphology, and staining distribution.

[0179] Example 4: An Example of Adversarial Training Incorporating Self-Supervised Feature Preservation Mechanisms

[0180] In this embodiment, the training of the virtual coloring model adopts a cyclic consistency generative adversarial framework, which includes two generation paths in opposite directions and corresponding discriminative paths. In each round of training, the model completes the generation from the source domain to the target domain, the reverse generation from the target domain to the source domain, and the cyclic reconstruction process after the two mappings, thereby ensuring the reversibility of cross-domain transformation.

[0181] Meanwhile, during generator training, encoder features are extracted from the input pathological images and their randomly augmented views, and feature consistency constraints are applied as regularization terms and introduced into the training objective. This mechanism enables the encoder to maintain stable feature representations formed during the self-supervised stage even in the dynamic game environment of adversarial training, significantly reducing the risk of structural degradation and improving the robustness and consistency of virtual staining results.

[0182] Example 5: Specific application of HER2 immunostaining prediction

[0183] In this embodiment, the above method is applied to the HER2 immunostaining prediction task. First, the encoder is pre-trained in a self-supervised manner using a large number of H&E images from a publicly available pathological dataset, allowing it to fully learn the general structural features of breast tissue. Then, the encoder is loaded and a complete virtual staining model is constructed, and adversarial fine-tuning training is performed on unpaired H&E and HER2 immunostaining images.

[0184] After training, untrained H&E pathological slides were input into the model for inference, and the model was able to directly generate corresponding HER2 virtual staining results. Experiments showed that the generated virtual staining images were highly consistent with the real immunostaining results in terms of cell membrane staining distribution, tissue structure continuity, and overall visual consistency, effectively assisting in pathological analysis and quantitative assessment.

[0185] Evidence related to the technical effects obtained by the embodiments of the present invention:

[0186] The preliminary experimental results of the self-supervised contrastive learning encoder described in Example 2 are as follows: Figure 3 As shown in the figure: The figure illustrates the clustering results of different datasets after the self-supervised contrastive learning encoder. Different categories represent different grading results of breast cancer in H&E and IHC images; for example, Class 3, 4, and 5 represent H&E 1+, 2+, and 3+, respectively; Class 0, 1, and 2 represent IHC 1+, 2+, and 3+. The figure shows that the clustering effect for H&E and IHC is significant, indicating that the self-supervised contrastive learning encoder has learned the specific properties of different stained images, significantly improving its ability to extract feature vectors from stained images and perform virtual staining.

[0187] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0188] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A self-supervised virtual staining method for pathological images, characterized in that, Includes the following steps: In the first stage, the pathological image encoder is self-supervised pre-trained. By constraining the feature similarity between different enhanced views generated from the same pathological image, the encoder learns feature representations that are sensitive to tissue structure and insensitive to staining differences. In the second stage, the self-supervised pre-trained encoder is embedded into the generative adversarial network for virtual staining training. During the adversarial training process, a contrastive regularization constraint based on feature consistency is introduced to enable the encoder to maintain the structure-aware ability formed in the pre-training stage in the high-noise environment of generative adversarial training, thereby realizing a stable virtual transformation of pathological images from the first staining domain to the second staining domain.

2. The method according to claim 1, characterized in that, The self-supervised pre-training stage generates different views by applying at least two random augmentation operations to the same original pathological image, and uses the similarity between the features extracted by the shared encoder of the different views as positive sample constraints and other image features as negative sample constraints for training.

3. The method according to claim 1, characterized in that, The contrastive regularization constraint in the second stage achieves stable constraint on the encoder feature distribution by constraining the consistency of the same pathological image and its enhanced view in the encoder feature space, thus preventing it from degrading during generative adversarial training.

4. An enhancement generator for virtual staining of pathological images, characterized in that, include: The encoder, pre-trained by self-supervised contrastive learning, is used to extract high-level semantic features that are stable to the tissue morphology and structure from the input pathological image. A multi-scale decoder, decoupled from the encoder function, is used to reconstruct the high-level semantic features into a target stained domain image step by step. In this process, the encoder acts as a structural feature extraction unit during virtual staining training, working in conjunction with the multi-scale decoder to achieve staining style conversion while maintaining the consistency of tissue structure.

5. The enhancement generator according to claim 4, characterized in that, The multi-scale decoder adopts a step-by-step upsampling structure and restores spatial resolution through a multi-stage feature reconstruction process, thereby achieving a continuous mapping from low-resolution semantic features to high-resolution image details.

6. The enhancement generator according to claim 4, characterized in that, The multi-scale decoder uses a channel rearrangement-based upsampling method to restore spatial dimensions, thereby reducing structural artifacts generated during upsampling and improving the detail consistency of the generated image.

7. A training method for a virtual staining model of pathological images, characterized in that, Construct a cycle-consistent generative adversarial network that includes a first generator, a second generator, and corresponding discriminators; Training is performed on unpaired pathological image data of the first and second staining domains, and the reversibility of cross-domain mapping is guaranteed by the cycle consistency constraint. During the training process, self-supervised contrastive regularization constraints are applied to the encoder features of the input image and its augmented view, enabling generative adversarial training and the self-supervised feature preservation mechanism to work synergistically, thereby improving the robustness of the virtual staining results in terms of structure preservation and staining consistency.

8. The method according to claim 7, characterized in that, The cycle consistency constraint achieves stable constraints on cross-coloring domain mapping relationships by limiting the difference between the reconstructed image after the first generator and the second generator are concatenated and the original input image.

9. The method according to claim 7, characterized in that, The self-supervised contrastive regularization constraint is introduced as a regularization term independent of adversarial loss and cycle consistency loss during the generator training phase to maintain the stability of the encoder feature distribution.

10. The method according to claim 7, characterized in that, During the model inference phase, the corresponding virtual staining result for the second staining domain can be output by using only the first generator that has been trained to perform forward computation on the input pathological image of the first staining domain.

Images