A pathological image virtual staining method and system based on a diffusion model

Abstract

The application relates to a pathological image virtual staining method and system based on a diffusion model, and belongs to the cross technical field of pathological image processing, deep learning and computer vision. Through an end-to-end process of data preprocessing-model construction-model training-virtual staining inference, non-paired virtual staining from AF to H&E is realized; through data preprocessing, non-paired AF pathological images and H&E pathological images are obtained; an STAIN model is constructed through a single-step diffusion generation main trunk, a multi-modal semantic enhancement module, a local saliency perception module and a composite loss function; the STAIN model is trained; and an AF pathological image to be processed is input into the trained STAIN model for virtual staining inference. The application breaks through the multi-step iteration bottleneck of the diffusion model, realizes single-step rapid inference, accurately retains pathological structures of cell nuclei and glands, guarantees that the staining style conforms to the H&E clinical standard, adopts a light-weight adaptation strategy, suppresses background noise interference and adapts to a non-paired data scene, thereby improving practicability and generality.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a virtual staining method and system for pathological images based on a diffusion model. Background Technology

[0002] In pathological diagnosis, hematoxylin-eosin (H&E) staining is the "gold standard" for disease diagnosis, but traditional chemical staining procedures have many limitations: sample preparation takes up to 24-48 hours, chemical reagents irreversibly consume precious tissue samples, and differences in scanning equipment and staining procedures among different hospitals lead to strong heterogeneity in staining results, which seriously affects multi-center collaboration and diagnostic consistency.

[0003] Existing virtual staining techniques are mainly divided into two categories: one is supervised methods based on convolutional neural networks such as UNET, such as the UNET supervised network slice HE virtual staining method disclosed in patent CN120148779A. This method relies on paired HE staining and fluorescence image training. While it can achieve basic staining transformation, its model feature extraction capability is limited. It only improves robustness through simple data augmentation, lacks semantic-level staining style constraints, and does not optimize for detail fidelity after staining homogenization. The structural reconstruction accuracy of complex tissues (such as glands and chromatin) is insufficient. It is particularly noteworthy that methods based on convolutional neural networks such as UNET are all supervised methods. These methods generally suffer from extremely high data acquisition costs, poor adaptability to small samples, and the risk of propagation of annotation bias. The other category is virtual staining methods based on diffusion models. These methods, with their progressive noise reduction generation characteristics, outperform GAN models in detail preservation. Patent CN119810254A discloses a method based on the StableDiffusion (SD) model, which achieves structural control by embedding weighted attention maps using ControlNet. However, this method relies on the dot product similarity between text prompts and image patches for attention map generation, resulting in coarse semantic constraints. Furthermore, it lacks a lightweight fine-tuning strategy, and full-parameter training leads to a high risk of overfitting in small-sample scenarios. Additionally, the ControlNet's control signals easily interfere with the original features, affecting generation stability. Moreover, the method in patent CN119810254A is primarily used for converting IHC-stained pathological sections to H&E sections.

[0004] Furthermore, existing technologies generally suffer from insufficient generalization ability and weak noise resistance: UNET-like methods are difficult to adapt to unpaired data scenarios, while diffusion model-like methods either have an excessively large number of trainable parameters or an imbalance between semantic guidance and structural control, failing to simultaneously meet the four core requirements of "inference efficiency, staining accuracy, structural fidelity, and lightweight adaptation." To address these bottlenecks, there is an urgent need for a virtual staining technique with accurate semantic guidance, lightweight training, strong noise resistance, and excellent generalization to meet clinical diagnostic requirements. Therefore, developing a virtual staining method that balances efficient inference, high-fidelity structure preservation, stable staining style, lightweight adaptation, and noise resistance is crucial for promoting the clinical application of this technology.

[0005] This invention provides a virtual staining method and system for pathological images based on a diffusion model, which is particularly suitable for cross-modal virtual staining from autofluorescence (AF) pathological images to hematoxylin-eosin (H&E) pathological images. It is applicable to cross-modal virtual staining of pathological tissues of multiple organs such as colorectal and breast tissues, and can meet the needs of clinical diagnosis and multi-center scientific research collaboration. It can be applied to scenarios such as clinical pathological diagnosis, multi-center pathological research, and data enhancement of computer-aided diagnostic (CAD) systems.

[0006] The Structure-preserving, Time-efficient, Adaptive Integrated Staining Network (STAIN) model proposed in this invention achieves unpaired virtual staining from AF to H&E through an end-to-end process of "data preprocessing - model building - model training - virtual staining inference".

[0007] This invention specifically achieves the following objectives:

[0008] Breaking through the multi-step iteration bottleneck of diffusion models, achieving rapid single-step inference to meet the needs of clinical batch processing;

[0009] Precisely preserve key pathological structures such as cell nuclei and glands to ensure the structural integrity of the generated images;

[0010] Ensure that the staining style conforms to H&E clinical standards to improve diagnostic consistency;

[0011] A lightweight adaptation strategy is adopted to reduce model training costs and adapt to small sample pathological data;

[0012] Suppress background noise interference and focus on the core diagnostic area;

[0013] It adapts to unpaired data scenarios, reduces the cost of clinical data preparation, and improves the practicality and generalization of the method. Summary of the Invention

[0014] To address the shortcomings of existing technologies, this invention provides a virtual staining method and system for pathological images based on a diffusion model, belonging to the interdisciplinary fields of pathological image processing, deep learning, and computer vision. Through an end-to-end process of "data preprocessing - model building - model training - virtual staining inference," unpaired virtual staining from AF (affected image) to H&E (hyper-and-exposed image) is achieved. Data preprocessing obtains unpaired AF and H&E pathological images. A STAIN model is constructed using a single-step diffusion-based backbone generation module, a multimodal semantic enhancement module, a local saliency perception module, and a composite loss function. The STAIN model is then trained. The AF pathological image to be processed is input into the trained STAIN model for virtual staining inference. This invention overcomes the multi-step iterative bottleneck of diffusion models, achieving rapid single-step inference; accurately preserves the pathological structures of cell nuclei and glands; ensures the staining style conforms to H&E clinical standards; employs a lightweight adaptation strategy; suppresses background noise interference; and adapts to unpaired data scenarios, improving practicality and generalization.

[0015] A virtual staining method for pathological images based on a diffusion model, the specific technical solution of which includes the following steps:

[0016] Step 1: Data preprocessing to obtain unpaired AF pathological images and H&E pathological images;

[0017] Step 2: Construct the STAIN virtual staining model;

[0018] The STAIN virtual staining model includes a single-step diffusion generation backbone module, a multimodal semantic enhancement module, a local saliency perception module, and a composite loss function module.

[0019] Step 3, train the STAIN virtual staining model: use a preset training strategy to train the STAIN virtual staining model end-to-end and save the optimal model weights;

[0020] Step 4, Virtual staining inference: The AF pathological image to be processed is input into the trained STAIN virtual staining model, and the corresponding H&E style virtual staining image is output through model inference.

[0021] Furthermore, in step 1, the data preprocessing process is as follows:

[0022] Step 1.1, Tissue sample preparation: Human tissue is fixed in formalin and embedded in paraffin to prepare a tissue chip. The tissue chip contains adjacent unlabeled sections and H&E stained sections to ensure that the two sets of sections come from the same tissue region and to ensure the consistency of tissue morphology and structural characteristics.

[0023] Step 1.2, Image Acquisition: Using a fully automated immunohistochemistry and fluorescence slide scanning system, fluorescent imaging was performed on unlabeled slides to obtain AF pathological images; bright-field imaging was performed on H&E stained slides to obtain H&E pathological images.

[0024] Step 1.3, Format standardization: AF pathological images are uniformly saved as 8-bit TIFF file format, and H&E pathological images are uniformly saved as 8-bit RGB file format;

[0025] Step 1.4, Data Adaptability Design: The VALIS method combined with flexible registration is used to achieve pixel-level alignment between AF pathological images and H&E pathological images.

[0026] During the model training phase, AF pathological images and H&E pathological images do not require pixel-level alignment processing; the similarity of human tissue features is ensured by the adjacent slice design of the tissue chip, while avoiding the model's dependence on paired training data and reducing data acquisition costs.

[0027] Furthermore, in step 2, the STAIN virtual coloring model uses a single-step diffusion generation backbone module as its core, and coordinates with a multimodal semantic enhancement module and a local saliency perception module to achieve high-quality virtual coloring, while ensuring training effectiveness through a composite loss function module; thus, inference time is compressed while ensuring the quality of virtual coloring.

[0028] First, the single-step diffusion generation backbone module receives the output of the multimodal semantic enhancement module as input, realizing the deep integration of the semantic enhancement module and the generation backbone, and explicitly constraining the target coloring style and semantic expression, reducing the uncertainty of coloring style under unpaired data conditions, and improving the stability and consistency of cross-domain transformation.

[0029] Then, the single-step diffusion generation backbone module, combined with the saliency weighted map output by the local saliency perception module, guides the generation process to prioritize the preservation of key structural features of cell morphology and human tissue arrangement, and suppresses structural drift in cross-modal mapping.

[0030] Furthermore, in step 2, the single-step diffusion generation backbone module uses Stable Diffusion as its architecture to reconstruct traditional Latent Diffusion Models (LDM), compressing the traditional multi-step diffusion process from the generation mechanism level to achieve efficient cross-modal mapping.

[0031] The single-step diffusion generation backbone module includes an improved VAE codec, a LoRA-enhanced U-Net, and a single-step scheduler.

[0032] Furthermore, the improved VAE codec introduces a multi-scale skip connection between the VAE encoder and decoder, directly transferring the intermediate feature maps of the VAE encoder. The corresponding layer of the VAE decoder is injected; to avoid disrupting the feature distribution of the pre-trained decoder in the early stages of training, the multi-scale skip connections are gated through "zero-convolution" (Zero-Conv) layers:

[0033] ;

[0034] in, This is a gated aggregation function, with initial weight values ​​set to less than or equal to... The value of y makes the initial output close to zero; i This is the output after gated aggregation;

[0035] The improved VAE codec maintains the stability of the generation process while enhancing the fidelity of the generated images in terms of glandular structure and cell morphology, achieving a balance between "efficiency improvement" and "tissue morphology integrity".

[0036] Furthermore, the LoRA-enhanced U-Net injects low-rank adaptive LoRA modules into the three types of substructures of U-Net; the three types of substructures include attention projection sublayers, feedforward network sublayers, and convolutional transformation sublayers; the three types of substructures have strong representation transformation capabilities;

[0037] The attention projection sublayer is a linear transformation component that realizes feature space transformation and information aggregation, including a Q / K / V projection stage and an output projection stage. The Q / K / V projection stage maps the input feature vector to three semantically orthogonal feature subspaces through independent learnable linear transformation matrices, generating a query vector Q, a key vector K, and a value vector V. The output projection stage obtains the attention output after weighted aggregation of the value vector V through attention weights. The attention output is then mapped back to the target feature space through the output projection matrix. A low-rank adaptive LoRA module is injected into the attention projection sublayer to achieve precise control of the attention mechanism.

[0038] The feedforward network sublayer, namely the two multi-layer perceptron (MLP) layers after the attention projection sublayer, undertakes the linear transformation function of channel dimension expansion and back projection, including two parts: dimension-up projection and dimension-down projection; the low-rank adaptive LoRA module is injected into the feedforward network sublayer to improve the adaptability of feature dimension transformation.

[0039] The convolutional transformation sublayer, namely the convolutional operators used for local texture modeling and scale transformation in the residual block and upsampling path, includes input mapping convolution, convolution in the residual block, and convolutional alignment units in the shortcut branch; the low-rank adaptive LoRA module is injected into the convolutional transformation sublayer to enhance the representation capability of local texture and scale transformation.

[0040] Furthermore, the single-step deterministic mapping process of the single-step scheduler:

[0041] ;

[0042] in, and These are the VAE encoder and decoder, respectively. For LoRA-enhanced U-Net networks; is a single-step scheduler; x is the input image; t is the number of iterations, ranging from [50, 1000]; c is the input condition; The image is the result of a single-step deterministic mapping;

[0043] The single-step deterministic mapping can complete cross-modal mapping with only one backdiffusion update, significantly reducing inference time.

[0044] Furthermore, the weight update rule of the low-rank adaptive LoRA module is as follows:

[0045] ;

[0046] in, This is the original weight matrix; This is the updated weight matrix; This refers to the amount of weight matrix update during the fine-tuning process; Let be the set of real numbers; d represents the feature dimension; BA is a trainable low-rank matrix, and the rank parameter is... The value ranges from 4 to 16; The scaling factor is linearly proportional to the rank. Freezing the backbone weights optimizes only the low-rank matrix, balancing expressive power and lightweight design.

[0047] Furthermore, in step 2, the multimodal semantic enhancement module uses the Pathology Language-Image Pre-training (PLIP) module as a pre-trained model to achieve semantic constraints on coloring styles;

[0048] In the STAIN virtual staining model, the multimodal semantic enhancement module is not a simple feature extractor, but a semantic navigator for the generation process. The text tower of the multimodal semantic enhancement module is used to process pathological staining description instructions, such as "H&E stained tissue with distinct nuclear chromatin". Unlike the approach of compressing the whole sentence into a single vector, the token-level semantic sequence output by the text tower is directly embedded into the LoRA-enhanced U-Net.

[0049] To adapt the token-level semantic sequences to the conditional input dimension of the LoRA-enhanced U-Net, a learnable linear projection layer is introduced to represent the semantic sequence at each token level. Perform mapping:

[0050] ;

[0051] Where C is the conditional sequence containing pathological semantics after linear projection; L is the length of the token-level semantic sequence;

[0052] The conditional sequence C is injected into each layer of the LoRA-enhanced U-Net through a cross-attention mechanism; the cross-attention mechanism :

[0053] ;

[0054] Where W is the original weight matrix, W Q W K W V Let W represent the three independent linear weight matrices, and T represent the transpose. In each cross-attention layer, the query vector Q comes from the latent features of the current layer of the LoRA-enhanced U-Net, and the key vector K and value vector V come from the conditional sequence C. Let the flattened spatial features of LoRA-enhanced U-Nett be... This allows for the generation of image styles at multiple scales with semantic constraints, where N represents the number of samples after feature flattening.

[0055] Injecting the multimodal semantic enhancement module into the single-step diffusion generation backbone module differs from simply adding semantic vectors to features. Instead, it uses the key vector K and value vector V as context to continuously adjust the attention weights, thereby guiding the staining style layer by layer and ensuring that the generated pathological staining images not only conform to the text instruction description but also meet the professional requirements of pathological diagnosis.

[0056] Furthermore, in step 2, the local saliency perception module is constructed based on the gray intensity distribution characteristics of pathological images. By setting a threshold and using the Sigmoid function, a smooth saliency weighted map is generated to achieve background noise suppression and weight enhancement of the core region of cell aggregation.

[0057] The local saliency perception module includes a grayscale threshold setting unit, a saliency weighted map generation unit, and a feature weighted fusion unit; each unit is connected in series to form a complete processing flow of "threshold differentiation - weight mapping - feature fusion";

[0058] The grayscale threshold setting unit presets two sets of intensity thresholds T based on the grayscale intensity distribution characteristics of pathological images. a T and Tᵦ correspond to different grayscale domains A and B of the image, respectively, and are used to accurately distinguish between the background region and the core foreground region; T a It adapts to low grayscale background regions and Tᵦ adapts to medium-high grayscale transition regions; the low grayscale background region includes the edge of the slide and blank background, and the high grayscale transition region includes bubbles and weak noise areas;

[0059] The saliency-weighted map generation unit constructs a smooth saliency-weighted map by applying a pixel-by-pixel sigmoid function. :

[0060] ;

[0061] in, (·) is the Sigmoid activation function, used to map the output to the interval [0, 1]; I is the pixel intensity of the input feature map; The background filtering threshold for the corresponding grayscale region is T, where region A is set to T. a Domain B takes Tᵦ; This is a scaling factor used to control the steepness of the Sigmoid function transformation, with a value ranging from 5 to 20.

[0062] The saliency-weighted map generation unit automatically assigns gradient weights ranging from 0.5 to 0.8 to the foreground tissue region through the saliency-weighted map mapping relationship to achieve basic focusing, and gradient weights ranging from 0.8 to 1.0 to achieve precise focusing for the foreground core region; the foreground core region includes, for example, glands and cell nuclei; the core foreground tissue includes, for example, cell nuclei and gland core areas;

[0063] AF pathological images suppress the gradient weight of the background tissue region to near zero, achieving preliminary filtering of background noise and focusing of the core region;

[0064] The feature weighted fusion unit performs pixel-wise weighted fusion of the generated saliency weighted map with the latent features of each layer of the LoRA-enhanced U-Net, strengthening the representation weight of the features in the foreground core region and guiding the model to prioritize maintaining the integrity of the core structure during the generation process.

[0065] Furthermore, in step 2, the composite loss function module includes visual-assisted adversarial loss, cycle consistency loss, identity regularization loss, and adaptive structural saliency loss; the visual-assisted adversarial loss, cycle consistency loss, identity regularization loss, and adaptive structural saliency loss are weighted to obtain the total loss;

[0066] The total loss for:

[0067] ;

[0068] in, for Weighting coefficients; for Weighting coefficients; for Weighting coefficients; for Weighting coefficients; To assist visual loss, a frozen PLIP visual encoder is used as the discriminator backbone to distinguish between real H&E images and generated images; To constrain the structural integrity of cross-modal mapping using a perception-based cyclic consistency loss, ensuring that the generated image is consistent with the pathological structure of the original AF image; To constrain the identity regularization loss, the target domain H&E pathological images are constrained to maintain their appearance after being input into the generator, thus avoiding color distortion caused by over-staining. To adapt to structural saliency loss, structural differences are calculated based on local saliency weighted maps to enhance the structural consistency of the core region of cell aggregation;

[0069] The composite loss function module employs a multi-dimensional balanced structure to maintain style matching.

[0070] Furthermore, in step 3, the process of training the STAIN virtual staining model is as follows:

[0071] The AdamW optimizer and cosine annealing learning rate scheduling strategy were used to train the STAIN virtual coloring model end-to-end.

[0072] S3.1, Parameter Settings: Initial learning rate range is [range missing]. The batch size is 2 to 8, the input image is normalized to the range of [-1,1], and the total number of training iterations is 10,000 to 20,000.

[0073] S3.2, Training strategy: Freeze the main parameters of the Stable Diffusion and PLIP modules, and only optimize the low-rank matrix and trainable parameters of the linear projection layer of the LoRA model to avoid overfitting on small samples.

[0074] The main parameters are the core parameters that have converged during the pre-training stage. They are not included in the optimization scope of the training, but are kept frozen to reuse the pre-trained feature extraction capabilities. The core parameters support the requirements of virtual staining of pathological images in terms of structural fidelity, style consistency and cross-modal conversion.

[0075] The main parameters include the weights of each convolutional layer in the convolutional transform sublayer, the VAE codec parameters, the weights of the multilayer perceptron in the attention projection sublayer, the embedding matrix in the PLIP module, and the cross-attention parameters.

[0076] The trainable parameters are lightweight parameters added to adapt to virtual staining of pathological images, including the LoRA low-rank adaptation matrix (the parameter size of each module does not exceed 10). 4 The linear projection layer (with 512 input and 256 output channels and approximately 130,000 parameters) is used for feature dimension matching, and the fine-tunable threshold parameter (T) in the local saliency sensing module is used. a (Tᵦ) and scaling factor ;

[0077] Trainable parameters account for less than 3% of the total parameters of the Stable Diffusion and PLIP models. By controlling the proportion of trainable parameters, the risk of overfitting in small sample scenarios can be significantly reduced.

[0078] S3.3, STAIN virtual staining model saving: Save the weights of the STAIN virtual staining model at fixed iteration intervals, and select the weights with the best quality generated on the validation set as the final STAIN virtual staining model.

[0079] A pathological image virtual staining system based on a diffusion model is provided for performing the aforementioned pathological image virtual staining method. The pathological image virtual staining system includes a data preprocessing module, a single-step diffusion backbone generation module, a multimodal semantic enhancement module, a local saliency perception module, and a composite loss function module.

[0080] The data preprocessing module is used to acquire pathological images. The data preprocessing module is connected to the semantic enhancement module and the local saliency perception module, respectively. The semantic enhancement module and the local saliency perception module are connected to the single-step diffusion generation backbone module. The single-step diffusion generation backbone module receives the output of the multimodal semantic enhancement module as input to achieve deep integration between the semantic enhancement module and the generation backbone, and to explicitly constrain the target staining style and semantic expression. The single-step diffusion generation backbone module receives the saliency weighted map output by the local saliency perception module, guiding the generation process to prioritize the preservation of key structural features of cell morphology and human tissue arrangement.

[0081] The single-step diffusion generation backbone module is the core backbone module, which works in conjunction with the multimodal semantic enhancement module and the local saliency perception module to achieve high-quality virtual staining of pathological images, and the training effect is guaranteed by the composite loss function module.

[0082] Compared with the prior art, the present invention has the following significant advantages:

[0083] Significantly improved inference efficiency: By using a single-step diffusion mechanism, the multi-step iteration of the traditional diffusion model is compressed into a single-step mapping. The inference time for a single 512×512 pixel image is ≤0.5 seconds, which is more than 90 times more efficient than the traditional diffusion model, meeting the needs of clinical batch processing.

[0084] Excellent structural fidelity: The improved VAE's multi-scale skip connections and Zero-Conv gating mechanism effectively preserve high-frequency pathological structures such as cell nuclear boundaries and chromatin texture, and the structural integrity of the generated image is highly consistent with the real H&E image;

[0085] The dual guidance mechanism, which provides low-level anatomical constraints through the VAE encoder and high-level staining style guidance through cross-attention by the PLIP model, effectively solves the style ambiguity problem common in traditional GAN ​​models, ensuring that the generated staining images accurately convey potential pathological diagnostic information while possessing the color features of the target domain.

[0086] Stable and controllable staining style: The PLIP module provides semantic constraints through a cross-attention mechanism, and the staining style of the generated images conforms to H&E clinical standards. The consistency of blind diagnosis by pathologists reaches more than 90%, avoiding diagnostic interference caused by color shift.

[0087] Lightweight and highly adaptable: The LoRA strategy significantly reduces the number of trainable parameters, lowers training computation costs, adapts to small sample pathological data, and enables effective training without large-scale annotation.

[0088] Outstanding noise resistance: The local saliency sensing module suppresses background noise interference, focuses on the core diagnostic region, and improves the model's robustness in processing complex background pathological images;

[0089] Excellent generalization ability: It can be directly transferred to the processing of pathological images of multiple organs such as colorectal and breast without modifying the model structure, adapts to unpaired data scenarios, supports multi-center pathological research and clinical collaboration, and has significant practical value. Attached Figure Description

[0090] Figure 1 This is an overall framework diagram of the STAIN virtual staining model of the present invention;

[0091] Figure 2 This is a comparison chart of the virtual staining visual results of the present invention with those of CycleGAN, CUT, UTOM, and PPT methods. Detailed Implementation

[0092] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0093] The architecture diagram of the STAIN virtual staining model of this invention includes AF image input, PLIP module, improved VAE codec, LoRA enhanced U-Net, single-step scheduler, local saliency perception module, and composite loss function constraint relationship. Arrows indicate the data flow direction and clearly show the collaborative working logic of each module.

[0094] Example 1: Virtual staining of colorectal tissue AF images to H&E images

[0095] (I) Dataset Construction

[0096] 1. Dataset source: The dataset used is a publicly available unpaired dataset of colorectal tissue, containing 13 AF images and 13 H&E images. The images were fixed in formalin and embedded in paraffin (FFPE) to form tissue microarrays (TMA), and then imaged using a Zeiss Axio Scan.Z1 scanner with an image resolution of approximately 5000×5000 pixels.

[0097] 2. Test dataset preprocessing:

[0098] The registration operation on the test dataset is a necessary prerequisite for the subsequent calculation of quantitative indicators such as structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR); however, the training set of the model does not require the registration process.

[0099] Registration: Global affine registration is performed using the VALIS method, combined with elastic registration (elastic coefficient 0.1) to correct local deformation. After registration, the pixel-level alignment error is ≤2 pixels.

[0100] Segmentation: Segmented by sliding at a size of 256×256 pixels with a step size of 128 pixels, with an overlap rate of 50%;

[0101] Filtering: Invalid patches with a white pixel ratio > 80% were removed, resulting in 1200 valid training patches;

[0102] Standardization: Staining standardization calibration was performed using H&E images acquired by the Aperio GT 450 scanner as a reference.

[0103] (II) Experimental Environment

[0104] Hardware: A GPU computing platform with ≥24GB of video memory (such as NVIDIA TITAN RTX).

[0105] Software: Linux operating system, Python 3.10 programming language, PyTorch 2.0.1+cu117 deep learning model, CUDA 11.7 parallel computing architecture, PLIP pathology-specific pre-trained weights.

[0106] (III) Training of the STAIN virtual staining model

[0107] 1. Model Configuration: LoRA Module Rank Parameter The initial weights of the Zero-Conv layer are set to The PLIP text instruction is “H&E stained tissue with distinct nuclear chromatin”;

[0108] 2. Loss function: The weight coefficients are set as follows: =1.0、 =10.0、 =1.0、 =1.0;

[0109] 3. Training parameters: Batch size 4, initial learning rate Cosine annealing learning rate scheduling, total number of iterations 15000, input image normalized to the [-1,1] interval;

[0110] 4. Model saving: The weights are saved every 1000 iterations, and the weights with the best quality generated on the validation set are selected as the final model.

[0111] (iv) Experimental Results

[0112] 1. Qualitative results: The generated images show cell nuclei in a blue-purple color consistent with pathological understanding, cytoplasm and matrix areas in a uniform pink color, no local offset or oversaturation of staining, clear cell nucleus boundaries, continuous glandular structure, and no obvious artifacts.

[0113] 2. Quantitative results: The generated image and the real H&E image have SSIM=0.426, PSNR=20.081 dB, FID=48.411, and KID=0.0162, which are significantly better than the existing mainstream methods;

[0114] 3. Clinical validation: Blind review by three senior pathologists showed that the diagnostic consistency between the virtual staining images and the real H&E images reached 90.1%, with Cohen's κ=0.83, which meets the requirements for clinical diagnosis.

[0115] Example 2: Cross-dataset generalization validation of breast tissue

[0116] (I) Dataset Construction

[0117] The BACH breast pathology dataset, containing 8,000 standard H&E images, was used. Fifty clinical breast AF images were selected and segmented into 480 valid patches according to the preprocessing procedure in Example 1, without the need for additional model training.

[0118] (II) Reasoning and Results

[0119] 1. Inference process: Load the model weights trained in Example 1, input the preprocessed breast AF image into the model, and output the H&E virtual staining image in a single step of inference. The average inference time is 0.45 seconds per image.

[0120] 2. Results Validation: The generated images maintained the intensity of nuclear staining in cancerous areas, and there were no physiological false colors in fat and loose stroma areas. The staining style was uniform and the tissue boundaries were clear. Among the quantitative indicators, FID=10.678, KID=0.0014, the consistency of blind diagnosis by pathologists reached 93.6%, and Cohen's κ=0.88, validating the model's cross-organ generalization ability.

[0121] (III) Validation of the effectiveness of stacking the Local Saliency Awareness Module, PLIP Semantic Enhancement Module, and High-Resolution Input Strategy. Using the "Stable Diffusion backbone + LoRA fine-tuning" model as the baseline, the Local Saliency Awareness Module, PLIP module, and high-resolution input strategy were gradually stacked to verify the role of each core module:

[0122] 1. After overlaying the local saliency perception module, background noise interference is significantly reduced, FID drops from 109.707 to 86.565, and the clarity of the core structure of the generated image is improved;

[0123] 2. After adding the PLIP module, the consistency of the coloring style was significantly improved, the KID dropped from 0.0299 to 0.0099, and the color shift phenomenon was effectively suppressed;

[0124] 3. After adopting the high-resolution input strategy, the accuracy of the restoration of fine-grained pathological structures (such as chromatin texture) was further improved. The final model had FID=48.411 and KID=0.0162, and all indicators reached the optimal level.

[0125] 4. The complete model outperforms the single-module configuration in all evaluation indicators, verifying that the modules achieve synergy through the constraints of the core formula, and achieve a balanced optimization of "efficiency-structure-style".

[0126] This invention solves the core bottleneck of existing virtual staining technology through a simplified technical solution supported by a core formula. The generated images meet clinical diagnostic standards and can be deployed without complex equipment. It provides reliable technical support for the digitization of pathological diagnosis, multi-center collaboration, and reuse of precious samples, and has broad clinical application prospects and scientific research value.

Claims

1. A virtual staining method for pathological images based on a diffusion model, characterized in that, The steps include the following: Step 1: Data preprocessing to obtain unpaired AF pathological images and H&E pathological images; Step 2: Construct the STAIN virtual staining model; The STAIN virtual staining model includes a single-step diffusion generation backbone module, a multimodal semantic enhancement module, a local saliency perception module, and a composite loss function module. The STAIN virtual staining model uses a single-step diffusion generation backbone module as its core, in conjunction with a multimodal semantic enhancement module and a local saliency perception module to achieve high-quality virtual staining, and uses a composite loss function module to ensure training effectiveness. First, the single-step diffusion generation backbone module receives the output of the multimodal semantic enhancement module as input, realizing deep integration between the semantic enhancement module and the generation backbone, and explicitly constraining the target staining style and semantic expression. Then, the single-step diffusion generation backbone module combines the saliency weighted map output by the local saliency perception module to guide the generation process to prioritize the preservation of key structural features of cell morphology and human tissue arrangement. The single-step diffusion generation backbone module includes an improved VAE codec, a LoRA-enhanced U-Net, and a single-step scheduler; The improved VAE codec introduces a multi-scale skip connection between the VAE encoder and decoder, directly transferring the intermediate feature maps of the VAE encoder. The corresponding layer is injected into the VAE decoder; multi-scale skip connections are gated through "zero-convolution" layers. ； in, This is a gated aggregation function, with initial weight values ​​set to less than or equal to... The value of y makes the initial output close to zero; i This is the output after gated aggregation; The single-step deterministic mapping process of the single-step scheduler: ； in, and These are the VAE encoder and decoder, respectively. For LoRA-enhanced U-Net networks; is a single-step scheduler; x is the input image; t is the number of iterations, ranging from [50, 1000]; c is the input condition; The image is the result of a single-step deterministic mapping; Single-step deterministic mapping can complete cross-modal mapping with only one back-diffusion update, significantly reducing inference time; Step 3, train the STAIN virtual staining model: use a preset training strategy to train the STAIN virtual staining model end-to-end and save the optimal model weights; Step 4, Virtual staining inference: The AF pathological image to be processed is input into the trained STAIN virtual staining model, and the corresponding H&E style virtual staining image is output through model inference.

2. The virtual staining method for pathological images based on a diffusion model according to claim 1, characterized in that, In step 1, the data preprocessing process is as follows: Step 1.1, Tissue sample preparation: Human tissue is fixed in formalin and embedded in paraffin to prepare a tissue chip. The tissue chip contains adjacent unlabeled sections and H&E stained sections to ensure that the two sets of sections come from the same tissue region and to ensure the consistency of tissue morphology and structural characteristics. Step 1.2, Image Acquisition: Using a fully automated immunohistochemistry and fluorescence slide scanning system, fluorescent imaging was performed on unlabeled slides to obtain AF pathological images; bright-field imaging was performed on H&E stained slides to obtain H&E pathological images. Step 1.3, Format standardization: AF pathological images are uniformly saved as 8-bit TIFF file format, and H&E pathological images are uniformly saved as 8-bit RGB file format; Step 1.4, Data Adaptability Design: The VALIS method combined with flexible registration is used to achieve pixel-level alignment between AF pathological images and H&E pathological images. During the model training phase, AF pathological images and H&E pathological images do not require pixel-level alignment.

3. The virtual staining method for pathological images based on a diffusion model according to claim 1, characterized in that, The LoRA-enhanced U-Net injects low-rank adaptive LoRA modules into three types of substructures of U-Net; the three types of substructures include attention projection sublayers, feedforward network sublayers, and convolutional transformation sublayers; The attention projection sublayer is a linear transformation component that realizes feature space transformation and information aggregation, including a Q / K / V projection stage and an output projection stage. The Q / K / V projection stage maps the input feature vector to three semantically orthogonal feature subspaces through independent learnable linear transformation matrices, generating a query vector Q, a key vector K, and a value vector V. The output projection stage obtains the attention output after weighted aggregation of the value vector V through attention weights. The attention output is then mapped back to the target feature space through the output projection matrix. A low-rank adaptive LoRA module is injected into the attention projection sublayer to achieve precise control of the attention mechanism. The feedforward network sublayer, namely the two multilayer perceptron layers after the attention projection sublayer, undertakes the linear transformation function of channel dimension expansion and back projection, and includes two parts: dimension-up projection and dimension-down projection; the low-rank adaptive LoRA module is injected into the feedforward network sublayer to improve the adaptability of feature dimension transformation. The convolutional transformation sublayer, namely the convolutional operators used for local texture modeling and scale transformation in the residual block and upsampling path, includes input mapping convolution, convolution in the residual block, and convolutional alignment units in the shortcut branch; the low-rank adaptive LoRA module is injected into the convolutional transformation sublayer to enhance the representation capability of local texture and scale transformation. The weight update rule for the low-rank adaptive LoRA module is as follows: ； in, This is the original weight matrix; This is the updated weight matrix; This refers to the amount of weight matrix update during the fine-tuning process; Let be the set of real numbers; d represents the feature dimension; BA is a trainable low-rank matrix, and the rank parameter is... The value ranges from 4 to 16; The scaling factor is linearly proportional to the rank. Freezing the backbone weights optimizes only the low-rank matrix, balancing expressive power and lightweight design.

4. The virtual staining method for pathological images based on a diffusion model according to claim 1, characterized in that, In step 2, the multimodal semantic enhancement module uses the PLIP module as a pre-trained model to realize the semantic constraints of coloring style; The multimodal semantic enhancement module is not a simple feature extractor, but a semantic navigator for the generation process. The text tower of the multimodal semantic enhancement module is used to process pathological staining description instructions, and the token-level semantic sequence output by the text tower is directly embedded into the LoRA-enhanced U-Net. To adapt the token-level semantic sequences to the conditional input dimension of the LoRA-enhanced U-Net, a learnable linear projection layer is introduced to represent the semantic sequence at each token level. Perform mapping: ； Where C is the conditional sequence containing pathological semantics after linear projection; L is the length of the token-level semantic sequence; The conditional sequence C is injected into each layer of the LoRA-enhanced U-Net through a cross-attention mechanism; cross-attention mechanism : ； Where W is the original weight matrix, W Q W K W V Let W represent the three independent linear weight matrices, and T represent the transpose. In each cross-attention layer, the query vector Q comes from the latent features of the current layer of the LoRA-enhanced U-Net, and the key vector K and value vector V come from the conditional sequence C. Let the flattened spatial features of LoRA-enhanced U-Nett be... This allows for the generation of image styles at multiple scales with semantic constraints, where N represents the number of samples after feature flattening.

5. The virtual staining method for pathological images based on a diffusion model according to claim 1, characterized in that, In step 2, the local saliency perception module includes a grayscale threshold setting unit, a saliency weighted map generation unit, and a feature weighted fusion unit; each unit is connected in series to form a complete processing flow of "threshold differentiation - weight mapping - feature fusion"; The grayscale threshold setting unit presets two sets of intensity thresholds based on the grayscale intensity distribution characteristics of pathological images. and These correspond to different grayscale regions A and B of the image, respectively, and are used to accurately distinguish between the background region and the core foreground region. Adapts to low grayscale background areas It adapts to medium and high grayscale transition regions; the low grayscale background region includes the edge of the slide and the blank background, and the high grayscale transition region includes bubbles and weak noise areas; The saliency-weighted map generation unit constructs a smooth saliency-weighted map by applying a pixel-by-pixel sigmoid function. : ； in, (·) is the Sigmoid activation function, used to map the output to the interval [0, 1]; I is the pixel intensity of the input feature map; The background filtering threshold for the corresponding grayscale range is set to A. Domain B ; This is a scaling factor used to control the steepness of the Sigmoid function transformation, with a value ranging from 5 to 20. The saliency-weighted map generation unit automatically assigns gradient weights ranging from 0.5 to 0.8 to the foreground tissue region through the saliency-weighted map mapping relationship to achieve basic focusing, and gradient weights ranging from 0.8 to 1.0 to achieve precise focusing for the foreground core region; the foreground core region includes, for example, glands and cell nuclei; the core foreground tissue includes, for example, cell nuclei and gland core areas; AF pathological images suppress the gradient weight of the background tissue region to near zero, achieving preliminary filtering of background noise and focusing of the core region; The feature weighted fusion unit performs pixel-wise weighted fusion of the generated saliency weighted map with the latent features of each layer of the LoRA-enhanced U-Net.

6. The virtual staining method for pathological images based on a diffusion model according to claim 1, characterized in that, In step 2, the composite loss function module includes visual-assisted adversarial loss, cycle consistency loss, identity regularization loss, and adaptive structural saliency loss; the visual-assisted adversarial loss, cycle consistency loss, identity regularization loss, and adaptive structural saliency loss are weighted to obtain the total loss; The total loss for: ； in, for Weighting coefficients; for Weighting coefficients; for Weighting coefficients; for Weighting coefficients; To assist visual loss, a frozen PLIP visual encoder is used as the discriminator backbone to distinguish between real H&E images and generated images; To constrain the structural integrity of cross-modal mapping using a perception-based cyclic consistency loss, ensuring that the generated image is consistent with the pathological structure of the original AF image; To constrain the identity regularization loss, the target domain H&E pathological images are constrained to maintain their appearance after being input into the generator, thus avoiding color distortion caused by over-staining. To adapt to structural saliency loss, structural differences are calculated based on local saliency weighted maps to enhance the structural consistency of the core region of cell aggregation; The composite loss function module employs a multi-dimensional balanced structure to maintain style matching.

7. The virtual staining method for pathological images based on a diffusion model according to claim 1, characterized in that, In step 3, the process of training the STAIN virtual staining model is as follows: The AdamW optimizer and cosine annealing learning rate scheduling strategy were used to train the STAIN virtual coloring model end-to-end. S3.1, Parameter Settings: Initial learning rate range is [range missing]. The batch size is 2 to 8, the input image is normalized to the range of [-1,1], and the total number of training iterations is 10,000 to 20,000. S3.2, Training strategy: Freeze the main parameters of the Stable Diffusion and PLIP modules, and only optimize the low-rank matrix and trainable parameters of the linear projection layer of the LoRA model; The main parameters are the core parameters that have converged in the pre-training stage. They are not included in the optimization scope of the training, but are kept frozen to reuse the pre-trained feature extraction capabilities. The main parameters include the weights of each convolutional layer in the convolutional transform sublayer, the VAE codec parameters, the weights of the multilayer perceptron in the attention projection sublayer, the embedding matrix in the PLIP module, and the cross-attention parameters. The trainable parameters are lightweight parameters added to adapt to virtual staining of pathological images, including the LoRA low-rank adaptation matrix, the linear projection layer for feature dimension matching, and the finely adjustable threshold parameters and scaling factors in the local saliency perception module. ; Trainable parameters account for less than 3% of the total parameters in the Stable Diffusion and PLIP models; S3.3, STAIN virtual staining model saving: Save the weights of the STAIN virtual staining model at fixed iteration intervals, and select the weights with the best quality generated on the validation set as the final STAIN virtual staining model.

8. A virtual staining system for pathological images based on a diffusion model, characterized in that, This system is used to perform the virtual staining method for pathological images as described in any one of claims 1 to 7. The virtual staining system includes a data preprocessing module, a single-step diffusion generation backbone module, a multimodal semantic enhancement module, a local saliency perception module, and a composite loss function module. The data preprocessing module is used to acquire pathological images. The data preprocessing module is connected to the semantic enhancement module and the local saliency perception module, respectively, and the semantic enhancement module and the local saliency perception module are connected to the single-step diffusion generation backbone module. The single-step diffusion generation backbone module receives the output of the multimodal semantic enhancement module as input, achieving deep integration between the semantic enhancement module and the generation backbone, and explicitly constraining the target staining style and semantic expression. The single-step diffusion generation backbone module receives the saliency weighted map output by the local saliency perception module, guiding the generation process to prioritize maintaining key structural features of cell morphology and human tissue arrangement. The single-step diffusion generation backbone module is the core backbone module, working in conjunction with the multimodal semantic enhancement module and the local saliency perception module to achieve high-quality virtual staining of pathological images, and the composite loss function module ensures training effectiveness.

Images