A virtual staining method and system for pathological section images

Abstract

The present invention discloses a virtual staining method and system for pathological section images, which belongs to the field of image processing technology. The method includes: constructing registration data pairs for different staining styles as training data through real staining, and also constructing a virtual staining lookup table model including an adaptive feature encoder, an adaptive weight predictor and 𝑛 basic virtual staining lookup tables. In the model, feature maps and weights are adaptively generated based on the input original image, and the 𝑛 basic virtual staining lookup tables are weighted based on the weights to construct an adaptively fused virtual staining lookup table for the original image. Then, the virtual staining lookup table is used to perform virtual staining in various chemical staining styles. This process has high staining efficiency, diversified staining styles, and staining is achieved by changing only the color value without changing the pixel position and morphology. The staining is highly adaptive, the staining quality is guaranteed, and the stained area is interpretable.

Description

Technical Field

[0001] The present invention belongs to the technical field of combining intelligent pathology-assisted diagnosis and image processing, and particularly relates to a virtual staining method and system for pathology section images. Background Art

[0002] Histopathological staining plays an irreplaceable and important role in clinical pathological diagnosis, effectively enhancing the contrast and visualization of cell and tissue structures. Among them, hematoxylin-eosin (HE) staining is currently the gold standard for clinical pathological diagnosis, while special staining techniques such as periodic acid-Schiff (PAS) and Masson's trichrome (MAS) are used to highlight specific tissue components to improve the diagnostic rate of pathologists. However, traditional chemical staining methods generally have problems such as cumbersome operating procedures, long time consumption, dependence on chemical reagents, and high costs. In addition, the staining process is irreversible, and in some cases, stained samples are difficult to use for analysis of other staining types. At the same time, due to differences in human operation, it is easy to cause inconsistency in staining quality.

[0003] With the rapid development of digital pathology and whole-slice scanning technology, high-resolution tissue slice images have been widely digitized, providing a technical foundation for remote diagnosis, image quantitative analysis, and the application of artificial intelligence in the field of pathology. In this context, virtual staining, as an emerging technology, has become a potential solution to replace traditional chemical staining. Virtual staining uses deep learning models to skip the traditional staining steps and directly generate target chemical staining images from unlabeled tissue, accelerating the diagnostic process. It can also avoid the damage to the sample caused by multiple stainings and directly convert the stained images into other stainings, supporting different staining schemes for the same slice and optimizing the diagnostic path. In addition, algorithm-driven virtual staining can reduce human errors and improve the repeatability of results; at the same time, it reduces the use of reagents, which is in line with the trend of green medicine.

[0004] Currently, the technology used to achieve virtual staining of pathology sections is primarily based on generative adversarial networks (GANs). These methods are categorized by whether or not paired data is required: ① Conditional Generative Adversarial Networks (cGANs)-based methods and ② Cycle-Consistent Adversarial Networks (Cycle-GANs)-based methods. The former requires strict pixel-level data pairing during training to achieve staining mapping, while the latter can achieve virtual staining through training with unpaired data. Furthermore, to enhance the structural consistency of virtual staining results generated through unpaired data training, ③ a method based on diffusion models to optimize GAN results has recently emerged.

[0005] However, the aforementioned first-category approaches require paired stained and unstained images of the same tissue sample, data that is often unavailable through direct microscopy and results in high experimental costs. Furthermore, adversarial training can lead to structural distortion or spurious details (e.g., erroneous virtual stained regions), producing hallucinatory artifacts. The cycle consistency constraints of the aforementioned second-category approaches cannot fully guarantee structural alignment between input and output, potentially missing key pathological features and resulting in poor structural consistency. Furthermore, precise control of the staining style is difficult, prone to color overflow or local inconsistencies (e.g., partial regions not being virtually stained). While the aforementioned third-category approaches can generate highly consistent virtual staining results, the iterative process of the diffusion model is complex, computationally expensive, and time-consuming, making it difficult to meet real-time requirements (e.g., intraoperative pathology analysis). Furthermore, the black-box nature of the generative model makes the generation process lacking interpretability, making it difficult to trace the source of errors. Summary of the Invention

[0006] In view of the above, an object of the present invention is to provide a virtual staining method and system for pathological section images, which can realize efficient and repeatable staining of pathological section images by constructing a virtual staining lookup table model.

[0007] To achieve the above-mentioned object of the invention, an embodiment provides a virtual staining method for pathological section images, comprising the following steps:

[0008] Obtain the original image and target chemical dye image corresponding to each round of real dyeing and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label.

[0009] A virtual coloring lookup table model is constructed, which includes an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual coloring lookup tables. The adaptive feature encoder extracts an adaptive feature map from the original image and splices it with the original image channel to obtain a channel splicing map. The adaptive weight predictor obtains the adaptive weights of the 𝑛 basic virtual coloring lookup tables from the original image. The adaptive weighted fusion of the 𝑛 basic virtual coloring lookup tables obtains an adaptive fused virtual coloring lookup table. The adaptive fused virtual coloring lookup table is used to perform continuous color mapping on the channel splicing map to obtain a virtual coloring image. A supervised learning loss function is constructed based on the supervised label and the virtual coloring image, and the virtual coloring lookup table model parameters are optimized based on the loss function.

[0010] The virtual staining lookup table model with optimized parameters is used to perform virtual staining of pathological section images.

[0011] Preferably, the original image and the target chemical staining image corresponding to each round of real staining include an unstained image as the original image and a first stained image as the target chemical staining image corresponding to the first staining, and the first stained image as the original image and a second stained image as the target chemical staining image corresponding to the second staining;

[0012] The original image and the target chemically stained image are processed to obtain a paired database, including:

[0013] Use unpaired virtual staining to generate paired virtual staining images as pseudo labels for the original images; or,

[0014] Utilize pixel-level image registration to generate paired registered target chemical staining images as true labels for the original images.

[0015] Among them, pseudo labels and true labels are collectively referred to as supervised labels.

[0016] Preferably, the unpaired virtual staining method is used to generate a paired virtual staining image as a pseudo label for the original image, including:

[0017] The original image and the non-registered target chemical staining image are used as data pairs to train the unpaired virtual staining model, and the trained unpaired virtual staining model is used to infer the virtual staining image corresponding to the original image as a pseudo label, or the virtual staining image obtained by inferring the original image with the unpaired virtual staining model is further optimized with a pre-trained diffusion model and then used as a pseudo label;

[0018] Utilize pixel-level image registration to generate paired registered target chemically stained images as true labels for the original images, including:

[0019] The original image and the target chemical staining image are cropped to filter out the invalid background, and the two remaining images after cropping are subjected to primary image registration. Then, the images after primary registration are subjected to block-level registration to obtain the registered target chemical staining image corresponding to the original image as the true label.

[0020] Preferably, the adaptive feature encoder includes an input layer, multiple residual blocks, and an output layer, wherein each residual block includes a jump connection with the initial features extracted from the input layer, and instance normalization is introduced to achieve deep feature extraction and adaptive output.

[0021] Preferably, the adaptive weight predictor adopts a downsampling coding architecture, and obtains adaptive weights by performing multi-layer downsampling coding on the original image. These weights are automatically divided into two categories, which are represented as:

[0022] ;

[0023] ;

[0024] in, represents the weight parameter, represents the bias parameter, They represent the weight parameters of the RGB three color channels in the 𝑛th basic virtual coloring lookup table, Represents the bias parameter of the 𝑛th base virtual coloring lookup table.

[0025] Preferably, each basic virtual coloring lookup table is constructed based on an M-dimensional lookup table, and the mapping relationship of all elements therein is initialized to an identity mapping, which is expressed as: ,in, represents the th lookup table, Respectively represent The color mapping relationship of the three RGB color channels in the lookup table, the element value therein is represented as mapping from one pixel value to another pixel value, and the mapping relationship of each element is changed through training;

[0026] The adaptive fusion virtual coloring lookup table is obtained by adaptive weight fusion of 𝑛 basic virtual coloring lookup tables, which is expressed as ,in,

[0027] ;

[0028] in, Represents the RGB color space One of three channels.

[0029] Preferably, the method of continuously color mapping the channel mosaic using the adaptively fused virtual coloring lookup table to obtain the virtual coloring image comprises:

[0030] The channel splicing map is input into the adaptive fused virtual coloring lookup table, and a continuous color map is generated through M linear interpolation operations to obtain the virtual coloring image. At the same time, since the interpolation operation is differentiable, the adaptive feature encoder, adaptive weight predictor and the learnable parameters in each basic virtual coloring lookup table are updated during the training process by back-propagating the gradient.

[0031] Preferably, a supervised learning loss function is constructed based on the supervised labels and the virtual dyed image for training to optimize the learnable parameters in the model, which includes a pixel-level loss function and a color difference loss function.

[0032] The pixel-level loss function uses the mean square error loss between the virtual dyed image and the supervision label;

[0033] The color difference loss function uses the brightness difference, chromaticity difference, and hue difference between the virtual dyed image and the supervised label in the Lab color space.

[0034] Preferably, performing virtual staining of pathological slice images using the virtual staining lookup table model after parameter optimization includes:

[0035] The pre-processed original pathological section image is input into the virtual staining lookup table model with optimized parameters for virtual staining to generate a virtual staining image with the target staining style;

[0036] The virtual stained image and the original pathological section image are displayed side by side or superimposed for pathologists to compare and observe; it also provides interactive image functions, including zooming, rotation, annotation, and ROI labeling; it also supports one-click switching of different chemical staining styles and displays the virtual stained image of the corresponding chemical staining style.

[0037] To achieve the above-mentioned object of the invention, an embodiment of the present invention further provides a virtual staining system for pathological section images, comprising:

[0038] A database construction module is used to obtain the original image and target chemical staining image corresponding to each round of real staining and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label;

[0039] A model construction and training module is used to construct a virtual coloring lookup table model including an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual coloring lookup tables, wherein the original image is subjected to an adaptive feature encoder to extract an adaptive feature map and is spliced ​​with the original image channel to obtain a channel splicing map, the original image is subjected to an adaptive weight predictor to obtain the adaptive weights fused with 𝑛 basic virtual coloring lookup tables, the 𝑛 basic virtual coloring lookup tables are weightedly fused with the adaptive weights to obtain an adaptively fused virtual coloring lookup table, the adaptively fused virtual coloring lookup table is used to perform continuous color mapping on the channel splicing map to obtain a virtual coloring image, a supervised learning loss function is constructed based on the supervised label and the virtual coloring image, and the virtual coloring lookup table model parameters are optimized based on the loss function;

[0040] The virtual staining module of the model is used to perform virtual staining of pathological section images using a parameter-optimized virtual staining lookup table model.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention uses real staining to construct registration data pairs for different staining styles as training data, and also constructs a virtual staining lookup table model including an adaptive feature encoder, an adaptive weight predictor and 𝑛 basic virtual staining lookup tables. In the model, feature maps and weights are adaptively generated based on the input original image, and the 𝑛 basic virtual staining lookup tables are weighted based on the weights to construct an adaptive fusion virtual staining lookup table for the original image. Then, the virtual staining lookup table is used to perform virtual staining in various pathological and chemical staining styles. This process has high staining efficiency, strong adaptability, and diversified staining styles. It achieves staining by changing the color value without changing the pixel position and morphology, thereby ensuring the staining quality, and the stained area has interpretable characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a flow chart of a virtual staining method for pathological section images provided in an embodiment;

[0045] Figure 2 It is a structural diagram of the virtual dyeing lookup table model and the training process provided by the embodiment;

[0046] Figure 3 Schematic diagram of the structure of the virtual staining system for pathological section images provided in the embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0048] The inventive concept of this invention is as follows: The embodiments of the present invention provide a method and system for virtual staining of pathology slide images, which comprises three phases: the first phase constructs a paired database to support virtual staining training of pathology slides; the second phase uses the paired database to train a virtual staining lookup table model; and the third phase uses the virtual staining lookup table model to complete virtual staining according to the pathologist's needs, enabling auxiliary diagnosis. These three phases can solve the following technical problems:

[0049] (1) To address the difficulty in obtaining paired data directly through pathology microscope equipment, the present invention proposes a hierarchical preprocessing scheme from whole-view slice (WSI) to image patch (Patch) registration in the first stage, realizing pixel-level training data registration.

[0050] (2) To address the time-consuming problem of generating GAN results optimized by the diffusion model, the table lookup operation constructed in the second stage of this invention has extremely low complexity, achieving millisecond-level virtual staining, far exceeding the diffusion model (seconds) and GAN (hundreds of milliseconds). This invention can be used as a supervised method for direct and efficient real-time inference and to assist in intraoperative pathology virtual staining. It can also be used as an accelerated solution to learn the results of existing generative models (such as high-quality virtual staining results optimized by the diffusion model) and achieve model distillation.

[0051] (3) In response to the problem that the Cycle-GAN-based method may lose key pathological features and have poor structural consistency when realizing pathological virtual staining, the method proposed in this paper has strict structural consistency. The learnable lookup table scheme it adopts only changes the color value without modifying the pixel position or morphology, and naturally maintains the input structure. However, GAN and diffusion models may introduce deformation due to the generation process.

[0052] (4) To address the problem that methods based on cGAN and Cycle-GAN may produce hallucination artifacts when implementing pathological virtual staining, the method proposed in this paper distinguishes different tissue regions (such as cell nucleus and cytoplasm) by generating adaptive feature maps. Independent color mapping can be applied to each type of region to achieve pixel-level adaptive virtual staining and content-aware fine-tuning, greatly reducing artifacts.

[0053] (5) In view of the black-box characteristics of the output results of existing generative models that are difficult to interpret, the method proposed in this invention can visualize the input-output relationship and has transparent mapping rules, which can enhance the pathologist's trust in the virtual staining results.

[0054] Specifically, if Figure 1 As shown, the virtual dyeing method provided in the embodiment includes the following steps:

[0055] S1, build a pairing database: obtain the original image and target chemical staining image corresponding to each round of real staining and process them to obtain a pairing database. Each pair of paired data includes the original image and its corresponding supervision label.

[0056] In this embodiment, each round of true dyeing refers to chemical dyeing of any dyeing style applied to any original image, resulting in a corresponding target chemical dyeing image, where dyeing styles include HE, PAS, MAS, etc. The original image can be an undyed image, and the target chemical dyeing image can be a dyed image after each round of dyeing. Specifically, the original image and target chemical dyeing image corresponding to each round of true dyeing include an undyed image corresponding to the first dyeing as the original image and a first dyeing image as the target chemical dyeing image, and may also include a first dyeing image corresponding to the second dyeing as the original image and a second dyeing image as the target chemical dyeing image.

[0057] In this embodiment, image acquisition is first performed using a pathology microscope. The acquired images are either unstained images or images that have undergone a primary staining (first staining). For unstained images, chemical staining is performed using standard pathology staining procedures, followed by image acquisition using the pathology microscope again. This produces a paired dataset of unstained and first staining images. However, these paired datasets often fail to achieve registration, making them difficult to use for supervised training. For images that have undergone a primary staining, chemical restaining (second staining) is performed using standard pathology staining procedures, followed by image acquisition using the pathology microscope again. This produces a paired dataset of first staining and second staining images. However, these paired datasets often fail to achieve registration, making them difficult to use for supervised training.

[0058] In order to obtain a paired database that can be used for virtual staining supervised training, the present invention proposes two schemes. The first scheme refers to generating paired virtual staining images as pseudo labels for the original images through an unpaired virtual staining method. The second scheme refers to generating paired registered target chemical staining images as true labels for the original images through a pixel-level image registration method. Among them, the pseudo labels and true labels are collectively referred to as supervised labels, which are used for the subsequent supervised training of the virtual staining lookup table model.

[0059] For the first solution, it includes: using existing unpaired data pairs in the field or collecting original images and target chemical staining images that have not been registered as data pairs to train an unpaired virtual staining model, and using the trained unpaired virtual staining model to infer the virtual staining image corresponding to the original image as a pseudo label, or optimizing the virtual staining image through a pre-trained diffusion model and then using it as a pseudo label; finally, the input original image and pseudo label are used as a paired database for training the virtual staining lookup table model of the present invention to achieve model distillation of the unpaired virtual staining model.

[0060] For the second solution, it includes: cropping the original image and the stained image to filter out the invalid background (i.e., cropping the invalid background), and performing primary image registration (i.e., WSI-level registration) on the two remaining images after cropping. Then, performing block-level registration (i.e., Patch-level registration) on the images after primary registration to obtain the registered target chemical staining image corresponding to the original image as the true label.

[0061] The specific operation is: first, the background without valid information in the whole field of view (WSI) of the paired data is cropped by human-computer interaction or by automatically calculating the minimum bounding rectangle (MBR), wherein MBR refers to the rectangle with the smallest area that can completely contain the tissue area. When multiple scattered tissue blocks appear, the joint MBR is calculated. At this time, it is usually difficult for the paired WSI to have the same image size; then the cropped paired WSI is image registered, and its specific operation is to use the unsupervised framework DeeperHistReg based on deep learning to complete the preliminary registration. At this time, the paired WSI has the same image size, but the paired image blocks cropped at the same position usually cannot achieve pixel-level matching, which is difficult to use for the subsequent supervised learning training of the present invention; therefore, Patch (block) level registration is finally performed, and its specific operation is to first perform the preliminary registered paired WSI crops image blocks based on the overcropping principle. The overcropping principle here means that when the goal is to obtain a 256x256 pixel-level matching image block, the cropping should be larger than this size (for example, the highest digit is rounded by 1 to 300×300. Similarly, when 512×512 is the target, it can be overcropped to 600×600). Then, the key points and feature descriptors in the overcropped paired image blocks are extracted using the SIFT algorithm, and linear or nonlinear alignment of the images is achieved based on these feature matching to generate aligned paired images. Finally, 256×256 pixel-level registered image blocks are cropped to form the paired database for virtual coloring training of the present invention, so as to further achieve subsequent efficient and real-time virtual coloring.

[0062] S2, build and train the virtual coloring lookup table model: the model includes an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual coloring lookup tables.

[0063] In the embodiment, the virtual coloring lookup table model is used to virtually color the original image. Figure 2 As shown, first the original image x Input to the adaptive feature encoder to extract the adaptive feature map for enhanced color feature representation h , then h and x Perform channel dimension stitching to obtain a channel stitching graph f ,in fThe channel dimension is set to M. That is, if the input original image x has a channel dimension of N (0 < N <= M), then the output h of the adaptive feature encoder has a channel dimension of M - N. Specifically, the adaptive feature encoder includes an input layer, multiple residual blocks, and an output layer. Each residual block includes a skip connection established with the initial features extracted by the input layer, and at the same time, instance normalization is introduced to achieve deep feature extraction and adaptive output. Specifically, a three-stage residual architecture can be adopted: 1) The input layer converts the input original image x into L-channel features through 3×3 convolution + instance normalization, where L is a given integer; 2) The middle layer contains three cascaded residual blocks (each block contains 3×3 convolution + LeakyReLU + instance normalization), maintains the L-channel resolution, and fuses the initial features extracted by the input layer through skip connections; 3) The output layer uses 3×3 convolution with Dropout to compress the features into M - N channels, and outputs a normalized adaptive feature map after Sigmoid activation h . The entire adaptive feature encoder maintains the input size through same-resolution convolution (stride = 1), and uses residual connections + multi-level instance normalization to achieve deep feature extraction and adaptive output of the staining features. Note that any modification to the adaptive feature encoder to complete the network architecture with the same purpose of the present invention should be within the protection scope of the present invention.

[0064] Secondly, the present invention synchronously inputs the original image x into the adaptive weight predictor to obtain the adaptive weights for fusing n basic virtual staining lookup tables. These weights are automatically divided into two categories, respectively denoted as and , where represents the weight parameter, represents the bias parameter, respectively represent the weight parameters of the RGB three color channels in the nth basic virtual staining lookup table, represents the bias parameter of the nth basic virtual staining lookup table.

[0065] Specifically, the adaptive weight predictor adopts a downsampling encoding architecture, and obtains the adaptive weights by performing multi-layer downsampling encoding on the original image: 1) First, the input original image xSampling to 256×256 resolution; 2) Initial downsampling through a 3×3 convolution (stride = 2) to convert the 3-channel input into 16-channel features; 3) Four cascaded downsampling blocks (each containing 3×3 convolution (stride = 2) + LeakyReLU + instance normalization) are connected in series, and the number of channels is doubled step by step (16→32→64→128→128); 4) Finally, 8×8 convolution is used to compress the 128-channel features into a 1×1×p output, where 𝑝 = 10𝑛 is the number of prediction weights. The entire adaptive weight predictor implements the input image through progressive downsampling (total downsampling rate 256 times) and Dropout regularization x Note that any modification of the adaptive weight predictor to achieve the same purpose of the network architecture in the present invention should be within the scope of protection of the present invention.

[0066] In the embodiment, each basic virtual coloring lookup table is constructed based on an M-dimensional lookup table, and the mapping relationship of all elements therein is initialized to an identity mapping, which is expressed as: . Indicates the A lookup table, , Respectively represent The color mapping relationship of the three RGB color channels in the lookup table, the element value therein is represented as a mapping from one pixel value to another pixel value, and the mapping relationship of each element can also be changed through training, that is, it is learnable.

[0067] Among them, M is preferably 3 or 4, and more preferably 4. Experiments have verified that virtual coloring constructed based on a 4-dimensional lookup table has strong effectiveness.

[0068] The adaptive fusion virtual coloring lookup table is obtained by adaptive weight fusion of 𝑛 basic virtual coloring lookup tables, which is expressed as ,in,

[0069] ;

[0070] in, Represents the RGB color space One of three channels.

[0071] Afterwards, the adaptive fusion virtual dyeing lookup table is used to continuously color map the channel mosaic to obtain a virtual dyeing image. Specifically, the channel mosaic is f Input to the adaptive fusion virtual dyeing lookup table, and generate continuous color mapping through M linear interpolation operations to obtain the virtual dyeing image zSpecifically, the channel mosaic of the M channel is sent to the M-dimensional lookup table for color lookup conversion of the pixel value. Because the M-dimensional lookup table is discrete and cannot accurately contain all color mapping relationships, it is necessary to obtain the color mapping relationship closest to the pixel value of the input channel mosaic through interpolation operation, that is, to generate a continuous color mapping here, and then obtain a virtual dyed image. z At the same time, since the interpolation operation is differentiable, the gradient can be transferred backward so that the parameters of the adaptive feature encoder, adaptive weight predictor, and each basic virtual coloring lookup table can be updated during the training process.

[0072] Finally, the virtual stained image obtained in the above operation is z Based on supervised labels y Training is performed under the optimization constraints of the loss function to obtain a virtual color lookup table model for inferring the initial color to the target color. Note that for the trained inference model, each input image generates an adaptively fused virtual color lookup table, and the color mapping relationship between input and output can be clarified through slice visualization, which has clear interpretability.

[0073] Specifically, the loss function of the present invention uses a combination of a pixel-level loss function and a color difference loss function for optimization training.

[0074] Among them, the pixel-level loss function uses the mean square error loss between the virtual dyed image and the supervision label , the purpose is to ensure the virtual dyeing image z and supervision labels y The pixel-level reconstruction accuracy is defined as follows:

[0075] ;

[0076] Note that modifications of selecting other pixel-level loss functions to achieve the same purpose of the present invention should be within the scope of protection of the present invention.

[0077] Among them, the color difference loss function uses the color difference between the virtual dyed image and the supervised label in the Lab color space, including brightness difference, chromaticity difference and hue difference. Specifically calculate the color difference in CIE94 Lab color space , the purpose is to reduce the virtual dyeing image z With supervision label y The specific steps are to virtually dye the image z and supervision labels y Convert from RGB space to Lab color space and calculate brightness ( ), chroma and hue differences; The specific definition is:

[0078] ;

[0079] In the Lab color space Directly corresponds to the brightness perceived by the human eye, here is the brightness channel difference, which is used to measure the generated virtual stained image and the supervised label y the difference between light and dark; is the chromaticity difference, which quantifies the difference in color saturation, Characterize saturation; is the hue difference, which measures the difference in color tone; the weight coefficient and Adjust the chroma and hue sensitivity of high saturation areas separately; By decoupling brightness, chromaticity, and hue, it is more in line with the human visual perception of color differences. Note that other methods of calculating the color difference loss function to achieve the same purpose of the present invention should be within the scope of protection of the present invention.

[0080] In summary, the total loss function adopted by the present invention is is defined as follows:

[0081] ;

[0082] Here is a given weight used to adjust the importance of the two. When the input image is not an RGB image, it is set to 0. Note that any addition (such as adding a regularization term) or deletion (such as removing a ) Any changes to achieve the virtual coloring goal described in the present invention should be within the scope of protection of the present invention.

[0083] S3. Virtual staining of the model: Virtual staining of pathological section images is performed using a parameter-optimized virtual staining lookup table model.

[0084] In an embodiment, a parameter-optimized virtual staining lookup table model is used to assist pathologists in clinical diagnosis. Specifically, the parameter-optimized virtual staining lookup table model can be deployed in the digital pathology system of a hospital's pathology department to perform virtual staining on unstained or stained tissue slice images in real time or in batches, thereby assisting pathologists in conducting efficient and accurate clinical diagnosis. The specific process includes the following steps:

[0085] First, model deployment and system integration are performed. 1) The parameter-optimized virtual staining lookup table model is encapsulated as an inference service module and integrated into the digital pathology image management system. 2) The model can be deployed on local servers, edge computing devices, or within cloud-based image processing interfaces, supporting automatic or manually triggered image virtual staining requests. 3) It supports interfacing with whole-slide scanners to receive real-time unstained or initially stained tissue image data.

[0086] The raw image is then input and preprocessed. 1) After scanning the tissue slice, the pathologist imports the raw image into the virtual staining lookup table system. 2) The system preprocesses the image, including resolution normalization, color normalization, and region of interest (ROI) extraction, to ensure that the input image meets the model inference requirements. 3) The image can be obtained from label-free methods such as traditional brightfield imaging, phase imaging, and fluorescence imaging.

[0087] Then, virtual staining inference and result generation are performed. 1) The preprocessed original pathology slide image is fed into the virtual staining lookup table model. 2) Based on the learned mapping relationship between the original image and the staining pattern, the model generates a virtual stained image with the target staining style (such as HE, PAS, MAS, etc.). 3) The virtual stained image retains the structural details of the original image and closely matches the color style of the real stained image.

[0088] The diagnostic interface then displays and interacts with the original image. 1) Virtually stained images are displayed side-by-side or superimposed with the original image for pathologists to compare and observe. 2) Interactive functions such as image zooming, rotation, annotation, and ROI labeling are provided. 3) One-click switching between different staining styles (e.g., switching from HE to PAS) improves the efficiency of multi-angle pathology feature recognition.

[0089] Finally, auxiliary diagnosis and data archiving are performed. 1) Pathologists use virtual stained images to analyze tissue morphology, identify lesions, and make diagnostic judgments. 2) Virtual stained images can be archived or exported as part of electronic pathology reports for in-hospital consultations or telemedicine. 3) They can be combined with subsequent AI-assisted diagnosis models (such as segmentation, classification, and prediction models) to improve the accuracy of automated diagnosis.

[0090] In general, the present invention has at least the following clinical integration advantages during application: 1) Improved diagnostic efficiency: avoiding waiting for physical staining and reducing sample processing time; 2) Tissue sample protection: non-destructive staining, preserving samples for subsequent molecular experiments or multimodal analysis; 3) Multi-staining simulation capability: a single pathology slide image supports virtual switching of multiple staining styles, assisting doctors in analyzing pathological features from multiple perspectives; 4) Standardization and consistency: stable and consistent model output, reducing the interference of human staining bias on diagnosis; 5) Remote diagnosis support: suitable for remote consultation, mobile terminal film reading, and deployment of pathology auxiliary systems in low-resource areas.

[0091] like Figure 3 As shown, the embodiment also provides a virtual staining system 30 for pathological section images, including: a database construction module 31, a model construction and training module 32, and a virtual staining module 33 of the model, wherein the database construction module 31 is used to obtain the original image and the target chemical staining image corresponding to each round of real staining and obtain a paired database after processing, and each pair of paired data includes an original image and its corresponding supervisory label; the model construction and training module 32 is used to construct a virtual staining lookup table model including an adaptive feature encoder, an adaptive weight predictor and 𝑛 basic virtual staining lookup tables, construct a supervised learning loss function based on the supervisory label and the virtual staining image, and update the virtual staining lookup table model parameters based on the loss function; the virtual staining module 33 of the model is used to use the parameter-optimized virtual staining lookup table model to perform virtual staining of pathological section images.

[0092] It should be noted that the virtual staining device for pathological section images provided in the above embodiment, when performing virtual staining of images, should be illustrated by the division of the above-mentioned functional modules. The above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the virtual staining device for pathological section images provided in the above embodiment and the embodiment of the virtual staining method for pathological section images are based on the same concept. The specific implementation process is detailed in the embodiment of the virtual staining method for pathological section images, and will not be repeated here.

[0093] Based on the same inventive concept, an embodiment further provides a computing device including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, the device is used to implement the above-mentioned virtual staining method for pathological section images, which specifically includes the following steps:

[0094] S1, build a paired database: obtain the original image and target chemical dyeing image corresponding to each round of real dyeing and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label;

[0095] S2, build and train the virtual coloring lookup table model: the model includes an adaptive feature encoder, an adaptive weight predictor, and n A basic virtual coloring lookup table;

[0096] S3. Virtual staining of the model: Virtual staining of pathological section images is performed using a parameter-optimized virtual staining lookup table model.

[0097] The computing device provided in the embodiment, at the hardware level, includes not only a processor and memory, but also hardware required for other services such as an internal bus, a network interface, and memory. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the virtual staining method of the pathological section image described in S1-S3 above. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0098] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for virtual staining of a pathological section image is implemented, specifically comprising the following steps:

[0099] S1, build a paired database: obtain the original image and target chemical dyeing image corresponding to each round of real dyeing and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label;

[0100] S2, build and train the virtual coloring lookup table model: the model includes an adaptive feature encoder, an adaptive weight predictor, and n A basic virtual coloring lookup table;

[0101] S3. Virtual staining of the model: Virtual staining of pathological section images is performed using a parameter-optimized virtual staining lookup table model.

[0102] In the embodiment, computer-readable media includes permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0103] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A virtual staining method for pathological section images, characterized in that: The following steps are involved: Obtain the original image and target chemical dye image corresponding to each round of real dyeing and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervised label. A virtual coloring lookup table model is constructed, which includes an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual coloring lookup tables. The adaptive feature encoder extracts an adaptive feature map from the original image and splices it with the original image channel to obtain a channel splicing map. The adaptive weight predictor obtains the adaptive weights of the 𝑛 basic virtual coloring lookup tables from the original image. The adaptive weighted fusion of the 𝑛 basic virtual coloring lookup tables obtains an adaptive fused virtual coloring lookup table. The adaptive fused virtual coloring lookup table is used to perform continuous color mapping on the channel splicing map to obtain a virtual coloring image. A supervised learning loss function is constructed based on the supervised label and the virtual coloring image, and the virtual coloring lookup table model parameters are optimized based on the loss function. The virtual staining lookup table model with optimized parameters is used to perform virtual staining of pathological section images.

2. The virtual staining method for pathological section images according to claim 1, characterized in that: The original image and target chemical staining image corresponding to each round of real staining include the unstained image as the original image and the first stained image as the target chemical staining image corresponding to the first staining, and the first stained image as the original image and the second stained image as the target chemical staining image corresponding to the second staining; The original image and the target chemically stained image are processed to obtain a paired database, including: Use unpaired virtual staining to generate paired virtual staining images as pseudo labels for the original images; or, Utilize pixel-level image registration to generate paired registered target chemical staining images as true labels for the original images. Among them, pseudo labels and true labels are collectively referred to as supervised labels.

3. The virtual staining method for pathological section images according to claim 2, characterized in that: The unpaired virtual staining method is used to generate paired virtual staining images as pseudo labels for the original images, including: The original image and the non-registered target chemical staining image are used as data pairs to train the unpaired virtual staining model, and the trained unpaired virtual staining model is used to infer the virtual staining image corresponding to the original image as a pseudo label, or the virtual staining image obtained by inferring the original image with the unpaired virtual staining model is further optimized with a pre-trained diffusion model and then used as a pseudo label; Utilize pixel-level image registration to generate paired registered target chemically stained images as true labels for the original images, including: The original image and the target chemical staining image are cropped to filter out the invalid background, and the two remaining images after cropping are subjected to primary image registration. Then, the images after primary registration are subjected to block-level registration to obtain the registered target chemical staining image corresponding to the original image as the true label.

4. The virtual staining method for pathological section images according to claim 1, characterized in that: The adaptive feature encoder includes an input layer, multiple residual blocks, and an output layer, wherein each residual block includes a jump connection with the initial features extracted by the input layer, and introduces instance normalization to achieve deep feature extraction and adaptive output.

5. The virtual staining method for pathological section images according to claim 1, characterized in that: The adaptive weight predictor adopts a downsampling coding architecture. It obtains adaptive weights by performing multi-layer downsampling coding on the original image. These weights are automatically divided into two categories, represented as: ; ; in, represents the weight parameter, represents the bias parameter, They represent the weight parameters of the RGB three color channels in the 𝑛th basic virtual coloring lookup table, Represents the bias parameter of the 𝑛th base virtual coloring lookup table.

6. The virtual staining method for pathological section images according to claim 5, characterized in that: Each basic virtual coloring lookup table is constructed based on an M-dimensional lookup table, and the mapping relationship of all elements in it is initialized to an identity mapping, which is expressed as: ,in, represents the th lookup table, Respectively represent The color mapping relationship of the three RGB color channels in the lookup table, the element value therein is represented as mapping from one pixel value to another pixel value, and the mapping relationship of each element is changed through training; The adaptive fusion virtual coloring lookup table is obtained by adaptive weight fusion of 𝑛 basic virtual coloring lookup tables, which is expressed as ,in, ; in, Represents the RGB color space One of three channels.

7. The virtual staining method for pathological section images according to claim 1, characterized in that: The channel mosaic is continuously color-mapped using an adaptively fused virtual dye lookup table to obtain a virtual dye image, including: The channel splicing map is input into the adaptive fused virtual coloring lookup table, and a continuous color map is generated through M linear interpolation operations to obtain the virtual coloring image. At the same time, since the interpolation operation is differentiable, the adaptive feature encoder, adaptive weight predictor and the learnable parameters in each basic virtual coloring lookup table are updated during the training process by back-propagating the gradient.

8. The virtual staining method for pathological section images according to claim 1, characterized in that: Based on the supervised labels and the virtual dyed images, a supervised learning loss function is constructed for training to optimize the learnable parameters in the model, which includes pixel-level loss function and color difference loss function. The pixel-level loss function uses the mean square error loss between the virtual dyed image and the supervision label; The color difference loss function uses the brightness difference, chromaticity difference, and hue difference between the virtual dyed image and the supervised label in the Lab color space.

9. The virtual staining method for pathological section images according to claim 1, characterized in that: The virtual staining lookup table model with optimized parameters is used to perform virtual staining of pathological slide images, including: The pre-processed original pathological section image is input into the virtual staining lookup table model with optimized parameters for virtual staining to generate a virtual staining image with the target staining style; The virtual stained image and the original pathological section image are displayed side by side or superimposed for pathologists to compare and observe; it also provides interactive image functions, including zooming, rotation, annotation, and ROI labeling; it also supports one-click switching of different chemical staining styles and displays the virtual stained image of the corresponding chemical staining style.

10. A virtual staining system for pathological section images, characterized in that: include: The database construction module is used to obtain the original image and the target chemical staining image corresponding to each round of real staining and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label; A model construction and training module is used to construct a virtual coloring lookup table model including an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual coloring lookup tables, wherein the original image is subjected to an adaptive feature encoder to extract an adaptive feature map and is spliced ​​with the original image channel to obtain a channel splicing map, the original image is subjected to an adaptive weight predictor to obtain the adaptive weights fused with 𝑛 basic virtual coloring lookup tables, the 𝑛 basic virtual coloring lookup tables are weightedly fused with the adaptive weights to obtain an adaptively fused virtual coloring lookup table, the adaptively fused virtual coloring lookup table is used to perform continuous color mapping on the channel splicing map to obtain a virtual coloring image, a supervised learning loss function is constructed based on the supervised label and the virtual coloring image, and the virtual coloring lookup table model parameters are optimized based on the loss function; The virtual staining module of the model is used to perform virtual staining of pathological section images using a parameter-optimized virtual staining lookup table model.

Images