Virtual staining method and system for pathological section image

Abstract

The invention discloses a virtual staining method and system for a pathological section image, and belongs to the technical field of image processing, and the method comprises the steps: constructing a registration data pair for different staining styles through real staining as training data, meanwhile, a virtual dyeing lookup table model comprising a self-adaptive feature encoder, a self-adaptive weight predictor and a basic virtual dyeing lookup table is constructed, and in the model, a feature map and a weight are generated in a self-adaptive mode based on an input original image; the basic virtual dyeing lookup tables are weighted based on the weights to construct the virtual dyeing lookup table for adaptive fusion of the original image, and then the virtual dyeing lookup table is utilized to perform virtual color dyeing of various chemical dyeing styles, so that the dyeing efficiency in the process is high, the dyeing styles can be diversified, and the image quality is improved. The positions and forms of the pixels are not changed, only color values are changed, dyeing is achieved, the dyeing self-adaption is high, the dyeing quality is guaranteed, and the dyeing area has interpretability.

Description

Technical Field

[0001] The present invention belongs to the technical field of combining intelligent pathological auxiliary diagnosis and image processing, and particularly relates to a virtual staining method and system for pathological section images. Background Art

[0002] Histopathological staining plays an irreplaceable and important role in clinical pathological diagnosis, and can effectively enhance the contrast and visualization effect of cells and tissue structures. Among them, hematoxylin-eosin (HE) staining is the current gold standard for clinical pathological diagnosis, while special staining techniques such as periodic acid-Schiff (PAS) and Masson's trichrome (MAS) are used to highlight specific tissue components to improve the diagnostic rate of pathologists. However, traditional chemical staining methods generally have problems such as cumbersome operation procedures, long time consumption, dependence on chemical reagents, and high costs. In addition, the staining process is irreversible, and the stained samples are difficult to be used for the analysis of other staining types in some cases. At the same time, due to human operation differences, it is also easy to cause inconsistencies in staining quality.

[0003] With the rapid development of digital pathology and whole slide scanning technology, high-resolution tissue section images have been widely digitized, providing a technical basis for remote diagnosis, image quantitative analysis, and the application of artificial intelligence in the field of pathology. In this context, virtual staining, as an emerging technology, has become a potential solution to replace traditional chemical staining. Virtual staining can skip the traditional staining steps by using a deep learning model, directly generate target chemical staining images from unlabeled tissues, and accelerate the diagnosis process; it can also avoid the damage to samples caused by multiple stainings, directly convert the stained images into other stains, support different staining schemes for the same section, and optimize the diagnosis path. In addition, algorithm-driven virtual staining can reduce human errors, improve the repeatability of results; at the same time, it reduces the use of reagents, in line with the trend of green medicine.

[0004] Currently, the technologies used to implement virtual staining of pathological sections are mainly based on generative adversarial networks (GANs), which are divided into ① conditional generative adversarial network (cGAN)-based methods and ② cycle-consistent adversarial network (Cycle-GAN)-based methods according to whether paired data is required. The former requires strict pixel-level data pairing during training to achieve staining mapping, while the latter can achieve virtual staining through unpaired data training. In addition, to enhance the structural consistency of the virtual staining results generated by unpaired data training, recently, ③ a method based on optimizing GAN results with a diffusion model has emerged.

[0005] However, the above-mentioned type ① solutions require paired stained and unstained images of the same tissue sample, and such data pairs are usually not directly obtainable by microscopic equipment, resulting in high experimental costs. At the same time, adversarial training may lead to structural distortion or false details (such as incorrect virtual staining areas), generating hallucination artifacts. The cycle consistency constraint of the above-mentioned type ② solutions cannot fully guarantee the structural alignment between the input and output, may lose key pathological features, and has poor structural consistency; in addition, it is difficult to precisely control the staining style, and color bleeding or local inconsistencies (such as virtual staining not being achieved in some areas) are likely to occur. Although the above-mentioned type ③ solutions can generate virtual staining results with highly consistent structures, the iterative process of the diffusion model is complex, with high computational costs and long time consumption, making it difficult to meet real-time requirements (such as intraoperative pathological analysis). In addition, the black-box nature of the generative model makes the generation process lack interpretability and it is difficult to trace the source of errors. Summary of the Invention

[0006] In view of the above, the objective of the present invention is to provide a method and system for virtual staining of pathological section images, which can achieve efficient and repeatable staining of pathological section images through the constructed virtual staining look-up table model.

[0007] To achieve the above-mentioned invention objective, a method for virtual staining of pathological section images provided by an embodiment includes the following steps: Obtain the original image corresponding to each round of real staining and the target chemical staining image, and after processing, obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label; Construct a virtual staining look-up table model including an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual staining look-up tables. Among them, the original image passes through the adaptive feature encoder to extract an adaptive feature map, and after being concatenated with the original image channels, a channel concatenated map is obtained. The original image passes through the adaptive weight predictor to obtain an adaptive weight that fuses the 𝑛 basic virtual staining look-up tables. The 𝑛 basic virtual staining look-up tables are weighted and fused through the adaptive weight to obtain an adaptively fused virtual staining look-up table. Use the adaptively fused virtual staining look-up table to perform continuous color mapping on the channel concatenated map to obtain a virtual staining image. Based on the supervision label and the virtual staining image, construct a loss function for supervised learning, and optimize the parameters of the virtual staining look-up table model based on the loss function; Use the virtual staining look-up table model with optimized parameters to perform virtual staining of pathological section images.

[0008] Preferably, the original image and the target chemical staining image corresponding to each round of real staining include the unstained image as the original image and the first staining image as the target chemical staining image corresponding to the first staining, and the first staining image as the original image and the second staining image as the target chemical staining image corresponding to the second staining; The original image and the target chemical staining image are processed to obtain a paired database, including: Using the unpaired virtual staining method to generate a paired virtual staining image for the original image as a pseudo-label; or, Using the pixel-level image registration method to generate a paired registered target chemical staining image for the original image as a true label; Among them, the pseudo-label and the true label are collectively referred to as the supervision label.

[0009] Preferably, using the unpaired virtual staining method to generate a paired virtual staining image for the original image as a pseudo-label, including: Using the original image and the unregistered target chemical staining image as a data pair to train an unpaired virtual staining model, and using the trained unpaired virtual staining model to infer and obtain the virtual staining image corresponding to the original image as a pseudo-label, or further using the virtual staining image obtained by the unpaired virtual staining model inferring the original image and optimizing it through a pre-trained diffusion model as a pseudo-label; Using the pixel-level image registration method to generate a paired registered target chemical staining image for the original image as a true label, including: Cropping the original image and the target chemical staining image to filter out the invalid background, performing primary image registration on the remaining two images after cropping, and then performing block-level registration on the images after primary registration to obtain the registered target chemical staining image corresponding to the original image as a true label.

[0010] Preferably, the adaptive feature encoder includes an input layer, multiple residual blocks, and an output layer, where each residual block includes establishing a skip connection with the initial features extracted by the input layer, and at the same time introducing instance normalization to achieve deep feature extraction and adaptive output.

[0011] Preferably, the adaptive weight predictor adopts a downsampling encoding architecture, and obtains adaptive weights by performing multi-layer downsampling encoding on the original image. These weights are automatically divided into two categories, which are respectively represented as: ; ; Among them, represents the weight parameter, represents the bias parameter, respectively represent the weight parameters of the RGB three color channels in the 𝑛th basic virtual staining lookup table, represents the bias parameter of the 𝑛th basic virtual staining lookup table.

[0012] Preferably, each basic virtual staining lookup table is constructed based on an M-dimensional lookup table, and the mapping relationship of all its elements is initialized as an identity mapping, which is represented as: , where represents the th lookup table, respectively represent the color mapping relationships of the RGB three color channels in the th lookup table. The element values therein represent mapping from one pixel value to another pixel value, and the mapping relationship of each element is changed through training; The n basic virtual staining lookup tables are adaptively weighted and fused to obtain an adaptively fused virtual staining lookup table, denoted as , where ; where represents one of the three channels in the RGB color space.

[0013] Preferably, the adaptively fused virtual staining lookup table is used to perform continuous color mapping on the channel stitching map to obtain a virtual staining image, including: Input the channel stitching map into the adaptively fused virtual staining lookup table, and generate continuous color mapping through M - time linear interpolation operation to obtain a virtual staining image. At the same time, since the interpolation operation is differentiable, the learnable parameters in the adaptive feature encoder, adaptive weight predictor, and each basic virtual staining lookup table are updated during training through backpropagation of gradients.

[0014] Preferably, a supervised learning loss function is constructed based on the supervised label and the virtual staining image for training to optimize the learnable parameters in the model, which includes a pixel - level loss function and a color difference loss function, where the pixel - level loss function uses the mean square error loss between the virtual staining image and the supervised label; The color difference loss function uses the brightness difference, chromaticity difference, and hue difference between the virtual staining image and the supervised label in the Lab color space.

[0015] Preferably, the virtual staining lookup table model with optimized parameters is used for virtual staining of pathological section images, including: Input the pre - processed original pathological section image into the virtual staining lookup table model with optimized parameters for virtual staining to generate a virtual staining image with the target staining style; The virtual staining image and the original pathological section image are displayed side - by - side or in an overlay manner for pathologists to compare and observe; an image interaction function is also provided, including zooming, rotating, annotating, and ROI marking; it also supports one - key switching of different chemical staining styles and displaying the virtual staining images corresponding to the chemical staining styles.

[0016] To achieve the above - mentioned invention purpose, an embodiment of the present invention also provides a virtual staining system for pathological section images, including: A database construction module, which is used to obtain the original image corresponding to each round of real staining and the target chemical staining image, and obtain a paired database after processing. Each pair of paired data includes the original image and its corresponding supervision label; A model construction and training module, which is used to construct a virtual staining lookup table model including an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual staining lookup tables. Among them, the original image is used to extract an adaptive feature map through the adaptive feature encoder and obtain a channel concatenated map after concatenating with the channels of the original image. The original image is used to obtain an adaptive weight that fuses 𝑛 basic virtual staining lookup tables through the adaptive weight predictor. The 𝑛 basic virtual staining lookup tables are weighted and fused through the adaptive weight to obtain an adaptively fused virtual staining lookup table. The adaptively fused virtual staining lookup table is used to perform continuous color mapping on the channel concatenated map to obtain a virtual staining image. A supervised learning loss function is constructed based on the supervision label and the virtual staining image, and the parameters of the virtual staining lookup table model are optimized based on the loss function; The virtual staining module of the model, which is used to perform virtual staining on the pathological section image by using the virtual staining lookup table model with optimized parameters.

[0017] Compared with the prior art, the beneficial effects of the present invention at least include: The present invention constructs registration data pairs for different staining styles through real staining as training data, and also constructs a virtual staining lookup table model including an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual staining lookup tables. In the model, an adaptive feature map and weight are generated based on the input original image, and the 𝑛 basic virtual staining lookup tables are weighted based on the weight to construct an adaptively fused virtual staining lookup table for the original image. Then, the virtual staining lookup table is used to perform virtual staining of various pathological chemical staining styles. This process has high staining efficiency, strong adaptability, diverse staining styles, and only changes the color value without changing the pixel position and morphology to achieve staining, ensuring the staining quality, and the stained area has an interpretable characteristic. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a flowchart of the virtual staining method for pathological section images provided by the embodiment; Figure 2It is a schematic structural diagram and training process of the virtual staining lookup table model provided by the embodiment; Figure 3 It is a schematic structural diagram of the virtual staining system for pathological section images provided by the embodiment. Specific embodiments

[0020] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0021] The inventive concept of the present invention is as follows: The embodiments of the present invention provide a virtual staining method and system for pathological section images, which include three stages: The first stage constructs a paired database to support the training of virtual staining for pathological sections; the second stage uses the paired database to train the virtual staining lookup table model; the third stage uses the virtual staining lookup table model to complete virtual staining according to the needs of pathologists to achieve auxiliary diagnosis. These three stages can solve the following technical problems: (1) Aiming at the problem of difficulty in directly obtaining paired data through pathological microscopy equipment, the present invention proposes a hierarchical preprocessing scheme for registration from whole-slide images (WSIs) to image patches (Patches) in the first stage, realizing pixel-level registration of training data.

[0022] (2) Aiming at the problem of time-consuming generation of GAN results optimized by diffusion models, the lookup table constructed in the second stage of the present invention has extremely low operation complexity, can achieve millisecond-level virtual staining, far exceeding diffusion models (second-level) and GANs (hundred-millisecond-level). The present invention can be used as a supervised method for direct, efficient and real-time inference to assist intraoperative pathological virtual staining, and can also be used as an acceleration scheme to learn the results of existing generative models (such as high-quality virtual staining results optimized by diffusion models) to achieve model distillation.

[0023] (3) Aiming at the problem that the method based on Cycle-GAN may lose key pathological features and have poor structural consistency when realizing pathological virtual staining, the method proposed by the present invention has strict structural consistency. The learnable lookup table scheme adopted only changes the color value, does not modify the pixel position or morphology, and naturally maintains the input structure, while GAN and diffusion models may introduce deformations during the generation process.

[0024] (4) Aiming at the problem that the methods based on cGAN and Cycle-GAN may generate hallucination artifacts when realizing pathological virtual staining, the method proposed by the present invention distinguishes different tissue regions (such as cell nuclei and cytoplasm) in the form of generating adaptive feature maps, and can apply independent color mappings to each type of region to achieve pixel-level adaptive virtual staining and content-aware fine adjustment, greatly reducing artifacts.

[0025] (5) Regarding the black-box feature that the output results of existing generation models are difficult to interpret, the method proposed in the present invention can visualize the input-output relationship, has a transparent mapping rule, and can enhance the trust of pathologists in virtual staining results.

[0026] Specifically, as Figure 1 shown, the virtual staining method provided by the embodiment includes the following steps: S1, constructing a paired database: obtaining the original image corresponding to each round of real staining and the target chemical staining image and processing them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label.

[0027] In the embodiment, each round of real staining refers to performing chemical staining in any staining style on an arbitrary original image to obtain a corresponding target chemical staining image. Among them, the staining styles include HE, PAS, MAS, etc. The original image can be an unstained image, and the target chemical staining image can be the stained image after each round of staining. Specifically, the original image and the target chemical staining image corresponding to each round of real staining include the unstained image as the original image and the first stained image as the target chemical staining image corresponding to the first staining, and can also include the first stained image as the original image and the second stained image as the target chemical staining image corresponding to the second staining.

[0028] In the embodiment, first, image acquisition is performed through a pathological microscope device. The acquisition object is an image without staining marks or an image with primary staining marks (first staining). For the unstained image, after performing chemical staining through standard pathological staining steps, image acquisition is performed again through the pathological microscope device to obtain an unstained-first stained paired data set. The paired data in it usually fails to achieve registration and is difficult to be used for supervised training; for the image with primary staining marks, after performing chemical restaining (second staining) through standard pathological staining steps, image acquisition is performed again through the pathological microscope device to obtain a first stained-second stained paired data set. The paired data in it usually fails to achieve registration and is difficult to be used for supervised training.

[0029] In order to obtain a paired database that can be used for virtual staining supervised training, the present invention proposes two solutions. The first solution refers to generating paired virtual staining images as pseudo-labels for the original image through an unpaired virtual staining method. The second solution refers to generating paired registered target chemical staining images as real labels for the original image through a pixel-level image registration method. Among them, the pseudo-labels and real labels are collectively referred to as supervision labels and are used for the supervised training of the subsequent virtual staining lookup table model.

[0030] For the first solution, it includes: using the unpaired data pairs existing in the field or collecting the original image and the target chemical staining image that has not been registered as a data pair to train the unpaired virtual staining model, and using the trained unpaired virtual staining model to infer and obtain the virtual staining image corresponding to the original image as a pseudo-label, or using the virtual staining image as a pseudo-label after optimizing it through a pre-trained diffusion model; finally, taking the input original image and the pseudo-label as the paired database for training the virtual staining lookup table model of the present invention to realize the model distillation of the unpaired virtual staining model.

[0031] For the second solution, it includes: cropping the original image and the stained image to filter out the invalid background (i.e., cropping the invalid background), and performing primary image registration (i.e., WSI-level registration) on the remaining two images after cropping, and then performing block-level registration (i.e., Patch-level registration) on the images after primary registration to obtain the registered target chemical staining image corresponding to the original image as the true label.

[0032] The specific operation is as follows: First, cut off the background without valid information in the whole slide image (WSI) of the paired data pair through human-computer interaction or by automatically calculating the minimum bounding rectangle (MBR). Here, MBR refers to the rectangle with the smallest area that can completely contain the tissue area. When there are multiple scattered tissue blocks, the combined MBR is calculated. At this time, it is usually difficult for the paired WSI to have the same image size; then perform image registration on the cropped paired WSI. The specific operation is to complete the primary registration using the unsupervised framework DeeperHistReg based on deep learning. At this time, the paired WSI has the same image size, but the paired image blocks cropped at the same position usually cannot achieve pixel-level matching and are difficult to be used for the subsequent supervised learning training of the present invention; therefore, finally perform Patch (block)-level registration. The specific operation is to first crop the paired WSI after primary registration based on the overcropping principle to obtain image blocks. Here, the overcropping principle means that when the target is to obtain a 256x256 pixel-level matching image block, the cropping size should be larger than this size (for example, the highest digit is rounded up to 300×300, and similarly, when 512×512 is the target, it can be overcropped to 600×600). Then, extract the key points and feature descriptors in the image from the overcropped paired image blocks through the SIFT algorithm and realize the linear or non-linear alignment of the images based on these feature matches to generate the aligned paired images. Finally, crop to obtain 256×256 pixel-level registered image blocks to form the paired database for virtual staining training of the present invention to further realize subsequent efficient and real-time virtual staining.

[0033] S2. Construct and train the virtual staining lookup table model: The model includes an adaptive feature encoder, an adaptive weight predictor, and n basic virtual staining lookup tables.

[0034] In the embodiment, the virtual staining lookup table model is used to perform virtual staining on the original image. As Figure 2 shown, first, the original image x is input into the adaptive feature encoder to extract an adaptive feature map that enhances the staining feature representation h , and then h and x are concatenated in the channel dimension to obtain a channel concatenated map f , where f has a channel dimension set to M. That is to say, if the channel dimension of the input original image x is N (0 < N <= M), then the h output by the adaptive feature encoder has a channel dimension of M - N. Specifically, the adaptive feature encoder includes an input layer, multiple residual blocks, and an output layer. Each residual block includes a skip connection established with the initial features extracted by the input layer, and at the same time introduces instance normalization to achieve deep extraction and adaptive output of features. Specifically, a three-stage residual architecture can be adopted: 1) The input layer converts the input original image x into L-channel features through 3×3 convolution + instance normalization, where L is a given integer; 2) The middle layer contains three cascaded residual blocks (each block contains 3×3 convolution + LeakyReLU + instance normalization), maintains the L-channel resolution, and fuses the initial features extracted by the input layer through skip connections; 3) The output layer uses 3×3 convolution with Dropout to compress the features into M - N channels, and outputs a normalized adaptive feature map h after Sigmoid activation. The entire adaptive feature encoder maintains the input size through same-resolution convolution (stride = 1), and uses residual connections + multi-level instance normalization to achieve deep extraction and adaptive output of staining features. Note that any modification to the adaptive feature encoder to complete the network architecture with the same purpose of the present invention should be within the protection scope of the present invention.

[0035] Secondly, the present invention synchronously inputs the original image x into the adaptive weight predictor to obtain adaptive weights that fuse n basic virtual staining lookup tables. These weights are automatically divided into two categories, respectively denoted as and , where represents the weight parameter, represents the bias parameter, respectively represent the weight parameters of the RGB three color channels in the nth basic virtual staining lookup table, represents the bias parameter of the nth basic virtual staining lookup table.

[0036] Specifically, the adaptive weight predictor adopts a downsampling coding architecture to obtain adaptive weights by performing multi-layer downsampling coding on the original image: 1) First, the input original image is converted into x 1) Sampling to 256×256 resolution; 2) Performing an initial downsampling through a 3×3 convolution (stride = 2) to convert the 3-channel input into a 16-channel feature; 3) Connecting four cascaded downsampling blocks in series (each block contains 3×3 convolution (stride = 2) + LeakyReLU + instance normalization), the number of channels is doubled step by step (16→32→64→128→128); 4) Finally, using 8×8 convolution to compress the 128-channel feature into a 1×1×p output, where 𝑝 = 10𝑛 is the number of prediction weights. The entire adaptive weight predictor implements the input image through progressive downsampling (total downsampling rate 256 times) and Dropout regularization x Note that any modification of the adaptive weight predictor to achieve the same purpose of the network architecture in the present invention should be within the protection scope of the present invention.

[0037] In the embodiment, each basic virtual coloring lookup table is constructed based on an M-dimensional lookup table, and the mapping relationship of all elements therein is initialized to an identity mapping, which is expressed as: . Indicates A lookup table, , Respectively represent The color mapping relationship of the three RGB color channels in the lookup table, the element value therein is represented as a mapping from one pixel value to another pixel value, and the mapping relationship of each element can also be changed through training, that is, it is learnable.

[0038] Among them, M is preferably 3 or 4, and more preferably 4. Experiments have verified that virtual coloring based on a 4-dimensional lookup table to construct a basic virtual coloring lookup table is highly effective.

[0039] The adaptive fused virtual coloring lookup table is obtained by adaptive weighted fusion of 𝑛 basic virtual coloring lookup tables, which is expressed as ,in, ; in, Represents the RGB color space One of three channels.

[0040] Afterwards, the channel mosaic is continuously color mapped using the adaptive fusion virtual dyeing lookup table to obtain a virtual dyeing image. fInput into the virtual staining lookup table for adaptive fusion, and generate a continuous color mapping through M - time linear interpolation operation to obtain the virtual staining image z Specifically, the channel - concatenated map of M channels is sent into the M - dimensional lookup table for color lookup conversion of pixel values. Since the M - dimensional lookup table is discrete and cannot precisely contain all color mapping relationships, interpolation operations are required to obtain the color mapping relationship closest to the pixel values of the input channel - concatenated map, that is, to generate a continuous color mapping here, and then obtain the virtual staining image z At the same time, because the interpolation operation is differentiable, the gradient can be back - propagated so that the parameters in the adaptive feature encoder, the adaptive weight predictor, and each basic virtual staining lookup table can be updated during training.

[0041] Finally, the virtual staining image obtained in the foregoing operations z Based on the supervision label y Is trained under the optimization constraint of the loss function to obtain a virtual staining lookup table model for inferring the primary staining to the target staining. Note that for the trained inference model, each input image will generate an adaptive - fusion virtual staining lookup table, and the color mapping relationship from input to output can be clarified by slicing and visualizing it, with clear interpretability.

[0042] Specifically, the loss function of the present invention uses a combination of a pixel - level loss function and a color - difference loss function for optimization training.

[0043] Among them, the pixel - level loss function adopts the mean - square error loss between the virtual staining image and the supervision label The purpose is to ensure the pixel - level reconstruction accuracy of the virtual staining image z And the supervision label y Is defined as follows: ; Note that any changes in choosing other pixel - level loss functions to achieve the same purpose of the present invention should be within the protection scope of the present invention.

[0044] Among them, the color - difference loss function adopts the color difference between the virtual staining image and the supervision label in the Lab color space, including luminance difference, chromaticity difference, and hue difference. Specifically, calculate the color difference in the CIE94 Lab color space The purpose is to reduce the color difference between the virtual staining image z And the supervision label y The specific steps are to convert the virtual staining image z And the supervision label y From the RGB space to the Lab color space, and calculate the luminance ( ), chromaticity, and hue differences respectively; The specific definition is as follows: ; In the Lab color space, directly corresponds to the brightness perceived by the human eye. Here, is the brightness channel difference, used to measure the light and dark difference between the generated virtual stained image and the supervised label y ; is the chromaticity difference, quantifying the difference in color saturation, representing saturation; is the hue difference, measuring the difference in color tone; the weight coefficients and respectively adjust the chromaticity and hue sensitivities in the high saturation region; By decoupling brightness, chromaticity, and hue, it better conforms to the perception characteristics of human vision for color differences. Note that any changes in calculating the color difference loss function in other ways to achieve the same purpose of the present invention should be within the protection scope of the present invention.

[0045] In summary, the total loss function adopted by the present invention is defined as follows: ; Here, is a given weight used to adjust the importance of the two. When the input image is not an RGB image, it is set to 0. Note that any addition (such as adding a regularization term) or deletion (such as removing )to this loss function to achieve the virtual staining goal described in the present invention should be within the protection scope of the present invention.

[0046] S3. Virtual staining of the model: Use the virtual staining lookup table model with optimized parameters to perform virtual staining on the pathological section images.

[0047] In the embodiment, the virtual staining lookup table model with optimized parameters is used to assist pathologists in clinical diagnosis. Specifically, the virtual staining lookup table model with optimized parameters can be deployed in the digital pathology system of the hospital pathology department to perform virtual staining processing on unstained or stained tissue section images in real time or in batches, thereby assisting pathologists in carrying out efficient and accurate clinical diagnosis. The specific process includes the following steps: First, perform model deployment and system integration. 1) Package the virtual staining lookup table model with optimized parameters into an inference service module and integrate it into the digital pathology image management system; 2) The model can be deployed on a local server, edge computing device, or image processing interface based on a cloud platform, supporting automatically or manually triggered image virtual staining requests; 3) Support connection with the whole slide scanner interface to receive tissue image data that is unstained or initially stained in real time.

[0048] Then, the input and preprocessing of the original image are carried out. 1) After the pathologist scans the tissue section, the original image is imported into the virtual staining lookup table system; 2) The system preprocesses the image, including operations such as resolution standardization, color normalization, and region cropping (ROI extraction), to ensure that the input image meets the requirements of model inference; 3) The image can be sourced from label-free methods such as traditional bright-field imaging, phase imaging, and fluorescence imaging.

[0049] Then, virtual staining inference and result generation are performed. 1) The preprocessed original pathological section image is fed into the virtual staining lookup table model; 2) Based on the learned mapping relationship between the original image and the stained image, the model generates a virtual stained image with the target staining style (such as HE, PAS, MAS, etc.); 3) The virtual stained image can retain the structural details of the original image and highly fit the real stained image in terms of color style.

[0050] Then, the diagnostic interface display and interaction are carried out. 1) The virtual stained image and the original image are displayed side by side or in an overlay manner for the pathologist to compare and observe; 2) Interactive functions such as image zooming, rotation, annotation, and ROI marking are provided; 3) One-key switching between different staining styles (such as virtual switching from HE to PAS) is supported to improve the efficiency of pathological feature recognition from multiple angles.

[0051] Finally, auxiliary diagnosis and data archiving are carried out. 1) The pathologist conducts tissue morphology analysis, lesion area identification, and diagnostic judgment based on the virtual stained image; 2) The virtual stained image can be archived or exported as part of an electronic pathology report for in-hospital consultation or telemedicine; 3) It supports joint use with subsequent AI auxiliary diagnosis models (such as segmentation, classification, and prediction models) to improve the accuracy of automatic diagnosis.

[0052] Generally speaking, the present invention has at least the following clinical integration advantages during the application process: 1) Improved diagnostic efficiency: Avoid waiting for physical staining and reduce sample processing time; 2) Tissue sample protection: Stain without damage and retain the sample for subsequent molecular experiments or multimodal analysis; 3) Multi-staining simulation ability: One pathological section image supports virtual switching of multiple staining styles to assist doctors in analyzing pathological features from multiple angles; 4) Standardization and consistency: The model output is stable and consistent, reducing the interference of human staining deviation on diagnosis; 5) Remote diagnosis support: Suitable for remote consultation, viewing images on mobile terminals, and deploying pathological assistance systems in low-resource areas.

[0053] Such as Figure 3As shown, the embodiment also provides a virtual staining system 30 for pathological section images, including: a database construction module 31, a model construction and training module 32, and a virtual staining module 33 of the model. Among them, the database construction module 31 is used to obtain the original image corresponding to each round of real staining and the target chemical staining image, and obtain a paired database after processing. Each pair of paired data includes the original image and its corresponding supervision label; the model construction and training module 32 is used to construct a virtual staining lookup table model including an adaptive feature encoder, an adaptive weight predictor, and 𝑛 basic virtual staining lookup tables, construct a loss function for supervised learning based on the supervision label and the virtual staining image, and update the parameters of the virtual staining lookup table model based on the loss function; the virtual staining module 33 of the model is used to perform virtual staining of pathological section images using the virtual staining lookup table model with optimized parameters.

[0054] It should be noted that when the virtual staining device for pathological section images provided in the above embodiment performs image virtual staining, the above-mentioned division of each functional module should be used for illustration. The above functions can be allocated to different functional modules according to needs, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the virtual staining device for pathological section images provided in the above embodiment and the embodiment of the virtual staining method for pathological section images belong to the same concept. For the specific implementation process, please refer to the embodiment of the virtual staining method for pathological section images, which will not be elaborated here.

[0055] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, it is used to implement the above-mentioned virtual staining method for pathological section images, specifically including the following steps: S1. Construct a paired database: Obtain the original image corresponding to each round of real staining and the target chemical staining image, and obtain a paired database after processing. Each pair of paired data includes the original image and its corresponding supervision label; S2. Construct and train a virtual staining lookup table model: The model includes an adaptive feature encoder, an adaptive weight predictor, and n basic virtual staining lookup tables; S3. Virtual staining of the model: Use the virtual staining lookup table model with optimized parameters to perform virtual staining of pathological section images.

[0056] The computing device provided by the embodiment, at the hardware level, in addition to including a processor and a memory, also includes an internal bus, a network interface, a memory, and other hardware required for other services. The memory is a non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the virtual staining method of the pathological section image described in S1-S3 above. Of course, in addition to the software implementation method, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or logic devices.

[0057] Based on the same inventive concept, the embodiment also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the virtual staining method of the pathological section image described above is implemented, which specifically includes the following steps: S1. Construct a paired database: Obtain the original image corresponding to each round of real staining and the target chemical staining image and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label; S2. Construct and train a virtual staining look-up table model: The model includes an adaptive feature encoder, an adaptive weight predictor, and n basic virtual staining look-up tables; S3. Virtual staining of the model: Use the virtual staining look-up table model with optimized parameters to perform virtual staining on the pathological section image.

[0058] In the embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0059] The specific embodiments described above have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual staining method for pathological section images, characterized in that, The steps include: Obtain the original image corresponding to each round of real staining and the target chemical staining image, and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervision label; Construct a virtual staining lookup table model including an adaptive feature encoder, an adaptive weight predictor, and n basic virtual staining lookup tables. Among them, the original image passes through the adaptive feature encoder to extract an adaptive feature map, and after concatenating with the original image channels, a channel concatenated map is obtained. The original image passes through the adaptive weight predictor to obtain an adaptive weight that fuses n basic virtual staining lookup tables. The n basic virtual staining lookup tables are weighted and fused adaptively to obtain an adaptively fused virtual staining lookup table. Use the adaptively fused virtual staining lookup table to perform continuous color mapping on the channel concatenated map to obtain a virtual staining image. Construct a loss function for supervised learning based on the supervision label and the virtual staining image, and optimize the parameters of the virtual staining lookup table model based on the loss function; Use the virtual staining lookup table model with optimized parameters to perform virtual staining on pathological section images.

2. The virtual staining method of the pathological section image according to claim 1, wherein The original image and the target chemical staining image corresponding to each round of real staining include the unstained image as the original image and the first staining image as the target chemical staining image corresponding to the first staining, and the first staining image as the original image and the second staining image as the target chemical staining image corresponding to the second staining; The original image and the target chemical staining image are processed to obtain a paired database, including: Adopt an unpaired virtual staining method to generate a paired virtual staining image for the original image as a pseudo label; or, Use a pixel-level image registration method to generate a paired registered target chemical staining image for the original image as a real label; Among them, the pseudo label and the real label are collectively referred to as the supervision label.

3. The virtual staining method for pathological section images according to claim 2, characterized in that, Adopt an unpaired virtual staining method to generate a paired virtual staining image for the original image as a pseudo label, including: Use the original image and the unregistered target chemical staining image as a data pair to train an unpaired virtual staining model, and use the trained unpaired virtual staining model to infer and obtain the virtual staining image corresponding to the original image as a pseudo label, or further optimize the virtual staining image obtained by the unpaired virtual staining model inferring the original image with a pre-trained diffusion model and then use it as a pseudo label; Use a pixel-level image registration method to generate a paired registered target chemical staining image for the original image as a real label, including: Crop the original image and the target chemical staining image to filter out the invalid background, perform primary image registration on the remaining two images after cropping, and then perform block-level registration on the images after primary registration to obtain the registered target chemical staining image corresponding to the original image as a real label.

4. The virtual staining method for pathological section images according to claim 1, wherein The adaptive feature encoder includes an input layer, multiple residual blocks, and an output layer. Each residual block includes establishing a skip connection with the initial feature extracted by the input layer, and at the same time introducing instance normalization to achieve deep feature extraction and adaptive output.

5. The virtual staining method for pathological section images according to claim 1, wherein, The adaptive weight predictor adopts a downsampling encoding architecture, and obtains an adaptive weight by performing multi-layer downsampling encoding on the original image. These weights are automatically divided into two categories, which are respectively represented as: ; ; Among them, represents a weight parameter, represents a bias parameter, respectively represent the weight parameters of the RGB three color channels in the 𝑛th basic virtual staining lookup table, represents the bias parameter of the 𝑛th basic virtual staining lookup table.

6. The virtual staining method of the pathological section image according to claim 1, wherein Each basic virtual color lookup table is constructed based on an M - dimensional lookup table, and the mapping relationships of all elements in it are initially set to identity mappings, expressed as: , where represents the th lookup table, respectively represent the color mapping relationships of the three RGB color channels in the th lookup table. The element values in it represent the mapping from one pixel value to another pixel value, and the mapping relationship of each element is changed through training; The adaptive fusion virtual color lookup table is obtained by adaptively weighted fusion of 𝑛 basic virtual color lookup tables, denoted as , where ; Among them, represents one of the three channels in the RGB color space.

7. The virtual staining method of the pathological section image according to claim 1, characterized in that, Obtaining a virtual staining image by performing continuous color mapping on a channel - spliced image using an adaptively - fused virtual staining look - up table, including: Inputting the channel - spliced image into the adaptively - fused virtual staining look - up table, generating continuous color mapping through M - time linear interpolation operations to obtain the virtual staining image. At the same time, since the interpolation operation is differentiable, the learnable parameters in the adaptive feature encoder, adaptive weight predictor, and each basic virtual staining look - up table are updated during the training process through back - propagation of gradients.

8. The virtual staining method of the pathological section image according to claim 1, characterized in that Constructing a loss function for supervised learning based on the supervised label and the virtual staining image for training to optimize the learnable parameters in the model, which includes a pixel - level loss function and a color - difference loss function. Among them, the pixel - level loss function uses the mean - square error loss between the virtual staining image and the supervised label. The color - difference loss function uses the luminance difference, chromaticity difference, and hue difference between the virtual staining image and the supervised label in the Lab color space.

9. The virtual staining method of the pathological section image according to claim 1, wherein Performing virtual staining of pathological section images using the virtual staining look - up table model with optimized parameters, including: Inputting the pre - processed original pathological section image into the virtual staining look - up table model with optimized parameters for virtual staining to generate a virtual staining image with the target staining style. Displaying the virtual staining image and the original pathological section image side - by - side or in an overlay manner for pathologists to compare and observe; also providing image interaction functions, including zooming, rotation, annotation, and ROI marking; and also supporting one - key switching of different chemical staining styles to display the virtual staining images corresponding to the chemical staining styles.

10. A virtual staining system for pathological section images, characterized in that, Including: A database construction module, which is used to obtain the original image corresponding to each round of real staining and the target chemical staining image and process them to obtain a paired database. Each pair of paired data includes the original image and its corresponding supervised label. A model construction and training module, which is used to construct a virtual staining look - up table model including an adaptive feature encoder, an adaptive weight predictor, and n basic virtual staining look - up tables. Among them, the original image is used to extract an adaptive feature map through the adaptive feature encoder and then obtain a channel - spliced image after channel splicing with the original image. The original image is used to obtain the adaptive weights for fusing n basic virtual staining look - up tables through the adaptive weight predictor. The n basic virtual staining look - up tables are weighted and fused through the adaptive weights to obtain an adaptively - fused virtual staining look - up table. Using the adaptively - fused virtual staining look - up table to perform continuous color mapping on the channel - spliced image to obtain the virtual staining image, constructing a loss function for supervised learning based on the supervised label and the virtual staining image, and optimizing the parameters of the virtual staining look - up table model based on the loss function. A virtual staining module of the model, which is used to perform virtual staining of pathological section images using the virtual staining look - up table model with optimized parameters.

Images