A WSI recoloring method based on self-supervised learning

Abstract

The present invention discloses a WSI recoloring method based on self-supervised learning. First, the WSI image is preprocessed to obtain image blocks of the same size; then, image blocks corresponding to different perspectives of the image blocks are obtained respectively, and all the obtained image blocks are input into the designed two-way recoloring network; finally, through self-supervised learning loss function training, images from different scanners and images obtained by data enhancement are made consistent with the normalized image of the current image, the palette matrix of the same source is made consistent, and images repaired by other palettes are made consistent with the corresponding image; in the inference stage, the reconstructed palette matrix of the reference image block and the normalized image of the target image are obtained to obtain the target image recolored by the reference image. The present invention can improve the consistency of the recolored image while taking up less time and space, generate clear and high-quality recolored images, and provide assistance for subsequent processing and judgment of doctors and models.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and in particular to a WSI recoloring method based on self-supervised learning. Background Art

[0002] Computational pathology has become increasingly popular with the advent of high-precision microscope slide scanners and the development of deep neural networks. It aims to objectively and quantitatively assess the disease microenvironment at the microscopic level, ultimately enabling precision medicine, encompassing disease diagnosis and treatment response. While CNN models have performed well in various computational pathology applications, most suffer from color inconsistency. This is caused by inconsistent tissue thickness during slicing, different dye manufacturers, and varying slicing protocols. Furthermore, scanning using different scanners can also lead to color inconsistency, which can affect the generalization performance of learning-based models, especially when applied to data from different institutions. This is why color normalization is always an essential preprocessing step before model training.

[0003] In recent years, a series of methods for stain normalization have emerged, such as Vahadane, Reinhard, and STST. However, most of these methods lack flexibility in practice. At the same time, these methods require the input of a reference image and the original image at the same time, and cannot reuse the color information of the reference image, making it inefficient. To solve the above problems, a model called RestainNet defines stain normalization as a digital re-staining process, learning how to re-stain an unstained (grayscale) image. This method separates the destaining and re-staining processes, but it uses a GAN network for re-coloring, which has problems with visual artifacts, large memory requirements, and slow color conversion speed. Visual artifacts will affect the judgment of doctors and models on the data, and WSI images are usually large in scale. Large memory requirements and slow color conversion speed will hinder the application of the model. Summary of the Invention

[0004] The purpose of the present invention is to overcome the problems existing in the above-mentioned prior art and to provide a WSI recoloring method based on self-supervised learning, which can complete the recoloring of WSI images with less time and space occupation, generate clear and high-quality recolored images, and provide assistance for subsequent processing and judgment of doctors and models.

[0005] In order to achieve the above object, the present invention provides a WSI recoloring method based on self-supervised learning, comprising the following steps:

[0006] S1, preprocess the WSI image to obtain image blocks of the same size;

[0007] S2, obtaining image blocks corresponding to different perspectives of the image block;

[0008] S3. Input all the obtained image blocks into a two-way recoloring network to obtain a normalized image containing structural information and a normalized palette matrix containing color information, and reconstruct the palette matrix;

[0009] S4. Using a self-supervised learning loss function, the two-way recoloring network is trained to make images from different scanners and data-augmented images consistent with the normalized image of the current image, the palette matrices from the same source consistent, and images restored with other palettes consistent with the corresponding images;

[0010] S5. Use the trained model for inference, input the reference image and the target image, and obtain the recolored reference image.

[0011] Preferably, step S1 specifically includes the following steps:

[0012] S11. Use the CLAM algorithm to segment the WSI image and remove the background image;

[0013] S12: Crop the remaining image into square image patches of equal size and without overlap.

[0014] Preferably, acquiring image blocks corresponding to different viewing angles of the image block includes:

[0015] Original image block, any available image block in the dataset can be used as the original image block I s ;

[0016] Image blocks I of the same WSI at the same position using different scanners p ; Form sample pairs with the original image blocks to explore the color changes that occur during scanning with different scanners;

[0017] Image patch I obtained by data augmentation in the hematoxylin and eosin color space a1 ,I a2 ,…;

[0018] Image patches I from different locations of the same WSI o1 ,I o2 ,…; The color difference between it and the original image block is minimal. By making the color palettes of the two as consistent as possible, the color vector of the original image can be corrected.

[0019] Preferably, the image block I obtained by performing data enhancement in the hematoxylin and eosin color space a1 ,I a2,…is to form data-enhanced sample pairs with the original image blocks. Through data enhancement, data-enhanced sample pairs are formed with the original image blocks to explore the color changes that may be caused by the dyeing process in simulated real situations. The specific steps are:

[0020] Convert an RGB image to optical density space:

[0021]

[0022] Where I is the pixel value and I0 is the reference light source intensity;

[0023] NMF-based staining separation separates the optical density data into hematoxylin and eosin components by estimating the staining matrix;

[0024] Perform perturbations on the two components separately to obtain new components;

[0025] The new components are combined and inversely transformed using the inverse matrix of the staining matrix to obtain the enhanced RGB image.

[0026] Preferably, the two-way recoloring network comprises a color path and a structure path.

[0027] Preferably, the color path is to obtain two palette matrices of shape k×k respectively through two feature extraction networks Efficient-Net-b0 according to the image I, and normalize the palette matrix c n Transform the original image into a normalized image and recolor the color matrix c r Recolor the normalized image;

[0028] The structure path is to use the normalized palette matrix c n , transform the color of image I to obtain the normalized image I n , use the original image to recolor the color palette matrix c for the normalized image r , obtain the reconstructed image, or use the reference image to recolor the color wheel matrix c for the normalized image r , and obtain the recolored image.

[0029] Preferably, the two-way recoloring network is trained by self-supervised learning loss function, specifically:

[0030] Normalized image loss loss1: Input the images from different scanners, the images obtained by data augmentation, and the current image into the two-way recoloring network to obtain the corresponding normalized image. Through the L2 loss function, the color palette matrix of the same source is made consistent with the color palette matrix of the original image.

[0031] Color loss loss2: The image from the same source and the original image are input into a two-way recoloring network to obtain the corresponding normalized palette matrix and recolored palette matrix. The L2 loss function is used to make the palette matrix of the same source and the palette matrix of the original image consistent.

[0032] Reconstruction loss loss3: regardless of other images, each image should get a reconstructed image that is the same as the original image after passing through the two-way recoloring network, and the L1 loss function is used to increase the similarity between the two;

[0033] Recolorization loss loss4, for images from different scanners and images obtained by data augmentation, the obtained normalized images are recolored by applying the recolored palette matrix of the original images to obtain the recolored images, and the L1 loss function is used to narrow the distance between the original image and the recolored image;

[0034] The normalized image loss loss1, color loss loss2, reconstruction loss loss3, and recoloring loss loss4 are combined into a total loss function.

[0035] Preferably, the total loss function is the weighted sum of the normalized image loss loss1, color loss loss2, reconstruction loss loss3, and recoloring loss loss4:

[0036] loss total =λ1loss1+λ2loss2+λ3loss3+λ4loss4

[0037] Among them, λ1, λ2, λ3, and λ4 are the weight hyperparameters of the corresponding loss function.

[0038] Preferably, the trained model is used for inference, specifically, the reference image I r With the original image I s Input the trained model and get I s Normalized image N s , and the palette matrix c rr , and recoloring is performed to obtain the final recolored image.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. This paper designs a two-way recolorization network. Compared with existing methods, this two-way recolorization network performs consistent color transformation on pixels, avoiding the common visual artifacts in GAN networks. This ensures the authenticity of the generated images and the reliability of diagnosis. In particular, it can better preserve histological structural information in high-resolution WSI images.

[0041] 2. The computational process used in the present invention significantly reduces the demand for hardware resources, especially when processing large-scale WSI data. It can complete the recoloring operation with lower memory usage, while significantly improving the color conversion speed, meeting the efficient processing needs in actual clinical and research.

[0042] 3. This paper enhances the color consistency of WSI images through a self-supervised learning framework and HE space color transformation technology, significantly reduces the color shift problem caused by dyeing differences or different scanning devices, and improves the generalization performance of the model in multiple institutions and multiple scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is an overall flow chart of the WSI recoloring method based on self-supervised learning provided by the present invention;

[0044] Figure 2 Schematic diagram of the data processing method of the present invention;

[0045] Figure 3 Schematic diagram of a two-way recoloring network of the present invention;

[0046] Figure 4 It is the self-supervised learning framework of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] To address the problems of existing technologies, this paper proposes a WSI recoloring method based on self-supervised learning. By designing a two-way recoloring network, the same color transformation is performed on each pixel, avoiding artifacts while achieving faster recoloring with lower memory usage. Furthermore, by fully leveraging the properties of WSI images, a recolored image that is most similar to the original image is obtained through HE space color transformation and strengthening color consistency from the same source.

[0049] The present invention proposes a WSI recoloring method based on self-supervised learning, such as Figure 1 The specific steps are as follows:

[0050] Step S1: Preprocess the WSI image to obtain image blocks of the same size; use the segmentation algorithm in clam to obtain useful images, remove the background image, and divide the image into 256×256 image blocks to facilitate neural network processing.

[0051] Step S2: Obtain image blocks corresponding to different viewpoints of the image block, including image blocks obtained by using different scanners for the same WSI and the same position, image blocks obtained by data enhancement in the hematoxylin and eosin color space, and image blocks from different positions of the same WSI.

[0052] like Figure 2 As shown, first, any available image block in the dataset can be used as the original image block I s ;

[0053] Reference Figure 2 Upper left corner a, image block I of the same WSI at the same position using different scanners p , which forms a sample pair with the original image block. Their structural information is the same. It aims to explore the color changes that occur during the scanning process of different scanners. The MITOS-ATYPIA14 dataset is used in the specific implementation, which contains three images with magnifications of 10X, 20X and 40X scanned by two different scanners: Aperio ScanscopeXT and Hamamatsu Nanozoomer 2.0-HT.

[0054] Reference Figure 2 c, Image patch I obtained by data augmentation in the hematoxylin and eosin color space a1 ,I a2 ,…, through data augmentation, a data augmented sample pair is formed with the original image block. Similarly, their structural information is the same, which is used to simulate the coloring difference in real situations.

[0055] Convert the image in RGB space to HE space. The specific steps are:

[0056] Convert RGB image to OD space (Optical Density space):

[0057]

[0058] Where I is the pixel value and I0 is the reference light source intensity.

[0059] Stain separation based on NMF (non-negative matrix factorization) separates the optical density data into hematoxylin and eosin components by estimating the staining matrix;

[0060] Perform perturbations on the two components separately to obtain new components;

[0061] The specific implementation is to set the perturbation factor to 0.1:

[0062] δ=factor·(max-min)·(2·random()-1)

[0063] Among them, max and min are the maximum and minimum values ​​in the channel, random generates a value between 0 and 1, and the obtained perturbation δ is added to the original component to obtain a new component;

[0064] I a =he_to_rgb(H,E)

[0065] In the specific implementation, I a The number of is set to 2;

[0066] Image patches I from different locations of the same WSI o1 ,I o2 ,…, the color difference between it and the original image block is minimal. By making the color palettes of the two as consistent as possible, the color vector of the original image can be corrected.

[0067] In the specific implementation, I o The number of is set to 2.

[0068] Step S3: Input all the obtained image blocks into the designed two-way recoloring network to obtain a normalized image full of structural information and a normalized palette matrix containing color information, and reconstruct the palette matrix.

[0069] like Figure 3 As shown, image I passes through two feature extraction networks Efficient-Net-b0 to obtain two palette matrices of shape k×k, and normalize the palette matrix c n It is used to transform the original image into a normalized image and recolor the color palette matrix c r It is used to recolor the normalized image. In the specific implementation, k is set to 16.

[0070] In order to obtain a color transformation matrix of appropriate dimension (3×3), two parameter matrices, W1 with dimension 3×16 and W2 with dimension 16×3, are used in the image normalization and recoloring processes, respectively.

[0071] S4. Using a self-supervised learning loss function, the two-way recoloring network is trained to make images from different scanners and data-augmented images consistent with the normalized image of the current image, the palette matrices from the same source consistent, and images restored with other palettes consistent with the corresponding images;

[0072] like Figure 4 As shown, in the specific implementation:

[0073] Normalized image loss loss1, input the images from different scanners and the images obtained by data enhancement and the current image into the two-way recoloring network to obtain the corresponding normalized images. Through the L2 loss function, the normalized images of the original image and the normalized images of the images from different scanners and the images obtained by data enhancement are made consistent:

[0074]

[0075] Among them, N s is the normalized image of the original image, N ai It is the normalized image obtained by data augmentation image. There are two of them, N p are normalized images of images from different scanners paired with the original images.

[0076] Color loss loss2, input the image from the same source and the original image into the two-way recoloring network, obtain the corresponding normalized palette matrix and recolored palette matrix, and use the L2 loss function to make the palette matrix of the same source and the palette matrix of the original image tend to be consistent:

[0077]

[0078] Among them, c rs and c ns Recolor the original image palette and normalize the palette, c ri and c ni For step S3 o Input the obtained palette matrix;

[0079] Reconstruction loss loss3, without considering other images, each image should get a reconstructed image that is the same as the original image after passing through the two-way recoloring network, and the L1 loss function is used to increase the similarity between the two:

[0080] loss3=||II′||1

[0081] Where I′ is the reconstructed image.

[0082] Recolorization loss loss4, for images from different scanners or images obtained by data augmentation, obtain normalized images, apply the recolored palette matrix of the original image to recolor them, obtain recolored images, and use the L1 loss function to narrow the distance between the original image and the recolored image:

[0083]

[0084] Among them, R is the color matrix of the original image re-colored by the color wheel matrix of N p , N aThe image obtained by recoloring;

[0085] Multiple loss functions are combined into a total loss function, which is a weighted sum of different loss functions:

[0086] loss total =λ1loss1+λ2loss2+λ3loss3+λ4loss4

[0087] Specifically, set λ1, λ2, λ3, and λ4 to 1, 1, 1, and 2;

[0088] In the specific implementation, the Adam optimizer is used for optimization, and the learning rate lr is set to 0.01.

[0089] Step S5: Use the trained model for inference, input the reference image and the target image, and obtain a recolored reference image.

[0090] Specifically, the reference image I r With the original image I s Input the trained model and get I s Normalized image N s , and the palette matrix c rr , and recoloring is performed to obtain the final recolored image.

[0091] In the specific implementation, 300 images from Aperio Scanscope XT were used as tests, one Hamamatsu Nanozoomer 2.0-HT scan image was used as a reference, and 300 corresponding images of Hamamatsu Nanozoomer 2.0-HT were used as labels. The PSNR indicator was used for evaluation, and results that exceeded existing methods were obtained.

[0092] Although the present invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A WSI recoloring method based on self-supervised learning, characterized in that: The following steps are involved: S1, preprocess the WSI image to obtain image blocks of the same size; S2, obtaining image blocks corresponding to different perspectives of the image block; S3. Input all the obtained image blocks into a two-way recoloring network to obtain a normalized image containing structural information and a normalized palette matrix containing color information, and reconstruct the palette matrix; S4. Using a self-supervised learning loss function, the two-way recoloring network is trained to make images from different scanners and data-augmented images consistent with the normalized image of the current image, the palette matrices from the same source consistent, and images restored with other palettes consistent with the corresponding images; S5. Use the trained model for inference, input the reference image and the target image, and obtain the recolored reference image.

2. A WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11. Use the CLAM algorithm to segment the WSI image and remove the background image; S12: Crop the remaining image into square image patches of equal size and without overlap.

3. A WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The obtaining of image blocks corresponding to different viewing angles of the image block includes: Original image block, any available image block in the dataset can be used as the original image block I s ; Image blocks I of the same WSI at the same position using different scanners p ; Image patch I obtained by data augmentation in the hematoxylin and eosin color space a1 ,I a2 ,…; Image patches I from different locations of the same WSI o1 ,I o2 ,….

4. A WSI recoloring method based on self-supervised learning according to claim 3, characterized in that: The image block I obtained by performing data enhancement in the hematoxylin and eosin color space a1 ,I a2 ,… is to form a data enhancement sample pair with the original image block. The specific steps are: Convert an RGB image to optical density space: Where I is the pixel value and I0 is the reference light source intensity; NMF-based staining separation separates the optical density data into hematoxylin and eosin components by estimating the staining matrix; Perform perturbations on the two components separately to obtain new components; The new components are combined and inversely transformed using the inverse matrix of the staining matrix to obtain the enhanced RGB image.

5. The WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The two-way recoloring network includes a color path and a structure path.

6. A WSI recoloring method based on self-supervised learning according to claim 5, characterized in that: The color path is based on the image I through two feature extraction networks Efficient-Net-b0, respectively obtaining two palette matrices of shape k×k, and normalizing the palette matrix c n Transform the original image into a normalized image and recolor the color matrix c r Recolor the normalized image; The structure path is to use the normalized palette matrix c n , transform the color of image I to obtain the normalized image I n , use the original image to recolor the color palette matrix c for the normalized image r , obtain the reconstructed image, or use the reference image to recolor the color wheel matrix c for the normalized image r , and obtain the recolored image.

7. The WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The two-way recoloring network is trained through self-supervised learning loss function, specifically: Normalized image loss loss1: Input the images from different scanners, the images obtained by data augmentation, and the current image into the two-way recoloring network to obtain the corresponding normalized image. Through the L2 loss function, the color palette matrix of the same source is made consistent with the color palette matrix of the original image. Color loss loss2: The image from the same source and the original image are input into a two-way recoloring network to obtain the corresponding normalized palette matrix and recolored palette matrix. The L2 loss function is used to make the palette matrix of the same source and the palette matrix of the original image consistent. Reconstruction loss loss3: regardless of other images, each image should get a reconstructed image that is the same as the original image after passing through the two-way recoloring network, and the L1 loss function is used to increase the similarity between the two; Recolorization loss loss4, for images from different scanners and images obtained by data augmentation, the obtained normalized images are recolored by applying the recolored palette matrix of the original images to obtain the recolored images, and the L1 loss function is used to narrow the distance between the original image and the recolored image; The normalized image loss loss1, color loss loss2, reconstruction loss loss3, and recoloring loss loss4 are combined into a total loss function.

8. A WSI recoloring method based on self-supervised learning according to claim 7, characterized in that: The total loss function is the weighted sum of the normalized image loss loss1, color loss loss2, reconstruction loss loss3, and recoloring loss loss4: loss total =λ1loss1+λ2loss2+λ3loss3+λ4loss4 Among them, λ1, λ2, λ3, and λ4 are the weight hyperparameters of the corresponding loss function.

9. The WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The trained model is used for inference, specifically, the reference image I r With the original image I s Input the trained model and get I s Normalized image N s , and the palette matrix c rr , and recoloring is performed to obtain the final recolored image.

Images