WSI re-coloring method based on self-supervised learning

Abstract

The invention discloses a WSI re-coloring method based on self-supervised learning. The method comprises the following steps: firstly, preprocessing a WSI picture to obtain image blocks with the same size; respectively acquiring image blocks corresponding to different visual angles of the image blocks, and inputting all the obtained image blocks into a designed double-path recoloring network; finally, through self-supervised learning loss function training, images from different scanners and images obtained through data enhancement tend to be consistent with a normalized image of a current image, palette matrixes of the same source tend to be consistent, and images repaired through other palettes tend to be consistent with corresponding images; in the reasoning stage, the reconstructed palette matrix of the reference image block and the normalized image of the target image are obtained, and the target image obtained after the reference image is re-colored is obtained.According to the method, the consistency of the re-colored image can be improved under the condition of lower time and space occupation, the clear and high-quality re-colored image is generated, and the image quality is improved. And help is provided for subsequent processing and judgment of doctors and models.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and in particular to a WSI recoloring method based on self-supervised learning. Background Art

[0002] Computational pathology has become increasingly popular with the invention of high-precision microscope slide scanners and the development of deep neural networks. It aims to objectively and quantitatively evaluate the disease microenvironment at the microscopic level and ultimately achieve precision medicine, including disease diagnosis, treatment response, etc. Although CNN models have performed well in various computational pathology applications, most models are still plagued by color inconsistency issues. This is caused by inconsistent tissue thickness during slicing, different dye manufacturers, different slicing protocols, etc. In addition, scanning with different scanners may also lead to color inconsistency issues, which affects the generalization performance of learning-based models, especially when applying them to data from different institutions. This is why color normalization is always an essential preprocessing step before model training.

[0003] In recent years, a series of methods for color normalization have emerged, such as Vahadane, Reinhard, and STST. However, most of these methods lack flexibility in practice. At the same time, these methods require the input of reference images and original images at the same time, and cannot reuse the color information of the reference image, making them inefficient. In order to solve the above problems, a model called RestainNet defines color normalization as a digital re-dyeing process, learning how to re-dye unstained (grayscale) images. This method separates the de-staining and re-dyeing processes, but it uses a GAN network for re-coloring, which has problems with visual artifacts, large memory requirements, and slow color conversion. Visual artifacts will affect the judgment of doctors and models on the data, and WSI images are usually large in scale. Large memory requirements and slow color conversion will hinder the application of the model. Summary of the invention

[0004] The purpose of the present invention is to overcome the problems existing in the above-mentioned prior art and to provide a WSI recolorization method based on self-supervised learning, which can complete the recolorization of WSI images with lower time and space occupancy, generate clear and high-quality recolored images, and provide assistance for subsequent processing and judgment of doctors and models.

[0005] In order to achieve the above object, the present invention provides a WSI recoloring method based on self-supervised learning, comprising the following steps:

[0006] S1, preprocess the WSI image to obtain image blocks of the same size;

[0007] S2, obtaining image blocks corresponding to different perspectives of the image block;

[0008] S3, inputting all the obtained image blocks into a two-way recoloring network to obtain a normalized image containing structural information and a normalized palette matrix containing color information, and reconstructing the palette matrix;

[0009] S4. Through the self-supervised learning loss function, the two-way recoloring network is trained so that the images from different scanners and the images obtained by data enhancement are consistent with the normalized images of the current image, the palette matrices from the same source are consistent, and the images repaired by other palettes are consistent with the corresponding images;

[0010] S5. Use the trained model for reasoning, input the reference image and the target image, and obtain the recolored reference image.

[0011] Preferably, step S1 specifically includes the following steps:

[0012] S11, segment the WSI image using the CLAM algorithm to remove the background image;

[0013] S12: Crop the remaining image into square image patches of equal size and without overlap.

[0014] Preferably, the acquiring of image blocks corresponding to different viewing angles of the image blocks comprises:

[0015] Original image block, any available image block in the dataset can be used as the original image block I s ;

[0016] Image patches I of the same WSI at the same location using different scanners p ; The sample pairs are formed with the original image blocks to explore the color changes that occur during scanning with different scanners;

[0017] Image patch I obtained by data augmentation in the hematoxylin and eosin color space a1 ,I a2 ,…;

[0018] Image patches I from different locations of the same WSI o1 ,I o2 ,…; the color difference between it and the original image block is minimal. By making the color palettes of the two as consistent as possible, the color vector of the original image can be corrected.

[0019] Preferably, the image block I obtained by performing data enhancement in the hematoxylin and eosin color space a1 ,I a2,…is to form a data enhancement sample pair with the original image block. Through data enhancement, the data enhancement sample pair is formed with the original image block to explore the color changes that may be caused by the dyeing process in simulated real situations. The specific steps are:

[0020] Convert an RGB image to optical density space:

[0021]

[0022] Among them, I is the pixel value, I0 is the reference light source intensity;

[0023] NMF-based staining separation separates the optical density data into hematoxylin and eosin components by estimating the staining matrix;

[0024] Perform perturbations on the two components respectively to obtain new components;

[0025] The new components are combined and inversely transformed using the inverse matrix of the dye matrix to obtain the enhanced RGB image.

[0026] Preferably, the two-way recoloring network comprises a color path and a structure path.

[0027] Preferably, the color path is to obtain two palette matrices of shape k×k respectively through two feature extraction networks Efficient-Net-b0 according to the image I, and normalize the palette matrix c n Transform the original image into a normalized image and recolor the color palette matrix c r Recolor the normalized image;

[0028] The structural path is to use the normalized palette matrix c n , transform the color of image I to obtain the normalized image I n , use the original image to recolor the color palette matrix c for the normalized image r , obtain the reconstructed image, or use the reference image to recolor the color palette matrix c for the normalized image r , and obtain the recolored image.

[0029] Preferably, the two-way recoloring network is trained by self-supervised learning loss function, specifically:

[0030] Normalized image loss loss1, input the images from different scanners and the images obtained by data enhancement and the current image into the two-way recoloring network to obtain the corresponding normalized image. Through the L2 loss function, the palette matrix of the same source is made consistent with the palette matrix of the original image;

[0031] Color loss loss2, input the image from the same source and the original image into the two-way recoloring network to obtain the corresponding normalized palette matrix and recolored palette matrix. Through the L2 loss function, the palette matrix of the same source is made consistent with the palette matrix of the original image;

[0032] Reconstruction loss loss3, regardless of other images, each image should get the same reconstructed image as the original image after passing through the two-way recoloring network, and the L1 loss function is used to increase the similarity between the two;

[0033] Recolorization loss loss4, for images from different scanners and images obtained by data augmentation, obtain normalized images, apply the recolored palette matrix of the original image to recolor them, obtain recolored images, and use the L1 loss function to narrow the distance between the original image and the recolored image;

[0034] The normalized image loss loss1, color loss loss2, reconstruction loss loss3, and recoloring loss loss4 are combined into a total loss function.

[0035] Preferably, the total loss function is a weighted sum of the normalized image loss loss1, the color loss loss2, the reconstruction loss loss3, and the recoloring loss loss4:

[0036] loss total =λ1loss1+λ2loss2+λ3loss3+λ4loss4

[0037] Among them, λ1, λ2, λ3, λ4 are the weight hyperparameters of the corresponding loss function.

[0038] Preferably, the trained model is used for reasoning, specifically, the reference image I r With the original image I s Input into the trained model and get I s The normalized image N s , and the palette matrix c rr , and recoloring is performed to obtain the final recolored image.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention designs a two-way recolorization network. Compared with the existing methods, the two-way recolorization network performs consistent color transformation on pixels, avoiding the common visual artifact problem in the GAN network, thereby ensuring the authenticity of the generated images and the reliability of the diagnosis, especially in high-resolution WSI images, which can better preserve the histological structure information.

[0041] 2. The computational process used in the present invention significantly reduces the demand for hardware resources, especially when processing large-scale WSI data, and can complete the recoloring operation with lower memory usage, while significantly improving the color conversion speed, meeting the efficient processing requirements in actual clinical and research.

[0042] 3. The present invention enhances the color consistency of WSI images through a self-supervised learning framework and HE space color transformation technology, significantly reduces the color shift problem caused by staining differences or different scanning devices, and improves the generalization performance of the model in multiple institutions and multiple scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The overall flow chart of a WSI recoloring method based on self-supervised learning provided by the present invention;

[0044] Figure 2 It is a schematic diagram of the data processing method of the present invention;

[0045] Figure 3 It is a schematic diagram of a two-way recoloring network of the present invention;

[0046] Figure 4 It is the self-supervised learning framework of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Aiming at the problems existing in the prior art, the present invention proposes a WSI recoloring method based on self-supervised learning, which designs a two-way recoloring network to perform the same color transformation on each pixel, thus avoiding artifacts and achieving faster recoloring with lower memory usage; at the same time, the properties of WSI images are fully utilized, and the recolored image that is most similar to the original image is obtained through HE space color transformation, strengthening the color consistency of the same source and other measures.

[0049] The present invention proposes a WSI recoloring method based on self-supervised learning, such as Figure 1 As shown, the specific steps are as follows:

[0050] Step S1, preprocess the WSI image to obtain image blocks of the same size; use the segmentation algorithm in clam to obtain useful images, remove the background image, and divide the image into 256×256 image blocks to facilitate neural network processing.

[0051] Step S2, obtaining image blocks corresponding to different perspectives of the image block, including image blocks obtained by using different scanners for the same WSI and the same position, image blocks obtained by performing data enhancement in the hematoxylin and eosin color space, and image blocks from different positions of the same WSI.

[0052] like Figure 2 As shown, first, any available image block in the dataset can be used as the original image block I s ;

[0053] Reference Figure 2 Top left corner a, image block I at the same position of the same WSI using different scanners p , and the original image blocks form sample pairs, their structural information is the same, aiming to explore the color changes that occur during scanning with different scanners. The MITOS-ATYPIA14 dataset is used in the specific implementation, which contains three images with magnifications of 10X, 20X and 40X scanned by two different scanners, Aperio ScanscopeXT and Hamamatsu Nanozoomer 2.0-HT.

[0054] Reference Figure 2 c, Image block I obtained by data enhancement in the hematoxylin and eosin color space a1 ,I a2 ,…, through data enhancement, a data enhanced sample pair is formed with the original image block. Similarly, their structural information is the same, which is used to simulate the coloring difference in real situations.

[0055] The image in RGB space is converted to HE space. The specific steps are:

[0056] Convert RGB image to OD space (Optical Density):

[0057]

[0058] Among them, I is the pixel value and I0 is the reference light source intensity.

[0059] Staining separation based on NMF (non-negative matrix factorization) separates the optical density data into hematoxylin and eosin components by estimating the staining matrix;

[0060] Perform perturbations on the two components respectively to obtain new components;

[0061] The specific implementation is to set the perturbation factor to 0.1:

[0062] δ=factor·(max-min)·(2·random()-1)

[0063] Among them, max and min are the maximum and minimum values ​​in the channel, random generates a value between 0 and 1, and the obtained perturbation δ is added to the original component to obtain a new component;

[0064] I a =he_to_rgb(H,E)

[0065] In the specific implementation, I a The number of is set to 2;

[0066] Image patches I from different locations of the same WSI o1 ,I o2 ,…, the color difference between it and the original image block is minimal. By making the color palettes of the two as consistent as possible, the color vector of the original image can be corrected.

[0067] In the specific implementation, I o The number of is set to 2.

[0068] Step S3: input all the obtained image blocks into the designed two-way recoloring network to obtain a normalized image full of structural information and a normalized palette matrix containing color information, and reconstruct the palette matrix.

[0069] like Figure 3 As shown, image I passes through two feature extraction networks Efficient-Net-b0 to obtain two palette matrices of shape k×k, and normalize the palette matrix c n It is used to transform the original image into a normalized image, recoloring the color palette matrix c r It is used to recolor the normalized image. In the specific implementation, k is set to 16.

[0070] In order to obtain a color transformation matrix of appropriate dimension (3×3), two parameter matrices, W1 with dimension 3×16 and W2 with dimension 16×3, are used in the image normalization and recoloring processes, respectively.

[0071] S4. Through the self-supervised learning loss function, the two-way recoloring network is trained so that the images from different scanners and the images obtained by data enhancement are consistent with the normalized images of the current image, the palette matrices from the same source are consistent, and the images repaired by other palettes are consistent with the corresponding images;

[0072] like Figure 4 As shown, in the specific implementation:

[0073] Normalized image loss loss1, input the images from different scanners and the images obtained by data enhancement and the current image into the two-way recoloring network to obtain the corresponding normalized image. Through the L2 loss function, the normalized image of the original image is made consistent with the normalized image of the images from different scanners and the images obtained by data enhancement:

[0074]

[0075] Among them, N s is the normalized image of the original image, N ai is the normalized image obtained by the data augmentation image. There are two of them, N p are normalized images of images from different scanners paired with the original images.

[0076] Color loss loss2, input the image from the same source and the original image into the two-way recoloring network, obtain the corresponding normalized palette matrix and recolored palette matrix, and through the L2 loss function, make the palette matrix of the same source and the palette matrix of the original image tend to be consistent:

[0077]

[0078] Among them, c rs and c ns Recolor the original image with a palette and a normalized palette, c ri and c ni The step S3 is described in o Input the obtained palette matrix;

[0079] Reconstruction loss loss3, regardless of other images, each image should get the same reconstructed image as the original image after passing through the two-way recoloring network, and the L1 loss function is used to increase the similarity between the two:

[0080] loss3=||II′||1

[0081] Where I′ is the reconstructed image.

[0082] Recoloring loss loss4, for images from different scanners or images obtained by data augmentation, obtain normalized images, apply the recolored palette matrix of the original image to recolor them, obtain recolored images, and use the L1 loss function to narrow the distance between the original image and the recolored image:

[0083]

[0084] Among them, R is the color palette matrix of the original image recolored by N p , N aThe image obtained by recoloring;

[0085] Multiple loss functions are combined into a total loss function, which is a weighted sum of different loss functions:

[0086] loss total =λ1loss1+λ2loss2+λ3loss3+λ4loss4

[0087] Specifically, set λ1,λ2,λ3,λ4 to 1,1,1,2;

[0088] In the specific implementation, Adam optimizer is used for optimization, and the learning rate lr is set to 0.01.

[0089] Step S5: Use the trained model for reasoning, input a reference image and a target image, and obtain a recolored reference image.

[0090] Specifically, the reference image I r With the original image I s Input into the trained model and get I s The normalized image N s , and the palette matrix c rr , and recoloring is performed to obtain the final recolored image.

[0091] In the specific implementation, 300 images from Aperio Scanscope XT are used as tests, one HamamatsuNanozoomer 2.0-HT scan image is used as a reference, and 300 corresponding images of Hamamatsu Nanozoomer 2.0-HT are used as labels. The PSNR indicator is used for evaluation, and results that exceed existing methods can be obtained.

[0092] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A WSI recoloring method based on self-supervised learning, characterized in that: The following steps are involved: S1, preprocess the WSI image to obtain image blocks of the same size; S2, obtaining image blocks corresponding to different perspectives of the image block; S3, inputting all the obtained image blocks into a two-way recoloring network to obtain a normalized image containing structural information and a normalized palette matrix containing color information, and reconstructing the palette matrix; S4. Through the self-supervised learning loss function, the two-way recoloring network is trained so that the images from different scanners and the images obtained by data enhancement are consistent with the normalized images of the current image, the palette matrices from the same source are consistent, and the images repaired by other palettes are consistent with the corresponding images; S5. Use the trained model for reasoning, input the reference image and the target image, and obtain the recolored reference image.

2. A WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11, segment the WSI image using the CLAM algorithm to remove the background image; S12: Crop the remaining image into square image patches of equal size and without overlap.

3. A WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The obtaining of image blocks corresponding to different viewing angles of the image block comprises: Original image block, any available image block in the dataset can be used as the original image block I s ; Image patches I of the same WSI at the same location using different scanners p ; Image patch I obtained by data augmentation in the hematoxylin and eosin color space a1 ,I a2 ,…; Image patches I from different locations of the same WSI o1 ,I o2 ,….

4. A WSI recoloring method based on self-supervised learning according to claim 3, characterized in that: The image block I obtained by performing data enhancement in the hematoxylin and eosin color space a1 ,I a2 ,…is to form a data enhancement sample pair with the original image block. The specific steps are: Convert an RGB image to optical density space: Among them, I is the pixel value, I0 is the reference light source intensity; NMF-based staining separation separates the optical density data into hematoxylin and eosin components by estimating the staining matrix; Perform perturbations on the two components respectively to obtain new components; The new components are combined and inversely transformed using the inverse matrix of the dye matrix to obtain the enhanced RGB image.

5. A WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The two-way recoloring network includes a color path and a structure path.

6. A WSI recoloring method based on self-supervised learning according to claim 5, characterized in that: The color path is based on the image I through two feature extraction networks Efficient-Net-b0, respectively obtaining two palette matrices of shape k×k, and normalizing the palette matrix c n Transform the original image into a normalized image and recolor the color palette matrix c r Recolor the normalized image; The structural path is to use the normalized palette matrix c n , transform the color of image I to obtain the normalized image I n , use the original image to recolor the color palette matrix c for the normalized image r , obtain the reconstructed image, or use the reference image to recolor the color palette matrix c for the normalized image r , and obtain the recolored image.

7. A WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The two-way recoloring network is trained by self-supervised learning loss function, specifically: Normalized image loss loss1, input the images from different scanners and the images obtained by data enhancement and the current image into the two-way recoloring network to obtain the corresponding normalized image. Through the L2 loss function, the palette matrix of the same source is made consistent with the palette matrix of the original image; Color loss loss2, input the image from the same source and the original image into the two-way recoloring network to obtain the corresponding normalized palette matrix and recolored palette matrix. Through the L2 loss function, the palette matrix of the same source is made consistent with the palette matrix of the original image; Reconstruction loss loss3, regardless of other images, each image should get the same reconstructed image as the original image after passing through the two-way recoloring network, and the L1 loss function is used to increase the similarity between the two; Recolorization loss loss4, for images from different scanners and images obtained by data augmentation, obtain normalized images, apply the recolored palette matrix of the original image to recolor them, obtain recolored images, and use the L1 loss function to narrow the distance between the original image and the recolored image; The normalized image loss loss1, color loss loss2, reconstruction loss loss3, and recoloring loss loss4 are combined into a total loss function.

8. A WSI recoloring method based on self-supervised learning according to claim 7, characterized in that: The total loss function is the weighted sum of the normalized image loss loss1, color loss loss2, reconstruction loss loss3, and recoloring loss loss4: loss total =λ1loss1+λ2loss2+λ3loss3+λ4loss4 Among them, λ1, λ2, λ3, λ4 are the weight hyperparameters of the corresponding loss function.

9. A WSI recoloring method based on self-supervised learning according to claim 1, characterized in that: The trained model is used for inference, specifically, the reference image I r With the original image I s Input into the trained model and get I s The normalized image N s , and the palette matrix c rr , and recoloring is performed to obtain the final recolored image.

Images