Method and system for color normalization of digital pathology images

Abstract

The application discloses a color standardization method and system for digital pathological images, and the method comprises the following steps: obtaining a colored digital pathological image from a target domain; correcting the digital color difference of the digital pathological image to serve as a reference image; performing color amplification on the reference image to simulate different staining differences, and obtaining a plurality of color amplified images; establishing a similarity map of the reference image and the corresponding plurality of color amplified images; inputting the similarity map into a de-staining model to eliminate the color information caused by staining, and outputting a plurality of gray feature maps corresponding to the plurality of color amplified images and retaining the information related to human tissues, cell structures and morphological distribution; inputting the gray feature maps corresponding to the color amplified images into a virtual staining network model to virtually color the gray feature maps of the color amplified images, and outputting a color image which is color-standardized and has similarity in color between the coloring color and the color of the reference image. The application is a color standardization method without using source domain images.

Description

TECHNICAL FIELD

[0001] The present application relates to the processing of medical images, and in particular to a method and system for color standardization of digital pathology images. BACKGROUND

[0002] Digital pathology refers to the use of a pathology scanner to digitize a pathology slide to obtain a color digital pathology whole slide image for pathological diagnosis. However, there are large differences in the color of pathology images. There are two main reasons. First, the staining difference of the pathology slide. For the same preparation method, different medical institutions use different brands, ingredients, concentrations of reagents and preparation processes, resulting in differences in the staining of pathology slides. Unstable preparation quality control further increases the staining difference. Second, the digitization stage. Different pathology scanners from different manufacturers have different color gamuts, resulting in differences in color restoration of the same pathology slide, causing digital color differences in the images. Therefore, the colors of digital pathology images from different medical institutions and different scanners always have obvious differences.

[0003] Color differences in digital pathology images not only cause deviations in pathological diagnosis by pathologists, but also significantly reduce the diagnostic accuracy of pathological artificial intelligence models. In order to avoid the influence of color differences on diagnosis, current color standardization techniques take a specific source image set (such as digital images of pathology slides prepared by a specific medical institution) as the target domain, assuming it to be a standard color image. Other sources of image sets are considered as the source domain, such as digital pathology images of pathology slides prepared by another medical institution. Color standardization techniques correct the color of source domain images to the color of target domain images, making them similar.

[0004] The first type of color standardization technique is to calculate the statistical properties of the color distribution of the source domain, and match the color distribution of the target domain so that the statistical quantities of the colors of the two are similar.

[0005] The second type of technique is based on a neural network model, which considers the color matching of source domain images to target domain images as a style transfer problem. The input source domain image is used to output an image similar in color to the target domain image using a style transfer model.

[0006] The third type of technique converts the color image of the target domain to a grayscale image. Then a neural network model, such as Pix2Pix, is trained to convert the grayscale image back to a color image. For source domain images, first grayscale, then recolor using the recoloring model established by the target domain, so that they obtain the same color distribution as the target domain images.

[0007] However, these techniques have obvious technical bottlenecks. The first two types of techniques need to establish a model for converting any one source domain image to the target domain. The main bottlenecks are: the need to obtain representative source domain images as a dataset. In actual work, it is generally impossible to obtain a sufficient number of source domain images. Even if source domain images can be obtained, it is unrealistic to establish a standardization method between each source domain and the target domain image. The third type of technology uses grayscale images of source domain and target domain images as intermediates, but grayscale images themselves are affected by color differences. When the color difference between source domain images and target domain images is large, the color of the recolored color image of the source domain is significantly different from the color of the target domain image. Moreover, the above-mentioned techniques all assume that the target domain image is standardized, but the target domain image itself is affected by digital color differences and is not a true color image of a pathological section.

[0008] Therefore, the existing color standardization technology cannot solve the color difference problem of digital pathology images, and a more feasible standardization technology needs to be established. SUMMARY

[0009] The present application provides a color standardization method and system for digital pathology images to solve the technical problems that the existing color standardization technology cannot obtain a sufficient number of source domain images for each source domain, and that the color of the recolored color image of the source domain is significantly different from the color of the target domain image due to the influence of grayscale images on color differences.

[0010] To solve the above technical problems, the technical solution provided by the present application is:

[0011] A color standardization method for digital pathology images, comprising the following steps:

[0012] Obtaining a colored digital pathology image from the target domain; and correcting the digital color difference of the digital pathology image to serve as a reference image;

[0013] Color augmentation is performed on the reference image to simulate different staining differences, and a plurality of color augmented images corresponding to different staining differences for each reference image are obtained; a similarity map of the reference image and the corresponding plurality of color augmented images is established;

[0014] The similarity map is input into a de-staining model to eliminate the color information caused by staining, and a plurality of grayscale feature maps corresponding to the color augmented images that retain information related to human tissue, cell structure, and morphological distribution are output;

[0015] The grayscale feature maps corresponding to the color augmented images are input into a virtual staining network model, the grayscale feature maps of the color augmented images are virtually colored, and color standardized color images with similar colors to the reference image colors are output.

[0016] Preferably, the de-staining model is trained by the following steps:

[0017] A plurality of color pathological images from a target domain are obtained, and a reference image dataset is generated after correcting digital color differences of the plurality of digital pathological images;

[0018] Each reference image in the reference image dataset is color augmented to simulate different staining differences, obtaining a plurality of color augmented images corresponding to different staining differences for each reference image; a similarity graph is established with all reference images and corresponding plurality of color augmented images as graph nodes, and the connection between the reference images and the color augmented images as edges; the similarity weight of the edge connected between each color augmented image and the reference image in the similarity graph is labeled;

[0019] The similarity graph and the similarity weight of all edges are taken as input, and the gray feature map corresponding to the plurality of color augmented images that retain information related to human tissues, cell structures and morphological distribution is taken as output to train the de-staining model.

[0020] Preferably, in the training of the de-staining model, the output gray feature map is a three-dimensional image matrix in which the color information is eliminated, and two dimensions are gray two-dimensional images of the same size as the reference images, and the third dimension is a feature channel.

[0021] Preferably, the de-staining model is a de-staining model realized by two or more layers of down-sampling encoders and corresponding two or more layers of up-sampling decoders, the output of the de-staining model is also compared with the output of the gray conversion network for image graying realized by a convolution layer, and the outputs of the de-staining model and the gray conversion network are provided with different weights.

[0022] Preferably, in the training of the de-staining model, the first similarity, the second similarity and the first discriminator similarity are taken as the training target, and the gradient descent method of back propagation is used to optimize the weights of the de-staining model and the gray conversion network;

[0023] The first similarity is the similarity between the gray feature map corresponding to the reference image and the gray feature map corresponding to the color augmented image of the reference image; the second similarity is the similarity between the gray feature map corresponding to the reference image and the gray image converted by the gray conversion network; and the first discriminator similarity is the similarity between the gray feature map of the reference image or the color augmented image calculated by the first discriminator and the gray image obtained by the gray conversion network.

[0024] Preferably, the first similarity and the second similarity both include pixel similarity, texture similarity and perceptual feature similarity.

[0025] Preferably, the virtual staining network model is trained by the following steps:

[0026] The gray scale feature map corresponding to the color augmented image output by the de-staining model is taken as input, the gray scale feature map of the color augmented image is virtually colored, and a color image after color standardization having similarity with the coloring color and the reference image color is taken as output, thereby training the virtual staining network model.

[0027] Preferably, the virtual staining network model is a virtual staining network model realized by two or more layers of down-sampling encoders and corresponding two or more layers of up-sampling decoders.

[0028] Preferably, in the training of the virtual staining network model, the third similarity, the fourth similarity and the second discriminator similarity are taken as training targets, and the gradient descent method of back propagation is used to optimize the weight values of the virtual staining network model.

[0029] The third similarity is the similarity between the color image corresponding to the reference image and the color augmented image of the reference image; the fourth similarity is the similarity between the reference image and the color image corresponding to the reference image; and the second discriminator similarity is the similarity between the color image after virtual staining of the gray scale feature map of the reference image or the color augmented image calculated by the second discriminator and the reference image.

[0030] The third similarity and the fourth similarity both include pixel similarity, texture similarity and perceptual feature similarity.

[0031] The present application also provides a computer system comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0032] The present application has the following advantages:

[0033] The color standardization method and system of the digital pathological image of the present application establish a digital pathological color standardization method based on the reference image of the target domain, which is universal and does not need to use the color standardization method of the source domain image; solves the problem that the source domain image is not easy to obtain and that a standardization method needs to be established for each source domain. At the same time, it solves the problem that the color difference between the source domain image and the target domain image cannot be too large, and is more suitable for color standardization of source domain images with obvious color changes.

[0034] In addition to the purposes, features and advantages described above, the present application has other purposes, features and advantages. The present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, illustrate the preferred embodiments of the application and assist in

[0036] Figure 1 is a flowchart of the color normalization method of the digital pathology image of the preferred embodiment of the present application;

[0037] Figure 2 is a schematic diagram of the color conversion function between the digital pathology image and the reference image of the preferred embodiment of the present application;

[0038] Figure 3 is a schematic diagram of the color card when the digital pathology image and the reference image are converted of the preferred embodiment of the present application;

[0039] Figure 4 is a schematic diagram of the similarity map of the reference image and the corresponding multiple color augmented images of the preferred embodiment of the present application;

[0040] Figure 5 is a structural schematic diagram of the de-staining model of the preferred embodiment of the present application;

[0041] Figure 6 is a schematic diagram of the training process of the de-staining model of the preferred embodiment of the present application;

[0042] Figure 7 is a structural schematic diagram of the virtual staining model of the preferred embodiment of the present application;

[0043] Figure 8 is a schematic diagram of the training process of the virtual staining model of the preferred embodiment of the present application;

[0044] Figure 9 is a schematic diagram of the joint training process of the de-staining model and the virtual staining model of the preferred embodiment of the present application;

[0045] Figure 10 is a schematic diagram of the joint training process of the de-staining model and the virtual staining model and the artificial intelligence model of the preferred embodiment of the present application;

[0046] Figure 11 is a standardization process of the source domain image of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application will be described in detail below with reference to the accompanying drawings, but the present application can be implemented in various different ways as limited and covered by the claims.

[0048] Referring to Figure 1 The color normalization method of the digital pathology image of the present application comprises the following steps:

[0049] S1: obtaining a digital pathological image of a color from a target domain; and correcting a digitalization color difference of the digital pathological image to serve as a reference image;

[0050] color augmenting the reference image to simulate different staining differences to obtain a plurality of color augmented images corresponding to different staining differences for each reference image; and establishing a similarity map of the reference image and the corresponding plurality of color augmented images;

[0051] S2: inputting the similarity map into a de-staining model to eliminate color information caused by staining, and outputting a plurality of grayscale feature maps corresponding to the color augmented images which retain information related to human tissue, cell structure and morphological distribution;

[0052] S3: inputting the grayscale feature maps corresponding to the color augmented images into a virtual staining network model, virtually coloring the grayscale feature maps of the color augmented images, and outputting color images which have similar color to the reference image color after color standardization.

[0053] The above steps use the reference image to simulate color images of multiple source domains by color augmentation, and establish a similarity map; remove color information of the source domain image, and extract the de-staining model of the grayscale features of human tissue, cell structure and morphological distribution; and realize the construction scheme of the virtual staining model for color standardization. The training process of the three-step transfer learning of the two models. The method for establishing digital pathology color standardization based on the target domain reference image is independent of the source domain and is a universal color standardization method that does not require the use of source domain images; solves the problem that source domain images are not easy to obtain and that a standardization method needs to be established for each source domain. At the same time, it solves the problem that the color difference between the source domain image and the target domain image cannot be too large, and is more suitable for color standardization of source domain images with obvious color changes.

[0054] In some embodiments, the de-staining model is trained by the following steps:

[0055] S101: obtaining a plurality of color digital pathological images from a target domain, and obtaining a reference image dataset after correcting the digitalization color difference of the digital pathological image. In implementation, the digitalization color difference of the target domain image is usually corrected by color calibration. The following steps can be used:

[0056] S1011: determining the data source of the target domain, i.e. the color digital pathological images of the pathological sections obtained by a certain preparation and digitalization protocol given in advance. Assuming that the target domain image is the target image for diagnosis by a pathologist or a pathological artificial intelligence model, the set of color pathological images of the target domain is defined as:

[0057] S1012: refer to Figure 3, a pre-prepared color card is used, which is composed of several color blocks, and the RGB values of the color blocks are known. The RGB values of the color card image are obtained by scanning the color card with a target domain scanner; the true RGB values of the color card are measured by a colorimeter. An interpolation method is used to establish a conversion function between the two, as shown in Figure 2 In this embodiment, Figure 2 The color card includes three single colors of red, green and blue, each from light to dark, covering the color value range of 0-255.

[0058] S1013: Read the RGB values of the pixel i of the target domain image i ′ i ′ i ′, and convert them into the true color values of each pixel r i ′ i ′ i ; obtain the reference image dataset R = {I0,....I m} of the target domain, i.e. the target domain digital pathology image after correcting the digital color difference.

[0059] S102: Color augmentation is performed on each reference image in the reference image dataset to simulate pathology images with different staining differences. A plurality of color augmented images corresponding to each reference image with different staining differences are obtained; a similarity graph is established, taking all reference images and corresponding plurality of color augmented images as graph nodes, and taking the connection between the reference image and the color augmented image as the edge; the similarity weight of the edge connecting each color augmented image and the reference image in the similarity graph is labeled.

[0060] In implementation, the following steps can be used:

[0061] S1021: Color data augmentation is performed on any reference image I i in the reference image dataset to obtain the color augmented image set E i = {I i，0 ,....I i，n} of the reference image I i . Augmentation is performed on all reference images to obtain the color augmented image dataset E = {E0,....E m}, i.e. E is the total color augmented image dataset composed of multiple E i . Various color differences that may occur in the source domain are simulated through color augmented images. In this embodiment, the augmentation methods include randomly changing the hue, saturation and gray intensity of I i .

[0062] Color augmentation image dataset is used to simulate different staining protocols and digital pathology images of source domain in different digitization ways. In order to expand the simulation range, a larger range of random parameters is selected when randomly color augmenting, so as to expand the range of hue, saturation and gray scale.

[0063] S1022: Constructing a similarity graph of reference images and color augmented images: taking each reference image and its color augmented image as a node of the similarity graph, calculating the color similarity of the reference image and the corresponding color augmented image as the weight of the edge of the similarity graph. See Figure 4 , for a reference image I i and a color augmented image E i , a similarity graph G i is established. j is a similarity weight, which depends on the color similarity of the reference image I i and its color augmented image I i，j . The more similar I i and I i，j are, the larger λ j is. In implementation, the mutual information or normalized correlation coefficient of the pixels of the two images can be used to calculate the image similarity as the similarity weight λ j .

[0064] S201: See Figure 5 , Figure 6 , the similarity graph and the similarity weight of all edges are taken as input, and the gray scale feature map corresponding to the multiple color augmented images related to human tissue, cell structure and morphological distribution is taken as output to train the destaining model.

[0065] In some embodiments, the destaining model first encodes and then decodes the input image, and is implemented using a neural network structure with an encoder-decoder. The destaining model is a destaining model implemented by more than two layers of down-sampling encoder and more than two layers of up-sampling decoder. The output of the destaining model is also connected with a gray scale conversion network implemented by a convolution layer for image graying. The outputs of the destaining model and the gray scale conversion network are provided with different weights.

[0066] In this embodiment, see Figure 5 , the destaining model is based on a multi-level convolutional neural network model, which includes multiple convolutional layers and pooling layers. The convolutional network can use a convolutional network module with channel and spatial attention mechanisms. The number of encoders and decoders is not limited. The number of channels gradually increases from I to III, and the image size is down-sampled. The number of channels gradually decreases from I to III, and the image size is up-sampled until it is the same size as the input image.

[0067] Figure 5In this process, the output grayscale feature map is a three-dimensional image matrix with color information removed. Two dimensions are grayscale two-dimensional images of the same size as the reference image, and the third dimension represents the feature channels. The height and width of the output grayscale feature map are H and W, respectively, and the number of channels is M > 3.

[0068] In some implementations, when training the desaturation model, the training objective is to maximize the first similarity, the second similarity, and the first discriminator similarity. The gradient descent method using backpropagation is used to optimize the weights of the desaturation model and the grayscale conversion network. The first similarity is the similarity between the grayscale feature map corresponding to the reference image and the grayscale feature map corresponding to the color augmented image of the reference image. The second similarity is the similarity between the grayscale feature map corresponding to the reference image and the grayscale image converted by the grayscale conversion network of the reference image. The first discriminator similarity is the similarity between the grayscale feature map of the reference image or the color augmented image calculated by the first discriminator and the grayscale image obtained by the grayscale conversion network.

[0069] In this embodiment, the desaturation model can be trained using the similarity graphs of the reference image dataset R and the color augmented image dataset E, following these steps:

[0070] The first step is to randomly initialize the weights of the desaturation model and the convolutional network for grayscale conversion.

[0071] The second step is to extract the reference image I in a batch process. i Similarity graph G i Acquire color augmented image E i Image data and similarity weights of similarity graphs.

[0072] Step 3: Input into the desaturation model, output reference image I i Corresponding grayscale feature map I′ i and reference image I i The grayscale feature map I′ corresponding to the color augmented image i，j .

[0073] Step 4: Use a convolutional network that performs grayscale conversion on the reference image I. i Perform grayscale conversion to obtain the grayscale image I″ of the reference image. i .

[0074] Fifth step, calculate I′ i and I′ i，j First similarity, I′ i and I″ i The second similarity and the first discriminator similarity.

[0075] In the sixth step, the similarity training of the de-staining model is maximized by taking the sum of the first similarity, the second similarity and the first discriminator similarity as a function of the similarity comparison of the de-staining model, and the weight values of the de-staining model and the gray conversion network are optimized by using the gradient descent method of back propagation.

[0076] In the seventh step, the input I is replaced by i and the color augmented image E i , and the process returns to the second step. The optimization algorithm is converged or the preset training times are reached.

[0077] In some embodiments, the de-staining model can be trained independently by using the above process, or can be trained jointly with the virtual staining model and the artificial intelligence model for a specific diagnostic task.

[0078] In some embodiments, when the memory capacity is sufficient, several similarity maps G of reference images can be loaded in one batch k , wherein the number of G k is greater than 1.

[0079] In some embodiments, the convolutional network selects a full convolutional neural network architecture, which mainly consists of convolutional layers and pooling layers and does not contain fully connected layers. Thus, the size of the input image is arbitrary, and the size of the input image and the output image is consistent. The gray conversion method of the reference image I i is set to be a plurality of 1x1 convolutional networks that can be learned, and a multi-channel (channel number M>3) gray image I" i is calculated. The similarity map G i of the reference image and its color augmented image shares the de-staining model, i.e., the output gray feature map is obtained by the same model, so as to reduce the total number of network parameters.

[0080] In some embodiments, the first similarity and the second similarity both include pixel similarity, texture similarity and perceptual feature similarity.

[0081] In this embodiment, the similarity comparison function SIM_Decolor of the de-staining model is defined as follows:

[0082]

[0083]

[0084] SIM_2 = SIM(I' i , I" i )

[0085] wherein the SIM calculates the similarity of the pixel-by-pixel, texture, and perceptual features of the two feature images. The pixel similarity calculates the gray value similarity, and the texture feature calculates the statistical similarity of the pixels. The perceptual similarity extracts the high-level features of the two feature images using a pre-trained feature extractor VGG and calculates the similarity. SIM_1 is the first similarity between the gray feature map corresponding to the reference image and the gray feature map corresponding to the color augmented image of the reference image. i is a similarity weight, which depends on the color similarity of I i and I i，j . The greater the color similarity of I i and I i，j , the greater λ j . That is, SIM_1 ensures that the greater the color similarity of the reference image and the color augmented image, the greater the similarity of the gray feature maps. SIM_2 is the second similarity between the gray feature map corresponding to the reference image and the gray image converted by the gray conversion network from the reference image.

[0086] SIM 1，D is the first discriminator similarity, which is defined as the similarity between the gray feature map of the reference image or the color augmented image calculated by the first discriminator and the gray image obtained by the gray conversion network.

[0087] SIM is defined as follows:

[0088] SIM(I, J) = SIM1(I, J) + SIM2(I, J) + SIM3(I, J)

[0089] wherein I and J are two input images; SIM1(I, J) is the pixel similarity; SIM2(I, J) is the texture similarity; and SIM3(I, J) is the perceptual feature similarity.

[0090] SIM1 is defined as the similarity of the pixels:

[0091]

[0092] wherein m is the number of pixels of the input image, and I and J are two input images.

[0093] SIM2(I, J) is defined as the texture similarity.

[0094] SIM2(I, J) = - [θMean(I, J) + aCon(I, J) + βEntropy(I, J)]

[0095] wherein I and J are two input images. θ, α, and β are constant coefficients; Mean(I, J) is the average value of the gray difference of the two input feature images I and J, Con(I, J) is the gray difference contrast of I and J, Con(I, J) = |∑ l l 2 p(l) - ∑ k k 2 p(k) |; Entropy(I, J) is the gray difference entropy of I and J, l and k are respectively a gray value of I and J; p(l) and p(k) represent respectively a probability of the feature image I and J taking the gray value l and k.

[0096] SIM3(I, J) is defined as perceptual similarity.

[0097]

[0098] wherein is a feature extractor; and MSE is a mean square error function.

[0099] Further, in some embodiments, the first discriminator is implemented using a generative adversarial network. In which, the de-staining model is regarded as a generator, and the gray feature image I' i and I' i，j are regarded as generated images; and I" i is regarded as a real image. The first discriminator is defined to calculate the first discriminator similarity SIM 1，D of the generated images and the real image.

[0100]

[0101] wherein E is an expectation function; D is the first discriminator; λ j is a similarity weight, which depends on the reference input to the de-staining model and the color augmented image I i and I i，j are output by the de-staining model as I' i and I' i，j In implementation, the discriminator adopts a convolutional neural network, which inputs the gray feature image and outputs a judgment result of true or false. The number of channels M of the gray feature image is greater than 3, so as to obtain more features related to the tissue.

[0102] In some embodiments, the virtual staining network model is trained by the following steps:

[0103] S301: Refer to Figure 7 , Figure 8The virtual dyeing network model is trained by taking the gray feature map corresponding to the color augmented image output by the dyeing model as input, virtually coloring the gray feature map of the color augmented image, and taking the color image after color standardization with similar color as the reference image color as output.

[0104] In some embodiments, the virtual dyeing network model is a virtual dyeing network model implemented by more than two layers of down-sampling encoders and corresponding more than two layers of up-sampling decoders. In implementation, the virtual dyeing model can be based on a multi-level convolutional neural network model including multiple convolutional layers and pooling layers. For example, see Figure 7 The virtual dyeing model first encodes and then decodes the input image, and is implemented using a neural network structure with an encoder-decoder. The input gray feature map has a height and a width of H and W, and a channel number of M>3. The output color image has a height and a width of H and W, and a channel number of 3. The convolutional network can use a convolutional network module with channel and spatial attention mechanisms. The number of encoders and decoders is not limited. The encoders gradually increase in channel number and down-sample the image size from I to III. The decoders gradually decrease in channel number and up-sample the image size from I to III until the image size is the same as the input image.

[0105] In some embodiments, in training the virtual dyeing network model, the third similarity, the fourth similarity, and the second discriminator similarity are taken as the training target, and the gradient descent method of back propagation is used to optimize the weight value of the virtual dyeing network model. The third similarity is the similarity between the color image corresponding to the color augmented image of the reference image and the color image of the reference image. The fourth similarity is the similarity between the reference image and the color image corresponding to the reference image. The third similarity and the fourth similarity both include pixel similarity, texture similarity, and perceptual feature similarity. The second discriminator similarity is the similarity between the color image after virtual dyeing of the gray feature map of the reference image or the color augmented image calculated by the second discriminator and the reference image.

[0106] In this embodiment, the reference image dataset R and the gray feature map can be used to train the virtual dyeing model according to the following steps.

[0107] First, the virtual dyeing model is randomly initialized.

[0108] Second, the similarity map G i of the reference image I i is extracted, and the dyeing model output is used to obtain the gray feature map I' i corresponding to the reference image I i and the gray feature map I' i，j corresponding to the color augmented image of the reference image I i .

[0109] Third step, I′ i and I′ i，j Input the virtual dyeing model to obtain the color image C(I′ i ) and C(I′ i，j ).

[0110] Fourth step, calculate the third similarity of C(I′ i ) and C(I′ i，j ), the fourth similarity of C(I′ i ) and the reference image I i and the second discriminator similarity.

[0111] Fifth step, maximize the similarity of the training virtual dyeing model with the sum of the third similarity, the fourth similarity and the second discriminator similarity as the function of the similarity comparison of the virtual dyeing model, and use the gradient descent method of back propagation to optimize the weight value of the virtual dyeing model.

[0112] Sixth step, replace the similarity graph G j of the other reference image I j , and return to the second step. Until the optimization algorithm converges or reaches the preset number of iterations. When the memory capacity is sufficient, several similarity graphs G k of reference images can be loaded in a batch, where the number of G k is greater than 1.

[0113] In some embodiments, the convolutional neural network of the virtual dyeing model selects a full convolutional neural network architecture mainly composed of convolutional layers and pooling layers without fully connected layers. This makes the input image size arbitrary and the input image and output image size consistent.

[0114] In some embodiments, the gray feature maps of the reference image and its color augmented image preferably share the virtual dyeing model, that is, the output is obtained through the same model, so as to reduce the number of network parameters. The shared weight refers to the parameter in the shared model.

[0115] In this embodiment, the function SIM_Color of the similarity comparison of the virtual dyeing model is defined as follows:

[0116]

[0117]

[0118] SIM_4 = SIM(C(I′ i ), I i )

[0119] wherein the SIM calculates the similarity of the pixel-by-pixel, texture, and perceptual features of the two color images. The pixel similarity calculates the similarity of the pixel red green blue values, and the texture feature calculates the statistical similarity of the pixels. The perceptual similarity extracts the high-level features of the two color images using a pre-trained feature extractor VGG, and calculates the similarity of the two color images. SIM_3 is the third similarity between the color image corresponding to the reference image and the color image corresponding to the color augmented image of the reference image; SIM_4 is the fourth similarity between the reference image and the color image corresponding to the reference image. SIM 2，D is the second discriminator similarity, which is defined as the similarity between the color image obtained by virtually coloring the gray feature map of the reference image or the color augmented image calculated by the second discriminator, and the reference image.

[0120] The SIM is defined as follows:

[0121] SIM(I, J) = SIM1(I, J) + SIM2(I, J) + SIM3(I, J)

[0122] wherein I and J are two input feature images; SIM1(I, J) is the pixel similarity; SIM2(I, J) is the texture similarity; and SIM3(I, J) is the perceptual feature similarity.

[0123] SIM1 is defined as the similarity of the pixels:

[0124]

[0125] wherein m is the number of pixel points of the input image, and I and J are two input images.

[0126] SIM2(I, J) is defined as the texture similarity.

[0127] SIM2(I, J) = - [θMean(I, J) + aCon(I, J) + βEntropy(I, J)]

[0128] wherein I and J are two input images. θ, a, and β are constant coefficients; Mean(I, J) is the average value of the gray difference of the two input feature images I and J, Con(I, J) is the gray difference contrast of I and J, Con(I, J) = |∑ l l 2 p(l) - ∑ k k 2 p(k) |; and Entropy(I, J) is the gray difference entropy of I and J. l and k are respectively a gray value of I and J; and p(l) and p(k) represent respectively the probability of the feature image I and J taking the gray values l and k.

[0129] SIM3(I, J) is defined as perceptual similarity.

[0130]

[0131] wherein is a feature extractor; MSE is a mean square error function. In this embodiment, the feature extractor is selected as a VGG network. λ j is a similarity weight, which depends on the color similarity of I i and I i，j . The more similar I i and I i，j are, the larger λ j is, that is, Loss2 ensures that the more similar the colors of the reference image and the color augmented image are, the more similar the corresponding color images output by the virtual staining model are.

[0132] Further, in some embodiments, the second discriminator is implemented using a generative adversarial network. In this case, the virtual staining model is regarded as a generator, and the color images C(I' i ) and C(I' i，j ) are regarded as generated images; the reference image I i is regarded as a real image. The second discriminator is defined to calculate a second discriminator similarity SIM 2，D of the generated images and the real image.

[0133]

[0134] wherein E is an expectation function; D is the second discriminator; λ i is a similarity weight, which depends on the similarity of the reference images and the color augmented images I i and I i，j in the similarity map, which are output as C(I' i ) and C(I' i，j ) after the de-staining model and the virtual staining model. In implementation, the discriminator is implemented using a convolutional neural network, which inputs the virtual staining image and the reference image and outputs a true or false determination result.

[0135] S302: After training the de-staining model and the virtual staining model respectively, the two models are jointly trained using the reference image dataset R and the similarity map, so as to realize transfer learning optimization of the two models.

[0136] In some embodiments, the virtual staining model can be trained separately using the above process, or can be jointly trained with the de-staining model and an artificial intelligence model for a specific diagnostic task, as described in Figure 9 , Figure 10 .

[0137] Figure 9 In the specific implementation, the reference image dataset R and the similarity map are used to jointly train the de-staining model and the virtual staining model after the training of the two models, so that the two models are optimized through transfer learning.

[0138] Figure 10 In the specific implementation, the reference image dataset R and the similarity map are used to train the artificial intelligence model combined with the specific task, so that the result output by the artificial intelligence model is consistent with the real result labeled by the reference image dataset R. The artificial intelligence model is trained together with the de-staining model and the virtual staining model, so that the two models achieve the optimal performance for the specific diagnostic task. In the implementation, the parameters of the artificial intelligence model can also be set to be fixed and not participate in the training, so as to optimize the training of the de-staining model and the virtual staining model.

[0139] Application example: when the color standardization method of the digital pathology image of the present application is used for color standardization of the source domain image, the following steps are adopted for application:

[0140] An image is obtained from the source domain, that is, a color digital image of a pathological section obtained by other preparations and digitalization protocols different from the target domain, and the color image set of the source domain is defined as: S={II0...II k}。

[0141] A digital pathology image II i of the source domain is extracted, input into the de-staining model and the virtual staining model, and color standardization operation is performed, and the standardization process is as shown in Figure 11 .

[0142] In the first step, for any source domain image II i different from the target domain, the de-staining model is input, and the color information of the image is removed, and only the grayscale feature map II i related to human tissue, cell structure and morphological distribution is reserved.

[0143] In the second step, II i is input into the virtual staining model, and a color virtual staining digital pathology image C(II i ) is obtained, that is, a color standardized pathological image.

[0144] The present application also provides a computer system comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above embodiments when executing the computer program.

[0145] In conclusion, the color standardization method and system of the digital pathological image of the present application is based on the reference image of the target domain to establish the digital pathological color standardization method, which is independent of the source domain and is universal without using the color standardization method of the source domain image; solves the problems that the source domain image is not easy to obtain and that a standardization method is established for each source domain. At the same time, the problem that the color difference between the source domain image and the target domain image cannot be too large is solved, and the color standardization of the source domain image with obvious color change is more suitable.

[0146] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A color standardization method for digital pathological images, characterized in that, Includes the following steps: A color digital pathology image is acquired from the target domain; and the digital pathology image is corrected for color differences and used as a reference image. The reference image is color augmented to simulate different staining differences, resulting in multiple color augmented images corresponding to different staining differences for each reference image; a similarity map between the reference image and the corresponding multiple color augmented images is then constructed. The similarity map is input into the destaining model to eliminate the color information brought by staining, and outputs grayscale feature maps corresponding to multiple color-amplified images that retain information related to human tissue, cell structure and morphological distribution. The grayscale feature map corresponding to the color augmented image is input into the virtual coloring network model, and virtual coloring is applied to the grayscale feature map of the color augmented image. The output is a color image that has been colorized and has similar color to the reference image.

2. The color normalization method for digital pathological images according to claim 1, characterized in that, The desaturation model is trained through the following steps: Multiple color digital pathology images from the target domain are acquired, and a reference image dataset is generated after correcting the digital color differences of the multiple digital pathology images. Each reference image in the reference image dataset is color augmented to simulate different coloring differences, resulting in multiple color augmented images corresponding to different coloring differences for each reference image; a similarity graph is constructed with all reference images and their corresponding multiple color augmented images as graph nodes and the lines connecting the reference images and the color augmented images as edges. The similarity weight of the edge connecting each color augmented image and the reference image is marked in the similarity map; The similarity graph and the similarity weights of all edges are used as input, and the grayscale feature maps corresponding to multiple color-amplified images that retain information related to human tissue, cell structure, and morphological distribution are used as output to train the destaining model.

3. The color standardization method for digital pathological images according to claim 2, characterized in that, When training the decolorization model, the output grayscale feature map is a three-dimensional image matrix with color information removed, where two dimensions are grayscale two-dimensional images of the same size as the reference image, and the third dimension is the feature channel.

4. The color standardization method for digital pathological images according to claim 2, characterized in that, The desaturation model is implemented by an encoder with two or more downsampling layers and a corresponding decoder with two or more upsampling layers. The output of the desaturation model is also compared with the output of a grayscale conversion network implemented by convolutional layers for image grayscale conversion. The output of the desaturation model and the output of the grayscale conversion network are assigned different weights.

5. The color normalization method for digital pathological images according to claim 4, characterized in that, When training the destaining model, the training objective is to maximize the first similarity, the second similarity, and the first discriminator similarity. The gradient descent method with backpropagation is used to optimize the weights of the destaining model and the grayscale conversion network. The first similarity is the similarity between the grayscale feature map corresponding to the reference image and the grayscale feature map corresponding to the color augmented image of the reference image; The second similarity is the similarity between the grayscale feature map corresponding to the reference image and the grayscale image converted from the reference image by the grayscale conversion network; The first discriminator similarity is the similarity between the grayscale feature map of the reference image or color augmented image calculated by the first discriminator and the grayscale image obtained by the grayscale conversion network.

6. The color normalization method for digital pathological images according to claim 5, characterized in that, Both the first similarity and the second similarity include pixel similarity, texture similarity, and perceptual feature similarity.

7. The color normalization method for digital pathological images according to any one of claims 2-6, characterized in that, The virtual staining network model is trained through the following steps: The grayscale feature map corresponding to the color augmented image output by the destaining model is used as input. Virtual coloring is performed on the grayscale feature map of the color augmented image. The color image after color normalization, which has similar color to the reference image, is used as output to train the virtual coloring network model.

8. The color normalization method for digital pathological images according to claim 7, characterized in that, The virtual coloring network model is a virtual coloring network model implemented by an encoder with two or more downsampling layers and a corresponding decoder with two or more upsampling layers.

9. The color normalization method for digital pathological images according to claim 8, characterized in that, When training the virtual staining network model, the training objective is to maximize the third similarity, fourth similarity, and second discriminator similarity. The weights of the virtual staining network model are optimized using the gradient descent method with backpropagation. The third similarity is the similarity between the color image corresponding to the reference image and the color image corresponding to the color augmented image of the reference image; The fourth similarity is the similarity between the reference image and the color image corresponding to the reference image; The second discriminator similarity is the similarity between the reference image or the color image after virtual coloring of the grayscale feature map of the color augmented image calculated by the second discriminator and the reference image. The third and fourth similarities both include pixel similarity, texture similarity, and perceptual feature similarity.

10. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images