H&E staining pathological image color normalization method and system based on improved cycle-gan

Abstract

The application discloses an H&E staining pathological image color normalization method and system based on an improved Cycle-GAN, wherein the method is divided into three stages: in the first stage, after an image is converted into an optical density space OD, SVD is used for color deconvolution, so that a staining color matrix is obtained; the staining color matrix of a training set is clustered and divided into domain A and domain B; in the second stage, the staining color matrix of the image is used as auxiliary input of a model, an improved color normalization generative adversarial network model of the Cycle-GAN is trained, and different color modes of pictures are converted into a relatively unified color mode through forward cycle and backward cycle learning; and in the third stage, the color normalization generative adversarial network model based on the Cycle-GAN can perform color normalization on the pictures to be normalized to the domain B, the above-mentioned generator A based on the improved Cycle-GAN is saved, and any picture input into the generator A can also obtain a normalized result with a unified color mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an H&E staining pathological image color normalization method and system based on an improved Cycle-GAN. BACKGROUND

[0002] Digital pathology whole slide is a high-power large-scale digital image that can be displayed, transmitted and processed by computer after the pathological section is digitized by a special scanning imaging system. With the deepening of research, computer-aided diagnosis technology based on digital pathology images is booming, and various auxiliary diagnosis methods suitable for multi-staining platforms and scanning platforms in the direction of pathological whole slide are becoming mature, which has become a trend in digital pathology. Among them, hematoxylin-eosin (H&E) staining agent is the most commonly used staining method for pathological tissue. However, due to various factors such as tissue preparation, staining agent, staining platform and scanner model, the color difference of H&E pathological tissue images from different channels can be very large, which will adversely affect the computer-aided diagnosis. Therefore, it is necessary to preprocess the input image before quantitative analysis. One solution is to normalize all H&E stained images to a similar color pattern and minimize the impact of color variation on subsequent computing processes.

[0003] For this reason, domestic and foreign scholars have proposed a variety of pathological image color normalization methods. In the traditional method, some scholars directly use the natural scene image color enhancement method to normalize the digital pathology image, such as histogram equalization, histogram standardization, etc. These methods do not consider the essential characteristics of pathological image color, and the quality largely depends on the histogram similarity of the two images, so they cannot achieve satisfactory pathological image normalization effect. Most effective pathological image normalization methods are based on color deconvolution-based staining separation methods. The color of the pathological image is mixed by multiple staining agents, and the color deconvolution algorithm uses this point to convert to the optical density space and transform the image to separate the independent staining color component, and then applies the staining color component of the target image to the input image. However, the color deconvolution algorithm needs to obtain the deconvolution parameters through experiments or calculations first, and the same set of parameters may not be suitable for pathological images from different platforms. In recent years, scholars have used deep learning models for image encoding or feature extraction, such as training unsupervised sparse autoencoder (StaNoSA) to subdivide the input image into multiple regions and perform independent color normalization to match each tissue type with a single template. The disadvantage of this method is that it may cause loss of image information. SUMMARY

[0004] The application provides an H&E staining pathological image color normalization method and system based on an improved Cycle-GAN, to solve or at least partially solve the technical problem of poor processing effect in the prior art.

[0005] To solve the above technical problems, the application provides an H&E staining pathological image color normalization method based on an improved Cycle-GAN, comprising:

[0006] S1: obtaining an original image dataset, wherein the original image dataset comprises H&E staining pathological images with multiple color modes;

[0007] S2: calculating a staining color matrix for each H&E staining pathological image in the original image dataset;

[0008] S3: performing clustering on the staining color matrix by using a K-means algorithm, dividing part of the images closest to the clustering center in the largest class to a domain B with uniform color modes, and dividing the remaining images to a domain A with non-uniform color modes, wherein the largest class refers to the class with the largest number of images;

[0009] S4: inputting the images divided to the domain B and the domain A and the corresponding staining color matrix to a pre-constructed network model based on Cycle-GAN for training, wherein the network model comprises two generators G A and G B , two discriminators D A and D B , for generating images in the forward cycle period according to the images a from the domain A to imitate the domain B, and generating images b' in the backward cycle period according to the images to imitate the domain A; for generating images in the backward cycle period according to the images b from the domain B to imitate the domain B, and generating images a' in the forward cycle period according to the images to imitate the domain B; the discriminator is used for judging whether the input image is a false picture generated by the generator or a true picture from the domain B, and the discriminator is used for judging whether the input image is a false picture generated by the generator or a true picture from the domain A, and the staining color matrix is used for guiding the color conversion of the two generators;

[0010] S5: performing color normalization processing on the H&E staining pathological images to be processed by using the trained network model, to obtain a color normalization result.

[0011] In an embodiment, step S2 comprises:

[0012] S21: obtaining a light density matrix corresponding to the RGB channel of the image according to the pixel values of the R, G and B channels :

[0013]

[0014] wherein is a color matrix of the RGB channel of the image, is a single-channel maximum value;

[0015] S22: performing color deconvolution based on the light density matrix using the SVD method to obtain a staining color matrix.

[0016] In an embodiment, wherein, is a staining color matrix, is a staining density map of the stain, and S22 comprises:

[0017] S221: calculating a plane formed by vectors: finding two vectors corresponding to the two largest singular values after SVD decomposition of the image, projecting all pixels converted to the OD space onto the plane formed by the two vectors, and normalizing to unit length;

[0018] S222: calculating the angle of each pixel point relative to the direction of the largest singular value;

[0019] S223: converting each pixel point back to the OD space according to the screen formed by the vectors and the angle of each point relative to the direction of the largest singular value to obtain the staining color matrix .

[0020] In an embodiment, the training process of the network model in step S4 comprises two cycles: a forward cycle and a backward cycle, wherein in the forward cycle, an image a from domain A is converted from domain A to domain B by , and the output is , which is then converted back to domain A by , and the output is a', and a and a' follow forward cycle consistency; in the backward cycle, an image b from domain B is converted from domain B to domain A by , and the output is , which is then converted back to domain B by , and the output is b', and b and b' follow backward cycle consistency.

[0021] In an embodiment, the calculation method of the total loss function used in the training process of the network model in step S4 is:

[0022]

[0023] wherein, is the total loss function, is the weight parameter for balancing the adversarial loss and the cycle consistency loss,

[0024] is the adversarial loss of the forward cycle period, and is calculated as:

[0025]

[0026] is the adversarial loss of the backward cycle period, and is calculated as:

[0027]

[0028] is the cycle consistency loss of the forward cycle period, and is calculated as:

[0029]

[0030] is the cycle consistency loss of the backward cycle period, and is calculated as:

[0031]

[0032] wherein, is the color staining matrix, a represents an image from domain A, and b represents an image from domain B, represents the distribution of image a, represents the distribution of image b.

[0033] Based on the same inventive concept, the second aspect of the present application provides an H&E staining pathological image color normalization system based on an improved Cycle-GAN, comprising:

[0034] a data set acquisition module, configured to acquire an original image data set, wherein the original image data set comprises H&E staining pathological images with multiple color modes;

[0035] a staining color matrix calculation module, configured to calculate a staining color matrix for each H&E staining pathological image in the original image data set;

[0036] a data set division module, configured to perform clustering on the staining color matrix by using a K-means algorithm, divide part of the images closest to the clustering center in the largest class to domain B with uniform color modes, and divide the remaining images to domain A with non-uniform color modes, wherein the largest class refers to the class with the largest number of images;

[0037] The training module is used to train a pre-built Cycle-GAN-based network model by inputting the images partitioned into domain B and domain A, along with the corresponding color matrices of the images. The network model includes two generators G. A and G B Two discriminators D A and D B , Used to generate an image from domain B based on an image a from domain A during a positive loop cycle. And in the backward loop cycle, based on the image Image b' is generated by mimicking domain A; Used to mimic the generation of an image from domain B based on the image b from domain B during the backward loop cycle. And in the positive cycle, based on the image Imitation domain B generates image a'; discriminator Used to determine the input image Is the image generated by the generator a fake image or a real image from domain B? The discriminator... Used to determine the input image Whether the image is a fake image generated by the generator or a real image from domain A, the color matrix is ​​used to guide the color conversion between the two generators;

[0038] The color normalization module is used to perform color normalization processing on the H&E stained pathological images to be processed using the trained network model, and obtain the color normalization result.

[0039] In one embodiment, the system further includes a visualization module for visualizing the color normalization results.

[0040] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in the first aspect.

[0041] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0042] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0043] The application provides an H&E staining pathological image color normalization method based on an improved Cycle-GAN. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0045] Figure 1 is a flowchart of the H&E staining pathological image color normalization method based on the improved Cycle-GAN provided in the embodiments of the present application;

[0046] Figure 2 is a principle schematic diagram of the network model in the embodiments of the present application;

[0047] Figure 3 is a schematic diagram of the color normalization result in the embodiments of the present application;

[0048] Figure 4 is a structural block diagram of the H&E staining pathological image color normalization system based on the improved Cycle-GAN in the embodiments of the present application;

[0049] Figure 5 is a structural schematic diagram of the computer readable storage medium provided in the embodiments of the present application;

[0050] Figure 6 is a structural schematic diagram of the computer device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0051] In the prior art, there is a pathological image staining normalization method which uses a method of extracting content encoding and style encoding respectively to perform style transfer for color normalization. The overall idea refers to the architecture of AdaIN, replaces the encoder of traditional AdaIN with a generator, and uses a back propagation algorithm and a loss function similar to Cycle-GAN. However, this method has the following disadvantages: the discriminator does not constrain the generator, resulting in the unnecessary existence of the discriminator; the distinction between style and color is ignored, and it cannot be ensured that the style encoder correctly extracts the color features of the picture. In contrast, the method of the present application uses a network model improved based on the Cycle-GAN as the main body for training, uses forward and backward cycle losses to ensure that the content details of the pathological image after conversion by the generator are not missing; and uses an adversarial loss to ensure that the color features of the picture generated by the generator are consistent with the target domain.

[0052] In addition, there is a method of using a cycle generative adversarial network for face illumination normalization, which is mainly used for face image processing. The present application applies Cycle-GAN to pathological image color normalization, and the use scenarios of the two are different. In addition, the Cycle-GAN in the present application is improved to make it more suitable for color normalization of digital pathological tissue staining images: the SVD is used to obtain the staining color matrix of the picture, and the original Cycle-GAN is improved, and the pre-calculated staining color matrix is additionally added as the auxiliary input of the generator. On this basis, a new cycle consistency loss containing the staining color matrix as auxiliary input is proposed (which is different from the loss function of the model in the existing face illumination normalization method). It should be noted that the staining color matrix plays an important role in the present application: it helps clustering in the division process of the training set, and guides the direction of color conversion of the generator during training, assisting training and accelerating convergence.

[0053] The present application has the following advantages: the present application adds the staining color matrix as auxiliary input based on the cycle generative adversarial network, which helps clustering and assists training, and achieves better results than similar methods in testing. The present application provides a new method for color normalization of H&E pathological images, which can be applied to medical systems, facilitating the diagnosis of medical workers, and can also be applied to computer-aided diagnosis systems, which is conducive to improving the accuracy of computer-aided diagnosis.

[0054] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0055] Embodiment one

[0056] The embodiments of the present application provide an H&E staining pathological image color normalization method based on an improved Cycle-GAN, comprising:

[0057] S1: Obtain an original image dataset, wherein the original image dataset comprises H&E staining pathological images with multiple color modes;

[0058] S2: For each H&E staining pathological image in the original image dataset, calculate a staining color matrix;

[0059] S3: Perform clustering on the staining color matrix by using a K-means algorithm, divide part of the images closest to the clustering center in the largest class to a domain B with uniform color modes, and divide the remaining images to a domain A with non-uniform color modes (each image has a staining color matrix, and the images are clustered according to the staining color matrix, and part of the images closest to the clustering center in the class with the most images (the largest class) are divided into the domain B);

[0060] S4: Input the images divided into the domain B and the domain A and the corresponding staining color matrices of the images into a pre-constructed network model based on Cycle-GAN for training, wherein the network model comprises two generators G A and G B , two discriminators D A and D B , for generating images in the forward cycle period according to the images a from the domain A to imitate the domain B, and generating images b' in the backward cycle period according to the images to imitate the domain A, for generating images in the backward cycle period according to the images b from the domain B to imitate the domain B, and generating images a' in the forward cycle period according to the images to imitate the domain B, for judging whether the input images is the staining color matrix used to guide the color conversion of the two generators, whether the generated picture is generated by the generator or the true picture from domain A;

[0061] S5: using the trained network model to perform color normalization processing on the to-be-processed H&E staining pathological image to obtain a color normalization result.

[0062] Specifically, please refer to Figure 1 , the flowchart of the H&E staining pathological image color normalization method based on the improved Cycle-GAN provided in the embodiment of the application.

[0063] The application proposes a new color normalization method and system based on cycle-GAN, the key idea is to construct a domain with relatively uniform color patterns and assist the generator to generate pictures, use SVD decomposition to obtain the staining color matrix of the picture, and use the staining color matrix obtained by decomposition as the auxiliary input of the generative adversarial network. The color normalization problem can be described as converting the color pattern of the image from one domain (domain A) to the color pattern in another domain (domain B). Wherein domain A has a wide color pattern spectrum, and domain B has a relatively uniform color pattern. Based on this idea, an improved cycle-GAN based method is developed, which is summarized as follows: given a training set of a group of images with various color patterns (domain A), we first extract a subset of the image as domain B. Then, train an improved cycle-GAN to convert the color pattern in domain A to domain B and convert it back. Most of the previous methods rely on a normalized template, but this template is unlikely to represent all types of images. While the present method is template-free. The normalized color space is determined by domain B, and domain B is formed through an unsupervised clustering step. In addition, the staining color matrix plays a crucial role. In the data preparation stage, the pre-processed staining color matrix is used to cluster the images to construct domain B with uniform color patterns. In the training step, the staining color matrix is used as the auxiliary input of the generator to guide the color conversion and stabilize the cycle consistency loss.

[0064] The following is the overall idea of the invention (a total of three stages):

[0065] The first stage (preprocessing stage, including steps S1-S3) first converts the image to optical density space OD based on Lambert-Beer's law, and then uses SVD to perform color deconvolution to obtain the staining color matrix. According to the staining color matrix of the training set, it is clustered and divided into domain A and domain B for the following training.

[0066] The second stage (color normalization model training stage, corresponding to step S4) uses the staining color matrix of the image as an auxiliary input of the model, trains the color normalization generative adversarial network model improved from the cycle-GAN, and learns to convert pictures of different color modes into a relatively unified color mode through a forward cycle and a backward cycle.

[0067] The third stage (normalization stage, corresponding to step S5) is that after the above model is trained, the color normalization generative adversarial network model based on the cycle-GAN can perform color normalization on the picture to be normalized to the domain B, and the generator A based on the improved cycle-GAN is saved, and the input of any picture to the generator A can also obtain a normalized result with a unified color mode. Please refer to Figure 3 , which is a schematic diagram of the color normalization result in the embodiment of the application.

[0068] In the specific implementation process, the original image data set in step S1 adopts level 0 from the CAMELYON16 data set, which includes twenty thousand 256*256 pictures.

[0069] Step S3 adopts the K-means algorithm to divide the domain A and the domain B, and the clustering number K can be selected according to the actual situation, for example, K=10 can be taken, the domain B takes part of the pictures closest to the clustering center in the largest class, and the selection ratio of the part of the pictures is determined according to the number of images in the data set to be normalized, for example, m / 10 is taken, and m is the number of pictures in the data set to be normalized.

[0070] The network model in S4 is improved on the basis of the cycle-GAN by adding the staining color matrix obtained through color deconvolution based on SVD decomposition as an auxiliary input of the generator based on the idea of generative adversarial. The implementation principle of the network model is shown in the accompanying Figure 2 . Among them, the generator can output a “fake” picture according to the input picture. And Both use two convolutional layers with a stride of two to downsample the input, followed by several residual blocks, and then use two convolutional layers with a stride of one-half to upsample. The staining color matrix calculated from the training set is used as an auxiliary input. The discriminator structure uses 70*70 Patch-GANs, which refers to the model of the cycle-GAN.

[0071] The network model adopted by the application improves the Cycle-GAN to make it more suitable for color normalization of digital pathological tissue staining images, mainly embodied in that the SVD is used to obtain a staining color matrix, and the matrix is taken as an auxiliary input of the generator; meanwhile, a new cycle consistency loss is proposed based on this, and the staining color matrix is added to constrain the color transfer of the generator. The staining color matrix can indicate the corresponding picture color mode, and the matrix can not only guide the clustering of the data set, but also guide the conversion direction of the generator when performing color conversion, thereby playing an important role in assisting training and promoting convergence.

[0072] In an embodiment, step S2 comprises:

[0073] S21: obtaining an optical density matrix corresponding to an RGB channel of an image according to pixel values of R, G and B channels :

[0074]

[0075] wherein is a color matrix of the picture RGB channel, is a single-channel maximum value;

[0076] S22: performing color deconvolution based on the optical density matrix by using the SVD method to obtain a staining color matrix.

[0077] In the specific implementation process, the single-channel maximum value can be 255.

[0078] In an embodiment, wherein, is a staining color matrix, is a staining density map of a staining agent, and S22 comprises:

[0079] S221: calculating a plane formed by vectors: finding two vectors corresponding to two maximum singular values after SVD decomposition of the image, projecting all pixels converted to the OD space onto the plane formed by the two vectors, and normalizing to unit length;

[0080] S222: calculating an angle of each pixel point relative to the direction of the maximum singular value;

[0081] S223: converting each pixel point back to the OD space according to the screen formed by the vectors and the angle of each point relative to the direction of the maximum singular value to obtain the staining color matrix .

[0082] In an embodiment, the training process of the network model in S4 comprises two cycles: a forward cycle and a backward cycle, wherein in the forward cycle, an image a from domain A is converted to the OD space by the generator G to obtain an image a' in the OD space, and then the image a' is converted back to the RGB space by the generator F to obtain an image a'' in the RGB space; in the backward cycle, an image b from domain B is converted to the OD space by the generator H to obtain an image b' in the OD space, and then the image b' is converted back to the RGB space by the generator F to obtain an image b'' in the RGB space. From domain A to domain B, the output is , and then converted back to domain A, the output is a', a and a' follow the forward cycle consistency; in the backward cycle period, the image b from domain B is converted by From domain B to domain A, the output is , and then converted back to domain B, the output is b', b and b' follow the backward cycle consistency.

[0083] The learning rate is set to 0.0002 during training, the weights are initialized using Xavier initialization, the bias is initialized to zero, the setting is 10, the total number of training periods is 200, and the batch size is 4.

[0084] In an embodiment, the calculation method of the total loss function used in the network model training process in step S4 is as follows:

[0085]

[0086] Wherein, is the total loss function, is the weight parameter for balancing the adversarial loss and the cycle consistency loss,

[0087] is the adversarial loss in the forward cycle period, and the calculation method is as follows:

[0088]

[0089] is the adversarial loss in the backward cycle period, and the calculation method is as follows:

[0090]

[0091] is the cycle consistency loss in the forward cycle period, and the calculation method is as follows:

[0092]

[0093] is the cycle consistency loss in the backward cycle period, and the calculation method is as follows:

[0094]

[0095] In the formula, is the color dyeing matrix, a represents an image from domain A, and b represents an image from domain B, represents the distribution of image a, represents the distribution of image b. ​​

[0096] Specifically, the greater the adversarial loss, the stronger the ability of the discriminator to distinguish true and false, so we want the adversarial loss to be greater. The cycle consistency loss is used to constrain the similarity of the image converted from domain A to domain B and then converted back to domain A with the original image, so we want the cycle consistency loss to be smaller.

[0097] The objective of the generative adversarial network model of the present application is to maximize the ability of the discriminator to judge the true and false of the picture, while minimizing the loss of the generator , wherein represents the original image input into the generator. Therefore, we have:

[0098] 。

[0099] Embodiment two

[0100] Based on the same inventive concept, the present embodiment provides an H&E staining pathological image color normalization system based on an improved Cycle-GAN, please refer to Figure 4 , which comprises:

[0101] The data set acquisition module 201 is used to acquire an original image data set, and the original image data set comprises H&E staining pathological images with multiple color modes;

[0102] The staining color matrix calculation module 202 is used to calculate the staining color matrix for each H&E staining pathological image in the original image data set;

[0103] The data set division module 203 is used to perform clustering on the staining color matrix by using the K-means algorithm, and divide the part of the images closest to the clustering center in the largest class to the domain B with uniform color mode, and divide the remaining images to the domain A with non-uniform color mode;

[0104] The training module 204 is used to input the images divided to the domain B and the domain A and the corresponding staining color matrix of the images into the pre-constructed network model based on Cycle-GAN for training, wherein the network model comprises two generators G A and G B , two discriminators D A and D B , for generating images in the forward cycle period according to the images a from the domain A to imitate the domain B, and generating images b' in the backward cycle period according to the images to imitate the domain A; for generating images in the backward cycle period according to the images b from the domain B to imitate the domain B, and generating images a' in the forward cycle period according to the images imitation domain B generates an image a'; the discriminator for judging the input image is a false picture generated by the generator or a true picture from domain B, the discriminator for judging the input image is a false picture generated by the generator or a true picture from domain A, the dye color matrix is used to guide the color conversion of the two generators;

[0105] The color normalization module 205 is configured to perform color normalization on the H&E stained pathological image to be processed by using the trained network model, and obtain a color normalization result.

[0106] Since the system introduced in the second embodiment of the present application is the system used in the implementation of the H&E stained pathological image color normalization method based on the improved Cycle-GAN in the first embodiment of the present application, the specific structure and deformation of the system can be understood by those skilled in the art based on the method introduced in the first embodiment of the present application, and therefore will not be described here. Any system used in the method in the first embodiment of the present application belongs to the scope of protection of the present application.

[0107] Embodiment three

[0108] Based on the same inventive concept, please refer to Figure 5 The present application also provides a computer readable storage medium 300, which stores a computer program 311, and the program is executed to realize the method as described in the first embodiment.

[0109] Since the computer readable storage medium introduced in the third embodiment of the present application is the computer readable storage medium used in the implementation of the H&E stained pathological image color normalization method based on the improved Cycle-GAN in the first embodiment of the present application, the specific structure and deformation of the computer readable storage medium can be understood by those skilled in the art based on the method introduced in the first embodiment of the present application, and therefore will not be described here. Any computer readable storage medium used in the method of the first embodiment of the present application belongs to the scope of protection of the present application.

[0110] Embodiment four

[0111] Based on the same inventive concept, the present application also provides a computer device, as shown in Figure 6 The computer device includes a memory 401, a processor 402, and a computer program 403 stored in the memory and executable on the processor, and the processor executes the above-mentioned program to realize the method in the first embodiment.

[0112] Since the computer device introduced in the embodiment four of the present application is the computer device used for implementing the H&E staining pathological image color normalization method based on the improved Cycle-GAN in the embodiment one of the present application, the specific structure and deformation of the computer device can be understood by the person skilled in the art based on the method introduced in the embodiment one of the present application, and thus will not be described here again. The computer device used in the method in the embodiment one of the present application belongs to the scope of the present application.

[0113] Those skilled in the art will appreciate that embodiments of the present application can be a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0114] The present application is described in reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The means for performing the functions specified in a flow or multiple flows and / or blocks.

[0115] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the present application.

[0116] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An H&E staining pathological image color normalization method based on an improved Cycle-GAN, characterized in that, Comprising: S1: obtaining an original image dataset, the original image dataset comprising H&E stained pathological images with multiple color modes; S2: for each H&E stained pathological image in the original image dataset, obtaining a staining color matrix; S3: using a K-means algorithm to cluster on the staining color matrix, dividing part of the images closest to the cluster center in the largest class to a domain B with uniform color modes, and dividing the remaining images to a domain A with non-uniform color modes, the largest class referring to the class with the largest number of images; S4: The images partitioned into domain B and domain A, along with their corresponding color matrices, are input into a pre-built Cycle-GAN-based network model for training. This network model includes two generators G. A and G B Two discriminators D A and D B , Used to mimic the generation of an image from domain B based on an image a from domain A during a positive loop cycle. And in the backward loop cycle, based on the image Image b' is generated by mimicking domain A; Used to mimic the image from domain B in the backward loop cycle to generate an image from domain B. And in the positive cycle, based on the image Image a' is generated by mimicking domain B; discriminator Used to determine the input image Is the image generated by the generator a fake image or a real image from domain B? The discriminator... Used to determine the input image Is the image a fake generated by the generator or a real image from domain A? The color matrix guides the color conversion between the two generators. The training process of the network model in S4 includes two cycles: a forward loop cycle and a backward loop cycle. In the forward loop cycle, the image a from domain A is generated by... Convert from domain A to domain B, output is Then by Transforming back to domain A, the output is a', and a and a' follow the forward loop consistency; in the backward loop cycle, the image b from domain B is transformed by... Convert from domain B to domain A, output: Then by Transform back to domain B, and output b'. b and b' follow backward loop consistency. S5: using the trained network model to perform color normalization processing on the H&E stained pathological image to be processed to obtain a color normalization result.

2. The H&E staining pathology image color normalization method based on improved Cycle-GAN of claim 1, wherein, Step S2 comprises: S21: obtaining an optical density matrix corresponding to RGB channels of the image according to pixel values of R, G and B channels : wherein is a color matrix for the picture RGB channels, is a single channel maximum value; S22: using an SVD method to perform color deconvolution based on the optical density matrix to obtain the staining color matrix.

3. The H&E staining pathology image color normalization method based on improved Cycle-GAN of claim 2, wherein, wherein, is a dye color matrix, is a dye density map of the dye, and S22 comprises: S221: calculating a plane formed by vectors: finding two vectors corresponding to the two largest singular values after SVD decomposition of the image, projecting all pixels converted to the OD space onto the plane formed by the two vectors, and normalizing to unit length; S222: calculating the angle of each pixel point relative to the direction of the largest singular value; S223: According to the angle of the screen formed by the vector and each point relative to the direction of the largest singular value, each pixel point is converted back to the OD space to obtain a dyeing color matrix .

4. The H&E staining pathology image color normalization method based on improved Cycle-GAN of claim 1, wherein, The calculation method of the overall loss function used in the network model training process in step S4 is: wherein, is the total loss function, is a weight parameter for balancing the adversarial loss and the cycle consistency loss, For the adversarial loss of the forward cycle, it is calculated as: For the adversarial loss of the backward cycle, it is calculated as: For the loss of cycle consistency for the forward cycle, the calculation is: The loss of cycle consistency for the backward cycle period is calculated as follows: wherein, is a color staining matrix, a denotes an image from domain A, b denotes an image from domain B, denotes the distribution of image a, denotes the distribution of image b.

5. An H&E staining pathological image color normalization system based on improved Cycle-GAN, characterized in that, Comprising: A dataset obtaining module configured to obtain an original image dataset, the original image dataset comprising H&E stained pathological images with multiple color modes; A staining color matrix obtaining module configured to obtain a staining color matrix for each H&E stained pathological image in the original image dataset; A dataset dividing module configured to use a K-means algorithm to cluster on the staining color matrix, divide part of the images closest to the cluster center in the largest class to a domain B with uniform color modes, and divide the remaining images to a domain A with non-uniform color modes, the largest class referring to the class with the largest number of images; The training module is used to train a pre-built Cycle-GAN-based network model by inputting the images partitioned into domain B and domain A, along with the corresponding color matrices of the images. The network model includes two generators G. A and G B Two discriminators D A and D B , Used to mimic the generation of an image from domain B based on an image a from domain A during a positive loop cycle. And in the backward loop cycle, based on the image Image b' is generated by mimicking domain A; Used to mimic the image from domain B in the backward loop cycle to generate an image from domain B. And in the positive cycle, based on the image Imitation domain B generates image a'; discriminator Used to determine the input image Is the image generated by the generator a fake image or a real image from domain B? The discriminator... Used to determine the input image Whether the image is a fake image generated by the generator or a real image from domain A, the color matrix is ​​used to guide the color conversion between the two generators; A color normalization module configured to use the trained network model to perform color normalization processing on the H&E stained pathological image to be processed to obtain a color normalization result.

6. The improved Cycle-GAN based H&E staining pathology image color normalization system as claimed in claim 5, wherein, The system further comprises a visual display module configured to visually display the color normalization result.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed to implement the method of any one of claims 1 to 4.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 4 when executing the program.

Images