Pathological tissue multi-domain virtual staining method and system based on generative adversarial network

Abstract

This invention discloses a method and system for multi-domain virtual staining of pathological tissues based on generative adversarial networks (GANs). The method includes: acquiring and segmenting a histopathological slide image to obtain multiple segmented images and segment coordinates; inputting each segmented image and its corresponding target staining style label into a generator of a GAN to output the segmented staining image; the generator includes a downsampling module, multiple cascaded GramLIN modules, and an upsampling module; the GramLIN module includes a style encoding dictionary storing information on various target staining styles; and stitching all the segmented staining images together according to their segment coordinates to obtain a histopathological slide image with the target staining style. The generator incorporates multiple cascaded GramLIN modules, each with a style encoding dictionary, enabling staining transfer between multiple staining methods using a single network.

Description

Technical Field

[0001] The invention relates to the field of pathological slide staining, and in particular to a method and system for multi-domain virtual staining of pathological tissues based on generative adversarial networks. Background Technology

[0002] Histological analysis is considered the gold standard for tissue-based diagnosis, and histological staining is an essential part of this process. The most common staining agents in histological staining (also known as routine staining agents) are hematoxylin and eosin (H&E), which are used to highlight the morphological characteristics of tissue components, assisting pathologists in making preliminary diagnoses. Besides H&E staining, pathologists use various histochemical staining agents with different properties to better highlight different tissue components. For example, in human kidney tissue pathology examination, MAS staining is used to distinguish collagen fibers from muscle fibers, PAS staining is used to better examine the glomerular basement membrane, and PASM staining and PAS staining show the same components in kidney tissue, but the black staining in PASM staining can better highlight the glomeruli and is often used to visualize glomerular structure. In human liver tissue examination, MAS staining mainly shows the distribution of collagen fibers in liver tissue, and can fully reveal the degree of liver fibrosis; reticular fiber staining is mainly used to show the distribution and orientation of the reticular fiber framework in liver tissue. These non-H&E staining agents, also known as special staining agents, are often used to assist pathologists in making further diagnoses of patients.

[0003] Compared to H&E staining, specialized staining typically requires more time and manpower. Furthermore, if pathologists need to perform multiple histological stainings on patient tissue for a more accurate diagnosis, multiple samples of the patient tissue are required for different staining methods. This is wasteful in terms of resources and materials and can burden the healthcare system and patients. More often than not, pathologists decide which specialized staining to perform only after examining H&E-stained sections, undoubtedly adding significant time costs for both doctors and patients. Therefore, developing a staining transfer system would greatly increase diagnostic efficiency, saving pathologists and patients considerable time and money.

[0004] Most existing staining transfer systems can only transfer between two staining types, which is very wasteful of computational and storage resources when there are many staining types. For example, transferring between k staining types requires k(k-1) networks. This invention overcomes this shortcoming by proposing a multi-domain virtual staining method and system for pathological tissues based on generative adversarial networks. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for multi-domain virtual staining of pathological tissues based on generative adversarial networks, which can achieve staining transfer between multiple staining methods using a single network.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] This invention provides a multi-domain virtual staining method for pathological tissues based on generative adversarial networks, comprising:

[0008] Acquire a histopathological slide image and segment the histopathological slide image to obtain multiple block images, and record the block coordinates of each block image in the histopathological slide image;

[0009] Each image block and its corresponding target coloring style label are input into the generator of a generative adversarial network (GAN) to output an image block with coloring. The generator includes a downsampling module, multiple cascaded GramLIN modules, and an upsampling module. The input of the downsampling module is used to input the image block and its corresponding target coloring style label. The output of the downsampling module is connected to the input of the first GramLIN module, and the input of the upsampling module is connected to the output of the last GramLIN module. The output of the upsampling module is the image block with coloring. The GramLIN module includes a style encoding dictionary that stores various target coloring style information.

[0010] All the segmented stained images are stitched together according to the segment coordinates to obtain a histopathological section image with the target staining style.

[0011] Optionally, for each of the GramLIN modules, processing the segmented image using the GramLIN module specifically includes:

[0012] The first feature map output by the previous adjacent module of the current GramLIN module is input into the current GramLIN module; the previous adjacent module is the downsampling module or the GramLIN module preceding the current GramLIN module.

[0013] The first feature map is processed through a convolutional layer to obtain the second feature map;

[0014] Calculate the feature structure of the second feature map;

[0015] Calculate the similarity between the feature style of the second feature map and the target coloring style corresponding to the target coloring style label;

[0016] GramLIN regularization is applied based on the feature structure and the similarity to obtain a hybrid feature map of structure and style.

[0017] The structure and style hybrid feature map is processed by a nonlinear activation function to obtain a third feature map;

[0018] A fourth feature map is obtained based on the first feature map and the third feature map; the fourth feature map serves as the input to the next GramLIN module after the current GramLIN module or as the input to the upsampling module.

[0019] Optionally, calculating the feature structure of the second feature map specifically includes:

[0020] The second feature map is subjected to instance regularization and layer regularization respectively to obtain the instance regularization result and the layer regularization result.

[0021] The feature structure of the second feature map is determined based on the instance regularization result and the layer regularization result.

[0022] Optionally, calculating the similarity between the feature style of the second feature map and the target coloring style corresponding to the target coloring style label specifically includes:

[0023] Calculate the feature Gram matrix of the second feature map, and transform the dimension of the feature Gram matrix to 1×C. 2 This yields the feature style vector; C represents the number of channels for the feature.

[0024] Based on the target coloring style label, the corresponding target coloring style vector is retrieved from the style encoding dictionary;

[0025] The similarity is obtained by taking the inner product of the feature style vector and the target coloring style vector.

[0026] Optionally, the expression used to obtain the hybrid structure and style feature map by applying GramLIN regularization based on the feature structure and the similarity is:

[0027]

[0028] in, and It is the affine transformation factor. γ1, γ2, β1, β2 are the learning parameters; Indicates similarity; ρ represents the scaling factor; This represents the result of instance regularization. The result of the regularization process for the representation layer.

[0029] Optionally, the style encoding dictionary follows regularization constraints based on Rényi entropy;

[0030] The expression for the regularization constraint based on Rényi entropy is:

[0031]

[0032] In the formula, α is an integer greater than 1, and S α (A) is the Rényi entropy of the positive definite matrix A; the positive definite matrix A is obtained by transposing the style coding dictionary matrix and the style coding dictionary matrix to obtain the style Gram matrix, and then normalizing the style Gram matrix by its diagonal; tr(A) α ) represents matrix A α The traces.

[0033] Optionally, the loss function of the generator is:

[0034]

[0035]

[0036]

[0037] l cyc =||G(G(x,L) t ),L o )-x||1

[0038] l idt =||G(x,L o )-x||1

[0039]

[0040] In the formula, λ1 = 10; λ2 = 1; x is the input image of the generator; l non-dia L1 loss for off-diagonal elements of a positive definite matrix; BCE() represents the cross-entropy loss function; G represents the generator; D adv () represents the image category discrimination branch of the discriminator in a generative adversarial network; D cls () represents the coloring-class discrimination branch of the discriminator in a generative adversarial network; L o It is the coloring style label of the input image; L t It is the target coloring style tag.

[0041] Optionally, after stitching together all the segmented stained images according to the segment coordinates to obtain a histopathological section image with the target staining style, the method further includes:

[0042] The histopathological slide image with the target staining style is input into the discriminator in the generative adversarial network;

[0043] The discriminator is used to distinguish whether the histopathological slide image with the target staining style is the original histopathological slide image or the image output by the generator.

[0044] The discriminator is used to classify the staining types of the histopathological slide images with the target staining style.

[0045] Optionally, the loss function of the discriminator is:

[0046]

[0047]

[0048]

[0049] Where x′ is the input image of the discriminator.

[0050] This invention also provides a multi-domain virtual staining system for pathological tissues based on generative adversarial networks, comprising:

[0051] An image acquisition and processing subsystem is used to acquire histopathological slide images and segment the histopathological slide images to obtain multiple block images, and record the block coordinates of each block image in the histopathological slide image.

[0052] A target coloring subsystem is used to input each of the segmented images and the corresponding target coloring style labels into the generator of a generative adversarial network (GAN), and output the segmented coloring image. The generator includes a downsampling module, multiple cascaded GramLIN modules, and an upsampling module. The GramLIN module includes a style encoding dictionary that stores various target coloring style information. The input terminal of the downsampling module is used to input the segmented image and the corresponding target coloring style label. The output terminal of the downsampling module is connected to the input terminal of the first GramLIN module, and the input terminal of the upsampling module is connected to the output terminal of the last GramLIN module. The output terminal of the upsampling module outputs the segmented coloring image.

[0053] The splicing subsystem is used to splice all the segmented stained images according to the segment coordinates to obtain histopathological slide images with the target staining style.

[0054] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0055] This invention relates to a method and system for multi-domain virtual staining of pathological tissues based on generative adversarial networks (GANs), comprising: acquiring and segmenting a histopathological slide image to obtain multiple segmented images and segment coordinates; inputting each segmented image and its corresponding target staining style label into a generator of a GAN to output a segmented staining image; the generator includes a downsampling module, multiple cascaded GramLIN modules, and an upsampling module; the GramLIN module includes a style encoding dictionary storing information on multiple target staining styles; and stitching all segmented staining images together according to the segment coordinates to obtain a histopathological slide image with the target staining style. By introducing multiple cascaded GramLIN modules into the generator, and by incorporating a style encoding dictionary storing information on multiple staining styles when processing feature images based on the GramLIN modules, staining transfer between multiple staining styles can be achieved using a single network. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 Here is a flowchart of a multi-domain virtual staining method for pathological tissues based on generative adversarial networks, provided in Embodiment 1 of the present invention.

[0058] Figure 2 This is a schematic diagram of a multi-domain virtual staining method for pathological tissues based on generative adversarial networks, provided in Embodiment 1 of the present invention.

[0059] Figure 3 This is a generator framework diagram provided in Embodiment 1 of the present invention;

[0060] Figure 4 This is a diagram of the GramLIN module framework provided in Embodiment 1 of the present invention;

[0061] Figure 5 This is a constraint framework diagram of the style encoding dictionary Q provided in Embodiment 1 of the present invention;

[0062] Figure 6 The image shows the result of an example of open-source software provided in Embodiment 1 of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The purpose of this invention is to provide a method and system for multi-domain virtual staining of pathological tissues based on generative adversarial networks (GANs). By introducing multiple cascaded GramLIN modules into the generator, and further incorporating a style encoding dictionary storing information on various staining styles when processing feature images using these GramLIN modules, staining transfer between different staining methods can be achieved using a single network. Specifically, according to this method, pathologists can generate more expensive and time-consuming staining methods from less costly and less time-consuming methods, significantly saving time and money.

[0065] This invention enables staining transfer between any two histochemical staining methods using a single network. The method and system can save pathologists and patients significant time and money in tissue analysis, increasing the prevalence of specific staining methods in pathological diagnosis. Furthermore, in achieving this task, the invention proposes an unsupervised multi-domain staining transfer framework based on a style-encoded dictionary. In this framework, various staining style features are encoded in the dictionary, allowing the model to query the similarity between the current feature and the target style feature during style transfer. By querying the similarity to the target staining style, the model can adaptively adjust the staining transfer direction, thus achieving accurate multi-domain staining transfer. Based on the framework proposed in this invention, any stained section can be quickly and accurately converted into other types of stained sections, greatly facilitating pathologists' diagnosis.

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Example 1

[0068] like Figure 1 and 2 As shown, this embodiment provides a multi-domain virtual staining method for pathological tissues based on generative adversarial networks, including:

[0069] S1: Acquire a histopathological slide image and segment the histopathological slide image to obtain multiple block images, and record the block coordinates of each block image in the histopathological slide image.

[0070] Pathologists use a holographic microscope to scan patient tissue slides, resulting in an input full-slide image (hereinafter referred to as a slide). The system then segments the input slide into numerous local tissue images (hereinafter referred to as patches) and assigns them coordinates (e.g., ...). Figure 2 The positions of these patches in the input slide are recorded in the form of (x1,y1), (x2,y2), etc.

[0071] Specifically, H&E-stained and specially stained pathological sections were scanned using a holographic microscope at a magnification of 20x. The scanned images were then divided into a large number of 256×256 patches, and the coordinate information of each patch was saved together with the patches.

[0072] S2: Input each of the segmented images and the corresponding target coloring style labels into the generator of the generative adversarial network, and output the segmented coloring images.

[0073] like Figure 3 As shown, the generator includes a downsampling module, multiple cascaded GramLIN modules (the number is not necessarily determined by the task; in a specific embodiment, there are 8 modules), and an upsampling module. The input of the downsampling module is used to input the patch image and the corresponding target coloring style label; the output of the downsampling module is connected to the input of the first GramLIN module, and the input of the upsampling module is connected to the output of the last GramLIN module; the output of the upsampling module is the patch coloring image. The downsampling module reduces the number of parameters learned by the network while increasing the receptive field. The GramLIN module adjusts the direction of coloring transfer by measuring the progress of the current coloring transfer; its detailed structure and function will be introduced later. The upsampling module infers the output patch based on the features output by the last GramLIN module.

[0074] The number of GramLIN modules connected in series is not fixed and depends on the actual task. For example, 8 modules or any number of modules can be selected.

[0075] Each image block and its corresponding target coloring style label (in the target coloring style label, one-hot encoding is used to represent different coloring styles) are first input into the downsampling module. The downsampling output is then input into the first GramLIN module. The output of the first GramLIN module is then input into the second GramLIN module, and so on. The output of the last GramLIN module is input into the upsampling module. The output of the upsampling module is the final image of the target coloring style.

[0076] Each GramLIN module completes a small step from the original style domain to the target style domain, and the entire color transfer is only completed when the output of the upsampling module is reached.

[0077] like Figure 4 As shown, each GramLIN module includes a style encoding dictionary, which stores information on various target coloring styles. The framework of each GramLIN module is... Figure 4 The module framework is shown.

[0078] The specific processes within each of the GramLIN modules are described in detail below:

[0079] For each of the GramLIN modules, the segmented image is processed using the GramLIN module, specifically including:

[0080] S21: Input the first feature map X output by the previous adjacent module of the current GramLIN module into the current GramLIN module; the previous adjacent module is the downsampling module or the GramLIN module preceding the current GramLIN module.

[0081] S22: Process the first feature map X through a convolutional layer to obtain the second feature map.

[0082] The essence of color transfer is to preserve the structure of the input features while transferring the style of the input features to the target style. Therefore, we will obtain the following... The structure, and calculation The similarity between the staining style and the target staining style guides the staining style transfer. Steps S23 and S24 are performed simultaneously.

[0083] S23: Calculate the second feature map The characteristic structure.

[0084] Obtain the second feature map Structure: Based on the existing method UGATIT, we obtain Right now The structure, in which and They are respectively for The results of instance regularization and layer regularization;

[0085] Specifically, step S23 includes:

[0086] (1) Perform instance regularization and layer regularization on the second feature map respectively to obtain the instance regularization result and the layer regularization result.

[0087] (2) Determine the feature structure of the second feature map based on the instance regularization result and the layer regularization result.

[0088] S24: Calculate the second feature map Similarity between the coloring style and the target coloring style corresponding to the target coloring style label

[0089] like Figure 4 As shown, the second feature map is calculated first. Gram matrix G F , Gram matrix G F Flattened to a dimension of 1×C 2 The feature style vector V is used to characterize The style is determined by the input target coloring style label L. Then, the target coloring style vector M is retrieved from the style encoding dictionary Q. Next, the inner product of the current feature style vector V and the target coloring style vector M is taken to obtain the similarity between the current feature style and the target coloring style. Its formula is shown below:

[0090]

[0091] L T Transpose the target coloring style label L of the input.

[0092] Specifically, step S24 includes:

[0093] (1) Calculate the feature Gram matrix of the second feature map, and transform the dimension of the feature Gram matrix to 1×C. 2 This yields the feature style vector. C represents the number of channels for the feature.

[0094] (2) Based on the target coloring style label, the corresponding target coloring style vector is found in the style encoding dictionary.

[0095] (3) Take the inner product of the feature style vector and the target coloring style vector to obtain the similarity.

[0096] S25: Apply GramLIN regularization based on the feature structure and the similarity to obtain a hybrid feature map of structure and style.

[0097] The expression used to obtain the hybrid feature map of structure and style in step S25 is:

[0098]

[0099] in,; and It is the affine transformation factor. γ1, γ2, β1, β2 are the learning parameters; ρ represents the similarity; ρ represents the scaling factor, which is the proportion of instance regularization and layer regularization respectively. This represents the result of instance regularization. The result of the regularization process for the representation layer.

[0100] S26: The structure and style hybrid feature map is processed by the nonlinear activation function ReLU to obtain the third feature map. Figure 4 In the diagram, H, W, and C represent the width, height, and number of channels of the feature block.

[0101] S27: Based on the first feature map X and the third feature map A fourth feature map is obtained; the fourth feature map is used as the input of the next GramLIN module after the current GramLIN module or as the input of the upsampling module.

[0102] The third feature map The fourth feature map is obtained by adding it to the first feature map X.

[0103] To enable the style encoding dictionary Q to better store coloring style information, and to maximize the amount of style information stored in the style encoding dictionary while ensuring that the differences between different coloring styles are as large as possible, a design was implemented as follows: Figure 5 The constraints are shown. Since each column of Q represents a coloring style, Q... T The matrix G obtained by Q Q The similarity between coloring styles is represented by a matrix, and diagonal normalization yields a positive definite matrix A that characterizes the relationships between different coloring styles. To enhance the differentiation between different coloring styles and increase the storage of style information in D, the style encoding dictionary follows a regularization constraint based on Rényi entropy.

[0104] The expression for the regularization constraint based on Rényi entropy is:

[0105]

[0106] In the formula, α is an integer greater than 1, and S α (A) is the Rényi entropy of the positive definite matrix A. The smaller its value, the greater the amount of coloring style feature information stored in Q; tr(A) α ) represents matrix A α The trace; the positive definite matrix A is obtained by transposing the style encoding dictionary matrix and the style encoding dictionary matrix to obtain the style Gram matrix, and then normalizing the style Gram matrix by diagonal.

[0107] In applying the method of this embodiment, in order to increase the amount of coloring style feature information stored in Q, a l was designed. R To reduce S α (A), as shown in the following formula:

[0108]

[0109] Among them, reducing S α (A) This not only increases the amount of coloring style feature information stored in Q, but also reduces the correlation between different coloring styles in Q, and enhances the specific features of each coloring style. To further reduce the correlation between different coloring styles in Q, the values ​​of the off-diagonal elements in the positive definite matrix A need to be minimized. The method provided in this embodiment adds an L1 loss to the off-diagonal elements of A, i.e., l... non-dia To minimize its value, thus assisting the Rényi entropy regularization constraint and further enhancing the coloring style transfer effect. R Add it to the generator loss function.

[0110] To more clearly illustrate the application of the style encoding dictionary Q, the following example is provided:

[0111] For example, assuming the style encoding dictionary Q stores four coloring styles: H&E, PAS, MAS, and PASM, then style encoding dictionary Q is a matrix with four columns, each column corresponding to a coloring style vector. When a target coloring style label is input into the generator, the corresponding target coloring style vector is extracted based on the label. Figure 3 In this example, if the target coloring style label for H&E is 1000, it means that the first column of the style coding dictionary Q is selected based on the label; if the target coloring style label for PAS is 0100, it means that the second column of the style coding dictionary Q is selected based on the label. It should be noted that the target coloring style coding here is merely an illustrative example and has no limiting effect.

[0112] S3: Stitch together all the segmented stained images according to the segment coordinates to obtain a histopathological section image with the target staining style.

[0113] The generator outputs the transformation result (i.e. Figure 2 After processing the output patches, these output patches are pieced back together into a slide according to the coordinates recorded when slicing the image, and then given to the pathologist for review.

[0114] Following step S3, a two-branch discriminator similar to the existing method StarGAN was designed, with one branch D. adv Another branch, D, is used to distinguish whether the input image is real or generated by the generator.cls This is used to classify the coloring types of the input image. That is, after step S3, it also includes:

[0115] The histopathological slide image with the target staining style is input into the discriminator in the generative adversarial network.

[0116] The discriminator is used to distinguish whether the histopathological slide image with the target staining style is the original histopathological slide image or the image output by the generator.

[0117] The discriminator is used to classify the staining types of the histopathological slide images with the target staining style.

[0118] For the generator, the loss function is:

[0119]

[0120]

[0121]

[0122] l cyc =||G(G(x,L) t ),L o )-x||1

[0123] l idt =||G(x,L o )-x||1

[0124]

[0125] In the formula, λ1 = 10; λ2 = 1; x is the input image of the generator; l non-dia L1 loss for off-diagonal elements of a positive definite matrix; BCE() represents the cross-entropy loss function; G represents the generator; D adv () represents the image category discrimination branch of the discriminator in a generative adversarial network; D cls () represents the coloring-class discrimination branch of the discriminator in a generative adversarial network; L o It is the coloring style label of the input image; L t It is the target coloring style tag.

[0126] For the discriminator, the loss function is:

[0127]

[0128]

[0129]

[0130] Where x′ is the input image of the discriminator.

[0131] In this embodiment, to facilitate the practical application of the method of the present invention, open-source software is designed, such as... Figure 6 As shown, it can automatically generate a slide with the target coloring style based on the input slide and the target coloring style label, which takes about 1 minute.

[0132] This invention enables color transfer between multiple coloring styles. Compared to existing color transfer methods that only handle two coloring styles, this method's network is fed with images of multiple coloring styles, thus enabling it to extract more diverse features and significantly improving performance. Furthermore, because this method uses a single network to transfer between multiple coloring styles, it greatly reduces memory consumption compared to existing methods.

[0133] More importantly, in clinical practice, this method can achieve staining migration quickly and accurately, saving patients and doctors a lot of time.

[0134] Example 2

[0135] This embodiment provides a multi-domain virtual staining system for pathological tissues based on generative adversarial networks, including:

[0136] The image acquisition and processing subsystem M1 is used to acquire histopathological slide images and segment the histopathological slide images to obtain multiple block images, and record the block coordinates of each block image in the histopathological slide image.

[0137] The target coloring subsystem M2 is used to input each of the segmented images and the corresponding target coloring style labels into the generator of the generative adversarial network (GAN), and output the segmented coloring image. The generator includes a downsampling module, multiple cascaded GramLIN modules, and an upsampling module. The GramLIN module includes a style encoding dictionary, which stores various target coloring style information. The input terminal of the downsampling module is used to input the segmented image and the corresponding target coloring style label. The output terminal of the downsampling module is connected to the input terminal of the first GramLIN module, and the input terminal of the upsampling module is connected to the output terminal of the last GramLIN module. The output terminal of the upsampling module outputs the segmented coloring image.

[0138] The splicing subsystem M3 is used to splice all the segmented stained images according to the segment coordinates to obtain histopathological slide images with the target staining style.

[0139] Each embodiment in this specification focuses on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be found in the method section.

[0140] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-domain virtual staining method for pathological tissues based on generative adversarial networks, characterized in that, include: Acquire a histopathological slide image and segment the histopathological slide image to obtain multiple block images, and record the block coordinates of each block image in the histopathological slide image; Each image block and its corresponding target coloring style label are input into the generator of a generative adversarial network (GAN) to output an image block with coloring. The generator includes a downsampling module, multiple cascaded GramLIN modules, and an upsampling module. The input of the downsampling module is used to input the image block and its corresponding target coloring style label. The output of the downsampling module is connected to the input of the first GramLIN module, and the input of the upsampling module is connected to the output of the last GramLIN module. The output of the upsampling module is the image block with coloring. The GramLIN module includes a style encoding dictionary that stores various target coloring style information. All the segmented stained images are stitched together according to the segment coordinates to obtain a histopathological section image with the target staining style; Specifically, for each of the GramLIN modules, the processing of the segmented image using the GramLIN module includes: The first feature map output by the previous adjacent module of the current GramLIN module is input into the current GramLIN module; the previous adjacent module is the downsampling module or the GramLIN module preceding the current GramLIN module. The first feature map is processed through a convolutional layer to obtain the second feature map; Calculate the feature structure of the second feature map; Calculate the similarity between the feature style of the second feature map and the target coloring style corresponding to the target coloring style label; GramLIN regularization is applied based on the feature structure and the similarity to obtain a hybrid feature map of structure and style. The structure and style hybrid feature map is processed by a nonlinear activation function to obtain a third feature map; A fourth feature map is obtained based on the first feature map and the third feature map; the fourth feature map serves as the input of the next GramLIN module after the current GramLIN module or as the input of the upsampling module; Specifically, calculating the feature structure of the second feature map includes: The second feature map is subjected to instance regularization and layer regularization respectively to obtain the instance regularization result and the layer regularization result. The feature structure of the second feature map is determined based on the instance regularization result and the layer regularization result.

2. The method according to claim 1, characterized in that, The calculation of the similarity between the feature style of the second feature map and the target coloring style corresponding to the target coloring style label specifically includes: Calculate the feature Gram matrix of the second feature map, and transform the dimension of the feature Gram matrix to... This yields the feature style vector; C The number of channels representing the feature; Based on the target coloring style label, the corresponding target coloring style vector is retrieved from the style encoding dictionary; The similarity is obtained by taking the inner product of the feature style vector and the target coloring style vector.

3. The method according to claim 2, characterized in that, The expression used to obtain the hybrid structure and style feature map by applying GramLIN regularization based on the feature structure and the similarity is: in, and It is the affine transformation factor. , , For learning parameters; Indicates similarity; Indicates the scaling factor; This represents the result of instance regularization. The result of the regularization process for the representation layer.

4. The method according to claim 1, characterized in that, The style encoding dictionary follows regularization constraints based on Rényi entropy; The expression for the regularization constraint based on Rényi entropy is: In the formula, It is an integer greater than 1. It is a positive definite matrix Rényi entropy; positive definite matrix It is obtained by converting the style encoding dictionary matrix and the style encoding dictionary matrix transpose to obtain the style Gram matrix, and then normalizing the style Gram matrix diagonally. tr ( ) represents a matrix The traces.

5. The method according to claim 4, characterized in that, The loss function of the generator is: In the formula, ; ; It is the input image of the generator; Let L1 loss be the L1 loss of the off-diagonal elements in the positive definite matrix; BCE() represents the cross-entropy loss function; G represents the generator; ( ) represents the image category discrimination branch of the discriminator in a generative adversarial network; ( ) represents the coloring-class discrimination branch of the discriminator in a generative adversarial network; It is the coloring style label of the input image; It is the target coloring style tag.

6. The method according to claim 1, characterized in that, After stitching together all the segmented stained images according to the segment coordinates to obtain a histopathological section image with the target staining style, the process further includes: The histopathological slide image with the target staining style is input into the discriminator in the generative adversarial network; The discriminator is used to distinguish whether the histopathological slide image with the target staining style is the original histopathological slide image or the image output by the generator. The discriminator is used to classify the staining types of the histopathological slide images with the target staining style.

7. The method according to claim 5, characterized in that, The loss function of the discriminator is: in, It is the input image for the discriminator.

8. A system based on the multi-domain virtual staining method for pathological tissues based on generative adversarial networks as described in any one of claims 1 to 7, characterized in that, include: An image acquisition and processing subsystem is used to acquire histopathological slide images and segment the histopathological slide images to obtain multiple block images, and record the block coordinates of each block image in the histopathological slide image. A target coloring subsystem is used to input each of the segmented images and the corresponding target coloring style labels into the generator of a generative adversarial network (GAN), and output the segmented coloring image. The generator includes a downsampling module, multiple cascaded GramLIN modules, and an upsampling module. The GramLIN module includes a style encoding dictionary that stores various target coloring style information. The input terminal of the downsampling module is used to input the segmented image and the corresponding target coloring style label. The output terminal of the downsampling module is connected to the input terminal of the first GramLIN module, and the input terminal of the upsampling module is connected to the output terminal of the last GramLIN module. The output terminal of the upsampling module outputs the segmented coloring image. The splicing subsystem is used to splice all the segmented stained images according to the segment coordinates to obtain histopathological slide images with the target staining style.

Images