A multi-modal general large field-of-view virtual staining post-processing method

Abstract

The present invention provides a multi-modal general large field-of-view virtual staining post-processing method, which includes the following steps: Step 1, establish a cyclic adversarial generation network, convert the red-green-blue (RGB) color space to the hue (H), saturation (S), and value (V) color space, and extract the value channel; Step 2, establish a large field-of-view stitching module based on confidence to process large field-of-view microscopic images and eliminate artifacts in the boundary inconsistency during the stitching process of small images. The method of the present invention can ensure correct color mapping relationships and retention of image content during the modality conversion process. At the same time, it eliminates artifacts generated during image sequence stitching with a plug-and-play post-processing method. This framework can handle various cross-modal conversions and is very suitable for processing large-scale and high-resolution images, effectively solving the boundary inconsistency problems encountered in many existing methods.

Description

Technical Field

[0001] The present invention relates to the field of virtual staining of microscopic histology, and particularly to a multi-modal general large-field virtual staining post-processing method. Background Art

[0002] In disease diagnosis and pathological research, tissue staining plays an important role as a basic technique. By using specific markers to label according to the inherent characteristics of biological elements, tissue staining can visualize different tissue and cell structures, thus providing support for subsequent pathological analysis and disease diagnosis.

[0003] The application of various staining techniques highlights diverse biological characteristics. Hematoxylin-eosin staining is an important method in pathological sections, providing rich information and widely used in tumor diagnosis. Immunohistochemical staining is a technique that combines immunology with traditional histology. Through the specific binding interaction between antigens and antibodies, it can detect and localize specific chemical substances in tissues and cells. For example, immunohistochemical staining of proliferating cell antigen is often used to evaluate cell proliferation activity, helping pathologists and researchers quantitatively evaluate the degree of tumor proliferation. However, these standard tissue staining procedures are usually carried out in pathological laboratories, involving multiple sample preparation processes, which are both laborious and time-consuming.

[0004] Deep learning has made remarkable progress in the field of optical microscopy, including the application of virtual tissue staining. Virtual staining bypasses the cumbersome sample preparation, professional laboratory equipment, and technical personnel requirements in traditional chemical tissue staining. By using deep learning to generate digital tissue staining, virtual staining provides an efficient, accurate, and cost-effective alternative. Deep learning-based virtual staining methods are mainly divided into two categories: supervised learning and unsupervised learning. Supervised learning methods rely on a large number of high-quality paired images; however, in optical microscopy applications, it is often challenging to provide a complete one-to-one correspondence between two data modalities. Therefore, unsupervised learning methods based on cycle generative adversarial networks are favored by relevant researchers because of their practical value in clinical imaging and scientific research. Summary of the Invention

[0005] Object of the Invention: The technical problem to be solved by the present invention is to provide a multi-modal general large-field virtual staining post-processing method in view of the deficiencies of the prior art, aiming to provide a general virtual staining framework that can achieve accurate color mapping between different pathological modalities.

[0006] The method of the present invention includes the following steps:

[0007] Step 1, establish a cycle adversarial generation network, convert the red-green-blue (RGB) color space to the hue (H), saturation (S), and value (V) color space, and extract the value channel;

[0008] Step 2: Establish a large field-of-view stitching module based on confidence to process large field-of-view microscopic images and eliminate artifacts in the inconsistent boundaries during the stitching process of small images;

[0009] In Step 1, set image datasets that are unpaired in A modality and B modality respectively. The cycle adversarial generation network includes adversarial loss, cycle consistency loss, and chroma-lightness loss;

[0010] The adversarial loss is used to train the generator to generate virtual images similar to real images. The mathematical expression is:

[0011] ,

[0012] ,

[0013] where and respectively represent the adversarial loss of generator and the adversarial loss of generator . denotes randomly sampling an image b from the image distribution of B modality, denotes randomly sampling an image from the image distribution of A modality. E(·) and p(·) respectively represent the expectation and distribution probability of data, and represent two generators, and represent two discriminators; Generator takes the real image from A modality as input and generates a virtual image in B modality. Discriminator is used to distinguish between real images and virtual images from B modality; Generator generates a virtual image in A modality based on the real image in B modality. Discriminator is used to distinguish between real images and virtual images in A modality;

[0014] The cycle consistency loss is used to maintain the integrity of the original image content. The mathematical expression is:

[0015] ,

[0016] where is L1 regularization; is the cycle consistency loss; represents the expectation of images in A modality, represents the expectation of images in B modality;

[0017] The color lightness loss is used to convert the RGB color space into the hue (H), saturation (S), and value (V) color space, extract the value channel, and perform color lightness constraint to ensure the accuracy of color mapping. The mathematical expression is:

[0018] ,

[0019] where, and correspond to the outputs of the generator for the input of the real image in modality A and the outputs of the generator for the input of the real image in modality B; value(·) represents the value of the value channel, is the color lightness loss, is the weight of the color lightness loss;

[0020] In step 2, the large field-of-view stitching module based on confidence crops the large-scale high-resolution full field-of-view image of the real image in modality A into a sequence of small images of every m pixels , and inputs the sequence into the network model trained in step 1 to obtain the output sequence of the virtual image in modality B;

[0021] According to the original positions of the sequence in the full field-of-view input, each image of the sequence is filled into a zero-padded matrix to form a sequence ;

[0022] Design a weight matrix with the same size as the full field-of-view input, multiply the weight matrix by the sequence using the Hadamard product, and perform normalization to obtain the final virtual full field-of-view output in modality B. The formula is:

[0023] ,

[0024] where, represents the i-th small image sequence after zero-padding after virtual staining, N represents the number of small images, represents the Hadamard product.

[0025] Step 1 also includes: adopting the following loss function :

[0026] ;

[0027] In step 1, the network model is trained according to the established cyclic adversarial generation network.

[0028] In step 2, the weight matrix is calculated using the following formula:

[0029] ,

[0030] where represents the value of the weight matrix at the x-th row and y-th column in the upper left corner among all the red, green, and blue RGB channels, is the ceiling function, n is the size of the cropped small patch, and m is the padding interval between adjacent small patches.

[0031] In step 2, is an integer.

[0032] The method is plug-and-play and suitable for processing large field-of-view super-resolution images.

[0033] The method is used for virtual staining between different types of pathological modalities and does not require adjusting hyperparameters, including autofluorescence, hematoxylin and eosin staining (H&E), and immunohistochemical staining (IHC).

[0034] The present invention also provides an electronic device, including a processor and a memory. The memory stores program code, and when the program code is executed by the processor, the processor is caused to execute the steps of the method.

[0035] The present invention also provides a storage medium storing a computer program or instruction, and when the computer program or instruction runs on a computer, the steps of the method are executed.

[0036] The present invention also provides a post-processing method based on confidence stitching for large field-of-view super-resolution image processing. This method tiles patches into a whole for large field-of-view imaging stitching by designing a weight matrix.

[0037] By optimizing the cyclic adversarial generation network, the present invention solves the problem of color distortion in virtual staining, ensures correct color mapping relationships and retention of image content during modality conversion. At the same time, it is very suitable for processing large-scale and high-resolution images and solves the boundary inconsistency problem encountered in many existing methods. It includes the following beneficial effects: (1) Stability in multi-modal conversion: The present invention can be widely applied to virtual staining under different modalities, provides a general and effective solution for different staining tasks, and meets the task requirements of various histological virtual staining.

[0038] (2) Practicality of large field-of-view super-resolution images: Through the present invention, for super-resolution large field-of-view microscopic imaging, artifacts generated by patch stitching can be eliminated in a plug-and-play and training-free manner, thus facilitating subsequent downstream analysis tasks. Description of the Drawings

[0039] The following further specific description of the present invention will be made in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become clearer.

[0040] Figure 1 It is a structural block diagram of the device of the present invention.

[0041] Figure 2 It is a schematic diagram of the structural flow of the training network module in the embodiment of the present invention.

[0042] Figure 3 It is a schematic diagram of the large field of view image stitching process in the embodiment of the present invention.

[0043] Figure 4 It is a result diagram of H&E staining of the autofluorescence image in the embodiment of the present invention. Specific Embodiments

[0044] The embodiment of the present invention takes autofluorescence - hematoxylin - eosin staining virtual staining as an example. Referring to Figure 1 As shown, based on the HSV space conversion, the embodiment of the present invention proposes a color lightness mapping virtual staining method based on a cyclic adversarial generation network. The present invention designs a new constraint by extracting the color lightness channel of the HSV space and correspondingly modifies the loss function of the cyclic adversarial generation network. The main structure of the framework of the present invention is as shown in reference to Figure 1 As shown, in which the present invention replaces the identity loss in the original network with a color lightness loss to ensure the correct color mapping relationship between the A modality and the B modality. It should be noted that in addition to calculating the color lightness loss, the main process of the method of the present invention is carried out separately in the RGB channels.

[0045] Referring to Figure 2 As shown, the method of the present invention establishes a training network module for unsupervised training of two unpaired modal image sequences, and the specific implementation method is as follows:

[0046] According to the original cyclic adversarial generation network, the present invention divides the unpaired images into the A modality and the B modality, so that the network can perform unsupervised training. The entire architecture includes two generators and and two discriminators and . The generator takes the real image from the A modality as input and generates a virtual image of the B modality, while the discriminator distinguishes between the real image and the virtual image from the B modality. Each generator has a corresponding discriminator that attempts to distinguish the image it generates from the real image. Similarly, the generator Generate the virtual image of modality A based on the real image of modality B, discriminator is responsible for distinguishing the real image and the virtual image of modality A. The loss function of the general virtual staining framework proposed by the present invention is:

[0047] ,

[0048] wherein, and respectively represent the adversarial loss of the generator and the adversarial loss of the generator . is the cycle consistency loss, is the weight of the cycle consistency loss. Specifically, the adversarial loss is used to train the model by minimizing the adversarial loss function between the generator and the discriminator. Through the optimization of the adversarial loss, the generator and the discriminator perform iterative adversarial training, and finally the generator can generate images similar to the real images in the target domain, while the discriminator can accurately distinguish the generated images from the real images. The introduction of the cycle consistency loss is to preserve the original content of the image and prevent it from being damaged during the generation process, and it is achieved by mutually reconstructing the images. By optimizing the cycle consistency loss, the generator can learn the mapping relationship between modality A and modality B, ensuring the image consistency and integrity during the modality conversion process. The adversarial loss and the cycle consistency loss between modality A and modality B can be expressed as:

[0049] ,

[0050] ,

[0051] ,

[0052] where E(·) and p(·) respectively represent the expectation and the distribution probability of the data.

[0053] The last two terms in the total loss function of the present invention are the chroma and lightness loss, wherein and respectively correspond to the outputs of the real A input of the generator and the real B input of the generator . value(·) represents the chroma and lightness of the chroma and lightness channel, is the L1 regularization, is the weight of the chroma and lightness loss.

[0054] The present invention designs the chroma and lightness loss to solve the problems of insufficient performance and weak constraints of the network in processing complex texture domain transformation. By optimizing the generator and the generator in the HSV space value channel, the color lightness loss can maintain the structural integrity and avoid color inversion during the virtual staining process. Taking the generator as an example, the present invention inputs an A-modal image and outputs a B-modal virtual image. The present invention converts the RGB channels of the original image into HSV channels and extracts the color lightness channel for loss calculation. Since it can avoid color inversion during the color conversion process, the newly designed loss function of the present invention is very suitable for various pathological patterns and can be used as a general framework without further dedicated loss function design and parameter setting.

[0055] Referring to Figure 3 as shown, the present invention proposes a confidence-based large field of view imaging stitching method, which is as follows:

[0056] First, the present invention crops the large-scale high-resolution full field of view image in the real domain A into a small image sequence of every m pixels , and inputs the sequence into the trained network model to obtain the output sequence in the virtual domain B. Subsequently, according to the original position of the sequence in the full field of view input, each image of the sequence is filled into a zero-padding matrix to form a sequence . Next, according to the characteristics of the virtual staining sequence, a weight matrix with the same size as the full field of view input is designed. Finally, the Hadamard product (element-wise product) is used to multiply by the sequence , and it is normalized to obtain the final virtual full field of view output in domain B. The red, green, and blue channels of the large-scale image are processed independently. The overall function of the confidence stitching scheme can be expressed as:

[0057] ,

[0058] where, represents the full field of view output, represents the sequence of zero-padded small images after virtual staining, N represents the number of small images, represents the Hadamard product.

[0059] The weight matrix is centrosymmetric and the same on the red, green, and blue channels, and can be calculated by the following formula:

[0060] ,

[0061] where represents the value of the x-th row and y-th column in the upper left corner of the weight matrix in all three red, green, and blue channels, is the ceiling function, n is the size of the cropped small patch, and m is the padding interval between adjacent small patches. The settings of m and n should ensure that is an integer. The weight matrix is designed based on two phenomena observed after virtual staining of small patches in the present invention: (1) Compared with the center, the edges of small patches are more likely to generate artifacts, which usually lead to cell structure damage and staining errors. In other words, the virtual staining in the central region is more reliable and realistic than that in the edge region. (2) The contrast inconsistency between small patches results in a decrease in visual quality when stitching the patches together to form a WSI. Through this post-processing method based on the confidence principle, the artifacts generated by virtual staining stitching in large fields of view of microscopic imaging are successfully eliminated.

[0062] Taking autofluorescence - hematoxylin and eosin staining as an example, the embodiments of the present invention implement its virtual staining. The specific implementation steps include:

[0063] Step 1: Crop the autofluorescence and hematoxylin - eosin staining images of the training set and input them into the network for training.

[0064] Step 2: Crop the autofluorescence images in the test set and input them into the trained network model for testing, and output a sequence of hematoxylin - eosin virtual staining images.

[0065] Step 3: Perform confidence stitching on the sequence of hematoxylin - eosin virtual staining images to obtain the full - field virtual staining output result.

[0066] As Figure 4 shown, the hematoxylin - eosin virtual staining results of the present invention are presented; in this experiment, a 4608 * 4608 full - field image was selected for testing. The first column shows the autofluorescence imaging, the second column shows the hematoxylin - eosin real staining image, and the third column shows the method of the present invention. In the figure, the first two rows show the staining results of two cropped small - image samples, which respectively belong to the frames in the last two rows, and the last three rows show the full - field images of the samples. Compared with the prior art, the present invention has the following several significant advantages: (1) The prior art is prone to color distortion and color inversion during virtual staining; the present invention can well retain its color characteristics. (2) The prior art is not stable enough in retaining cell structure during staining and is prone to morphological distortion, while the present invention can well overcome this phenomenon. (3) The full - field output obtained by the prior art will show obvious block effects, affecting the overall output of the staining result, while the present invention can achieve contrast uniformity for the entire image at the full - field scale.

[0067] An embodiment of the present invention proposes a microscopic imaging virtual staining method based on a cyclic adversarial generation network and is enhanced by a newly designed chromatic lightness loss. Among them, the chromatic lightness loss ensures an accurate color and structural mapping relationship between the input and output images during the color conversion process. In addition, when dealing with large-field-of-view microscopic imaging staining, processes such as patch segmentation, virtual staining, and final stitching are required, and existing methods usually face the problem of block effects between patches. To solve this problem, the present invention adopts a confidence-based and plug-and-play post-processing method, which effectively improves the continuity of the stained patches and eliminates stitching artifacts. The present invention conducts multiple experiments to prove the robustness and efficiency of the method of the present invention through virtual staining between three different modes. All in all, the present invention has excellent generality and large-field-of-view imaging capabilities, and it has important reference value for future applications of microscopic imaging and pathological analysis.

[0068] The present invention provides a multi-modal general large-field-of-view virtual staining post-processing method. There are many methods and ways to specifically implement this technical solution. The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and retouches can be made, and these improvements and retouches should also be regarded as the protection scope of the present invention. Each component not clearly defined in this embodiment can be implemented by existing technologies.

Claims

1. A multi-modal general large-field virtual staining post-processing method, characterized in that: The following steps are involved: Step 1, establish a recurrent adversarial generative network, convert the red, green and blue RGB color space into hue H, saturation S, and color brightness V color space, and extract the color brightness channel; Step 2, establish a confidence-based large field of view stitching module to process large field of view microscopic images and eliminate artifacts in the boundary inconsistency of small image stitching process; In step 1, unpaired image datasets of modality A and modality B are set, and the cyclic adversarial generative network includes adversarial loss, cycle consistency loss, and color brightness loss; The adversarial loss is used to train the generator to generate virtual images similar to real images. The mathematical expression is: ， ， in and Respectively represent the generator Adversarial loss and generator The adversarial loss, It means randomly sampling an image b from the image distribution of modality B. Indicates randomly sampling an image from the image distribution of modality A , E(·) and p(·) represent the expected and distribution probability of the data, respectively. and represents two generators, and Represents two discriminators; generator Taking real images from modality A as input and generating virtual images from modality B, the discriminator Used to distinguish between real and virtual images from B modality; Generator Generate virtual images of modality A based on real images of modality B, discriminator Used to distinguish between real and virtual images of A modality; The cycle consistency loss is used to maintain the integrity of the original image content, and the mathematical expression is: ， in is L1 regularization; is the cycle consistency loss; represents the expectation of A modality image, represents the expectation of B-mode image; The color-lightness loss is used to convert the red, green, and blue (RGB) color space into a hue H, saturation S, and color-lightness V color space, and extract the color-lightness channel to perform color-lightness constraints to ensure the accuracy of color mapping. The mathematical expression is: ， in, and Corresponding to the generator A modality for real image input output and generator The output of the B-mode real image input; value(·) represents the color brightness of the color brightness channel, is the color brightness loss, is the weight of color brightness loss; In step 2, the confidence-based large field of view stitching module crops the large-scale high-resolution full-field of view image of the A-modality real image into a small image sequence of m pixels per pixel. , convert the sequence Input into the network model trained in step 1 to obtain the output sequence of B-mode virtual images ; According to the sequence The original position in the full field input, the sequence Each image of is filled into a zero-filled matrix to form a sequence ; Design a weight matrix with the same size as the full field of view input , use the Hadamard product to convert the weight matrix Multiplying a sequence , and normalize to obtain the final virtual full field of view output of B mode , the formula is: ， in, represents the i-th small image sequence with zero padding after virtual dyeing, N represents the number of small images, Represents the Hadamard product.

2. The method according to claim 1, characterized in that Step 1 also includes: using the following loss function : ， The network model is trained based on the established cyclic adversarial generation network.

3. The method according to claim 2, characterized in that In step 2, the weight matrix is ​​calculated using the following formula: ， in Represents the weight matrix of all red, green and blue RGB channels The value of the top left corner in row x and column y, is a ceiling function, n is the size of the cropped block, and m is the padding interval between adjacent blocks.

4. The method according to claim 3, characterized in that In step 2, is an integer.

5. The method according to claim 4, characterized in that The method is plug-and-play and suitable for processing large-field-of-view super-resolution images.

6. The method according to claim 5, characterized in that The method is used for virtual staining between different types of pathological modalities, including autofluorescence, hematoxylin and eosin staining H&E and immunohistochemistry staining IHC.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

8. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 6 are executed.

Images