Multi-mode general large-view-field virtual dyeing post-processing method

Abstract

The invention provides a multi-mode general large-view-field virtual dyeing post-processing method, which comprises the following steps of: 1, establishing a cyclic generative adversarial network, converting a red, green and blue RGB color space into a color space of hue H, saturation S and color lightness V, and extracting a color lightness channel; step 2, establishing a large-view-field splicing module based on confidence, processing a large-view-field microscopic image, and eliminating artifacts in boundary inconsistency in a small image splicing process; according to the method, the correct color mapping relation and image content retention in the modal conversion process can be ensured, meanwhile, artifacts generated in image sequence splicing are eliminated through a plug-and-play post-processing method, and the frame can process various cross-modal conversions, is very suitable for processing large-scale and high-resolution images, and has a wide application prospect. The problem of boundary inconsistency encountered in many existing methods is effectively solved.

Description

Technical Field

[0001] The invention relates to the field of microscopic histology virtual staining, and in particular to a multi-modal universal large-field virtual staining post-processing method. Background Art

[0002] In disease diagnosis and pathological research, tissue staining plays an important role as a basic technology. By using specific markers and labeling according to the inherent characteristics of biological elements, tissue staining can visualize different tissues and cell structures, thus providing support for subsequent pathological analysis and disease diagnosis.

[0003] The application of a variety of staining techniques highlights diverse biological characteristics. Hematoxylin-eosin staining is an important method in pathological sections, providing rich information and widely used in tumor diagnosis. Immunohistochemical staining is a technique that combines immunology with traditional histology. Through the specific binding interaction between antigens and antibodies, it can detect and locate specific chemicals in tissues and cells. For example, proliferating cell antigen immunohistochemical staining is often used to evaluate cell proliferation activity, helping pathologists and researchers to quantitatively assess the extent of tumor proliferation. However, these standard tissue staining procedures are usually performed in pathology laboratories and involve multiple sample preparation processes, which are laborious and time-consuming.

[0004] Deep learning has made significant progress in the field of optical microscopy, including the application of virtual tissue staining. Virtual staining bypasses the tedious sample preparation, specialized laboratory equipment, and technician requirements of traditional chemical tissue staining. By leveraging deep learning to generate digital tissue staining, virtual staining provides an efficient, accurate, and cost-effective alternative. Deep learning-based virtual staining methods are mainly divided into two categories: supervised learning and unsupervised learning. Supervised learning methods rely on a large number of high-quality paired images; however, in optical microscopy applications, it is often challenging to provide a complete one-to-one correspondence between two data modalities. Therefore, unsupervised learning methods based on recurrent generative adversarial networks are favored by relevant researchers because they have practical value in clinical imaging and scientific research. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a multi-modal universal large-field virtual staining post-processing method in view of the deficiencies in the prior art, aiming to provide a universal virtual staining framework that can achieve accurate color mapping between different pathological modalities.

[0006] The method of the present invention comprises the following steps: Step 1, establish a recurrent adversarial generative network, convert the red, green and blue RGB color space into hue H, saturation S, and color brightness V color space, and extract the color brightness channel; Step 2, establish a confidence-based large field of view stitching module to process large field of view microscopic images and eliminate artifacts in the boundary inconsistency of small image stitching process; In step 1, unpaired image datasets of modality A and modality B are set, and the cyclic adversarial generative network includes adversarial loss, cycle consistency loss, and color brightness loss; The adversarial loss is used to train the generator to generate virtual images similar to real images. The mathematical expression is: , , in and Respectively represent the generator Adversarial loss and generator The adversarial loss, represents randomly sampling an image b from the image distribution of modality B, Indicates randomly sampling an image from the image distribution of modality A , E(·) and p(·) represent the expected and distribution probabilities of the data, respectively. and represents two generators, and Represents two discriminators; generator Taking real images from modality A as input and generating virtual images from modality B, the discriminator Used to distinguish between real and virtual images from B modality; Generator Generate virtual images of modality A based on real images of modality B, discriminator Used to distinguish between real and virtual images of A modality; The cycle consistency loss is used to maintain the integrity of the original image content, and the mathematical expression is: , in is L1 regularization; is the cycle consistency loss; represents the expectation of A modality image, represents the expectation of B-mode image; The color-lightness loss is used to convert the red, green, and blue (RGB) color space into a hue H, saturation S, and color-lightness V color space, and extract the color-lightness channel to perform color-lightness constraints to ensure the accuracy of color mapping. The mathematical expression is: , in, and Corresponding to the generator A modality for real image input output and generator The output of the B-mode real image input; value(·) represents the color brightness of the color brightness channel, is the color brightness loss, is the weight of color brightness loss; In step 2, the confidence-based large field of view stitching module crops the large-scale high-resolution full-field of view image of the A-modality real image into a small image sequence of m pixels per pixel. , convert the sequence Input into the network model trained in step 1 to obtain the output sequence of B-mode virtual images ; According to the sequence The original position in the full field input, the sequence Each image of is filled into a zero-filled matrix to form a sequence ; Design a weight matrix with the same size as the full field of view input , use the Hadamard product to convert the weight matrix Multiplying a sequence , and normalize to obtain the final virtual full field of view output of B mode , the formula is: , in, represents the i-th small image sequence with zero padding after virtual dyeing, N represents the number of small images, represents the Hadamard product.

[0007] Step 1 also includes: using the following loss function : ;

[0008] In step 1, the network model is trained based on the established cyclic adversarial generation network.

[0009] In step 2, the weight matrix is ​​calculated using the following formula: , in Represents the weight matrix of all red, green and blue RGB channels The value of the top left corner in row x and column y, is a ceiling function, n is the size of the cropped block, and m is the padding interval between adjacent blocks.

[0010] In step 2, is an integer.

[0011] The method is plug-and-play and suitable for processing large-field-of-view super-resolution images.

[0012] The method is used for virtual staining between different types of pathological modalities without adjusting hyperparameters, including autofluorescence, hematoxylin and eosin staining H&E, and immunohistochemistry staining IHC.

[0013] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0014] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0015] The present invention also provides a post-processing method based on confidence stitching for large-field super-resolution image processing. The method designs a weight matrix to tile the patch to the whole for large-field imaging stitching.

[0016] The present invention solves the color distortion problem in virtual staining by optimizing the recurrent adversarial generative network, ensuring the correct color mapping relationship and image content retention during the modality conversion process. At the same time, it is very suitable for processing large-scale and high-resolution images, and solves the boundary inconsistency problem encountered in many existing methods. The following beneficial effects are included: (1) Stability of multi-modal conversion: The present invention can be widely applied to virtual staining under different modalities, and provides a general and effective solution for different staining tasks, which can meet the task requirements of various histological virtual staining tasks.

[0017] (2) Practicality of large-field super-resolution images: Through the present invention, for super-resolution large-field microscopic imaging, artifacts generated by patch stitching can be eliminated in a plug-and-play manner without the need for training, thereby facilitating subsequent downstream analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0019] Figure 1 It is a structural block diagram of the device of the present invention.

[0020] Figure 2 Schematic diagram of the training network module structure process in an embodiment of the present invention.

[0021] Figure 3 Schematic diagram of a large field of view image stitching process in an embodiment of the present invention.

[0022] Figure 4This is a result diagram of H&E staining of the autofluorescence image in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiment of the present invention takes the virtual staining of autofluorescence-hematoxylin-eosin staining as an example. Figure 1 As shown, based on HSV space conversion, the embodiment of the present invention proposes a color-value mapping virtual coloring method based on a recurrent adversarial generative network. The present invention designs a new constraint by extracting the color-value channel of the HSV space and modifies the loss function of the recurrent adversarial generative network accordingly. The main structure of the framework of the present invention is shown in FIG. Figure 1 As shown, the present invention replaces the identity loss in the original network with the color-lightness loss to ensure the correct color mapping relationship between the A mode and the B mode. It is worth noting that, in addition to calculating the color-lightness loss, the main process of the method of the present invention is performed separately in the RGB channels.

[0024] Reference Figure 2 As shown, the method of the present invention establishes a training network module for unsupervised training of unpaired two-modality image sequences. The specific implementation method is as follows: Based on the original recurrent adversarial generative network, the present invention divides unpaired images into A-mode and B-mode, so that the network can be trained unsupervised. The whole architecture consists of two generators and and two discriminators and . Generator Takes as input a real image from modality A and generates a virtual image from modality B, while the discriminator Then, we distinguish between real images and virtual images from B modalities. Each generator has a corresponding discriminator that tries to distinguish its generated images from real images. Similarly, the generator Generate virtual images of modality A based on real images of modality B, discriminator Responsible for distinguishing the real image and virtual image of modality A. The loss function of the general virtual coloring framework proposed in this invention is: , in, and Respectively represent the generator Adversarial loss and generator The adversarial loss, is the cycle consistency loss, Weight of the cycle consistency loss. Specifically, the adversarial loss is used to train the model by minimizing the adversarial loss function between the generator and the discriminator. Through the optimization of the adversarial loss, the generator and the discriminator are iteratively adversarially trained, and finally the generator is able to generate images similar to real images in the target domain, while the discriminator can accurately distinguish between the generated images and the real images. The cycle consistency loss is introduced to preserve the original content of the image and prevent it from being destroyed during the generation process, and it is achieved by reconstructing the images from each other. By optimizing the cycle consistency loss, the generator can learn the mapping relationship between the A modality and the B modality, ensuring the consistency and integrity of the image during the modality conversion process. The adversarial loss and cycle consistency loss between the A modality and the B modality can be expressed as: , , , Where E(·) and p(·) represent the expectation and distribution probability of the data, respectively.

[0025] The last two terms in the total loss function of the present invention are color brightness losses, where and Corresponding to the generator The real A input and generator The output of the real B input. value(·) represents the color brightness of the color brightness channel. is L1 regularization, is the weight of color brightness loss.

[0026] The present invention designs a color brightness loss to solve the problem of insufficient performance and weak constraints when the network processes complex texture domain transformations. By optimizing the generator in the HSV space value channel and generator , color brightness loss can maintain structural integrity and avoid color inversion during virtual dyeing. For example, the present invention inputs an A-modality image and outputs a B-modality virtual image. The present invention converts the RGB channel of the original image into an HSV channel and extracts the color-brightness channel for loss calculation. Since it can avoid color inversion during the color conversion process, the newly designed loss function of the present invention is very suitable for various pathological patterns and can be used as a general framework without the need for further specialized loss function design and parameter setting.

[0027] Reference Figure 3 As shown, the present invention proposes a large field of view imaging stitching method based on confidence, which is as follows: First, the present invention crops the large-scale high-resolution full-field image of the real domain A into a sequence of small images with m pixels per image. , convert the sequence Input into the trained network model to obtain the output sequence of virtual domain B Then, according to the sequence The original position in the full field input, the sequence Each image of is filled into a zero-filled matrix to form a sequence Next, based on the characteristics of the virtual staining sequence, a weight matrix with the same size as the full field input was designed. Finally, we use the Hadamard product (element-wise product) to convert Multiplying a sequence , and normalize them to obtain the final virtual full-field-of-view output in domain B. The red, green, and blue channels of the large-scale image are processed independently. The overall function of the confidence stitching scheme can be expressed as: , in, Indicates full field of view output, represents a sequence of small images filled with zero after virtual dyeing, N represents the number of small images, represents the Hadamard product.

[0028] Weight Matrix The red, green and blue channels are all centrally symmetric and identical, and can be calculated using the following formula: , in Represents the weight matrix in all three red, green and blue channels The value of the top left corner in row x and column y, is the ceiling function, n is the size of the cropped patch, and m is the padding interval between adjacent patches. The settings of m and n should ensure is an integer. The weight matrix The design of the proposed method is based on two phenomena observed by the present invention after virtual staining of small patches: (1) Compared with the center, the edges of small patches are more prone to artifacts, which often lead to cell structure damage and staining errors. In other words, virtual staining in the central area is more reliable and realistic than that in the edge area. (2) The contrast between small patches is inconsistent, resulting in a decrease in visual quality when the patches are stitched together to form a WSI. Through this post-processing method based on the confidence principle, the artifacts generated by virtual staining stitching in a large field of view of microscopic imaging are successfully eliminated.

[0029] The embodiment of the present invention takes autofluorescence-hematoxylin-eosin staining as an example to implement its virtual staining, and the specific implementation steps include: Step 1: crop the autofluorescence and hematoxylin-eosin staining images of the training set and input them into the network for training.

[0030] Step 2: crop the autofluorescence images in the test set, input them into the trained network model for testing, and output a sequence of hematoxylin-eosin virtual staining images.

[0031] Step 3: Confidence stitching of the hematoxylin-eosin virtual staining image sequence is performed to obtain the full-field virtual staining output result.

[0032] like Figure 4 As shown, the virtual staining result of hematoxylin-eosin of the present invention is shown; the experiment selected a 4608*4608 full-field image for testing, the first column shows the autofluorescence imaging, the second column shows the real staining image of hematoxylin-eosin, and the third column shows the method of the present invention. In the figure, the first two rows show the staining results of two cropped small sample images, which belong to the boxes in the last two rows respectively, and the last three rows show the full-field image of the sample. Compared with the prior art, the present invention has the following significant advantages: (1) The prior art is prone to color distortion and color inversion when performing virtual staining; the present invention can well retain its color characteristics. (2) The prior art is not stable enough in retaining the cell structure during staining, and morphological distortion is prone to occur. The present invention can well overcome this phenomenon. (3) The full-field output obtained by the prior art will have an obvious block effect, which affects the overall output of the staining result. The present invention can achieve uniform contrast of the entire image at the full-field scale.

[0033] The embodiment of the present invention proposes a microscopic imaging virtual staining method based on a recurrent adversarial generative network, and is enhanced by a newly designed color brightness loss. Among them, the color brightness loss ensures the accurate color and structure mapping relationship between the input and output images during the color conversion process. In addition, when processing large-field microscopic imaging staining, it is necessary to perform patch segmentation, virtual staining and final splicing processes, and existing methods usually face the problem of block effect between patches. In order to solve this problem, the present invention adopts a confidence-based and plug-and-play post-processing method, which effectively improves the continuity of the stained patches and eliminates splicing artifacts. The present invention has conducted multiple experiments to demonstrate the robustness and efficiency of the method of the present invention through virtual staining between three different modes. In summary, the present invention has excellent versatility and large-field imaging capabilities, and it can have important reference value for future applications in microscopic imaging and pathological analysis.

[0034] The present invention provides a multi-modal general large-field virtual staining post-processing method. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A multi-modal general large-field virtual staining post-processing method, characterized in that: The following steps are involved: Step 1, establish a recurrent adversarial generative network, convert the red, green and blue RGB color space into hue H, saturation S, and color brightness V color space, and extract the color brightness channel; Step 2, establish a confidence-based large field of view stitching module to process large field of view microscopic images and eliminate artifacts in the boundary inconsistency of small image stitching process; In step 1, unpaired image datasets of modality A and modality B are set, and the cyclic adversarial generative network includes adversarial loss, cycle consistency loss, and color brightness loss; The adversarial loss is used to train the generator to generate virtual images similar to real images. The mathematical expression is: ， ， in and Respectively represent the generator Adversarial loss and generator The adversarial loss, It means randomly sampling an image b from the image distribution of modality B. Indicates randomly sampling an image from the image distribution of modality A , E(·) and p(·) represent the expected and distribution probability of the data, respectively. and represents two generators, and Represents two discriminators; generator Taking real images from modality A as input and generating virtual images from modality B, the discriminator Used to distinguish between real and virtual images from B modality; Generator Generate virtual images of modality A based on real images of modality B, discriminator Used to distinguish between real and virtual images of A modality; The cycle consistency loss is used to maintain the integrity of the original image content, and the mathematical expression is: ， in is L1 regularization; is the cycle consistency loss; represents the expectation of A modality image, represents the expectation of B-mode image; The color-lightness loss is used to convert the red, green, and blue (RGB) color space into a hue H, saturation S, and color-lightness V color space, and extract the color-lightness channel to perform color-lightness constraints to ensure the accuracy of color mapping. The mathematical expression is: ， in, and Corresponding to the generator A modality for real image input output and generator The output of the B-mode real image input; value(·) represents the color brightness of the color brightness channel, is the color brightness loss, is the weight of color brightness loss; In step 2, the confidence-based large field of view stitching module crops the large-scale high-resolution full-field of view image of the A-modality real image into a small image sequence of m pixels per pixel. , convert the sequence Input into the network model trained in step 1 to obtain the output sequence of B-mode virtual images ; According to the sequence The original position in the full field input, the sequence Each image of is filled into a zero-filled matrix to form a sequence ; Design a weight matrix with the same size as the full field of view input , use the Hadamard product to convert the weight matrix Multiplying a sequence , and normalize to obtain the final virtual full field of view output of B mode , the formula is: ， in, represents the i-th small image sequence filled with zero after virtual dyeing, N represents the number of small images, Represents the Hadamard product.

2. The method according to claim 1, characterized in that Step 1 also includes: using the following loss function : ， The network model is trained based on the established cyclic adversarial generation network.

3. The method according to claim 2, characterized in that In step 2, the weight matrix is ​​calculated using the following formula: ， in Represents the weight matrix of all red, green and blue RGB channels The value of the top left corner in row x and column y, is a ceiling function, n is the size of the cropped block, and m is the padding interval between adjacent blocks.

4. The method according to claim 3, characterized in that In step 2, is an integer.

5. The method according to claim 4, characterized in that The method is plug-and-play and suitable for processing large-field-of-view super-resolution images.

6. The method according to claim 5, characterized in that The method is used for virtual staining between different types of pathological modalities, including autofluorescence, hematoxylin and eosin staining H&E and immunohistochemistry staining IHC.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

8. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 6 are executed.

Images