A Fourier stacking microscopic color reconstruction method, device and equipment

Abstract

The present invention discloses a Fourier ptychographic microscopy color reconstruction method, apparatus and device, relating to the field of microscopic imaging, and is used to solve the problems of poor reconstruction efficiency and poor image quality in the existing Fourier ptychographic microscopy imaging. It includes: obtaining a high-resolution data set; constructing a Fourier ptychographic microscopy color reconstruction network based on a self-supervised generative adversarial network; setting a loss function for training the Fourier ptychographic microscopy color reconstruction network; using the data set and the loss function to train the Fourier ptychographic microscopy color reconstruction network to obtain a trained Fourier ptychographic microscopy color reconstruction network; inputting the data to be tested into the trained Fourier ptychographic microscopy color reconstruction network to obtain a target FPM color image. The present invention can improve the quality of the FPM color image, reduce the FPM acquisition time, and improve the reconstruction efficiency.

Description

Technical Field

[0001] The present invention relates to the field of microscopic imaging technology, and in particular to a Fourier stack microscopic color reconstruction method, device and equipment. Background Art

[0002] The ability of optical microscopy to perform minimally invasive, high-resolution imaging is fundamental to human understanding of biological systems and processes. In clinical pathology, the analysis of high-resolution color pathology images using optical microscopy remains the gold standard for diagnosing diseases ranging from cancer to blood-borne infections. Large-field-of-view, high-resolution color imaging has long been the imaging goal pursued in the field of digital pathology, but there are insurmountable limitations between high resolution and large field of view in traditional imaging systems.

[0003] The introduction of Fourier ptychographic microscopy (FPM) technology has transferred the challenges of large field of view and high resolution imaging from the field of physical limitations to the field of computation. It has become an important research topic in the field of computational imaging because of its advantages of large field of view, high resolution and quantitative phase imaging.

[0004] However, the application of FPM in pathology still faces some challenges. On the one hand, the time efficiency of FPM acquisition and reconstruction is relatively poor. FPM usually requires continuous acquisition using red, green, and blue illumination to form hundreds of low-resolution (LR) images to obtain better color reconstruction, resulting in slower acquisition and reconstruction speeds. On the other hand, due to coherence artifacts caused by light interference and reconstruction errors, FPM color reconstructed images are usually of poor quality. In addition, due to the influence of factors such as acquisition personnel, technology, and external environment, the color reconstruction results of the same pathological section will have significant visual differences. In biological and clinical medical applications, the color information of the sample after staining with reagents helps the observer to quickly locate the area of interest and interpret relevant information about tissues and cells.

[0005] Therefore, there is an urgent need to provide a more reliable Fourier stack microscopy color reconstruction solution. Summary of the invention

[0006] The object of the present invention is to provide a Fourier stack microscopic color reconstruction method, device and equipment, which are used to solve the problems of poor reconstruction efficiency and poor reconstructed image quality in the prior art of Fourier stack microscopic imaging.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a method for Fourier stacking microscopic color reconstruction, the method comprising:

[0009] Obtain a high-resolution data set; the data set is a high-resolution single-channel grayscale data set generated by simulating using a publicly available data set as the ground truth through a Fourier ptychography imaging model;

[0010] Construct a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure;

[0011] Set the loss function for training the Fourier ptychography microscopic color reconstruction network;

[0012] Use the data set and the loss function to train the Fourier ptychography microscopic color reconstruction network to obtain a trained Fourier ptychography microscopic color reconstruction network;

[0013] Input the data to be tested into the trained Fourier ptychography microscopic color reconstruction network to obtain a target FPM color image.

[0014] Compared with the prior art, a Fourier ptychography microscopic color reconstruction method provided by the present invention. By obtaining a high-resolution data set; constructing a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; setting the loss function for training the Fourier ptychography microscopic color reconstruction network; using the data set and the loss function to train the Fourier ptychography microscopic color reconstruction network to obtain a trained Fourier ptychography microscopic color reconstruction network; inputting the data to be tested into the trained Fourier ptychography microscopic color reconstruction network to obtain a target FPM color image. The self-supervised generative adversarial network in the present invention includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure; the network adaptively obtains efficient cross-channel interaction information in a lightweight manner through an efficient channel residual (ECR) block; the generator network outputs the reconstructed FPM color image by gradually performing the encoding, RCR feature transformation, and decoding processes, improving the quality of the stained image in an extremely lightweight manner while obtaining efficient cross-channel interaction information; while effectively staining the FPM image, the high-frequency information of the input image is maximally retained, having the best visual effect. And the Fourier ptychography microscopic color reconstruction method based on a self-supervised generative adversarial network can reduce coherent artifacts, further improve the quality of the FPM color image, reduce the FPM acquisition time, and improve the reconstruction efficiency.

[0015] In a second aspect, the present invention provides a Fourier ptychography microscopic color reconstruction device, the device includes:

[0016] A dataset acquisition module for acquiring a high-resolution dataset; the dataset is a high-resolution single-channel grayscale dataset generated by simulating using a publicly available dataset as the ground truth through a Fourier ptychography imaging model.

[0017] A Fourier ptychography microscopic color reconstruction network construction module for constructing a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure.

[0018] A loss function setting module for setting a loss function for training the Fourier ptychography microscopic color reconstruction network.

[0019] A Fourier ptychography microscopic color reconstruction network training module for training the Fourier ptychography microscopic color reconstruction network using the dataset and the loss function to obtain a trained Fourier ptychography microscopic color reconstruction network.

[0020] A target FPM color image prediction module for inputting test data into the trained Fourier ptychography microscopic color reconstruction network to obtain a target FPM color image.

[0021] In a third aspect, the present invention provides a Fourier ptychography microscopic color reconstruction device, the device includes:

[0022] A communication unit / communication interface for acquiring a high-resolution dataset; the dataset is a high-resolution single-channel grayscale dataset generated by simulating using a publicly available dataset as the ground truth through a Fourier ptychography imaging model.

[0023] A processing unit / processor for constructing a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure.

[0024] Setting a loss function for training the Fourier ptychography microscopic color reconstruction network.

[0025] Training the Fourier ptychography microscopic color reconstruction network using the dataset and the loss function to obtain a trained Fourier ptychography microscopic color reconstruction network.

[0026] Inputting test data into the trained Fourier ptychography microscopic color reconstruction network to obtain a target FPM color image.

[0027] Fourthly, the present invention provides a computer storage medium, in which instructions are stored, and when the instructions are run, the above-mentioned Fourier stacking microscopic color reconstruction method is implemented.

[0028] The technical effects achieved by the device-related solution provided in the second aspect, the equipment-related solution provided in the third aspect, and the computer storage medium solution provided in the fourth aspect are the same as those of the method-related solution provided in the first aspect, and will not be elaborated here. Description of the Drawings

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0030] Figure 1 It is a schematic flow chart of a Fourier stacking microscopic color reconstruction method provided by the present invention;

[0031] Figure 2 It is a schematic diagram of the overall network structure of the self-supervised generative adversarial network provided by the present invention;

[0032] Figure 3 It is a schematic diagram of the network structure of the generator provided by the present invention;

[0033] Figure 4 It is a schematic diagram of the structure of a Fourier stacking microscopic color reconstruction device provided by the present invention;

[0034] Figure 5 It is a schematic diagram of the structure of a Fourier stacking microscopic color reconstruction equipment provided by the present invention. Detailed Embodiments

[0035] In order to facilitate a clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and roles. For example, the first threshold and the second threshold are only used to distinguish different thresholds and do not limit their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first" and "second" do not necessarily mean different.

[0036] It should be noted that in the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific way.

[0037] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or similar expressions refer to any combination of these items, including any combination of single item(s) or plural item(s). For example, at least one (item) of a, b, or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b, and c, where a, b, and c can be single or multiple.

[0038] Based on the generative adversarial network, the present invention combines the virtual staining method with FPM color reconstruction, and proposes a Fourier ptychographic microscopy color reconstruction method based on self-supervised generative adversarial network to reduce coherent artifacts, further improve the quality of FPM color images, reduce the 2 / 3 FPM acquisition time, and improve the time efficiency. This virtual staining method can avoid the differences in staining results caused by the randomness of personnel and technology, has the ability of standardized staining, and has wide applications in future digital pathology.

[0039] The overall network structure includes a generator structure and a discriminator structure. The network based on encoding-decoding and efficient channel residual (ECR) modules is used as the generator, and the network based on five-layer convolution is the discriminator structure. The present invention designs a generator based on ECR to adaptively obtain efficient cross-channel interaction information in a lightweight manner, and introduces content consistency loss to learn the high-frequency information of the image and improve the quality of the stained image. In addition, the effectiveness of this method is verified through specific objective indicators and visual evaluations.

[0040] Next, the solution provided in the embodiments of this specification will be described in conjunction with the accompanying drawings:

[0041] As Figure 1 shown, the process may include the following steps:

[0042] Step 110: Obtain a high-resolution dataset; the dataset is a high-resolution single-channel grayscale dataset generated by simulating using a publicly available dataset as the ground truth through the Fourier ptychography imaging model.

[0043] Constructing a large-scale dataset containing a training set and a test set is required for training a neural network. However, directly collecting high-resolution color images from FPM has a series of disadvantages:

[0044] (1) The amount of raw data collected is too large to construct a dataset of sufficient size;

[0045] (2) The ground truth of the dataset is obtained through traditional reconstruction methods, so the effect of network reconstruction is limited by traditional methods;

[0046] (3) The dataset is bound to a specific system and is difficult to be flexibly applied to other systems.

[0047] To avoid the above problems, the present invention directly uses a publicly available dataset as the ground truth and generates corresponding single-channel high-resolution input data through a Fourier ptychography imaging model simulation. The simulated dataset includes 4363 single-channel FPM high-resolution grayscale images and 4362 unpaired three-channel color FPM images, among which 3208 single-channel grayscale images and 3279 color images are used for network training, and the rest are used for network testing.

[0048] Step 120: Construct a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure.

[0049] The self-supervised generative adversarial network consists of a generator network and a discriminator network. The generator network is responsible for converting single-channel FPM grayscale images (input domain X) into high-resolution FPM color images (target domain Y), while the discriminator network determines whether the colored input image is a reconstructed image of the generator network or a real image.

[0050] The generator network G includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure, and outputs a reconstructed FPM high-resolution color image by gradually performing the encoding, ECR conversion, and decoding processes.

[0051] The discriminator D acts as a competitor competing with the generator during the training process. The PatchGAN (Markov discriminator) is used as the discriminator D, and the receptive field of the PatchGAN is set to 70×70, which makes the PatchGAN faster and guides the generator G to produce more realistic results.

[0052] Step 130: Set the loss function for training the Fourier ptychography microscopic color reconstruction network.

[0053] In a neural network, the loss function serves as the objective function to evaluate the difference between the model prediction and the target value (label). In the technical solution provided by the present invention, content-consistency loss, identity loss, adversarial loss, and PatchNCE loss are introduced for joint training as the overall objective function of the network.

[0054] The content-consistency loss learns the high-frequency information of the image to avoid color inversion and feature distortion between the input and output images.

[0055] The identity loss prevents the generator G from making unnecessary changes and encourages the input-output mapping to maintain color and brightness composition.

[0056] The adversarial loss encourages the output to be visually as similar as possible to the images in the target domain Y.

[0057] The objective of the PatchNCE loss is to maximize the mutual information between the corresponding patches of the input and output images.

[0058] Step 140: Use the dataset and the loss function to train the Fourier ptychographic microscopy color reconstruction network to obtain a trained Fourier ptychographic microscopy color reconstruction network.

[0059] Using a simulation dataset and a set loss function, the effectiveness of self-supervised generative adversarial network virtual staining was verified from the specific indicators and visual analysis of the FPM color reconstruction results. The single-channel grayscale images in the test set were input into the trained self-supervised generative adversarial network model, and the corresponding high-resolution FPM color images were output. The test passed, and the Fourier ptychographic microscopy color reconstruction network was trained successfully.

[0060] Step 150: Input the data to be tested into the trained Fourier ptychographic microscopy color reconstruction network to obtain the target FPM color image.

[0061] In an actual application scenario, the data to be tested can be a high-resolution single-channel grayscale image. Inputting the data to be tested into the trained Fourier ptychographic microscopy color reconstruction network can obtain the target FPM color image.

[0062] Figure 1The method in [description] includes: obtaining a high-resolution dataset; constructing a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; setting a loss function for training the Fourier ptychography microscopic color reconstruction network; using the dataset and the loss function to train the Fourier ptychography microscopic color reconstruction network to obtain a trained Fourier ptychography microscopic color reconstruction network; inputting the data to be tested into the trained Fourier ptychography microscopic color reconstruction network to obtain a target FPM color image. The self-supervised generative adversarial network in the present invention includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure; the network adaptively obtains efficient cross-channel interaction information in a lightweight manner through an efficient channel residual (ECR) block; the generator network outputs a reconstructed FPM color image by gradually performing an encoding, RCR feature transformation, and decoding process, improving the quality of the stained image in an extremely lightweight manner while obtaining efficient cross-channel interaction information; while effectively staining the FPM image, maximizing the retention of the high-frequency information of the input image, with the best visual effect. And the Fourier ptychography microscopic color reconstruction method based on the self-supervised generative adversarial network can reduce coherent artifacts, further improve the quality of the FPM color image, reduce the FPM acquisition time, and improve the reconstruction efficiency.

[0063] Based on Figure 1 the method described above, some specific embodiments of this method are also provided in the embodiments of this specification, which will be described below.

[0064] The construction of the Fourier ptychography microscopic color reconstruction network based on the self-supervised generative adversarial network may specifically include:

[0065] Constructing a generator network including an encoding structure and a decoding structure; the encoding structure includes a downsampling module; the downsampling module includes a convolutional layer, an instance normalization layer, and an activation function layer; the generator network uses a preset convolutional layer, instance normalization layer, and activation function layer to initially extract shallow features from the input single-channel image; extracts low-frequency features from the input single-channel image through continuous downsampling modules and inputs the low-frequency features into the ECR module; the decoding structure includes an upsampling module; the upsampling module includes a transposed convolutional layer, an instance normalization layer, and an activation function layer; the feature map output by the ECR module reconstructs the details and resolution of the image through continuous upsampling modules and outputs a color image through a convolutional layer and an activation function layer;

[0066] PatchGAN is used as the discriminator in the discriminator network;

[0067] Based on the generator network and the discriminator network, a Fourier ptychography microscopic color reconstruction network based on the self-supervised generative adversarial network is constructed.

[0068] Furthermore, the ECR module includes an original residual block and an ECA module; the original residual block includes an input convolutional layer, an output convolutional layer, and an activation function layer; the ECA module includes a global average pooling layer, a one-dimensional convolutional layer with an adaptive convolutional kernel, and a Sigmoid activation layer, and an identity mapping is added in the ECR module.

[0069] The overall network structure of the above self-supervised generative adversarial network can be combined with Figure 2 for illustration:

[0070] The Fourier ptychography microscopic color reconstruction method provided in this specification can be a self-supervised generative adversarial network Fourier ptychography microscopic color reconstruction method based on the idea of virtual staining, abbreviated as ECSGAN. ECSGAN consists of a generator network G and a discriminator network D. G(target domain Y), and D is trained to distinguish real samples in domain Y and color images generated in domain X. The overall network structure is as Figure 2 shown. The generator G is divided into two parts, an encoder Genc and a decoder Gdec. Among them, Genc and a two-layer MLP projection head (Hl) are used as the shared embeddings of domain X and domain Y. Figure 2 The content consistency loss (the line labeled 1), the identity loss (the line labeled 2), the GAN loss (the line labeled 3), and the patch-based PatchNCE loss (the line labeled 4) are described in

[0071] The PatchNCE loss helps the generated virtual image to be similar to the real input image x, but different from other color patches of x. To ensure that the finally reconstructed color image performs well in subjective evaluation and objective metrics, the above losses are jointly trained to further improve the detailed information of color reconstruction.

[0072] Furthermore, the structure of the generator network can be combined with Figure 3 for illustration: As Figure 3 shown, the generator G includes 3 downsampling modules, 9 ECR blocks, 2 upsampling modules, and 1 convolutional layer. It is divided into two parts, an encoder Genc and a decoder Gdec, and the specific structure is as Figure 3 shown. The network adaptively obtains efficient cross-channel interaction information in a lightweight manner through the efficient channel residual (ECR) block. The generator network outputs the reconstructed FPM color image by gradually performing encoding, RCR feature transformation, and decoding processes.

[0073] The encoding structure is mainly composed of downsampling modules. The downsampling module includes a convolutional layer with a stride of 2 and a kernel of 3, an instance normalization layer, and a ReLU activation function layer. The network first uses a convolutional layer with k = 7, an instance normalization layer, and a ReLU activation function layer to receive the input single-channel image and initially extract shallow features. Then, two consecutive downsampling modules are used to further extract low-frequency features from the input and input them into the ECR module.

[0074] The present invention also introduces an efficient channel attention mechanism (ECA) to design the ECR module, which is formed by residual connection of the original residual block and the ECA module. The first half of the ECR is the original residual block, including a 3×3 convolutional layer, a ReLU activation layer, and a 3×3 convolutional layer, with a shortcut connection set between the input and output convolutional layers. The second half is the effective channel attention (ECA) block, which includes a global average pooling layer, a one-dimensional convolutional layer with an adaptive convolutional kernel, and a Sigmoid activation layer, with a shortcut connection also set between the input and output. The residual block can truly deepen the depth of the convolutional neural network without reducing the network accuracy, and its powerful representation ability can significantly improve the network performance. The ECA block improves the network performance by avoiding reducing the channel dimension to learn effective channel attention and obtaining cross-channel interaction information in an extremely lightweight manner. Adding an identity mapping in the ECR block not only ensures that there is no gradient vanishing problem during the backpropagation process, but also makes the network turn to learning the residual function, ensuring that the decision function of the entire network is smoother and improving the generalization performance.

[0075] The decoding structure upsamples the feature map. The upsampling module is composed of a transposed convolutional layer with a stride of 2 and a kernel of 3, an instance normalization layer, and a ReLU activation function layer. The feature map output by the ECR is reconstructed for the details and resolution of the image through two consecutive upsampling modules, and finally a three-channel, 256×256 resolution color image is output through a convolutional layer with k = 7 and a Tanh activation layer.

[0076] The discriminator network is used to identify whether the input is a fake image output by the generator or a real high-resolution color image. Since there is a zero-sum game relationship between the generator and the discriminator, the performance of the discriminator will also affect the specific performance of the generator. If the discriminator is not well trained and the model cannot accurately identify the data, then the fake image generated by the generator will also be judged as real by the discriminator, making the training of the generator difficult. Only when the discriminator is correctly trained can it provide more effective gradient information for training the generator, and finally optimize both network models after iterative training. Figure 3It is the specific structure of the discriminator network. The present invention introduces PatchGAN as the discriminator, enabling the model to pay more attention to image details. The receptive field is set to 70×70, which makes PatchGAN faster and still guides the generator to produce realistic results. The network structure specifically includes five 4×4 convolutional layers. The first four layers apply LeakyReLU, and the middle three layers apply batch normalization. During training, the discriminator acts as a competitor to play a game with the generator. A high-resolution image is used as the input, passes through five convolutional layers, and finally outputs the predicted result.

[0077] Optionally, when designing the loss function (PatchNCE Loss), it can be implemented based on the following method:

[0078] To maximize the mutual information between the input and output corresponding patches of the network, two signals, a "query" and its "positive" sample, are associated without being associated with other samples ("negatives") in the dataset. First, the cross-entropy loss is calculated to represent the probability of choosing the "positive" rather than the "negatives", as shown in Equation (1):

[0079]

[0080] where, v, v + ∈R K and v _ ∈R N×K respectively represent mapping the query, positive, and N negatives to k-dimensional vectors. v n _ ∈R K represents the nth negative. τ is a hyperparameter used to scale the distance between the query and other samples, and the default value is set to 0.07.

[0081] Genc and H l are used to share weights and extract features from domains X and Y. First, select L layers and pass the feature maps through H l Then generate a feature stack where the output of the lth layer is denoted as Similarly, the output image G(x) belonging to domain Y is encoded as To match the input and output patches corresponding to a specific location, the PatchNCE Loss is defined as in Equation (2):

[0082]

[0083] where, the spatial positions of each layer in Equation (2) are represented by Sl representation represents the corresponding feature (“positive”), represents other features (“negatives”), where C l represents the number of channels in each layer.

[0084] For Content - consistency Loss: The content - consistency loss is designed in the loss function of joint training. The content - consistency loss obtained by the multi - scale structural similarity index (MS - SSIM) enhances the high - frequency information of the image. It is defined as Equation (3):

[0085] L Content (G,X)=E x～X [1 - msSSIM(G(x),x)] (3)

[0086] In Equation (3), where G(x) represents the generated color image; x represents the input single - channel image, G represents the generator, X represents the input domain, and E x～X represents the expectation, and msSSIM represents the multi - scale structural similarity index. The content change between the generated color image G(x) and the input single - channel image x ∈ X is minimized as much as possible. The color inversion and feature distortion between the input and output images are avoided through the content - consistency loss.

[0087] For Adversarial Loss: The adversarial loss is used to make the output image G(x) of the generator G visually as similar as possible to the color image of the target domain Y. For the one - way mapping of the generator / discriminator (G / D) pair, the GAN loss is represented as Equation (4):

[0088]

[0089] For Identity Loss: The identity loss is added to avoid unnecessary changes in the generator G and encourage the input and output mappings to maintain the color and brightness composition, as shown in Equation (5).

[0090]

[0091] In this specification, the ultimate goal of the network is to make the patches of the input and output images learn the same corresponding relationships, so as to generate more realistic color images. The overall loss function is designed as Equation (6):

[0092]

[0093] where, λ GAN is the weight coefficient of the adversarial loss, λ NCEis the weight coefficient of the patchNCE loss, λ CCL is the weight coefficient of the content consistency loss, λ Idt is the weight coefficient of the identity loss. By default, the weight coefficients of the losses are set to 1, 10, 1, and 1 respectively.

[0094] Optionally, before training the Fourier ptychographic microscopy color reconstruction network using the dataset and the loss function to obtain the trained Fourier ptychographic microscopy color reconstruction network, it may further include:

[0095] Set the parameter settings for network learning and training; the parameters at least include the initial learning rate, the number of samples selected for each training, and the pixels of the training images;

[0096] When performing network learning and training, use the Adam optimizer and train based on the PyTorch deep learning framework.

[0097] Specifically, during the training process, use the Adam optimizer, the initial learning rate is 0.0002, and the batch size is 2. All training images are adjusted to 256×256 pixels and trained for 200 epochs. The learning rate starts to decay linearly after half of the total epochs. The model is based on the PyTorch deep learning framework and is deployed on a GPU that meets the requirements for experiments.

[0098] The solution provided by the present invention can be used for color reconstruction of single-channel FPM pathological sections. This solution designs a generator based on the ECR module and content consistency loss for joint training to obtain efficient cross-channel interaction information and improve the quality of stained images in an extremely lightweight manner. Experimental results show that this method effectively stains FPM images while maximizing the retention of high-frequency information of the input images, with the best visual effects. This virtual staining method can avoid differences in staining results caused by the randomness of personnel and technology, has the ability to standardize staining, and has broad application prospects in future digital pathology.

[0099] For the technical solution provided by the present invention, specific experiments were conducted, and the corresponding implementation data is as follows:

[0100] (1) Experimental parameter settings

[0101] The solution provided by the present invention is implemented on a server equipped with an NVIDIA GPU (RTX 3080, 10GB) and a Linux operating system. The initial learning rate of the ECSGAN network is set to 0.0002, the batch size is set to 2, and the Adam optimizer is selected to adaptively update the model parameters. All training images are adjusted to 256×256 pixels and trained for 200 epochs. When the network is trained to half of the total number of epochs, the learning rate starts to decay linearly.

[0102] (2) Comparative experiments with unsupervised deep learning methods

[0103] Through the qualitative comparison results of different comparison methods, it can be determined that ECSGAN successfully stains the unpaired single-channel FPM input image into a three-channel color FPM target domain image through image conversion. While performing good virtual coloring on the input image, ECSGAN can maximize the retention of the structure and content of the input image, thus obtaining the highest color image quality.

[0104] It can be observed from visual analysis that CUT cannot accurately identify the unstained parts in the image; CycleGAN and UNIT can well identify the white parts of the pathological sections, but lose the internal texture details; DCLGAN introduces artifacts and has low image quality. In addition, there are certain differences in color between other competing methods and the ground truth. ECSGAN has the highest visual similarity with the ground truth, proving the effectiveness of the network.

[0105] As shown in Table 1, the present invention selects FID, LPIPS, SSIM, and PSNR as evaluation indicators to measure the quality of the generated color images. Among them, FID and LPIPS have a high correspondence with human visual perception. Lower FID and LPIPS mean that the generated images are more realistic. On the contrary, the larger the values of SSIM and PSNR, the better the image quality. Table 1 shows the evaluation indicators of different comparative experiments. ECSGAN is significantly superior to other competing methods in terms of FID, LPIPS, SSIM, and PSNR indicators, and the specific indicators strongly prove its effectiveness.

[0106] Table 1. Quantitative results of different methods

[0107]

[0108] (3) Comparative results with classical virtual coloring methods

[0109] ECSGAN was compared with other classic and recent virtual coloring methods to further verify the effectiveness and time efficiency of this network in grayscale image coloring. The experimental results show that ECSGAN has significant advantages in terms of color and time, etc.

[0110] Through visual analysis, it can be concluded that ECSGAN has achieved the best results in terms of color and content.

[0111] The specific evaluation metrics are shown in Table 2. The number of iterations and inference time are selected as the metrics to measure time efficiency. ECSGAN has the minimum time cost and the best performance.

[0112] Table 2. Time efficiency metrics of different staining methods

[0113]

[0114] (4) Ablation study

[0115] To deeply analyze the effectiveness of ECSGAN, several ablation experiments were conducted to separately study each contribution of this paper. The first ablation experiment eliminated the content consistency loss and identity loss, and used the PatchNCE loss as a learnable, domain-specific identity loss. Another ablation experiment replaced the proposed ECR module in the generator G with the original residual block.

[0116] The experimental results are shown in Table 3. The removal of the content consistency loss and identity loss will lead to the loss of high-frequency information in the output image and deteriorate the visual effect of coloring. The ability of ECSGAN to accurately identify the blank parts of pathological sections decreases with the removal of the ECR block, resulting in poor image quality. ECSGAN shows superior performance in all metrics and visual comparisons.

[0117] Table 3. Evaluation metrics of ablation experiments

[0118]

[0119] Based on the same idea, the present invention also provides a Fourier ptychographic microscopic color reconstruction device, as Figure 4 shown, the device may include:

[0120] A dataset acquisition module 410, configured to acquire a high-resolution dataset; the dataset is a high-resolution single-channel grayscale dataset generated by using a publicly available dataset as the ground truth and simulating through a Fourier ptychographic imaging model;

[0121] The Fourier ptychography microscopic color reconstruction network construction module 420 is used to construct a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes an encoding structure based on downsampling, an ECR module based on efficient channel attention, and a decoding structure based on upsampling;

[0122] The loss function setting module 430 is used to set the loss function for training the Fourier ptychography microscopic color reconstruction network;

[0123] The Fourier ptychography microscopic color reconstruction network training module 440 is used to train the Fourier ptychography microscopic color reconstruction network by using the dataset and the loss function to obtain a trained Fourier ptychography microscopic color reconstruction network;

[0124] The target FPM color image prediction module 450 is used to input the data to be tested into the trained Fourier ptychography microscopic color reconstruction network to obtain a target FPM color image.

[0125] Based on Figure 4 the device in, some specific implementation units may further be included:

[0126] Optionally, the generator network is used to convert a single-channel FPM grayscale image into a high-resolution FPM color image by sequentially performing an encoding, ECR conversion, and decoding process; the discriminator network is used to determine whether the colored color input image is a reconstructed image of the generator network or a real image.

[0127] Optionally, the Fourier ptychography microscopic color reconstruction network construction module 420 may specifically include:

[0128] The generator network construction unit is used to construct a generator network including an encoding structure and a decoding structure; the encoding structure includes a downsampling module; the downsampling module includes a convolutional layer, an instance normalization layer, and an activation function layer; the generator network uses a preset convolutional layer, instance normalization layer, and activation function layer to initially extract shallow features from the input single-channel image; low-frequency features are extracted from the input single-channel image through continuous downsampling modules, and the low-frequency features are input into the ECR module; the decoding structure includes an upsampling module; the upsampling module includes a transposed convolutional layer, an instance normalization layer, and an activation function layer; the feature map output by the ECR module reconstructs the details and resolution of the image through continuous upsampling modules, and outputs a color image through a convolutional layer and an activation function layer;

[0129] The discriminator selection unit is used to use PatchGAN as the discriminator in the discriminator network;

[0130] The building unit of the Fourier ptychography microscopic color reconstruction network is used to construct a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network by using the generator network and the discriminator network.

[0131] Optionally, the ECR module includes an original residual block and an ECA module; the original residual block includes an input convolutional layer, an output convolutional layer, and an activation function layer; the ECA module includes a global average pooling layer, a one-dimensional convolutional layer with an adaptive convolutional kernel, and a Sigmoid activation layer, and an identity mapping is added in the ECR module.

[0132] Optionally, the loss function setting module 430 may specifically include:

[0133] A plurality of loss introduction units, which are used to introduce a content consistency loss, an identity loss, an adversarial loss, and a PatchNCE loss for joint training to determine the loss function of the Fourier ptychography microscopic color reconstruction network; the loss function is expressed as:

[0134] L(G,D,H)=λ GAN L GAN (G,D,X,Y)+λ NCE L PatchNCE (G,H,X)+λ CCL L Content (G,X)+λ Idt L Identity (G,Y)

[0135] Where λ GAN is the weight coefficient of the adversarial loss, λ NCE is the weight coefficient of the patchNCE loss, λ CCL is the weight coefficient of the content consistency loss, λ Idt is the weight coefficient of the identity loss.

[0136] Optionally, the content consistency loss is expressed as:

[0137] L Content (G,X)=E x～X [1 - msSSIM(G(x),x)].

[0138] Where G(x) represents the generated color image; x represents the input single-channel image, G represents the generator, X represents the input domain, and E x～X represents the expectation, and msSSIM represents the multi-scale structural similarity index.

[0139] Optionally, the device may further include:

[0140] A parameter setting unit for setting parameter settings for network learning and training; the parameters at least include an initial learning rate, the number of samples selected for each training, and the pixels of the training images.

[0141] A training unit for using the Adam optimizer and performing training based on the PyTorch deep learning framework when conducting network learning and training.

[0142] Based on the same idea, the embodiments of this specification also provide a Fourier ptychographic microscopy color reconstruction device. As Figure 5 shown, it may include:

[0143] A communication unit / communication interface for obtaining a high-resolution data set; the data set is a high-resolution single-channel grayscale data set generated by simulating using a publicly available data set as the ground truth and through a Fourier ptychographic imaging model.

[0144] A processing unit / processor for constructing a Fourier ptychographic microscopy color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure.

[0145] Set a loss function for training the Fourier ptychographic microscopy color reconstruction network.

[0146] Use the data set and the loss function to train the Fourier ptychographic microscopy color reconstruction network to obtain a trained Fourier ptychographic microscopy color reconstruction network.

[0147] Input the data to be tested into the trained Fourier ptychographic microscopy color reconstruction network to obtain a target FPM color image.

[0148] As Figure 5 shown, the above terminal device may further include a communication line. The communication line may include a path for transmitting information between the above components.

[0149] Optionally, as Figure 5 shown, the terminal device may further include a memory. The memory is used to store computer execution instructions for implementing the solution of the present invention and is controlled by the processor to execute. The processor is used to execute the computer execution instructions stored in the memory, thereby implementing the method provided by the embodiments of the present invention.

[0150] In a specific implementation, as an embodiment, as Figure 5 shown, the processor may include one or more CPUs, such as Figure 5 CPU0 and CPU1 in

[0151] In a specific implementation, as an example, as Figure 5 shown, the terminal device may include multiple processors, such as Figure 5 the processors in

[0152] Based on the same idea, the embodiments of this specification also provide a computer storage medium corresponding to the above embodiments. Instructions are stored in the computer storage medium, and when the instructions are run, the methods in the above embodiments are implemented.

[0153] The above mainly introduces the solutions provided by the embodiments of the present invention from the perspective of interactions between various modules. It can be understood that, in order to implement the above functions, each of the various modules includes corresponding hardware structures and / or software units for executing the respective functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed in this article, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or in the form of computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0154] The embodiments of the present invention can perform functional module division according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present invention is illustrative, merely a logical functional division, and there may be other division methods in actual implementation.

[0155] The processor in this specification can also have the function of a memory. The memory is used to store computer execution instructions for executing the solution of the present invention and is controlled by the processor for execution. The processor is used to execute the computer execution instructions stored in the memory, thereby implementing the methods provided by the embodiments of the present invention.

[0156] The memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor through a communication line. The memory may also be integrated with the processor.

[0157] Optionally, the computer-executable instructions in the embodiments of the present invention may also be referred to as application program code, and the embodiments of the present invention do not make specific limitations thereto.

[0158] The methods disclosed in the above embodiments of the present invention may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above processor may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention may be directly embodied as being executed and completed by a hardware decoding processor, or completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0159] In one possible implementation, a computer-readable storage medium is provided, in which instructions are stored and, when the instructions are run, are used to implement the method in the above-mentioned embodiments.

[0160] In the above-mentioned embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device, or other programmable devices. The computer program or instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid state drive (SSD).

[0161] Although the present invention has been described in conjunction with various embodiments herein, however, in the process of implementing the claimed invention, those skilled in the art can understand and achieve other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality. A single processor or other unit can implement several functions recited in the claims. Certain measures are recited in mutually different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0162] Although the present invention has been described in connection with specific features and their embodiments, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the invention. Accordingly, the present specification and the drawings are merely illustrative of the invention as defined by the appended claims and are considered to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A Fourier stacking microscopic color reconstruction method, characterized in that The method includes: Obtaining a high-resolution dataset; The dataset is a high-resolution single-channel grayscale dataset generated by simulating using a publicly available dataset as the ground truth through a Fourier ptychography imaging model; Constructing a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure; the generator network is used to convert a single-channel FPM grayscale image into a high-resolution FPM color image by gradually performing the encoding, ECR conversion, and decoding processes; the discriminator network is used to determine whether the stained color input image is a reconstructed image of the generator network or a real image; the ECR module includes an original residual block and an ECA module; the original residual block includes an input convolutional layer, an output convolutional layer, and an activation function layer; the ECA module includes a global average pooling layer, a one-dimensional convolutional layer with an adaptive convolutional kernel, and a Sigmoid activation layer, and an identity mapping is added in the ECR module; Setting a loss function for training the Fourier ptychography microscopic color reconstruction network; Using the dataset and the loss function to train the Fourier ptychography microscopic color reconstruction network to obtain a trained Fourier ptychography microscopic color reconstruction network; Inputting the data to be tested into the trained Fourier ptychography microscopic color reconstruction network to obtain a target FPM color image.

2. The Fourier stacking microscopic color reconstruction method according to claim 1, wherein The constructing of the Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network specifically includes: Constructing a generator network including an encoding structure and a decoding structure; the encoding structure includes a downsampling module; the downsampling module includes a convolutional layer, an instance normalization layer, and an activation function layer; the generator network uses a preset convolutional layer, an instance normalization layer, and an activation function layer to initially extract shallow features from the input single-channel image; extracts low-frequency features from the input single-channel image through continuous downsampling modules and inputs the low-frequency features into the ECR module; the decoding structure includes an upsampling module; the upsampling module includes a transposed convolutional layer, an instance normalization layer, and an activation function layer; the feature map output by the ECR module reconstructs the details and resolution of the image through continuous upsampling modules and outputs a color image through a convolutional layer and an activation function layer; PatchGAN is used as the discriminator in the discriminator network; Based on the generator network and the discriminator network, a Fourier ptychography microscopic color reconstruction network based on a self-supervised generative adversarial network is constructed.

3. The Fourier stacking microscopic color reconstruction method according to claim 2, wherein, The setting of the loss function for training the Fourier ptychography microscopic color reconstruction network specifically includes: Introducing content consistency loss, identity loss, adversarial loss, and PatchNCE loss for joint training and determining it as the loss function of the Fourier ptychography microscopic color reconstruction network; the loss function is expressed as: L(G, D, H) = λ GAN L GAN (G, D, X, Y) + λ NCE L PatchNCE (G, H, X) + λ CCL L Content (G, X) + λ Idt L Identity (G, Y) Among them, λ GAN is the weight coefficient of the adversarial loss, λ NCE is the weight coefficient of the patchNCE loss, λ CCL is the weight coefficient of the content consistency loss, λ Idt is the weight coefficient of the identity loss.

4. The Fourier stacking microscopic color reconstruction method according to claim 3, wherein, The content consistency loss is expressed as: L Content (G, X) = Ε x～X [1 - msSSIM(G(x), x)] Among them, G(x) represents the generated color image; x represents the input single-channel image, G represents the generator, X represents the input domain, and Ε x～X represents expectation, and msSSIM represents the multi-scale structural similarity index.

5. The Fourier stacking microscopic color reconstruction method according to claim 1, wherein Before training the Fourier ptychography color reconstruction network using the dataset and the loss function to obtain a trained Fourier ptychography color reconstruction network, the following steps are also included: Set the parameter settings for network learning and training; the parameters at least include the initial learning rate, the number of samples selected for each training, and the pixels of the training images; When performing network learning and training, use the Adam optimizer and train based on the PyTorch deep learning framework.

6. A Fourier stacking microscopic color reconstruction device, characterized in that, The device includes: A dataset acquisition module for acquiring a high-resolution dataset; The dataset is a high-resolution single-channel grayscale dataset generated by simulating using a publicly available dataset as the ground truth through a Fourier ptychography imaging model; A Fourier ptychography color reconstruction network construction module for constructing a Fourier ptychography color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure; A loss function setting module for setting the loss function for training the Fourier ptychography color reconstruction network; A Fourier ptychography color reconstruction network training module for training the Fourier ptychography color reconstruction network using the dataset and the loss function to obtain a trained Fourier ptychography color reconstruction network; A target FPM color image prediction module for inputting the data to be tested into the trained Fourier ptychography color reconstruction network to obtain a target FPM color image.

7. A Fourier stacking microscopic color reconstruction device, characterized in that the device It includes: A communication unit / communication interface for acquiring a high-resolution dataset; The dataset is a high-resolution single-channel grayscale dataset generated by simulating using a publicly available dataset as the ground truth through a Fourier ptychography imaging model; A processing unit / processor for constructing a Fourier ptychography color reconstruction network based on a self-supervised generative adversarial network; the self-supervised generative adversarial network includes a generator network and a discriminator network; the generator network includes a downsampling-based encoding structure, an ECR module based on efficient channel attention, and an upsampling-based decoding structure; the generator network is used to convert a single-channel FPM grayscale image into a high-resolution FPM color image by gradually performing the encoding, ECR conversion, and decoding processes; the discriminator network is used to determine whether the colored input image after staining is the reconstructed image of the generator network or a real image; the ECR module includes an original residual block and an ECA module; the original residual block includes an input convolutional layer, an output convolutional layer, and an activation function layer; the ECA module includes a global average pooling layer, a one-dimensional convolutional layer with an adaptive convolutional kernel, and a Sigmoid activation layer, and an identity mapping is added in the ECR module; Set the loss function for training the Fourier ptychography color reconstruction network; Train the Fourier ptychography color reconstruction network using the dataset and the loss function to obtain a trained Fourier ptychography color reconstruction network; Input the data to be tested into the trained Fourier ptychography microscopic color reconstruction network to obtain the target FPM color image.

8. A computer storage medium, characterized in that, Instructions are stored in the computer storage medium, and when the instructions are run by the processor, the Fourier ptychography microscopic color reconstruction method according to any one of claims 1 to 5 is implemented.

Images