A method for image grayscale stereo conversion based on GSCGAN

Abstract

The present invention discloses a method for grayscale stereo conversion of images based on GSCGAN, comprising the following steps: Step 1: constructing a bright field mode image set, a phase contrast mode image set, and a differential interference mode image set; Step 2: using the light propagation convolution formula in phase contrast imaging and differential interference imaging to replace the convolution propagation operation in the traditional GAN ​​to construct a complex-valued GAN and establish a preliminary GSCGAN model; Step 3: inputting the training image into the preliminary GSCGAN model for training to obtain the GSCGAN model; Step 4: inputting the transparent target sample image into the GSCGAN model for testing to obtain the result image. The present invention uses a model based on GSCGAN to perform grayscale stereo conversion on the transparent sample image, and obtains an image with a phase contrast interference effect or a differential interference effect. The constructed pattern discriminator D m It can determine whether an image is a phase contrast image or a differential interference effect image, allowing the input of one image to output two different effect images. It also proposes a grayscale loss function to effectively preserve the correct information while avoiding the effect of color confusion on the result.

Description

Technical Field

[0001] The present invention belongs to the fields of image processing technology and microscopic imaging technology, and particularly relates to an image grayscale stereo conversion method based on GSCGAN. Background Art

[0002] When observing transparent samples under brightfield light, conventional microscopes often fail to clearly and accurately visualize the sample's hierarchical structure and textural details. While staining and other treatments allow for visualization, there's a risk of damaging the sample's cells. Phase contrast microscopy utilizes the principle that varying density and composition of the object produce varying phase differences under illumination. This phase difference is converted into light and dark differences discernible to the naked eye, allowing for more precise visualization of the fine structure of biological samples. Differential interference contrast, however, reveals a distinct, embossed sample structure, facilitating the identification of structural details that are invisible or difficult to discern under brightfield light. Currently, phase contrast and differential interference contrast microscopy systems require expensive optical accessories such as interference prisms, resulting in a high system cost. Therefore, using computational imaging to convert ordinary, inexpensive microscopes into phase contrast or differential interference contrast microscopes can significantly reduce costs.

[0003] In recent years, deep learning has been widely applied in fields such as machine learning, image recognition, remote sensing imaging, and medical imaging. Image style transfer involves applying an image style to a given image to enhance its presentation. Using deep learning methods for style transfer is a hot topic in imaging research. Generative Adversarial Networks (GANs) have garnered significant attention in recent years, with applications in a variety of areas, including image super-resolution, image transfer, and image segmentation.

[0004] In his paper "Image Style Transfer Network Incorporating Texture Feature Analysis," Yu Yingdong proposed a feedforward image style transfer network algorithm that combines image texture feature analysis. This algorithm performs style transfer by training a feedforward residual network with a perceptual loss function. By analyzing the information entropy and invariant moments of the input image, the algorithm discovered the relationship between the input image's features and the output, thereby achieving targeted improvements to the network. However, the connection between color, depth, and other feature information is severed, and their correlation and complementarity are not fully considered. This can lead to false color errors when testing grayscale images. Deep learning-based methods for enhancing image contrast primarily rely on training data to update network parameters, thus requiring a large amount of training data. However, current datasets for medical and biological image research are limited due to the confidentiality of hospital patient samples, making it difficult to achieve higher prediction accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide an image grayscale stereo conversion method based on GSCGAN, which solves the problem that it is difficult to observe clear structures and details of transparent biomedical samples under ordinary microscopes.

[0006] The technical solution to realize the present invention is: a method for grayscale stereo conversion of an image based on GSCGAN, characterized by comprising the following steps:

[0007] Step 1: Using a scientific microscope, collect P transparent samples and obtain corresponding images in bright field mode, phase contrast mode, and differential interference mode, respectively, for a total of 3P images, where P ≥ 200; each image has a pixel size of 1000 × 1000, and the images are converted into grayscale images to form a bright field mode image set, a phase contrast mode image set, and a differential interference mode image set, respectively;

[0008] Step 2: Use the light propagation convolution formula in phase contrast imaging and differential interference imaging to replace the convolution propagation operation in traditional GAN ​​to construct a complex-valued GAN. Connect the two complex-valued GANs end to end to form a loop structure, and add a pattern discriminator D after each layer outputs the image. m , obtain the preliminary GSCGAN model;

[0009] Step 3: Use the grayscale images in the brightfield mode image set as the original training images, and the grayscale images in the phase contrast mode image set and the differential interference mode image set as the ideal training target images. Perform Fourier transform on the original training images and the ideal training target images, convert each pixel value in the above images into a complex number, and obtain image feature maps of real and imaginary pairs corresponding to the amplitude and phase, respectively. These complex-valued pairs are input into the preliminary GSCGAN model for training to obtain the GSCGAN model.

[0010] Step 4: Collect a transparent target sample image as real data with a pixel size of 1000×1000, convert the image into a grayscale image, and input it into the GSCGAN model to obtain the corresponding phase contrast interference image or differential interference image, thereby achieving a stereoscopic imaging effect with distinct light and dark.

[0011] Compared with the prior art, the present invention has the following significant advantages:

[0012] (1) The light propagation convolution formula in phase contrast imaging and differential interference imaging is used to replace the convolution propagation operation in traditional GAN ​​to construct a complex-valued GAN, which is more accurate than the traditional real-valued convolution.

[0013] (2) Added pattern discriminator D m The converted output image or the ideal training target image is analyzed to determine whether it is a symmetrical effect image or a differential interference effect image, so that two different effect images can be output by inputting one image.

[0014] (3) A grayscale loss function is incorporated into the loss function. The root mean square (RMS) calculation is performed on the grayscale difference between the X-domain real data and the Y-domain real data after single-channel grayscale conversion. Single-channel grayscale conversion only takes the channel with the largest proportion among the three RGB channels, discards the other two channels, and only analyzes the data information of the single channel. This effectively preserves the correct information while avoiding the influence of color confusion on the results caused by pseudo-color. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of the image grayscale stereo conversion method based on GSCGAN of the present invention.

[0016] Figure 2(a) is a set of bright field mode images, Figure 2(b) is a set of phase contrast mode images, and Figure 2(c) is a set of differential interference mode images.

[0017] Figure 3 is a transparent target sample image.

[0018] Figure 4(a) is the phase contrast interferogram output by GSCGAN, and Figure 4(b) is the differential interference image output by GSCGAN. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below with reference to the accompanying drawings.

[0020] Combine Figure 1 , a method for image grayscale stereo conversion based on GSCGAN, comprising the following steps:

[0021] Step 1: Use a scientific research microscope to collect P transparent samples and obtain corresponding images in bright field mode, phase contrast mode, and differential interference mode, respectively, for a total of 3P images, where P ≥ 200; the pixel size of each of the above images is 1000 × 1000, and the images are converted into grayscale images, which correspond to the bright field mode image set, phase contrast mode image set, and differential interference mode image set, respectively.

[0022] Furthermore, corresponding images of the same transparent sample in bright field mode, phase contrast mode, and differential interference mode are collected without changing the angle and position of the scientific research microscope.

[0023] Step 2: Use the light propagation convolution formula used in phase contrast imaging and differential interference imaging to replace the convolution propagation operation in traditional GANs (Generative Adversarial Networks) to construct a complex-valued GAN. That is, the convolution propagation in each layer of the network is no longer the real-valued propagation in ordinary GANs, but the optical complex amplitude and phase propagation. Connect the two complex-valued GANs end to end to form a loop structure, and add a pattern discriminator Dm after each layer output image to obtain the preliminary grayscale stereo conversion generative adversarial network (GSCGAN) model, as follows:

[0024] The complex-valued GAN is constructed by using the light propagation convolution formula in phase contrast imaging and differential interference imaging to replace the convolution propagation operation in the traditional GAN. The light propagation convolution formula is obtained based on the phase contrast and differential interference imaging theory, as follows:

[0025] For phase contrast imaging, it is assumed that the complex amplitude distribution of the object transmitted light is By changing the phase spectrum information of the object, the transformed complex amplitude distribution is The light intensity distribution is where δ represents the zero-order phase shift, represents phase information, i represents the imaginary unit, and (x′, y′) represents the coordinates of the information point on the image plane. From the above formula, we can see that the light intensity and phase are in a linear relationship. If the phase difference is converted into amplitude information, the image after conversion can show the difference in brightness and darkness.

[0026] For differential interference imaging, the light intensity distribution of the sample imaging is in It represents the interference difference between orthogonal linear polarized lights, E0 represents the amplitude of the polarization component, H(o,e) represents the differential of the sample surface morphology, o represents the position of the prism, and the measured phase distribution Where λ is the wavelength of light, Δo is the transverse shear between orthogonal linearly polarized lights, and β(o) is the prism aberration, which is a linear function of the prism position.

[0027] The method of representing complex values ​​in the network is as follows: a complex number z = A + ib has a real component A and an imaginary component b. We represent the real part A and the imaginary part b of a complex number as logically different real-valued entities, and use real-valued algorithms to simulate complex arithmetic internally, where i represents the imaginary unit. If a real-valued two-dimensional convolutional layer has N feature maps, such that N is divisible by 2, in order to represent them as complex numbers, the first N / 2 feature maps are assigned to represent the real component, and the remaining N / 2 feature maps are assigned to represent the imaginary component. Therefore, for the concatenated input feature map Nin To the output feature map N out The four-dimensional weight tensor W has a kernel size of m×m, that is, there is a size of (N out ×N in ×m×m) / 2 complex weight tensor.

[0028] Connect the two GANs end to end to form a loop structure, and add a pattern discriminator D after each layer outputs the image. m , we get the preliminary GSCGAN model, as follows:

[0029] The first GAN is used to generate images from domain X to domain Y, consisting of the first generator G and the first discriminator D x The second GAN is used to generate images from domain Y to domain X, consisting of the second generator F and the second discriminator D y The two GANs are connected end to end to form a loop structure.

[0030] The first generator G generates a fake image containing Y domain information based on the original training image input in the X domain and the information of the Y domain in the ideal training target image. The first discriminator D x To analyze whether the fake image is the Y domain image in the ideal training target image, if it is, it is output, if not, the information is fed back to the first generator G to let it update the fake image until the first discriminator D x The second generator F generates a fake image containing X domain information based on the original training image input in the Y domain and the information of the X domain in the ideal training target image. The second discriminator D y To analyze whether the fake image is the X domain image in the ideal training target image, and feedback information to the second generator F to update the image; the pattern discriminator D m The converted output image or the ideal training target image is analyzed to determine whether it is a symmetrical effect image or a differential interference effect image, and the feedback information is used to guide the two generators respectively.

[0031] The first generator G, its loss function during training is L GSCGAN (G,D x ,M) are as follows:

[0032] L GSCGAN (G,D x ,M)=L GAN (G,D x ,X,Y)+αL GAN (G,M)+βL Grayscale (G,X,Y)

[0033] Among them, the first generator G and the first discriminator D x The adversarial loss function represents the mathematical expectation of the real data in the X domain, Represents the mathematical expectation of the real data in the Y domain; the first generator G and the pattern discriminator D m The adversarial loss function L GAN (G,M)=∑ X p(x)log q(x), p(x) represents the probability of generating a phase contrast mode image in the X domain, and q(x) represents the probability of generating a differential interference mode image in the X domain. The generator is trained with different mode images; L Grayscale (G, X, Y) is the grayscale loss function, α and β represent the weights of each loss in the network, D x (x,y) is the first discriminator D x The guidance information returned to the first generator G, G(x,y) represents the image information output by the generator G based on the original training image input in the X domain and the information in the Y domain of the ideal training target image;

[0034] The loss function L of the second generator F during training GSCGAN (F,D y ,M) are as follows:

[0035] L GSCGAN (F,D y ,M)=L GAN (F,D y ,X,Y)+αL GAN (F,M)+βL Grayscale (F,X,Y)

[0036] Among them, the second generator F and the second discriminator D y The adversarial loss function represents the mathematical expectation of the real data in the X domain, Represents the mathematical expectation of the real data in the Y domain; the second generator F and the pattern discriminator D m The adversarial loss function L GAN (F,M)=∑ Y p(y)log q(y), p(y) represents the probability of generating a phase contrast mode image in the Y domain, and q(y) represents the probability of generating a differential interference mode image in the Y domain, allowing the generator to train different mode images; L Grayscale (F, X, Y) is the grayscale loss function, α and β represent the weights of each loss in the network, D Y (y,x) is the second discriminator D yThe guidance information returned to the second generator F, F(y,x) represents the image information output by the generator F based on the original training image input in the Y domain and the information in the X domain of the ideal training target image;

[0037] The grayscale loss function L of the generator during training Grayscale :

[0038] The root mean square calculation is performed on the grayscale difference between the real data in the X domain and the real data in the Y domain after single-channel grayscale conversion. Single-channel grayscale conversion only takes the channel with the largest proportion among the three RGB channels, discards the other two channels, and only analyzes the data information of the single channel.

[0039] The first discriminator D x and the second discriminator D y The structure is the same, both consisting of 5 convolutional layers. After extracting features from the image, it is determined whether these features belong to a specific category. The last convolutional layer is used to produce a one-dimensional output.

[0040] Step 3: Use the grayscale images from the brightfield image set as the original training images, and the grayscale images from the phase contrast and differential interference image sets as the ideal training target images. By performing a Fourier transform on the original training images and the ideal training target images, each pixel value in the image is converted to a complex number. Image feature maps of real and imaginary pairs corresponding to amplitude and phase are obtained, respectively. These complex-valued pairs are input into the preliminary GSCGAN model for training, resulting in the GSCGAN model.

[0041] Step 4: Collect a transparent target sample image as real data with a pixel size of 1000×1000, convert the image into a grayscale image, and input it into the GSCGAN model to obtain the corresponding phase contrast interference image or differential interference image, thereby achieving a stereoscopic imaging effect with distinct light and dark.

[0042] Example 1

[0043] Step 1: Use a scientific microscope to collect 400 transparent samples and obtain corresponding images in bright field mode, phase contrast mode, and differential interference mode, for a total of 3P images, P ≥ 200; the pixel size of each image is 1000 × 1000, and the image is converted into a grayscale image, which corresponds to a bright field mode image set, a phase contrast mode image set, and a differential interference mode image set, as shown in Figure 1. Figure 2(a) to Figure 2(c) shown.

[0044] Step 2: Use the light propagation convolution formula in phase contrast imaging and differential interference imaging to replace the convolution propagation operation in traditional GAN ​​to construct a complex-valued GAN. That is, the convolution propagation in each layer of the network is no longer the real-valued propagation in ordinary GAN, but the complex amplitude and phase propagation in optics. Connect the two complex-valued GANs end to end to form a loop structure, and add a pattern discriminator D after each layer outputs the image. m , obtain the preliminary GSCGAN model;

[0045] Step 3: Use the grayscale images from the brightfield image set as the original training images, and the grayscale images from the phase contrast and differential interference image sets as the ideal training target images. By performing a Fourier transform on the original training images and the ideal training target images, each pixel value in the image is converted to a complex number. Image feature maps of real and imaginary pairs corresponding to amplitude and phase are obtained, respectively. These complex-valued pairs are input into the preliminary GSCGAN model for training, resulting in the GSCGAN model.

[0046] Step 4: Collect transparent target sample images (such as Figure 3 As shown in Figure 4(a) and Figure 4(b)), the real data is taken as the image pixel size of 1000×1000, and the image is converted into a grayscale image and input into the GSCGAN model to obtain the corresponding phase contrast interference image or differential interference image (as shown in Figure 4(a) and Figure 4(b)), thereby achieving a stereoscopic imaging effect with distinct light and dark.

[0047] In summary, the GSCGAN-based grayscale stereo image conversion method provided in embodiments of the present invention trains and constructs a model using GSCGAN, original training images, and ideal training target images. A transparent target sample image is then input into the GSCGAN model to generate a phase contrast interferometry image or a differential interference effect image. Using the GSCGAN-based grayscale stereo image conversion method of the present invention, a transparent sample image can be directly converted into a phase contrast interferometry image or a differential interference effect image.

Claims

1. A method for image grayscale stereo conversion based on GSCGAN, characterized in that: The following steps are involved: Step 1: Using a scientific microscope, collect P transparent samples and obtain corresponding images in bright field mode, phase contrast mode, and differential interference mode, respectively, for a total of 3P images, where P ≥ 200; each of the above images has a pixel size of 1000 × 1000, and convert the images into grayscale images to form a bright field mode image set, a phase contrast mode image set, and a differential interference mode image set, respectively; Step 2: Use the light propagation convolution formula in phase contrast imaging and differential interference imaging to replace the convolution propagation operation in traditional GAN ​​to construct a complex-valued GAN. Connect the two complex-valued GANs end to end to form a loop structure, and add a pattern discriminator D after each layer outputs the image. m , obtain the preliminary GSCGAN model; Step 3: Use the grayscale images in the brightfield mode image set as the original training images, and the grayscale images in the phase contrast mode image set and the differential interference mode image set as the ideal training target images. Perform Fourier transform on the original training images and the ideal training target images to convert each pixel value in the above images into a complex number. Obtain image feature maps of real and imaginary pairs corresponding to amplitude and phase, respectively. These complex-valued pairs are input into the preliminary GSCGAN model for training to obtain the GSCGAN model. Step 4: Collect a transparent target sample image as real data with a pixel size of 1000×1000, convert the image into a grayscale image, and input it into the GSCGAN model to obtain the corresponding phase contrast interferogram or differential interference image, thereby achieving a stereoscopic imaging effect with distinct light and dark. In step 1, the angle and position of the scientific microscope are maintained, and images of the same transparent sample are collected in bright field mode, phase contrast mode, and differential interference mode. In step 2, the light propagation convolution formula in phase contrast imaging and differential interference imaging is obtained based on the phase contrast and differential interference imaging theory, as follows: For phase contrast imaging, it is assumed that the complex amplitude distribution of the object transmitted light is By changing the phase spectrum information of the object, the transformed complex amplitude distribution is The light intensity distribution is where δ represents the zero-order phase shift, represents phase information, i represents the imaginary unit, and (x', y') represents the coordinates of the information point on the image plane. From the above formula, we can see that the light intensity and phase are in a linear relationship. The phase difference is converted into amplitude information, and the image after conversion can show the difference in brightness and darkness. For differential interference imaging, the light intensity distribution of the sample imaging is in It represents the interference difference between orthogonal linear polarized lights, E0 represents the amplitude of the polarization component, H(o,e) represents the differential of the sample surface morphology, o represents the position of the prism, and the measured phase distribution Where λ represents the wavelength of light, Δo is the transverse shear between orthogonal linearly polarized light, and β(o) is the prism aberration, which is a linear function of the prism position. The complex value is represented in the network as follows: a complex number z = A + ib has a real component A and an imaginary component b. We represent the real part A and the imaginary part b of a complex number as logically different real-valued entities, and use real-valued algorithms to simulate complex arithmetic internally, where i represents the imaginary unit. If a real-valued two-dimensional convolutional layer has N feature maps, such that N is divisible by 2, in order to represent them as complex numbers, the first N / 2 feature maps are assigned to represent the real component, and the remaining N / 2 feature maps are assigned to represent the imaginary component. Therefore, for the concatenated input feature map N in To the output feature map N out The four-dimensional weight tensor W has a kernel size of n×n, that is, there is a size of (N out ×N in ×n×n) / 2 complex weight tensor; In step 2, the two complex-valued GANs are connected end to end to form a loop structure, and a pattern discriminator D is added after each layer outputs the image. m , we get the preliminary GSCGAN model, as follows: The first GAN is used to generate images from domain X to domain Y, consisting of the first generator G and the first discriminator D x The second GAN is used to generate images from domain Y to domain X, consisting of the second generator F and the second discriminator D y Composition: connect two GANs end to end to form a loop structure; The first generator G generates a fake image containing Y domain information based on the original training image input in the X domain and the information of the Y domain in the ideal training target image. The first discriminator D x To analyze whether the fake image is the Y domain image in the ideal training target image, if it is, it is output, if not, the information is fed back to the first generator G to let it update the fake image until the first discriminator D x The second generator F generates a fake image containing X domain information based on the original training image input in the Y domain and the information of the X domain in the ideal training target image. The second discriminator D y To analyze whether the fake image is the X domain image in the ideal training target image, and feedback information to the second generator F to update the image; the pattern discriminator D m Analyze the converted output image or the ideal training target image to determine whether it is a symmetry effect image or a differential interference effect image, and provide feedback to guide the two generators respectively; The first generator G, its loss function during training is L GSCGAN (G,D x ,M) are as follows: L GSCGAN (G,D x ,M)=L GAN (G,D x ,X,Y)+αL GAN (G,M)+βL Grayscale (G,X,Y) Among them, the first generator G and the first discriminator D x The adversarial loss function represents the mathematical expectation of the real data in the X domain, Represents the mathematical expectation of the real data in the Y domain; the first generator G and the pattern discriminator D m The adversarial loss function L GAN (G,M)=∑ X p(x)log q(x), p(x) represents the probability of generating a phase contrast mode image in the X domain, and q(x) represents the probability of generating a differential interference mode image in the X domain. The generator is trained with different mode images; L Grayscale (G, X, Y) is the grayscale loss function, α and β represent the weights of each loss in the network, D x (x,y) is the first discriminator D x The guidance information returned to the first generator G, G(x,y) represents the image information output by the generator G based on the original training image input in the X domain and the information in the Y domain of the ideal training target image; The second generator F, whose loss function during training is L GSCGAN (F,D y ,M) are as follows: L GSCGAN (F,D y ,M)=L GAN (F,D y ,X,Y)+αL GAN (F,M)+βL Grayscale (F,X,Y) Among them, the second generator F and the second discriminator D y The adversarial loss function represents the mathematical expectation of the real data in the X domain, Represents the mathematical expectation of the real data in the Y domain; the second generator F and the pattern discriminator D m The adversarial loss function L GAN (F,M)=∑ Y p(y)log q(y), p(y) represents the probability of generating a phase contrast mode image in the Y domain, and q(y) represents the probability of generating a differential interference mode image in the Y domain, allowing the generator to train different mode images; L Grayscale (F, X, Y) is the grayscale loss function, α and β represent the weights of each loss in the network, D y (y,x) is the second discriminator D y The guidance information returned to the second generator F, F(y,x) represents the image information output by the generator F based on the original training image input in the Y domain and the information in the X domain of the ideal training target image.

2. The method for image grayscale stereo conversion based on GSCGAN according to claim 1, characterized in that: The grayscale loss function L of the generator during training Grayscale : The root mean square calculation is performed on the grayscale difference between the real data in the X domain and the real data in the Y domain after single-channel grayscale conversion. Single-channel grayscale conversion only takes the channel with the largest proportion among the three RGB channels, discards the other two channels, and only analyzes the data information of the single channel.

3. The method for image grayscale stereo conversion based on GSCGAN according to claim 2, characterized in that: The first discriminator D x and the second discriminator D y The structure is the same, both consisting of 5 convolutional layers. After extracting features from the image, it determines whether these features belong to the specified category. The last convolutional layer is used to produce a one-dimensional output.

Images