Microscopic image super-resolution method based on mathematical imaging theory and generative adversarial network

Abstract

The present application relates to a kind of microscopic image super-resolution method based on mathematical imaging theory and generative adversarial network, belong to image processing and computer vision field;Based on the imaging model of both wide-field microscope and confocal microscope, the present application proposes a kind of method for predicting high-resolution microscopic image using mathematical theory and improved super-resolution generative adversarial network, using the point spread function accurately derived to simulate generation high, low resolution image pair as the data set required for network training, without image registration.In network design part, combined with U-Net and residual network, an improved generative adversarial network is proposed, and the generator and discriminator in the network are redefined.The generator adopts 4-layer convolutional network and 16 residual module structures composed of skip connection structure to collect image details, and the discriminator is optimized by deepening CNN layer and fusing feature mapping, to solve the problem that the discriminator is difficult to judge whether the generated predicted image is real.

Description

Technical Field

[0001] The present invention belongs to the field of biological microscopic image processing, and mainly relates to a microscopic image super-resolution method based on mathematical imaging theory and generative adversarial networks. Background Art

[0002] Microscopy is a powerful tool for observing intracellular microstructures and dynamic processes. Traditional widefield (WF) microscopy offers fast imaging speeds but low resolution. Confocal microscopy (CM) uses point illumination and detection to isolate background stray light and collect only signals from the focal plane. This improves resolution compared to widefield microscopy, increasing resolution by 1.4 times at a pinhole size of 1 AU (Airy Unit). In microscopic imaging, there is a trade-off between imaging speed and resolution, making it difficult to achieve optimal results simultaneously.

[0003] To explore dynamic changes in samples at the cellular level, continuous measurement and quantitative analysis of both overall structure and local details at the microscopic scale are necessary. Therefore, large-field-of-view, high-resolution, and long-time imaging are essential in the life sciences. Existing super-resolution microscopy methods struggle to strike a balance between these three key performance indicators. The emergence of deep learning has, to some extent, addressed this issue. Research has demonstrated that deep learning is highly effective in recovering fluorescence images with low signal-to-noise ratios or low resolution. However, to ensure accurate deep learning predictions, these methods require a large number of registered high- and low-resolution image pairs during dataset preparation, resulting in high dataset preparation costs. Furthermore, obtaining a large number of fluorescence image pairs with the same field of view is challenging, particularly in in vivo imaging, where samples may exhibit different morphologies at different times, potentially leading to changes in the field of view during objective lens switching. Furthermore, optical distortion and chromatic aberration between objective lenses are unavoidable, making it difficult to obtain perfectly aligned images, significantly impacting deep learning training effectiveness. Most deep learning reconstruction methods adopt a simple end-to-end strategy that ignores the physical laws of the imaging process, making the preparation of training data very challenging and prone to generating false structures or artifacts from specific datasets.

[0004] Therefore, a technical problem that technicians in this field urgently need to solve is: when predicting super-resolution images through deep learning, how to generate high- and low-resolution image pairs that conform to the intrinsic physical laws of the actual imaging process as training data sets while reducing the actual acquisition of fluorescence images, and at the same time propose an improved network for the traditional deep learning network model to achieve a low-cost, high-quality, and high-efficiency super-resolution image prediction method. Summary of the Invention

[0005] The present invention aims to overcome the shortcomings of existing technologies by proposing a microscopic image super-resolution method based on mathematical imaging theory and an improved generative adversarial network. This method eliminates the need to acquire actual microscopic images. Instead, it establishes a mathematical model of the microscopic imaging process to calculate an accurate point spread function. This function is then used to simulate paired low- and high-resolution images, which serve as a dataset for deep learning. Because this dataset is simulated using an imaging model, the relationship between the low- and high-resolution image pairs conforms to actual imaging relationships. After network training is complete, prediction of high-resolution microscopic images can be achieved.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Based on mathematical models of widefield and confocal microscopy imaging processes, this paper simulates and generates high- and low-resolution image pairs as network training datasets. This eliminates the need to acquire actual widefield and confocal microscopy images, facilitating dataset preparation using open-source image data and eliminating the need for further image alignment preprocessing. Furthermore, a modified generative adversarial network (GAN) is proposed. This network combines U-Net and residual networks, redefining the generator and discriminator in GANs. The generator utilizes a 4-layer convolutional network with 16 residual modules composed of skip connections to capture image details. Two sub-pixel convolutional networks upscale the input 128×128 pixel low-resolution widefield image to a 512×512 pixel predicted image. The target image is further optimized using a mean squared error loss function. The discriminator comprises 10 convolutional layers, each batch-normalized and activated by Reinforced Unit (ReLU). The number of feature maps is increased from 64 to 2048 and then reduced to 512. The output tensor image is then fed into the flattening layer and the fully connected layer, where it is activated using the Sigmoid activation function to produce the final discriminant result. The discriminator is optimized by deepening the CNN layer and fusing feature maps, addressing the discriminator's difficulty in determining whether the generated predicted image is realistic.

[0008] Specifically, the deep learning super-resolution microscopy image prediction method based on the physical imaging model mainly includes the following steps:

[0009] Step 1: Establish widefield microscopy and confocal microscopy imaging system models, and derive and calculate the point spread functions of the two microscopy imaging systems;

[0010] Step 2: Using the imaging system point spread function established in step 1, perform deconvolution calculations on the open-source image dataset and add Gaussian white noise to obtain simulated wide-field images and confocal microscopy images, respectively. The generated simulated images are then translated, rotated, and flipped 50% to complete image augmentation.

[0011] Step 3. Propose an improved generative adversarial network, which redefines the generator and discriminator in combination with the residual network, and inputs the simulated low-resolution wide-field microscopy images of size 128×128 pixels into the network as a training dataset.

[0012] Step 4: Train the network's generator and discriminator until the loss function is minimized, completing network training. At this point, the network inputs a 128×128 pixel low-resolution wide-field microscopy image, and outputs a predicted 512×512 pixel high-resolution microscopy image.

[0013] The first step is to establish the point spread function of the microscopic imaging system. Based on the Richards-Wolf vector diffraction integral and Tony Wilson confocal imaging theory, a wide-field imaging model and a confocal microscopic imaging model are established, and accurate point spread functions are derived for each.

[0014] In the step 2, based on the imaging model, the point spread function derived from the mathematical imaging model established in step 1 is used to calculate the same image through the deconvolution method, and simulated wide-field microscopic images and confocal microscopic images are obtained as low- and high-resolution image pairs, respectively. The low-resolution image and the high-resolution image pair conform to the intrinsic relationship in the physical imaging process. During the simulation process, the system noise is set to Gaussian white noise in combination with the imaging parameters of the actual microscopic imaging system. The generation of simulated images can utilize open source data without the need to collect a large number of real microscopic images. And because the low-resolution image and the high-resolution image pair are generated by simulating the same image, there is no need to perform preprocessing such as image alignment in the input network training model, which simplifies the training process. In order to expand the data set, the image was rotated and flipped 50%.

[0015] Step 3 involves an improved generative adversarial network (GAN) that combines a U-Net and a residual network, redefining the generator and discriminator within the GAN. The generator utilizes a 4-layer convolutional network with 16 residual modules composed of skip connections to capture image details. Two sub-pixel convolutional networks upscale the input 128×128 pixel low-resolution wide-field image to a 512×512 pixel predicted image. The target image is further optimized using a mean squared error loss function. Each network module uses a 3×3 convolution kernel for high-dimensional representation, and the network is deepened to improve model training performance. The discriminator comprises 10 convolutional layers, each with batch normalization and Reluctant Unit (ReLU) activation. The number of feature maps is increased from 64 to 2048, and then reduced to 512. The output tensor image is then fed into a flattening layer and a fully connected layer, activated with a sigmoid activation function, to produce the final discrimination result. The discriminator is optimized by deepening the CNN layer and fusing feature maps to solve the problem that the discriminator has difficulty in judging whether the generated predicted image is real.

[0016] The method of the present invention differs from the existing method of using GAN to generate predicted microscopic images in that:

[0017] 1. Image datasets are generated differently

[0018] The present invention establishes rigorous widefield imaging and confocal microscopy models, deriving precise point spread functions, respectively, and then obtaining simulated widefield microscopy and confocal microscopy images through deconvolution. The present invention can utilize open-source image datasets to simulate and generate a large number of low- and high-resolution image pairs that conform to actual physical imaging relationships, eliminating the need to collect a large number of actual microscopy images. Furthermore, because the low- and high-resolution image pairs are generated from the same image, precise image registration is not required before input into the network. This simulation-driven approach to generating image datasets reduces experimental costs and simplifies dataset preparation, completely different from traditional dataset preparation methods.

[0019] 2. GANs are different in form

[0020] The present invention combines U-Net and residual network to propose an improved generative adversarial network. The generator and discriminator are redefined, wherein the generator adopts a six-layer convolutional network and a jump structure to form a residual module, and further optimizes the target image in combination with the mean square error loss function. To provide rich feature information for the training model, combined with the characteristics of the U-Net network, the 16 residual modules are designed as a "U" structure containing 6 convolutional layers, wherein the jump connection structure can effectively avoid the loss of additional information caused by pooling and upsampling. The discriminator network uses 10 convolutional layers, and the image convolution layer uses a jump structure and superimposes the corresponding feature layers to obtain high-dimensional feature information to optimize the discriminator, so as to solve the problem that the discriminator has difficulty in judging whether the generated super-resolution image is real.

[0021] 3. Different network compatibility

[0022] The present invention uses simulated images for network training. During the training process, the imaging principles in the image generation process between low-resolution image pairs can be learned. Super-resolution images of different categories from the simulated data set can be predicted. The prediction results are accurate, have strong network generalization, and high network compatibility.

[0023] The beneficial effects of the present invention are:

[0024] This invention integrates physical imaging models and deep learning networks. Compared with traditional deep learning methods, it does not need to collect a large number of real microscopic images, and does not require image alignment and other tasks, which reduces experimental costs and realizes a low-cost, high-quality and high-efficiency super-resolution confocal microscopic image prediction method. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is the overall flow chart of the present invention;

[0026] Figure 2 The illumination system and detection system of the microscopic imaging model of the present invention;

[0027] Figure 3 is a generator network of the present invention;

[0028] Figure 4 is the discriminator network of the present invention;

[0029] Figure 5 The comparison results of the method of the present invention are as follows;

[0030] Figure 6 This is a numerical distribution diagram of the structural similarity between 200 network input images and output images of the method of the present invention. DETAILED DESCRIPTION

[0031] To make the above-mentioned objects, features and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation examples. The specific implementation examples include the following:

[0032] like Figure 1 As shown in FIG, this embodiment shows the overall flow chart of the microscopic image super-resolution method based on mathematical imaging theory and generative adversarial networks, which mainly includes four parts: data set preparation, image augmentation, network training, and network output. Figure 2 As shown in the figure, the illumination system and detection system of the microscopic imaging model are established respectively, and the point spread function of the microscopic imaging model and the confocal microscopic imaging model are derived by combining the Richards-Wolf vector diffraction integral and confocal imaging theory. The specific derivation process is as follows:

[0033] like Figure 2 As shown in (a), the incident light is focused onto the illumination focal plane by objective lens 1. The electric field component of the incident light is expressed in cylindrical coordinates (ρ, z) indicates that the light beam propagates along the positive direction of the z axis. In the focal plane, that is, z = 0, any point P near the focus (polar coordinates are r P ,θ P , The Cartesian component of the electric field at is expressed as:

[0034]

[0035] in

[0036] cosε=cosθcosθ P +sinθsinθ P cos(φ-φ P ),

[0037] Where α is the half-angle of the objective lens aperture, α = arcsin (NA / n), n is the refractive index of the imaging medium, and A is a constant. Therefore, the excitation point spread function can be expressed as

[0038] PSF exc =|E x | 2 +|E y | 2 +|E z | 2 .

[0039] The point spread function of a confocal microscopy system is affected by both the excitation light and the detection light and is expressed as the product of the excitation point spread function and the detection point spread function. Figure 2(b) The imaging model of the detection system of the microscopic imaging system. The light emitted from the light source (spherical coordinates φ1, θ1) is focused on the planar detector (spherical coordinates φ2, θ2 of the center point of the detector) after passing through the lens 2 and the objective 2. Unlike the wide-field microscopic imaging process, a pinhole is used to achieve point detection in the confocal microscopic imaging process, so the effect of the pinhole needs to be considered when calculating the final point spread function of the detector. The Cartesian components of the electric field of the detector plane in the focal region are represented as:

[0040]

[0041] where (r d ,φ d ,z d ) represents the coordinates of the detector plane in the cylindrical coordinate system, k d is the wave number, and k d = 2π / λ. (p x ,p y ,p z ) T represents the Cartesian components of the electric dipole moment. The expressions of the functions K and O are as follows:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] where (φ1, θ1) and (φ2, θ2) are the spherical coordinates of the excitation point and the detection point, respectively, α2 = arcsin(NA / n2), and n2 is the refractive index of the imaging medium of the detection region. The point spread function of the detection point PSF det , i.e., the point spread function of the wide-field imaging system, can be represented as:

[0049] PSF WF = PSF det = |E dx | 2 + |E dy | 2 + |E dz | 2 .

[0050] The point spread function of the confocal system can be represented as

[0051]

[0052] Where p(s) represents the transmission function of the pinhole.

[0053] To make our simulated images more similar to real images captured by a microscope, our imaging model requires the addition of noise during low-resolution imaging. Literature suggests that additive Gaussian white noise best matches the noise conditions in real imaging situations and can effectively restore the real image. Therefore, we added Gaussian white noise to our widefield imaging model for simulation.

[0054] Based on the point spread functions of the wide-field microscopy system and confocal microscopy system derived above, the MCF-7 breast cancer cell images in the open source image dataset BBBC021v were downloaded as the original dataset. Simulated wide-field microscopy images and confocal microscopy images were obtained through deconvolution calculation and used as the dataset for network training.

[0055] A widefield microscopy image and a confocal microscopy image, generated from the same image using different point spread functions, are used as a low-resolution image pair without the need for image registration. The image generation conforms to the physical imaging principles of real microscopy. The simulated images undergo image segmentation, rotation, and flipping before being fed into the subsequent network training.

[0056] During the network training phase, the improved generative adversarial network proposed in this paper is used to complete image training and prediction. Compared with the traditional generative adversarial network, the improved generative adversarial network redefines the structure of the generator and discriminator. Figure 3 and Figure 4 The generator and discriminator structures of the generative adversarial network proposed in this invention are shown in Figure 1. The convolutional layer parameters are given in the form of ksn, where k is the convolution kernel size, s is the step size, and n represents the number of feature maps.

[0057] The generator uses a four-layer convolutional network with a skip architecture to capture image details and further optimizes the target image using a mean squared error loss function. To provide rich feature information for the training model, 16 residual modules are designed, with skip connections effectively avoiding the extra information loss caused by pooling and upsampling. The discriminator network utilizes 10 convolutional layers. The image convolution layers use a skip architecture and stack corresponding feature layers to obtain high-dimensional feature information to optimize the discriminator, addressing the discriminator's difficulty in determining the authenticity of the generated super-resolution image. The trained network can input a low-resolution image and generate a high-resolution predicted image. Figure 5The comparison results of the method of the present invention are shown in Figure (a) as the network input image, Figure (b) as the network output image, and Figure (c) as the true value. It can be seen from the figure that the resolution of the network output image is significantly improved compared with the input image, and is close to the true value image. Figure 6 This figure shows the distribution of structural similarity values ​​for 200 network input and output images using the method of the present invention. The hollow circles represent the input images, and the solid circles represent the output images. As can be seen from the figure, the structural similarity values ​​of the output images are significantly improved compared to the input images, verifying the effectiveness of the method of the present invention.

[0058] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It is apparent that various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the present invention. Thus, the present invention is intended to encompass such modifications and variations as long as they fall within the scope of the claims and their equivalents.

Claims

1. A microscopic image super-resolution method based on an imaging model and an improved generative adversarial network, characterized in that: The method trains a generative adversarial network by building simulated images generated by a physical imaging model to achieve super-resolution microscopic image prediction. The steps are as follows: Step 1: Based on the physical imaging process, establish widefield microscopy and confocal microscopy imaging system models and derive the point spread functions of the two microscopy imaging systems. The specific derivation process is as follows: The incident light is focused onto the illumination focal plane through objective lens 1, and the electric field component of the incident light is expressed in cylindrical coordinates. Indicates that the light beam propagates along the positive direction of the z axis. At the focal plane, that is, z = 0, any point P near the focus (polar coordinates are r P ,θ P , ) is expressed as: in cosε=cosθcosθ P +sinθsinθ P cos(φ-φ P ), Where α is the half-angle of the objective lens aperture, α = arcsin(NA / n), n is the refractive index of the imaging medium, and A is a constant. Therefore, the excitation point spread function is expressed as: PSF exc =|E x | 2 +|E y | 2 +|E z | 2 . The point spread function of the confocal microscopy system is affected by both the excitation light and the detection light and is expressed as the product of the excitation point spread function and the detection point spread function. The light emitted from the light source (spherical coordinates φ1, θ1) is focused onto the array detection plane (spherical coordinates φ2, θ2 of the detector center point) after passing through lens 2 and objective lens 2. Unlike the widefield microscopy process, the confocal microscopy process requires the use of a pinhole to achieve point detection. Therefore, the effect of the pinhole is taken into account when calculating the final detection point spread function. The Cartesian component of the electric field on the detector plane in the focused area is expressed as: Among them, (r d ,φ d ,z d ) represents the detector plane coordinate in the cylindrical coordinate system, k d is the wave number, and k d =2π / λ, (p x ,p y ,p z ) T The expressions of the Cartesian components of the electric dipole moment, function K and function O are: Where (φ1, θ1) and (φ2, θ2) are the spherical coordinates of the excitation point and the detection point, respectively. α2 = arcsin(NA / n2), n2 is the refractive index of the imaging medium in the detection area, and the detection point spread function (PSF) is det , that is, the point spread function of the wide-field imaging system is expressed as: PSF WF =PSF det =|E dx | 2 +|E dy | 2 +|E dz | 2 . The point spread function of the confocal system is expressed as follows, taking into account the illumination point spread function, the detection point spread function and the pinhole: Among them, p(s) represents the transmission function of the pinhole; Step 2: Using the imaging system point spread function established in step 1, deconvolution calculations are performed on the open-source image dataset to obtain simulated wide-field microscopy images and confocal microscopy images, which serve as the low-resolution image pairs required for network training. Step 3: During the network training phase, an improved super-resolution generative adversarial network is designed by combining the U-Net structure and the residual network. Step 4: Train the generator and discriminator until the loss function is minimized, completing network training. At this point, a low-resolution wide-field microscopic image is input and a predicted high-resolution microscopic image is output.

2. The microscopic image super-resolution method based on an imaging model and an improved generative adversarial network according to claim 1, characterized in that: In step 3, the improved generative adversarial network combines U-Net and residual network to redefine the generator and discriminator in the generative adversarial network; the generator uses a 16 residual module structure composed of a 4-layer convolutional network and a jump connection structure to collect image details, and uses two sub-pixel convolutional networks to enlarge the input 128×128 pixel low-resolution wide-field image to a 512×512 pixel prediction image, and further optimizes the target image in combination with the mean square error loss function; the discriminator includes 10 convolutional layers, each convolutional layer is batch normalized and activated by ReLU; the number of feature maps is increased from 64 to 2048, and then reduced to 512; finally, the output tensor image is input into the flattening layer and the fully connected layer, activated by the Sigmoid activation function, and the final discrimination result is obtained; The discriminator is optimized by deepening the CNN layer and fusing feature maps to solve the problem that the discriminator has difficulty in judging whether the generated predicted image is real.

Images