A Microscope Image Deblurring Method Based on Adversarial Networks and Iterative Filters

Abstract

This application relates to a microscope image deblurring method based on adversarial networks and iterative filters. This method utilizes deep learning to learn the relationship between deblurring in out-of-focus and fully focused images under an optical microscope, thereby achieving deblurring of data acquired through optical microscopy. The iterative filtering approach effectively removes localized blur under the microscope, solving the problems of image blurring and incomplete detail in optical microscope images.

Description

Technical Field

[0001] This application relates to the fields of digital signal processing, image processing, machine learning, optical imaging, and biology. Background Technology

[0002] Optical microscopes are important and routine equipment for microscopic detection and analysis, and are widely used in fields such as biology, medicine, geology, materials, and textiles. Their price is much lower than that of electron microscopes, so they are favored by researchers. However, the resolution of optical microscopes is limited by factors such as the wavelength of light and the direction of the lens, making it impossible to observe finer cellular structures. Furthermore, due to the unevenness of the sample surface, problems such as local blurring and loss of detail often occur.

[0003] With the development of deep learning in the field of computer vision, deep learning has been widely applied to image super-resolution reconstruction tasks and has achieved excellent reconstruction results. Today, deep learning-based SISR methods have become mainstream. From the initial algorithms based on Convolutional Neural Networks (CNNs) to the use of continuously proposed new network structures, including Deep Residual Networks (ResNets), Recursive Neural Networks (RNNs), Densely Connected Convolutional Networks (DenseNets), and Generative Adversarial Networks (GANs), they have all shown good performance, making the reconstructed HR images more realistic.

[0004] Therefore, this application proposes a method that combines super-resolution algorithm and deblurring algorithm to solve the problems of local blurring and incomplete details in optical microscopes. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this application provides a microscope image deblurring method based on adversarial networks and iterative filters. This method can effectively solve the problems of local blurring and loss of detail caused by the unevenness of the surface of the observed sample under an optical microscope.

[0006] This application provides a method for deblurring microscope images based on adversarial networks and iterative filters, comprising the following steps:

[0007] Biological samples were collected using an optical microscope as the initial sample set: due to the uneven surface of the biological samples, the initial sample set contained locally blurred images.

[0008] To obtain a full-focus sample set, the initial sample set is manually segmented to crop out the clearly focused areas, thereby obtaining the full-focus sample set.

[0009] Gaussian blur downsampling of the full-focus sample set: The full-focus sample set is downsampled using the Gaussian blur method to obtain an image with a defocused blur effect, which is used as the defocused sample set. The defocused sample set and the full-focus sample set are normalized and used as the deblurred dataset;

[0010] Bicubic downsampling of the full-focus sample set: The full-focus image is downsampled using the Bicubic interpolation method to obtain a low-resolution full-focus sample set. The low-resolution full-focus sample set and the full-focus sample set are normalized and used as the super-resolution dataset.

[0011] Model training: The iterative filter network is trained using the deblurred dataset to obtain the deblurred model, and the adversarial network is trained using the super-resolution dataset to obtain the super-resolution model.

[0012] Optimization of images under an optical microscope: Deblurring model and super-resolution model are used to optimize images under an optical microscope;

[0013] In one embodiment, using an optical microscope to collect biological samples as an initial sample set further includes:

[0014] The experimental setup employed a Nikon Eclipse Ti2-U inverted optical microscope. Compared to a conventional microscope, its objective and eyepiece positions are reversed, resulting in a longer working distance and allowing for the observation of thicker samples. Magnification was set to 100x, using Nikon CFI60 series objectives, a 488nm laser source, and a DS-Ri2 camera interface. The initial sample set was obtained using NIS-Elements microscope image processing software.

[0015] In one implementation, obtaining a full-focus sample set by region segmentation of the initial sample set further includes:

[0016] The images in the initial sample set are manually cropped, and the in-focus areas are cropped using a 1024*728 matrix to serve as the full-focus sample set.

[0017] In one implementation, Gaussian blur downsampling of the full-focus sample set further includes:

[0018] Gaussian blur downsampling is applied to the manually cropped full-focus sample set. Gaussian blur is a commonly used image processing method that achieves a blurring effect similar to defocusing by applying Gaussian filtering to the image. The Gaussian blur formula is as follows:

[0019]

[0020] Where G(x,y) represents the Gaussian kernel value at (x,y), and σ represents the standard deviation of the Gaussian kernel. In this method, the Gaussian kernel size is set to (9,9), and σ is 1.5. The downsampled images are used as the out-of-focus sample set. The out-of-focus sample set and the full-focus sample set are normalized and used as the deblurred dataset.

[0021] In one implementation, bicubic downsampling of the full-focus sample set further includes:

[0022] Bicubic interpolation is used to downsample a full-focus image, resulting in a low-resolution full-focus sample set. Bicubic interpolation is a commonly used image processing method that can perform operations such as image enlargement, reduction, and rotation, as well as image denoising and sharpening, by interpolating the image. The Bicubic downsampling formula is as follows:

[0023]

[0024] Where P(x,y) represents the value of pixel (x,y) in the downsampled image, and Q... m+i,n+j w represents the values ​​of the 4×4 pixels closest to (x,y) in the original image. i,j (x,y) represents the weight of the Bicubic interpolation kernel at position (i,j) of (x,y). There are various forms of Bicubic interpolation kernels; a commonly used form is as follows:

[0025]

[0026] During bicubic downsampling, this formula is applied to each pixel in the downsampled image to calculate its corresponding pixel value in the original image. After bicubic downsampling, the resulting low-resolution image is used as a low-resolution full-focus sample.

[0027] The obtained low-resolution full-focus samples and full-focus samples are normalized and used as the super-resolution dataset.

[0028] In one implementation, model training includes:

[0029] An iterative filtering network is trained using a deblurring dataset to obtain a deblurring deep learning model. The architecture of the deep learning model is as follows: Figure 2 As shown:

[0030] The deblurring deep learning model used in this method is the IFAN (Iterative Filter Adaptive Network), whose main architecture is based on the Unet network architecture, including a feature extractor, a reconstructor, and the intermediate IFAN module. The feature extractor extracts out-of-focus features and inputs them into the Iterative Filter Adaptive Network (IFAN). The Iterative Filter Adaptive Network obtains deblurring features by predicting spatially varying deblurring containers, and then transmits the deblurring features to the reconstructor, which restores the full-focus image.

[0031] A super-resolution model is obtained by training an adversarial network using a super-resolution dataset. The super-resolution deep learning model used in this method is ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks), and the generator framework of the network is as follows: Figure 3 As shown:

[0032] The low-resolution image first passes through a Conv layer and is then input into RRDBNet. After continuous feature extraction, it passes through another Conv layer, and then the original low-resolution images are superimposed and input into an Upsample layer. After two Conv layers, the final high-resolution image is obtained. The framework of the RRDB structure in the ESRGAN network is as follows: Figure 4 As shown:

[0033] The RRDBNet layer consists of 23 RRDB blocks, and each RRDB block consists of three Dense Blocks, which in turn consist of 5 Conv layers and 4 ReLU activation layers densely connected together. This structure ensures that the ESRGAN network can extract enough details to generate more realistic high-resolution images.

[0034] In one embodiment, optimizing images under an optical microscope further includes:

[0035] First, a high-resolution image is generated using a trained super-resolution model, and then the image is optimized for deblurring using a deblurring model.

[0036] The technical solution provided in this application has the following beneficial effects: it can solve the problems of insufficient magnification and local blurring under optical microscopes. The technology used in this method can automatically learn more useful features from the original data, and it will not fail when faced with unfamiliar data, and can still achieve the effect of restoring high-definition images.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0038] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0039] Figure 1 This is a schematic flowchart of a microscope image deblurring method shown in Embodiment 1 of this application;

[0040] Figure 2 This is an architecture diagram of the deep learning-based IFAN network shown in Embodiment 1 of this application;

[0041] Figure 3 This is a diagram of the generative network architecture of the ESRGAN network based on deep learning, as shown in Embodiment 1 of this application;

[0042] Figure 4 This is an RRDB block architecture diagram of the ESRGAN network shown in Embodiment 1 of this application;

[0043] Figure 5 This is a microscopic illustration of the deblurring effect shown in Embodiment 1 of this application; Detailed Implementation

[0044] Preferred embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0045] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0046] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0047] Example 1

[0048] Deep learning-based image processing technology has several advantages over traditional image processing techniques. Firstly, it offers higher accuracy: deep learning models can automatically learn more effective feature representations, thus improving the accuracy of super-resolution algorithms. Secondly, it provides better generalization ability: deep learning models can better adapt to different scenes and data, exhibiting better generalization capabilities. Thirdly, it enables faster processing speeds: deep learning models can utilize hardware acceleration such as GPUs to improve processing speed, thus completing image super-resolution tasks more quickly. Fourthly, it handles complex scenes better: deep learning models can handle more complex image scenes, such as those with low light, noise, and blur.

[0049] Therefore, this application proposes a microscopic deblurring method that performs end-to-end model training on the sample to obtain a super-resolution and deblurring model under the microscope, which has excellent generalization ability.

[0050] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0051] Figure 1 This is a schematic flowchart of a microscope image deblurring method shown in Embodiment 1 of this application;

[0052] Figure 2 This is an architecture diagram of the deep learning-based IFAN network shown in Embodiment 1 of this application;

[0053] Figure 3 This is a diagram of the generative network architecture of the ESRGAN network based on deep learning, as shown in Embodiment 1 of this application;

[0054] Figure 4 This is an RRDB block architecture diagram of the ESRGAN network shown in Embodiment 1 of this application;

[0055] Figure 5 This is a microscopic illustration of the deblurring effect shown in Embodiment 1 of this application;

[0056] See Figure 1

[0057] An embodiment of this application provides a method for predicting the cadmium content in rice seedlings, comprising the following steps:

[0058] S101 uses an optical microscope to collect biological samples as the initial sample set:

[0059] The experimental setup employed a Nikon Eclipse Ti2-U inverted optical microscope. Compared to a conventional microscope, its objective and eyepiece positions are reversed, resulting in a longer working distance and allowing for the observation of thicker samples. Magnification was set to 100x, using Nikon CFI60 series objectives, a 488nm laser source, and a DS-Ri2 camera interface. The initial sample set was obtained using NIS-Elements microscope image processing software.

[0060] The embodiments of this application do not impose limitations on object distance, exposure time, and focal length, but are set according to actual conditions. The scope of the invention regarding object distance, exposure time, and focal length all fall within the protection scope of this application.

[0061] S102 performs region segmentation on the initial sample set to obtain the full-focus sample set:

[0062] The initial samples obtained in the above steps are preprocessed by manually segmenting the initial sample set to crop out the clearly focused areas, thereby obtaining a full-focus sample set.

[0063] S103 performs Gaussian blur downsampling on the full-focus sample set to obtain the out-of-focus sample set. After normalizing the full-focus and out-of-focus samples, it serves as the deblurred dataset.

[0064] Gaussian blur downsampling is performed on the manually cropped full-focus sample set. Gaussian blur is a commonly used image processing method that can achieve a blur effect similar to defocusing by applying Gaussian filtering to the image.

[0065] For example, the Gaussian blur formula is as follows:

[0066]

[0067] Where G(x,y) represents the value of the Gaussian kernel at (x,y), and σ represents the standard deviation of the Gaussian kernel. For example, the Gaussian kernel size in this application is (9,9), and σ is 1.5. The downsampled images are used as the out-of-focus sample set.

[0068] S104 performs Bicubic downsampling on the full-focus sample set to obtain a low-resolution full-focus image sample set. The low-resolution full-focus image sample set and the full-focus image sample set are then normalized and used as the super-resolution dataset.

[0069] Bicubic interpolation is used to downsample full-focus images to obtain a low-resolution full-focus sample set. Bicubic (Bicubic Interpolation) is a commonly used image processing method that can achieve image enlargement, reduction, rotation, and other operations, as well as image denoising and sharpening effects by interpolating images.

[0070] For example, the Bicubic downsampling formula is as follows:

[0071]

[0072] Where P(x,y) represents the value of pixel (x,y) in the downsampled image, and Q... m+i,n+j w represents the values ​​of the 4×4 pixels closest to (x,y) in the original image. i,j (x,y) represents the weight of the Bicubic interpolation kernel at position (i,j) of (x,y). There are various forms of Bicubic interpolation kernels; a commonly used form is as follows:

[0073]

[0074] During bicubic downsampling, this formula is applied to each pixel in the downsampled image to calculate its corresponding pixel value in the original image. After bicubic downsampling, the resulting low-resolution image is used as a low-resolution full-focus sample.

[0075] S105 uses a deblurred dataset to train the iterative filtering network, obtaining a deblurred model for the optical microscope. It then uses a super-resolution dataset to train the generative network, obtaining a super-resolution model for the optical microscope.

[0076] The datasets obtained in the above steps are used to train the generative network and the adaptive iterative network, respectively, to obtain the super-resolution model and the deblurring model under the microscope.

[0077] S106 uses the obtained deblurring model and super-resolution model to optimize the images under the optical microscope:

[0078] First, a high-resolution image is generated using a trained super-resolution model. Then, a deblurring model is used to optimize the image for deblurring. From Figure 5 As can be seen, the left side is the image observed under a microscope, and the right side is the optimized image. It is evident that the optimized image under the microscope shows a significant improvement.

[0079] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for deblurring optical microscope images based on adversarial networks and iterative filters, comprising the following steps: Biological samples were collected using an optical microscope as the initial sample set: images of biological samples were obtained under the microscope using an optical microscope at a magnification of 100. Due to the uneven surface of the biological samples, the initial sample set contained locally blurred images. To obtain a full-focus sample set, the initial sample set is manually segmented to crop out the clearly focused areas, thereby obtaining the full-focus sample set. Gaussian blur downsampling of the full-focus sample set: The full-focus sample set is downsampled using the Gaussian blur method to obtain the image with out-of-focus blur effect, which is used as the out-of-focus sample set. The out-of-focus sample set and the full-focus sample set are normalized and used as the deblurred dataset. Bicubic downsampling of the full-focus sample set: The full-focus image is downsampled using the Bicubic interpolation method to obtain a low-resolution full-focus sample set. The low-resolution full-focus sample set and the full-focus sample set are normalized and used as the super-resolution dataset. Model training: An iterative filtering network is trained using a deblurring dataset to obtain a deblurring model, and an adversarial network is trained using a super-resolution dataset to obtain a super-resolution model. The deblurring deep learning model is the IFAN (Iterative Filter Adaptive Network), whose main architecture is based on the Unet network architecture. It includes a feature extractor, a reconstructor, and an intermediate IFAN module. The feature extractor extracts out-of-focus features and inputs them into the Iterative Filter Adaptive Network (IFAN). The Iterative Filter Adaptive Network obtains deblurred features by predicting spatially varying deblurred containers, and then transmits the deblurred features to the reconstructor, which restores the full-focus image. A super-resolution model is obtained by training an adversarial network using a super-resolution dataset. The super-resolution deep learning model is ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks). The network's generator works as follows: the low-resolution image first passes through a Conv layer and is then input into RRDBNet. After continuously extracting features, it passes through another Conv layer, and then the original low-resolution image is superimposed and input into the Upsample layer. After two Conv layers, the final high-resolution image is obtained. The RRDB structure in the ESRGAN network consists of 23 RRDB blocks in the RRDBNet layer, and each RRDB block consists of three Dense Blocks. Each Dense Block is composed of 5 Conv layers and 4 ReLU activation layers densely connected. This structure ensures that the ESRGAN network can extract enough details to generate more realistic high-resolution images. Optimization of images under an optical microscope: Deblurring model and super-resolution model are used to optimize images under an optical microscope.

2. The optical microscope image deblurring method according to claim 1, characterized in that: Using optical microscopy to collect biological samples as an initial sample set includes: The experimental setup used a Nikon Eclipse Ti2-U inverted optical microscope. Compared to a regular microscope, the objective and eyepiece are inverted, resulting in a longer working distance and allowing observation of thicker samples. The magnification was 100x, the objective lens was a Nikon CFI60 series lens, the laser source was 488nm, and a DS-Ri2 camera interface was used. The initial sample set was obtained using NIS-Elements microscope image processing software.

3. The optical microscope image deblurring method according to claim 1, characterized in that: The process of segmenting the initial sample set to obtain the full-focus sample set includes: The images in the initial sample set are manually cropped, and the in-focus areas are cropped using a 1024*728 matrix to serve as the full-focus sample set.

4. The optical microscope image deblurring method according to claim 3, characterized in that: Gaussian blur downsampling of the full-focus sample set includes: Gaussian blur downsampling is applied to the manually cropped full-focus sample set. Gaussian blur is a commonly used image processing method that achieves a defocused blur effect by applying Gaussian filtering to the image. The Gaussian blur formula is as follows: Where G(x,y) represents the value of the Gaussian kernel at (x,y), and σ represents the standard deviation of the Gaussian kernel. In this method, the size of the Gaussian kernel is set to (9,9), and σ is 1.

5. The images obtained after downsampling are used as the out-of-focus sample set.

5. The optical microscope image deblurring method according to claim 4, characterized in that: After Gaussian blur downsampling the full-focus sample set, the following steps are also included: The out-of-focus sample set and the full-focus sample set are normalized and used as the deblurred dataset.

6. The optical microscope image deblurring method according to claim 1, characterized in that: Bicubic downsampling of the full-focus sample set includes: Bicubic interpolation is used to downsample a full-focus image to obtain a low-resolution full-focus sample set. Bicubic (Bicubic Interpolation) is a commonly used image processing method that can perform image magnification, reduction, rotation, denoising, and sharpening effects by interpolating the image. The Bicubic downsampling formula is as follows: Where P(x,y) represents the value of pixel (x,y) in the downsampled image, and Q... m+i,n+j w represents the values ​​of the 4×4 pixels closest to (x,y) in the original image. i,j (x,y) represents the weight of the Bicubic interpolation kernel at position (i,j) of (x,y); the Bicubic interpolation kernel has the following form: When performing Bicubic downsampling, this formula is applied to each pixel in the downsampled image to calculate its corresponding pixel value in the original image; after completing Bicubic downsampling, the resulting low-resolution image is used as a low-resolution full-focus sample.

7. The optical microscope image deblurring method according to claim 6, characterized in that: After performing Bicubic downsampling on the full-focus sample set, the following steps are also included: The obtained low-resolution full-focus samples and full-focus samples are normalized and used as the super-resolution dataset.

8. The optical microscope image deblurring method according to claim 1, characterized in that: Optimization of images under an optical microscope includes: The image evaluation metric is PSNR (Peak Signal-to-Noise Ratio). The formulas for PSNR are as follows: Signal-to-Noise Ratio (SNR) and SSIM (Structural Similarity). Among them, a max The mean squared error (MSE) represents the maximum pixel value in the image, and its formula is as follows: Where a i and Let represent the i-th pixel of the real image and the reconstructed image, respectively. Minimizing MSE will result in maximizing PSNR. SSIM can be used to measure the structural similarity between images, and its formula is as follows: in, and Representing I y and The mean, and For I y and The variance, and For I y and The covariance, where C1 and C2 represent constant terms.

Images