A Fourier ptychography microscopic image denoising method based on convolutional neural network

Abstract

The present invention discloses a Fourier ptychography microscopic image denoising method based on a convolutional neural network, comprising the following steps: S1, using a Fourier ptychography microscopic system to image a human blood cell sample to produce a data set; S2, building a convolutional neural network; S3, feeding the input of the training set into the encoding module, denoising module and decoding module of the convolutional neural network, and using the characteristics of the convolutional neural network to suppress the expression of the noise information channel; S4, using the L1 loss function to iteratively optimize the convolutional neural network repeatedly to complete the training of the convolutional neural network; S5, using the convolutional neural network to denoise the actually collected Fourier microscopic image to obtain a high-quality reconstructed image. The present invention combines the intensity map and phase map obtained by the Fourier ptychography microscopic system, utilizes the advantages of the deep learning method, improves the noise problem of the traditional reconstruction algorithm, and provides an accurate and scientific algorithm for high-quality medical imaging.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer microscopy imaging, and particularly relates to a Fourier ptychographic microscopy image denoising method based on a convolutional neural network. Background Technique

[0002] Fourier ptychographic microscopy is a recently emerging computational imaging technology that realizes high-resolution image reconstruction under a large field of view. It has been widely used in digital pathology, cell counting, surface defect detection, etc., and has received extensive attention and research. This method combines the related concepts of phase retrieval and synthetic aperture. It has multiple modes such as bright-field imaging, dark-field imaging, and phase-contrast imaging, can achieve quantitative phase imaging, can obtain information about cell structure and position, etc., and solves the problem that transparent samples cannot be imaged. This technology uses a programmable light-emitting diode (LED) array to illuminate the sample from different angles to achieve frequency-domain scanning, collects the intensity information of low-resolution images as the spatial-domain amplitude constraint, and uses a circular pupil function as the Fourier-domain constraint. Based on these two constraints, iterative calculations are performed repeatedly to obtain the high-resolution complex amplitude information of the sample.

[0003] Although Fourier ptychographic microscopy has many advantages, traditional Fourier ptychographic microscopy still faces many challenges in the imaging and reconstruction processes, such as various aberrations inevitably introduced by lenses, position errors that cannot be avoided due to the process of the light-emitting diode array, various inevitable system noises in the imaging system, and the low speed of the reconstruction algorithm. In addition, compared with the intensity map of the sample, its phase map is more significantly affected by system noise.

[0004] With the development of deep learning, using convolutional neural networks to process medical images has become a research hotspot. With the huge computing power of computers, neural networks are trained based on a large number of data sets, and medical images are optimized in an automated manner, effectively removing the background noise of the images, improving the image clarity, obtaining richer detailed information, enhancing the image imaging quality, and obtaining more accurate and reliable results. Summary of the Invention

[0005] Aiming at the above deficiencies in the prior art, a Fourier ptychographic microscopy image denoising method based on a convolutional neural network provided by the present invention solves the problems of large noise in traditional reconstruction algorithms and low medical imaging quality.

[0006] In order to achieve the above invention purpose, the technical solution adopted by the present invention is: a Fourier ptychographic microscopy image denoising method based on a convolutional neural network, including the following steps:

[0007] S1. Use a Fourier ptychographic microscopy system to image a human blood cell sample and make a data set, including a training set and a validation set;

[0008] S2. Build a convolutional neural network, including an encoding module, a denoising module, and a decoding module;

[0009] S3. Feed the input of the training set into the encoding module, denoising module, and decoding module of the convolutional neural network. Utilize the characteristics of the convolutional neural network to automatically learn the target information and noise information of the image, assign a larger weight to the target information channel of the feature map, and assign a very small weight to the noise information channel of the feature map, thereby suppressing the expression of the noise information channel;

[0010] S4. Use the L 1 loss function to iteratively optimize the convolutional neural network. After each iteration, use the validation set to test the trained model. Adopt the peak signal-to-noise ratio and structural similarity as evaluation metrics. When there are no obvious changes in the loss function, peak signal-to-noise ratio, and structural similarity, complete the training of the convolutional neural network;

[0011] S5. Use the convolutional neural network to denoise the actually collected Fourier microscopy images and obtain high-quality reconstructed images.

[0012] Furthermore: The specific steps of step S1 are as follows:

[0013] S11. Use a Fourier ptychography microscopy system to image a human blood cell sample and collect 400 intensity maps under a 20x objective lens;

[0014] S12. Randomly select high-resolution intensity maps as intensity maps and phase maps respectively, and obtain 1600 groups of high-resolution complex amplitudes through random combination as the true value data of the neural network; Combine the Fourier ptychography imaging simulation algorithm and add random noise to obtain low-resolution intensity maps;

[0015] S13. Use the traditional Fourier ptychography microscopy reconstruction algorithm to iterate the low-resolution intensity maps once to obtain low-resolution complex amplitudes as the input of the neural network;

[0016] S14. Crop the input and true value data during the simulation process to obtain 25,600 groups of input and true value data. Randomly select 23,040 groups of input and true value data as the training set, and use the remaining 2,560 groups of input and true value data as the validation set.

[0017] Furthermore: The encoding module in step S2 consists of 4 convolutional pooling blocks, which perform feature extraction and downsampling on the input image.

[0018] Furthermore: The convolutional pooling block includes a 1*1 convolutional layer, a batch normalization layer, an activation function layer, a 3*3 convolutional layer, a batch normalization layer, an activation function layer, and a max pooling layer connected in sequence.

[0019] Furthermore, in step S2, the denoising module includes a global average pooling layer, a 1×1 convolutional layer, an activation function layer, a 1×1 convolutional layer, and an activation function layer connected in sequence.

[0020] Furthermore, in step S2, the input of the denoising module is the feature maps of different levels obtained before the pooling operation of the encoding module. Global pooling operations are respectively performed on the feature maps of different levels, and the channel features of the feature maps of different levels are fused. The fused channel feature information is optimized through a convolutional layer and an activation function, and then segmented according to the number of channels of the feature maps of different levels. The segmented channel feature information is multiplied with the corresponding level feature maps respectively to achieve the reallocation of channel feature weights, thereby highlighting the expression of information channels and suppressing the expression of noise information.

[0021] Furthermore, in step S2, the decoding module consists of 4 convolutional upsampling blocks, which extract features from the high-level feature maps, fuse with the same-level feature maps optimized by the denoising module, and jointly perform upsampling operations to finally obtain a high-resolution complex amplitude, including intensity information and phase information.

[0022] Furthermore, the convolutional upsampling block includes an upsampling layer, a 1×1 convolutional layer, a batch normalization layer, an activation function layer, a 3×3 convolutional layer, a batch normalization layer, and an activation function layer connected in sequence.

[0023] The beneficial effects of the present invention are as follows: The present invention uses a Fourier ptychography microscopy system to image a human blood cell sample, obtains the intensity map and phase map of the sample, inputs the two as dual-channel features into a convolutional neural network for image denoising, thereby removing background noise and improving the imaging quality. The multi-level channel attention mechanism convolutional neural network proposed by the present invention combines the intensity map and phase map obtained by the Fourier ptychography microscopy system, utilizes the advantages of deep learning methods, improves the noise problem of traditional reconstruction algorithms, and provides an accurate and scientific algorithm for high-quality medical imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is the overall flowchart of the present invention;

[0025] Figure 2 is the structural diagram of the convolutional neural network built by the present invention;

[0026] Figure 3 is the structural diagram of the encoding module of the convolutional neural network in the present invention;

[0027] Figure 4 is the structural diagram of the denoising module of the convolutional neural network in the present invention;

[0028] Figure 5This is the structural diagram of the decoding module of the convolutional neural network in the present invention. Detailed implementation manners

[0029] The following describes the detailed implementation manners of the present invention to facilitate those skilled in the art of this technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the detailed implementation manners. For those of ordinary skill in the art of this technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions created using the concept of the present invention are within the scope of protection.

[0030] As Figure 1 shown, a Fourier ptychographic microscopy image denoising method based on a convolutional neural network includes the following steps:

[0031] S1. Use a Fourier ptychographic microscopy system to image a human blood cell sample and make a data set, including a training set and a validation set. The specific steps are as follows:

[0032] S11. Use a Fourier ptychographic microscopy imaging system to image a human blood cell sample and collect 400 intensity maps under a 20x objective lens.

[0033] S12. Randomly select high-resolution intensity maps as intensity maps and phase maps respectively, and obtain 1,600 groups of high-resolution complex amplitudes through random combination as the true value data of the neural network; combine the Fourier ptychographic imaging simulation algorithm and add random noise to obtain low-resolution intensity maps.

[0034] S13. Use the traditional Fourier ptychographic microscopy reconstruction algorithm to iterate the low-resolution intensity map once to obtain low-resolution complex amplitudes as the input of the neural network.

[0035] S14. Crop the input and true value data in the simulation process to obtain 25,600 groups of input and true value data. Randomly select 23,040 groups of input and true value data as the training set, and use the remaining 2,560 groups of input and true value data as the validation set.

[0036] S2. Build a convolutional neural network, including an encoding module, a denoising module, and a decoding module; as Figure 2As shown, the size of the input image is 192*192. The encoding module consists of 4 convolutional pooling blocks, which mainly perform feature extraction and downsampling on the input image; the input of the denoising module is the feature maps of different levels obtained before the pooling operation of the encoding module. Global pooling operations are respectively performed on the feature maps of different levels, and the channel feature information of the feature maps of different levels is fused. The fused channel feature information is optimized through a convolutional layer and an activation function, and then segmented according to the number of channels of the feature maps of different levels. The segmented channel feature information is multiplied with the corresponding-level feature maps. Through the deep learning framework, the information channel and the noise channel features are adaptively learned. The multiplication operation realizes the reallocation of the feature channel weights, thereby highlighting the expression of the information channel and suppressing the expression of the noise information; the decoding module consists of 4 convolutional upsampling blocks, which mainly perform feature extraction on the high-level feature maps and fuse them with the same-level feature maps optimized by the denoising module, and jointly perform upsampling operations to finally obtain a high-resolution complex amplitude, including intensity information and phase information.

[0037] S3. Feed the input of the training set into the encoding module, denoising module, and decoding module of the convolutional neural network. Utilize the characteristics of the convolutional neural network to automatically learn the target information and noise information of the image, assign a larger weight to the target information channel of the feature map, and assign a very small weight to the noise information channel of the feature map, thereby suppressing the expression of the noise information channel;

[0038] S4. Use the L 1 loss function to iteratively optimize the convolutional neural network. After each iteration, use the validation set to test the trained model. Adopt the peak signal-to-noise ratio and structural similarity as evaluation indicators. When there is no obvious change in the loss function, peak signal-to-noise ratio, and structural similarity, complete the training of the convolutional neural network;

[0039] S5. Use the convolutional neural network to denoise the actually collected Fourier microscopy image to obtain a high-quality reconstructed image.

[0040] Figure 3 Figure 15 is a schematic diagram of one convolutional pooling block in the encoding module of the convolutional neural network, including two convolutional layers, two batch normalization layers, two activation function layers, and one max pooling layer. First, use a 1*1 convolutional layer to perform a dimensionality increase operation on the input feature map. Except for the first convolutional pooling block that raises the two-channel input dimension to 64, the other convolutional pooling blocks double the number of channels of the input feature map, and then use a 3*3 convolutional layer to perform feature extraction on the input image. In addition, batch normalization layers are added to accelerate the training and convergence speed of the convolutional neural network and prevent overfitting, and activation function layers are added to increase the non-linearity of the convolutional neural network. Finally, use the max pooling layer to perform downsampling operations on the feature map. After four convolutional pooling blocks, 4 different-level feature maps can be generated.

[0041] Figure 4 It is a schematic diagram of the denoising module in the convolutional neural network, which shows the optimization of the feature map by the denoising module at a certain level. It includes a global average pooling layer, two 1×1 convolutional layers, and two activation functions. The feature maps at different levels obtained by the encoding module are respectively passed through the global average pooling layer, so that the height and width of the feature maps at different levels both become 1, without changing the number of channels. Therefore, the feature maps at different levels can be concatenated in the channel dimension. The fused features jointly enter the optimization module composed of the convolutional layer and the activation function layer, so as to realize the feature sharing of the feature maps at different levels in the channel dimension. The optimized features are sliced according to the original number of channels of the feature maps at different levels, and then multiplied by the input feature maps at the corresponding levels to realize the reallocation of the weights of the information channels and the noise channels, and further realize the prominent expression of the information channels and suppress the display of the information in the noise channels, so as to achieve the purpose of removing background noise.

[0042] Figure 5 It is a schematic diagram of a convolutional upsampling block of the decoding module in the convolutional neural network, which includes an upsampling layer, two convolutional layers, two batch normalization layers, and two activation function layers. The upsampling layer upsamples the feature map at a higher level and fuses it with the feature map at the corresponding level optimized by the denoising module, and they jointly enter the convolutional layer, the batch normalization layer, and the activation function layer to realize the dimensionality reduction of the feature map and feature extraction. When the feature map at the lowest level is fused, no upsampling is performed anymore, and the 1×1 convolutional layer is used to reduce the dimension to two channels, which is used as the final output of the convolutional neural network, that is, the high-resolution intensity map and the phase map.

Claims

1. A Fourier ptychography microscopic image denoising method based on a convolutional neural network, characterized in that, it includes the following steps: S1. Use a Fourier ptychography microscopic system to image a human blood cell sample, obtain an intensity map and a phase map, and make a data set, including a training set and a validation set; S2. Build a convolutional neural network, including an encoding module, a denoising module and a decoding module; S3. Feed the input of the training set into the encoding module, denoising module and decoding module of the convolutional neural network. Utilize the characteristics of the convolutional neural network to automatically learn the target information and noise information of the image, assign a first weight to the target information channel of the feature map, and assign a second weight to the noise information channel of the feature map, thereby suppressing the expression of the noise information channel; S4. Use L 1 The loss function is used to iteratively optimize the convolutional neural network. After each iteration, the trained model is tested using the validation set, and the peak signal-to-noise ratio and structural similarity are used as evaluation metrics. When there are no obvious changes in the loss function, peak signal-to-noise ratio, and structural similarity, the training of the convolutional neural network is completed; S5. Use the convolutional neural network to denoise the actually collected Fourier microscopic image to obtain a high-quality reconstructed image; In step S2, the encoding module is composed of 4 convolutional pooling blocks, which perform feature extraction and downsampling on the input image; the convolutional pooling block includes a 1*1 convolutional layer, a batch normalization layer, an activation function layer, a 3*3 convolutional layer, a batch normalization layer, an activation function layer and a max pooling layer connected in sequence; The denoising module in step S2 includes a global average pooling layer, a 1*1 convolutional layer, an activation function layer, a 1*1 convolutional layer and an activation function layer connected in sequence; the input of the denoising module in step S2 is the feature maps of different levels obtained before the pooling operation of the encoding module. Perform global pooling operations on the feature maps of different levels respectively, and fuse the channel features of the feature maps of different levels. The fused channel feature information is optimized by a convolutional layer and an activation function, and then segmented according to the number of channels of the feature maps of different levels. The segmented channel feature information is multiplied with the corresponding level feature maps respectively to realize the reallocation of channel feature weights, thereby highlighting the expression of the information channel and suppressing the expression of the noise information; The decoding module in step S2 is composed of 4 convolutional upsampling blocks, which perform feature extraction on the high-level feature maps, fuse them with the optimized same-level feature maps of the denoising module, and jointly perform upsampling operations to finally obtain a high-resolution complex amplitude, including intensity information and phase information; the convolutional upsampling block includes an upsampling layer, a 1*1 convolutional layer, a batch normalization layer, an activation function layer, a 3*3 convolutional layer, a batch normalization layer and an activation function layer connected in sequence.

2. The Fourier ptychography microscopic image denoising method based on a convolutional neural network according to claim 1, characterized in that, the specific steps of step S1 are: S11. Use a Fourier ptychography microscopic imaging system to image a human blood cell sample, and collect 400 intensity maps under a 20x objective lens; S12. Randomly select high-resolution intensity maps as intensity maps and phase maps respectively, and obtain 1600 groups of high-resolution complex amplitudes through random combination as the true value data of the neural network; combine the Fourier ptychography imaging simulation algorithm and add random noise to obtain low-resolution intensity maps; S13. Use the traditional Fourier ptychography microscopic reconstruction algorithm to iterate the low-resolution intensity map once to obtain a low-resolution complex amplitude as the input of the neural network; S14. Crop the input and ground truth data during the simulation process to obtain 25,600 groups of input and ground truth data. Randomly select 23,040 groups of input and ground truth data as the training set, and use the remaining 2,560 groups of input and ground truth data as the validation set.

Images