Step-by-step supervised training generation type deep learning single-frame super-resolution imaging algorithm
Through step-by-step supervised training and data augmentation technology, data dependence and artifact problems in deep learning super-resolution microscopy are solved, image quality and model performance are improved, and high-quality super-resolution imaging with a small amount of training data is achieved. .
Patent Information
- Application Number
- CN202411181526.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
AI Technical Summary
In super-resolution microscopy, deep learning models require a large number of high-quality training data sets, but data sets are difficult to obtain and model output may produce artifacts, affecting the accurate observation of biological structures.
Using step-by-step supervised training generated deep learning single-frame super-resolution imaging algorithm, through independent training of encoder and decoder, GT images are used multiple times for loss calculation, combining data enhancement and specific loss function design, reducing the amount of training data and suppress artifact generation.
This improves the usage rate of training data, enhances the authenticity of the output image and model performance, reduces the dependence on the amount of training data, reduces artifact generation, and improves the quality of super-resolved images.
Smart Images

Figure CN119067850A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deep learning super-resolution imaging, and specifically refers to a step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm. Background Art
[0002] Super-resolution microscopy is one of the important tools in biomedical research, but existing super-resolution microscopy technologies have mutually restrictive factors such as resolution, imaging speed, imaging field of view, phototoxicity and photobleaching, and the cost of purchasing and building the optical system is also relatively high. Therefore, in recent years, super-resolution microscopy technology based on deep learning has shown great application potential. Its later use cost is low, and the image processing speed is fast. It can perform super-resolution imaging on large quantities of images in a short time. However, in the context of super-resolution microscopy, the training data set required for deep learning is not so easy to obtain, and the quality of the training data set directly affects the quality of the trained neural network model. At the same time, the model output may also produce artifacts, which will also lead to inaccurate observation of biological structures.
[0003] Therefore, based on the problems existing in this technology, it is urgent to develop a method that can ensure the quality of the output super-resolution image as much as possible while using a small amount of training data. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm, in which the encoder and decoder are trained independently in each epoch of training, and the parameters are updated independently. In this process, the model uses the GT image twice for loss calculation, which greatly improves the utilization rate of the GT image and the multi-channel feature map output by the encoder, thereby reducing the dependence on the amount of training data. In addition, through a series of data enhancement, network module and structure and loss function design, the authenticity of the output image is guaranteed and the generation of artifacts is suppressed.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:
[0006] A step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm, the algorithm comprising the following steps:
[0007] S1 prepares the training data set. First, obtain the low-resolution and high-resolution paired data sets required for training the neural network, that is, the diffraction-limited low-resolution microscopic image as input, and the corresponding super-resolution microscopic image obtained by the traditional super-resolution method as the true value (GT) (such as STED, SIM and other methods, it is necessary to ensure that the corresponding images have the same field of view (ROI)). Here, only one or two images are collected for different ROIs, and 8-9 different signal-to-noise ratios are collected for each ROI. The field of view of the collected image here should preferably not be less than 32.5 microns. In the present invention, a data set collected based on SIM technology is used as a training set and a validation set. The data set uses SIM to reconstruct a super-resolution image, so the spatial size of the GT image is twice that of the input image. This needs to be taken into consideration when designing the neural network decoder structure. At the same time, since the images collected by the microscope are generally large in size, and considering the limitations of computer video memory and the diversity of the training set, it is generally necessary to randomly crop the large images into small images as the final training set. Here, two pairs of images, 9 different signal-to-noise ratios (SNRs), and a total of 18 pairs of large images (502x502 and 1004x1004) were randomly cropped into 640 pairs of small images (128x128 and 256x256) for training.
[0008] S2 Data preprocessing. Before the image data is formally input into the neural network training, some preprocessing and data enhancement are required. First, read the image data and convert it into a 32-bit integer grayscale image. Then flip the image pair horizontally or vertically. At the same time, use the "elastic deformation" function to randomly modify the tiny details in the image pair, and set two parameters to change the amplitude and smoothness of the image deformation. By controlling the parameter strategy to maximize the diversity of the image and not too distorted, the random number is also used to control whether the image is elastically deformed. If the transformation is performed, first generate a displacement field for the GT image and obtain the size of the image, then perform Gaussian smoothing on the displacement field, and then generate the coordinate grid of the image, then add the smoothed displacement field to the corresponding coordinates to generate new mapping coordinates, and then resample the original image according to the mapping coordinates, use the linear interpolation method, and finally convert the converted image array back to the image format. At this point, the data required for training is ready.
[0009] S3 network training. After initializing the codec and training and validating the data set, a series of hyperparameters need to be set to control the network training process. The Adam optimizer is used to iteratively optimize the training data. The loss function of the encoder uses the Gram matrix function. The loss function of the decoder is weighted by multiple loss functions, namely the MSE mean square error loss function, the SSIM structural similarity loss function, and the perceptual loss function. The pre-trained model used by the perceptual loss function is vgg16. During the training process, the weights of the codec are saved at the same time every 50 epochs, and the loss changes of the encoder on the training set and validation set are recorded in the form of text and drawn curves.
[0010] S4 super-resolution image reconstruction. Use a uniform light field to illuminate the sample to obtain a wide-field diffraction-limited fluorescence image of the sample, and input the optimal network parameters trained in step S3 to obtain a super-resolution reconstructed image output by the network. The image preprocessing process is the same as that mentioned in S2.
[0011] Preferably, the present invention adds a global residual connection to the overall structure of the U-Net class, further increasing the stability of the model. At the same time, the GT image is used twice for loss calculation in each epoch of training, which greatly improves the utilization rate of the GT image and the multi-channel feature map output by the encoder, and enhances the performance of the encoder.
[0012] Preferably, step S3 includes the following sub-steps:
[0013] S3.1. Construct the submodules required for the network. The overall neural network consists of multiple self-attention mechanism modules, channel attention mechanism modules, and spatial attention mechanism modules.
[0014] S3.2, encoder and decoder construction. The encoder downsamples the input image four times in total. If the input is a GT image, an additional downsampling is required. The feature map in the downsampling process needs to be spliced into the corresponding upsampling process of the decoder. The decoder needs to restore the input feature map to the normal spatial size through four upsampling. Finally, the original input image needs to be adjusted to the normal spatial size through a convolutional layer and added to the final output to form a global residual connection.
[0015] Preferably, the random cropping used in step S1 requires first selecting a random coordinate point on the original large image, and then cropping with the coordinate point as the center according to the spatial size to be cropped, and the cropped image is randomly flipped horizontally and vertically.
[0016] Preferably, in step S3, the wide-field low-resolution image and the super-resolution true value image are simultaneously input into the encoder for downsampling to obtain the corresponding multi-channel feature map, and then the loss calculation and back propagation are performed to update the encoder parameters. Then, the feature map of the low-resolution image is input into the decoder for upsampling to obtain the final super-resolution reconstructed image, and the loss calculation and back propagation are performed on the image and the original super-resolution true value image, at which time the encoder parameters are fixed and only the decoder parameters are updated.
[0017] Preferably, if the image pair in step S1 is acquired using SIM technology, it is necessary to add multiple original SIM images to obtain a single low-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 (a) A diffraction-limited wide-field low-resolution microtubule image, which serves as the input of the model.
[0019] Figure 1 (b) is a super-resolution microtubule image obtained using the traditional SIM reconstruction algorithm, which is used as the true value during model training, namely the GT map.
[0020] Figure 1 (c) is a super-resolution microtubule microscopy image inferred using the deep learning algorithm model proposed in the present invention, that is, the model output image.
[0021] Figure 1 (d)(e) are super-resolution microtubule microscopy images inferred using DFCAN and DFGAN proposed in the paper Evaluation and development of deep neural networks for image super-resolution in optical microscopy (using the pre-trained models provided by the authors).
[0022] Figure 2 Comparison of the effects of elastic deformation used. (a) is the original GT image, and (b) is the GT image obtained after elastic deformation. DETAILED DESCRIPTION
[0023] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0024] S1. First obtain the low-resolution and high-resolution paired data sets required for training the neural network, that is, the diffraction-limited low-resolution microscopic image as input, and the corresponding super-resolution microscopic image obtained by the traditional super-resolution method as the true value (GT) (such as STED, SIM and other methods, it is necessary to ensure that the corresponding images have the same field of view (ROI)). Here, only one or two images are collected for different ROIs, and 8-9 different signal-to-noise ratios are collected for each ROI. The field of view of the collected images here should preferably not be less than 32.5 microns. The present invention uses a data set collected based on SIM technology as a training set and a validation set. The data set uses SIM to reconstruct the super-resolution image, so the spatial size of the GT image is twice that of the input image, which needs to be paid attention to when designing the neural network decoder structure. At the same time, since the images collected by the microscope are generally large in size, and considering the limitations of computer video memory and the diversity of the training set, it is generally necessary to randomly crop the large images into small images as the final training set. Here, two pairs of images, 9 different signal-to-noise ratios (SNRs), and a total of 18 pairs of large images (502x502 and 1004x1004) were randomly cropped into 640 pairs of small images (128x128 and 256x256) for training.
[0025] S2. Before the image data is formally input into the neural network training, some preprocessing and data enhancement are required. First, read the image data and convert it into a 32-bit integer grayscale image. Then flip the image pair horizontally or vertically. At the same time, use the "elastic deformation" function to randomly modify the tiny details in the image pair, and set two parameters to change the amplitude and smoothness of the image deformation. By controlling the parameter strategy to maximize the diversity of the image and not too distorted, the random number is also used to control whether the image is elastically deformed. If the transformation is performed, first generate a displacement field for the GT image and obtain the size of the image, then perform Gaussian smoothing on the displacement field, and then generate the coordinate grid of the image, and then add the smoothed displacement field to the corresponding coordinates to generate new mapping coordinates, and then resample the original image according to the mapping coordinates, use the linear interpolation method, and finally convert the converted image array back to the image format. At this point, the data required for training is ready.
[0026] S3. After initializing the codec and training and validating the data set, a series of hyperparameters need to be set to control the training process of the network. The Adam optimizer is used to iteratively optimize the training data. The loss function of the encoder uses the Gram matrix function. The loss function of the decoder uses a variety of weighted loss functions, namely the MSE mean square error loss function, the SSIM structural similarity loss function, and the perceptual loss function. The pre-trained model used for the perceptual loss function is vgg16. During the training process, the weights of the codec are saved at the same time every 50 epochs, and the loss changes of the encoder on the training set and validation set are recorded in the form of text and curve graphs.
[0027] S3.1. Construct the submodules required for the network. The overall neural network consists of multiple self-attention mechanism modules, channel attention mechanism modules, and spatial attention mechanism modules.
[0028] S3.2, encoder and decoder construction. The encoder downsamples the input image four times in total. If the input is a GT image, an additional downsampling is required. The feature map in the downsampling process needs to be spliced into the corresponding upsampling process of the decoder. The decoder needs to restore the input feature map to the normal spatial size through four upsampling. Finally, the original input image needs to be adjusted to the normal spatial size through a convolutional layer and added to the final output to form a global residual connection.
[0029] S4, using a uniform light field to illuminate the sample, obtain a wide-field diffraction-limited fluorescence image of the sample, and input the optimal network parameters trained in step S3 to obtain a super-resolution reconstructed image output by the network. The image preprocessing process is the same as that mentioned in S2.
[0030] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.
Claims
1. A step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm, comprising, characterized in that: The following steps are involved: S1 prepares training data sets: obtains paired low-resolution and high-resolution data sets required for training neural networks; S2 data preprocessing: read image data and convert it into a 32-bit integer grayscale image, flip the image pair horizontally or vertically, randomly modify the details of the image pair and change the amplitude and smoothness of the image deformation; S3 network training: After initializing the encoder, decoder, and training and validation data sets, iteratively optimize the training data; S4 super-resolution image reconstruction: Use a uniform light field to illuminate the sample to obtain a wide-field diffraction-limited fluorescence image of the sample. After the data preprocessing in S2, the optimal network parameters trained in step S3 are input to obtain the super-resolution reconstructed image output by the network.
2. According to claim 1, a step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm is characterized by: The corresponding images in the data set in S1 have the same ROI, and the number of different ROIs collected is ≤ two; the number of single ROI signal-to-noise ratio collection is 8 to 9, and the field of view of the collected image is greater than 32.5 μm;.
3. According to claim 2, a step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm is characterized by: The large images in the dataset S1 have a low resolution of 502×502 and a high resolution of 1004×1004, which are randomly cropped into small images of 128×128 and 256×256 for training; The random cropping in S1 includes selecting random coordinate points on the original large image, cropping with the coordinate points as the center according to the spatial size to be cropped, and randomly flipping the cropped image horizontally and vertically.
4. The step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm according to claim 1, characterized in that: S2 includes randomly modifying details in the image pair using an elastic deformation function, wherein the random modification controls whether the image is elastically deformed by a random number; When performing elastic deformation, a displacement field is generated for the image, the size of the image is obtained, the displacement field is Gaussian smoothed, the coordinate grid of the image is generated, the Gaussian smoothed displacement field is added to the corresponding coordinates, new mapping coordinates are generated, the original image is resampled according to the mapping coordinates, and the transformed image array is converted back to image format using linear interpolation.
5. The step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm according to claim 1, characterized in that: The Adam optimizer is used in S3 to iteratively optimize the training data, wherein the loss function of the encoder uses the Gram matrix function, and the loss function of the decoder uses multiple loss functions for weighting, and the loss changes of the encoder on the training set and the validation set are recorded in the form of text and curve graphs.
6. The step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm according to claim 5, characterized in that: S3 includes the following sub-steps: S3.
1. Construct submodules required for the network: The overall neural network consists of multiple self-attention mechanism modules, channel attention mechanism modules, and spatial attention mechanism modules; S3.2, encoder and decoder construction: The encoder downsamples the input image four times in total, and performs an additional downsampling when the input is a GT image. The feature map in the downsampling process is spliced into the corresponding upsampling process of the decoder; The decoder restores the input feature map to its normal spatial size by upsampling four times. Finally, the original input image is resized through a convolutional layer and added to the final output to form a global residual connection.
7. The step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm according to claim 6, characterized in that: In S3, the wide-field low-resolution image and the super-resolution true value image are simultaneously input into the encoder for downsampling to obtain the corresponding multi-channel feature map, and then the loss calculation and back propagation are performed to update the parameters of the encoder. The feature map of the low-resolution image is then input into the decoder for upsampling to obtain the final super-resolution reconstructed image, and the loss calculation and back propagation are performed on it and the original super-resolution true value image. At this time, the parameters of the encoder are fixed and only the parameters of the decoder are updated.
8. The step-by-step supervised training generative deep learning single-frame super-resolution imaging algorithm according to claim 3, characterized in that: The acquisition of the image pair in step S1 includes acquisition using SIM technology, and when acquiring using SIM technology, multiple original SIM images are added together to obtain a single low-resolution image.
