Real-time denoising method for optical microscopic imaging, optical microscopic imaging system, device and medium

Abstract

The application relates to a real-time denoising method, an optical microscopic imaging system, an equipment and a medium for optical microscopic imaging, which comprises the following steps: acquiring image data output by an optical microscopic imaging system; adopting a denoising model based on a U-Net network to perform denoising processing on the image data, so as to obtain an optical microscopic imaging result after denoising; wherein a training pair is generated by adopting a space-time redundancy sampling strategy, and the denoising model is trained by adopting a self-supervised training mode. Compared with the prior art, the application has the advantages of good real-time performance, good scene generalization and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical microscopic imaging, and in particular to a real-time denoising method for optical microscopic imaging, an optical microscopic imaging system, a device and a medium. BACKGROUND

[0002] Due to the differences in various microscopic techniques, tissue types, cell types and probes, optical microscopic imaging is highly diversified, and different imaging scenarios have different requirements. For example, structural imaging usually aims to maintain higher spatial resolution, while functional imaging seeks higher temporal resolution and imaging depth. In addition, the real-time nature of imaging experiments is crucial, as scientists and researchers need high signal-to-noise ratio images for timely image quality assessment or to implement closed-loop feedback. The diversity of observation methods and observation objects, as well as the need for real-time acquisition of high signal-to-noise ratio images, poses great challenges to denoising methods.

[0003] Since the advent of deep learning denoising methods, self-supervised methods have become an essential image enhancement step due to their superior denoising performance over traditional methods and the absence of the need for high signal-to-noise ratio ground truth. These methods exploit the significant spatio-temporal redundancy in fluorescence time-lapse imaging, i.e., repeated information between temporally adjacent frames and spatially adjacent pixels, to extract clean image features from the redundant information using neural networks for denoising.

[0004] Existing self-supervised denoising methods can be divided into two categories: neighbor-based methods and blind spot-based methods. In neighbor-based methods, one method samples the time interval between frames in multiple frames of original data acquired continuously to construct a training framework using temporal redundancy; another method samples the spatial pixel interval in a single frame to construct a training framework using spatial redundancy. However, these methods have strict requirements for high temporal or high spatial resolution of imaging, otherwise they may produce over-smoothing and artifacts. Blind spot-based methods rely on the degradation of blind spots in the center frame, using unobstructed pixels from the center frame and adjacent frames to reconstruct blind spots. This pixel-level reconstruction task requires a large amount of training time. In addition, since this method can only obtain the denoising result of the center frame after inputting multiple frames of images, it also takes longer to process complete imaging data, which cannot meet the real-time requirement.

[0005] Both of the two self-supervised learning denoising methods provide an improvement in image signal-to-noise ratio for optical microscopy. However, there are still many challenges, including dependence on high temporal or spatial resolution, demand for high computational resources, which seriously hinder the realization of real-time denoising in different optical microscopy scenes. Most of the existing image enhancement self-supervised learning methods use large parameter networks or use complex training strategies, which will cause a lot of space and time consumption. The three-dimensional U-shaped network receives a large amount of three-dimensional data and performs complex three-dimensional convolution operations, which makes its space consumption huge. The Transformer network has huge time consumption due to the large amount of calculation required by the self-attention mechanism. The complex training strategy proposed for a specific imaging scene will cause it to be unable to adapt to flexible scene changes, further limiting the application potential. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the prior art and provide a real-time denoising method for optical microscopy with good real-time performance and scene generalization, an optical microscopy system, an apparatus and a medium.

[0007] The purpose of the present application can be achieved by the following technical solutions:

[0008] According to a first aspect of the present application, a real-time denoising method for optical microscopy is provided, comprising:

[0009] obtaining image data output by an optical microscopy system;

[0010] using a denoising model based on a U-Net network to perform denoising processing on the image data to obtain a denoised optical microscopy result; wherein a training pair is generated using a spatio-temporal redundancy sampling strategy, and the denoising model is trained in a self-supervised manner.

[0011] Preferably, the training pair is generated using a spatio-temporal redundancy sampling strategy, wherein the temporal sampling process is specifically:

[0012] for a training stack with HxWxT pixels, wherein H, W and T are the height, width and length of the input image stack, respectively;

[0013] the training stack is divided into an input stack Xi and a target stack Yi in a sliding window manner in time, and the size of the input stack and the target stack is HxWxC, the initial frame differs by S frames, and C and S are the window width and step length of the sliding window, respectively, and S≤C.

[0014] Preferably, the training pair is generated using a spatio-temporal redundancy sampling strategy, wherein the spatial sampling process is specifically:

[0015] The input stack Xi and the target stack Yi are respectively spatially sampled by using a sparse convolution calculation mode for each image to obtain two sub-stacks G1(Xi), G2(Xi) of the input stack Xi and two sub-stacks G1(Yi), G2(Yi) of the target stack Yi;

[0016] The same window of the input stack Xi and the target stack Yi uses the same random seed, so that the original pixel positions of the sub-stacks G1(Xi) and G1(Yi) are the same, the original pixel positions of the sub-stacks G2(Xi) and G2(Yi) are the same, the sub-stacks G1(Xi) and G2(Yi) satisfy a spatiotemporal adjacent relationship, and the sub-stacks G2(Xi) and G1(Yi) satisfy a spatiotemporal adjacent relationship;

[0017] Xi and G1(Xi) are selected as the input stack, and Yi and G2(Yi) are selected as the target stack to generate a training pair for self-supervised training of the denoising model.

[0018] Preferably, spatial sampling is performed by using a sparse convolution calculation mode, and adjacent pixels are selected by spatial sampling, and specific parameter settings include:

[0019] The window size of the sparse convolution calculation is a x a pixels, the step size is b, the convolution kernel of each window is randomly selected from a mask cell list, and the mask cell is a a binary matrix, in which the values of two elements are 1 and used to select adjacent pixel positions; wherein a and b are self-defined according to the requirements of processing speed and the spatial resolution of the image.

[0020] Preferably, the U-Net network model is a lightweight U-Net network model, and the lightweight U-Net network model reduces the number of down-sampling times to twice and cancels the jump connection of the topmost layer based on the original U-Net network model.

[0021] Preferably, the lightweight U-Net network model includes two encoding blocks and two decoding blocks, and specifically includes:

[0022] Two encoding blocks are used to reduce the dimension, and each encoding block is composed of a convolution layer followed by a BatchNorm, a ReLU and a maximum pooling layer;

[0023] Two decoding blocks are used to restore the input dimension, and each decoding block includes a nearest neighbor interpolation followed by a convolution layer, a BatchNorm and a ReLU;

[0024] At each resolution depth, a jump connection is used to connect the output of the second encoding block to the input of the same level encoding block.

[0025] According to a second aspect of the present application, there is provided an optical microscopic imaging system, wherein a de-noising module is installed in the graphic processor of the system, and the de-noising module adopts any of the methods.

[0026] Preferably, the images before and after de-noising are synchronously displayed on the display of the system.

[0027] According to a third aspect of the present application, there is provided an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods.

[0028] According to a fourth aspect of the present application, there is provided a computer readable storage medium, wherein a computer program is stored on the medium, and the program is executed by a processor to implement any of the methods.

[0029] Compared with the prior art, the present application has the following beneficial effects:

[0030] (1) According to the characteristics that the signals between time adjacent frames are similar and the noises are independent, the sub-images obtained by time-space adjacent down-sampling in the original imaging data can be used as the training pairs of the neural network, and the de-noising network parameters can be optimized. The training method does not have strict requirements on the time resolution and spatial resolution of the original image, and thus can be applied to various imaging scenes.

[0031] (2) The present application regards the spatial sampling as a down-sampling, and makes a lightweight improvement on the U-Net network by reducing the number of down-sampling to twice, thereby reducing the de-noising network parameter quantity and the calculation amount, and achieving a de-noising processing speed exceeding the imaging speed. The real-time capability is not dependent on high-performance hardware, and the de-noising can be performed on mainstream consumer-grade graphics cards. Meanwhile, the skip connection in the top layer of the U-Net network is cancelled, which can effectively suppress the noise features of the high layer, and further improve the de-noising effect.

[0032] (3) The microscopic imaging system of the present application realizes continuous data acquisition, multi-thread parallel processing, and real-time synchronous display of images before and after de-noising through a double-queue data inflow and outflow mode comprising data acquisition, multi-thread parallel processing, and real-time synchronous display. The microscopic imaging real-time de-noising can be realized on mainstream consumer-grade graphics cards without relying on high-performance hardware. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flowchart of the method of the present application;

[0034] Figure 2A schematic diagram of a real-time denoising method for optical microscopic imaging according to the present application; wherein (a) is the input and output of the network after time sampling and spatial sampling of the collected image, and the network parameters are optimized through scale constraint and spatio-temporal self-supervision, and (b) is a schematic diagram of a spatial sampling strategy;

[0035] Figure 3 A network structure diagram; wherein (a) is a classic two-dimensional U-Net, (b) is a three-dimensional U-Net, and (c) is a super-light two-dimensional U-Net according to the present application;

[0036] Figure 4 The enhanced neuron segmentation effect in the embodiment; wherein (a)-(f) are the image denoising and neuron segmentation result comparison of the method of the present application and other methods, (g) is a quantitative evaluation index of neuron segmentation, and (h) is a comparison of denoising segmentation results on three specific neurons;

[0037] Figure 5 A schematic diagram of recovering high temporal precision neuron activity using the present application in the embodiment; wherein (a) is the image and signal difference before and after the voltage imaging data is processed by the method in this paper, (b) is a detailed presentation of the time range marked by the red dashed line in (a) after enlargement, and (c) is the Pearson correlation coefficient of the signal obtained after processing by the method of the present application and the true value of the electrophysiological signal;

[0038] Figure 6 A schematic diagram of volume structure imaging results in the embodiment; wherein (a) is a comparison before and after the skin optical coherence tomography volume imaging is enhanced using the method of the present application, and (b) is a single slice display of the results in (a). DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0040] EMBODIMENT

[0041] As shown in the drawings, Figure 1 The present embodiment provides a real-time denoising method for optical microscopic imaging, comprising:

[0042] acquiring image data output by an optical microscopic imaging system;

[0043] The image data is denoised by using a denoising model based on a U-Net network to obtain a denoised optical microscopic imaging result; wherein, a training pair is generated by using a spatiotemporal redundancy sampling strategy, and the denoising model is trained by using a self-supervised training method.

[0044] Next, the method of the embodiment is described in detail.

[0045] (1) Spatiotemporal redundancy sampling

[0046] After the collected images are time sampled and space sampled, the images are used as the input and output of the denoising model based on the U-Net network, and the parameters of the denoising model are optimized by scale constraint and spatiotemporal self-supervision, and the specific process is as follows:

[0047] The spatiotemporal redundancy sampling strategy is used to generate a training pair. For a training stack with HxWxT pixels, wherein H, W and T are the height, width and length of the input image stack, respectively;

[0048] First, time sampling is performed, and the training stack is divided into an input stack Xi and a target stack Yi in a sliding window manner in time. The size of the input stack and the target stack is HxWxC, the initial frame is S frames apart, and C and S are the window width and step length of the sliding window, respectively, and S≤C;

[0049] For the input stack Xi and the target stack Yi, each image is space sampled in a manner similar to neural network sparse convolution calculation to obtain two sub-stacks G1(Xi), G2(Xi) of the input stack Xi and two sub-stacks G1(Yi), G2(Yi) of the target stack Yi;

[0050] The same window of the input stack Xi and the target stack Yi uses the same random seed, so the original pixel positions of the sub-stacks G1(Xi) and G1(Yi) are the same, the original pixel positions of G2(Xi) and G2(Yi) are the same, G1(Xi) and G2(Yi) satisfy the spatiotemporal adjacent relationship, and G2(Xi) and G1(Yi) satisfy the spatiotemporal adjacent relationship;

[0051] Xi and G1(Xi) are selected as the input stack, and Yi and G2(Yi) are selected as the target stack for self-supervised training of the denoising network.

[0052] Then, space sampling is performed to select adjacent pixels. For space sampling, the image randomly selects a mask from a mask unit list, and splits the complete image into two sub-images according to the mask marking.

[0053] The spatial sampling can be realized by sparse convolution or other manners. In order to accelerate the sampling speed, the sparse convolution manner is selected in the embodiment, and the parameter settings include that the window size of the sparse convolution calculation is a x a pixels, the step length is b, the convolution kernel of each window is randomly selected from a mask unit list, and the mask unit is a 2 x a binary matrix, in which the values of two elements are 1 and used for selecting adjacent pixel positions. The parameters a and b can be set according to the processing speed requirement and the spatial resolution of the image.

[0054] In the embodiment, the window size of the sparse convolution calculation is 2 x 2 pixels, the step length is 2, the convolution kernel of each window is randomly selected from 16 mask unit lists, and the mask unit is a 2 x 2 binary matrix, in which the values of two elements are 1 and used for realizing the selection of adjacent pixel positions.

[0055] (2) Denoising model based on U-Net network

[0056] In the embodiment, the U-Net network selects a lightweight U-Net network.

[0057] Similar to the original U-Net model, the lightweight U-Net network in the embodiment also uses an encoder-decoder architecture and a skip connection. Two encoding blocks are used to reduce the dimension, each encoding block is composed of a 3 x 3 convolution layer, followed by a BatchNorm, a ReLU and a 3 x 3 maximum pooling layer, and two decoding blocks are used to restore the input dimension, each decoding block contains a nearest neighbor interpolation, followed by a 3 x 3 convolution layer, a BatchNorm and a ReLU. At each resolution depth, the number of channels in each feature map is 64, 64, 64 respectively, and the output of the second encoding block is connected to the input of the same level encoding block using a skip connection.

[0058] Compared with the original U-Net network, the lightweight U-Net network of the application has the following two changes:

[0059] Firstly, the number of down-sampling is reduced to twice, and the down-sampling can increase the receptive field, so that the convolution kernel can extract features on a larger image range, but with the increase of the number of down-sampling, the parameter amount of the network will also increase sharply, and the spatial sub-sampler of the application can be regarded as once down-sampling, which has the function of expanding the receptive field, so that the number of down-sampling of the U-net can be reduced to reduce the network.

[0060] Second, the topmost skip connection is removed. The skip connection can fuse the bottom features and the high-level features. However, in the denoising task, the high-level features are noise and need to be suppressed. Therefore, removing the topmost skip connection is beneficial to improve the denoising performance. By minimizing the network depth, the number of channels of each feature map, and the number of skip connections, in the embodiment, the network only trains 0.13M parameters.

[0061] The embodiment also provides an optical microscopic imaging system, a denoising module is loaded in a graphics processor of the system, the denoising module uses the method to perform denoising, and the images before and after denoising are synchronously displayed on a display of the system.

[0062] The denoising model is pre-trained by using the self-supervised training method, and the trained denoising model is loaded into the graphics processor (GPU) as the denoising module.

[0063] The original output data stream of the imaging system is input as an input into a to-be-processed queue, the images in the to-be-processed queue are continuously input into the graphics processor in which the denoising model is loaded, the data after denoising is input into a completed queue, and finally the images before and after denoising are synchronously displayed on the display.

[0064] The microscopic imaging system in the embodiment realizes continuous data acquisition, multi-thread parallel processing, and real-time synchronous display of the images before and after denoising by using a double-queue data inflow and outflow mode including data acquisition, multi-thread parallel processing, and real-time synchronous display, and can realize real-time denoising of the microscopic imaging on a mainstream consumer-grade graphics card without relying on high-performance hardware.

[0065] Next, the effect of the denoising method of the application is verified by experiments.

[0066] Experiment 1: Enhancing the neuron segmentation effect

[0067] Background: Fluorescent genetically encoded calcium indicators and two-photon microscopy can simultaneously measure the activity of hundreds to thousands of neurons, which is used to study the correlation between neural activity and neural function. Neuron segmentation is an important step in analyzing the calcium imaging of neuron groups, and noise reduces the number and accuracy of segmentation. The loss of neurons submerged in noise can adversely affect the analysis.

[0068] Data preparation: The Neurofinder dataset is used for verification, and the dataset contains manually labeled segmentation benchmarks. The original data is contaminated by noise and shot noise, as shown in Figure 4 Part (a) of FIG. 1, which seriously damages the morphology and contour of the neurons, which are key features of segmentation.

[0069] Noise processing: The two-photon calcium imaging data is processed by using the application and four other denoising methods, as shown in Figure 4Part (b)-(f) of the middle. After processing by the present invention, the morphology of neurons recovered from noise, cell boundaries were clear, and synapses of neurons were visible in the enlarged area and emitted fluorescence together with neurons. The results of DeepCAD-RT showed smooth transitions, and the synaptic structure was almost indistinguishable from the background. The results of DeepVid and SUPPORT not only failed to observe image details, but also had some artifacts. The results of SRDTrans were clear and sharp, but due to the complex Transformer network, the processing consumed a large amount of time and memory.

[0070] Segmentation analysis: Cellpose was used to segment neurons from five denoising results. SRDTrans segmented the most additional neurons (29), and SUPPORT lost the most neurons (57). The enhanced image of the present invention segmented the most neurons (125), and the least additional neurons (19) were segmented. Enlarged images of segmentation of three denoising methods were selected, as shown in Figure 4 Part (g) in FIG. 6, where only the present invention can completely identify neurons, with a deviation of only 1-2 pixels from the benchmark.

[0071] Performance evaluation: The denoising method of the present invention achieved the highest F1 score (0.64), indicating that the denoising method of the present invention can enhance calcium imaging of neuron populations to facilitate accurate and comprehensive neuron segmentation.

[0072] Results: After processing, the morphology of neurons recovered significantly, cell boundaries were clear, and synapses were visible. The present invention performed best in terms of the number and accuracy of segmented neurons, with the highest F1 score.

[0073] Conclusion: The denoising method of the present invention significantly improves the accuracy and comprehensiveness of neuron segmentation, providing reliable data support for the study of neural activity and neural function.

[0074] Experiment 2: Recovering high temporal precision neuronal activity

[0075] Background: To study the patterns of neuronal activity, high temporal precision recording of activity is crucial. With the development of voltage indicators, this requires higher imaging speed, requiring voltage imaging speed to reach 1 kHz. However, due to the inherent trade-off between imaging speed and the maximum number of photons collected, a reduction in signal-to-noise ratio (SNR) is inevitable. This reduction compromises the recorded neuronal morphology and signals, and neuronal activity signals can be obscured or even overwhelmed by noise. Through denoising, these signals can be recovered, enabling accurate high-precision recording of neuronal activity.

[0076] Data preparation: Wide-field microscope images were used to record the activity of individual QuasAr6a-expressing neurons in layer 2 / 3 of mouse cortex (imaging speed 1 kHz) with simultaneous patch-clamp electrophysiology. The raw data were significantly contaminated by noise, which severely corrupted the spatial footprints and temporal traces of the neurons, as shown in Figure 5 (a) of the partial view.

[0077] Noise processing: The data were enhanced using the present application, and the cell boundaries became identifiable, and the voltage traces emerged from the noise. The fluorescence of 100 randomly selected cytoplasmic pixels was extracted, and it was found that after enhancement by the present application, the voltage fluctuations could be displayed at the single-pixel level, as shown in Figure 5 (b) of the partial view.

[0078] Performance evaluation: The average Pearson correlation coefficient between the voltage traces and the electrophysiological recordings increased from 0.13 to 0.29, as shown in Figure 5 (c) of the partial view.

[0079] Results: After processing by the present application, the morphology and voltage traces of the neurons were restored significantly, the cell boundaries were clear, and the voltage fluctuations were visible. The correlation between the voltage traces and the electrophysiological recordings was significantly improved by the present application, increasing from 0.13 to 0.29. The denoising results of the in vivo population neuron voltage imaging data showed that the neuronal morphological structure was restored, and the voltage transients were preserved.

[0080] Conclusion: The present application can effectively denoise high-time-resolution single-neuron and population neuron voltage imaging data while faithfully restoring voltage transients, providing reliable technical support for high-precision recording of neuronal activity.

[0081] Experiment 3: Volume structure imaging enhancement

[0082] Background: Optical coherence tomography (OCT) is a non-invasive imaging technique widely used in the diagnosis and research of skin diseases. OCT can provide high-resolution two-dimensional and three-dimensional images, revealing the microscopic structure of skin tissue. However, due to the presence of noise during the OCT imaging process, the image quality is severely affected, especially in volumetric imaging, where the noise problem is more prominent. Noise not only reduces the clarity of the image, but also can lead to misjudgment of the skin tissue structure and pathological changes. Therefore, developing an effective denoising method is crucial for improving the quality of OCT volumetric imaging. In skin OCT volumetric imaging, noise mainly comes from several aspects: first, the hardware limitations of the OCT system itself, such as the stability of the light source, the sensitivity of the detector, etc., will introduce noise; second, the influence of ambient light, sample scattering and absorption during the imaging process will also increase noise; finally, algorithm errors and calculation accuracy problems in the data processing process will also lead to the accumulation of noise.

[0083] Conclusion: Figure 6 The results of the volumetric structure imaging using the method of the present application are shown in (a), which is a comparison of the results before and after the enhancement of skin optical coherence tomography volumetric imaging using the method of the present application, and (b) is a single slice display of the results in (a). It can be seen that by using the method of the present application, the noise in volumetric imaging can be effectively removed, and the clarity and resolution of the image can be improved. The experimental results show that this method performs well in terms of image quality improvement and noise level reduction, and has significant potential application value. For example, it helps to improve the resolution of skin OCT volumetric imaging, and can clearly display the fine structure of skin tissue, such as the boundary between the epidermis, dermis and subcutaneous tissue, making the detection of pathological regions more accurate and complete, and the denoised image can better reflect the location and range of the lesion, providing more reliable basis for early diagnosis and treatment of skin diseases.

[0084] Conclusion: The application of the present application in the volumetric time sequence structure and functional imaging of astrocytes shows its scalability in different fluorescence and dimensions, providing reliable technical support for the observation of heterogeneous activities.

[0085] The electronic device of the present application includes a central processing unit (CPU) and a graphics processor (GPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded into a random access memory (RAM) from a storage unit. In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0086] A number of components in the device are connected to the I / O interface, including: input units, such as a keyboard, a mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as a magnetic disk, an optical disk, etc.; and communication units, such as a network card, a modem, a wireless communication transceiver, etc. The communication units allow the device to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0087] The processing units perform the various methods and processes described above. For example, in some embodiments, the methods can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage units. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the device via the ROM and / or the communication units. When the computer program is loaded onto the RAM and executed by the central processing unit (CPU) and the graphics processor (GPU), one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the central processing unit (CPU) and the graphics processor (GPU) can be configured, by way of other any suitable means (e.g., by way of firmware), to perform the methods.

[0088] The functionality described above above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0089] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0090] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage medium would include one or more lines of electrical wire, portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0091] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A real-time denoising method for optical microscopy imaging, characterized in that: include: Acquiring image data output by an optical microscopy imaging system; A denoising model based on a U-Net network was used to denoise the image data and obtain the denoised optical microscopy imaging results. A spatiotemporal redundant sampling strategy was used to generate training pairs, and the denoising model was trained through self-supervised training. The spatiotemporal redundant sampling strategy is used to generate training pairs, wherein the time sampling process is specifically as follows: For a training stack with H × W × T pixels, where H, W, and T are the height, width, and length of the input image stack, respectively; The training stack is divided into an input stack Xi and a target stack Yi in a sliding window manner in time. The size of the input stack and the target stack are both H×W×C. The initial frames differ by S frames. C and S are the window width and step size of the sliding window respectively, and S≤C. The spatial sampling process is specifically as follows: For the input stack Xi and the target stack Yi, each image is spatially sampled using a sparse convolution calculation method to obtain two sub-stacks G1(Xi) and G2(Xi) of the input stack Xi, and two sub-stacks G1(Yi) and G2(Yi) of the target stack Yi; The same random seed is used for the same window of the input stack Xi and the target stack Yi, so the original pixel positions of the substacks G1(Xi) and G1(Yi) are the same, the original pixel positions of G2(Xi) and G2(Yi) are the same, G1(Xi) and G2(Yi) satisfy the spatiotemporal neighbor relationship, and G2(Xi) and G1(Yi) satisfy the spatiotemporal neighbor relationship; Xi and G1(Xi) are selected as the input stack, and Yi and G2(Yi) are selected as the target stack to generate training pairs for self-supervised training of the denoising model.

2. A real-time denoising method for optical microscopy imaging according to claim 1, characterized in that: Sparse convolution is used for spatial sampling to select adjacent pixels. The specific parameter settings include: The window size of the sparse convolution calculation is a×a pixels, with a step size of b. The convolution kernel of each window is randomly selected from the list of mask units. The mask unit is an a×a binary matrix with two elements with the value of 1, which is used to select adjacent pixel positions. Among them, a and b are customized according to the processing speed requirements and the spatial resolution of the image.

3. The real-time denoising method for optical microscopy imaging according to claim 1, characterized in that: The U-Net network model is a lightweight U-Net network model. The lightweight U-Net network model reduces the number of downsampling times to two and cancels the top-level skip connection based on the original U-Net network model.

4. A real-time denoising method for optical microscopy imaging according to claim 3, characterized in that: The lightweight U-Net network model includes two encoding blocks and two decoding blocks, specifically: Use two encoding blocks to reduce the dimensionality. Each encoding block consists of a convolutional layer followed by a BatchNorm, a ReLU, and a max pooling layer. Use two decoding blocks to restore the input dimension. Each decoding block contains a nearest neighbor interpolation followed by a convolution layer, a BatchNorm and a ReLU. At each resolution depth, the output of the second encoder block is connected to the input of the same-level encoder block using skip connections.

5. An optical microscopy imaging system, characterized in that: The graphics processor of the system is equipped with a denoising module, and the denoising module performs denoising using the method according to any one of claims 1 to 4.

6. The system according to claim 5, characterized in that The system displays the images before and after denoising synchronously on a display.

7. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images