Real-time denoising method for optical microscopic imaging, optical microscopic imaging system, equipment and medium

Abstract

The invention relates to a real-time denoising method for optical microscopic imaging, an optical microscopic imaging system, equipment and a medium. The method comprises the following steps: acquiring image data output by the optical microscopic imaging system; performing de-noising processing on the image data by adopting a de-noising model based on a U-Net network to obtain a de-noised optical microscopic imaging result; wherein a training pair is generated by adopting a space-time redundancy sampling strategy, and the denoising model is trained in a self-supervised training mode. Compared with the prior art, the method has the advantages of good real-time performance, good scene generalization and the like.

Description

Technical Field

[0001] The present invention relates to the field of optical microscopy, and in particular to a real-time denoising method, an optical microscopy system, a device and a medium for optical microscopy. Background Art

[0002] Optical microscopy is highly diverse due to the diverse range of microscopy techniques, tissue types, cell types, and probes, with different imaging scenarios presenting distinct requirements. For example, structural imaging typically aims to maintain higher spatial resolution, while functional imaging seeks higher temporal resolution and imaging depth. Furthermore, real-time imaging experiments are crucial, as scientists and researchers require images with high signal-to-noise ratios for timely image quality assessment or closed-loop feedback. The diversity of observation methods and objects, coupled with the need to acquire high signal-to-noise ratio images in real time, poses significant challenges to denoising methods.

[0003] Since the introduction of deep learning denoising methods, self-supervised methods have become a crucial step in image enhancement due to their superior denoising performance compared to traditional methods and the lack of a high signal-to-noise ratio ground truth. These methods exploit the significant spatiotemporal redundancy in fluorescence time-lapse imaging—that is, repeated information between temporally adjacent frames and between spatially adjacent pixels—and use neural networks to extract clean image features from this redundant information for denoising.

[0004] Existing self-supervised denoising methods can be divided into two categories: neighbor-based methods and blind-spot-based methods. Among the neighbor-based methods, one is to construct a training framework by sampling the time intervals between frames in multiple frames of continuously acquired raw data, thereby exploiting temporal redundancy; the other is to construct a training framework by sampling the spatial pixel intervals in a single frame of image, thereby exploiting spatial redundancy. However, these methods have strict requirements for high temporal or high spatial resolution of imaging, otherwise oversmoothing and artifacts may occur. Blind-spot-based methods rely on the blind spot degradation of the center frame and use unoccluded pixels from the center frame and adjacent frames to reconstruct the blind spot. This pixel-level reconstruction task requires a lot of training time. In addition, since this method can only obtain the denoising result of the center frame after inputting multiple frames of images, it takes longer to process the complete imaging data, which cannot meet real-time requirements.

[0005] Both types of self-supervised learning denoising methods provide improvements in image signal-to-noise ratio for optical microscopy. However, many challenges remain, including dependence on high temporal or spatial resolution and the need for high computational resources. These challenges pose serious obstacles to achieving real-time denoising in different optical microscopy scenarios. Most existing self-supervised learning methods for image enhancement use networks with large parameters or use complex training strategies, which results in a large amount of space and time consumption. The three-dimensional U-shaped network receives a large amount of three-dimensional data and performs complex three-dimensional convolution operations, which makes it consume a lot of space. The Transformer network has a huge time consumption due to the large amount of computation required by the self-attention mechanism. The complex training strategies proposed for specific imaging scenarios make it unable to adapt to flexible scene changes, further limiting its application potential. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a real-time denoising method for optical microscopy imaging, an optical microscopy imaging system, an apparatus and a medium with good real-time performance and good scene generalization.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to a first aspect of the present invention, there is provided a real-time denoising method for optical microscopy imaging, comprising:

[0009] Acquiring image data output by an optical microscopy imaging system;

[0010] A denoising model based on the U-Net network was used to denoise the image data to obtain the denoised optical microscopy imaging results. A spatiotemporal redundant sampling strategy was used to generate training pairs, and the denoising model was trained through self-supervised training.

[0011] Preferably, the spatiotemporal redundant sampling strategy is used to generate training pairs, wherein the time sampling process is specifically as follows:

[0012] For a training stack with H×W×T pixels, where H, W, and T are the height, width, and length of the input image stack, respectively;

[0013] The training stack is divided into an input stack Xi and a target stack Yi in a sliding window manner in time. The sizes of the input stack and the target stack are both H×W×C. The initial frames differ by S frames. C and S are the window width and step size of the sliding window, respectively, and S≤C.

[0014] Preferably, the spatiotemporal redundant sampling strategy is used to generate training pairs, wherein the spatial sampling process is specifically as follows:

[0015] For the input stack Xi and the target stack Yi, each image is spatially sampled using a sparse convolution calculation method to obtain two sub-stacks G1(Xi) and G2(Xi) of the input stack Xi, and two sub-stacks G1(Yi) and G2(Yi) of the target stack Yi;

[0016] The same random seed is used for the same window of the input stack Xi and the target stack Yi, so the original pixel positions of the substacks G1(Xi) and G1(Yi) are the same, the original pixel positions of G2(Xi) and G2(Yi) are the same, G1(Xi) and G2(Yi) satisfy the spatiotemporal neighbor relationship, and G2(Xi) and G1(Yi) satisfy the spatiotemporal neighbor relationship;

[0017] Xi and G1(Xi) are selected as the input stack, and Yi and G2(Yi) are selected as the target stack to generate training pairs for self-supervised training of the denoising model.

[0018] Preferably, sparse convolution calculation is used for spatial sampling, and adjacent pixels are selected through spatial sampling. Specific parameter settings include:

[0019] The window size of the sparse convolution calculation is a×a pixels, with a step size of b. The convolution kernel of each window is randomly selected from the list of mask units. The mask unit is an a×a binary matrix with two elements with the value of 1, which is used to select adjacent pixel positions. Among them, a and b are customized according to the processing speed requirements and the spatial resolution of the image.

[0020] Preferably, the U-Net network model is a lightweight U-Net network model, which reduces the number of downsampling times to twice and cancels the top-level jump connection based on the original U-Net network model.

[0021] Preferably, the lightweight U-Net network model includes two encoding blocks and two decoding blocks, specifically:

[0022] Use two encoding blocks to reduce the dimensionality. Each encoding block consists of a convolutional layer followed by a BatchNorm, a ReLU, and a max pooling layer.

[0023] Use two decoding blocks to restore the input dimension. Each decoding block contains a nearest neighbor interpolation followed by a convolution layer, a BatchNorm and a ReLU.

[0024] At each resolution depth, the output of the second encoder block is connected to the input of the same-level encoder block using skip connections.

[0025] According to a second aspect of the present invention, an optical microscopy imaging system is provided, wherein a graphics processor of the system is equipped with a denoising module, and the denoising module performs denoising using any one of the methods described above.

[0026] Preferably, the system displays the images before and after denoising synchronously on a display.

[0027] According to a third aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0028] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any one of the methods described above is implemented.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) Based on the characteristics of similar signals and independent noise between temporally adjacent frames and spatially adjacent pixels, the present invention utilizes the fact that sub-images obtained through temporally and spatially adjacent downsampling of the original imaging data contain similar signals and independent noise. These sub-images can be used as training pairs for a neural network to optimize the denoising network parameters. This training method does not have strict requirements on the temporal and spatial resolution of the original image and is therefore applicable to a variety of imaging scenarios.

[0031] (2) The present invention regards spatial sampling as a downsampling and makes a lightweight improvement to the U-Net network by reducing the number of downsampling times to two, thereby reducing the number of denoising network parameters and the amount of calculation, and can achieve a denoising processing speed that exceeds the imaging speed. The real-time capability does not rely on high-performance hardware and can run on mainstream consumer-grade graphics cards. At the same time, the top-level jump connection in the U-Net network is eliminated, which can effectively suppress the noise characteristics of the high layer and further improve the denoising effect.

[0032] (3) The microscopic imaging system of the present invention realizes continuous data acquisition, multi-threaded parallel processing and real-time synchronous display of images before and after denoising through a dual-queue data inflow and outflow method including data acquisition, multi-threaded parallel processing and real-time synchronous display. Real-time denoising of microscopic imaging can be achieved on mainstream consumer-grade graphics cards without relying on high-performance hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2This is a diagram of the architecture of the real-time denoising method for optical microscopy proposed in this invention. (a) shows the temporal and spatial sampling of the acquired image as the input and output of the network, and the optimization of network parameters through scale constraints and spatiotemporal self-supervision. (b) is a schematic diagram of the spatial sampling strategy.

[0035] Figure 3 Figure 2 is a network structure diagram; (a) is the classic two-dimensional U-Net, (b) is the three-dimensional U-Net, and (c) is the ultra-light two-dimensional U-Net proposed in this paper.

[0036] Figure 4 The enhanced neuron segmentation effect in the embodiment; wherein, (a) to (f) are comparisons of image denoising and neuron segmentation results of the method of the present invention and other methods, (g) is a quantitative evaluation index of neuron segmentation, and (h) is a comparison of denoising and segmentation results on three specific neurons;

[0037] Figure 5 Schematic diagram of using the present invention to restore high temporal precision neuronal activity in an embodiment; wherein (a) shows the image and signal difference of voltage imaging data before and after processing by the method of this invention; (b) shows the details after magnification of the time range marked by the red dashed box in (a); and (c) shows the Pearson correlation coefficient between the signal obtained after processing by the method of the present invention and the true value of the electrophysiological signal.

[0038] Figure 6 Schematic diagram of the volume structure imaging results in the embodiment; wherein (a) is a before-and-after comparison diagram of the enhanced skin optical coherence tomography volume imaging using the method of the present invention, and (b) is a single slice display of the result in (a). DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts should fall within the scope of protection of the present invention.

[0040] Example

[0041] like Figure 1 As shown, this embodiment provides a real-time denoising method for optical microscopy imaging, comprising:

[0042] Acquiring image data output by an optical microscopy imaging system;

[0043] A denoising model based on the U-Net network was used to denoise the image data to obtain the denoised optical microscopy imaging results. A spatiotemporal redundant sampling strategy was used to generate training pairs, and the denoising model was trained through self-supervised training.

[0044] Next, the method of this embodiment is introduced in detail.

[0045] (1) Spatiotemporal redundant sampling

[0046] After temporal and spatial sampling, the captured images are used as the input and output of the denoising model based on the U-Net network. The parameters of the denoising model are optimized through scale constraints and spatiotemporal self-supervision. The specific process is as follows:

[0047] A spatiotemporal redundant sampling strategy is used to generate training pairs. For a training stack with H×W×T pixels, H, W, and T are the height, width, and length of the input image stack, respectively.

[0048] First, temporal sampling is performed. The training stack is divided into an input stack Xi and a target stack Yi in a sliding window manner. The size of the input stack and the target stack are both H×W×C. The initial frames differ by S frames. C and S are the window width and step size of the sliding window, respectively, and S≤C.

[0049] For the input stack Xi and the target stack Yi, each image is spatially sampled using a method similar to the sparse convolution calculation of the neural network, and two sub-stacks G1(Xi) and G2(Xi) of the input stack Xi and two sub-stacks G1(Yi) and G2(Yi) of the target stack Yi are obtained;

[0050] The same random seed is used for the same window of the input stack Xi and the target stack Yi, so the original pixel positions of the substacks G1(Xi) and G1(Yi) are the same, the original pixel positions of G2(Xi) and G2(Yi) are the same, G1(Xi) and G2(Yi) satisfy the spatiotemporal neighbor relationship, and G2(Xi) and G1(Yi) satisfy the spatiotemporal neighbor relationship;

[0051] Xi and G1(Xi) are selected as the input stack, and Yi and G2(Yi) are selected as the target stack for self-supervised training of the denoising network.

[0052] Then spatial sampling is performed to select adjacent pixels. For spatial sampling, the image randomly selects a mask from the mask unit list and splits the full image into two sub-images according to the mask mark.

[0053] Spatial sampling can be achieved through sparse convolution or other methods. To speed up sampling, this embodiment uses sparse convolution. The parameter settings include: the window size for sparse convolution is a×a pixels, the step size is b, and the convolution kernel for each window is randomly selected from a list of mask cells. The mask cell is an a×a binary matrix with two elements set to 1, which is used to select adjacent pixel positions. Parameters a and b can be customized based on the processing speed requirements and the spatial resolution of the image.

[0054] In this embodiment, the window size of the sparse convolution calculation is 2×2 pixels, the step size is 2, and the convolution kernel of each window is randomly selected from a list of 16 mask units. The mask unit is a 2×2 binary matrix in which the values ​​of two elements are 1, which is used to realize the selection of adjacent pixel positions.

[0055] (2) Denoising model based on U-Net network

[0056] In this embodiment, the U-Net network uses a lightweight U-Net network.

[0057] Similar to the original U-Net model, the lightweight U-Net network in this embodiment also uses an encoder-decoder architecture and skip connections. Two encoding blocks are used to reduce the dimensionality. Each encoding block consists of a 3×3 convolution layer followed by a BatchNorm, a ReLU, and a 3×3 max pooling layer. Two decoding blocks are used to restore the input dimension. Each decoding block contains a nearest neighbor interpolation followed by a 3×3 convolution layer, a BatchNorm, and a ReLU. At each resolution depth, the number of channels in each feature map is 64, 64, and 64, respectively. Skip connections are used to connect the output of the second encoding block to the input of the same-level encoding block.

[0058] Compared with the original U-Net network, the lightweight U-Net network of the present invention has made the following two changes:

[0059] First, the number of downsampling is reduced to two. Downsampling can increase the receptive field and enable the convolution kernel to extract features on a larger image range. However, as the number of downsampling increases, the number of network parameters will also increase dramatically. The spatial subsampler of the present invention can be regarded as a downsampling, which has the function of expanding the receptive field. Therefore, the number of downsampling of U-net can be reduced to shrink the network.

[0060] Second, we remove the top-level skip connections. Skip connections can fuse low-level features with high-level features. However, in denoising tasks, high-level features are noise and need to be suppressed. Therefore, removing the top-level skip connections improves denoising performance. By minimizing the network depth, the number of channels per feature map, and the number of skip connections, in this example, the network only trained 0.13M parameters.

[0061] This embodiment further provides an optical microscopy imaging system, in which a graphics processor is equipped with a denoising module. The denoising module uses the above-mentioned method to perform denoising, and the system display synchronously displays the images before and after denoising.

[0062] The denoising model is pre-trained using the self-supervised training method, and the trained denoising model is loaded into a graphics processing unit (GPU) as a denoising module.

[0063] The original output data stream of the imaging system is used as input and enters the processing queue. The images in the processing queue continuously enter the graphics processor loaded with the denoising model. The denoised data enters the completion queue, and finally the images before and after denoising are synchronously displayed on the monitor.

[0064] The microscopic imaging system in this embodiment realizes continuous data acquisition, multi-threaded parallel processing, and real-time synchronous display of images before and after denoising through a dual-queue data inflow and outflow method including data acquisition, multi-threaded parallel processing, and real-time synchronous display. Real-time denoising of microscopic imaging can be achieved on mainstream consumer-grade graphics cards without relying on high-performance hardware.

[0065] Next, the effect of the denoising method of the present invention is experimentally verified.

[0066] Experiment 1: Enhanced neuron segmentation

[0067] Background: Fluorescent genetically encoded calcium indicators and two-photon microscopy can simultaneously measure the activity of hundreds to thousands of neurons, enabling investigation of the correlation between neural activity and function. Neuronal segmentation is a crucial step in analyzing calcium imaging in neuronal populations. However, noise can reduce the number and accuracy of segmentations, and the loss of neurons overwhelmed by noise can adversely affect analysis.

[0068] Data preparation: We use the Neurofinder dataset for validation, which contains manually labeled segmentation benchmarks. The original data is contaminated by noise and shot noise, such as Figure 4 In part (a), the morphology and outline of neurons, which are key features for segmentation, are severely destroyed.

[0069] Noise processing: Two-photon calcium imaging data were processed using the present invention and four other denoising methods, such as Figure 4Parts (b) to (f) in the middle. After processing by the present invention, the morphology of the neurons is restored from the noise, the cell boundaries are obvious, the synapses of the neurons are visible in the magnified area, and they emit fluorescence together with the neurons. The results of DeepCAD-RT show smooth transitions, and the synaptic structure is almost indistinguishable from the background. The results of DeepVid and SUPPORT not only fail to observe image details, but also have some artifacts. The results of SRDTrans are clear and sharp, but due to the complex Transformer network, the processing consumes a lot of time and memory.

[0070] Segmentation analysis: Cellpose was used to perform neuron segmentation on the five denoising results. SRDTrans segmented the most extra neurons (29), and SUPPORT lost the most neurons (57). The image enhanced by the present invention segmented the most neurons (125) and segmented the least extra neurons (19). The segmented magnified images of the three denoising methods were selected, as shown in Figure 2. Figure 4 In part (g), only the present invention can completely identify neurons, with a deviation of only 1 to 2 pixels from the benchmark.

[0071] Performance evaluation: The denoising method of the present invention achieved the highest value (0.64) in F1 score, indicating that the denoising method of the present invention can enhance calcium imaging of neuronal populations to promote accurate and comprehensive neuronal segmentation.

[0072] Results: The treated neurons showed significant morphological restoration, with clear cell boundaries and visible synapses. The proposed method performed best in terms of the number and accuracy of segmented neurons, achieving the highest F1 score.

[0073] Conclusion: The denoising method of the present invention significantly improves the accuracy and comprehensiveness of neuron segmentation, and provides reliable data support for the study of neural activity and neural function.

[0074] Experiment 2: Restoring high temporal precision neuronal activity

[0075] Background: To study patterns of neuronal activity, high temporal precision activity recording is crucial. With the development of voltage indicators, this requires higher imaging speeds, with voltage imaging speeds reaching 1kHz. However, due to the inherent trade-off between imaging speed and the maximum number of photons collected, a reduction in the signal-to-noise ratio (SNR) is inevitable. This reduction compromises the recorded neuronal morphology and signals, with neuronal activity signals potentially being masked or even overwhelmed by noise. Through denoising, these signals can be restored, enabling accurate, high-precision recording of neuronal activity.

[0076] Data preparation: The activity of single QuasAr6a-expressing neurons in layer 2 / 3 of the mouse cortex was recorded using wide-field microscopy (imaging speed 1kHz) and synchronized patch-clamp electrophysiological recordings. The raw data were contaminated by significant noise, which severely destroyed the spatial footprint and temporal trajectory of the neurons, such as Figure 5 As shown in part (a).

[0077] Noise processing: Using the present invention to enhance these data, cell boundaries become identifiable and voltage traces emerge from the noise. The fluorescence of 100 cytoplasmic pixels is randomly extracted, and it is found that after the enhancement of the present invention, voltage fluctuations can be displayed at the single pixel level, such as Figure 5 As shown in part (b) of .

[0078] Performance evaluation: The average Pearson correlation coefficient between voltage traces and electrophysiological recordings increased from 0.13 to 0.29, as shown in Figure 5 As shown in part (c) of .

[0079] Results: Neuronal morphology and voltage traces were significantly restored after treatment with the present invention, with clear cell boundaries and visible voltage fluctuations. The present invention significantly improved the correlation between voltage traces and electrophysiological recordings, increasing from 0.13 to 0.29. Denoising of in vivo population neuronal voltage imaging data revealed restoration of neuronal morphology and structure, with preserved voltage transients.

[0080] Conclusion: This method can effectively denoise high temporal resolution single neuron and group neuron voltage imaging data while faithfully restoring voltage transients, providing reliable technical support for high-precision recording of neuronal activity.

[0081] Experiment 3: Volume Structure Imaging Enhancement

[0082] Background: Optical coherence tomography (OCT) is a non-invasive imaging technique widely used in the diagnosis and research of skin diseases. OCT can provide high-resolution two-dimensional and three-dimensional images, revealing the microstructure of skin tissue. However, due to the presence of noise interference during OCT imaging, image quality is severely affected, especially in volumetric imaging, where the noise problem is more prominent. Noise not only reduces image clarity but can also lead to misinterpretation of skin tissue structure and pathological changes. Therefore, developing an effective denoising method is crucial to improving the quality of OCT volumetric imaging. In skin OCT volumetric imaging, noise mainly comes from several aspects: First, the hardware limitations of the OCT system itself, such as the stability of the light source and the sensitivity of the detector, can introduce noise; second, the influence of ambient light during imaging, sample scattering and absorption, etc. can also increase noise; finally, algorithmic errors and computational accuracy issues during data processing can also lead to noise accumulation.

[0083] in conclusion: Figure 6 The figure is a schematic diagram of the volume structure imaging results using the method of the present invention, wherein (a) is a before-and-after comparison diagram of the enhanced volume imaging of skin optical coherence tomography using the method of the present invention, and (b) is a single slice display of the result in (a). It can be seen that the method of the present invention can effectively remove the noise in the volume imaging and improve the clarity and resolution of the image. The experimental results show that this method performs well in improving image quality and reducing noise levels, and has significant potential application value. For example, it helps to improve the resolution of skin OCT volume imaging, and can clearly display the fine structure of skin tissue, such as the boundary between the epidermis, dermis and subcutaneous tissue, making the detection of the lesion area more accurate and complete. The denoised image can better reflect the location and range of the lesion, providing a more reliable basis for the early diagnosis and treatment of skin diseases.

[0084] Conclusion: The application of the present invention in volumetric time-series structural and functional imaging of astrocytes demonstrates its scalability across different fluorescence and dimensions, providing reliable technical support for the observation of heterogeneous activities.

[0085] The electronic device of the present invention includes a central processing unit (CPU) and a graphics processing unit (GPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0086] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0087] The processing unit performs the various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a central processing unit (CPU) and a graphics processing unit (GPU), one or more steps of the method described above can be performed. Alternatively, in other embodiments, the central processing unit (CPU) and the graphics processing unit (GPU) can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0088] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0089] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A real-time denoising method for optical microscopy imaging, characterized in that: include: Acquire image data output by an optical microscopy imaging system; The image data was denoised using a denoising model based on the U-Net network to obtain the denoised optical microscopy imaging results. A spatiotemporal redundant sampling strategy was used to generate training pairs, and the denoising model was trained using a self-supervised training method.

2. A real-time denoising method for optical microscopy imaging according to claim 1, characterized in that: The time-space redundant sampling strategy is used to generate training pairs, wherein the time sampling process is specifically as follows: For a training stack with H×W×T pixels, where H, W, and T are the height, width, and length of the input image stack, respectively; The training stack is divided into an input stack Xi and a target stack Yi in a sliding window manner in time. The sizes of the input stack and the target stack are both H×W×C. The initial frames differ by S frames. C and S are the window width and step length of the sliding window, respectively, and S≤C.

3. A real-time denoising method for optical microscopy imaging according to claim 2, characterized in that: The spatiotemporal redundant sampling strategy is used to generate training pairs, wherein the spatial sampling process is specifically as follows: For the input stack Xi and the target stack Yi, spatial sampling is performed on each image using a sparse convolution calculation method to obtain two sub-stacks G1(Xi) and G2(Xi) of the input stack Xi, and two sub-stacks G1(Yi) and G2(Yi) of the target stack Yi; The same window of the input stack Xi and the target stack Yi uses the same random seed, so the original pixel positions of the substacks G1(Xi) and G1(Yi) are the same, the original pixel positions of G2(Xi) and G2(Yi) are the same, G1(Xi) and G2(Yi) satisfy the spatiotemporal neighbor relationship, and G2(Xi) and G1(Yi) satisfy the spatiotemporal neighbor relationship; Xi and G1(Xi) are selected as the input stack, Yi and G2(Yi) are selected as the target stack, and training pairs are generated for self-supervised training of the denoising model.

4. A real-time denoising method for optical microscopy imaging according to claim 3, characterized in that: The sparse convolution calculation method is used for spatial sampling. Adjacent pixels are selected through spatial sampling. The specific parameter settings include: The window size of sparse convolution calculation is a×a pixels, the step size is b, and the convolution kernel of each window is randomly selected from the mask unit list. The mask unit is an a×a binary matrix with two elements with the value of 1, which is used to select adjacent pixel positions. Among them, a and b are customized according to the processing speed requirements and the spatial resolution of the image.

5. A real-time denoising method for optical microscopy imaging according to claim 1, characterized in that: The U-Net network model is a lightweight U-Net network model. Based on the original U-Net network model, the lightweight U-Net network model reduces the number of downsampling times to two and cancels the top-level jump connection.

6. A real-time denoising method for optical microscopy imaging according to claim 5, characterized in that: The lightweight U-Net network model includes two encoding blocks and two decoding blocks, specifically: Use two encoding blocks to reduce the dimensionality, each encoding block consists of a convolutional layer followed by a BatchNorm, a ReLU, and a max pooling layer; Use two decoding blocks to restore the input dimension, each decoding block contains a nearest neighbor interpolation followed by a convolution layer, a BatchNorm and a ReLU; At each resolution depth, the output of the second encoder block is connected to the input of the sibling encoder block using skip connections.

7. An optical microscopic imaging system, characterized in that: The graphics processor of the system is equipped with a denoising module, and the denoising module performs denoising using the method described in any one of claims 1 to 6.

8. The system according to claim 7, characterized in that The images before and after denoising are synchronously displayed on a display of the system.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images