A self-supervised structured light microscopy reconstruction method and system based on pixel rearrangement
By employing a self-supervised multimodal structured light microscopy reconstruction method, a training set is generated using pixel rearrangement technology to perform self-supervised training on the neural network. This solves the problem of low signal-to-noise ratio in live cell imaging, achieves high-quality super-resolution image reconstruction, reduces costs and phototoxicity, and improves imaging performance.
Patent Information
- Application Number
- CN202310325238.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-03-30
AI Technical Summary
Existing structured light microscopy techniques suffer from poor image quality in live cell imaging due to low signal-to-noise ratio. Furthermore, traditional denoising algorithms and supervised neural network training require high costs and multiple sampling, which affect imaging performance and temporal resolution.
A self-supervised multimodal structured light microscopy reconstruction method is adopted. A training set is generated by pixel rearrangement technology, and the neural network is trained in a self-supervised manner. The super-resolution image is reconstructed using the standard structured light reconstruction algorithm and the denoising neural network, which reduces the signal-to-noise ratio requirement and improves the image quality.
It effectively denoises and reconstructs high-quality super-resolution images under extremely low signal-to-noise ratio conditions, reducing experimental costs and phototoxicity, and improving the applicability and imaging rate of live cell imaging.
Smart Images

Figure CN116402681B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to a method and system for self-supervised structured light microscopy reconstruction, and in particular to self-supervised structured light microscopy reconstruction trained using pixel rearrangement technology. Background Art
[0002] Structured light microscopy uses multiple modulated (such as sinusoidal moiré) excitation lights to illuminate the sample, and then reconstructs it through a specific algorithm. Compared with traditional fluorescence microscopy, it can obtain microscopic imaging results with significantly improved resolution, so as to achieve more detailed microstructural analysis. Structured light microscopy has a very wide range of applications in the field of living cell imaging due to its fast imaging speed, low phototoxicity, and wide sample applicability.
[0003] However, unlike traditional fluorescence microscopy, where the original image collected is the final visualization result, the super-resolution reconstructed image using structured light microscopy needs to be obtained from a series of original images through a reconstruction algorithm, where the reconstruction algorithm includes multiple complex steps such as Fourier transform, inverse Fourier transform, frequency domain translation, frequency domain filtering, and frequency domain splicing. Therefore, if there is noise in the original image, the noise will be significantly amplified in the image processed by the reconstruction algorithm. If the signal-to-noise ratio of the original image is low, this will significantly affect the quality of the final super-resolution reconstructed image, that is, produce serious reconstruction "artifacts". The presence of "artifacts" will seriously affect the quality of the final super-resolution reconstructed image, making it impossible to distinguish between the real sample information and the "artifacts" generated during the reconstruction process, thereby affecting the microscopic observation effect.
[0004] When using structured light microscopy to image living cells, the signal-to-noise ratio of the original fluorescence image obtained is usually low due to the following factors:
[0005] 1. Limited by the types of fluorescent dyes and fluorescent proteins that can be used for fluorescent labeling of living cells, the efficiency of fluorescent labeling is usually low, resulting in low emission efficiency of fluorescent signals when the sample is excited by light;
[0006] 2. In order to avoid damage to living cells, usually only low-intensity excitation light sources can be used for irradiation, which will also reduce the light intensity of fluorescence;
[0007] 3. Living cells usually move at high speed. In order to capture the movement process of living cells, the exposure time of the photo needs to be shortened as much as possible, which will lead to a corresponding decrease in the intensity of the acquired fluorescence signal.
[0008] For the above reasons, especially for living cell observation, how to denoise the image has become a very critical breakthrough point in structured light microscopy. In the existing technical research on image denoising in structured light microscopy, denoising algorithms such as Wiener filtering and BM3D are mainly used to suppress noise in a targeted manner through the statistical difference between image noise and sample information. However, for the case where the signal-to-noise ratio of the original fluorescence image obtained by observing living cells is extremely low, because the image noise is deeply coupled with the sample information, the algorithm using this statistical difference will not be able to perform effective denoising.
[0009] In addition, in recent years, due to the rapid development of deep learning technology, the use of neural network methods for image denoising has attracted more and more attention. However, if the original fluorescence images of living cells are used to train the denoising neural network, a "supervised" training mechanism is usually required, which requires that a large number of matching "high signal-to-noise ratio-low signal-to-noise ratio" image pairs must be collected in advance to construct a training set. However, the large-scale collection of "high signal-to-noise ratio-low signal-to-noise ratio" image pairs will not only cause damage to living cells due to the need to increase the intensity of the excitation light, but also significantly increase the experimental cost due to the increased requirements for the selection of fluorescent dyes and fluorescent proteins. In addition, for living cell observations, it is also necessary to sample the same living sample multiple times, which will reduce the temporal resolution and make it impossible to process video data. Summary of the invention
[0010] In response to the above problems, the present application proposes a novel self-supervised multimodal structured light microscopy image super-resolution reconstruction and denoising technology solution. When training the neural network used in this technical solution, the training set is generated by only a set of single-acquired fluorescence images through the use of a novel "pixel rearrangement technology" (no need to use high signal-to-noise ratio images for supervision or multiple sampling of the same living biological sample) to perform "self-supervised" training on the neural network. The trained neural network ensures that the denoised super-resolution image can be reconstructed based on the original fluorescence image with a lower signal-to-noise ratio, thereby significantly improving the image quality of structured light microscopy and expanding its scope of application.
[0011] According to one aspect of the present application, a self-supervised multimodal structured light microscopy reconstruction method is provided, comprising:
[0012] A biological sample is excited by structured light to obtain J original fluorescence image sequences (Y) generated by the stimulation of the biological sample, wherein J is an integer greater than or equal to 1, each original fluorescence image sequence (Y) includes S fluorescence images, S is an integer greater than or equal to 2, and each fluorescence image has a pixel size of M*N, wherein M and N are even numbers,
[0013] For each of the J original fluorescence image sequences (Y), a program is run in a computer, wherein:
[0014] 1) For the i-th fluorescence image (y i ) extracts the first sub-image, the second sub-image, the third sub-image and the fourth sub-image whose pixel sizes are all (M / 2)*(N / 2) by pixel rearrangement, wherein the two-dimensional pixel matrix y of the first sub-image i,(2m-1)(2n-1) , the two-dimensional pixel matrix y of the second sub-image i,(2m-1)(2n) , the two-dimensional pixel matrix y of the third sub-image i,(2m)(2n-1) , and the two-dimensional pixel matrix y of the fourth sub-image i,(2m)(2n) Respectively from the i-th fluorescence image (y i ) of the two-dimensional pixel matrix y i,MN Select, where m is an integer and = 1, 2, 3, ..., M / 2, n is an integer and = 1, 2, 3, ..., N / 2;
[0015] 2) performing a 2x pixel upsampling process on the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image to obtain a first upsampled sub-image, a second upsampled sub-image, a third upsampled sub-image, and a fourth upsampled sub-image, all of which have a pixel size of M*N;
[0016] 3) Performing a pixel shift of [0.5, 0.5] on the two-dimensional pixel matrix of the first upsampled sub-image to obtain a first sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, 0.5] on the two-dimensional pixel matrix of the second upsampled sub-image to obtain a second sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [0.5, -0.5] on the two-dimensional pixel matrix of the third upsampled sub-image to obtain a third sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, -0.5] on the two-dimensional pixel matrix of the fourth upsampled sub-image to obtain a fourth sub-image after two-dimensional sub-pixel shift;
[0017] 4) classifying the S two-dimensional sub-pixel-shifted first sub-images obtained from the S fluorescence images into a first sub-image group, classifying the S two-dimensional sub-pixel-shifted second sub-images obtained from the S fluorescence images into a second sub-image group, classifying the S two-dimensional sub-pixel-shifted third sub-images obtained from the S fluorescence images into a third sub-image group, and classifying the S two-dimensional sub-pixel-shifted fourth sub-images obtained from the S fluorescence images into a fourth sub-image group;
[0018] 5) performing structured light image reconstruction on the first sub-image group, the second sub-image group, the third sub-image group, and the fourth sub-image group using a standard structured light reconstruction algorithm to obtain a first super-resolution sub-image, a second super-resolution sub-image, a third super-resolution sub-image, and a fourth super-resolution sub-image;
[0019] 6) selecting two different super-resolution sub-images from the first super-resolution sub-image, the second super-resolution sub-image, the third super-resolution sub-image, and the fourth super-resolution sub-image to form a training super-resolution image pair, and grouping them into a training set;
[0020] A program is run in a computer, the program causing:
[0021] 7) training a denoising neural network based on the training set;
[0022] 8) performing super-resolution reconstruction on the S fluorescence images in each of the J original fluorescence image sequences (Y) using a standard structured light super-resolution reconstruction algorithm to form a super-resolution image, and using the super-resolution image as an input of the denoising neural network to obtain a final super-resolution reconstructed image.
[0023] Optionally, when training a denoising neural network, one super-resolution sub-image in the training super-resolution image pair is used as training input data, and the other super-resolution sub-image is used as training target data.
[0024] Optionally, if the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image of the original fluorescence image sequence is an odd number, the corresponding number of row or column pixels are deleted to ensure that the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image are both even numbers.
[0025] Optionally, when training a denoising neural network based on the training set, in each training cycle, a training super-resolution image pair is randomly extracted from the training set, and pixel blocks at the same position in the two super-resolution sub-images are randomly extracted, and the pixels are randomly rotated and folded to serve as the denoising neural network input image and target image respectively, the error between the network output and the target image is calculated, and the gradient is back-propagated to update the network parameters.
[0026] Optionally, the denoising neural network includes but is not limited to a U-type neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
[0027] Optionally, an optical imaging system (100) is used to excite a biological sample using structured light to obtain j original fluorescence image sequences (Y) generated by the stimulation of the biological sample, wherein the optical imaging system (100) includes but is not limited to a two-dimensional structured light system (2D-SIM), a three-dimensional structured light system (3D-SIM), a lattice light sheet structured light system (LLS-SIM), and a grazing incidence illumination structured light system (GI-SIM).
[0028] Optionally, the 2x pixel upsampling process is implemented using nearest neighbor interpolation, bilinear interpolation or biquadratic spline interpolation.
[0029] According to another aspect of the present application, a self-supervised multimodal structured light microscopy reconstruction system is provided, comprising:
[0030] A super-resolution reconstruction module (210), wherein the super-resolution reconstruction module (210) is configured to perform super-resolution reconstruction on the image using a standard structured light super-resolution reconstruction algorithm; and
[0031] A denoising module (220), wherein the denoising module (220) is provided with a denoising neural network,
[0032] The self-supervised multimodal structured light microscopy reconstruction system is configured as follows:
[0033] A biological sample is excited by structured light to obtain J original fluorescence image sequences (Y) generated by the stimulation of the biological sample, wherein J is an integer greater than or equal to 1, each original fluorescence image sequence (Y) includes S fluorescence images, S is an integer greater than or equal to 2, and each fluorescence image has a pixel size of M*N, wherein M and N are even numbers,
[0034] The self-supervised multimodal structured light microscopy reconstruction system is further configured to:
[0035] For each of the J original fluorescence image sequences (Y),
[0036] 1) For the i-th fluorescence image (y i ) extracts the first sub-image, the second sub-image, the third sub-image and the fourth sub-image whose pixel sizes are all (M / 2)*(N / 2) by pixel rearrangement, wherein the two-dimensional pixel matrix y of the first sub-image i,(2m-1)(2n-1) , the two-dimensional pixel matrix y of the second sub-image i,(2m-1)(2n) , the two-dimensional pixel matrix y of the third sub-image i,(2m)(2n-1) , and the two-dimensional pixel matrix y of the fourth sub-image i,(2m)(2n) Respectively from the i-th fluorescence image (y i ) of the two-dimensional pixel matrix y i,MNSelect, where m is an integer and = 1, 2, 3, ..., M / 2, n is an integer and = 1, 2, 3, ..., N / 2;
[0037] 2) performing a 2x pixel upsampling process on the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image to obtain a first upsampled sub-image, a second upsampled sub-image, a third upsampled sub-image, and a fourth upsampled sub-image, all of which have a pixel size of M*N;
[0038] 3) Performing a pixel shift of [0.5, 0.5] on the two-dimensional pixel matrix of the first upsampled sub-image to obtain a first sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, 0.5] on the two-dimensional pixel matrix of the second upsampled sub-image to obtain a second sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [0.5, -0.5] on the two-dimensional pixel matrix of the third upsampled sub-image to obtain a third sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, -0.5] on the two-dimensional pixel matrix of the fourth upsampled sub-image to obtain a fourth sub-image after two-dimensional sub-pixel shift;
[0039] 4) classifying the S two-dimensional sub-pixel-shifted first sub-images obtained from the S fluorescence images into a first sub-image group, classifying the S two-dimensional sub-pixel-shifted second sub-images obtained from the S fluorescence images into a second sub-image group, classifying the S two-dimensional sub-pixel-shifted third sub-images obtained from the S fluorescence images into a third sub-image group, and classifying the S two-dimensional sub-pixel-shifted fourth sub-images obtained from the S fluorescence images into a fourth sub-image group;
[0040] 5) performing structured light image reconstruction on the first sub-image group, the second sub-image group, the third sub-image group, and the fourth sub-image group using a standard structured light reconstruction algorithm to obtain a first super-resolution sub-image, a second super-resolution sub-image, a third super-resolution sub-image, and a fourth super-resolution sub-image;
[0041] 6) selecting two different super-resolution sub-images from the first super-resolution sub-image, the second super-resolution sub-image, the third super-resolution sub-image, and the fourth super-resolution sub-image to form a training super-resolution image pair, and grouping them into a training set;
[0042] The self-supervised multimodal structured light microscopy reconstruction system is further configured to:
[0043] 7) training a denoising neural network based on the training set;
[0044] 8) performing super-resolution reconstruction on the S fluorescence images in each of the J original fluorescence image sequences (Y) using a standard structured light super-resolution reconstruction algorithm to form a super-resolution image, and using the super-resolution image as an input of the denoising neural network to obtain a final super-resolution reconstructed image.
[0045] Optionally, when training a denoising neural network, one super-resolution sub-image in the training super-resolution image pair is used as training input data, and the other super-resolution sub-image is used as training target data.
[0046] Optionally, if the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image of the original fluorescence image sequence is an odd number, the corresponding number of row or column pixels are deleted to ensure that the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image are both even numbers.
[0047] Optionally, when training a denoising neural network based on the training set, in each training cycle, a training super-resolution image pair is randomly extracted from the training set, and pixel blocks at the same position in the two super-resolution sub-images are randomly extracted, and the pixels are randomly rotated and folded to serve as the denoising neural network input image and target image respectively, the error between the network output and the target image is calculated, and the gradient is back-propagated to update the network parameters.
[0048] Optionally, the denoising neural network includes but is not limited to a U-type neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
[0049] Optionally, an optical imaging system is used to excite a biological sample using structured light to obtain j original fluorescence image sequences generated by the stimulation of the biological sample, wherein the optical imaging system includes but is not limited to a two-dimensional structured light system (2D-SIM), a three-dimensional structured light system (3D-SIM), a lattice light sheet structured light system (LLS-SIM), and a grazing incidence illumination structured light system (GI-SIM).
[0050] Optionally, the 2x pixel upsampling process is implemented using nearest neighbor interpolation, bilinear interpolation or biquadratic spline interpolation.
[0051] By adopting the technical means of the present application, especially for long-term video data of living cells, a training set can be constructed from the original data itself and a neural network can be trained, and denoised by itself, thus eliminating the need for any additional data and pre-trained models, thereby effectively reducing experimental costs and improving the generalization of the present technology.
[0052] In addition, by using the technical means of this application, extremely low excitation light power and extremely short exposure time can be used in the process of fluorescence imaging of cells, especially living cells, which significantly reduces phototoxicity and can capture high-speed dynamic processes, thereby improving the applicability of living cell imaging. In summary, it is of great significance to improve the quality of structured light microscopy images and expand the application of structured light microscopy technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The principles and various aspects of the present application can be more fully understood from the following detailed description in conjunction with the following drawings. It should be noted that the scales of the drawings may be different for the purpose of clear description, but this will not affect the understanding of the present application. In the drawings:
[0054] Figure 1 A basic block diagram of a structured light microscopy imaging system is schematically shown;
[0055] Figure 2A The process of training the neural network of the denoising module of the self-supervised multimodal structured light microscopy reconstruction system according to one embodiment of the present application is schematically shown;
[0056] Figure 2B The process of reconstructing the original fluorescence image sequence using the self-supervised multimodal structured light microscopy reconstruction system with a trained neural network according to one embodiment of the present application is schematically shown;
[0057] Figure 3 A schematic diagram of a two-dimensional pixel matrix representing a fluorescence image is shown;
[0058] Figure 4A , 4B 4C and 4D schematically show the use of "pixel rearrangement" by Figure 3 A schematic diagram of a two-dimensional pixel matrix of a sub-image extracted by a two-dimensional pixel matrix of ;
[0059] Figure 5 Schematic diagram of "2x pixel upsampling" Figure 4A A schematic diagram of processing a two-dimensional pixel matrix of a sub-image;
[0060] Figure 6 A flowchart of a self-supervised multimodal structured light microscopy reconstruction method according to an embodiment of the present application is schematically shown;
[0061] Fig. 7A and 7B Schematic diagrams schematically illustrate example diagrams of microscopic image processing of the training process and reconstruction process involved in the present application. DETAILED DESCRIPTION
[0062] In the various figures of the present application, features with the same structure or similar functions are represented by the same reference numerals.
[0063] Figure 1 The basic block diagram of a structured light microscopy imaging system is schematically shown, which generally includes an optical imaging system 100 and a control and data processing system 200. The optical imaging system 100 includes an excitation light path and a detection light path, wherein the excitation light path includes an excitation objective lens and other optical components for generating excitation light, and the excitation light beam can be emitted through the excitation objective lens in the form of periodic stripes of structured light, so as to excite fluorescence on the biological sample, and the detection light path includes a detection objective lens and other optical components for imaging, which are used to receive and detect the excited fluorescence. It should be clear to those skilled in the art that, depending on the configuration of the structured light microscopy imaging system, the excitation objective lens and the detection objective lens can be the same objective lens or different objective lenses. When performing three-dimensional fluorescence microscopy imaging on biological samples, especially living biological samples, multi-layer fluorescence images are continuously scanned and sampled along the optical axis direction of the detection objective lens, that is, along the axial direction, so that each time a scanning sampling is completed, the acquired multi-layer fluorescence images constitute a fluorescence image stack (also referred to as a "sequence").
[0064] The control and data processing system 200 mainly includes a computer and related components (such as a data storage device, etc.), which can control the operation of the optical imaging system 100 and can receive image data from the optical imaging system 100 and perform corresponding post-processing. For example, the acquired fluorescent image volume stack is provided to the control and data processing system 200, and reconstructed into a high signal-to-noise ratio three-dimensional microscopic image after a series of data processing. For this purpose, the control and data processing system 200 may include a self-supervised multi-modal structured light microscopic image reconstruction module or system (also referred to as a "self-supervised multi-modal structured light microscopic reconstruction module or system"). The self-supervised multi-modal structured light microscopic image reconstruction module or system includes a super-resolution reconstruction submodule 210 and a denoising submodule 220. It should be pointed out that within the scope of the present application, the modules and / or submodules described here can be understood to include a data storage device, such as a computer-readable storage medium, in which programs or subroutines and denoising neural network models called and run by a computer, especially a computer of the control and data processing system 200, can be stored. These programs or subroutines and denoising neural network models can implement the methods / steps described below, especially the self-supervised multimodal structured light microscopy image reconstruction method / steps, when called and executed by a computer. The specific programming method for the programs and / or subroutines is not discussed in this application, and those skilled in the art can implement the relevant functions with any well-known programming software and / or commercial software. Therefore, when describing the operation of the relevant system or the operation or method of the module below, it should be understood that they can also be written as programs to be called and executed by a computer.
[0065] The super-resolution reconstruction submodule 210 can select a standard structured light super-resolution reconstruction algorithm to perform super-resolution reconstruction on the fluorescence image acquired by the optical imaging system 100. In the scope of the present application, the standard structured light super-resolution reconstruction algorithm can be considered as an algorithm already known in the field of microscopic imaging. As an example, the standard structured light super-resolution reconstruction algorithm can refer to the public document Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J 94, 4957-4970 (2008) by Gustafsson, MG et al.
[0066] The denoising submodule 220 can be implemented by using any neural network architecture in a manner known to those skilled in the art for image denoising. For example, the neural network model used in the denoising submodule 220 includes but is not limited to a U-type neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. When training the neural network of the denoising submodule 220, the relevant network model is optimized using a loss function, and the loss function includes but is not limited to mean square error (MSE), mean absolute error (MAE), structural similarity (SSIM) or a weighted sum thereof.
[0067] Therefore, the super-resolution reconstruction submodule 210 (or “super-resolution reconstruction module”) and the denoising submodule 220 (or “denoising module”) constitute the self-supervised multimodal structured light microscopy reconstruction system according to the present application. Figure 2A The process of training the neural network of the denoising module 220 of the self-supervised multimodal structured light microscopy reconstruction system using the original fluorescence image sequence Y obtained by the optical imaging system 100 according to one embodiment of the present application is schematically shown; Figure 2B The schematic diagram shows a process of reconstructing the original fluorescence image sequence Y obtained by the optical imaging system 100 using a self-supervised multimodal structured light microscopy reconstruction system with a trained neural network according to an embodiment of the present application.
[0068] The optical imaging system 100 to which the self-supervised multimodal structured light microscopy reconstruction system of the present application can be applied may include but is not limited to a two-dimensional structured light system (2D-SIM), a three-dimensional structured light system (3D-SIM), a lattice light sheet structured light system (LLS-SIM), and a grazing incidence illumination structured light system (GI-SIM). Therefore, the term "multimodal structured light microscopy reconstruction" means that this "structured light microscopy reconstruction" can be applied to a variety of optical imaging systems. In addition, in order to overcome the various deficiencies mentioned in the background technology section, the multimodal structured light microscopy reconstruction system of the present application adopts a unique self-supervised method for neural network training. Refer to the attached figure below. Figure 2A The basic principle of training the neural network of the denoising module 220 of the self-supervised multimodal structured light microscopy reconstruction system of the present application is introduced.
[0069] First, for example, a two-dimensional structured light system is used as an example of the optical imaging system 100 to complete a fluorescent image scan of a biological sample to obtain an original image sequence Y. For example, the original image sequence Y acquired by the optical imaging system 100 can be represented as a series of noisy fluorescent images y under different illumination modes: i (i=1, 2, . . . , S), where S represents the number of illumination modes of the optical imaging system 100, and S is an integer greater than or equal to 2.
[0070] It should be clear to those skilled in the art that each fluorescent image can be represented in the form of a two-dimensional pixel matrix that can be read and processed by a computer of the control and data processing system 200. For example, for the fluorescent image y i In terms of y i,MN , where M is the number of row pixels of the fluorescence image and N is the number of column pixels of the fluorescence image. For the fluorescence image involved in this application, M and N can be the same or different and both are even numbers. For example, the fluorescence image can be y i,512*512 , indicating that the size of the i-th image y is 512*512 pixels.
[0071] Of course, for the original fluorescence image where M or N is an odd number, in order to apply the technical solution of the present application, those skilled in the art should be aware that a row or a column of pixels of the fluorescence image (for example, at the boundary) can be removed accordingly to ensure that the maximum number of pixels in the row direction or the column direction of the pixel size of the original fluorescence image used in the technical solution of the present application is an even number.
[0072] Figure 3 Schematically shows the i-th fluorescence image y in the original image sequence Y i The two-dimensional pixel matrix y i,MNAccording to an embodiment of the present application, the two-dimensional pixel matrix of all fluorescent images in the original image sequence Y is used to obtain a training set for the neural network used to train the denoising module 220 in a "pixel rearrangement" manner. Figure 3 Take the i-th fluorescence image y as an example to illustrate i By adopting the “pixel rearrangement” approach, those skilled in the art should be aware that the same approach can be applied to other fluorescent images in the original image sequence Y.
[0073] (1) Downsampling the fluorescence image y by 2 times i The two-dimensional pixel matrix y i,MN Extract and generate four sub-graphs y i,(2m-1)(2n-1) ,y i,(2m-1)(2n) ,y i,(2m)(2n-1) ,y i,(2m)(2n) , where m is an integer and = 1, 2, 3, ..., M / 2, and n is an integer and = 1, 2, 3, ..., N / 2. Therefore, if the two-dimensional pixel matrix y i,MN If the pixel size is M*N, then the four sub-images y i,(2m -1)(2n-1),y i,(2m -1)(2n),y i,(2m)(2n -1), y i,(2m)(2n) The pixel size will be (M / 2)*(N / 2), such as Figures 4A to 4D In the context of the present application, when referring to a fluorescent image, a two-dimensional pixel matrix of a fluorescent image, a sub-image, or a two-dimensional pixel matrix of a sub-image, the pixel points of the image are counted starting from the leftmost pixel point as 1 in the row direction, and are counted starting from the topmost pixel point as 1 in the column direction.
[0074] (2) For the four sub-graphs y i,(2m-1)(2n-1) ,y i,(2m-1)(2n) ,y i,(2m)(2n-1) ,y i,(2m)(2n) Perform 2x pixel upsampling to obtain four upsampled sub-images ys1' with a pixel size of M*N i,MN 、ys2' i,MN 、ys3' i,MN 、ys4' i,MN . Only one of the subgraphs y' i,(2m -1)(2n-1) upsampled sub-image ys1' i,MN exist Figure 5 It is schematically shown in FIG. Those skilled in the art should be aware that when performing a 2x pixel upsampling process on an image, the methods that can be used include but are not limited to any known methods for processing images in a computer, such as nearest neighbor interpolation, bilinear interpolation, and biquadratic spline interpolation. Therefore, no redundant description is given in this article.
[0075] (3) Due to the four sub-images ys1' after upsampling i,MN 、ys2' i,MN 、ys3' i,MN 、ys4' i,MN The pixel size is the same as the original two-dimensional pixel matrix y i,MN The same, so considering the difference in the previous downsampling method, each sub-image ys1' i,MN 、ys2' i,MN 、ys3' i,MN 、ys4' i,MN The image center and the original two-dimensional pixel matrix y i,MN Therefore, in order to eliminate the influence caused by the deviation between the image center of the sub-image and the image center of the original image, each sub-image ys1' i,MN 、ys2' i,MN 、ys3' i,MN 、ys4' i,MN Perform sub-pixel registration separately, that is, for sub-image ys1' i,MN , perform a two-dimensional sub-pixel translation of [0.5,0.5] to obtain the sub-image ys1 i,MN ; For subgraph ys2' i,MN , perform a two-dimensional sub-pixel translation of [-0.5, 0.5] to obtain the sub-image ys2 i,MN ; For subgraph ys3' i,MN , perform a two-dimensional sub-pixel translation of [0.5,-0.5] to obtain sub-image ys3 i,MN ; and for subgraph ys4' i,MN , perform a two-dimensional sub-pixel translation of [-0.5,-0.5] to obtain sub-image ys4 i,MN .
[0076] Within the scope of the present application, when processing the translation image (sub-pixel registration), translation to the right in the row direction is positive, and translation to the left is negative; translation downward in the column direction is positive, and translation upward is negative. For example, [0.5, 0.5] should be understood as the sub-image to be processed is translated half a pixel to the right in the row direction and half a pixel downward in the column direction; [-0.5, -0.5] should be understood as the sub-image to be processed is translated half a pixel to the left in the row direction and half a pixel upward in the column direction. The translation image processing involved in the technical solution of the present application can be implemented by any known software, program or command in computer image processing technology, which will not be described in detail here.
[0077] Those skilled in the art should be aware that in the process of performing two-dimensional sub-pixel translation on a sub-image, invalid pixel values (for example, pixel values cleared to zero due to translation) may exist in the corresponding boundary rows and columns of the sub-image. Figure 5As shown, with subgraph ys1' i,MN For example, after a two-dimensional sub-pixel translation of [0.5, 0.5], the translated sub-image ys1 i,MN The pixels in the leftmost column and the topmost row may have null values (zeroed pixel values), as marked by "x" in the figure. However, the null values at the boundaries do not affect the subsequent results of overall image processing and recognition, so the impact of such null values can be ignored in the technical solution of this application.
[0078] (4) Repeat the above (1) to (3) for each fluorescence image in the original image sequence Y, and then the sub-image group ys1 can be obtained. i,MN 、ys2 i,MN 、ys3 i,MN 、ys4 i,MN , where i = 1, 2, ..., S. Then, the sub-image group ys1 is reconstructed using a standard structured light reconstruction algorithm. i,MN 、ys2 i,MN 、ys3 i,MN 、ys4 i,MN Structured light reconstruction is performed to obtain four super-resolution sub-images Y1, Y2, Y3, and Y4. As mentioned above, the standard structured light super-resolution reconstruction algorithm can refer to the public document Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J94, 4957-4970 (2008) by Gustafsson, MG et al. or any other known reconstruction algorithm.
[0079] (5) For the original image sequence Y, when four super-resolution sub-images Y1, Y2, Y3, and Y4 are obtained, two different super-resolution sub-images are selected to form a training super-resolution image pair. For example, for an original image sequence, there are 12 combinations of training super-resolution image pairs, namely (Y1, Y2), (Y1, Y3), (Y1, Y4), (Y2, Y3), (Y2, Y4), (Y3, Y4), (Y2, Y1), (Y3, Y1), (Y4, Y1), (Y3, Y2), (Y4, Y2), (Y4, Y3). In each training super-resolution image pair composed of two super-resolution images, the first super-resolution image is used as training input data, and the second super-resolution image is used as training target data. In other words, under the premise of ignoring the order of the two super-resolution images in a training super-resolution image pair, it can also be considered that one super-resolution image in a training super-resolution image pair is used as training input data, and the other super-resolution image is used as training target data.
[0080] If, for a living cell, an optical imaging system 100 is used to obtain J original fluorescence image sequences (where J can be an integer greater than or equal to 1), then for each of the J original fluorescence image sequences (for example, each original fluorescence image sequence can be composed of S fluorescence images), the above method can be used to obtain 12 combinations of training super-resolution image pairs. Finally, for all J original fluorescence image sequences, J*12 training super-resolution image pairs will be obtained. These training super-resolution image pairs constitute a training set for training the neural network of the denoising module 220. The neural network of the denoising module 220 can be trained using a training super-resolution image pair selected from the training set. For example, in each training cycle, a training super-resolution image pair is randomly selected from the training set, and pixel blocks at the same position are randomly extracted, and randomly rotated and folded as the neural network input image and target image respectively, and the error between the network output and the target image is calculated, and its gradient is back-propagated to update the network parameters. After the network input error converges, the training is stopped and the network parameters are stored. Those skilled in the art should be aware that the training methods of the neural network are not limited to those listed. It can be seen that the neural network training of the present application is actually a self-supervised training.
[0081] Those skilled in the art should be aware that when only J=1, it can be considered that the optical imaging system 100 is performing static observation of the biological sample. In this case, the optical imaging system 100 obtains a static fluorescence imaging sequence consisting of multiple original fluorescence images for the biological sample. When J>=2, it can be considered that the optical imaging system 100 is performing dynamic observation of the biological sample (for example, continuously shooting videos). In this case, multiple fluorescence imaging sequences are obtained, and each fluorescence imaging sequence includes multiple original fluorescence images.
[0082] After the neural network training of the denoising module 220 is completed, the self-supervised multimodal structured light microscopy imaging system reconstructs (or predicts) the original fluorescence image sequence obtained by the optical imaging system 100, such as Figure 2B As shown. The noisy fluorescence image in each original fluorescence image sequence is super-resolved using the super-resolution reconstruction submodule 210 to obtain a super-resolution image. Then, the obtained super-resolution image is used as the neural network input of the denoising module 220 to obtain the final super-resolution image after denoising.
[0083] Figure 6 An embodiment of the self-supervised multimodal structured light microscopy reconstruction method according to the present application is schematically shown. Assume that an optical imaging system 100 is used to acquire J original fluorescence image sequences Y of a biological sample or a living cell, where J is an integer greater than or equal to 1. In the jth original fluorescence image sequence Y j (j=1,2,…,J), including the fluorescence image y i (i=1,2,…,S), where S is an integer greater than or equal to 2, and each fluorescence image y i The pixel size is M*N, where M and N are integers at least greater than 2 and are even numbers.
[0084] In step S10, starting from j=1, for the jth original fluorescence image sequence Y j Fluorescence image y i For example, starting from i=1, a computer is used to downsample the fluorescence image y by 2 times. i The two-dimensional pixel matrix y i,MN Extract and generate four sub-graphs y i,(2m-1)(2n-1) ,y i,(2m-1)(2n) ,y i,(2m)(2n-1) ,y i,(2m)(2n) , where m is an integer and = 1, 2, 3, ..., M / 2, and n is an integer and = 1, 2, 3, ..., N / 2.
[0085] In step S20, the four sub-images y are processed by a computer. i,(2m-1)(2n-1) ,y i,(2m-1)(2n),y i,(2m)(2n-1) ,y i,(2m)(2n) Perform 2x pixel upsampling to obtain four upsampled sub-images ys1' with a pixel size of M*N i,MN 、ys2' i,MN 、ys3' i,MN 、ys4' i,MN When performing a 2x pixel upsampling process on an image, the methods that may be used include, but are not limited to, any known methods for processing images in a computer, such as nearest neighbor interpolation, bilinear interpolation, and biquadratic spline interpolation.
[0086] In step S30, a computer is used to calculate each sub-image ys1' i,MN 、ys2' i,MN 、ys3' i,MN 、ys4' i,MN Perform sub-pixel registration separately, that is, for sub-image ys1' i,MN , perform a two-dimensional sub-pixel translation of [0.5,0.5] to obtain the sub-image ys1 i,MN ; For subgraph ys2' i,MN , perform a two-dimensional sub-pixel translation of [-0.5, 0.5] to obtain the sub-image ys2 i,MN ; For subgraph ys3' i,MN , perform a two-dimensional sub-pixel translation of [0.5,-0.5] to obtain sub-image ys3 i,MN ; and for subgraph ys4' i,MN , perform a two-dimensional sub-pixel translation of [-0.5,-0.5] to obtain sub-image ys4 i,MN .
[0087] In step S31, a computer is used to determine the original image sequence Y j The above steps S10 to S30 have been repeated for each fluorescent image in, for example, if i is not equal to S, then i=i+1 and steps S10 to S30 are repeated; if i=S, go to step S32.
[0088] In step S32, since the original image sequence Y has been j The above steps S10 to S30 have been repeated for each fluorescent image in , so four sub-image groups ys1 can be obtained i,MN 、ys2 i,MN 、ys3 i,MN 、ys4 i,MN , where i = 1, 2, ..., S. The super-resolution reconstruction module 210 is used to reconstruct each sub-image group ys1 i,MN 、ys2 i,MN 、ys3 i,MN 、ys4 i,MNSuper-resolution reconstruction is performed to obtain four super-resolution sub-images Y1, Y2, Y3, and Y4.
[0089] In step S33, two different super-resolution sub-images are randomly selected from the four super-resolution sub-images Y1, Y2, Y3, and Y4 obtained in step S32 to form a training super-resolution image pair, wherein the first super-resolution image in the training super-resolution image pair is used as training input data, and the second super-resolution image is used as training target data. For example, for a single original fluorescence image sequence, since four super-resolution sub-images can be obtained, there are 12 combinations of training super-resolution image pairs. These training super-resolution image pairs can be used as part of the training set for training a neural network later.
[0090] In step S40, it is determined whether the above steps S10 to S33 have been performed on all original fluorescence image sequences. For example, it can be determined whether j has reached the maximum value J. If not, j=j+1 and steps S10 to S32 are repeated; if yes, go to step S50.
[0091] In step S50, the neural network of the denoising module 220 is trained using the training set obtained above. It should be clear to those skilled in the art that the training can be performed in any suitable manner. For example, in each training cycle, any number of training super-resolution image pairs are randomly selected from the training set, and then pixel blocks at the same position of the two super-resolution images before and after the training super-resolution image pair are randomly extracted, and after random rotation and folding, they are used as neural network input images and target images respectively, and the error between the network output and the target image is calculated, and its gradient is back-propagated to update the network parameters. After the network input error converges, the training is stopped and the network parameters are stored.
[0092] In step S60, starting from j=1, for the jth original fluorescence image sequence Y j Fluorescence image y j (j=1, 2, 3, ..., S), and super-resolution reconstruction is performed using the super-resolution reconstruction module 210 to obtain a super-resolution image.
[0093] In step S70, the super-resolution image obtained in step S70 is used as the input of the neural network of the denoising module 220, and the obtained output image is the final denoised super-resolution image.
[0094] In step S80, it is determined whether the above steps S60 to S70 have been performed on all original fluorescence sequences. For example, it can be determined whether j has reached the maximum value J. If not, j=j+1 and steps S60 and S70 are repeated; if yes, go to step S100. In step S100, the self-supervised multimodal structured light limited reconstruction method is exited.
[0095] According to the present application, as an example, a typical neural network that can be used by the denoising module 220 is a U-net network (U-net), and the reference values of its main parameters are shown in Table 1. It should be noted that the neural network that can be used in the present application does not require the use of a specific neural network structure, and typical network structures such as residual network (ResNet), residual channel attention network (RCAN), residual dense neural network (RDN), self-attention network (Transformer) can all achieve the above functions.
[0096]
[0097] Table 1 Main parameters of the neural network
[0098] In addition, the main parameters used in the neural network training process of the present application need to be adjusted according to the specific situation of the data set (signal-to-noise ratio, structural complexity, etc.). A set of parameters for reference is shown in Table 2.
[0099]
[0100] Table 2 Reference parameters of network training process
[0101] Fig. 7A and 7B The schematic diagram shows the training and reconstruction using the original fluorescence image sequence using the above-mentioned method, wherein Fig. 7A The process of training a neural network is Figure 7B It is the process of reconstructing the original fluorescence image sequence using the reconstruction system trained by the neural network.
[0102] This application uses a unique "pixel rearrangement" method combined with super-resolution reconstruction to obtain a training set for the neural network training denoising module. Its main advantages are as follows:
[0103] 1. When training a denoising neural network, the technical solution of the present application can obtain a training set sufficient for training the denoising neural network based on an original fluorescence image sequence with an extremely low signal-to-noise ratio. There is no need to sample the same biological sample multiple times, nor is there any need to irradiate the same biological sample with high-intensity excitation light to obtain an original fluorescence image with a high signal-to-noise ratio, thereby reducing the possibility of damage to the biological sample.
[0104] 2. This application adopts a unique "pixel rearrangement" technology to split four sub-images from the original fluorescence image, and then obtains super-resolution image pairs for training through upsampling, sub-pixel alignment, and super-resolution reconstruction to achieve a fully self-supervised neural network training mechanism.
[0105] 3. The technical solution of the present application can be directly applied to the processing of long-term video data, so that the video data of the present application directly constructs a training set, trains the neural network of the denoising module, and then uses the trained denoising module to denoise the super-resolution microscopic image.
[0106] This application proposes a self-supervised structured light microscopy reconstruction method and system based on pixel rearrangement, which can effectively denoise the structured light microscopy data on the original image with extremely low signal-to-noise ratio, and reconstruct a super-resolution noise-free image with high fidelity. The application of this method significantly reduces the requirements for the signal-to-noise ratio of the original image during structured light microscopy imaging, thereby reducing the light damage to biological samples during imaging and improving the imaging rate, significantly improving the image quality of structured light microscopy and expanding its application range.
[0107] Although the specific embodiments of the present application are described in detail herein, they are provided only for the purpose of explanation and should not be considered to limit the scope of the present application. In addition, it should be clear to those skilled in the art that the various embodiments described in this specification can be used in combination with each other. Various replacements, changes and modifications can be conceived without departing from the spirit and scope of the present application.
Claims
1. A self-supervised multimodal structured light microscopy reconstruction method, comprising: A biological sample is excited by structured light to obtain J original fluorescence image sequences Y generated by the stimulation of the biological sample, wherein J is an integer greater than or equal to 1, each original fluorescence image sequence Y includes S fluorescence images, S is an integer greater than or equal to 2, and each fluorescence image has a pixel size of M*N, wherein M and N are even numbers, For each of the J original fluorescence image sequences Y, a program is run in a computer, wherein: 1) For the i-th fluorescence image y among the S fluorescence images i The first sub-image, the second sub-image, the third sub-image and the fourth sub-image with pixel sizes of (M / 2)*(N / 2) are extracted by rearranging pixels, wherein the two-dimensional pixel matrix y of the first sub-image is i,(2m-1)(2n-1) , the two-dimensional pixel matrix y of the second sub-image i,(2m-1)(2n) , the two-dimensional pixel matrix y of the third sub-image i,(2m)(2n-1) , and the two-dimensional pixel matrix y of the fourth sub-image i,(2m)(2n) Respectively from the i-th fluorescence image y i The two-dimensional pixel matrix y i,MN Select, where m is an integer and = 1, 2, 3, ..., M / 2, n is an integer and = 1, 2, 3, ..., N / 2; 2) performing a 2x pixel upsampling process on the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image to obtain a first upsampled sub-image, a second upsampled sub-image, a third upsampled sub-image, and a fourth upsampled sub-image, all of which have a pixel size of M*N; 3) Performing a pixel shift of [0.5, 0.5] on the two-dimensional pixel matrix of the first upsampled sub-image to obtain a first sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, 0.5] on the two-dimensional pixel matrix of the second upsampled sub-image to obtain a second sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [0.5, -0.5] on the two-dimensional pixel matrix of the third upsampled sub-image to obtain a third sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, -0.5] on the two-dimensional pixel matrix of the fourth upsampled sub-image to obtain a fourth sub-image after two-dimensional sub-pixel shift; 4) classifying the S two-dimensional sub-pixel-shifted first sub-images obtained from the S fluorescence images into a first sub-image group, classifying the S two-dimensional sub-pixel-shifted second sub-images obtained from the S fluorescence images into a second sub-image group, classifying the S two-dimensional sub-pixel-shifted third sub-images obtained from the S fluorescence images into a third sub-image group, and classifying the S two-dimensional sub-pixel-shifted fourth sub-images obtained from the S fluorescence images into a fourth sub-image group; 5) performing structured light image reconstruction on the first sub-image group, the second sub-image group, the third sub-image group, and the fourth sub-image group using a standard structured light reconstruction algorithm to obtain a first super-resolution sub-image, a second super-resolution sub-image, a third super-resolution sub-image, and a fourth super-resolution sub-image; 6) selecting two different super-resolution sub-images from the first super-resolution sub-image, the second super-resolution sub-image, the third super-resolution sub-image, and the fourth super-resolution sub-image to form a training super-resolution image pair, and grouping them into a training set; 7) training a denoising neural network based on the training set; 8) performing super-resolution reconstruction on the S fluorescence images in each of the J original fluorescence image sequences Y using a standard structured light super-resolution reconstruction algorithm to form a super-resolution image, and using the super-resolution image as an input of the denoising neural network to obtain a final super-resolution reconstructed image.
2. The method according to claim 1, characterized in that When training a denoising neural network, one super-resolution sub-image in the training super-resolution image pair is used as training input data, and the other super-resolution sub-image is used as training target data.
3. The method according to claim 2, characterized in that If the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image of the original fluorescence image sequence is an odd number, the corresponding number of row or column pixels are deleted to ensure that the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image are both even numbers.
4. The method according to any one of claims 1 to 3, characterized in that: When training a denoising neural network based on the training set, in each training cycle, a training super-resolution image pair is randomly extracted from the training set, and pixel blocks at the same position in two super-resolution sub-images are randomly extracted, and after randomly rotating and folding, they are used as the denoising neural network input image and target image respectively, the error between the network output and the target image is calculated, and the gradient is back-propagated to update the network parameters.
5. The method according to claim 4, characterized in that The denoising neural network includes but is not limited to a U-type neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
6. The method according to claim 5, characterized in that An optical imaging system (100) is used to excite a biological sample using structured light to obtain j original fluorescence image sequences generated by the stimulation of the biological sample, wherein the optical imaging system (100) includes but is not limited to a two-dimensional structured light system, a three-dimensional structured light system, a lattice light sheet structured light system, and a grazing incidence illumination structured light system.
7. The method according to claim 6, characterized in that The 2x pixel upsampling process is implemented by using nearest neighbor interpolation, bilinear interpolation or biquadratic spline interpolation.
8. A self-supervised multimodal structured light microscopy reconstruction system, comprising: A super-resolution reconstruction module (210), wherein the super-resolution reconstruction module (210) is configured to perform super-resolution reconstruction on the image using a standard structured light super-resolution reconstruction algorithm; as well as A denoising module (220), wherein the denoising module (220) is provided with a denoising neural network, The self-supervised multimodal structured light microscopy reconstruction system is configured as follows: A biological sample is excited by structured light to obtain J original fluorescence image sequences Y generated by the stimulation of the biological sample, wherein J is an integer greater than or equal to 1, each original fluorescence image sequence Y includes S fluorescence images, S is an integer greater than or equal to 2, and each fluorescence image has a pixel size of M*N, wherein M and N are even numbers, The self-supervised multimodal structured light microscopy reconstruction system is further configured to: For each of the J original fluorescence image sequences Y, 1) For the i-th fluorescence image y among the S fluorescence images i The first sub-image, the second sub-image, the third sub-image and the fourth sub-image with pixel sizes of (M / 2)*(N / 2) are extracted by rearranging pixels, wherein the two-dimensional pixel matrix y of the first sub-image is i,(2m-1)(2n-1) , the two-dimensional pixel matrix y of the second sub-image i,(2m-1)(2n) , the two-dimensional pixel matrix y of the third sub-image i,(2m)(2n-1) , and the two-dimensional pixel matrix y of the fourth sub-image i,(2m)(2n) Respectively from the i-th fluorescence image y i The two-dimensional pixel matrix y i,MN Select, where m is an integer and = 1, 2, 3, ..., M / 2, n is an integer and = 1, 2, 3, ..., N / 2; 2) performing a 2x pixel upsampling process on the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image to obtain a first upsampled sub-image, a second upsampled sub-image, a third upsampled sub-image, and a fourth upsampled sub-image, all of which have a pixel size of M*N; 3) Performing a pixel shift of [0.5, 0.5] on the two-dimensional pixel matrix of the first upsampled sub-image to obtain a first sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, 0.5] on the two-dimensional pixel matrix of the second upsampled sub-image to obtain a second sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [0.5, -0.5] on the two-dimensional pixel matrix of the third upsampled sub-image to obtain a third sub-image after two-dimensional sub-pixel shift; performing a pixel shift of [-0.5, -0.5] on the two-dimensional pixel matrix of the fourth upsampled sub-image to obtain a fourth sub-image after two-dimensional sub-pixel shift; 4) classifying the S two-dimensional sub-pixel-shifted first sub-images obtained from the S fluorescence images into a first sub-image group, classifying the S two-dimensional sub-pixel-shifted second sub-images obtained from the S fluorescence images into a second sub-image group, classifying the S two-dimensional sub-pixel-shifted third sub-images obtained from the S fluorescence images into a third sub-image group, and classifying the S two-dimensional sub-pixel-shifted fourth sub-images obtained from the S fluorescence images into a fourth sub-image group; 5) performing structured light image reconstruction on the first sub-image group, the second sub-image group, the third sub-image group, and the fourth sub-image group using a standard structured light reconstruction algorithm to obtain a first super-resolution sub-image, a second super-resolution sub-image, a third super-resolution sub-image, and a fourth super-resolution sub-image; 6) selecting two different super-resolution sub-images from the first super-resolution sub-image, the second super-resolution sub-image, the third super-resolution sub-image, and the fourth super-resolution sub-image to form a training super-resolution image pair, and grouping them into a training set; 7) training a denoising neural network based on the training set; 8) performing super-resolution reconstruction on the S fluorescence images in each of the J original fluorescence image sequences Y using a standard structured light super-resolution reconstruction algorithm to form a super-resolution image, and using the super-resolution image as an input of the denoising neural network to obtain a final super-resolution reconstructed image.
9. The self-supervised multimodal structured light microscopy reconstruction system according to claim 8, characterized in that: When training a denoising neural network, one super-resolution sub-image in the training super-resolution image pair is used as training input data, and the other super-resolution sub-image is used as training target data.
10. The self-supervised multimodal structured light microscopy reconstruction system according to claim 9, characterized in that: If the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image of the original fluorescence image sequence is an odd number, the corresponding number of row or column pixels are deleted to ensure that the maximum number of pixels in the row direction and / or the maximum number of pixels in the column direction of the fluorescence image are both even numbers.
11. The self-supervised multimodal structured light microscopy reconstruction system according to any one of claims 8 to 10, characterized in that: When training a denoising neural network based on the training set, in each training cycle, a training super-resolution image pair is randomly extracted from the training set, and pixel blocks at the same position in two super-resolution sub-images are randomly extracted, and after randomly rotating and folding, they are used as the denoising neural network input image and target image respectively, the error between the network output and the target image is calculated, and the gradient is back-propagated to update the network parameters.
12. The self-supervised multimodal structured light microscopy reconstruction system according to claim 11, characterized in that: The denoising neural network includes but is not limited to a U-type neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
13. The self-supervised multimodal structured light microscopy reconstruction system according to claim 12, characterized in that: An optical imaging system (100) is used to excite a biological sample using structured light to obtain j original fluorescence image sequences generated by the stimulation of the biological sample, wherein the optical imaging system (100) includes but is not limited to a two-dimensional structured light system, a three-dimensional structured light system, a lattice light sheet structured light system, and a grazing incidence illumination structured light system.
14. The self-supervised multimodal structured light microscopy reconstruction system according to claim 13, characterized in that: The 2x pixel upsampling process is implemented by using nearest neighbor interpolation, bilinear interpolation or biquadratic spline interpolation.
