Multi-point confocal image scanning microscope and imaging method

Abstract

The present application relates to the technical fields of optical elements, systems, instruments and imaging, and discloses a multi-point confocal image scanning microscope and an imaging method. According to the present application, an imaging light source assembly, a light source collimation assembly, a multi-focus generation assembly, a multi-focus moving assembly, a multi-focus projection assembly, and a camera detection assembly are arranged along a light path, so as to achieve complete image recording. A unique multi-focus illumination mode provided by the present application is used in combination with an optical locked-phase detection technology, so that a defocus-induced stray signal during thick sample imaging can be effectively removed, and depth imaging is performed; the multi-focus generation assembly is switched, so that switching between a wide-field mode, a confocal mode and a super-resolution mode is easily achieved. Moreover, the present application uses a pixel redistribution algorithm to perform super-resolution reconstruction, and uses a multi-image deconvolution reconstruction algorithm and redundant information in a raw data set to perform frame reduction reconstruction, thereby increasing the imaging speed. Compared with the existing methods, the present application can increase the imaging speed, improve the imaging depth, reduce phototoxicity, and widen the diversity of imaging modes.

Description

Multi-point confocal image scanning microscope and imaging method

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 5, 2023, with application number 202311647921.2 and invention name “A multi-point confocal image scanning microscope and imaging method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the fields of optical elements, systems or instruments and imaging, and in particular to a multi-point confocal image scanning microscope and an imaging method. Background Art

[0003] Microscopic imaging technology is currently widely used in fields such as biomedicine and materials science. However, the resolution of traditional microscopes is limited by the optical diffraction limit, making it unable to meet the demand for high-resolution observation of fine structures. To address this problem, super-resolution imaging technology has been proposed and widely studied.

[0004] Confocal microscopy has become the preferred technique for biologists in fluorescence imaging due to its excellent optical sectioning capabilities and versatility. However, the limited resolution and relatively slow imaging speed of confocal microscopy have hindered its wider application in life sciences. By replacing the single-point detector of confocal microscopy with a detector array, a two-fold resolution enhancement is achieved in image scanning microscopy (ISM) by combining pixel reallocation and deconvolution. To address the relatively slow imaging speed of image scanning microscopy with single-point excitation and camera recording, various extension techniques have been developed based on the ideas of multi-point excitation and optical or digital reconstruction.

[0005] Techniques that rely on optical reconstruction, such as rescanning confocal microscopy, optical photon redistribution microscopy, instant structured light illumination microscopy, and optical photon redistribution spinning disk confocal microscopy, have extremely complex optical pathways. Furthermore, pixel redistribution requires the fluorescence signal to be descanned, rescanned, or passed through a microlens array, which can significantly reduce the fluorescence signal and increase phototoxicity. Furthermore, while full optical reconstruction can replace digital reconstruction, the potential information inherent in image scanning microscopy remains unutilized. Digital reconstruction techniques, such as multifocus structured light illumination microscopy and spinning disk confocal image scanning microscopy, cleverly utilize existing devices or modules, such as digital micromirror devices (DMDs) and spinning disks, seamlessly enabling multi-point data acquisition. However, the ratio between scanning step length and pinhole size is similarly limited by the DMDs or spinning disks, resulting in an excessive number of raw frames required for reconstruction, limiting imaging speed. Furthermore, the redundant information collected in these methods remains largely unutilized. Summary of the Invention

[0006] In order to solve the above-mentioned problems existing in the prior art, the present application provides a multi-point confocal image scanning microscope and an imaging method.

[0007] To achieve the above objectives, this application provides the following solutions.

[0008] In the first aspect, the present application provides a multi-point confocal image scanning microscope, comprising: a signal control component and an imaging light source component, a light source collimation component, a multi-focus generating component, a multi-focus moving component, a multi-focus projection component and a camera detection component arranged along the optical path.

[0009] The imaging light source assembly is used to output light source; the light source collimation assembly is used to collimate the light source output by the imaging light source assembly to obtain a collimated light spot; the multi-focus generation assembly is used to generate multiple excitation focus illumination lights from a single collimated light spot; the multi-focus movement assembly is used to translate the excitation focus illumination light horizontally, vertically or obliquely; the multi-focus projection assembly is used to project the excitation focus illumination light onto the sample surface, and at the same time cooperate with the multi-focus movement assembly to realize multi-focus parallel excitation and scanning of the sample surface; the camera detection assembly is used to realize the detection and output of the fluorescence signal generated by the multi-focus parallel excitation of the sample surface.

[0010] The signal control component is connected to the multi-focus moving component and the camera detection component respectively; the signal control component is used to generate a control signal; the control signal is used to control the multi-focus moving component and the camera detection component to work synchronously.

[0011] In an exemplary embodiment, the imaging light source assembly includes a multi-channel switchable light emitting diode; the light emitted by the light emitting diode is output through a liquid light waveguide, a single-mode optical fiber, or a multi-mode optical fiber of a set size.

[0012] In an exemplary embodiment, the light source collimating component is an aspheric lens, a 90° off-axis parabolic reflector, an aspheric mirror with adjustable focus, an achromatic doublet lens, or an air-spaced doublet lens.

[0013] In an exemplary embodiment, a pinhole array is used to etch small holes of a set size on chrome-plated glass using photolithography technology to form the multi-focus generating component; or, a microlens array, a digital micromirror device, a grating and / or a diffractive optical element (DOE) is used to form the multi-focus generating component.

[0014] In an exemplary embodiment, the multi-focus moving component is a two-dimensional galvanometer mirror.

[0015] In one exemplary embodiment, the multi-focal projection assembly includes a first lens, a second lens, a dichroic mirror, a tube lens, and a microscope objective lens arranged along an optical path.

[0016] After the excitation focus illumination light passes through the first lens, it is translated horizontally, vertically or obliquely by the multi-focus moving component and then enters the second lens.

[0017] The tube lens, the microscope objective lens and the sample surface constitute a microscope host; the microscope host is used to achieve the nominal magnification of the microscope objective lens and clamp the sample.

[0018] The dichroic mirror is used to separate the excitation focus illumination light emitted by the second lens and the fluorescence signal of the sample surface; the excitation focus illumination light emitted by the second lens is reflected into the tube lens; the fluorescence signal of the sample surface is transmitted into the camera detection component.

[0019] In an exemplary embodiment, the camera detection assembly includes a third lens and a camera disposed along the optical path.

[0020] The fluorescence signal of the sample surface is incident into the camera through the third lens; the camera is connected to the signal control component; the camera is used to detect and output the fluorescence signal generated by multi-focus parallel excitation of the sample surface.

[0021] In a second aspect, the present application provides an imaging method using a multi-point confocal image scanning microscope, the method comprising:

[0022] The detection results are obtained using the multi-point confocal image scanning microscope provided above, and the detection results are used as the original image.

[0023] Based on the original image, a pixel redistribution algorithm or a multi-image deconvolution algorithm is used to achieve imaging reconstruction.

[0024] In an exemplary embodiment, the process of implementing imaging reconstruction based on the original image using a pixel redistribution algorithm includes:

[0025] The pixel-level center coordinates are located based on the method of taking the maximum pixel in the neighborhood grayscale value to obtain the candidate center set.

[0026] A plurality of sub-images are intercepted from the defocused image based on the candidate center set to form a sub-image stack.

[0027] Perform sub-pixel center positioning on the sub-image to obtain the sub-pixel coordinates of the sub-image center.

[0028] A two-dimensional Gaussian mask having the same size as the sub-image is generated with the sub-pixel coordinates as the center.

[0029] The two-dimensional Gaussian mask is multiplied with the sub-image as a digital pinhole.

[0030] The sub-images obtained by multiplication are integrated into a canvas having the same size as the detection result according to the sub-pixel coordinates or the pixel center coordinates to obtain a confocal image.

[0031] In an exemplary embodiment, the process of implementing imaging reconstruction based on the original image using a multi-image deconvolution algorithm includes: constructing an imaging model; the imaging model is: I(x, y; x i ,y i )=[M(x,y;x i ,y i )*PSF ILL (x,y)]·Obj(x,y)*PSF em (x,y);

[0032] Where, M(x,y;x i ,y i ) represents the pinhole array, (x,y) represents the spatial coordinates, (x i ,y i ) represents the relative position of the lighting mode of the i-th picture, Obj(x,y) and I(x,y; x i ,y i ) represent the real object and the detected image, PSF ILL (x,y) and PSF em(x, y) represents the excitation point spread function and the detection point spread function, respectively, * represents the convolution operation, and · represents the Hadamard product operation.

[0033] The pinhole array is estimated based on the imaging model.

[0034] Based on the constructed imaging model and the estimation result of the pinhole array, the imaging model is iteratively solved to obtain a reconstructed image.

[0035] According to the specific embodiments provided by this application, the application discloses the following technical effects: This application uses an imaging light source assembly, a light source collimation assembly, a multi-focus generation assembly, a multi-focus movement assembly, a multi-focus projection assembly, and a camera detection assembly arranged along the optical path. The generated multi-focus illumination combined with optical phase-locked detection technology can remove defocus signals in depth imaging. The unique multi-focus illumination mode makes the phototoxicity lower than that of conventional structured light illumination super-resolution microscopes. In addition, by setting up a multi-focus generation assembly, it is easy to switch freely between wide-field, confocal, and super-resolution modes.

[0036] This application uses a pixel reallocation algorithm or a multi-image deconvolution algorithm for super-resolution imaging reconstruction. By utilizing redundant information through the multi-image deconvolution algorithm, the number of reconstruction frames required for super-resolution reconstruction can be reduced, thereby increasing imaging speed. Compared with existing methods, this method can improve 3D resolution, accelerate imaging speed, increase imaging depth, reduce phototoxicity, and enhance the diversity of imaging modes.

[0037] Figures in the specification

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] FIG1 is a schematic diagram of the structure of a multi-point confocal image scanning microscope using a pinhole array provided in this application.

[0040] FIG2 is a schematic diagram of the structure of a multi-point confocal image scanning microscope using a microlens array provided in this application.

[0041] FIG3 is a matrix diagram showing the effects of adapting the multi-point confocal image scanning microscope provided in this application to various types of light sources including single-mode lasers, multi-mode lasers and high-power light-emitting diodes (LEDs).

[0042] Figure 4 shows the principle and effect of optical lock-in detection during the algorithm processing process provided by this application. Figure 4(a) is a raw data diagram, Figure 4(b) is a schematic diagram of elemental data defocusing, Figure 4(c) is a schematic diagram of the Fourier transform of raw data, Figure 4(d) is a comparison diagram of the raw data defocused background processing results, Figure 4(e) is a schematic diagram of raw data superimposed wide field, and Figure 4(f) is a schematic diagram of raw data defocused superimposed wide field.

[0043] Figure 5 is a schematic diagram of the results and resolution of wide-field, confocal, and multi-point confocal image scanning microscopy under fixed cell actin filament samples provided by the present application. Figure 5(a) is a wide-field result diagram under fixed cell actin filament samples, Figure 5(b) is a confocal result diagram under fixed cell actin filament samples, Figure 5(c) is a multi-point confocal image scanning microscopy result diagram under fixed cell actin filament samples, and Figure 5(d) is a schematic diagram of the resolution under fixed cell actin filament samples.

[0044] Figure 6 is a schematic diagram of the results and resolution of wide-field, confocal, and multi-point confocal image scanning microscopes in the YZ plane of a fixed mouse kidney slice sample provided by the present application. Wherein, (a) of Figure 6 is a wide-field result diagram in the YZ plane of a fixed mouse kidney slice sample, (b) of Figure 6 is a confocal result diagram in the YZ plane of a fixed mouse kidney slice sample, (c) of Figure 6 is a multi-point confocal image scanning microscope result diagram in the YZ plane of a fixed mouse kidney slice sample, (d) of Figure 6 is a schematic diagram of the resolution of a fixed mouse kidney slice sample along the Z axis, and (e) of Figure 6 is a schematic diagram of the resolution of a fixed mouse kidney slice sample along the Y axis.

[0045] Figure 7 is a comparison of the frame reduction reconstruction results provided by this application with the wide-field, confocal, and classical image scanning microscope reconstruction results of normal frame number reconstruction. Figure 7(a) is a schematic diagram of the original wide-field data, Figure 7(b) is a confocal result diagram of normal frame number reconstruction, Figure 7(c) is a classical image scanning microscope reconstruction result diagram of normal frame number reconstruction, Figure 7(d) is a joint RL deconvolution result diagram of frame reduction reconstruction, and Figure 7(e) is a fast iterative shrinkage threshold algorithm-group sparsity deconvolution result diagram of frame reduction reconstruction.

[0046] FIG8 is a flow chart of an imaging method using a multi-point confocal image scanning microscope provided in this application.

[0047] Figure numerals: 10 - imaging light source assembly, 20 - light source collimation assembly, 30 - multi-focus generating assembly, 40 - multi-focus moving assembly, 50 - multi-focus projection assembly, 51 - first lens, 52 - second lens, 53 - dichroic mirror, 54 - tube lens, 55 - microscope objective lens, 56 - sample surface, 60 - camera detection assembly, 61 - third lens, 62 - camera, 70 - signal control assembly. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0049] The purpose of this application is to provide a multi-point confocal image scanning microscope and imaging method, which can improve three-dimensional resolution, accelerate imaging speed, increase imaging depth, reduce phototoxicity, and enhance the diversity of imaging modes.

[0050] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] As shown in Figures 1 and 2, the multi-point confocal image scanning microscope provided in the present application includes: a signal control component 70 and an imaging light source component 10, a light source collimation component 20, a multi-focus generating component 30, a multi-focus moving component 40, a multi-focus projection component 50 and a camera detection component 60 arranged along the optical path.

[0052] The imaging light source assembly 10 is used to excite the fluorescent sample (ie, emit light). The light source collimating assembly 20 is used to collimate the output of the imaging light source assembly 10 to achieve uniform illumination of a large light spot.

[0053] The single collimated light spot output by the light source collimating component 20 generates multiple excitation focus illumination lights after passing through the multi-focus generating component 30 .

[0054] The multi-focus moving component 40 is used to translate the multi-focus illumination light generated by the multi-focus generating component 30 horizontally, vertically or obliquely to achieve uniform scanning of the entire sample.

[0055] The multi-focus projection assembly 50 comprises a first lens 51, a second lens 52, a dichroic mirror 53, a tube lens 54, and a microscope objective 55. The multi-focus projection assembly 50 is used to project the generated multi-focus illumination light onto a sample surface 56, while cooperating with the multi-focus movement assembly 40 to achieve multi-focus parallel excitation and uniform scanning of the sample surface 56. The tube lens 54, microscope objective 55, and sample surface 56 constitute the microscope mainframe, which is used to achieve the nominal magnification of the microscope objective and to clamp the sample.

[0056] The camera detection assembly 60 includes a third lens 61 and a camera 62. The camera detection assembly 60 is used to detect and output fluorescence signals. The signal control assembly 70 generates the necessary control signals and outputs them to synchronize the operation of all instruments. Fluorescence signals generated by the sample surface 56 pass through the microscope objective 55, tube lens 54, and dichroic mirror 53 in the multi-focus projection assembly 50 and then enter the third lens 61 and camera 62.

[0057] In an exemplary embodiment, the imaging light source assembly 10 can use a multi-channel fast-switching LED. Other single-mode lasers and multi-mode lasers are also applicable. The effects of adapting multiple types of light sources such as single-mode lasers, multi-mode lasers, and LEDs are shown in Figure 3.

[0058] When using multi-channel, fast-switching, high-power LEDs, the LEDs are output through a liquid light guide of a set size (e.g., 3 mm), or through single-mode or multimode optical fibers. LEDs and multimode lasers offer advantages in eliminating speckle artifacts, providing large field-of-view illumination, and high power. Liquid light guides and multimode optical fibers also offer advantages in transmission efficiency.

[0059] In one exemplary embodiment, the light source collimation assembly 20 may be an aspheric lens. In this case, the light output from the liquid light waveguide is collimated by the aspheric lens. For example, the light source collimation assembly 20 may also employ a 90° off-axis parabolic reflector, an aspheric mirror with adjustable focal length, an achromatic doublet, an air-spaced doublet, or the like for collimation. Light output from single-mode and multimode optical fibers can also be collimated using the aforementioned methods.

[0060] In one exemplary embodiment, the multi-focus generating assembly 30 can be constructed in two ways. One approach is to use a pinhole array (as shown in FIG1 ). For example, photolithography is used to etch small holes of a predetermined size on chrome-plated glass. The light spot emitted by the light source collimation assembly illuminates the pinhole array, creating multi-focus illumination. The pinhole diameter and spacing can be adjusted to accommodate different samples. Another approach is to use a microlens array (as shown in FIG2 ), a DMD, a grating, and a DOE to achieve multi-focus illumination.

[0061] In one exemplary embodiment, the diameter of the pinhole array is large enough to fill the entrance pupil of microscope objective 55 after passing through the multifocal projection assembly. In this optical system configuration, a 40μm diameter pinhole is most suitable. Smaller excitation pinholes can enhance imaging resolution to a certain extent, but will significantly reduce the signal-to-noise ratio. The spot size after focusing through the microlens array should also meet the requirement of a diameter of 40μm or less. For example, a square microlens with a microcell size of 125μm × 125μm, a focal length of 2.7mm, and a numerical aperture of 0.033 is used.

[0062] In an exemplary embodiment, the multi-focus moving component 40 can use a two-dimensional galvanometer, and the generated multi-focus can be evenly covered on the plane by galvanometer scanning. The two-dimensional galvanometer includes two galvanometers. The two galvanometers are fixed together by a mechanical structure to realize two-dimensional scanning, and the two analog signals generated by the signal control component 70 control the movement of the two galvanometers. A one-dimensional galvanometer can also be used to realize scanning of the entire plane through an inclined pinhole arrangement, and scanning can also be realized through a MEMS galvanometer, a resonant mirror or a rotating mirror. The stepping of the galvanometer is strictly limited, and the stepping of the scanning light beam through the second lens 52 at its focus is generally equal to or less than half the pinhole diameter. Smaller steps will result in too many original images being collected, prolonging the final imaging time and increasing phototoxicity. Larger steps will increase the imaging speed, but will reduce the resolution and introduce artifacts.

[0063] In an exemplary embodiment, the dichroic mirror 53 is used to separate the excitation light and the emission light, reflecting the excitation light into the tube lens 54 and transmitting the fluorescent emission light into the camera detection assembly 60 .

[0064] In one exemplary embodiment, the multi-focus moving assembly 40 is generally located between the first lens 51 and the second lens 52. This position is strictly defined, and is conjugated to the entrance pupil of the microscope objective 55 via the second lens 52 and the tube lens 54 in the multi-focus projection assembly. Theoretically, the light beam should remain stationary at this position, but due to the assembly of the two-dimensional galvanometer mirror, there is no way to ensure that the two lenses coincide, resulting in movement of the light spot at the entrance pupil of the microscope objective 55. Alternatively, the two-dimensional galvanometer mirror can be disassembled, and a 4F imaging system can be formed using two lenses, conjugating the two galvanometer mirrors.

[0065] In one exemplary embodiment, the tube lens 54 in the camera detection assembly 60 can utilize a single-lens reflex (SLR) lens to collect the returned fluorescence. This SLR lens enables large-field proportional conjugate imaging. A scientific-grade camera with a large image area can detect the returned fluorescence with high sensitivity. Other relay lens systems can also be used to achieve conjugate imaging, and conventional cameras can also be used for fluorescence detection.

[0066] In an exemplary embodiment, in the present application, the signal control component 70 can output three analog signals and two digital signals. The two analog signals control the multi-focus movement component 40 to achieve scanning in the x- and y-directions. Another analog signal controls the camera in the camera detection component 60, and the two digital signals serve as the address code for the data distributor 74HS138. The global exposure timing output of the camera is connected to the enable port of the data distributor 74HS138. The output of the data distributor 74HS138 is connected to the external trigger of the imaging light source component to achieve wavelength switching and on / off of the light source. Due to multi-point parallel excitation scanning, the total scan length needs to be precisely equal to the pinhole spacing in the multi-focus generation component. In addition, multi-point parallel excitation results in a scanning angle of the galvanometer of approximately 40 μrad. At small mechanical scanning angles, the rapid start and stop of the galvanometer can easily cause uneven scanning steps. Voltage compensation can correct for scanning unevenness and accurately adjust the total voltage, avoiding image artifacts caused by uneven scanning steps.

[0067] In an exemplary embodiment, the present application further provides an imaging method using a multi-point confocal image scanning microscope, as shown in FIG8 , the method includes step 100 and step 101 .

[0068] In step 100, detection results are acquired using the provided multi-point confocal image scanning microscope, and the detection results are used as the original image. Before acquiring detection results using the multi-point confocal image scanning microscope, parameters such as exposure time, multi-focus step distance and number of steps, laser power, number of frames, whether to use multicolor, and imaging field of view need to be set. Different sample types require different pinhole diameters and distances, and therefore the multi-focus step distance and number of steps are also different. For thin samples (thickness < 20 μm), a diameter and pitch ratio of 40 μm:120 μm is generally used, and the number of horizontal and vertical steps is set to 7. For thicker samples (20 μm < thickness < 80 μm), a diameter and pitch ratio of 40 μm:200 μm is generally used, and the number of horizontal and vertical steps is set to 11. For very thick samples (thickness > 80 μm), a diameter and pitch ratio of 40 μm:240 μm is generally used, and the number of horizontal and vertical steps is set to 14. For live cell imaging of tissue samples, a diameter and pitch ratio of 40 μm:160 μm is generally used, and the number of horizontal and vertical steps is set to 9. Exposure time and laser power are generally determined based on sample brightness.

[0069] Step 101: Based on the original image, a pixel redistribution algorithm or a multi-image deconvolution algorithm is used to achieve imaging reconstruction.

[0070] In an exemplary embodiment, in the above step 101 of the present application, the process of implementing imaging reconstruction based on the original image using a pixel redistribution algorithm includes steps 1 to 5.

[0071] Among them, step 1: defocused background processing (optical phase-locked detection).

[0072] Since the illumination light excites the fluorescence signal of the focal plane, it also excites the defocused layer sample, and the emitted fluorescence does not pass through the pinhole again, there is some background signal in the collected original image. In order to improve the positioning accuracy of the illumination bright spot and realize light layer cutting, the acquired raw data needs to be defocused. The characteristics of these signals are that they appear constantly during the scanning process, and the defocus information will appear with the point scanning, so the background defocus signal will show DC characteristics in the time frame. The signal is Fourier transformed pixel by pixel in the acquisition direction. The zero frequency contains the sum of all information, that is, the sum of the DC component and all AC components. Therefore, the background defocus signal can be removed by suppressing the DC component. The principle and results of optical phase-locked detection are shown in Figure 4.

[0073] Step 2: Pixel-level center positioning.

[0074] The main purpose of this step is to extract the sub-image. Since the sub-image is read at the pixel level, it is necessary to locate the pixel-level center of the sub-image (i.e., the center of the bright spot) when the sub-image size is defined in advance. After the original image is processed in step 1, most of the out-of-focus information is removed, and the remaining interference information outside the point illumination will no longer interfere with the location of the bright spot center, so no additional preprocessing is required. If step 1 is not performed and the pixel-level center of the original image is directly located, the following preprocessing process is required.

[0075] First, the image spectrum is subjected to a Butterworth high-pass filter to suppress low-frequency noise and retain high-frequency fluorescence information. Second, morphological processing is performed using an opening operation to smooth the outline of the illumination point, eliminate thin protrusions, and clean up isolated noisy pixels.

[0076] Subsequently, the image is subjected to median filtering to remove isolated pixels in the original image, and Gaussian filtering is performed to make the grayscale value-spatial distribution curve of the illumination point tend to a Gaussian line shape, thereby meeting the conditions for locating the center coordinates.

[0077] By comparing the grayscale values ​​of each pixel with its eight neighborhoods, the local maximum points are screened out as the candidate center point set of the center coordinates.

[0078] Step 3: Extract sub-images.

[0079] A hard threshold is set. Pixels with grayscale values ​​below this threshold in the candidate center point set obtained in step 2 are considered background pixels and removed from the candidate center point set. Coordinates above this threshold are the center coordinates of the sub-image. With this coordinate as the center, a series of sub-images are extracted from the resulting image processed in step 1 with a suitable side length (which should completely encompass the illumination point and exclude other illumination points), forming a sub-image stack.

[0080] Step 4: Apply the digital pinhole.

[0081] In an exemplary embodiment, in order to perform precise positioning and add a digital pinhole, it is necessary to determine the sub-pixel center. First, the sub-image is Gaussian filtered so that the grayscale value-spatial distribution curve of the illumination point tends to a Gaussian line shape, so that it meets the requirements of sub-pixel center positioning. Sub-pixel center positioning can use a quadratic function fitting to determine the center, and the center is determined by the maximum coordinates of the quadratic function fitted in two directions to obtain the sub-pixel coordinates of the sub-image center. With the sub-pixel coordinates as the center, a two-dimensional Gaussian mask of the same size as the sub-image is generated, and it is multiplied with the sub-image as a digital pinhole to achieve further digital defocusing operations.

[0082] Step 5: Pixel redistribution and deconvolution.

[0083] Based on the sub-pixel center and sub-image size, the sub-image coordinate area within the original image size canvas is determined. Since the canvas coordinates are pixel-level coordinates, the sub-image needs to be resampled to the corresponding canvas pixel-level coordinate area. All sub-images are integrated into the canvas in the same way and the corresponding results are saved to obtain the theoretical confocal image.

[0084] Based on the above description, compared to the processing method of integrating the image directly processed by the confocal microscope into the corresponding position of the original image according to the center point coordinates, the pixel redistribution algorithm needs to interpolate the canvas of the original image size, so that the number of canvas coordinate indexes is doubled, resulting in non-integer pixel points relative to the original image, which is equivalent to expanding the width and height of the canvas by twice. Then, the sub-pixel coordinate point closest to the sub-pixel center is found, and its corresponding array index value is the pixel-level center coordinate of the sub-image in the expanded canvas to achieve pixel redistribution. When integrating the sub-image into the expanded canvas, in order to realize the integration of sub-pixel coordinates into pixel-level coordinates, the sub-image needs to be resampled. Finally, the resolution can be further improved by the deconvolution algorithm to achieve a 2-fold image resolution improvement.

[0085] In an exemplary embodiment, multiple images are generated from a single sample. Each image contains only partial information about the sample, but the superposition of all images can form a uniform and complete illumination field. Images that meet these conditions can be reconstructed into a single, higher-quality image through "joint" deconvolution. Frame reduction reconstruction algorithms can fully utilize redundant information and achieve super-resolution reconstruction using fewer original frames. Based on this, multi-image deconvolution algorithms can be divided into joint RL (joint Richardson-Lucy, jRL) deconvolution and fast iterative shrinkage-thresholding algorithm-Group Sparsity (FISTA-GS) deconvolution.

[0086] In an exemplary embodiment, when jRL deconvolution is used to implement imaging reconstruction based on the original image, the process includes the following steps.

[0087] Step 1: Build an imaging model.

[0088] The imaging process of a multi-point confocal image scanning microscope is as follows: the excitation light generated by the imaging light source component is used to generate a multi-focus excitation pattern through the multi-focus generation component. The multi-focus projection component projects the generated multi-focus illumination excitation pattern onto the sample surface, and at the same time cooperates with the multi-focus movement component to achieve uniform scanning of the sample. Finally, the fluorescence signal is amplified by the microscope host component and enters the camera detection component and is detected by the camera. The multi-focus excitation pattern can be represented by the convolution of the pinhole array and the excitation point spread function. The multi-focus illumination excitation pattern illuminates the sample, which is the multi-focus excitation pattern multiplied by the sample function. The process of being received by the camera through the detection light path is the convolution with the detection point spread function. The specific imaging process is shown in the following formula:

[0089] Where, M(x,y;x i ,y i ) represents the pinhole array, (x,y) represents the spatial coordinates, (x i ,y i ) represents the relative position of the lighting mode of the i-th picture, Obj(x,y) and I(x,y; x i ,y i ) represent the real object and the detected image, PSF ILL (x,y) and PSF em (x,y) represents the excitation point spread function and the detection point spread function respectively, * represents the convolution operation, · Represents the Hadamard product operation, pinhole(x,y;x i ,y i) represents the multi-focus illumination mode, which is the convolution of the pinhole array and the excitation point spread function.

[0090] Step 2: Pinhole array estimation.

[0091] In the imaging model (see above), I(x,y; x i ,y i ) is the obtained detection result, the real object Obj(x,y) is the target to be solved, the excitation light wavelength and the emission light wavelength are known, and the excitation point spread function PSF can be approximated according to the Gaussian model ILL (x,y) and the detection point spread function PSF em (x, y), so we also need to solve the pinhole array M(x, y; x i ,y i Because the multi-focus illumination mode excites the sample to emit fluorescence signals, the detected peak light intensity deviates from the pinhole center due to the convolution of the detection point spread function. This makes it impossible to directly solve the pinhole array in the spatial domain. Therefore, we choose to use the frequency domain to solve the pinhole array. The specific solution steps are as follows.

[0092] Step 2.1: Perform a direct Fourier transform on each frame of the original image. Since the original image has a spatially periodic lattice, its Fourier spectrum also has periodically distributed characteristic frequency points, which can be directly obtained by selecting the local highest point. The highest pixel point in both directions is retained as the basis vector in the frequency domain.

[0093] Step 2.2: After the two basic vectors are linearly combined, their corresponding positions should also be characteristic frequency points. Therefore, through polynomial fitting, the sub-pixel characteristic frequency point positions are obtained. The linear coordinates and frequency point positions are fitted with the least squares method to obtain the sub-pixel basic vectors.

[0094] Step 2.3: Obtain accurate spatial basis vectors by coordinate transformation of the frequency domain basis vectors.

[0095] Step 2.4: Generate a spatial lattice using the spatial basic vectors. At this time, the relative positions between the points in the spatial lattice are correct, but there is an overall displacement between the absolute positions and the actual excitation lattice. Compare it with the bright spot of the actual image, sum the deviation vectors of the spatial lattice point and the nearest bright spot center to obtain the offset, and calculate the sub-pixel offset through polynomial fitting. The lattice generated by the spatial basic vector plus the offset is the position on the sample surface corresponding to the actual pinhole array.

[0096] Step 3: Iterative solution.

[0097] The jRL deconvolution is based on the Poisson response of the array detector to the photon and the maximum likelihood estimation, and the reconstructed image is obtained by iterative solution.i (·) is simplified, and the reverse process is expressed as Indicates that I i Represents the i-th image, n represents the number of iterations, and the iterative process is shown in the following formula.

[0098] In the formula, the superscript ∧ means the estimated value of the corresponding parameter, which is used to represent the intermediate results in the iterative process. For I i The estimated value produced at the current iteration with I i Element-wise division of the measured values ​​to measure the deviation between the two, and They are respectively the estimated values ​​of the real object in the nth iteration and the n+1th iteration.

[0099] In an exemplary embodiment, when FISTA-GS deconvolution is used to implement imaging reconstruction based on the original image, the process includes the following steps.

[0100] Step 1: Design the optimization model.

[0101] In addition to jRL deconvolution, constraints can also be imposed on the image reconstruction process. The images collected by the multi-point confocal image scanning microscope are dot matrix images. If constraints are imposed on each frame, it may lead to the loss of image information. Therefore, using group sparsity to impose sparse constraints on the superimposed images of the dot matrix images can further improve the reconstruction quality. The imaging model constructed when FISTA-GS deconvolution realizes imaging reconstruction is the same as the imaging model established in the jRL deconvolution realization presentation reconstruction. However, FISTA-GS deconvolution does not directly use the real object Obj(x,y) as the solution parameter, but instead pinhole(x,y;x) on the real object Obj(x,y) is used. i ,y i ) The information in the pinhole area corresponding to i (r) is used as the solution target, and the relative position (x i ,y i ) is directly represented by the subscript i, then the multi-focus excitation mode is pinhole i (r), the imaging result is I i (r), multi-focus detection diffusion mode is PSF em (r). Based on this, the imaging model can be simplified as: i (r)=[pinhole i (r)·O i (r)]*PSF em (r).

[0102] The design optimization model is:

[0103] Among them, I i,noise (r) represents the original image, L(O i ) represents the objective function. i O i (r) is simplified, representing the i-th original image. f(O) represents the first term on the right side of the equation. This term is the fidelity term, which is used to ensure that the image of the reconstructed object formed by the optical system is as close as possible to the measured image. g(O) represents the second term on the right side of the equation. This term is the group sparsity regularization term, which is a priori constraint imposed on the optimization problem. λ is the regularization coefficient that controls the degree of influence between the two. * is the norm.

[0104] Step 2: Find the gradient.

[0105] For differentiable and continuous convex function f(O) and closed but non-differentiable convex function g(O), the above optimization problem can be solved by proximal gradient descent method. The proximal gradient operator p L (Y) can be expressed as follows:

[0106] Among them, L represents the Lipschitz constant, which plays the role of step size, O represents the vector containing the image in the iterative process, and Y represents another value in the neighborhood of O. is the Hamiltonian operator, which here represents the gradient operation. Similar to the operation of the soft shrinkage threshold, The area, the derivative can be obtained exist The area, the derivative can be obtained exist The area, the derivative can be obtained

[0107] In seeking , it can be decomposed into To explain convolution, define the component I i The coordinates are r′, and the component O i The coordinates of r0, then we can get I i (r')=pinhole i (r0)·O i (r0)·PSF em (r'-r0), so:

[0108] Where, I i It represents the imaging of the object by the optical system. Indicates that the i-th image is the gradient of the function f as a parameter, Indicates that the i-th image is parameter I i The gradient, Represents the gradient of the i-th image, pinhole i (·) represents the multi-focus excitation pattern of the i-th image, PSF em (·) represents the excitation point spread function, and They are and The simplified expression is: and They are and The weight.

[0109] Step 3: Iterative solution.

[0110] Step 3.1: k = 0, give the initial value of image information and the background bias initial value b 0 ,in Can be set directly to I i,noise (r), b k Set to zero initial value, and finally get y k+1 =x k ,t k+1 =1.0.

[0111] Step 3.2: Calculate the forward propagation result: And calculate the forward propagation result f according to the above formula k The gradient of . Indicates the result of the k-th iteration of the i-th picture, b k Represents the background value of the kth iteration, where k represents the number of iterations.

[0112] Step 3.3: According to the forward propagation result f k and the image I formed by the optical system on the original image i,noise Calculate the objective function L(O i ), and get the loss value.

[0113] Step 3.4: Iteratively update x according to the FISTA algorithm k and t k+1 are the intermediate variables in the iterative process.

[0114] Step 3.5: Repeat steps 3.2 to 3.4 until the number of iterations is met, which is:

[0115] Based on the above description, compared with the prior art, this application has the following advantages:

[0116] 1) The use of high-power LEDs or multi-mode lasers in the imaging light source assembly of this application can effectively remove speckle artifacts, provide a large illumination field of view, and shorten exposure time when sufficient power is available.

[0117] 2) The multi-focus generating component uses a photolithographic pinhole array, which has greater flexibility in optimizing and adjusting the ratio of pinhole diameter to spacing. In addition, the price of photolithographic pinhole arrays is very low. When using microlenses to generate multi-focus, the efficiency is relatively high. Compared with the DMD that generates multi-focus and provides scanning, the generation of multi-focus is decoupled from the stepping. When the step size satisfies the Nyquist sampling theorem, the reduction of the pinhole size is not limited by it, which can achieve a larger imaging field of view and reduce the scanning step length. Under small-angle (0.2°) scanning, the switching speed of the galvanometer can reach 6.7KHz, which is similar to the switching speed of the DMD.

[0118] 3) The third lens 61 in the camera detection assembly uses a single-lens reflex lens, which can effectively increase the imaging field of view.

[0119] 4) Changing the spacing between different pinhole arrays or microlens arrays in the multi-focus generating component can balance the imaging speed and the removal of defocus signals, and adapt to samples with different requirements.

[0120] 5) The sparse multi-focal illumination generated by the multi-focal generation component, combined with optical lock-in detection technology and digital pinhole technology, can physically remove out-of-focus signals from depth imaging. Compared to using a digital pinhole alone, out-of-focus background removal is much cleaner. Furthermore, after the original image is subjected to optical lock-in detection technology to remove background, most out-of-focus information is removed. Interference information other than point illumination no longer interferes with the location of the center of the illumination spot, facilitating accurate sub-image extraction (as shown in Figure 4).

[0121] 6) By utilizing the principles of image scanning microscopy, a lateral resolution of 110 nm can be achieved on fixed actin filament samples, doubling the diffraction limit (as shown in Figure 5). On tissue samples, compared to widefield and confocal microscopy, multi-point confocal image scanning microscopy significantly eliminates out-of-focus signal interference, resolving structures more clearly. The lateral and axial resolutions reach 131 nm and 336 nm, respectively, doubling the three-dimensional resolution (as shown in Figure 6).

[0122] 7) Based on the principle of image reconstruction achieved by the pixel reallocation algorithm, it has significantly reduced phototoxicity compared to other super-resolution imaging methods. Due to the extremely short pixel residence time, the single-point power of the single-point scanning confocal is relatively high, which is not friendly to the sample. Since the detection light path of the spinning disk confocal super-resolution microscope has to pass through the microlens array again, the fluorescence signal attenuation is relatively large, so the phototoxicity is relatively large. Compared with the classic structured light illumination super-resolution microscope, the multi-point confocal image scanning microscope is equivalent to the superposition of two wide-field images, while the structured light illumination super-resolution microscope is equivalent to the superposition of three wide-field images. Therefore, the multi-point confocal image scanning microscope has the lowest phototoxicity.

[0123] 8) The multi-image deconvolution algorithm uses redundant information to achieve frame-reduction reconstruction, reducing the number of original frames required for reconstruction, thereby improving the imaging speed (as shown in Figure 7). jRL deconvolution integrates the sample information contained in multiple images from different perspectives to reconstruct a super-resolution image. Since the deconvolution step takes into account the Poisson response of the array detector to photons, jRL deconvolution can also remove Poisson noise in the image to a certain extent. The optimization model of FISTA-GS deconvolution includes a fidelity term and a group sparsity regularization term. The regularization coefficient λ is used to weigh the effects of the two. Compared with general sparse constraints, group sparsity imposes sparse constraints on the superimposed images of the dot matrix images to avoid possible loss of image information, which can further improve the reconstruction quality. Compared with wide-field images and jRL deconvolution, Fista-GS deconvolution can achieve a two-fold resolution improvement.

[0124] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0125] Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of this application. In summary, the contents of this specification should not be construed as limiting this application.

Claims

1. A multi-point confocal image scanning microscope, characterized in that: include: A signal control component and an imaging light source component, a light source collimation component, a multi-focus generating component, a multi-focus moving component, a multi-focus projection component and a camera detection component arranged along the light path; The imaging light source assembly is used to output light source; the light source collimation assembly is used to collimate the light source output by the imaging light source assembly to obtain a collimated light spot; the multi-focus generation assembly is used to generate multiple excitation focus illumination lights from a single collimated light spot; the multi-focus moving assembly is used to translate the excitation focus illumination light horizontally, vertically or obliquely; the multi-focus projection assembly is used to project the excitation focus illumination light onto the sample surface, and cooperate with the multi-focus moving assembly to realize multi-focus parallel excitation and scanning of the sample surface; the camera detection assembly is used to realize the detection and output of the fluorescence signal generated by the multi-focus parallel excitation of the sample surface; The signal control component is connected to the multi-focus moving component and the camera detection component respectively; the signal control component is used to generate a control signal; the control signal is used to control the multi-focus moving component and the camera detection component to work synchronously.

2. The multi-point confocal image scanning microscope according to claim 1, characterized in that: The imaging light source assembly comprises a multi-channel switchable light emitting diode; the light source emitted by the light emitting diode is output through a liquid optical waveguide, a single-mode optical fiber or a multi-mode optical fiber of a set size.

3. The multi-point confocal image scanning microscope according to claim 1, characterized in that: The light source collimating component is an aspheric lens, a 90° off-axis parabolic reflector, an aspheric mirror with adjustable focal length, an achromatic doublet lens or an air-spaced doublet lens.

4. The multi-point confocal image scanning microscope according to claim 1, characterized in that: The multi-focus generating component is formed by etching small holes of set sizes on chrome-plated glass using a pinhole array by photolithography technology; or, the multi-focus generating component is formed by using a microlens array, a digital micromirror device, a grating and / or a diffraction optical element.

5. The multi-point confocal image scanning microscope according to claim 1, characterized in that: The multi-focus moving component is a two-dimensional galvanometer.

6. The multi-point confocal image scanning microscope according to claim 1, characterized in that: The multi-focus projection assembly includes a first lens, a second lens, a dichroic mirror, a tube lens and a microscope objective lens arranged along the optical path; After the excitation focal illumination light passes through the first lens, it is translated horizontally, vertically or obliquely by the multi-focus moving component and then incident on the second lens; The tube lens, the microscope objective lens and the sample surface constitute a microscope host; the microscope host is used to achieve the nominal magnification of the microscope objective lens and the clamping of the sample; The dichroic mirror is used to separate the excitation focus illumination light emitted by the second lens and the fluorescence signal of the sample surface; the excitation focus illumination light emitted by the second lens is reflected into the tube lens; the fluorescence signal of the sample surface is transmitted into the camera detection component.

7. The multi-point confocal image scanning microscope according to claim 1, characterized in that: The camera detection assembly includes a third lens and a camera arranged along the optical path; The fluorescence signal of the sample surface is incident into the camera through the third lens; the camera is connected to the signal control component; the camera is used to detect and output the fluorescence signal generated by multi-focal parallel excitation of the sample surface.

8. An imaging method using a multi-point confocal image scanning microscope, characterized in that: include: Acquire a detection result using the multi-point confocal image scanning microscope as described in any one of claims 1 to 7, and use the detection result as an original image; Based on the original image, a pixel reallocation algorithm or a multi-image deconvolution algorithm is used to achieve imaging reconstruction.

9. The imaging method using a multi-point confocal image scanning microscope according to claim 8, characterized in that: Based on the original image, the process of implementing imaging reconstruction using a pixel redistribution algorithm includes: Based on the method of taking the maximum pixel in the neighborhood grayscale value, the pixel-level center coordinates are located to obtain the candidate center set; intercepting a plurality of sub-images from the defocused image based on the candidate center set to form a sub-image stack; Perform sub-pixel center positioning on the sub-image to obtain the sub-pixel coordinates of the sub-image center; Generate a two-dimensional Gaussian mask with the same size as the sub-image, centered at the sub-pixel coordinates; multiplying the two-dimensional Gaussian mask with the sub-image as a digital pinhole; The sub-images obtained by multiplication are integrated into a canvas having the same size as the detection result according to the sub-pixel coordinates or the pixel center coordinates to obtain a confocal image.

10. The imaging method using a multi-point confocal image scanning microscope according to claim 8, characterized in that: Based on the original image, the process of implementing imaging reconstruction using a multi-image deconvolution algorithm includes: Construct an imaging model; the imaging model is: I(x,y;x i ,y i )=[M(x,y;x i ,y i )*PSF ILL (x,y)]·Obj(x,y)*PSF em (x,y); In the formula, M(x,y;x i ,y i ) represents the pinhole array, (x,y) represents the spatial coordinates, (x i ,y i ) represents the relative position of the lighting mode of the i-th image, Obj(x,y) and I(x,y; x i ,y i ) represent the real object and the detected image, PSF ILL (x,y) and PSF em (x, y) represents the excitation point spread function and the detection point spread function, * represents the convolution operation, and · represents the Hadamard product operation; completing an estimation of a pinhole array based on the imaging model; Based on the constructed imaging model and the estimation result of the pinhole array, the imaging model is iteratively solved to obtain a reconstructed image.

Images