Digital micromirror device-based compressed confocal imaging apparatus and method

Abstract

A kind of compression confocal imaging device and method based on digital micro-mirror device, the device includes confocal microscopic imaging module and single-pixel imaging compression detection module, the confocal microscopic imaging module and the single-pixel imaging compression detection module have the digital micro-mirror device and the sample detection light path between the digital micro-mirror device and sample, wherein, when confocal image acquisition is carried out to sample, the digital micro-mirror device becomes the confocal pinhole of the confocal microscopic imaging module, when single-pixel sampling is carried out to sample, the digital micro-mirror device becomes the confocal pinhole and single-pixel sampling light coding element of the single-pixel imaging compression detection module.Using the device and method of the application, confocal microscopic imaging can be realized under the condition of high speed low light toxicity sampling, the activity of biological sample can be fully guaranteed, and high spatiotemporal resolution imaging effect is provided.

Description

Technical Field

[0001] This invention relates to confocal microscopy imaging technology, and in particular to a compressed confocal imaging device and method based on digital micromirror devices (DMDs). Background Technology

[0002] Fluorescence microscopy is a crucial tool for monitoring cell physiology and solving various biological problems. A major problem with fluorescence microscopy is that defocused light interferes with focal plane imaging, leading to blurred details and reduced contrast. Optical slicing techniques are typically used to address this issue by eliminating defocused light in the focused imaging scene. Among these techniques, laser scanning confocal microscopy is the most widely used in fluorescence imaging. It employs a point-by-point scanning detection method, utilizing a pinhole placed in front of the detector to eliminate defocused light. However, due to the limitations of point-by-point imaging, this technique is slow and requires higher-energy lasers to enhance the signal-to-noise ratio, resulting in bleaching of fluorescent labels and photodamage to the sample.

[0003] To improve the speed of confocal microscopy and reduce phototoxicity, researchers have proposed other confocal imaging schemes, such as rotating disk confocal microscopy and light sheet microscopy. However, the former suffers from fluorescence crosstalk between pinholes and lacks the flexibility to be combined with other imaging techniques. Light sheet microscopy requires specialized sample preparation and fixation and suffers from non-uniform illumination. Furthermore, both solutions require highly sensitive detector arrays, such as scientific complementary metal-oxide-semiconductor (sCMOS) and electron-multiplying charge-coupled devices (EMCCDs). Compared to single-pixel detectors, these arrays have narrow spectral response ranges, low temporal resolution, and high cost.

[0004] In summary, traditional confocal microscopy suffers from problems such as high phototoxicity, slow imaging speed, and high cost.

[0005] Achieving confocal microscopy with low phototoxicity and high spatiotemporal resolution is a problem that needs to be solved by existing technologies.

[0006] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The main objective of this invention is to overcome the deficiencies of the aforementioned background technology and provide a compression confocal imaging device and method based on digital micromirror devices.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A compressed confocal imaging device based on a digital micromirror device includes a confocal microscopy imaging module and a single-pixel imaging compressed detection module. The confocal microscopy imaging module and the single-pixel imaging compressed detection module share a digital micromirror device and a sample detection optical path between the digital micromirror device and the sample. When confocal image acquisition is performed on the sample, the digital micromirror device becomes the confocal pinhole of the confocal microscopy imaging module. When single-pixel sampling is performed on the sample, the digital micromirror device becomes both the confocal pinhole and the single-pixel sampling optical encoding element of the single-pixel imaging compressed detection module.

[0010] Furthermore, the sample detection optical path includes an objective lens and a tube lens disposed between the objective lens and the digital micromirror device.

[0011] Furthermore, the single-pixel imaging compression detection module includes a first dichroic mirror, a first lens, a single-photon detector, a second lens, and a first laser. The laser emitted by the first laser passes through the second lens and the first dichroic mirror and is incident on the digital micromirror device. After being modulated by the digital micromirror device, it enters the sample detection optical path. The signal light returning from the sample detection optical path is reflected on the digital micromirror device, reflected again by the first dichroic mirror, and reaches the first lens. The light passing through the first lens is converged at a point and compressed and collected by the single-photon detector. Preferably, the single-photon detector is a photomultiplier tube (PMT).

[0012] Furthermore, the confocal microscopy imaging module includes a second dichroic mirror, a third lens, a two-dimensional image acquisition device, a fourth lens, and a second laser. The laser emitted by the second laser passes through the fourth lens and the second dichroic mirror and is incident on the digital micromirror device. After being modulated by the digital micromirror device, it enters the sample detection optical path. The signal light returning from the sample detection optical path is reflected on the digital micromirror device, reflected again by the second dichroic mirror, and reaches the third lens. The light passing through the third lens is then relayed and imaged onto the two-dimensional image acquisition device. Preferably, the two-dimensional image acquisition device is an sCMOS camera.

[0013] Furthermore, when acquiring confocal images of a sample, the digital micromirror device completes all modulation mode changes within one exposure time of the camera, achieving full coverage detection of all points in the sample detection area and obtaining a complete confocal two-dimensional image; or, each modulation mode change of the digital micromirror device triggers an exposure, and multiple exposures are performed until all points in the sample detection area are covered, and then the images acquired in each exposure are stacked to obtain a complete confocal image.

[0014] A compression confocal imaging method based on a digital micromirror device includes the following steps:

[0015] S1. The single-pixel sampling data and confocal image of the sample are obtained using the compressed confocal imaging device based on the digital micromirror device, respectively, as the paired dataset required for training the deep learning network.

[0016] S2. Train the deep learning network using the paired dataset;

[0017] S3. Collect single-pixel sampling data of the target sample, and reconstruct a two-dimensional confocal image using the one-dimensional compressed single-pixel sampling data through the deep learning network.

[0018] Further, in step S1, obtaining the single-pixel sampling data of the sample includes: generating a random binary sparse matrix as the measurement matrix for modulating the digital micromirror device, where an element "1" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "on" state, and an element "0" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "off" state, with at least 4 "0"s between adjacent "1"s; changing the modulation mode of the digital micromirror device multiple times, and obtaining a one-dimensional intensity signal of the single-pixel sample after multiple acquisitions.

[0019] Further, in step S1, obtaining the confocal image of the sample includes: generating a parallel fringe binary matrix to modulate the digital micromirror device as a confocal pinhole for confocal image acquisition. The elements in the same column of the parallel fringe matrix are all "1" or "0". The element "1" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "on" state, and the element "0" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "off" state. Every n columns make one microreflection unit in an "on" state, preferably n is 10 or more, more preferably n∈[10,20]. Among them, when all the units of the digital micromirror device in the "on" state are translated by one unit, the camera is triggered to expose and the corresponding two-dimensional image data is recorded until the micromirror array has been scanned once. The recorded two-dimensional image data are stitched together to obtain the confocal two-dimensional true image signal of the entire field of view.

[0020] Further, in step S3, reconstructing the two-dimensional confocal image includes:

[0021] Based on the principle of compressed sensing, the following algorithm is used to reconstruct a two-dimensional confocal image signal x from a compressed one-dimensional intensity signal y:

[0022] y = Ax + e (1)

[0023] Where A is the measurement matrix and e is the noise, the solution is optimized using the following soft thresholding iterative algorithm:

[0024]

[0025] Where F(·) is the trained convolutional neural network, k represents the number of iterations, and x (k) Let r represent the signal value at the k-th iteration, r be the intermediate iteration value, and θ be the regularization parameter.

[0026] Furthermore, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the method.

[0027] The present invention has the following beneficial effects:

[0028] This invention proposes a compressed confocal imaging device and method based on digital micromirror devices that combines single-pixel imaging technology with confocal microscopy imaging technology, effectively solving the problems of slow imaging speed and high phototoxicity faced by traditional confocal imaging.

[0029] Unlike traditional confocal microscopy imaging devices that use scanning galvanometers to scan points to obtain two-dimensional confocal images, the compressed confocal imaging device based on digital micromirror devices of this invention can obtain single-pixel sampling data and confocal images of samples. These can be used as paired datasets to train a deep learning network. Based on this, by collecting single-pixel sampling data of the target sample, the two-dimensional confocal image can be reconstructed using the one-dimensional compressed single-pixel sampling data through the deep learning network.

[0030] This invention employs a high-throughput compressed acquisition scheme using single-pixel sampling. After modulating light with a digital micromirror device, the light is collected by a single-point detector, acquiring single-pixel sampled data of the target sample. Then, the one-dimensional compressed single-pixel sampled data is used to reconstruct a two-dimensional confocal image through a trained deep learning network. Through this invention, utilizing compressed sensing technology, only undersampled data acquisition is required to perfectly reconstruct a two-dimensional confocal microscopy image using one-dimensional compressed single-pixel sampled data, significantly increasing light throughput, reducing phototoxicity, and achieving low-phototoxicity, high spatiotemporal resolution confocal microscopy. Furthermore, the equipment used in this invention is simple, effectively saving costs.

[0031] Because this invention enables confocal microscopy imaging under high-speed, low-phototoxicity sampling conditions, it fully preserves the viability of biological samples and provides high spatiotemporal resolution imaging. Compared to traditional confocal imaging systems, this system significantly improves sampling speed, reduces phototoxicity, and better meets the needs of biological observation, providing a valuable research tool for understanding cellular life processes. This invention can be widely applied in the biomedical and materials science fields, providing an effective tool for observing biological microscopic samples and detecting surface morphology features of materials. Attached Figure Description

[0032] Figure 1AThis is a schematic diagram of the light illumination path of a compression confocal microscope system according to an embodiment of the present invention.

[0033] Figure 1B This is a schematic diagram of the light acquisition path of a compression confocal microscope system according to an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the reflection mechanism of a single micromirror in a digital micromirror device (DMD) according to an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the modulation method of a digital micromirror device (DMD) according to an embodiment of the present invention.

[0036] Figure 4 This is a flowchart of a compressed confocal imaging process according to an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram of a random sparse sampling matrix according to an embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram of a parallel stripe matrix according to an embodiment of the present invention. Detailed Implementation

[0039] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0040] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for fixing, coupling, or communication.

[0041] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0043] Confocal microscopy, with its high image contrast and 3D imaging capabilities, is widely used in biological research. However, its detection requires strong laser illumination of the sample, which often leads to photodamage to cells and fluorescent dyes, severely affecting the observation of cellular life activities. Furthermore, the relatively slow imaging speed of confocal microscopy also limits its application in biological research.

[0044] Single-pixel imaging (SPI) is an emerging imaging technology that can replace detector array imaging. Based on compressed sensing theory, it utilizes the encoded modulation of signals in the observed scene, requiring only undersampled data acquisition to reconstruct the original signal. It possesses various desirable characteristics, including extremely weak signal detection, undersampling, and higher temporal resolution.

[0045] See Figure 1A and Figure 1B This invention provides a compressed confocal imaging device based on a digital micromirror device 4, comprising a confocal microscopy imaging module and a single-pixel imaging compressed detection module. The confocal microscopy imaging module and the single-pixel imaging compressed detection module share a common digital micromirror device 4 and a sample detection optical path between the digital micromirror device 4 and the sample. When confocal image acquisition is performed on the sample, the digital micromirror device 4 becomes the confocal pinhole of the confocal microscopy imaging module. When single-pixel sampling is performed on the sample (e.g., biofluorescent sample 1), the digital micromirror device 4 becomes both the confocal pinhole and the single-pixel sampling optical encoding element of the single-pixel imaging compressed detection module.

[0046] like Figure 1A and Figure 1B As shown, in a preferred embodiment, the sample detection optical path includes an objective lens 2 and a tube lens 3 disposed between the objective lens 2 and the digital micromirror device 4.

[0047] like Figure 1A and Figure 1BAs shown, in a preferred embodiment, the single-pixel imaging compression detection module includes a first dichroic mirror 5, a first lens 6, a single-photon detector 7, a second lens 8, and a first laser 9. The laser emitted by the first laser 9 passes through the second lens 8 and the first dichroic mirror 5 and is incident on the digital micromirror device 4. After being modulated by the digital micromirror device 4, it enters the sample detection optical path. The signal light returning from the sample detection optical path is reflected on the digital micromirror device 4, reflected again by the first dichroic mirror 5, and reaches the first lens 6. The light passing through the first lens 6 is converged at a point and compressed and collected by the single-photon detector 7. Preferably, the single-photon detector 7 is a photomultiplier tube (PMT).

[0048] like Figure 1A and Figure 1B As shown, in a preferred embodiment, the confocal microscopy imaging module includes a second dichroic mirror 10, a third lens 11, a two-dimensional image acquisition unit 12, a fourth lens 13, and a second laser 14. The laser emitted by the second laser 14 passes through the fourth lens 13 and the second dichroic mirror 10 and is incident on the digital micromirror device 4. After being modulated by the digital micromirror device 4, it enters the sample detection optical path. The signal light returning from the sample detection optical path is reflected on the digital micromirror device 4, reflected again by the second dichroic mirror 10, and reaches the third lens 11. The light transmitted through the third lens 11 is relayed and imaged onto the two-dimensional image acquisition unit 12. Preferably, the two-dimensional image acquisition unit 12 is an sCMOS (scientific complementary metal-oxide-semiconductor) camera.

[0049] In some embodiments, when a sample is acquired using confocal imaging, the digital micromirror device 4 completes all modulation mode changes within a single camera exposure time, achieving full coverage detection of all points in the sample detection area and obtaining a complete confocal two-dimensional image.

[0050] In other embodiments, each modulation mode change of the digital micromirror device 4 triggers an exposure, and multiple exposures are performed until all points in the sample detection area are covered. Then, the images acquired from each exposure are stacked to obtain a complete confocal image.

[0051] Single-pixel sampling utilizes a digital micromirror device (DMD) to spatially modulate the fluorescence emitted by the sample, which is then detected by a single-pixel detector (e.g., a photomultiplier tube, PMT) with high temporal resolution. Furthermore, single-pixel imaging (SPI) employs compressed sensing, requiring only undersampled data, thus reducing data acquisition time and increasing imaging speed. Additionally, single-pixel sampling utilizes a high-throughput convergent collection detection method, enhancing the detection capability for weak signals and thus providing low phototoxicity detection capabilities.

[0052] This invention proposes a compressed confocal microscopy imaging system based on a digital micromirror device (DMD). The DMD serves simultaneously as the pinhole detector for confocal microscopy and the optical encoding element for single-pixel imaging. This design enables both confocal image acquisition and single-pixel sampling. The resulting data from both methods are used to construct the training dataset required for a deep learning network. The trained deep learning network can then directly reconstruct high-quality confocal images using the single-pixel sampled data.

[0053] This invention also provides a compressed confocal imaging method based on a digital micromirror device, comprising the following steps:

[0054] S1. A compression confocal imaging device based on a digital micromirror device as described in the foregoing embodiments (e.g., Figure 1A and Figure 1B As shown, the single-pixel sampling data and confocal images of the samples are obtained respectively, which serve as the paired dataset required for training the deep learning network;

[0055] S2. Train the deep learning network using the paired dataset;

[0056] S3. Collect single-pixel sampling data of the target sample, and reconstruct a two-dimensional confocal image using the one-dimensional compressed single-pixel sampling data through the deep learning network.

[0057] In a preferred embodiment, step S1, obtaining single-pixel sampling data of the sample includes: generating a random binary sparse matrix as a measurement matrix for modulating the digital micromirror device, where an element "1" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "on" state, and an element "0" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "off" state, with at least 4 "0"s between adjacent "1"s; changing the modulation mode of the digital micromirror device multiple times, and obtaining a one-dimensional intensity signal of the single-pixel sample after multiple acquisitions.

[0058] In a preferred embodiment, step S1, obtaining the confocal image of the sample includes: generating a parallel fringe binary matrix to modulate the digital micromirror device as a confocal pinhole for confocal image acquisition; all elements in the same column of the parallel fringe matrix are either "1" or "0"; an element "1" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "on" state, and an element "0" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "off" state; every n columns, a microreflection unit is made to be in an "on" state, preferably n is 10 or more, more preferably n∈[10,20]; wherein, when all units of the digital micromirror device in the "on" state are translated by one unit, the camera is triggered to expose, and the corresponding two-dimensional image data is recorded until the micromirror array has been scanned once, and the recorded two-dimensional image data is stitched together to obtain the confocal two-dimensional true image signal of the entire field of view.

[0059] In a preferred embodiment, step S3, reconstructing the two-dimensional confocal image, includes:

[0060] Based on the principle of compressed sensing, the following algorithm is used to reconstruct a two-dimensional confocal image signal x from a compressed one-dimensional intensity signal y:

[0061] y = Ax + e (1)

[0062] Where A is the measurement matrix and e is the noise, the solution is optimized using the following soft thresholding iterative algorithm:

[0063]

[0064] Where F(·) is the trained convolutional neural network, k represents the number of iterations, and x (k) Let r represent the signal value at the k-th iteration, r be the intermediate value at the k-th iteration, and θ be the regularization parameter.

[0065] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the compression confocal imaging method based on a digital micromirror device.

[0066] The following describes specific embodiments of the present invention.

[0067] In some embodiments of the present invention, a compressed confocal imaging device based on a digital micromirror device is provided, which can achieve low phototoxicity and high spatiotemporal resolution confocal data acquisition, thereby obtaining the paired dataset required for training deep learning networks. The specific structure of this compressed confocal imaging device is as follows: Figure 1A and Figure 1B As shown, where:

[0068] Biofluorescent sample 1 is the object being observed.

[0069] Objective lens 2 is used for optical magnification of the observed object.

[0070] The tube lens 3 is used in conjunction with the objective lens 2 to achieve magnified imaging, converging the magnified light from the objective lens 2 onto the plane of the digital micromirror device 4.

[0071] The digital micromirror device (DMD) 4 can display different coding matrices and simultaneously realize the confocal detection pinhole function and single-pixel sampling coding function. It can reflect light to the first dichroic mirror 5 or the second dichroic mirror 10.

[0072] The first dichroic mirror 5 allows laser light to be transmitted to the digital micromirror device 4 and reflects fluorescence to the first lens 6.

[0073] The first lens 6 can focus light into a single point.

[0074] The single-photon detector 7 is used to detect the light signal converged by the first lens 6.

[0075] The second lens 8 can expand the laser beam.

[0076] The first laser 9 is used to emit laser light to excite the sample, thereby emitting fluorescence.

[0077] The second dichroic mirror 10 allows laser light to be transmitted to the digital micromirror device 4 and reflects fluorescence to the third lens 1.

[0078] The third lens 11 is used to relay the image on the plane of the digital micromirror device 4 onto the two-dimensional image acquisition unit 12.

[0079] The two-dimensional image acquisition device 12 is a camera capable of capturing two-dimensional digital images, such as an sCMOS camera.

[0080] The fourth lens 13 can expand the laser beam.

[0081] The second laser 14 is used to emit laser light to excite the sample, thereby emitting fluorescence.

[0082] In some embodiments of the present invention, the specific steps for achieving compressed confocal imaging are as follows:

[0083] Modulated excitation light

[0084] Laser 9 emits a laser beam, which is expanded by lens 8 and irradiates digital micromirror device 4 in parallel. The excitation light is encoded and modulated using the independent addressing modulation function of digital micromirror device 4. The modulation principle is as follows: Figure 2 and Figure 3 As shown.

[0085] like Figure 2The figure illustrates the reflection mechanism of a single micromirror in a digital micromirror device (DMD). The thin solid lines in the figure represent the baseline and normal of a single micromirror at its initial position. Each micromirror in the DMD can be independently controlled to flip in +12° and -12° directions, thus reflecting incident light along the initial normal direction to two symmetrical flip directions. This achieves light modulation. This modulation method can be arbitrarily designed, such as... Figure 3 As shown in the diagram, different elements in the matrix represent individual micromirrors of the Digital Micromirror Device (DMD). The binary "1" and "0" states of the control signal correspond to the "on" (+12°) and "off" (-12°) states of the micromirror, respectively. A micromirror in the "on" state can reflect light onto the corresponding area of ​​the sample. A micromirror in the "off" state cannot emit light onto the sample area. The core of confocal microscopy is the "illumination pinhole" and the "detection pinhole." The function of the "illumination pinhole" is to ensure that the laser light passing through it only illuminates the sample at the focal plane, while samples off-focus are not illuminated. In this case, the modulation of the DMD is equivalent to the "illumination pinhole" of the confocal microscope.

[0086] The compression detection module obtains a one-dimensional intensity signal.

[0087] The modulated laser excites the fluorescence in the corresponding region of the sample. Specifically, the fluorescence of the sample plane at the confocal position of the digital micromirror device (DMD) in the "on" state is excited, while the fluorescence of the sample plane at the confocal position of the DMD in the "off" state is not excited. The excited fluorescence returns along the original path and passes through the DMD in the "on" state (at this time, the DMD acts as a "detector pinhole" for confocal focusing). It is reflected again by the dichroic mirror 5 and reaches the lens 6. The lens 6 focuses the light to a point and is compressed and collected by the single-photon detector 7. Finally, by changing the modulation mode of the DMD and collecting the light multiple times, a one-dimensional intensity signal is obtained.

[0088] Acquisition of True Value Data for High-Intensity Light Samples

[0089] The laser emitted by the second laser 14 is expanded by the lens 13 and modulated by the digital micromirror device 4 to excite fluorescence in the corresponding area of ​​the sample. The fluorescence returns along the excitation path and is reflected by the second dichroic mirror 10 before being imaged onto the two-dimensional image acquisition device 12 (sCOMS camera). At this time, the lens is adjusted so that the pixels of the sCOMS camera correspond one-to-one with the micromirrors of the digital micromirror device (DMD). The modulation scanning mode on the DMD is changed rapidly. Within one exposure time of the camera, the DMD completes all modulation mode changes and achieves full coverage detection of all points in the detection area, obtaining a complete confocal two-dimensional image.

[0090] True data acquisition of weak fluorescence samples

[0091] The laser emitted by the second laser 14 is expanded by the lens 13 and modulated by the digital micromirror device 4 to excite fluorescence in the corresponding area of ​​the sample. The fluorescence returns along the excitation path and is reflected by the second dichroic mirror 10 before being imaged onto the two-dimensional image acquisition device 12 (sCOMS camera). At this time, the lens is adjusted so that the pixels of the sCOMS camera correspond one-to-one with the micromirrors of the digital micromirror device (DMD). To ensure a sufficiently high signal-to-noise ratio for the weak fluorescence sample signal, a slow-speed change in the modulation scanning mode on the DMD is used, and each mode change triggers camera exposure. Multiple scans are performed until all points in the sample detection area are covered. Finally, the camera exposure images from all scanning modes are stacked to obtain a complete confocal image.

[0092] In other embodiments of the present invention, a compression reconstruction method based on a deep learning model is provided. This method uses confocal data acquired by a compression confocal imaging device based on a digital micromirror device, as described in the preceding embodiments of the present invention, to obtain a paired dataset required for training a deep learning network. Through deep learning network training, high-quality confocal images can be reconstructed using compressed data obtained by a molding device employing single-pixel imaging compression detection. The specific method is as follows:

[0093] Confocal sampling and reconstruction methods

[0094] By sampling confocal image data using compressed sensing principles, image reconstruction can be achieved using undersampled data, significantly improving imaging speed. Furthermore, this high-throughput collection method is highly sensitive to extremely weak signals, enabling observation of samples with low phototoxicity. However, traditional compressed sensing reconstruction algorithms suffer from poor reconstruction quality, failing to meet the requirements of high contrast and high spatial resolution in confocal images. To achieve high-quality confocal microscopy, a compressed confocal microscopy system based on symmetric measurement using a digital micromirror device (DMD), as described in the preceding embodiments of this invention, can be used. For different samples or scenarios, steps 3 and 4 can be used to acquire corresponding one-dimensional (one-dimensional intensity signal) and two-dimensional (confocal two-dimensional image) paired datasets for deep learning network model training. In response, the applicant proposes a compressed reconstruction algorithm based on a deep learning model to reconstruct high-quality two-dimensional confocal images using one-dimensional compressed data. The compressed confocal imaging process is as follows: Figure 4 As shown, the specific steps are as follows:

[0095] Random Compressed Confocal Sparse Matrix Generation

[0096] A random binary sparse matrix was written using MATLAB software as the measurement matrix. In this matrix, an element "1" corresponds to a single micromirror of the Digital Micromirror Device (DMD) being in the "on" state, and an element "0" corresponds to a single micromirror of the DMD being in the "off" state. Furthermore, to ensure the DMD performs pinhole functionality, the number of "1"s in the matrix must be much smaller than the number of "0"s, and adjacent "1"s must be separated by at least four "0"s. This effectively eliminates interference from defocused light and yields high-quality one-dimensional light intensity measurement signal data. The random sparse sampling matrix is ​​as follows: Figure 5 As shown.

[0097] Generating true confocal images of weak fluorescence

[0098] Parallel fringe binary matrices are generated using MATLAB software to serve as the "illumination pinhole" and "detection pinhole" for confocal image acquisition. Elements in the same column of the parallel fringe matrix are identical (either "1" or "0"). An element "1" in the matrix corresponds to a single micromirror of the Digital Micromirror Device (DMD) being in the "on" state, and an element "0" corresponds to a single micromirror of the DMD being in the "off" state. Every n columns are arranged so that one microreflection unit is in the "on" state, where n ∈ [10, 20]. Figure 6 The parallel fringe matrix is ​​shown. The digital micromirror device (DMD) is controlled by a program to shift all "on" cells by one unit and trigger camera exposure, recording and storing the corresponding fluorescence data until all microreflective cells have been scanned. The fluorescence data recorded by all cameras are then stitched together to obtain the true-to-life two-dimensional fluorescence confocal image data of the entire field of view.

[0099] Deep learning-based compression and reconstruction algorithm

[0100] To reconstruct a two-dimensional confocal image signal (x) from a compressed one-dimensional intensity signal (y), the principle of compressed sensing needs to be employed.

[0101] y = Ax + e (1)

[0102] Where A is the measurement matrix and e is noise. Since the dimension of y is much smaller than that of x, optimization is required. A soft thresholding iterative algorithm is used:

[0103]

[0104] Where F(·) is the trained convolutional neural network, k represents the number of iterations, and x (k) Let r represent the signal value at the k-th iteration, r be the intermediate iteration value, and θ be the regularization parameter.

[0105] For training the neural network, steps 3 and 4 can be used to collect corresponding one-dimensional and two-dimensional paired training datasets. Since the data comes from the actual sampling results of the current system, the performance of the network is greatly improved.

[0106] Compared with traditional confocal imaging methods, the present invention has the following significant advantages:

[0107] This invention proposes a compressed confocal imaging device and method based on digital micromirror devices that combines single-pixel imaging technology with confocal microscopy imaging technology, effectively solving the problems of slow imaging speed and high phototoxicity faced by traditional confocal imaging.

[0108] Unlike traditional confocal microscopy imaging devices that use scanning galvanometer point scanning to obtain two-dimensional confocal images, the compressed confocal imaging device based on digital micromirror devices of this invention can obtain single-pixel sampling data and confocal images of samples. These can be used as paired datasets to train deep learning networks. Based on this, by collecting single-pixel sampling data of the target sample, the two-dimensional confocal image can be reconstructed using the one-dimensional compressed single-pixel sampling data through the deep learning network.

[0109] This invention employs a high-throughput compressed acquisition scheme using single-pixel sampling. After modulating light with a digital micromirror device, the light is collected by a single-point detector, acquiring single-pixel sampled data of the target sample. Then, the one-dimensional compressed single-pixel sampled data is used to reconstruct a two-dimensional confocal image through a trained deep learning network. Through this invention, utilizing compressed sensing technology, only undersampled data acquisition is required to perfectly reconstruct a two-dimensional confocal microscopy image using one-dimensional compressed single-pixel sampled data, significantly increasing light throughput, reducing phototoxicity, and achieving low-phototoxicity, high spatiotemporal resolution confocal microscopy. Furthermore, the equipment used in this invention is simple, effectively saving costs.

[0110] Because this invention enables confocal microscopy imaging under high-speed, low-phototoxicity sampling conditions, it fully preserves the viability of biological samples and provides high spatiotemporal resolution imaging. Compared to traditional confocal imaging systems, this system significantly improves sampling speed, reduces phototoxicity, and better meets the needs of biological observation, providing a valuable research tool for understanding cellular life processes. This invention can be widely applied in the biomedical and materials science fields, providing an effective tool for observing biological microscopic samples and detecting surface morphology features of materials.

[0111] Because this invention offers advantages such as low cost, low phototoxicity, and high imaging speed in microscopic imaging systems, it effectively addresses the needs of biological researchers for low-phototoxicity, rapid confocal imaging. It can widely replace existing microscopic confocal imaging systems, realizing confocal microscopy with low phototoxicity, high imaging speed, and low cost, and has broad application prospects in the field of microscopic imaging. This invention can be widely applied in the biomedical and materials science fields, providing an effective tool for observing biological microscopic samples and detecting surface morphology features of materials.

[0112] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0113] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0114] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0115] The storage medium can be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0116] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0119] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0121] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0122] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0123] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0124] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A compression confocal imaging method based on a digital micromirror device, using a compression confocal imaging apparatus, characterized in that, The compressed confocal imaging device includes a confocal microscopy imaging module and a single-pixel imaging compressed detection module. The confocal microscopy imaging module and the single-pixel imaging compressed detection module share a common digital micromirror device and a sample detection optical path between the digital micromirror device and the sample. When acquiring a confocal image of the sample, the digital micromirror device becomes the confocal pinhole of the confocal microscopy imaging module; when performing single-pixel sampling of the sample, the digital micromirror device becomes both the confocal pinhole and the single-pixel sampling optical encoding element of the single-pixel imaging compressed detection module. The method includes the following steps: S1. Using the aforementioned compressed confocal imaging device based on a digital micromirror device, single-pixel sampling data and confocal images of the samples are obtained respectively, serving as the paired dataset required for training the deep learning network; wherein, obtaining the single-pixel sampling data of the samples includes: generating a random binary sparse matrix as the measurement matrix for modulating the digital micromirror device, where an element "1" in the matrix corresponds to a single micromirror of the digital micromirror device being in an "on" state, and an element "0" in the matrix corresponds to a single micromirror of the digital micromirror device being in a "off" state, with at least 4 "0"s between adjacent "1"s; changing the modulation mode of the digital micromirror device multiple times, and obtaining a one-dimensional intensity signal of the single-pixel sample after multiple acquisitions; wherein, obtaining the confocal image of the samples The process includes: generating a parallel fringe binary matrix to modulate the digital micromirror device (DMD), which serves as a confocal pinhole for confocal image acquisition. Elements in the same column of the parallel fringe matrix are either "1" or "0". An element "1" in the matrix corresponds to a single micromirror of the DMD being in an "on" state, and an element "0" corresponds to a single micromirror of the DMD being in a "off" state. Every n columns, a microreflection unit is made to be in an "on" state. The camera is triggered to expose when all units of the DMD in the "on" state are translated by one unit, recording the corresponding two-dimensional image data. This process continues until the entire micromirror array has been scanned. The recorded two-dimensional image data is then stitched together to obtain the confocal two-dimensional ground truth image signal of the entire field of view. S2. Train the deep learning network using the paired dataset; S3. Collect single-pixel sampling data of the target sample, and reconstruct a two-dimensional confocal image using the one-dimensional compressed single-pixel sampling data through the deep learning network.

2. The compression confocal imaging method based on digital micromirror devices as described in claim 1, characterized in that, The sample detection optical path includes an objective lens and a tube lens disposed between the objective lens and the digital micromirror device.

3. The compression confocal imaging method based on digital micromirror devices as described in claim 1, characterized in that, The single-pixel imaging compression detection module includes a first dichroic mirror, a first lens, a single-photon detector, a second lens, and a first laser. The laser emitted by the first laser passes through the second lens and the first dichroic mirror and is incident on the digital micromirror device. After being modulated by the digital micromirror device, it enters the sample detection optical path. The signal light returning from the sample detection optical path is reflected by the digital micromirror device, reflected again by the first dichroic mirror, and reaches the first lens. The light passing through the first lens is converged at a point and compressed and collected by the single-photon detector.

4. The compression confocal imaging method based on digital micromirror devices as described in claim 1, characterized in that, The confocal microscopy imaging module includes a second dichroic mirror, a third lens, a two-dimensional image acquisition device, a fourth lens, and a second laser. The laser emitted by the second laser passes through the fourth lens and the second dichroic mirror and is incident on the digital micromirror device. After being modulated by the digital micromirror device, it enters the sample detection optical path. The signal light returning from the sample detection optical path is reflected on the digital micromirror device, reflected again by the second dichroic mirror, and reaches the third lens. The light passing through the third lens is then relayed and imaged onto the two-dimensional image acquisition device.

5. The compression confocal imaging method based on a digital micromirror device as described in any one of claims 1 to 4, characterized in that, When acquiring a confocal image of a sample, the digital micromirror device completes all modulation mode changes within one exposure time of the camera, achieving full coverage detection of all points in the sample detection area and obtaining a complete confocal two-dimensional image; or, each modulation mode change of the digital micromirror device triggers an exposure, and multiple exposures are performed until all points in the sample detection area are covered, and then the images acquired in each exposure are stacked to obtain a complete confocal image.

6. The compression confocal imaging method according to any one of claims 1 to 4, characterized in that, In step S1, n is 10 or more.

7. The compressed confocal imaging method as described in claim 6, characterized in that, In step S1, .

8. The compression confocal imaging method according to any one of claims 1 to 4, characterized in that, In step S3, reconstructing the two-dimensional confocal image includes: Based on the principle of compressed sensing, the following algorithm is used to reconstruct a two-dimensional confocal image signal x from a compressed one-dimensional intensity signal y: （1） Where A is the measurement matrix, It is noise, and the following soft threshold iterative algorithm is used to optimize the solution: （2） in, For the convolutional neural network being trained, k represents the number of iterations. This represents the signal value in the k-th iteration. This is the value of the kth intermediate iteration. This is the regularization parameter.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images