A high-precision timing synchronous dual-mode multi-focal structured light microscopic imaging system and method

Abstract

The application discloses a high-precision timing synchronous bimodal multi-focus structured light microscopic imaging system and method, and belongs to the technical field of optical microscopic imaging. The system comprises an excitation light source module, a light field regulation module, a scanning deflection module, a light splitting module, a first area array detector, a second area array detector and a control module. The excitation light source module outputs excitation light; the light field regulation module modulates the excitation light into a multi-focus array light beam; the scanning deflection module controls the multi-focus array light beam to perform two-dimensional step scanning on a sample surface; single-photon fluorescence and two-photon fluorescence of the sample are excited; the control module synchronously drives two-dimensional deflection of the scanning deflection module and synchronously controls exposure of the two area array detectors, so that an effective exposure window falls within a stable residence period of single step of the scanning deflection module. The application completely eliminates invalid empty sampling and timing misplacement caused by software asynchrony, and is suitable for high-resolution, high-contrast multi-modal super-resolution imaging of deep biological tissues.

Description

Technical Field

[0001] This invention relates to the field of optical microscopy imaging technology, and more specifically to a high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging system and method, which can simultaneously acquire near-infrared single-photon and visible two-photon data to achieve complementary multi-dimensional information of deep tissues. Background Technology

[0002] Optical microscopy, with its advantages of high sensitivity, high spatial resolution, and non-invasiveness, has been widely used in biomedical research. However, traditional optical microscopy is limited by the optical diffraction limit, with a lateral resolution typically around 250–300 nm, which is insufficient to meet the high-resolution observation requirements of cells and subcellular microstructures. To overcome the diffraction limit, image scanning microscopy (ISM) uses an area array detector to replace the single-point detector in traditional laser scanning confocal microscopy. Each pixel of the area array detector functions as a point detector with a built-in pinhole, thereby improving light signal utilization and achieving super-resolution imaging. However, ISM typically uses single-point excitation and point-by-point scanning, resulting in a relatively slow imaging speed. Multifocal structured illumination microscopy (MSIM) is an optical super-resolution imaging technique that combines multifocal parallel excitation, structured light modulation, and computational reconstruction. Compared with traditional ISM technology, MSIM improves imaging throughput through multifocal parallel illumination and enhances spatial resolution and suppresses defocus background by utilizing structured light modulation and reconstruction algorithms. Therefore, it has important application value in rapid fluorescence imaging of cell and tissue samples.

[0003] Current micro-simulation imaging (MSIM) techniques are primarily based on visible light single-photon excitation. These systems typically use visible light continuous-wave lasers or LEDs as the light source, resulting in relatively simple optical path structures, a wide selection of fluorescent probes, and high detection efficiency, making them suitable for high-resolution imaging of thin-layer cells, tissue sections, or shallow biological samples. However, visible light single-photon excitation exhibits strong scattering in biological tissues, limiting imaging depth; furthermore, visible light excitation easily induces strong autofluorescence background and phototoxicity. When samples have a certain thickness or require long-term imaging, conventional visible light MSIM often struggles to balance resolution, imaging depth, and biocompatibility.

[0004] To address these issues, one approach is to extend the excitation wavelength of single-photon MSIM to the near-infrared band. Near-infrared single-photon MSIM utilizes near-infrared continuous-wave lasers or narrow-band near-infrared light sources for excitation. Compared to visible light excitation, it exhibits lower tissue scattering and autofluorescence background, and can be combined with near-infrared fluorescent probes to improve the signal-to-noise ratio and penetration capability of thick sample imaging. Therefore, near-infrared single-photon MSIM can, to some extent, compensate for the shortcomings of traditional visible-light MSIM in tissue imaging. However, near-infrared single-photon excitation is still a linear excitation process, and its fluorescence signal can be generated throughout the illumination path, limiting defocus background and axial optical sectioning capabilities. In deeper tissues or strongly scattering samples, single-photon excitation still struggles to completely avoid background accumulation and spatial resolution degradation.

[0005] Another approach is to develop two-photon MSIM. Two-photon excitation typically uses near-infrared femtosecond pulsed lasers, where the fluorescence signal is mainly generated near the focal point, offering inherent optical slicing capabilities, lower defocused excitation background, and better imaging capabilities for deep tissues. Therefore, two-photon MSIM has significant advantages in imaging thick tissues, biological samples, and three-dimensional structures. However, two-photon MSIM also suffers from system complexity, high cost, limited excitation efficiency, and a trade-off between imaging speed and light dose. Because the two-photon signal exhibits a non-linear relationship with the excitation light intensity, its structured light modulation characteristics, focal array distribution, reconstruction model, and overall performance differ significantly from those of single-photon MSIM.

[0006] From an application perspective, near-infrared single-photon MSIM and two-photon MSIM are not simply substitutes. Near-infrared single-photon MSIM is more suitable for high-throughput, low-cost, and low-optical-damage near-infrared fluorescence imaging; two-photon MSIM is more suitable for imaging deep, thick samples and three-dimensional optical sections. If rapid registration and acquisition of the same sample and field of view using both imaging modes can be achieved on the same microscopic imaging platform, near-infrared single-photon fluorescence information and two-photon deep optical section information can be obtained simultaneously, which is of great significance in the observation of multi-labeled samples, thick tissue samples, and dynamic processes. For example, near-infrared single-photon MSIM can be used to quickly acquire near-infrared label signals for large fields of view or targets, while two-photon MSIM can be used to acquire high axial resolution structural information or deep signals in the same region. Combining the two is beneficial for multimodal comparative analysis under the same experimental conditions, reducing errors caused by sample movement, repeated positioning, and temporal drift.

[0007] Although existing research has attempted to combine conventional single-photon and two-photon imaging, such as adding a near-infrared single-photon detection module to a traditional two-photon point scanning system, or using a beam splitter combined with visible light and infrared cameras to achieve wide-field dual-channel acquisition in the field of macroscopic imaging, integrating near-infrared single-photon MSIM and two-photon MSIM into the same system still faces many technical challenges.

[0008] First, the two modes require different light sources: near-infrared single-photon MSIM typically uses continuous or quasi-continuous near-infrared lasers, while two-photon MSIM requires femtosecond pulsed lasers. These differ in power, pulse width, beam quality, and modulation methods. Second, the excitation mechanisms differ: single-photon fluorescence intensity is approximately linearly related to excitation light intensity, while two-photon fluorescence intensity is correlated with the square of the excitation light intensity. This results in differences in the effective modulation depth, spatial frequency response, and reconstruction parameters for the same focal array or structured light pattern in the two modes. Third, different excitation wavelengths and emission channels can lead to problems such as chromatic aberration, focal plane shift, optical path registration, and signal crosstalk, requiring unified correction in optical path design, filtering and beam splitting, detection synchronization, and image reconstruction. Furthermore, the optimal exposure time, excitation power, focal spacing, and acquisition strategies for the two modes are not entirely consistent. Simply building two systems side-by-side not only increases size and cost but also makes it difficult to guarantee spatial registration and temporal consistency within the same sample area.

[0009] Therefore, current technologies still lack an imaging method and system that can integrate near-infrared single-photon MSIM and two-photon MSIM within a single microscopic imaging platform. This system should retain the advantages of single-photon MSIM (fast, high-throughput, and low optical damage) while leveraging the deep imaging and optical slicing capabilities of two-photon MSIM, and enabling rapid switching or registration acquisition of the same sample within the same field of view. To address these issues, it is necessary to propose a novel multimodal MSIM optical path design, excitation modulation, and reconstruction method to achieve effective fusion of near-infrared single-photon MSIM and two-photon MSIM, thereby meeting the demands of high resolution, high signal-to-noise ratio, and multimodal information acquisition for thick samples and multi-label dynamic imaging.

[0010] In summary, how to construct a single- and dual-photon dual-modal MSIM system with a shared core optical path, and how to achieve precise timing binding between the multifocal scanning module and multiple area array detectors through hardware-level synchronous control, thereby reducing spatial mismatch caused by optical path switching or independent stitching, while avoiding invalid acquisition and motion blur caused by asynchronous software triggering, are the technical problems that urgently need to be solved in dual-modal super-resolution microscopy of thick biological tissues. Summary of the Invention

[0011] In view of the above problems, the present invention is proposed to provide a high-precision time-synchronized dual-modal multifocal structured light microscopy imaging system and method that overcomes or at least partially solves the above problems.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] In a first aspect, embodiments of the present invention provide a high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging system, comprising: Excitation light source module, used to output excitation light; The light field modulation module is used to modulate the excitation light into a multi-focus array beam; The scanning deflection module is used to control the multi-focus array beam to perform two-dimensional step scanning on the sample surface; The spectrometer module is located between the scanning deflection module and the objective lens. It is used to guide the excitation light to the objective lens and divide the fluorescence emitted by the sample into a first band fluorescence and a second band fluorescence. A first array detector is used to receive fluorescence in the first band and output a first image signal; A second array detector is used to receive the fluorescence in the second band and output a second image signal; and The control module includes a data acquisition card, which synchronously outputs two analog signals to drive the two-dimensional deflection of the scanning deflection module, and synchronously outputs two trigger signals to control the exposure of the first array detector and the second array detector. The control module is configured such that, at the underlying hardware level, the trigger release times of the first array detector and the second array detector are forcibly bound to the actual physical output time of the analog signal of the scanning deflection module, so that the effective exposure windows of the first array detector and the second array detector strictly fall within the stable dwell period of a single step of the scanning deflection module.

[0014] In one embodiment, the excitation source module is a femtosecond pulsed laser with an output wavelength in the near-infrared band, used to simultaneously excite single-photon fluorescence and two-photon fluorescence of the sample.

[0015] In one embodiment, the control module uses the data flow state control line in the LabVIEW programming environment to time-bind the start-up execution of the trigger task nodes of the first array detector and the second array detector with the analog signal output node of the scanning deflection module.

[0016] In one embodiment, the data stream state control line is an error line. Through the series connection of the error lines, the triggering tasks of the first array detector and the second array detector can only be executed after the analog signal of the scanning deflection module is actually output.

[0017] In one embodiment, the light field modulation module includes a spatial light modulator or a digital micromirror device for loading phase patterns or amplitude patterns to generate a multifocal array.

[0018] In one embodiment, the system further includes an image processing module, which is used to perform spatial registration of the first image signal and the second image signal. The spatial registration adopts a reflection-free similarity transformation model, which extracts the sub-pixel-level centroid coordinates of the fluorescence dot matrix, solves the transformation matrix, and uses bicubic interpolation for resampling.

[0019] In one embodiment, the image processing module is further configured to perform background suppression and super-resolution reconstruction on the registered image data. The background suppression employs optical phase-locked detection processing and / or Dark sectioning algorithm, and the super-resolution reconstruction employs pixel relocation and deconvolution techniques.

[0020] In one embodiment, the first array detector is a visible light array detector used to receive visible light fluorescence excited by two photons; the second array detector is a near-infrared array detector used to receive near-infrared fluorescence excited by single photons.

[0021] In one embodiment, the system further includes a Z-axis electric displacement stage, and the control module performs time-synchronous control on the Z-axis electric displacement stage to achieve three-dimensional dual-modal multi-focus super-resolution synchronous scanning imaging.

[0022] In a second aspect, embodiments of the present invention provide a high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging method, applied to the system described in any one of the embodiments of the first aspect, comprising the following steps: Step S1: The excitation light is modulated into a multi-focus array beam by the light field modulation module, and the multi-focus array beam is guided to the sample surface by the scanning deflection module. Step S2: The data acquisition card synchronously outputs multiple control signals, including two analog signals that drive the scanning deflection module to perform two-dimensional step scanning, and two trigger signals that control the first array detector and the second array detector to be exposed synchronously. Step S3: At the underlying hardware level, the trigger release times of the first and second array detectors are forcibly bound to the actual physical output time of the analog signal of the scanning deflection module, so that the effective exposure window of the array detector strictly falls within the stable dwell period of a single step of the scanning deflection module. Step S4: Collect and save the first image signal and the second image signal output by the first area array detector and the second area array detector respectively, to obtain the first mode dot matrix image sequence and the second mode dot matrix image sequence with time synchronization.

[0023] In one embodiment, it also includes: Step S5: Perform high-precision spatial registration on the first modal dot matrix image sequence and the second modal dot matrix image sequence; and perform super-resolution reconstruction of the registered data using the background suppression module.

[0024] As can be seen from the above technical solutions, compared with the prior art, the present invention has the following technical advantages: 1. The system can simultaneously acquire single-photon MSIM and two-photon MSIM data without mechanical optical path switching. Since the single-photon excitation optical path and the two-photon excitation optical path share the scanning deflection module, relay optics system, and objective lens illumination path, the system does not require an additional independent wide-field positioning light source or mechanical switching module. This reduces optical path complexity, lowers system debugging difficulty, and improves optical axis consistency and acquisition stability during dual-modal imaging.

[0025] 2. Regarding sample localization and scanning addressing, this invention utilizes the near-wide-field imaging characteristics of single-photon array illumination in thick tissue imaging to rapidly locate regions of interest within the sample, and combines this with a two-dimensional scanning deflection module to achieve precise scanning of the target region. This approach avoids the repetitive operation of first observing the wide-field and then switching to the scanning imaging mode in traditional procedures, thus shortening the experimental process and improving the efficiency of target region localization and data acquisition. Simultaneously, the system can adjust the scanning range, scanning step size, or number of multifocal points according to the sample area size and imaging requirements, thereby achieving a flexible balance between imaging field of view, acquisition speed, and spatial sampling density.

[0026] 3. Regarding synchronization control, this invention employs a low-level data stream state binding mechanism to forcibly associate the scanning position, stable dwell state, area array detector exposure trigger signal, and image frame acquisition state of the scanning deflection module. Only after the scanning deflection module reaches the designated scanning position and enters a stable dwell period are the two area array detectors allowed to perform effective exposure; after the exposure and data readout of the two area array detectors are completed, the system triggers the next scanning position. This mechanism effectively avoids premature exposure, delayed exposure, and invalid data acquisition problems caused by asynchronous software control, command delays, or inconsistent buffer states of the area array detectors, improving the one-to-one correspondence between the acquired frames and scanning coordinates, and ensuring the temporal reliability and spatial validity of the original data.

[0027] 4. Regarding the correspondence of dual-modal data, this invention addresses the potential physical differences between the two detection channels in terms of magnification, field of view position, rotation angle, and aberrations by introducing spatial registration processing. Coordinate mapping and field of view correction are performed on the single-photon MSIM dataset and the two-photon MSIM dataset. This processing reduces the field of view mismatch between the two detectors, ensuring that the structural information obtained from the two modes corresponds spatially, providing a reliable foundation for subsequent dual-modal fusion analysis and structural comparison of the same region.

[0028] 5. Regarding image quality improvement, this invention further inputs the registered single- and two-photon MSIM data into background suppression and super-resolution reconstruction algorithms. By suppressing defocused background, scattered background, and signal interference from non-target areas in thick tissue imaging, the image contrast and edge sharpness of fine structures can be improved. Through MSIM reconstruction, the resolution limitations of conventional wide-field or direct scanning imaging can be overcome to a certain extent, improving the resolvability of cells, fibers, blood vessels, or subcellular structures in deep biological samples. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0030] Figure 1 This is a structural diagram of a high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging system provided in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram illustrating the timing synchronization principle of the control module of the present invention.

[0032] Figure 3 This is a flowchart of a high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging method provided in an embodiment of the present invention.

[0033] Figure 4 This is an example diagram of the control program interface of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This invention proposes a high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging system and method, primarily applied in the fields of biomedical microscopy and cutting-edge research in life sciences. It is particularly suitable for high-fidelity, multi-dimensional in-situ optical observation of complex biological tissues or samples with a certain thickness. In specific applications, this invention can be widely used for super-resolution imaging of deep, complex tissues. For biological samples with strong scattering and strong background, such as solid tumor models and thick organ sections, it can overcome the depth and resolution limitations of single imaging modalities, acquiring microscopic images with both great depth, high contrast, and high resolution. Simultaneously, thanks to the shared optical path eliminating mechanical switching vibrations (such as switching the optical path by moving and flipping the lens) and the forced binding of the underlying state to achieve zero temporal misalignment, this invention can continuously capture rapidly changing dynamic physiological processes within organisms, completely avoiding motion blur and data redundancy, and achieving long-term, artifact-free in-situ tracking. Furthermore, this invention can also be used for high-fidelity fusion analysis of multi-target molecular and structural information, enabling the simultaneous acquisition of highly sensitive probe information of near-infrared single photons and high-resolution structural information of two photons within the same scanning coordinates, the same exposure sequence, and the same target field of view. This provides absolutely reliable multidimensional data support for disease pathological mechanism research, cross-analysis of deep tissue morphology, and truth verification of super-resolution reconstruction algorithms.

[0036] This invention discloses a high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging system, comprising: an excitation source module, a light field modulation module, a scanning deflection module, a beam splitting module, a first array detector, a second array detector, and a control module.

[0037] The excitation light source module is used to output excitation light. In a preferred embodiment, such as... Figure 1 As shown, the excitation source module is a femtosecond pulsed laser 1. The laser emitted by the laser first passes through the excitation source module, which consists of a first half-wave plate 2, a polarizing beam splitter prism 3, and a second half-wave plate 4, to precisely adjust the power of the incident light. Its output wavelength is in the near-infrared band, for example, 920 nm, and is used to simultaneously excite single-photon fluorescence and two-photon fluorescence of the sample. Because femtosecond pulsed lasers possess both high peak power and wide spectral characteristics, under suitable power conditions, they can excite both two-photon fluorescence (a nonlinear process) and single-photon fluorescence (a linear process), thus enabling a single laser to simultaneously drive two imaging modes. Furthermore, the configuration of the excitation source is not limited to using a single laser; it can also be equivalent to using multiple lasers of different wavelengths combined as the excitation source of the system.

[0038] The light field modulation module is used to modulate the excitation light into a multifocal array beam. For example... Figure 1As shown, the optical field manipulation module may include optical field manipulation devices such as a spatial light modulator (SLM) or a digital micromirror device (DMD). The laser beam is spatially modulated by loading a corresponding phase pattern onto the spatial light modulator 8. After passing through the third lens 9 and the second reflecting mirror 10, the emitted light undergoes spatial filtering via the aperture 11, removing unwanted diffraction order light and leaving only the first-order diffraction light required for subsequent optical paths, thereby generating a high-quality multifocal excitation array.

[0039] like Figure 1 As shown, the scanning deflection module 13 is used to control the multifocal array beam to perform a two-dimensional step scan on the sample surface. This scanning deflection module 13 can be a two-dimensional galvanometer scanning module or other devices with similar beam scanning deflection functions. By controlling the X-axis and Y-axis deflection angles of the scanning deflection module, precise translation of the multifocal array on the sample surface can be achieved, thereby enabling the excitation lattice to cover the entire desired region of interest with high density and without blind spots.

[0040] The spectrometer module is positioned between the scanning deflection module 13 and the tube lens 16. It guides the excitation light to the objective lens and separates the fluorescence emitted by the sample into a first-band fluorescence and a second-band fluorescence. For example... Figure 1 As shown, in a preferred embodiment, the spectrophotometer module includes a first dichroic mirror 14 and a second dichroic mirror 15. The deflected excitation light passes sequentially through the first dichroic mirror 14 and the second dichroic mirror 15, and is focused onto the sample surface via the tube mirror 16 and the microscope objective 17. The fluorescence signal emitted by the sample is collected by the same microscope objective 17 and returns along the original path. The near-infrared single-photon fluorescence is reflected by the second dichroic mirror 15 and reaches the near-infrared surface array detector 19; while the visible light two-photon fluorescence passes through the second dichroic mirror 15 and is reflected by the first dichroic mirror 14, reaching the visible light surface array detector 18.

[0041] The first array detector, the visible light array detector 18, is used to receive the first band of fluorescence (i.e., visible light two-photon fluorescence) and output the first image signal. The second array detector, the near-infrared array detector 19, is used to receive the second band of fluorescence (i.e., near-infrared single-photon fluorescence) and output the second image signal. Both array detectors can be scientific complementary metal-oxide-semiconductor (sCMOS), electron-multiplying charge-coupled device (EMCCD), indium gallium arsenide (InGaAs) detectors, or other array detectors with high sensitivity.

[0042] The control module includes a data acquisition card (DAQ card), a computer terminal (PC), and LabVIEW control and data processing programs running on it. The PC communicates with the DAQ card and the visible light array detector 18 and the near-infrared array detector 19, respectively; the DAQ card, in turn, is physically connected to the scanning deflection module and the visible light array detector 18 and the near-infrared array detector 19. Figure 2 As shown, the system outputs two analog output (AO) signals through the DAQ card to drive and control the X-axis and Y-axis deflection of the two-dimensional scanning deflection module. Simultaneously, based on the trigger interface characteristics of different area array detectors, the DAQ card synchronously outputs one AO ​​signal to trigger and control the exposure of the visible light area array detector, and outputs one digital pulse signal (i.e., transistor-transistor logic, TTL) to trigger and control the exposure of the near-infrared area array detector. This enables strictly spatiotemporal synchronous acquisition of multi-channel heterogeneous signals, obtaining spatiotemporally completely corresponding MSIM fluorescence array images of single photons and two photons.

[0043] like Figure 2 As shown, the data acquisition card synchronously outputs two analog signals to drive the two-dimensional deflection of the scanning deflection module, and synchronously outputs two trigger signals to control the exposure of the visible light array detector 18 and the near-infrared array detector 19. In a preferred embodiment, the data acquisition card synchronously outputs three analog signals (AO) and one digital pulse signal (TTL), wherein two AO signals control the X-axis and Y-axis deflection of the two-dimensional scanning deflection module respectively, one AO ​​signal triggers the exposure of the visible light array detector 18, and one TTL signal triggers the exposure of the near-infrared array detector 19.

[0044] The control module of the present invention is configured such that, at the underlying hardware level, the trigger release times of the two area array detectors are forcibly bound to the actual physical output time of the analog signal of the scanning deflection module, so that the effective exposure windows of the two area array detectors strictly fall within the stable dwell period of a single step of the scanning deflection module.

[0045] Specifically, the step deflection of the 2D scanning deflection module 13 is controlled by the AO signals output from the DAQ card. One AO ​​signal controls the deflection in the X direction, and the other AO signal controls the deflection in the Y direction. Each physical position of the multifocal array on the sample surface corresponds to a pair of corresponding X-axis and Y-axis deflection voltage values. Therefore, by outputting the corresponding voltage step sequence in a timely manner through the control program, the 2D scanning deflection module 13 can quickly and accurately deflect the multifocal array to different pixel positions on the sample, realizing 2D step scanning.

[0046] To achieve super-resolution using the MSIM principle, the system needs to acquire several fluorescence array images at different locations on the sample. In this method, the single-photon and two-photon fluorescence signals generated by the scanning deflection module 13 are simultaneously acquired by two high-sensitivity area array detectors. Since the multifocal array can only cover a portion of the field of view, the scanning deflection module 13 needs to perform high-frequency scanning to fill the gaps between the focal points and achieve complete sampling of the field of view. When the scanning deflection module 13 controls the multifocal array to move to the target coordinate, the two area array detectors are triggered to begin exposure and acquire sample data at that coordinate. After the exposure, the scanning deflection module 13 is controlled to move to the next adjacent target coordinate. Upon reaching the target coordinate, the area array detectors are triggered to perform the next exposure and acquire sample data at this coordinate. This process is repeated to achieve overall translation of the multifocal array on the sample surface, thereby enabling the excitation array to cover the entire desired region of interest with high density and without blind spots.

[0047] To ensure that the effective exposure time of the area array detector falls entirely within the single step cycle of the scanning deflection module 13 (i.e., the effective exposure time of the area array detector is less than the single mechanical deflection cycle), this invention must address the inherent timing misalignment problem under conventional asynchronous software control. In conventional non-strictly bound control logic, when the control program sends commands to the detector and deflection module respectively, due to differences in the underlying execution sequence and signal conversion response, the system often prioritizes releasing the TTL pulse signal that triggers the area array detector, while the AO signal sent to the scanning deflection module exhibits execution lag. This results in the area array detector immediately starting actual exposure acquisition after receiving a high-level trigger, but during the initial few frames of exposure time, the scanning deflection module 13 has not yet received the AO signal and started deflecting (i.e., it remains stationary at the origin). This asynchronous response of software commands directly leads to a large amount of invalid "empty acquisition frames" data redundancy, and causes a timing misalignment between the subsequent dot matrix mechanical stepping and the effective exposure window of the area array detector.

[0048] To completely solve this underlying control challenge, this invention proposes a hardware-software collaborative synchronization principle based on mandatory data flow state binding. Specifically, in the LabVIEW program, the TTL trigger task controlling the exposure of the area array detector and the AO output task controlling the deflection of the scanning deflection module 13 are configured independently. In the initial stage of operation, the system pre-initializes the parameters of the area array detector trigger task and places it in a "ready" suspended state. Subsequently, using data flow state control lines (such as error wires) in the underlying programming environment, a strict timing dependency binding is established between the "start execution" node of the area array detector trigger task and the "physical signal output" node of the 2D scanning deflection module AO task.

[0049] Then, a master-slave triggering architecture based on the data acquisition card (DAQ) hardware is adopted. The corresponding data stream channel is turned on only when the system bottom layer confirms that the AO signal has been actually output to the scan deflection module 13, thereby driving the suspended TTL trigger signal to be released instantaneously with zero hysteresis. This underlying strong binding control mechanism effectively suppresses the premature response of the area array detector, so that it is only activated and triggered after the AO signal of the scan deflection module 13 has actually taken effect. This completely eliminates the defect of premature exposure of the area array detector caused by asynchronous software instructions, and ensures that every effective exposure of the area array detector falls precisely within the mechanical deflection cycle of the scan deflection module 13.

[0050] After acquiring temporally synchronized single-photon and two-photon fluorescence array image sequences, eliminating the spatial mismatch between independent detection channels is another key aspect of the post-processing method in this invention. Due to the physical difference in magnification between the detection optical paths of two-photon and single-photon detectors, the equivalent pixel sizes corresponding to the two area array detectors on the object surface are not entirely the same. Furthermore, the target surfaces of the two detectors inevitably exhibit slight relative deflection angles during actual physical assembly. These physical factors prevent the fields of view captured by the two area array detectors from achieving 100% natural overlap in physical pixel coordinates. Therefore, this invention also introduces a high-precision image spatial registration mechanism. To achieve this mechanism, the system constructs a spatial geometric transformation model based on feature extraction and sub-pixel centroid adsorption. Considering that the spatial deformation between the dual-channel imaging optical paths mainly manifests as proportional scaling and mechanical rotation and translation of the area array detectors, without shear deformation, the system employs a non-reflection similarity transformation model for registration.

[0051] Let the pixel coordinates in the single-photon image to be registered be (x, y), and the coordinates mapped to the two-photon reference image after registration be (x, y). , The mathematical calculation process of its transformation matrix M is defined as follows:

[0052] In the formula, S represents the proportional optical magnification scaling factor between the dual-channel array detectors; θ represents the relative mechanical rotation angle when the target surfaces of the two array detectors are installed. and These represent the lateral and longitudinal translational offsets of the centers of the two target surfaces in physical space, respectively.

[0053] During the calculation of the transformation matrix parameters, considering that the actual acquired raw data is a continuous multi-focus dot matrix data stack, the system first selects a corresponding dot matrix image frame as the feature registration frame. In this single-frame dot matrix image, the system uses a feature recognition algorithm to extract local pixel windows of the fluorescent dot matrix spots in the image. In this single-frame dot matrix image, the system uses an existing local peak-finding algorithm to extract local pixel windows of the bright fluorescent dot matrix spots in the image. To eliminate the defocus noise interference of the single-photon channel and improve the coordinate positioning accuracy, the system subtracts the minimum gray value from the extracted local pixel blocks to obtain a pure signal intensity matrix I(x,y), and then uses the gray-level centroid method to calculate the sub-pixel level centroid coordinates of the dot matrix spot. , ):

[0054] Through the conventional feature point extraction steps on a single-frame dot matrix image described above, the system obtains at least three pairs of high-precision conjugate control points and uses the least squares method to solve the system of equations, thereby calculating the optimal spatial transformation matrix M.

[0055] After acquiring the aforementioned high-precision spatial transformation matrix, the system uses the overall dot matrix data stack of the dual-photon channel as the absolute spatial reference to perform global mapping reconstruction on the 16-bit single-photon dot matrix data to be registered. Considering that the target coordinates after matrix mapping are usually floating-point numbers containing decimals, using conventional forced rounding operations would severely damage the high-frequency modulation information in multifocal structured light illumination imaging. Therefore, the system uses a bicubic interpolation algorithm to resample and calculate the grayscale values ​​of the target pixels to smoothly approximate and preserve the energy distribution characteristics of the original dot matrix. After completing the coordinate remapping and grayscale interpolation, the system physically clips any pixels exceeding the spatial reference boundary and fills the blank areas generated during registration with zeros, ultimately outputting a dual-modal dot matrix dataset with a field of view scale perfectly aligned with the spatial grid, providing a high-fidelity data source for subsequent image processing.

[0056] After completing spatial registration and obtaining a dataset with a completely consistent field of view, this invention further processes the registered dual-modal data to address the defocus background signal that is easily generated by thick biological tissue in single-photon MSIM imaging. By combining super-resolution reconstruction and background suppression algorithms, the defocused and blurred background caused by deep biological tissue is objectively removed. The super-resolution reconstruction algorithm uses pixel reassignment (PR) and deconvolution techniques; the background suppression algorithm combines optical lock-in detection (OLID) processing and dark sectioning algorithm, ultimately obtaining high-resolution, high-contrast dual-modal super-resolution microscopic images.

[0057] Furthermore, as a natural extension of the system of the present invention in the spatial dimension, additional three-dimensional control devices can be introduced into the optical path, or a timing synchronization control mechanism for the Z-axis electric displacement stage can be further added to the LabVIEW synchronous control program, thereby expanding the two-dimensional region of interest to three dimensions and realizing three-dimensional dual-modal multi-focus super-resolution synchronous scanning imaging.

[0058] In summary, this invention has two main advantages. First, it utilizes a state-forced binding mechanism in the underlying data stream to tightly link the physical deflection output of the two-dimensional scanning deflection module with the ready-state triggering of the two area array detectors, fundamentally avoiding the premature exposure disconnect of the area array detectors caused by software and hardware asynchrony. Second, it employs a rigorous data processing flow, which first overcomes the physical difference between the magnification of the detector optical path and the target angle to achieve strict registration of the single and dual photon array fields of view, and then combines the image reconstruction and background suppression algorithms in the backend to ultimately achieve high-precision, artifact-free dual-modal multifocal super-resolution microscopic imaging.

[0059] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0060] Reference Figure 1 The dual-mode system all-optical-path architecture of the present invention is as follows: The laser is a femtosecond pulsed laser 1, with an output wavelength in the near-infrared band (e.g., 920 nm). The emitted femtosecond pulsed laser is sequentially passed through a first half-wave plate 2 and a polarizing beam splitter prism 3 to adjust its power, then through a second half-wave plate 4 to adjust its polarization state, and finally through lenses 5 and 6 to expand and collimate the beam. The expanded laser beam is projected onto a spatial light modulator 8 via a first reflecting mirror 7, onto which a corresponding multifocal phase pattern is loaded, for example, generating a 7×7 or 9×9 multifocal array. The modulated beam is focused by a third lens 9 and reflected by a second reflecting mirror 10 into an aperture 11, where zero-order and higher-order stray light are filtered out, thereby generating a pure multifocal excitation array. This array beam enters a scanning deflection module 13 via a third reflecting mirror 12 to achieve two-dimensional spatial deflection scanning. The deflected excitation light passes sequentially through the first dichroic mirror 14 and the second dichroic mirror 15, is collimated by the tube mirror 16, and is finally focused onto the sample by the microscope objective 17, exciting fluorescence and achieving simultaneous two-photon and single-photon excitation of biological tissue. The dual-mode fluorescence emitted by the sample is collected by the same microscope objective 17 and returns along the same path. Specifically, the near-infrared single-photon fluorescence is reflected by the second dichroic mirror 15 and projected onto the near-infrared surface array detector 19; the visible light two-photon fluorescence passes through the second dichroic mirror 15 and is reflected by the first dichroic mirror 14, and projected onto the visible light surface array detector 18. The entire excitation and detection of single and two-photon modes share the same core excitation optical path, requiring no mechanical plugging or component switching.

[0061] To address the dual-modal physical excitation differences (i.e., the high efficiency and strong background of single-photon versus the low efficiency of two-photon) caused by sharing a single femtosecond laser and multi-focus generation device, this invention employs parameter equalization and pre-calibration strategies in the excitation source module and the optical field control module. In the excitation source module, the power mismatch caused by differences in physical excitation efficiency is addressed by pre-calibrating the excitation power of the target sample before formal acquisition. By gradient-adjusting the laser output power, an optimal power range is found that balances the contrast of single-photon array modulation and the signal-to-noise ratio of two-photon images, serving as a benchmark for synchronous concurrent acquisition and ensuring the validity of dual-modal image data at the energy source. In the optical field control module, to overcome the spatial crosstalk between adjacent multi-focus points caused by the strong defocus background of single-photon images, the array pitch of the multi-focus array generation device is also pre-calibrated. Specifically, the system performs imaging tests on a uniform fluorescent solution to intuitively assess the global background interference level, thereby obtaining a compatible and acceptable array spacing range. The calibrated spacing avoids both the failure of the backend MSIM reconstruction algorithm due to excessively small background overlap and the decrease in the overall scanning frame rate of the dual-modal system due to excessively large spacing.

[0062] The PC outputs two AO signals via the DAQ card to control the XY axis deflection of the scanning deflection module 13, and simultaneously outputs one AO ​​signal and one TTL signal to trigger the two area array detectors to start exposure and data acquisition.

[0063] Reference Figure 2 In terms of control logic, the system adopts a mandatory binding mechanism for underlying data flow states. For example... Figure 2 As shown, during the acquisition cycle, the DAQ card synchronously generates four control signals: three analog signals (x-axis signal controlling the x-axis of the two-dimensional scanning deflection module, y-axis signal controlling the y-axis, and sCMOS trigger signal for the visible light area array detector) and one digital pulse signal (TTL, i.e., MC trigger signal for the near-infrared area array detector). Through the cascading of the error wires at the bottom layer of the control program, the trigger task node of the area array detector is forced to belong to the drive node of the scanning deflection module. Combining the timing correspondence of the vertical dashed lines in the figure, it can be seen that the square wave trigger pulses of sCMOS and MC are strictly aligned and released simultaneously only at the physical instant when the x and y drive signals undergo a voltage step transition (i.e., the scanning deflection module executes a step). This mechanism ensures a strict hardware timing lock between the area array detector trigger pulse and the stepping action of the scanning deflection module, thereby ensuring that the effective exposure windows of the two area array detectors precisely fall within each step dwell period of the mechanical deflection, completely avoiding premature exposure and invalid data acquisition caused by asynchronous software instructions.

[0064] The high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging system provided by this invention operates on the following principle: First, a multifocal array is loaded onto the spatial light modulator and the excitation source is turned on. The focus is adjusted to find the single-photon and dual-photon confocal surfaces. Taking advantage of the wide-field effect of single-photon real-time imaging, the XY-axis displacement stage is adjusted to select the region of interest. The physical offset voltage of the XY-axis of the two-dimensional scanning deflection module is recorded at this time as the spatial reference for subsequent dot matrix scanning.

[0065] Then, the two array detectors were adjusted to external trigger mode to begin acquiring dual-channel MSIM data. The underlying software controlled the data acquisition card to synchronously output three AO signals and one TTL signal: two AO signals drove the XY-axis addressing deflection of the scanning deflection module, and the remaining AO and TTL signals triggered synchronous exposure of the dual detectors. After acquisition, single and dual photon array image sequences with strict time-series correspondences were saved.

[0066] Finally, offline processing was performed using a data processing program written in MATLAB. This included: performing high-precision spatial registration on the acquired single and two-photon dot matrix sequences to eliminate field-of-view mismatch caused by the hardware optical path; then inputting the registered dataset into the background suppression module to remove out-of-focus blurred backgrounds; and finally reconstructing and superimposing the dot matrices to obtain a dual-modal multifocal super-resolution image with a completely consistent field of view.

[0067] Based on the same inventive concept, referring to Figure 3 The present invention also provides a high-precision time-synchronized dual-modal multifocal structured light microscopy imaging method, applied to the above-mentioned high-precision time-synchronized dual-modal multifocal structured light microscopy imaging system, comprising the following steps: Step S1: The excitation light is modulated into a multi-focus array beam by the light field modulation module, and the multi-focus array beam is guided to the sample surface by the scanning deflection module. Step S2: The data acquisition card synchronously outputs multiple control signals, including two analog signals that drive the scanning deflection module to perform two-dimensional step scanning, and two trigger signals that control the first array detector and the second array detector to be exposed synchronously. Step S3: At the underlying hardware level, the trigger release times of the first and second array detectors are forcibly bound to the actual physical output time of the analog signal of the scanning deflection module, so that the effective exposure window of the array detector strictly falls within the stable dwell period of a single step of the scanning deflection module. Step S4: Collect and save the first image signal and the second image signal output by the first area array detector and the second area array detector respectively, to obtain the first mode dot matrix image sequence and the second mode dot matrix image sequence with time synchronization.

[0068] Step S5: Perform high-precision spatial registration on the first modal dot matrix image sequence and the second modal dot matrix image sequence; and perform super-resolution reconstruction of the registered data using the background suppression module.

[0069] Specifically, firstly, a phase pattern is loaded onto the spatial light modulator (SLM) to generate a multifocal array, and the excitation source is turned on. By adjusting the focusing mechanism of the microscope system, the confocal plane of single-photon and two-photon fluorescence signals is accurately located within the sample. Subsequently, utilizing the physical characteristics of single-photon imaging—that is, although a multifocal array beam is excited, a global fluorescence image with a similar wide-field illumination effect can still be observed in the real-time imaging mode of the single-photon array detector branch—this wide-field effect is used to locate and select the region of interest to be imaged by adjusting the XY-axis displacement stage. Once the region of interest is selected, the physical offset voltage of the two-dimensional scanning deflection module on the X and Y axes is recorded. This initial offset voltage data will serve as the spatial reference for subsequent lattice-addressed scanning.

[0070] Secondly, the control software of both area array detectors was uniformly adjusted to external trigger mode, placing them in a standby state awaiting external hardware pulses. Subsequently, the system began acquiring dual-channel MSIM data. Based on the determined initial offset reference, the underlying control software controlled the data acquisition card (DAQ) to synchronously output three analog (AO) signals and one digital pulse (TTL) signal. Two of the AO signals served as drive levels, controlling the two-dimensional scanning deflection module to perform precise positioning and dot matrix addressing deflection on the X and Y axes. Simultaneously, to ensure spatiotemporal consistency between mechanical deflection and area array detector exposure, the system released the remaining AO pulse signal and TTL pulse signal while outputting the deflection drive signal. These two trigger signals respectively triggered the visible light and near-infrared dual-area array detectors for synchronous and effective exposure. This "step-synchronous exposure" cycle was executed alternately until the scanning of the entire region of interest was completed. After acquisition, the system saved single-photon and two-photon mode dot matrix fluorescence image sequences with strict temporal correspondence for subsequent data post-processing.

[0071] Finally, the acquired data was processed offline using data processing programs written in MATLAB and other software. This included: addressing the field-of-view mismatch caused by hardware optical paths (such as differences in lens magnification and physical factors like the mechanical mounting angle of the detector target surface) between the two detection channels by performing high-precision spatial registration of the acquired single-photon and two-photon dot matrix fluorescence image sequences. Through the aforementioned reflectionless similarity transformation model, the two datasets were precisely aligned in pixel coordinates, effectively eliminating the field-of-view mismatch caused by the hardware optical paths. Subsequently, the registered, field-of-view consistent dataset was input into the backend background suppression module. Combined with optical lock-in detection (OLID) processing and the Dark sectioning algorithm, defocused and blurred background signals, which are easily generated in deep tissue imaging, were objectively and efficiently removed. Finally, super-resolution reconstruction operations, such as pixel repositioning, were performed on the processed dot matrix dataset, and the reconstructed scan dots were superimposed separately to output a dual-modal multifocal super-resolution microscopic image with a completely consistent field of view, combining high resolution and high contrast.

[0072] Example: Taking the experiment on U87-MG-GFP cell samples that have endocytosed mPBTQ-4FPdots as an example, the experimental steps are as follows: Step 1: Power on the system hardware and software control environment. Turn on the power to the femtosecond pulse laser, spatial light modulator (SLM), data acquisition card (DAQ), two-dimensional scanning deflection module, electric displacement stage, electric turntable for controlling the first half-wave plate 2, computer terminal (PC), and controllers of the two area array detectors in sequence, and run the synchronization control program on the PC.

[0073] Step 2: Turn on the 920nm femtosecond pulsed laser 1, and precisely adjust the power of the incident excitation light by rotating the first half-wave plate 2 using an electric turntable. Adjust the Z-axis electric displacement stage to clearly focus and image the excited fluorescence signal on the detection surfaces of the two area array detectors; then, taking advantage of the near-infrared single-photon detection branch's inherent wide-field characteristics in thick tissue imaging, adjust the XY-axis displacement stage and set the initial offset voltage value of the two-dimensional scanning deflection module to find the region of interest.

[0074] Step 3: Switch both the visible light and near-infrared area array detectors to the "external trigger" mode, which awaits hardware pulse activation, and then enter the following... Figure 4The control program interface is shown. Configure the four physical signal output channels of the DAQ card in the program (i.e., X-axis AO0, Y-axis AO1, area array detector trigger AO2, and area array detector trigger ctr0). Input and set key parameters such as the initial offset values ​​for the X and Y axes, the single-point scanning range, the single-scan step size, and the actual number of images captured by the area array detectors. Run the program; with the help of the underlying data stream binding mechanism, the DAQ card begins to output the above four signals in strict synchronization. At the instant the 2D scanning deflection module performs precise step scanning, the two area array detectors are instantly triggered and synchronously exposed. After scanning, save the stacked data of the single and dual-photon images with a time-corresponding relationship for subsequent processing.

[0075] Step 4: Import the saved original image stacks using a Matlab program, register the two sets of pixel stacks, and crop the field of view to obtain a one-to-one high-fidelity dual-modal dataset. Finally, perform depth processing on the registered pixel stacks using background suppression and reconstruction algorithms to output a high-resolution, high-contrast dual-modal super-resolution micrograph with defocus interference eliminated.

[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0077] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging system, characterized in that, include: The excitation light source module is used to output excitation light; the excitation light source module is a femtosecond pulsed laser with an output wavelength of 920nm in the near-infrared band, used to simultaneously excite the single-photon fluorescence and two-photon fluorescence of the sample. The light field modulation module is used to modulate the excitation light into a multi-focus array beam; The scanning deflection module is used to control the multi-focus array beam to perform two-dimensional step scanning on the sample surface; The spectrometer module is located between the scanning deflection module and the objective lens. It is used to guide the excitation light to the objective lens and divide the fluorescence emitted by the sample into a first band fluorescence and a second band fluorescence. The first array detector is used to receive the fluorescence of the first band and output the first image signal; the first array detector is a visible light array detector used to receive visible light fluorescence excited by two photons. The second array detector is used to receive the second band fluorescence and output the second image signal; the second array detector is a near-infrared array detector used to receive near-infrared fluorescence excited by a single photon; as well as The control module includes a data acquisition card, which synchronously outputs two analog signals to drive the two-dimensional deflection of the scanning deflection module, and synchronously outputs two trigger signals to control the exposure of the first array detector and the second array detector. The control module is configured to: at the underlying hardware level, forcibly bind the trigger release times of the first array detector and the second array detector to the actual physical output time of the analog signal of the scanning deflection module, so that the effective exposure windows of the first array detector and the second array detector strictly fall within the stable dwell period of a single step of the scanning deflection module. The control module uses the data flow state control line in the LabVIEW programming environment to time-series bind the start-up execution of the trigger task nodes of the first array detector and the second array detector with the analog signal output node of the scanning deflection module. The data stream status control line is an error line. Through the series connection of the error lines, the triggering tasks of the first array detector and the second array detector can only be executed after the analog signal of the scanning deflection module is actually output. It also includes an image processing module, which is used to perform spatial registration on the first image signal and the second image signal. The spatial registration adopts a reflection-free similarity transformation model, which extracts the sub-pixel level centroid coordinates of the fluorescence dot matrix, solves the transformation matrix, and uses bicubic interpolation for resampling.

2. The system according to claim 1, characterized in that, The light field modulation module includes a spatial light modulator or a digital micromirror device, used to load phase patterns or amplitude patterns to generate a multifocal array.

3. The system according to claim 1, characterized in that, The image processing module is also used to perform background suppression and super-resolution reconstruction on the registered image data. The background suppression adopts optical phase-locked detection processing and / or Darksectioning algorithm, and the super-resolution reconstruction adopts pixel relocation and deconvolution technology.

4. The system according to claim 1, characterized in that, It also includes a Z-axis electric displacement stage, and the control module performs time-synchronous control on the Z-axis electric displacement stage to achieve three-dimensional dual-modal multi-focus super-resolution synchronous scanning imaging.

5. A high-precision, time-synchronized, dual-modal, multi-focus structured light microscopy imaging method, applied to the system described in any one of claims 1-4, characterized in that, Includes the following steps: Step S1: The excitation light is modulated into a multi-focus array beam by the light field modulation module, and the multi-focus array beam is guided to the sample surface by the scanning deflection module. Step S2: The data acquisition card synchronously outputs multiple control signals, including two analog signals that drive the scanning deflection module to perform two-dimensional step scanning, and two trigger signals that control the first array detector and the second array detector to be exposed synchronously. Step S3: At the underlying hardware level, the trigger release times of the first and second array detectors are forcibly bound to the actual physical output time of the analog signal of the scanning deflection module, so that the effective exposure window of the array detector strictly falls within the stable dwell period of a single step of the scanning deflection module. Step S4: Collect and save the first image signal and the second image signal output by the first area array detector and the second area array detector respectively, to obtain the first mode dot matrix image sequence and the second mode dot matrix image sequence with time synchronization.

Images