A three-dimensional single molecule localization imaging system, method, device, medium, and product

Abstract

The application discloses a three-dimensional single molecule positioning imaging system, method, device, medium and product, relates to the technical field of single molecule positioning microscopic imaging, and the system comprises a light source module, which is used for generating an excitation light beam for irradiating a single molecule sample; an intensity modulation module, which is used for intensity modulation on the excitation light beam to obtain a modulated excitation light beam; a depth modulation module, which is used for axial position modulation on the fluorescent pulse to obtain an encoded fluorescent pulse; an event detection module, which is used for generating a photon event sequence according to the encoded fluorescent pulse; and a positioning imaging module, which is used for solving three-dimensional space coordinates of each single molecule in the single molecule sample according to the photon event sequence, and reconstructing and generating a three-dimensional single molecule image corresponding to the single molecule sample according to the three-dimensional space coordinates of all single molecules in the single molecule sample. The application improves the utilization rate of single molecule positioning information, the fidelity of a three-dimensional image and the imaging flux.

Description

Technical Field

[0001] This application relates to the field of single-molecule localization microscopy imaging technology, and in particular to a three-dimensional single-molecule localization imaging system, method, device, medium and product. Background Technology

[0002] Single-Molecule Localization Microscopy (SMLM) separates spatially overlapping single-molecule emission events along the time axis by controlling the random scintillation of fluorescent molecules and successfully breaks the optical diffraction limit by performing high-precision fitting of the center position of the point spread function (PSF). Traditional SMLM systems heavily rely on frame array detectors (such as scientific complementary metal-oxide-semiconductor (sCMOS) cameras or electron-multiplying charge-coupled devices (EMCCDs). These detectors output a two-dimensional intensity matrix by integrating photons over a fixed exposure time. However, the globally fixed exposure mechanism of frame cameras inevitably leads to extremely high spatiotemporal data redundancy and is limited by the readout frame rate, making it difficult to capture microsecond-level high-frequency dynamic processes in biological samples.

[0003] In recent years, with the maturation of neuromorphic vision sensors (i.e., event cameras), they have begun to be introduced into the field of single-molecule imaging due to their high temporal resolution at the microsecond level, extremely low data redundancy, and extremely high dynamic range. However, event cameras are only sensitive to relative changes in light intensity (time derivative), and do not respond to absolute light intensity. This luminescence characteristic leads to the following problems for existing event-based imaging sensors (SMLMs): 1. Severe waste of photon information during the bright plateau phase: single molecules are "invisible" to event cameras during the golden window of maximum brightness and photon emission; 2. Extremely sparse event clouds cause 3D localization failure: sparse event point clouds generated by passive observation cannot support complex 3D morphological representations and are easily submerged by thermal noise or background stray light, resulting in a break in the continuity of super-resolution reconstruction and a significant decrease in fidelity. Summary of the Invention

[0004] The purpose of this application is to provide a three-dimensional single-molecule localization imaging system, method, device, medium, and product that can improve the utilization rate of single-molecule localization information, the fidelity of three-dimensional images, and the imaging throughput.

[0005] To achieve the above objectives, this application provides the following solution.

[0006] In a first aspect, this application provides a three-dimensional single-molecule localization imaging system, comprising: a light source module for generating an excitation beam that irradiates a single-molecule sample; an intensity modulation module for modulating the intensity of the excitation beam to obtain a modulated excitation beam; wherein the single-molecule sample emits a fluorescence pulse under the excitation of the modulated excitation beam; a depth modulation module for modulating the axial position of the fluorescence pulse to obtain an encoded fluorescence pulse; an event detection module for generating a photon event sequence based on the encoded fluorescence pulse; wherein each photon event in the photon event sequence includes fluorescence spot coordinates, a fluorescence trigger timestamp, and fluorescence intensity change characteristic parameters; and a localization imaging module for calculating the three-dimensional spatial coordinates of each single molecule in the single-molecule sample based on the photon event sequence, and reconstructing a three-dimensional single-molecule image corresponding to the single-molecule sample based on the three-dimensional spatial coordinates of all single molecules in the single-molecule sample.

[0007] Secondly, this application provides a three-dimensional single-molecule localization imaging method, comprising: generating an excitation beam irradiating a single-molecule sample through a light source module; modulating the intensity of the excitation beam through an intensity modulation module to obtain a modulated excitation beam; the single-molecule sample emitting a fluorescence pulse under the excitation of the modulated excitation beam; modulating the axial position of the fluorescence pulse through a depth modulation module to obtain an encoded fluorescence pulse; generating a photon event sequence based on the encoded fluorescence pulse through an event detection module; each photon event in the photon event sequence includes fluorescence spot coordinates, a fluorescence trigger timestamp, and fluorescence intensity change characteristic parameters; and calculating the three-dimensional spatial coordinates of each single molecule in the single-molecule sample based on the photon event sequence through a localization imaging module, and reconstructing a three-dimensional single-molecule image corresponding to the single-molecule sample based on the three-dimensional spatial coordinates of all single molecules in the single-molecule sample.

[0008] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described three-dimensional single-molecule localization imaging method.

[0009] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described three-dimensional single-molecule localization imaging method.

[0010] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described three-dimensional single-molecule localization imaging method.

[0011] Based on the specific embodiments provided in this application, the following technical effects are disclosed.

[0012] This application provides a three-dimensional single-molecule localization imaging system, method, device, medium, and product. The system utilizes a light source module to stably output an excitation beam, providing a reliable light source foundation for subsequent optical signal modulation and sample excitation, ensuring the continuity and consistency of the entire imaging system's light input. An intensity modulation module modulates the intensity of the excitation beam, thereby controllably exciting single-molecule samples to emit fluorescence pulses. This forces the originally continuous bright-state emission platform of the single molecule to be re-encoded into a high-frequency or quasi-high-frequency time pulse sequence, enabling the event detection module to output an extremely dense, highly decodeable photon event sequence during the duration of the single-molecule's bright state. A depth modulation module modulates the axial position of the fluorescence pulses, transmitting molecular depth information. Coupled to the fluorescence pulse, the two-dimensional fluorescence signal is converted into a signal carrying spatial depth information, supplementing the key depth dimension data for subsequent three-dimensional spatial coordinate calculation. The event detection module generates a photon event sequence containing fluorescence spot coordinates, fluorescence trigger timestamps, and fluorescence intensity change characteristic parameters based on the coded fluorescence pulse, completely preserving the spatiotemporal variation details of the fluorescence signal and converting the optical signal into a digital data sequence that can be analyzed by the algorithm. The positioning imaging module completes the accurate calculation of the three-dimensional spatial coordinates of a single molecule based on the standardized photon event sequence, and reconstructs a complete three-dimensional single-molecule image based on all molecular coordinates, ultimately realizing the visualization of the spatial morphology and distribution of the single-molecule sample. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the overall structure of a three-dimensional single-molecule localization imaging system according to an embodiment of this application.

[0015] Figure 2 This is a schematic diagram comparing the effects of global modulation and local modulation according to an embodiment of this application, wherein, Figure 2 Part (a) in the diagram is a schematic diagram of the field of view of global modulation. Figure 2 Part (b) is a schematic diagram of the field of view of local modulation. Figure 2 Part (c) is a schematic diagram of the time response curve of global modulation. Figure 2 Part (d) in the figure is a schematic diagram of the time response curve of local modulation.

[0016] Figure 3 This is a schematic diagram illustrating the principle of active modulation to increase the number of events, provided in an embodiment of this application. Figure 3Part (a) in the figure is a schematic diagram of the time series curve of single-molecule fluorescence intensity without modulation. Figure 3 Part (b) is a schematic diagram of the event generation rate curve without modulation. Figure 3 Part (c) in the diagram is a schematic diagram of the time-series curve of single-molecule fluorescence intensity under modulation. Figure 3 Part (d) in the figure is a schematic diagram of the event generation rate curve when there is modulation.

[0017] Figure 4 This is a schematic diagram of the architecture of different three-dimensional spatial feature encoding components provided in an embodiment of this application.

[0018] Figure 5 This is a schematic diagram comparing single-molecule imaging results with and without modulation and with a frame camera, provided as an embodiment of this application.

[0019] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0020] Figure reference numerals: 10-Light source module, 11-Excitation beam, 12-0th order light, 13-1st order diffracted light, 14-Plain-top uniform beam, 15-Main excitation beam, 20-Global modulation unit, 30-Beam collimation and shaping assembly, 40-Local modulation unit, 41-Microlens array, 42-First lens, 43-Galvanometer, 44-Second lens, 45-Galvanometer driver, 46-Modulated light field, 50-Main beam splitter assembly, 51-Emission filter assembly, 60-Microscopic imaging assembly, 61-Lens tube lens, 62-Microscopic objective lens, 63-Sample scanning assembly, 64- Sample focal plane, 65-Target excitation region, 66-Lateral drive component, 67-Axial drive component, 70-Three-dimensional spatial feature encoding component, 71-Converging lens, 72-Cylindrical lens, 73-Pure phase liquid crystal spatial light modulator, 74-Third lens, 75-Fourth lens, 76-Fifth lens, 77-Beam splitter, 80-Event detection module, 81-Detection plane, 82-Second event detection module, 83-Second detection surface, 90-Synchronization control and event reconstruction component, 100-Two-dimensional spatial geometric features, 101-Second two-dimensional spatial geometric features. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Event cameras can respond to changes in the emission of fluorescent molecules, especially at the instants when a single molecule changes from dark to bright and from bright to dark, outputting a polarized asynchronous event stream, demonstrating their potential in super-resolution imaging. However, despite the significant advantages of event detectors, simply following the existing approach of "using event cameras to directly replace traditional cameras for passive observation" encounters a physical conflict: event cameras are only sensitive to relative changes in light intensity (time derivative), and do not respond to absolute light intensity. In actual photophysics processes of single molecules, the typical state is characterized by a near-continuous and stable "bright emission plateau" after transitioning from a dark state to a bright state, before returning to a dark state. This emission characteristic means that event cameras are only triggered at the instant of molecular state switching, and during the duration of the bright plateau, the emission signal is approximately constant, and the event camera almost no longer outputs events. Furthermore, to obtain the Z-axis depth of a single molecule, three-dimensional spatial encoding techniques such as astigmatism or double helix are usually required. These techniques rely on rich PSF geometry (such as precise elliptical profiles or clear two-point rotations) for fitting and inversion. However, sparse event point clouds generated by passive observation are simply unable to support complex three-dimensional morphological representations and are easily overwhelmed by thermal noise or background stray light, resulting in discontinuity breaks and a significant decrease in fidelity in super-resolution reconstruction.

[0024] This application is proposed against the backdrop of the aforementioned physical conflict. The research and development concept of this application is based on a completely new premise: instead of passively waiting for and recording the inherent random flickering start and end of a single molecule, it "actively changes" the emission behavior of a single molecule along the time axis through hardware means. Based on this concept, this application aims to solve the following problems.

[0025] 1. Addressing the issue of low information utilization: By introducing active modulation at the excitation / illumination end (such as using an acousto-optic modulator (AOM) for global high-frequency amplitude cutting, or using a microlens array for local spatial scanning), the originally smooth and continuous "bright-state emission platform" of a single molecule is forcibly re-encoded into a high-frequency event pulse sequence with drastic temporal fluctuations. This allows the event camera to be continuously triggered at high frequencies throughout the entire bright state of a single molecule, achieving an exponential increase in the amount of usable event information.

[0026] 2. Solving the problem of large field of view data congestion: By introducing a local temporal control mechanism, the light intensity fluctuations in different areas of the field of view are physically misaligned in time, which cleverly resolves the problem of event sensor bandwidth saturation caused by the simultaneous flickering of a large number of single molecules.

[0027] 3. Solve the problems of poor robustness of 3D localization and computing power bottleneck: The dense modulation event stream provides the underlying data with extremely high signal-to-noise ratio for subsequent local image generation, enabling the system to clearly resolve the 3D geometric deformation introduced by the cylindrical lens or phase mask; combined with the separable projection sub-pixel fitting algorithm optimized for batch processing, the complete data link from hardware high-frequency modulation to back-end ultra-fast reconstruction is completely opened up.

[0028] In one exemplary embodiment, a three-dimensional single-molecule localization imaging system is provided, including: a light source module, an intensity modulation module, a depth modulation module, an event detection module, and a localization imaging module.

[0029] The system comprises the following modules: a light source module to generate an excitation beam that irradiates the single-molecule sample; an intensity modulation module to modulate the intensity of the excitation beam to obtain a modulated excitation beam; and a depth modulation module to modulate the axial position of the fluorescence pulses to obtain coded fluorescence pulses. An event detection module to generate a photon event sequence based on the coded fluorescence pulses. Each photon event in the photon event sequence includes fluorescence spot coordinates, a fluorescence trigger timestamp, and fluorescence intensity change characteristic parameters. Specifically, the fluorescence intensity change characteristic parameter is the fluorescence change polarity. A localization imaging module to calculate the three-dimensional spatial coordinates of each single molecule in the single-molecule sample based on the photon event sequence, and to reconstruct a three-dimensional single-molecule image corresponding to the single-molecule sample based on the three-dimensional spatial coordinates of all single molecules in the single-molecule sample.

[0030] In one example, the intensity modulation module includes a global modulation unit and a local modulation unit. The global modulation unit is used to pulse-modulate the excitation beam to obtain a pulse-modulated excitation beam. The local modulation unit is used to spatially modulate the pulse-modulated excitation beam to obtain the modulated excitation beam.

[0031] In this example, as Figure 1 As shown, the light source module 10 is used to output the excitation beam 11. The light source module 10 can employ a single-mode laser or a multi-wavelength laser engine to support monochromatic or multi-color single-molecule localization imaging.

[0032] The global modulation unit 20 is used to apply time-waveform modulation to the entire excitation beam 11, with the ultimate goal of applying preset time-domain modulation to the single-molecule luminescence process within the target excitation region 65. In one example, the global modulation unit 20 employs a combination of a polarizer and an acousto-optic modulator. The polarizer is used to adjust the polarization state of the excitation beam 11 so that it enters the aperture of the acousto-optic modulator, and the acousto-optic modulator's RF driver receives an analog voltage signal to perform high-frequency amplitude modulation. The first-order diffracted light 13 of this acousto-optic modulator is used as the main excitation beam, while the residual zero-order light 12 is blocked to improve the switching modulation contrast. In addition to the aforementioned acousto-optic modulator, the global modulation unit 20 can also be used for electro-optic modulators, pulsed laser drive modules, light-emitting diode (LED) fast gating modules, current modulation drive modules, and mechanical or electronic shutter modules.

[0033] The beam collimation and shaping component 30 collimates and shapes the first-order diffracted light 13 to obtain a relatively ideal flat-top uniform beam 14.

[0034] The local modulation unit 40 performs time-domain modulation on the flat-top uniform beam 14, with the ultimate goal of applying preset time-domain modulation to the single-molecule luminescence process within the target excitation region 65. Its function is similar to that of the global modulation unit 20, but there are also differences. For example... Figure 2 As shown, in global modulation, the waveforms of points A and B in the field of view are synchronous and uniform high-frequency square waves after modulation, indicating that the entire field of view is modulated simultaneously, and the modulation time of all regions is consistent. In local modulation, the modulation waveforms of points A and B are envelope waves that are staggered, indicating a time difference. The 't' indicates that the modulation beam is scanning; a point is only modulated when the beam sweeps across it. Therefore, the difference between global modulation and local modulation lies in the fact that the fluctuations caused by the global modulation unit 20 occur synchronously across different regions of the entire field of view, while the intensity fluctuations caused by the local modulation unit 40 in different regions of the field of view are staggered in time. This helps reduce data congestion on the detector. The global modulation unit 20 and the local modulation unit 40 can be located at other equivalent positions in the system and can be used individually or in combination.

[0035] like Figure 1As shown, the local modulation unit 40 consists of a microlens array 41 and corresponding lenses. The microlens array 41 is used to perform wavefront modulation on the flat-top uniform beam 14 to obtain a multi-focal illumination pattern. The first lens 42 is used to focus the beam. The galvanometer 43 is used to scan the illumination pattern, causing light illumination fluctuations within the target excitation region 65 through scanning. The galvanometer driver 45 is used to drive the galvanometer 43 to move. The second lens 44 is used to restore the beam to a parallel light state for emission, forming a modulated light field 46. In addition to the above configuration, the local modulation unit 40 can also be a digital micromirror device, a spatial light modulator, a diffractive optical element, a speckle generator, a micro-electro-mechanical systems scanner (MEMS), or other structured light or local scanning generation devices.

[0036] like Figure 3 As shown, introducing multi-dimensional "active modulation" into the optical path can forcibly re-encode the originally continuous bright-state emission plateau of a single molecule into a high-frequency or quasi-high-frequency time pulse sequence. Without modulation, the fluorescence intensity of a single molecule during the bright-state plateau (t...) on t off The start and end times of the bright plateau phase remain stable, with only a small number of positive and negative events triggered at the rising / falling edges of the signal. No events are generated during the bright plateau phase, resulting in a small overall number of events. However, with modulation, the fluorescence signal undergoes rapid changes in brightness during the bright plateau phase by periodically modulating the excitation light intensity. The event detection module 80 can continuously generate events for each intensity change, significantly increasing the number and density of events generated per molecule, and providing richer temporal data support for subsequent three-dimensional single-molecule localization imaging.

[0037] like Figure 1 As shown, the main beam splitter 50 (such as a dichroic mirror) is used to receive the main excitation beam 15 modulated by the local modulation unit 40 and guide it to the microscopic imaging assembly 60 by reflection (or transmission).

[0038] The microscopic imaging assembly 60 is configured to ensure that the modulated light field 46 output by the local modulation unit 40 is accurately projected onto the sample focal plane 64, while simultaneously collecting the emitted light. It consists of a tube lens 61, a microscope objective 62, and necessary conjugate relay mirrors to ensure a strict conjugate relationship between the modulated light field 46, the probe plane 81, and the sample focal plane 64. The sample scanning assembly 63 and its associated lateral drive assembly 66 and axial drive assembly 67 are used to perform three-dimensional spatial movement of the sample.

[0039] A three-dimensional spatial feature encoding component (depth modulation module) 70 is configured in the emission optical path to receive the emitted fluorescence wavefront and actively introduce a preset optical distortion or optical path difference. Its function is to physically encode the depth information of a single molecule in the Z-axis direction of the object space into a two-dimensional spatial geometric feature of the point spread function of the emitted beam. In one example, the depth modulation module is a cylindrical lens, a phase mask, or a multi-path axial differential optical path.

[0040] In one example, such as Figure 4 As shown, to adapt to different axial detection depth and resolution requirements, the three-dimensional spatial feature encoding component 70 can be configured into the following three typical implementation paradigms based on different physical modulation mechanisms.

[0041] like Figure 4 As shown in part (a), when used only for two-dimensional single-molecule imaging, the three-dimensional spatial feature encoding component 70 degenerates into a single converging lens 71 to converge parallel emitted light from infinity onto the detection plane 81.

[0042] like Figure 4 As shown in section (b), the three-dimensional spatial feature encoding component 70 can be configured as a deformation encoding architecture based on astigmatism modulation. In this configuration, the three-dimensional spatial feature encoding component 70 employs a cylindrical lens 72 that disrupts the system's axisymmetry. When a single-molecule fluorescence wavefront passes through the cylindrical lens, different focal lengths are generated in two mutually perpendicular lateral meridional planes. This causes the distribution of the single-molecule event point cloud captured by the event detection module 80 to exhibit depth-dependent deformation, imaging different two-dimensional spatial geometric features 100 at different locations off the focal plane. At the focal plane, there is a clustered circular point cloud; above the focal plane, there is a longitudinally distributed elliptical point cloud; and below the focal plane, there is a laterally distributed elliptical point cloud. By fitting the two-dimensional spatial covariance or major-minor axis ratio of the event cloud clusters using an algorithm, the axial depth of the single molecule is realized. z The monotonic inversion.

[0043] like Figure 4As shown in section (c), the three-dimensional spatial feature encoding component 70 can be configured as a wavefront rotation encoding architecture based on a phase mask. In this configuration, the three-dimensional spatial feature encoding component 70 employs a cascaded combination of a 4f relay system and a spatial phase modulator. A 4f system consisting of a third lens 74 and a fourth lens 75 is added to the emission optical path, and a phase mask (such as a double-helix phase plate) or a pure phase liquid crystal spatial light modulator 73 is placed on its Fourier surface (pupil surface). This modulation changes the spatial spectral distribution of the wavefront, causing the originally single PSF to split into two main and side lobes. On the detection plane 81, the same scintillation-emitting single molecule will simultaneously trigger two clusters of mutually separated event streams (i.e., double-helix event clusters). As the axial depth of the single molecule changes, the angle of the line connecting these two clusters of event streams rotates continuously, imaging different two-dimensional spatial geometric features 100 at different positions away from the focal plane. By calculating the rotation angle of the centers of the two clusters of events, high-precision positioning along the Z-axis over a wide range is achieved.

[0044] like Figure 4 As shown in section (d), the three-dimensional spatial feature encoding component 70 can be configured as an axial differential architecture based on beam splitting and multipath propagation. In this configuration, the three-dimensional spatial feature encoding component 70 uses a combination of beam splitters and unequal-spacing probe surfaces to perform depth encoding using the principle of axial optical tomography.

[0045] Before the emitted fluorescence wave enters the component, it is split into at least a first detection optical path and a second detection optical path by the beam splitter 77 according to energy. The first optical path is the original detection optical path. In the second optical path, a second event detection module 82 is added. However, there is an optical field difference between the sample focal plane corresponding to the second detection plane 83 of the second event detection module 82 and the sample focal plane corresponding to the detection plane 81. Therefore, a fifth lens 76 is added. Thus, the event detection module 80 and the second event detection module 82 are respectively conjugated to two different focal planes with a preset axial spacing in the sample space. For the same modulated single molecule, the two regions of the detection module will capture two clusters of event streams with completely different spatial broadening (one side is focused, such as the two-dimensional spatial geometric feature 100, and the other side is defocused and diffused, such as the second two-dimensional spatial geometric feature 101). By combining spatial domain differential fitting, its true Z-axis coordinates are directly locked. This architecture maintains a pure diffraction-limited PSF morphology.

[0046] like Figure 1 As shown, the emission filter component 51 is disposed on the transmission (or reflection) optical path of the main beam splitter component 50, and includes an emission filter for filtering out residual excitation light and background noise.

[0047] The event detection module 80 is positioned at the final imaging target surface of the system, and its detection plane 81 is strictly conjugate with the sample focal plane 64. It is used to acquire single-molecule emission signals after three-dimensional spatial encoding and output the raw event stream.

[0048] The event detection module 80 is an event-based detector (such as an event camera). During operation, driven by the inherent photophysical scintillation of molecules or by high-frequency modulation applied by the global modulation unit 20 and the local modulation unit 40, the photosensitive pixels on the detector target surface asynchronously output discrete events containing spatial coordinates, timestamps, and polarities only when the relative change in the received light intensity exceeds a preset threshold. Therefore, the 3D encoded PSF morphological features are transformed into a high-temporal-resolution asynchronous event point cloud set.

[0049] The synchronization control and event reconstruction component 90 is configured as the central hub of the system. It is responsible for unified timing coordination and reference synchronization of processes such as light source excitation, active modulation, and event detection. In one example, the positioning imaging module is built into the synchronization control and event reconstruction component 90, which receives the raw single-molecule event stream (i.e., photon event sequence) output by the event detection module 80, extracts the encoded PSF features from the asynchronous event cloud by performing cluster analysis and joint parameter fitting, and finally calculates the super-resolution three-dimensional spatial coordinates and high-frequency dynamic evolution information of the single molecule.

[0050] In another example, the three-dimensional single-molecule localization imaging system includes a single-molecule excitation and imaging module, an active modulation module, a three-dimensional spatial feature encoding component, an event detection module, an event preprocessing and candidate detection module, a local point spread function image generation module, and a sub-pixel localization and reconstruction module.

[0051] The single-molecule excitation and imaging module is used to establish the optical input and detection path required for single-molecule localization, including a laser source, a beam splitter, and a microscope objective, to excite the sample and collect the emission signal.

[0052] The active modulation module is the core hardware, located in the excitation or detection path, used to apply a preset spatiotemporal joint modulation to the single-molecule luminescence process, converting the single-molecule bright state plateau into a time-fluctuating signal. In this example, active modulation includes global amplitude modulation (such as AOM) and local spatial modulation (such as microlens array + galvanometer).

[0053] The 3D spatial feature encoding component is a key extension hardware component, configured in the emission optical path. It actively introduces pre-defined optical distortions or optical path differences to physically encode the molecular Z-axis depth information into two-dimensional morphological features of the point spread function. It encompasses switchable paradigms such as astigmatism modulation (cylindrical lens), wavefront rotation encoding (phase mask / 4f optical system), or beam-splitting multi-path axial differential architecture. The event detection module is configured on the final target surface to respond to the relative changes in high-frequency light intensity induced by active modulation, acquiring and outputting a raw single-molecule event stream with high temporal resolution. In this example, the event detection module is an event camera. The event preprocessing and candidate detection module performs noise reduction and spatiotemporal clustering on the raw single-molecule event stream, accurately extracting candidate single-molecule signal clusters. The local point spread function image generation module extracts the depth morphological features (such as major and minor axes, rotation angles) contained in the raw single-molecule event stream, transforming them into local images or event stacks suitable for algorithm fitting. The subpixel localization and reconstruction module uses a multimodal spatiotemporal joint fitting algorithm to estimate the four-dimensional spatial coordinates (three dimensions + time) of a single molecule, and finally outputs a high-fidelity super-resolution three-dimensional image reconstruction result.

[0054] In one example, the 3D spatial feature encoding component employs cylindrical lenses (astigmatic method), double-helix phase masks (wavefront rotation method), or beam splitting multipaths (axial difference method) for depth encoding. Alternatively, the 3D spatial feature encoding component can also be implemented using other complex pure phase modulation masks such as tetrapods or saddle-points, multi-wavelength / chromatic aberration depth encoding devices, or self-interfering optical architectures (such as 4Pi or interferometric photoactivated localization microscopy (iPALM) interference paths). The essence of the 3D spatial feature encoding component lies in its ability to actively introduce optical distortion or optical path difference into the emitted optical path, physically encoding the Z-axis depth information of a single molecule into a two-dimensional morphological or energy distribution feature recognizable by an event camera.

[0055] Event preprocessing and denoising methods employ either 3D point cloud analysis based on the covariance matrix or dual-scale frame sequence processing based on multi-scale convolution. Alternatively, event preprocessing and denoising can also be implemented using methods such as graph neural network point cloud analysis, spatiotemporal convolutional neural network (3D-CNN) filtering, physical filtering based on polarity matching (positive (ON) / negative (OFF) events appear in pairs), Bayesian filtering, or Kalman filtering. The essence lies in utilizing spatiotemporal aggregation characteristics to separate real single-molecule signals from massive active modulation event streams and remove background / hot pixel noise.

[0056] The sub-pixel 3D joint solution algorithm employs the variable projection Levenberg-Marquardt (LM) algorithm for Gaussian fitting. Alternatively, the sub-pixel 3D joint solution algorithm can also be implemented using: Maximum Likelihood Estimation (MLE) based on a Poisson / Gaussian noise model; 3D surface fitting directly to the 3D point cloud (without explicitly generating a 2D local image); end-to-end single-molecule coordinate regression based on deep learning (e.g., inputting a local event stream and directly outputting 3D spatial coordinates); and phase manifold extraction algorithms. Essentially, it can inversely solve for the single-molecule 3D spatial coordinates, including the Z-axis depth, based on the extracted local event stream or its 2D reduced-dimensional feature image.

[0057] In another example, the intensity modulation module is a global modulation module or a local modulation module. The modulation excitation beam is a pulse modulation excitation beam or a spatial modulation excitation beam.

[0058] The global modulation module is used to pulse-modulate the excitation beam to obtain the pulse-modulated excitation beam. The local modulation module is used to spatially modulate the excitation beam to obtain the spatially modulated excitation beam.

[0059] In one example, the excitation-side global modulation unit uses an acousto-optic modulator (AOM) and analog voltage drive to acquire first-order diffracted light. Alternatively, the global modulation unit can be implemented using an electro-optic modulator (EOM), direct drive control of a pulsed laser, a high-power LED fast gating module, an intensity modulation mode of an acousto-optic deflector (AOD), or a high-frequency mechanical / electronic shutter module. The essence of the global modulation unit is its ability to apply high-frequency time waveform modulation to the overall illumination input, transforming single-molecule bright-state plateaus within the field of view into a time-pulse sequence of continuously triggering events.

[0060] The local modulation unit on the emitter side employs a single-axis scanning galvanometer combined with a rotating disk (or microlens array) featuring an array of tilted pinholes. Alternatively, the local modulation unit can be implemented using: a digital micromirror device (DMD) or spatial light modulator (SLM), a two-dimensional optical scanning galvanometer, a conventional Nipkow porous disk, a random speckle generator or dynamic optical diffraction element (DOE), or a high-frequency relative micro-displacement (e.g., piezoelectric ceramic drive) controlling the illumination pattern and the sample stage. The essence of the local modulation unit lies in its ability to introduce spatially non-uniform light field scanning or fluctuations during the duration of a single-molecule bright state, causing the emitted signals from different regions to fluctuate temporally and generate a continuous event response.

[0061] The dual-layer active modulation acts on the excitation path and the emission filter path (galvanometer + disk) respectively. Alternatively, the action point of active modulation can be equivalently transferred to the following locations: inside the event detector (e.g., high-frequency modulation of the internal bias voltage, photosensitive gain, or global gating of an event camera), only at the excitation end (e.g., high-frequency switching in full-field structured light), or only at the emission end (e.g., a high-speed liquid crystal grating in the emission optical path). Its essence lies in its ability to frequently alter the effective photon flux representation of a single molecule entering the target surface of the event detector.

[0062] High-throughput computing acceleration architectures employ Graphics Processing Units (GPUs) (developed and computed based on the Compute Unified Device Architecture (CUDA)), spatial mesh partitioning, prefix-sum indexing, and atomic addition operations to achieve parallel acceleration. Alternatively, underlying hardware acceleration can be implemented using: Field Programmable Gate Array (FPGA) hardware-level pipelined parallel processing, application-specific integrated circuit (ASIC) chips with embedded algorithms, multi-core CPU thread pool distributed architectures, and cloud cluster distributed computing. Essentially, the underlying computing power delivery platform can be replaced as computer engineering technology advances, enabling efficient processing of asynchronous event streams within an active modulation event coding framework.

[0063] In one example, the localization and imaging module includes: a feature extraction unit, a point spread function generation unit, a coordinate calculation unit, and an image generation unit.

[0064] The system comprises several components: a feature extraction unit for the photon event sequence to obtain spatiotemporal feature parameters of each molecule in the single-molecule sample; a point spread function (DFS) generation unit for obtaining the DFS image of each molecule in the single-molecule sample using the event projection accumulation method based on the photon event sequence and the spatiotemporal feature parameters of each molecule in the single-molecule sample; a coordinate calculation unit for obtaining the three-dimensional spatial coordinates of each molecule in the single-molecule sample by performing nonlinear least squares fitting using the variable projection Levenberg-Marquardt algorithm based on the DFS image of each molecule in the single-molecule sample; and an image generation unit for reconstructing and generating the three-dimensional single-molecule image using Gaussian rendering based on the three-dimensional spatial coordinates of all molecules in the single-molecule sample.

[0065] In another example, the positioning imaging module further includes a coordinate filtering unit, which is used to perform temporal filtering of the three-dimensional spatial coordinates of each single molecule in the single-molecule sample based on the molecular fluorescence activity time of each single molecule in the single-molecule sample.

[0066] In one example, the three-dimensional single-molecule image is a dynamic imaging image, in which each imaging point corresponds one-to-one with each single molecule in the single-molecule sample, and each imaging point in the three-dimensional single-molecule image is dynamically displayed according to the corresponding molecular fluorescence activity time.

[0067] In one example, the three-dimensional single-molecule localization imaging system further includes a denoising module, which is used to denoise the photon event sequence using a covariance matrix method or a multi-scale convolution method.

[0068] In another exemplary embodiment of this application, a three-dimensional single-molecule localization imaging method is provided, including the following steps 001 to 005.

[0069] Step 001: Generate an excitation beam to irradiate the single-molecule sample using the light source module.

[0070] Step 002: The intensity of the excitation beam is modulated using an intensity modulation module to obtain a modulated excitation beam. The single-molecule sample emits fluorescence pulses under the excitation of the modulated excitation beam.

[0071] Step 003: The fluorescence pulse is axially position modulated by the depth modulation module to obtain an encoded fluorescence pulse.

[0072] Step 004: The event detection module generates a photon event sequence based on the encoded fluorescence pulses. Each photon event in the photon event sequence includes fluorescence spot coordinates, a fluorescence trigger timestamp, and a fluorescence intensity change characteristic parameter. Specifically, the fluorescence intensity change characteristic parameter is the fluorescence change polarity.

[0073] Step 005: The positioning imaging module calculates the three-dimensional spatial coordinates of each single molecule in the single-molecule sample according to the photon event sequence, and reconstructs the three-dimensional single-molecule image corresponding to the single-molecule sample based on the three-dimensional spatial coordinates of all single molecules in the single-molecule sample.

[0074] In this example, the three-dimensional single-molecule localization imaging method follows the following temporal workflow: starting from the initial excitation of the single-molecule sample, the three-dimensional single-molecule localization imaging system synchronously performs active modulation (i.e., cutting the excitation light field through acousto-optic devices or scanning mechanisms); then, the event detection module captures the emission polarity event stream triggered by this modulation at high frequency; subsequently, after event stream preprocessing and candidate molecule detection to remove redundancy and noise, it further combines the three-dimensional distortion applied by optical hardware to perform local point spread function image / feature generation; finally, it performs sub-pixel spatiotemporal joint localization and completes the structural reconstruction beyond the diffraction limit.

[0075] Specifically, the three-dimensional single-molecule localization imaging method uses the synchronization control and event reconstruction component 90 as the global central hub, coordinating multiple hardware execution entities such as the light source module 10, global modulation unit 20, local modulation unit 40, and event detection module 80 to complete the task collaboratively. The specific process is as follows.

[0076] Step S1: Excite the single-molecule sample and establish a three-dimensional coded imaging state.

[0077] The synchronous control and event reconstruction component 90 issues control commands to drive the light source module 10 to output the excitation beam 11, which is then irradiated by the microscopic imaging component 60 onto the single-molecule sample, causing the single-molecule emitter in the target excitation region 65 to enter the sparse luminescence state required for single-molecule localization imaging. Simultaneously, the three-dimensional spatial feature encoding component 70, configured in the emission light path, receives the fluorescence wavefront emitted by the single molecule and actively introduces preset optical distortions (such as astigmatism, wavefront rotation, or axial difference) to physically encode the depth information of the single molecule in the Z-axis direction of the object space into the two-dimensional spatial geometric features of the point spread function of the emitted beam.

[0078] Step S2: Apply active spatiotemporal modulation to the single-molecule luminescence process.

[0079] In order to convert the near-continuous single-molecule bright-state emission platform into a high-frequency pulse, the synchronization control and event reconstruction component 90 drives the modulation component to perform preset intensity modulation on the excitation beam using one or two of the following combinations.

[0080] Global time waveform modulation: Drive the global modulation unit 20 (such as an acousto-optic modulator), input a 0V~1V analog voltage signal to perform high-frequency amplitude switching, so that the single-molecule excitation intensity in the entire field of view changes with time.

[0081] Local spatiotemporal fluctuation modulation: drive the local modulation unit 40 (such as microlens array and scanning galvanometer) to scan the multifocal illumination mode so that single molecules in different regions of the field of view experience periodic fluctuations in local illumination intensity during the bright state, and the fluctuations are staggered in time to reduce data congestion.

[0082] By using global or local modulation, the physical limitation of traditional event cameras—the inability to continuously respond to slow, bright states—can be overcome. By actively segmenting the continuous light field, the near-constant emission plateau of a single molecule can be forcibly re-encoded into a high-frequency, dense sequence of modulated photon pulses.

[0083] Step S3: Acquire the actively modulated raw single-molecule event stream.

[0084] The event detection module 80 continuously monitors the relative changes in light intensity on the detection plane 81. When the rate of change of light intensity received by its pixels exceeds a preset threshold, it asynchronously outputs a series of discrete event signals to obtain the original single-molecule event stream. The original single-molecule event stream is defined in a three-dimensional spatiotemporal coordinate system. point cloud collection , ,in, x , y and t These are the horizontal spatial coordinates, vertical spatial coordinates, and time coordinates, respectively. For the first i discrete event points and These represent the horizontal and vertical coordinates of the spatial pixels where the event occurred, i.e., the horizontal and vertical coordinates of the fluorescent points. This is the timestamp of the event, i.e., the fluorescence trigger timestamp. The event polarity, i.e., the characteristic parameter of fluorescence intensity change, represents whether the light intensity increases or decreases. N The total number of discrete event points. This step captures the high-frequency dynamic features and three-dimensional coded morphology of single molecules introduced by active modulation with extremely low data redundancy and microsecond-level time resolution.

[0085] Step S4: Perform noise reduction preprocessing on the original single-molecule event stream from a dual perspective.

[0086] The synchronous control and event reconstruction component 90 receives the raw single-molecule event stream output by the event detection module 80, and selects one of the following two noise reduction modes to remove transient environmental interference, sensor hot pixels and disordered background, and separates signal events with high positioning value from massive events according to the data processing strategy.

[0087] Mode 1: 3D point cloud denoising based on covariance matrix. (For a 3D spatiotemporal coordinate system...) Constructing the covariance matrix from local neighborhoods Eigenvalue decomposition is performed to extract three eigenvalues. Utilizing the prior knowledge that single-molecule event points exhibit isotropic aggregation in space and time, while noise events exhibit isolated one-dimensional linear extensions, a maximum eigenvalue threshold is set. and isotropic threshold Remove hot pixels and low-density backgrounds. When and When this occurs, it is retained as a valid signal event.

[0088] Mode 2: Denoising of 2D cumulative frame sequences based on multi-scale convolution. The asynchronous event stream is divided into fixed time windows. Discretized into a two-dimensional event frame sequence .right Perform dual-scale convolutional difference with Gaussian smoothing kernel. To suppress high-frequency noise, use sparse convolution kernels. Extract and subtract low-frequency, slowly changing background, and finally segment using a dynamic threshold (e.g., greater than 100%). (Number of noise standard deviations) preserves signal pixels.

[0089] Step S5: Detect candidate single-molecule centers and their active time intervals.

[0090] Based on the denoised data, the synchronous control and event reconstruction component 90 detects the geometric center and active region of each single-molecule event cluster to separate and lock the physical location boundary of the spot on the target surface and the time window of occurrence of each potential single-molecule signal from the global event data pool. Similar to step S4, there are two processing modes based on different data perspectives.

[0091] Mode 1: From a point cloud perspective, perform density-based spatial clustering (such as Density-Based Spatial Clustering of Applications with Noise, DBSCAN) on the denoised event set. Utilizing GPU parallel acceleration, core points that overlap and meet the minimum neighborhood density requirement are grouped into the same event cluster, and the spatial bounding box boundary of each cluster is calculated. and the start and end range of time And solve for its geometric centroid. .in, and These are the minimum and maximum horizontal coordinates of all event points within the cluster, respectively. and These are the minimum and maximum coordinates of all event points within the cluster in the vertical direction, respectively. and These are the minimum and maximum timestamps of all event points within the event cluster, respectively. and These are the geometric centers of the horizontal and vertical coordinates of all event points within the event cluster, respectively.

[0092] Mode 2: From the perspective of frame sequence, local maxima search is performed on the accumulated two-dimensional event frames, and non-maximum suppression (setting an exclusion radius R) is used for filtering. Subsequently, through connected component area analysis, false positive signals with areas outside the reasonable physical range are eliminated, and the coordinates of the retained maxima and the corresponding starting frame time are recorded as candidate monomolecular centers.

[0093] Step S6: Generate the local point spread function (PSF).

[0094] For each candidate molecule extracted in step S5, the synchronous control and event reconstruction component 90 extracts its corresponding local events and generates a two-dimensional local PSF image suitable for sub-pixel fitting. The non-standard asynchronous event point cloud is dimensionality-reduced and transformed into a two-dimensional image matrix suitable for classical optimization algorithms, fully preserving the depth morphological features introduced by the three-dimensional spatial feature encoding component 70. Depending on the perspective adopted in step S5, step S6 has the following two processing modes.

[0095] Mode 1: Generation method based on point cloud clustering. This involves generating independent clusters pre-defined by clustering algorithms such as DBSCAN. Project all its internal events directly along the time axis and accumulate them to generate a size of A two-dimensional local image, wherein, Let be the side length of the two-dimensional local image.

[0096] Mode 2: Generation method based on candidate centers. This method uses local maxima centers. Based on this, define a spatial radius. R and time extension The spatiotemporal neighborhood of. Among them, and These are the x and y coordinates of the local maximum center, respectively. A GPU-accelerated solution using spatial grid partitioning and prefix-sum indexing is employed to quickly retrieve events within the neighborhood from the global event stream and project and accumulate them.

[0097] Step S7: Subpixel 3D joint solution based on separable projection algorithm.

[0098] The synchronous control and event reconstruction component 90 performs sub-pixel localization and depth calculation on the local PSF image. Since the 3D spatial feature encoding component 70 maps the Z-axis depth to 2D PSF parameters (such as standard deviation broadening of a Gaussian model), this step uses an isotropic or anisotropic Gaussian model to perform nonlinear least-squares fitting on the PSF image. This accurately extracts the spatial position information and Z-axis depth information of single molecules, achieving spatial resolution exceeding the optical diffraction limit and significantly improving the algorithm's throughput.

[0099] In this example, to achieve high-throughput batch processing, instead of directly solving the massive three-dimensional Jacobian Tensor, the variable projection LM algorithm is introduced. This leverages the row-column separability of the Gaussian model to transform the Hessian matrix terms... and gradient vector terms The two-dimensional pixel traversal and accumulation operation in the mathematical decomposition is as follows: x shaft and y Independent one-dimensional projection and one-dimensional inner product calculations along the axial direction avoid the massive memory read / write bottleneck caused by explicitly constructing the Jacobian matrix. Taking two-dimensional single-molecule reconstruction as an example, the parameter vector is finally obtained by solving the Cholesky decomposition method. .in, For background noise, For single-molecule peak intensity, and These represent the x and y coordinates of the sub-pixel two-dimensional center of a single molecule. This is the broadening parameter of the PSF. Substituting this broadening parameter into the 3D encoding calibration curve (such as the axial position calibration function of the astigmatism method), the axial depth of the single molecule is calculated. z Specific solution z The method is as follows.

[0100] for Figure 4 The configuration shown in part (b) involves substituting the extracted width value into the preset astigmatism calibration model shown in the following formula. By minimizing the residual between the measured value and the model value, the solution is obtained. z .

[0101] .

[0102] in, For local PSF images in x or y Fit width in the direction, is the minimum beam waist width in the direction corresponding to the focal plane of the system, and is a calibration parameter. for x or y The axial position of the directional focal point deviating from zero is determined by the amount of astigmatism introduced by the cylindrical lens. A These are the third-order aberration correction coefficients. B These are the fourth-order aberration correction coefficients. D For depth of focus parameters, z The axial depth of a single molecule is the objective to be solved.

[0103] for Figure 4 The configuration shown in section (c) performs bimodal center recognition within a local PSF image region to extract the centroid coordinates of the two main and side lobes. and Calculate the rotation angle of the line connecting the two points relative to the reference axis. Using the linear or polynomial mapping relationship between the rotation angle and the axial depth shown in the following formula, the angle search is used to inversely calculate... z .

[0104] .

[0105] in, The angle of rotation of the line connecting the two side petals. For the first k Phase coding sensitivity coefficient, n This represents the order of the polynomial. Simultaneously, the transverse coordinate of the single molecule... From the geometric midpoint of the two side lobes Provided.

[0106] for Figure 4 The configuration shown in section (d) synchronously retrieves local PSF images corresponding to two different focal planes of the same single molecule. and Calculate the broadening width (or peak light intensity density) of the event stream in each of the two images. and Combined with the preset optical path difference between the two detection surfaces The following formula is used to perform a joint search based on the ratio or difference of the width of the two sides and the corresponding curve of the depth.

[0107] .

[0108] in, This is the axial difference ratio function.

[0109] Step S8: Super-resolution image four-dimensional reconstruction.

[0110] The synchronous control and event reconstruction component integrates the 3D sub-pixel coordinates of all single molecules calculated in step S7 and the temporal active windows of single molecules extracted in step S5. Utilizing the known temporal prior information of the actively modulated waveform, stray molecules with low location confidence are removed. Finally, through Gaussian rendering and other methods, a super-resolution structural image of the target sample containing high-frequency temporal evolution characteristics is reconstructed. The final high-fidelity, high spatiotemporal resolution multimodal reconstruction result is output, completing the closed loop of the 3D single-molecule localization microscopy imaging workflow. Figure 5 As shown, the imaging results of ordinary frame cameras are limited by exposure time and frame rate, and can only capture a portion of single-molecule signals, resulting in generally poor image contrast and continuity. Although the imaging results of unmodulated event cameras can respond to changes in light intensity, the low number of single-molecule bright plateau events and low signal density lead to insufficient image details. In contrast, the imaging results of event cameras with active modulation significantly increase the number of events generated and improve signal continuity because the single-molecule bright plateau is re-encoded into a high-frequency pulse sequence. The microtubule skeleton structure details in the image are clearer, the contrast is higher, and the overall imaging quality is significantly better than the former two. This directly demonstrates the improvement effect of the active modulation technology of this application on the single-molecule imaging performance of event cameras.

[0111] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores fluorescence pulses, coded fluorescence pulses, photon event sequences, and the three-dimensional spatial coordinates of individual molecules. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a three-dimensional single-molecule localization imaging method.

[0112] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include, for example, [the following is a list of possible additional structures]. Figure 6The diagram shows more or fewer components, or combinations of certain components, or different component arrangements.

[0113] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0114] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0115] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0116] This application addresses the pain point of single molecules exhibiting near-continuous, slow-changing behavior during their bright state, making it difficult to sustain event camera responses. By introducing multi-dimensional "active modulation" into the optical path, the originally continuous bright-state emission platform of a single molecule is forcibly re-encoded into a high-frequency or quasi-high-frequency time pulse sequence. This modification enables the event detection module to output an extremely dense and highly decodeable event stream during the continuous bright state of a single molecule. Based on this, feature extraction and computation are performed using a specific hardware architecture, significantly improving the utilization rate of positioning information, the fidelity of 3D reconstruction, and the imaging throughput. The "active modulation" includes global modulation and local modulation. Global modulation applies high-frequency time waveform modulation to the overall illumination input (e.g., the overall excitation intensity varies according to a preset frequency), while local modulation applies non-uniform scanning or fluctuation modulation to the spatial illumination field, causing asynchronous time signals in different regions of the field of view and preventing data congestion. Global and local modulation can be used individually or combined to construct a "dual-layer joint modulation," with the core purpose of "modifying the single-molecule bright-state platform to drive the event detection module."

[0117] This application breaks away from the inherent framework of "single-molecule direct passive event detection" in existing technologies, and perfectly bridges the gap between the working principle of event sensors and the luminescence characteristics of single molecules with "active modulation coding" as the core upstream mechanism. Specifically, compared with existing technologies, this application has the following significant technical effects and advantages.

[0118] First, it breaks through the information bottleneck of single-molecule detection in event cameras, greatly improving the utilization rate of bright-state photons.

[0119] In existing technologies, event cameras only produce a small response at the instant of a single molecule's illumination and extinguishing, remaining "blind" to its longer and nearly continuous "bright plateau period," resulting in a significant waste of photon information. This application successfully transforms the originally silent bright plateau into a high-frequency, continuous sequence of event pulses, significantly increasing the effective data density required for localization. This advantage mainly stems from the global modulation unit and / or local modulation unit. By applying preset active cutting and fluctuation modulation to the excitation beam in the time or spatial domain, the steady state of single-molecule luminescence is broken, thereby continuously triggering the event detection module.

[0120] Second, it helps to alleviate event data congestion and improve the ability to perform high-throughput imaging with a wide field of view.

[0121] If only global modulation is used, all excited single molecules within the field of view will flicker simultaneously, which can easily lead to pixel bandwidth saturation and data congestion in the event camera. This application achieves a staggered distribution of event triggering on the time axis within the field of view, significantly reducing the instantaneous data throughput pressure. This advantage mainly comes from the local modulation unit (such as a microlens array combined with a scanning galvanometer). The spatially non-uniform light field generated by this component scans within the field of view, causing single molecules in different physical regions to experience light intensity fluctuations at asynchronous times, thus achieving smooth staggering of the event data stream in the time dimension.

[0122] Third, it enables high-precision three-dimensional positioning that transcends two-dimensional planes.

[0123] This application endows the system with high-precision, interference-resistant Z-axis depth detection capabilities without sacrificing the ultra-high temporal resolution of the event camera, enabling the system to reconstruct the 3D super-resolution continuous structure of complex biological samples. This advantage mainly stems from the joint design of the 3D spatial feature encoding component and the data reconstruction process. By flexibly configuring cylindrical lenses (astigmatism method), phase masks (wavefront rotation method), or beam splitting multipaths (axial difference method) in the emission optical path, the Z-axis depth information is physically "dimensionally reduced and encoded" into the form of a two-dimensional point spread function. Then, through the geometric feature inversion of the event cloud, the asynchronous event flow and 3D spatial coordinates are perfectly bridged.

[0124] Fourth, it significantly reduces the pressure on video memory bandwidth, enabling ultra-fast sub-pixel fitting of massive events.

[0125] Traditional 3D optimization algorithms (such as the conventional LM algorithm) suffer from severe GPU memory read / write bottlenecks due to the frequent construction of high-dimensional Jacobian tensors when processing batches of local PSF images generated from massive events. This application significantly reduces memory access overhead during computation, resulting in a geometric increase in the throughput of sub-pixel 3D localization algorithms, laying a computational foundation for real-time / near-real-time single-molecule super-resolution imaging. This advantage primarily stems from the variable projection LM algorithm deployed in the synchronization control and event reconstruction components. This algorithm cleverly utilizes the separability of rows and columns in the Gaussian model, reducing the time-consuming 2D pixel accumulation to 1D projection and 1D inner product calculations, and combining it with Cholesky decomposition to solve symmetric positive definite systems, thus completely unlocking the underlying logic for computational acceleration.

[0126] Fifth, it provides extremely strong engineering scalability.

[0127] This application establishes a general technical framework of "active modulation + 3D encoding + event detection + joint solution". On the excitation side, it includes a dual path from global slicing of the acousto-optic modulator (AOM) to local scanning of the microlens array; on the detection side, it is compatible with almost all mainstream 3D extension components such as cylindrical lenses, double-helix phase plates, and double-sided beam splitters.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0129] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0130] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A three-dimensional single-molecule localization imaging system, characterized in that, The system includes: The light source module is used to generate an excitation beam that irradiates a single-molecule sample; An intensity modulation module is used to modulate the intensity of the excitation beam to obtain a modulated excitation beam; the single-molecule sample emits fluorescence pulses under the excitation of the modulated excitation beam; A depth modulation module is used to modulate the axial position of the fluorescence pulse to obtain an encoded fluorescence pulse; An event detection module is used to generate a photon event sequence based on the encoded fluorescence pulses; each photon event in the photon event sequence includes fluorescence spot coordinates, fluorescence trigger timestamp, and fluorescence intensity change characteristic parameters; The positioning and imaging module is used to calculate the three-dimensional spatial coordinates of each single molecule in the single-molecule sample based on the photon event sequence, and to reconstruct and generate a three-dimensional single-molecule image corresponding to the single-molecule sample based on the three-dimensional spatial coordinates of all single molecules in the single-molecule sample.

2. The three-dimensional single-molecule localization imaging system according to claim 1, characterized in that, The intensity modulation module is a global modulation module or a local modulation module; the modulation excitation beam is a pulse modulation excitation beam or a spatial modulation excitation beam. The global modulation module is used to pulse modulate the excitation beam to obtain the pulse-modulated excitation beam; The local modulation module is used to spatially modulate the excitation beam to obtain the spatially modulated excitation beam.

3. The three-dimensional single-molecule localization imaging system according to claim 1, characterized in that, The intensity modulation module includes a global modulation unit and a local modulation unit; The global modulation unit is used to pulse modulate the excitation beam to obtain a pulse-modulated excitation beam. The local modulation unit is used to spatially modulate the pulse-modulated excitation beam to obtain the modulated excitation beam.

4. The three-dimensional single-molecule localization imaging system according to claim 1, characterized in that, The positioning and imaging module includes: The feature extraction unit is used to extract features from the photon event sequence to obtain the spatiotemporal feature parameters of each single molecule in the single-molecule sample; the spatiotemporal feature parameters include the geometric center coordinates of the light spot, the physical location boundary of the light spot, and the molecular fluorescence activity time; The point spread function generation unit is used to obtain the point spread function image of each single molecule in the single molecule sample by using the event projection accumulation method based on the photon event sequence and the spatiotemporal characteristic parameters of each single molecule in the single molecule sample. The coordinate calculation unit is used to obtain the three-dimensional spatial coordinates of each single molecule in the single molecule sample by performing nonlinear least squares fitting using the variable projection Levenberg-Marquardt algorithm based on the point diffusion function image of each single molecule in the single molecule sample. The image generation unit is used to reconstruct and generate the three-dimensional single-molecule image based on the three-dimensional spatial coordinates of all single molecules in the single-molecule sample using a Gaussian rendering method.

5. The three-dimensional single-molecule localization imaging system according to claim 4, characterized in that, The positioning and imaging module further includes a coordinate filtering unit, which is used to perform temporal filtering of the three-dimensional spatial coordinates of each single molecule in the single-molecule sample based on the molecular fluorescence activity time of each single molecule in the single-molecule sample.

6. The three-dimensional single-molecule localization imaging system according to claim 4, characterized in that, The three-dimensional single-molecule image is a dynamic imaging image. Each imaging point in the three-dimensional single-molecule image corresponds one-to-one with each single molecule in the single-molecule sample, and each imaging point in the three-dimensional single-molecule image is dynamically displayed according to the corresponding molecular fluorescence activity time.

7. A three-dimensional single-molecule localization imaging method, using the three-dimensional single-molecule localization imaging system according to any one of claims 1-6, characterized in that, The method includes: An excitation beam is generated by the light source module to irradiate the single-molecule sample; The intensity of the excitation beam is modulated by an intensity modulation module to obtain a modulated excitation beam; the single-molecule sample emits fluorescence pulses under the excitation of the modulated excitation beam. The fluorescence pulses are axially position modulated using a depth modulation module to obtain encoded fluorescence pulses. The event detection module generates a photon event sequence based on the encoded fluorescence pulses; each photon event in the photon event sequence includes fluorescence spot coordinates, fluorescence trigger timestamp, and fluorescence intensity change characteristic parameters. The positioning and imaging module calculates the three-dimensional spatial coordinates of each single molecule in the single-molecule sample based on the photon event sequence, and reconstructs the three-dimensional single-molecule image corresponding to the single-molecule sample based on the three-dimensional spatial coordinates of all single molecules in the single-molecule sample.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the three-dimensional single-molecule localization imaging method of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional single-molecule localization imaging method as described in claim 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional single-molecule localization imaging method as described in claim 7.

Images