Event-based fourier-on-layers super-resolution microscopy system and method

Abstract

The application discloses an event-based Fourier ptychographic super-resolution microscopic system and method, and belongs to the technical field of optical super-resolution imaging. The system adopts central LED of normal incidence illumination and side illumination LED capable of rotating around the optical axis under different illumination polar angles, records the first positive event time delay of each pixel after each LED is turned on through an event camera, obtains a normalized reference intensity map, and collects a rotating scanning event stream generated by the side illumination LED rotating around the optical axis. The normalized reference intensity map and the rotating scanning event stream are input into a feedforward neural network after preprocessing, and high-resolution complex amplitude images of a sample are reconstructed through central illumination feature extraction, side illumination double-branch coding, angle modulation, cross-polar angle feature integration and fusion of the central illumination feature and the side illumination feature. The application can reduce the data volume and acquisition time of Fourier ptychographic microscopic imaging, and is suitable for complex amplitude imaging of biological cells and transparent thin objects.

Description

Technical Field

[0001] This invention relates to the field of computational optical super-resolution imaging technology, and in particular to an event-based Fourier stacked super-resolution microscopy imaging system and method. This method combines an event camera with continuous modulation of the illumination direction to solve the problems of slow acquisition speed and high data redundancy in traditional FPM, achieving low-cost dynamic super-resolution imaging of cell samples. This invention is applicable to applications such as biological cell imaging and phase imaging of transparent thin objects. Background Technology

[0002] Fourier Ptychographic Microscopy (FPM) is a computational imaging technique that combines synthetic aperture and phase retrieval. Traditional FPM involves placing a programmable LED array beneath the sample, sequentially illuminating LEDs at different positions to illuminate the sample from various angles. A series of low-resolution intensity images are acquired using a low numerical aperture (NA) objective and a frame-type image sensor (such as CCD or CMOS). An iterative phase retrieval algorithm then synthesizes an equivalent high numerical aperture spectrum in the Fourier domain, ultimately reconstructing a complex amplitude image of the sample with both a large field of view and high resolution. However, traditional FPM has the following drawbacks: First, it requires the sequential acquisition of a large number (usually hundreds) of intensity images, resulting in long data acquisition times and significant data redundancy. Second, traditional frame-type image sensors have limited dynamic range, making it difficult to acquire large-angle illumination signals under dark conditions, typically requiring multiple exposures for high dynamic range synthesis. Third, the information on illumination angle variations between adjacent LEDs is not fully utilized, leading to low acquisition efficiency.

[0003] An event camera is a neuromorphic visual sensor where each pixel independently senses logarithmic brightness changes. When the brightness change exceeds a preset threshold, it asynchronously outputs an event signal, offering advantages such as microsecond-level temporal resolution, high dynamic range, and low data redundancy. However, event cameras only output brightness change events and cannot directly acquire absolute intensity information. Furthermore, their output is an asynchronous sparse event stream, fundamentally different from the frame-based intensity maps output by traditional image sensors, making it difficult to directly apply to traditional FPM iterative algorithms based on intensity map constraints. Therefore, how to combine the unique advantages of event cameras with the computational imaging capabilities of Fourier layered microscopy to significantly reduce acquisition time and data redundancy while maintaining high-resolution reconstruction quality is a pressing technical problem to be solved in this field. Summary of the Invention

[0004] To address the problems in the prior art, this invention provides an event-based Fourier stacked super-resolution microscopy system and method.

[0005] The technical solution of the present invention is as follows:

[0006] In a first aspect, the present invention discloses an event-based Fourier stacked super-resolution microscopy method, comprising:

[0007] 1) Acquire samples for Fourier layered microscopy imaging, and true complex amplitude images of the samples; wherein the samples include biological cell samples and tissue section samples;

[0008] 2) Construct a physical optical model including a central LED, LEDs distributed on circumferences of different radii, an objective lens, and an event camera. The central LED illuminates the sample along the optical axis of the objective lens, and the LEDs on each circumference illuminate the sample at a fixed tilt angle relative to the optical axis. The LEDs on the circumference can rotate around the optical axis to continuously change the lateral illumination direction. The physical optical model is used to image the sample to generate normalized reference intensity maps under normal incidence conditions and under each lateral illumination condition, as well as a rotating scan event stream.

[0009] 3) Input the rotational scan event stream and the normalized baseline intensity map into the feedforward neural network. The feedforward neural network performs dual-branch encoding, angle modulation and cross-polar angle fusion to obtain the predicted complex amplitude image. Then, the feedforward neural network is trained based on the predicted complex amplitude image and the real complex amplitude image.

[0010] 4) The physical optics model is used to image the sample to be generated with complex amplitude images. The imaging results are then input into the trained feedforward neural network to obtain the complex amplitude images of the sample.

[0011] Further, in step 2), the method for generating the rotational scanning event stream includes: sequentially controlling the LEDs on each circumference to maintain constant brightness and rotate around the optical axis at a constant angular velocity, acquiring pixel events generated by brightness changes in each pixel during the LED rotation process in the event camera; simultaneously determining the illumination azimuth angle according to the angular velocity and the timestamp in the pixel event, arranging each pixel event and the corresponding illumination azimuth angle according to the timestamp, and obtaining the rotational scanning event stream under the corresponding side illumination conditions.

[0012] Furthermore, before inputting the rotating scan event stream and the normalized reference intensity map into the feedforward neural network, each rotating scan event stream is divided into multiple azimuth intervals according to the illumination azimuth angle. The number of positive events and the number of negative events in the pixel events of each pixel in each azimuth interval are counted to form an event counting tensor. Each normalized reference intensity map is then upsampled to a preset reconstruction resolution. The angle between the direction of the LED illuminating the sample on each circumference and the optical axis of the objective lens is used as the illumination polar angle of the corresponding side illumination condition.

[0013] Further, in step 3), the feedforward neural network includes a central illumination unit, a side illumination group coding unit, and a Transformer coding layer; the method for the feedforward neural network to obtain the predicted complex amplitude image includes: inputting the normalized reference intensity map under the upsampled normal incidence condition into the central illumination unit to obtain central illumination features; inputting the normalized reference intensity map under each side illumination condition and the corresponding event count tensor into the side illumination group coding unit for bi-branch coding to obtain side illumination features under the corresponding illumination polar angle; performing angle modulation on the side illumination features according to the corresponding illumination polar angle, and performing cross-polar angle fusion on the angle-modulated side illumination features through the Transformer coding layer to obtain cross-polar angle fused features; fusing and decoding the cross-polar angle fused features with the central illumination features to obtain a normalized amplitude transmittance map and a phase delay map, which constitute the predicted complex amplitude image.

[0014] In a second aspect, the present invention also provides an event-based Fourier stacked super-resolution microscopy system for implementing the method, comprising an illumination and imaging module, a data acquisition module, and a data processing module.

[0015] The illumination and imaging module includes a rotating LED illumination unit, a microscopic imaging unit, an event-based camera-based imaging data acquisition unit, and a control unit. The rotating LED illumination unit includes a central LED located at the center of the objective lens optical axis and side illumination LEDs arranged on concentric rings of different radii. The central LED provides normal incident illumination, and the side illumination LEDs illuminate the sample at a fixed tilt angle and rotate around the objective lens optical axis to change the illumination azimuth angle. The microscopic imaging unit is used to transmit the sample image to the imaging data acquisition unit. The control unit is used to control the emission and / or rotation of each LED and establish a temporal correspondence between pixel events and illumination azimuth angles.

[0016] The data acquisition module generates a normalized reference intensity map under normal incidence conditions and under each side illumination condition based on the delay of the first positive event of each pixel after each LED is turned on, and generates a rotational scan event stream based on the pixel events generated during the rotation of the side illumination LED.

[0017] The data processing module is used to process the data acquired by the data acquisition module using a feedforward neural network to generate a complex amplitude image.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] (1) In view of the problems of low acquisition efficiency and large data redundancy in the existing FPM technology, the present invention decouples the illumination polar angle and azimuth angle by rotating LED lighting design, and uses the characteristics of high temporal resolution and high dynamic range of event camera to replace the traditional frame image acquisition with a dual-modal acquisition protocol of absolute intensity reference and relative brightness change event, which compresses the acquisition of hundreds of intensity images into the acquisition of a small number of reference frames and sparse event streams, significantly reducing the amount of data and acquisition time;

[0020] (2) In view of the problem of insufficient dynamic range and loss of intermediate process information of traditional sensors, the present invention uses an event camera to capture the delay of the first positive event during the light source turn-on stage, obtains the high dynamic range reference absolute intensity under each illumination polar angle, and then continuously captures brightness change events during the rotation scanning stage, completely records the spectral offset signal caused by the change of wave vector direction, so that the information acquisition changes from discrete frame to continuous stream, and obtains complete intermediate process information.

[0021] (3) To address the problem of computationally intensive traditional iterative reconstruction, this invention constructs a physically guided pure feedforward neural network model. The central illumination branch extracts low-frequency priors, and the side illumination group coding branch extracts high-frequency information in different polar angle frequency bands. The angle embedding and feature linear modulation are combined to achieve adaptive feature extraction across illumination conditions. Then, the Transformer is integrated and the decoder is reconstructed. High-resolution complex amplitude images can be output end-to-end without any iterations during the inference stage, thus improving the reconstruction speed.

[0022] Therefore, this invention provides a new computational optical imaging approach with high speed and high information fidelity for dynamic super-resolution imaging of living cells. Attached Figure Description

[0023] Figure 1 This is an overall block diagram of the event-based Fourier stacked super-resolution microscopy system proposed in this invention;

[0024] Figure 2 This is a top view of the rotating LED lighting unit used in this invention;

[0025] Figure 3 This is a side view of the lighting and imaging module used in this invention;

[0026] Figure 4 This is a flowchart of the central lighting branch unit of the data processing module used in this invention;

[0027] Figure 5 This is a flowchart of the side illumination group coding unit of the data processing module used in this invention;

[0028] Figure 6 This is a flowchart of the side illumination feature integration unit of the data processing module used in this invention;

[0029] Figure 7 This is a flowchart of the feature fusion and complex amplitude reconstruction unit of the data processing module used in this invention;

[0030] Figure 8 This is a visualization of some data on the reconstruction results and process of HeLa cells in an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0032] Existing Fourier layered microscopy (FPM) generally faces four key bottlenecks in practical applications: First, data acquisition efficiency is low. Traditional methods require the sequential acquisition of hundreds of complete intensity images, resulting in long acquisition times and large data redundancy, making it difficult to meet the needs of live dynamic imaging. Second, traditional frame-based image sensors have limited dynamic range, making it difficult to acquire large-angle illumination signals under dark conditions. Multiple exposures are usually required for high dynamic range synthesis, further increasing the acquisition burden. Third, the continuous light intensity change information during the switching process between adjacent illumination angles is not fully utilized. Only the steady-state intensity after the switching is completed is recorded, resulting in the loss of a large amount of intermediate process information. Fourth, traditional FPM reconstruction relies on iterative phase retrieval algorithms, which require repeated projection between the Fourier domain and the spatial domain, resulting in time-consuming calculations.

[0033] Therefore, this invention provides an event-based Fourier stacked super-resolution microscopy system, which combines hardware and software to form a complete chain from illumination acquisition to image reconstruction. (See attached...) Figure 1 The system block diagram shown comprises three main parts: an illumination and imaging module S1, a data acquisition module S2, and a data processing module S3. The illumination and imaging module S1 provides programmable multi-angle illumination and optical imaging; the data acquisition module S2 executes a specific acquisition protocol to obtain a reference intensity map and a rotational scan event stream; and the data processing module S3 reconstructs a high-resolution complex amplitude image from the acquired data. Each module is described in detail below.

[0034] Illumination and Imaging Module S1

[0035] The illumination and imaging module S1 is the hardware foundation of the system and consists of four units: a rotating LED illumination unit S1.1, a microscopic imaging unit S1.2, an event camera-based imaging data acquisition unit S1.3, and a control unit S1.4.

[0036] Rotating LED lighting unit S1.1

[0037] like Figure 2 and Figure 3 As shown, the rotating LED lighting unit S1.1 includes an LED fixed at the center of the optical axis and... LEDs are distributed on concentric radii and can rotate around a center. The central LED provides positive incident illumination (illumination angle of 0°), and the LEDs on each radius correspond to different illumination angles. Illumination angle Based on geometric relationships Confirmed, among which For the first A concentric radius, This is the distance from the LED plane to the sample plane. One LED is positioned on each radius, and driven by the rotating arm, the LEDs can rotate around the optical axis at a constant angular velocity. Uniform rotation, i.e., illumination azimuth angle The linear change over time is: Each LED contains three light-emitting channels: blue, green, and red, which can emit quasi-monochromatic light with center wavelengths of approximately 475nm, 532nm, and 630nm, respectively, exhibiting standard Lambertian light emission characteristics. (See attached image.) Figure 2 A top view of the unit is provided, along with an illustration of the illumination azimuth. Meaning. Illumination azimuth angle. The definition is: within the LED plane, with the center of the optical axis as the vertex, along a specified rotation direction, from a preset reference direction (e.g., Figure 2 The angle formed by the radial line connecting the optical axis (in the horizontal direction) to the current position of the LED. The angular velocity of the LED at different radii represents the rotational speed of the LED. same.

[0038] Microscopic imaging unit S1.2

[0039] like Figure 3 As shown, the microscopic imaging unit S1.2 includes an objective lens and a tube lens arranged sequentially along the optical axis. The sample is located at the object-side focal plane of the objective lens. The objective lens and the tube lens are coaxially arranged, and the image-side focal plane of the objective lens coincides with the object-side focal plane of the tube lens. The imaging data acquisition unit based on the event camera is located at the image-side focal plane of the tube lens, thus forming an infinity-corrected 4f (four focal lengths) imaging system. The objective lens is an infinity-corrected conjugate plan apochromatic objective lens with a magnification of 2x and a numerical aperture of... The magnification is 0.08; the focal length of the tube lens is 200mm. The low magnification and low aberration of the objective lens ensure a large imaging field of view and small aberrations, providing a good optical foundation for subsequent Fourier super-resolution microscopy.

[0040] One of the key technical features of the rotating LED lighting unit S1.1 is the decoupling of the illumination polar angle and azimuth angle through a concentric ring lighting structure. When rotating on the same radius, the illumination polar angle remains constant, only the illumination azimuth angle changes. This means that the brightness variation of the pixels is mainly caused by the change in illumination direction, rather than by the attenuation of illumination intensity as the illumination polar angle increases. This provides the event camera with a physically clean spectral offset signal. The rotating scan event stream output by the event camera during this rotation primarily characterizes the pixel brightness variation caused by the change in the spectral sampling region between adjacent illumination azimuth angles, providing a spatial frequency constraint that continuously varies with the illumination azimuth angle for the reconstruction of high-resolution complex amplitude images.

[0041] The second key technical point of the rotating LED lighting unit S1.1 is that the selection of the lighting polar angle must follow the principle of uniformly covering the range of lighting angles from low to high, ensuring sufficient overlap of the spectral sampling regions corresponding to adjacent lighting polar angles in the Fourier domain (an overlap rate greater than 60% is recommended) to ensure that the Fourier stacked reconstruction can converge. The spectral overlap contains the spatial frequency information jointly collected by adjacent lighting polar angles, which can provide correlation constraints for the data corresponding to different lighting polar angles. This is beneficial for the data processing module to integrate the comprehensive features corresponding to different lighting polar angles and improve the stability and accuracy of high-resolution complex amplitude image reconstruction.

[0042] Illumination polar angle The specific selection process: Given the numerical aperture of the objective lens Starting with normal incident illumination, the radii of each concentric ring are set outwards in stages. For any candidate radius Its corresponding illumination polar angle for Its corresponding numerical aperture for illumination for In the Fourier domain, each illumination polar angle corresponds to an illumination numerical aperture. Center offset, expressed as objective lens numerical aperture The radius is the spectral sampling region. (Circular radius) With the radius of the ring The selection should ensure sufficient overlap between the corresponding spectral sampling regions, preferably with an overlap rate greater than 60%.

[0043] Imaging data acquisition unit S1.3 based on event camera

[0044] This unit is located on the back focal plane of the telescope (i.e., the system image plane, or image-side focal plane). Each pixel of the event camera operates independently, continuously monitoring the logarithmic intensity (i.e., brightness) of each pixel on the back focal plane of the telescope. The brightness of a pixel changes. When the change in brightness of a pixel exceeds a preset positive or negative threshold, an event signal is output asynchronously. , Event signal Contains the coordinates of pixels Event polarity and timestamp Event polarity The value can be +1 or -1, where +1 indicates an increase in brightness (positive event) and -1 indicates a decrease in brightness (negative event). The event camera outputs a timestamp. With microsecond-level precision, this high temporal resolution ensures both the accuracy of intensity acquisition and the sensitivity of brightness change perception.

[0045] Appendix Figure 3 The side view shown illustrates the relative positions of the rotating LED illumination unit S1.1, the microscopic imaging unit S1.2, and the event camera-based imaging data acquisition unit S1.3, and also shows the illumination polar angle. The meaning of .

[0046] Control unit S1.4

[0047] This unit is responsible for the timing control of lighting and data acquisition in the system. Specifically, it is responsible for the switching and brightness modulation of each LED, the angle control of the rotating arm, and the control of the lighting azimuth angle. Real-time location feedback and reception and caching of event camera data streams.

[0048] In a specific embodiment of the present invention, the control unit S1.4 includes a microcontroller, an LED driving module, a rotation driving module, an angle feedback module, a synchronization trigger interface, and a communication interface. The microcontroller controls the switching and brightness of each LED through the LED driving module, controls the rotation of the rotating arm through the rotation driving module, and reads the angle data output by the angle feedback module. Before scanning, the preset reference direction of the rotating arm is set to the illumination azimuth angle of 0°; during scanning, the microcontroller determines the illumination azimuth angle θ based on the angle data, and adjusts the rotation driving module according to the deviation between the actual angular velocity and the preset angular velocity to keep the rotating arm at the preset angular velocity.

[0049] Illumination azimuth Every time the angle changes by a preset interval, the microcontroller outputs a synchronization trigger signal to the event camera through the synchronization trigger interface, while simultaneously latching the current illumination azimuth angle. And generate a sequence number and illumination azimuth angle. The camera records the angle of the event. The event camera timestamps the synchronization trigger signal and writes it as an external trigger event into the event data stream.

[0050] The event camera transmits event data streams to the host computer via a high-speed USB interface, and the control unit transmits angle records to the host computer via a communication interface. The host computer, according to the order of occurrence, maps each externally triggered event to a corresponding angle record, and then determines the illumination azimuth angle based on the adjacent externally triggered events. Angle interpolation is performed on the pixel events between the two to determine the illumination azimuth angle at which each pixel event occurs. .

[0051] Data acquisition module S2

[0052] The data acquisition module S2 is responsible for executing a specific acquisition protocol to sequentially acquire the reference intensity maps and rotating scan event streams of the normal incidence and illumination from each side. This module is divided into three units according to the operation stage: the reference intensity map acquisition unit S2.1, the rotating scan event stream acquisition unit S2.2, and the acquired data integration and storage unit S2.3.

[0053] Reference strength map acquisition unit S2.1

[0054] The reference intensity map acquisition unit S2.1 is used to acquire the intensity map of the central LED and LEDs on each radius at different illumination wavelengths. Illumination polar angle and lighting azimuth angle The corresponding reference intensity map is obtained. When the reference intensity map for one LED is acquired, the remaining LEDs are turned off, and the event camera establishes the reference logarithmic light intensity of each pixel while the LEDs are off.

[0055] For a certain illumination wavelength of a certain LED When the sample is illuminated at its maximum illumination intensity, the light intensity distribution formed on the image plane is denoted as . The first item within the superscript parentheses (i.e. ) is the illumination polar angle, the second term (i.e. The azimuth angle is the illumination angle. The reference intensity map is obtained by using an event camera to measure the intensity of each pixel after the light source is turned on. Starting from zero lighting, Photoelectrons continuously accumulate under irradiation; when the accumulated amount reaches the trigger threshold, the first positive event is generated, and its time delay is recorded. . and They are inversely proportional, from which we can deduce that the illumination polar angle is... The illumination azimuth angle is Reference intensity diagram under illumination conditions .

[0056] The specific steps for obtaining the baseline intensity map for each lighting LED are as follows:

[0057] (1) Control the rotation of the rotating arm to bring the LED under test to the preset initial illumination azimuth angle and turn off the remaining LEDs. Among them, for LEDs on each radius, the preset initial illumination azimuth angle is set to 0; for the central LED located at the center of the optical axis, since its illumination azimuth angle does not change with the rotation of the rotating arm, it is used for easy uniform representation. After the LED under test is turned off and the background event output by the event camera tends to stabilize, the event camera uses the current logarithmic light intensity as the reference logarithmic light intensity of each pixel.

[0058] (2) In At any given moment, the control unit outputs a step control signal to the LED driver module, causing the driving current of the LED under test to quickly switch from 0 to the preset maximum driving current and remain unchanged within the preset acquisition time. As a result, each pixel on the system image plane begins to accumulate photoelectrons from a zero illumination state. Simultaneously, the control unit inputs a synchronization trigger signal to the event camera to mark the moment when the LED under test is turned on in the event data stream.

[0059] (3) The event camera continuously monitors the brightness changes of each pixel on the image plane of the system. For each pixel... When the accumulated photoelectrons first reach the trigger threshold of the event camera, the first positive event is generated, and the timestamp of the event is recorded. ;

[0060] (4) According to and The inverse relationship between them is used, with the maximum value of the reference intensity map itself as the normalization reference, to obtain the normalized reference intensity. :

[0061]

[0062] in, Indicates the illumination polar angle is and lighting azimuth angle Under the lighting conditions, the pixel coordinates are The normalized baseline intensity corresponding to the pixel; This represents the coordinates of all pixels within the same baseline intensity map. corresponding The maximum value, Used as a normalized reference value for this baseline intensity map.

[0063] This yields the illumination polar angle The illumination azimuth angle is Reference intensity map normalized to the [0,1] interval under the given conditions This normalization operation standardizes the intensity scale under various lighting conditions, making it easier for subsequent data processing modules to process the data uniformly.

[0064] Following the above steps, the reference intensity map acquisition unit S2.1 sequentially acquires the following reference intensity maps: the normal incident reference intensity map corresponding to the central LED (illumination polar angle is...). The illumination azimuth angle is ), and the side lighting reference intensity diagram corresponding to the LEDs on each concentric ring of side lighting (lighting polar angle is ), and The illumination azimuth angle is ), , This represents the total number of concentric rings under side illumination. The normalized reference intensity maps acquired in this unit at various illumination polar angles record the spatial frequency information of the sample after low-pass filtering by the objective lens at different illumination angles. Among them, the normalized reference intensity map for normal incidence... The sample's low-frequency spatial frequency components and normalized reference intensity maps of illumination from each side are provided. The spatial frequency components, after spectral shifting, are provided for different illumination polar angles. These baseline intensity maps collectively provide the absolute intensity constraints required for Fourier stack reconstruction in subsequent data processing modules.

[0065] Rotational scanning event stream acquisition unit S2.2

[0066] This unit performs a rotational scan on the LEDs on each concentric ring sequentially, collecting data on the changes in each pixel's position with the illumination azimuth angle as the LEDs rotate uniformly around the optical axis. The brightness change events resulting from the changes constitute a rotational scan event stream at each illumination polar angle. The central LED does not need to perform this unit operation.

[0067] For each illumination polar angle The specific steps for acquiring rotational scan events are as follows:

[0068] (1) After the baseline intensity map acquisition phase is completed, the LEDs on the ring remain on, from the initial illumination azimuth angle. Start with a constant angular velocity It rotates at a constant speed around the optical axis.

[0069] (2) During the LED rotation process, the event camera continuously operates, detecting the brightness changes of each pixel in real time. For each pixel... The event camera maintains a reference brightness value internally. Its initial value is the logarithm of the reference intensity value of the pixel. Let the current brightness value of the pixel be... Brightness increment .when When the event trigger threshold is exceeded: If If so, a positive event is generated; if Then a negative event is generated; where, The threshold for triggering positive events. This is the threshold for triggering a negative event. After the event is triggered, the event camera uses the reference brightness value of that pixel. Update to the current brightness value The updated brightness value is used as the comparison benchmark to determine whether the next event will occur for that pixel.

[0070] (3) After the LED rotates one revolution, the data acquisition of the illumination polar angle ends, and the corresponding rotation scan event stream is output. Event Stream timestamps Under the action of control unit S1.4, the azimuth angle of the illumination was maintained. The precise temporal correspondence between them. For example, for a timestamp of... The event, whose corresponding illumination azimuth angle can be determined by... Confirmed, among which The initial illumination azimuth angle is 0.

[0071] Through the above steps, this unit sequentially acquires the rotational scan event streams of all side-illumination concentric rings. Each event stream is a sparse asynchronous event sequence that records the brightness of each pixel as a function of the illumination azimuth angle at the corresponding illumination polar angle. The changing information provides the subsequent data processing module with spatial frequency constraints caused by the change in wave vector direction.

[0072] Specifically, the rotation scan event stream refers to the event data collected and arranged in chronological order during the rotation of the side-illumination LED around the optical axis. It includes the pixel coordinates, event polarity, and timestamp of each event, as well as the illumination wavelength, illumination polar angle, and illumination azimuth angle at the time of the event, which are determined by time synchronization. Among them, the event polarity indicates whether the brightness of the corresponding pixel increases or decreases, the timestamp indicates the time when the event occurs, and the illumination azimuth angle is obtained by matching the rotation angle information recorded by the control unit with the event timestamp.

[0073] Data collection, integration, and storage unit S2.3

[0074] This unit integrates and stores the outputs of the reference intensity map acquisition unit S2.1 and the rotating scan event stream acquisition unit S2.2 according to the specified conditions. After all acquisitions are completed, the data is stored in groups according to the specified polar angle, specifically including:

[0075] Normal incidence illumination data: Central (normal incidence) normalized baseline intensity map One;

[0076] Side lighting data (total) Groups): Each group contains the illumination polar angle. Corresponding normalized baseline intensity map A single and rotating scan event stream A group.

[0077] The normalized baseline intensity map is stored in the form of a two-dimensional matrix, with matrix elements being 32-bit single-precision floating-point numbers in the interval [0,1]. The rotational scan event stream is stored in the form of a structured array, and the raw pixel events output by the event camera include the pixel x-coordinate. Pixel ordinate Event polarity and event timestamp The data collection, integration, and storage unit S2.3 collects data based on event timestamps. In addition, the angle record output by the control unit S1.4 is used to determine the illumination azimuth angle at the time the pixel event occurs through time matching and angle interpolation. And record each pixel event after integration as .in, and Represents pixel coordinates, pixel coordinates and Stored as a 16-bit unsigned integer (in pixels), polarity Stored as a boolean type. Indicates a positive event. Indicates a negative event. Indicates the time when the event occurred. Stored as a 32-bit unsigned integer (in microseconds). Indicates the lighting azimuth angle corresponding to the occurrence of the event.

[0078] Data processing module S3

[0079] The data processing module is the core of the system's algorithm, responsible for reconstructing high-resolution complex amplitude images of the sample from the acquired baseline intensity map and rotating scan event stream. This module includes a data preprocessing unit S3.1, a central illumination branch unit S3.2, a side illumination grouping and coding unit S3.3, a side illumination feature integration unit S3.4, and a feature fusion and complex amplitude reconstruction unit S3.5, operating in a pure feedforward mode.

[0080] Data preprocessing unit S3.1

[0081] This unit preprocesses the acquired data output from the data integration and storage unit S2.3 to adapt it to the input format of the neural network. The preprocessing operations include discretization of the rotational scan event stream and upsampling of the normalized baseline intensity map.

[0082] Rotational scan event stream discretization: Rotational scan event stream for each side illumination ring (i.e., for the illumination polar angle) Corresponding rotational scan event stream ), and the rotation angle range of its corresponding lighting azimuth angle Divided into The azimuth angle is divided into several equally spaced intervals, and the width of each interval is... Statistical analysis of each pixel within each azimuth angle interval. The counts of positive events and the counts of negative events constitute the event count tensor. , The first dimension represents The second dimension has two channels corresponding to the cumulative number of positive and negative events within each azimuth angle interval, respectively. The third dimension... and the fourth dimension These represent the height and width of the event camera image plane, respectively.

[0083] Specifically, for the first Each side-lit LED has an installation radius corresponding to a fixed lighting angle. During the rotation of the LED on this side, the illumination azimuth angle range from 0° to 360° is divided into M azimuth angle intervals. For the m-th azimuth angle interval, the number of positive events and negative events generated by each pixel of the event camera are counted to obtain an event count map. Its dimensions are 2×H×W. The same illumination polar angle... The M event count maps below are arranged in order of illumination azimuth, forming an event count tensor. , .

[0084] Upsampling of normalized reference intensity map: For normalized reference intensity map of normal incidence and normalized reference intensity diagram of lighting on each side They were upsampled to the target reconstruction resolution using bicubic interpolation. This matches the desired resolution of the output complex amplitude image. This represents the height of the desired output complex amplitude image. This represents the width of the desired output complex amplitude image. The upsampled normalized reference intensity map of normal incidence and the normalized reference intensity maps of illumination from each side are denoted as follows: and .

[0085] After the above preprocessing, the data are grouped according to the polar angle and sent into the unit. The sampled normalized reference intensity map is shown below. Feed into the central lighting branch unit S3.2; event counting tensor and the upsampled side illumination normalized baseline intensity map Paired side lighting group coding unit S3.3.

[0086] Central lighting branch unit S3.2

[0087] This unit processes the upsampled normalized reference intensity map. This unit extracts low-frequency structural information from the sample. In classical Fourier layered microscopy, the intensity map corresponding to central normal incidence illumination is assigned zero or random phase and used as the initial spectral estimate after Fourier transform. High-frequency components are then gradually recovered through alternating projections. This strategy is effective because, under normal incidence illumination, the objective lens only allows the low-frequency spatial frequency components of the sample to pass through, and these low-frequency components typically constitute the majority of the sample's spectral energy. Based on this physical principle, this unit extracts the sampled normal incidence normalized reference intensity map. As a low-frequency information entry point for the network.

[0088] The center illumination branch unit S3.2 is implemented by a two-dimensional convolutional neural network encoder. This encoder consists of multiple residual convolutional blocks (i.e., Figure 4 The convolutional blocks are composed of two-dimensional residual convolutional blocks stacked sequentially. Each residual convolutional block contains 3×3 convolutional layers. Batch Normalization Layer and nonlinear activation functions Furthermore, by using skip connections to add the input feature map to the convolutional output, the gradient vanishing problem in deep networks is alleviated while preserving the propagation of shallow features.

[0089] Within this unit and the following units, This refers to various nonlinear activation functions, which can be independently selected. Function or other functions. A two-dimensional convolutional neural network encoder outputs a sampled, normally incident, normalized baseline intensity map. Layer-by-layer feature extraction is performed, with the spatial resolution of the output gradually decreasing and the number of channels gradually increasing at each layer, ultimately generating a central feature map. , ,in Centered feature map The number of channels, and Centered feature map Height and width. Center feature map. The low-frequency spatial frequency components of the sample after low-pass filtering by the objective lens were encoded, providing a reliable structural basis for subsequent fusion with the high-frequency features of side illumination.

[0090] Side illumination group coding unit S3.3

[0091] This unit contains A set of encoders sharing weights, each encoder set corresponding to a specific illumination polar angle. This is used to extract the spatial frequency information of the frequency band corresponding to the illumination polar angle.

[0092] In Fourier layered microscopy, different illumination polar angles correspond to different spectral offsets—the larger the illumination polar angle, the farther the sample spectrum is shifted into the objective passband region from the origin, and the higher the captured spatial frequency components. This unit follows this physical principle, setting an independent encoding channel for each illumination polar angle, but sharing encoder weights. This shared weight design allows the network to learn the general operation of extracting effective information from the corresponding intensity map and event stream given a spectral offset, rather than learning a separate set of parameters for each illumination polar angle. Differences between different illumination polar angles are addressed through angle embedding vectors. To distinguish them.

[0093] For the Each lighting polar angle The encoding process is as follows:

[0094] (1) Illumination condition coding: This refers to the illumination polar angle Other system physical parameters, independent of the samples, are encoded as fixed-dimensional angular embedding vectors. The parameters constituting the angular embedding vector include: illumination polar angle. Numerical aperture of objective lens Illumination wavelength azimuth interval number The above parameters are input into a learnable shallow multi-layer perceptron (MLP), which is then mapped to a fixed dimension. Angle Embedding Vector , The angle embedding vector encodes complete lighting condition information for that lighting polar angle.

[0095] (2) Reference frame encoding: The illumination polar angle Corresponding upsampled normalized baseline intensity map of side illumination Feature extraction is performed using a two-dimensional convolutional neural network encoder. The basic structure of this encoder is the same as the two-dimensional convolutional neural network encoder used in the center illumination branch unit S3.2 to process the upsampled normalized baseline intensity map, i.e., it also consists of multiple residual convolutional blocks stacked sequentially, with the spatial resolution of each layer's output gradually decreasing and the number of channels gradually increasing. Unlike the encoder in the center illumination branch unit S3.2, this encoder inserts a feature-wise linear modulation (FiLM) layer after the output of each residual convolutional block to accept the angle embedding vector. Modulation. FiLM works as follows: Assume the feature map output by a certain layer is... , , This represents the number of feature map channels output by the current residual convolutional block. and These represent the number of pixels in the feature map output by the current residual convolutional block in the height and width directions, respectively; the angle embedding vector... Through two learnable linear transformation matrices respectively and , and Generate a channel-by-channel scaling factor and bias , , , , ; then the feature map Each channel is scaled and offset independently to obtain the result after independent scaling and offset of each channel. , ,in Representation of feature map The One channel, ; and Scaling factors and bias The Each component. Linear transformation matrix. and Shared across all illumination polar angles, its function is to embed the angle into a vector. The mapping is adapted to modulation parameters that fit different feature levels and channels. Through this layer-by-layer injection method, the angle embedding vector... The illumination condition information is progressively integrated into the feature maps extracted by each layer of the encoder, enabling the encoder, which shares convolutional weights, to produce differentiated responses based on different illumination polar angles. The encoder output is denoted as the baseline frame feature. , .in To achieve the required number of output channels for the reference frame encoder in reference frame encoding, and This represents the output space dimension of the reference frame encoder.

[0096] (3) Event stream encoding: the event count tensor corresponding to the illumination polar angle Feature extraction is performed using a 3D convolutional neural network encoder. This differs from 2D convolution, which only operates in the spatial dimension (…). The sliding motion of the 3D convolution kernel differs in both the spatial dimension and the azimuth dimension. Simultaneous sliding on the objective lens allows for the capture of dynamic changes in the intensity of each pixel as the illumination azimuth angle changes. This design corresponds to the physical process of Fourier layer-by-layer microscopy: as the illumination azimuth angle increases from 0 to 360°, the illumination wave vector rotates around the optical axis, and circular sub-regions corresponding to different illumination azimuth angles in the sample spectrum are sequentially moved into the objective lens passband; the difference in event counts within different azimuth angle intervals encodes the rate and direction of brightness changes in each pixel during this dynamic movement. Similar to the reference frame encoder, this encoder inserts a FiLM after the output of each 3D convolutional block to accept the same angle embedding vector. The modulation. FiLM operates in the same way as FiLM in the reference frame coding process, that is, through a linear transformation matrix shared between illumination polar angles. and Embedding the angle vector The mapping is expressed as a channel-wise scaling factor and bias, independently scaling and offsetting each feature channel of the event feature map output by the current residual convolutional block. It is worth noting that the event stream coding branch and the baseline frame coding branch each have their own independent set of linear transformation matrices. and That is, event stream encoding branches use and The base frame coding branch uses and The two do not share the same linear transformation matrix. and At all lighting polar angles Shared among them. The output of the event stream encoder that implements event stream encoding is denoted as the event feature. , ,in This refers to the number of output channels of the event stream encoder. and This is the output space size of the event stream encoder. The differential information of brightness variation along the illumination azimuth direction at this illumination polar angle is encoded. (Compared with the features of the reference frame.) The provided static spatial frequency information is complementary, and the event characteristics reflect the spectral overlap constraints between different azimuth angles, providing additional information for subsequent reconstruction.

[0097] (4) Feature fusion: Baseline frame features and event characteristics The same illumination polar angle was encoded separately. The system incorporates static intensity information and dynamic event information that varies with the illumination azimuth angle. These two types of information are physically complementary; the former corresponds to the absolute intensity constraint at that illumination angle in classical FPM, while the latter corresponds to the relative change constraint between adjacent illumination azimuth angles caused by spectral shifting. To integrate these two types of information into a unified representation, this step performs feature fusion. First, through spatial dimension alignment operations (such as zero-padding if necessary), the features of the reference frame are aligned... and event characteristics The spatial dimensions are consistent. The aligned height and width are denoted as follows: and Then, the two aligned features are concatenated along the channel dimension to obtain a feature with [number of channels]. intermediate features, The intermediate features are then fed into a 1×1 convolutional layer and a non-linear activation function. The fusion layer, composed of these layers, generates a comprehensive feature for the illumination polar angle by linearly combining and compressing information between channels using learnable 1×1 convolutional kernels. , The weights of the 1×1 convolutional kernels in the fusion layer are shared across all illumination polarities. This means that the network learns a general fusion strategy that optimally combines static intensity information and dynamic event information, rather than learning a separate set of fusion parameters for each illumination polarity.

[0098] Side lighting feature integration unit S3.4

[0099] This unit will use the S comprehensive features output by the side illumination group coding unit S3.3. Integrate into a single global side illumination feature map In Fourier layered microscopy, the spectral sub-regions corresponding to different illumination polar angles are arranged in an ordered manner in the Fourier domain according to the size of the polar angle: the smaller the polar angle, the closer the center of the spectral sub-region is to the origin, carrying low-frequency spatial frequency components; the larger the polar angle, the farther the offset, carrying high-frequency components. There is approximately 60% overlap between the frequency bands of adjacent illumination polar angles; this redundancy is the physical basis for the convergence of classical FPM iterative reconstruction. This unit follows this physical structure, arranging the comprehensive characteristics of each illumination polar angle according to the illumination polar angle... The features are arranged in ascending order so that the features of adjacent positions exactly correspond to adjacent frequency bands with overlapping physical spectra, and then integrated.

[0100] Specifically, this unit is implemented according to the following steps:

[0101] First, the comprehensive characteristics corresponding to each illumination polar angle are expanded in the spatial dimension. Let: Then the first Each lighting polar angle The corresponding comprehensive features can be expressed as: ,in Indicates the first The first illumination polar angle Feature vectors at spatial locations , For spatial location index.

[0102] For the same spatial location Arrange the feature vectors corresponding to all illumination polar angles in ascending order of illumination polar angle to form a cross-polar angle feature sequence. , Among them, the first in the sequence Each element corresponds to the illumination polar angle. This sequence can represent multi-band side illumination information from low to high illumination polar angles at the same spatial location. To enable the subsequent Transformer encoder to distinguish the illumination polar angles to which different feature vectors belong, this unit sets a learnable polar angle position embedding matrix. , ,in For the first The polar position embedding vector corresponding to each illumination polar angle. The polar position is embedded in the matrix. For the learnable parameters of the network, in all spatial locations The parameters are shared and updated along with the parameters of the Transformer encoder during training. For the... The feature vector corresponding to each illumination polar angle Embed it with the corresponding polar position vector Adding them together yields a feature vector containing polar position information. , Therefore, we obtain the first... Transpolar feature sequences corresponding to spatial locations with polar angle location information , .

[0103] Subsequently, the cross-polar angle feature sequence Input a Transformer encoder. This Transformer encoder is used in all spatial locations. Shared parameters. The Transformer encoder is composed of... The coding layers with the same structure are stacked sequentially. Each coding layer includes a multi-head self-attention sublayer, residual connections, a layer normalization sublayer, and a feedforward network sublayer. Let the first layer be... The input of the layer coding layer is , The output is , ,in ,and ; For the first Layer coding layer in the first The output transpolar angle feature sequence at the nth spatial location, i.e. the nth Layer coding layer in the first Input transpolar angle feature sequences at spatial locations; when hour, That is, the cross-polar angle feature sequence For the first The multi-head self-attention sub-layer processes the input cross-polar angle feature sequence through a learnable linear transformation as follows: First, the multi-head self-attention sub-layer transforms the input cross-polar angle feature sequence through a learnable linear transformation. Mapped to query matrix Key matrix Sum matrix ,Right now: , , ,in , and The first The learnable weight matrix of the layer is then calculated. Next, the attention weights between the integrated features corresponding to different illumination polar angles are calculated, and the self-attention output is obtained. , ,in, This represents the feature dimension of a single attention head. Multi-head self-attention is used to calculate the correlation between the integrated features corresponding to any two illumination polar angles at the same spatial location, enabling the network to adaptively integrate side illumination information from different frequency bands based on the overlap between spectral sub-regions corresponding to different illumination polar angles. Self-attention output. With the Layer input sequence Perform residual connections and layer normalization to obtain intermediate feature sequences. , ,in The representation layer is normalized. Then, the intermediate feature sequences are... Input feedforward network The feedforward network consists of two fully connected layers and a nonlinear activation function located between them. This is composed of components used to further map the feature vectors at each illumination polar angle location. Its output is compared with the intermediate feature sequence. Perform residual connections and layer normalization again to obtain the first... The output polar angle feature sequence of the layer : .go through After sequential stacking of the coding layers, the first layer is obtained. Transpolar integrated feature sequences at various spatial locations , ,in Indicates the first At the first spatial location Each lighting polar angle passes through The feature vectors of cross-polar angle information interaction are then used in this unit. Next, this unit integrates the cross-polar angle feature sequences. Average pooling is performed along the illumination polar angle dimension to obtain the first... Side lighting blending features at each spatial location , No. Side lighting blending features at each spatial location The expression is:

[0104]

[0105] Finally, the side illumination fused features at all spatial locations are rearranged according to their original spatial locations to restore them to a two-dimensional feature map form, resulting in the global side illumination feature map. , This global side illumination feature map preserves the structural information of the sample in the spatial dimension and integrates multi-band spatial frequency information corresponding to different side illumination angles in the illumination polar angle dimension. As a comprehensive representation of the sample's high-frequency spatial frequency information, the central feature map is fed into the output of the subsequent feature fusion and complex amplitude reconstruction unit S3.5 and the central illumination branch unit S3.2. The images are fused to reconstruct high-resolution complex amplitude images of the sample.

[0106] Feature fusion and complex amplitude reconstruction element S3.5

[0107] This unit is used to output the center feature map from the center lighting branch unit S3.2. Global side illumination feature map output by side illumination feature integration unit S3.4 The samples were fused and reconstructed into high-resolution complex amplitude images.

[0108] Center feature map output by center lighting branch unit S3.2 It mainly includes low-frequency spatial structure information obtained under normal incident illumination conditions, and a global side illumination feature map output by the side illumination feature integration unit S3.4. It mainly contains high-frequency spatial information corresponding to multiple side illumination polar angles. Since the two originate from different coding branches, their spatial dimensions may be different. Therefore, this unit first performs feature alignment on the two.

[0109] First, the central feature map Perform spatial dimension alignment. The center feature map is then aligned using bilinear interpolation. Upsampling or downsampling to the global side illumination feature map For the same spatial dimensions, obtain the center feature map after spatial dimension alignment. , .

[0110] Then With global side illumination feature map Concat the features along the feature channel dimension to obtain the fused input feature map. , The fused input feature map contains both low-frequency structural information provided by the center illumination and high-frequency detail information provided by the side illumination.

[0111] Subsequently, the input feature maps will be fused. Input the fused convolutional layer. The fused convolutional layer consists of a 3×3 convolutional layer. Batch Normalization Layer and nonlinear activation functions The composition is used to fuse central illumination features and side illumination features within a local spatial neighborhood to obtain an initial reconstructed feature map. And used as input to the decoder, .

[0112] The decoder in this unit consists of multiple residual decoding blocks stacked sequentially. Each residual decoding block includes two 3×3 convolutional layers, a batch normalization layer, a non-linear activation function, and residual connections. For example, for the... Each residual decoding block has the following input: The output is :

[0113]

[0114] in, , This represents the number of residual decoding blocks. For the first Layer-by-layer fused feature maps output by each residual decoding block.

[0115] By using multiple residual decoding blocks, the decoder can gradually recover high-frequency detail information provided by the side illumination features while maintaining the stability of the low-frequency structure. After processing by the residual decoding blocks, a deep fusion feature map is obtained. .

[0116] Subsequently, the feature maps will be deeply fused. The sampled data is fed into the upsampling decoding module to restore its spatial dimensions to the target high-resolution reconstruction size. The upsampling decoding module consists of several upsampling blocks, each including a bilinear upsampling layer, a 3×3 convolutional layer, and a nonlinear activation function. After step-by-step upsampling, a high-resolution reconstructed feature map is obtained. , , The number of channels for high-resolution reconstructed feature maps.

[0117] Finally, the high-resolution reconstructed feature map The data is fed into the output convolutional layer. The output convolutional layer is a 3×3 convolutional layer with 2 output channels. It reconstructs the feature map at high resolution. Convert to a prediction tensor with two channels , , Prediction tensor The two channels correspond to the amplitude prediction value and phase prediction value of the sample, respectively, along the prediction tensor. The channel dimension will predict the tensor. The first channel is denoted as the amplitude prediction value graph. The second channel is denoted as the phase prediction value map. Amplitude prediction graph The Sigmoid function is applied to each pixel value in the image to obtain the normalized amplitude transmittance map. , ; Phase prediction value map Each pixel value in the array is then subjected to a hyperbolic tangent function and multiplied by . The phase delay map is obtained. , .

[0118] Based on the normalized amplitude transmittance diagram and phase delay diagram Construct high-resolution complex amplitude images of the sample , ,in, The imaginary unit; For the sample in pixel coordinates Normalized amplitude transmittance at the location; For the sample in pixel coordinates Phase delay at that point.

[0119] The complex amplitude image simultaneously contains high-resolution amplitude and phase information of the sample. The low-frequency structural features provided by the central illumination branch unit S3.2 ensure the overall contour and low-frequency stability of the reconstruction result, while the multi-angle high-frequency features provided by the side illumination grouping encoding unit S3.3 enhance the detail resolution of the reconstruction result. Through the aforementioned data processing module S3, this system can reconstruct the high-resolution complex amplitude image of the sample end-to-end from the reference intensity map acquired by the event camera and the rotating scan event stream during the inference stage, without the iterative optimization process in the classic FPM reconstruction algorithm.

[0120] The implementation process of the present invention is illustrated below through specific embodiments:

[0121] Application Example: Quantitative Phase Imaging of Biological Cells

[0122] The following uses quantitative phase imaging of HeLa cells as an example to illustrate the complete workflow of the above-mentioned event-based Fourier super-resolution microscopy method, including the training method of the neural network model and the actual imaging effect.

[0123] (a) Construction of the training dataset

[0124] In this embodiment, the training data for the neural network model is generated through simulation. The specific construction method is as follows: First, a set of sample images with known complex amplitude ground truth values ​​is acquired or generated as training labels. These sample images can be real cell images selected from publicly available biological image datasets, their phase information extracted as phase ground truth values, and corresponding amplitude ground truth values ​​generated. Alternatively, complex amplitude images with real cell morphological features can be synthesized using methods such as generative adversarial networks.

[0125] Secondly, the data acquisition process of the illumination and imaging module S1 is simulated using a physical optics model. For each training label image, corresponding simulated acquisition data is generated according to the acquisition protocol described in the data acquisition module S2. The generation process includes a baseline intensity map acquisition stage and a rotational scan event stream acquisition stage. During the simulation, the actual physical parameters of the system (illumination wavelength) are used to simulate the data acquisition process. Numerical aperture of objective lens Illumination polar sequence azimuth interval number Rigorous optical propagation calculations are performed on events (such as trigger thresholds) to make the simulation data as close as possible to the real experimental conditions.

[0126] Using the above method, a set of training inputs is generated for each training label, including: a normalized baseline intensity map of normal incidence, Group side lighting normalized reference intensity map and Grouped rotational scan event streams. Each training sample is labeled with its true complex amplitude image (containing both amplitude and phase channels). By changing the sample type, imaging parameters, and noise conditions, diverse datasets containing tens of thousands to hundreds of thousands of training samples can be constructed.

[0127] (II) Loss Function and Training Process

[0128] In this embodiment, the training of the neural network model adopts a combination of supervised learning and physical constraints, and the loss function consists of the following four components:

[0129] Monitoring losses At the same illumination wavelength Below, the high-resolution complex amplitude image predicted by the network is used to measure its performance. Compared with true high-resolution complex amplitude images Pixel-wise mean square error, supervised loss Defined as the main monitoring signal in this embodiment.

[0130] Reference frame consistency loss : High-resolution complex amplitude images obtained from network prediction The input data is fed into a differentiable physical model (the differentiable physical model is derived from ZHENG G, SHEN C, JIANG S, et al. Concept, implementations and applications of Fourier ptychography[J / OL]. Nature Reviews Physics, 2021, 3(3): 207-223. DOI:10.1038 / s42254-021-00280-y.). The input data for this differentiable physical model includes: high-resolution complex amplitude images predicted by the network. Illumination wavelength Numerical aperture of objective lens The illumination wave vector when the illumination light is incident along the optical axis of the system. Objective lens pupil function (i.e., the range of spatial frequency components that the objective lens transmits) , and These represent the spatial frequency coordinates in two mutually perpendicular directions. Among them, the high-resolution complex amplitude image... The output of the network to be constrained is shown, while the remaining parameters are pre-calibrated or known physical parameters of the system. The differentiable physical model is based on the forward imaging process of Fourier layered microscopy for high-resolution complex amplitude images. The process involves sequentially performing a Fourier transform, spectral shift based on the normal incident illumination wave vector, multiplication by the objective pupil function, inverse Fourier transform, and modulus squaring to output a predicted normalized reference intensity map under normal incident illumination conditions. , where superscript This represents the normal incidence illumination condition with an illumination polar angle of 0 and an illumination azimuth angle of 0. Subsequently, the predicted normal incidence normalized reference intensity map under the normal incidence condition is calculated. Normalized reference intensity map of normal incidence compared with actual sampling The per-pixel mean square error between the two frames yields the baseline frame consistency loss. This loss term is used to ensure that the complex amplitude image output by the network, after being generated into predicted intensity by the forward imaging model, remains consistent with the actual acquired normal incidence reference intensity map.

[0131] Event consistency loss : High-resolution complex amplitude images obtained from network prediction The input data is fed into a differentiable physical model. This differentiable physical model's input data includes: high-resolution complex amplitude images predicted by the network. Illumination wavelength Numerical aperture of objective lens Each lighting polar angle Initial illumination azimuth angles corresponding to each azimuth angle interval and objective pupil function Among them, high-resolution complex amplitude images The output of the network to be constrained is shown, while the remaining parameters are pre-calibrated or known physical parameters of the system. For an illumination polar angle of... The initial illumination azimuth angle is The illumination conditions are such that the corresponding illumination wave vector is: , The above differentiable physical model outputs predicted intensity maps under various illumination polar angles and initial illumination azimuth angles. Subsequently, the predicted intensity map was analyzed. Take the logarithm and calculate the predicted logarithmic intensity variation matrix between adjacent initial illumination azimuth angles. :

[0132]

[0133] Event polarity marking The event count tensor obtained by the data preprocessing unit S3.1 Confirmed. Specifically, Indicates the first The first illumination polar angle, the first Pixels within each azimuth angle range The cumulative number of positive events, Indicates the first The first illumination polar angle, the first Pixels within each azimuth angle range The cumulative number of negative events. When season ; ,make When both are equal or both are zero, the pixel does not participate in the event consistency loss calculation within that azimuth angle range. The event consistency loss is calculated using the following hinge loss function. :

[0134]

[0135] in, This represents the set of all valid indexes involved in calculating the consistency loss of an event, i.e., those that satisfy... of combination, This indicates the number of elements in the set. The preset interval threshold; Indicates the illumination polar angle is The initial illumination azimuth angle is At that time, pixel coordinates The predicted logarithmic intensity change value at that location.

[0136] This loss term is used to constrain the direction of brightness change between adjacent illumination azimuth angles of the high-resolution complex amplitude image output by the network to be consistent with the event polarity recorded in the event count tensor after the predicted intensity is generated by the forward imaging model.

[0137] Total variational regularization loss It is defined as the high-resolution complex amplitude image obtained by network prediction. Total variational regularization (TVGF) is the sum of the absolute values ​​of the differences between adjacent pixels in the horizontal and vertical directions. It measures the total variation between adjacent pixels in an image. Adding TVGF to the loss function tends to suppress random noise and unwanted detail oscillations in the image while preserving realistic structures with large gradient changes, such as cell edges, making the reconstructed image smoother and more physically consistent.

[0138] Total loss function The weighted sum of the above four items:

[0139]

[0140] in, , , These are the weighting coefficients for the baseline frame consistency loss, event consistency loss, and total variational regularization loss, respectively, which are determined based on experimental tuning.

[0141] During training, each training iteration randomly selects a batch of training samples from the training set. The input data is then fed into the data processing module S3 for forward propagation to obtain the predicted high-resolution complex amplitude image. The total loss is then calculated, and the learnable parameters of each unit in the network are updated via gradient backpropagation. Training iterations are repeated until the loss function converges, and the trained network parameters are saved for subsequent inference.

[0142] (III) Actual Work Process

[0143] The following uses quantitative phase imaging of HeLa cells as an example to illustrate the complete workflow of this system from data acquisition to image reconstruction.

[0144] Data acquisition phase: The hardware system is built according to the illumination and imaging module S1. The prepared HeLa cell samples are placed on the system's sample stage, and the sample position is adjusted so that the region of interest is centered in the field of view. Data acquisition is completed sequentially according to the acquisition protocol described in the data acquisition module S2: First, a reference intensity map is acquired from the central LED to obtain a normalized reference intensity map of normal incidence; then, data is acquired sequentially according to the illumination polar angle from smallest to largest. Data acquisition is performed on the LEDs on each side of the concentric ring of illumination. For each illumination polarity... First, rotate the LED to the initial azimuth angle. The reference intensity map acquisition unit S2.1 acquires the normalized reference intensity map of the side illumination at each illumination polar angle; then, a rotational scan event stream acquisition is performed to make the LED move at a constant angular velocity. The event camera records the rotational scan event stream as the device rotates around the optical axis for one revolution. The total data volume of the entire acquisition process is approximately several MB, and the acquisition time is within a few seconds.

[0145] Data preprocessing stage: The acquired data is sent to the data preprocessing unit S3.1 of the data processing module S3. For each group of rotational scan event streams... The event count tensor is discretized into an event count tensor according to the preset number of azimuth angle intervals M (set to 32 in this embodiment). For each normalized reference intensity map, it is upsampled to the target reconstruction resolution by bicubic interpolation to obtain the upsampled normalized reference intensity map of normal incidence and normalized reference intensity maps of each side illumination.

[0146] Network inference stage: The preprocessed data is fed into the trained neural network model. Following the data processing flow from the central illumination branch unit S3.2, the side illumination grouping and coding unit S3.3, the side illumination feature integration unit S3.4 to the feature fusion and complex amplitude reconstruction unit S3.5, the following operations are performed in a pure feedforward manner: First, the central feature map is extracted from the upsampled normalized reference intensity map. Then for each illumination polar angle Reference frame features are extracted from the upsampled side illumination normalized reference intensity map and the event count tensor, respectively. and event characteristics And by feature fusion, comprehensive features for each illumination polar angle are generated. Then The comprehensive characteristics of each illumination polar angle are based on the illumination polar angle. The features are integrated into a global side illumination feature map in ascending order. Finally, the central feature map With global side illumination feature map The images were fused and then decoded to reconstruct high-resolution complex amplitude images of the HeLa cell samples.

[0147] In this embodiment, the original size of the input low-resolution image is . , The numerical aperture of the objective lens used is The output high-resolution image size is , The reconstructed numerical aperture is , ,in This is the maximum illumination polar angle.

[0148] (iv) Expected Results

[0149] Figure 8 Figure (a) shows the centrally normalized baseline intensity map acquired by the present invention, which presents the imaging results of the low numerical aperture microscope objective under positive illumination and reflects the low frequency intensity information of the sample. Figure 8 Figures (b) and (d) show the illumination intensity maps of the first and S sides, respectively, presenting the imaging results of a low numerical aperture microscope objective under small-angle and large-angle side illumination. They reflect the mid-frequency and high-frequency intensity information of the sample, respectively. Figure 8 Figure (b) shows the imaging results of a low numerical aperture microscope objective under small illumination polar angle conditions, mainly reflecting the mid-frequency intensity information of the sample; Figure 8 Figure (d) in the figure presents the imaging results of a low numerical aperture microscope objective under large illumination polar angle conditions, mainly reflecting the high frequency intensity information of the sample. Figure 8 Figures (c) and (e) show a selected visualization of the event count tensors for events 1 and S within the fourth azimuth angle interval, reflecting the brightness changes of each pixel as the side-illuminated LED rotates within this azimuth angle interval at the corresponding illumination polarity. Red pixels represent positive event counts exceeding negative event counts, indicating an increase in brightness; blue pixels represent negative event counts exceeding positive event counts, indicating a decrease in brightness. Figure 8Figures (c) and (e) show that the events generated under different illumination polar angles differ in spatial distribution and quantity, indicating that the rotating scan event stream can record the changes in sample imaging brightness caused by changes in illumination azimuth angle.

[0150] Figure 8 Figure (f) shows the high-resolution normalized amplitude transmittance map of HeLa cells reconstructed in this invention, compared to... Figure 8 Figures (a), (b), and (d) show low-resolution normalized baseline intensity maps under single illumination conditions. Figure 8 The cell outlines and boundaries between adjacent cells in Figure (f) are clearer, and the texture and local details inside the cells are richer, indicating that the present invention can fuse the low-frequency structural information provided by the central positive incident illumination, the mid-to-high frequency intensity information provided by the polar angles of different side illuminations, and the brightness change information provided by the rotating scan event stream, thereby improving the spatial resolution of the sample amplitude image.

[0151] Figure 8 Figure (g) shows the high-resolution phase delay map of HeLa cells reconstructed by the present invention. Figure 8 Figures (a), (b), and (d) in the original text only show intensity imaging results obtained using a low numerical aperture microscope objective, which cannot directly characterize the phase delay information of transparent samples. This invention uses a feedforward neural network to jointly process the normalized baseline intensity map and the event count tensor to recover the phase distribution of HeLa cells, thus displaying the cell morphology and internal phase differences. Figure 8 As shown in Figures (f) and (g), the present invention is able to reconstruct a complex amplitude image of a sample that simultaneously contains high-resolution amplitude information and phase information.

[0152] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An event-based Fourier stacked super-resolution microscopy method, characterized in that, include: 1) Acquire the sample for Fourier layered microscopy imaging, and the true complex amplitude image of the sample; 2) Construct a physical optical model including a central LED, LEDs distributed on circumferences of different radii, an objective lens, and an event camera. The central LED illuminates the sample along the optical axis of the objective lens, and the LEDs on each circumference illuminate the sample at a fixed tilt angle relative to the optical axis. The LEDs on the circumference can rotate around the optical axis to continuously change the lateral illumination direction. The physical optical model is used to image the sample to obtain normalized reference intensity maps under normal incidence conditions and under each lateral illumination condition, as well as a rotating scan event flow. 3) Input the rotational scan event stream and the normalized baseline intensity map into the feedforward neural network. The feedforward neural network performs dual-branch encoding, angle modulation and cross-polar angle fusion to obtain the predicted complex amplitude image. Then, the feedforward neural network is trained based on the predicted complex amplitude image and the real complex amplitude image. 4) The physical optics model is used to image the sample to be generated with complex amplitude images. The imaging results are then input into the trained feedforward neural network to obtain the complex amplitude images of the sample.

2. The event-based Fourier stacked super-resolution microscopy method according to claim 1, characterized in that, In step 2), the method for generating the normalized reference intensity map includes: sequentially switching each LED from the off state to the maximum illumination intensity while keeping the luminous brightness unchanged; collecting and imaging the light field modulated by the sample under the current LED illumination by the objective lens; having the event camera record the time delay of the first positive event generated by each pixel from the moment the current LED is turned on on the sample image surface formed by the objective lens; calculating the reciprocal of each time delay; using the maximum value of the reciprocal of the time delay of each pixel under the LED illumination as the normalization reference value; normalizing the reciprocal of each time delay and arranging them according to the pixel position to obtain the normalized reference intensity map corresponding to the LED.

3. The event-based Fourier stacked super-resolution microscopy method according to claim 1, characterized in that, In step 2), the method for generating the rotating scan event stream includes: sequentially controlling the LEDs on each circumference to maintain constant brightness and rotate around the optical axis at a constant angular velocity, acquiring pixel events generated by brightness changes in each pixel during the LED rotation process in the event camera; simultaneously determining the illumination azimuth angle according to the angular velocity and the timestamp in the pixel event, arranging each pixel event and the corresponding illumination azimuth angle according to the timestamp, and obtaining the rotating scan event stream under the corresponding side illumination conditions.

4. The event-based Fourier stacked super-resolution microscopy method according to claim 3, characterized in that, Before inputting the rotating scan event stream and the normalized reference intensity map into the feedforward neural network, each rotating scan event stream is divided into multiple azimuth intervals according to the illumination azimuth angle. The number of positive events and the number of negative events in the pixel events of each pixel in each azimuth interval are counted to form an event counting tensor. Each normalized reference intensity map is then upsampled to a preset reconstruction resolution. The angle between the direction of the LED illuminating the sample on each circumference and the optical axis of the objective lens is used as the illumination polar angle of the corresponding side illumination condition.

5. The event-based Fourier stacked super-resolution microscopy method according to claim 4, characterized in that, In step 3), the feedforward neural network includes a central illumination unit, a side illumination group coding unit, and a Transformer coding layer; The method for obtaining the predicted complex amplitude image by the feedforward neural network includes: inputting the normalized reference intensity map under the upsampled normal incidence condition into the center illumination unit to obtain the center illumination feature; inputting the normalized reference intensity map under each side illumination condition and the corresponding event count tensor into the side illumination grouping encoding unit for bi-branch encoding to obtain the side illumination feature under the corresponding illumination polar angle; performing angle modulation on the side illumination feature according to the corresponding illumination polar angle, and performing cross-polar angle fusion on the angle-modulated side illumination feature through the Transformer encoding layer to obtain the cross-polar angle fused feature; fusing and decoding the cross-polar angle fused feature with the center illumination feature to obtain the normalized amplitude transmittance map and phase delay map, which constitute the predicted complex amplitude image.

6. The event-based Fourier stacked super-resolution microscopy method according to claim 5, characterized in that, The side illumination group coding unit includes a reference frame encoder and an event stream encoder; both the reference frame encoder and the event stream encoder include multiple residual convolutional blocks connected in sequence, and each residual convolutional block has a feature linear modulation layer at its output end; the reference frame encoder and the event stream encoder respectively output the reference frame feature and the event stream feature modulated by the illumination polar angle, and the two are fused to form the side illumination feature under the corresponding illumination polar angle; The angle modulation includes converting the illumination polar angle into an angle embedding vector, and generating scaling and offset parameters for each feature channel in the side illumination features based on the angle embedding vector through the feature linear modulation layer of the feedforward neural network; the cross-polar angle fusion includes performing feature fusion based on the correlation between the side illumination features after angle modulation under different illumination polar angles through the Transformer encoding layer.

7. The event-based Fourier stacked super-resolution microscopy method according to claim 4, characterized in that, In step 3), the loss function used to train the feedforward neural network includes supervisory loss, reference frame consistency loss, event consistency loss, and total variational regularization loss. The supervisory loss is determined based on the pixel-wise mean square error between the predicted complex amplitude image and the real complex amplitude image. The reference frame consistency loss is determined based on the pixel-wise mean square error between the predicted intensity map of the predicted complex amplitude image generated by the physical optics model and the normalized reference intensity map under the normal incidence condition. The event consistency loss is determined based on the consistency between the logarithmic intensity change of the predicted intensity map under adjacent illumination azimuth angles generated by the physical optics model and the event polarity represented by the event count tensor, and the hinge loss is used to penalize pixels whose change directions are inconsistent. The total variational regularization loss is determined based on the sum of the absolute values ​​of the differences between adjacent pixels in the horizontal and vertical directions in the predicted complex amplitude image.

8. An event-based Fourier stacked super-resolution microscopy system for implementing the method of claim 1, characterized in that, It includes a lighting and imaging module, a data acquisition module, and a data processing module; The illumination and imaging module includes a rotating LED illumination unit, a microscopic imaging unit, an event-based camera-based imaging data acquisition unit, and a control unit. The rotating LED illumination unit includes a central LED located at the center of the objective lens optical axis and side illumination LEDs arranged on concentric rings of different radii. The central LED provides normal incident illumination, and the side illumination LEDs illuminate the sample at a fixed tilt angle and rotate around the objective lens optical axis to change the illumination azimuth angle. The microscopic imaging unit is used to transmit the sample image to the imaging data acquisition unit. The control unit is used to control the emission and / or rotation of each LED and establish a temporal correspondence between pixel events and illumination azimuth angles. The data acquisition module generates a normalized reference intensity map under normal incidence conditions and under each side illumination condition based on the delay of the first positive event of each pixel after each LED is turned on, and generates a rotational scan event stream based on the pixel events generated during the rotation of the side illumination LED. The data processing module is used to process the data acquired by the data acquisition module using a feedforward neural network to generate a complex amplitude image.

9. The event-based Fourier stacked super-resolution microscopy system according to claim 8, characterized in that, The microscopic imaging unit includes an objective lens and a tube lens, and the event camera is disposed on the back focal plane of the tube lens; LEDs on different radii correspond to different side illumination polar angles, and the Fourier spectrum sampling regions corresponding to adjacent side illumination polar angles have an overlap rate of more than 60%.

10. The event-based Fourier stacked super-resolution microscopy system according to claim 8, characterized in that, The control unit includes a microcontroller, an LED driver module, a rotation driver module, an angle feedback module, a synchronization trigger interface, and a communication interface. The microcontroller controls the light emission state of each LED through the LED driver module, controls the LEDs on each circumference to rotate around the optical axis through the rotation driver module, and determines the illumination azimuth angle based on the angle information collected by the angle feedback module. The microcontroller sends a synchronization trigger signal to the event camera through the synchronization trigger interface, so that pixel events are associated with the illumination azimuth angle according to a unified time reference.

Images