SLO, SS-OCT and ERG synchronous ophthalmic multi-mode imaging system

Abstract

The invention discloses an SLO, SS-OCT and ERG synchronous ophthalmology multimodal imaging system, and belongs to the field of medical imaging equipment. The system comprises an SLO module, an SS-OCT module, an ERG module, an FPGA main control board and an image processing unit. The FPGA main control board generates a unified hardware clock and a global synchronous trigger signal, controls the three modules to work synchronously based on the same time reference, and adds a unified timestamp to all collected data. And the image processing unit realizes spatial positioning of SS-OCT fault data in an SLO image coordinate system and time sequence alignment of an ERG electrophysiological signal and an SS-OCT scanning position on the basis of the timestamp and a coordinate mapping relation between the SLO and the SS-OCT. According to the method, through hardware-level synchronization and accurate registration, the problem of space-time non-uniformity of multi-modal data is solved, point-to-point fusion of retina structure and function information is realized, and the early diagnosis and evaluation capability of ophthalmic diseases is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging equipment technology, specifically relating to an ophthalmic multimodal imaging system that synchronizes SLO, SS-OCT and ERG. Background Technology

[0002] In ophthalmic clinical diagnosis and scientific research, obtaining high-resolution structural information of the retina and its corresponding functional status is crucial for early disease detection, disease assessment, and mechanistic research. Currently, commonly used ophthalmic imaging and functional testing equipment mainly includes: Scanning laser ophthalmoscope (SLO): It can achieve high-speed, high-resolution two-dimensional fundus reflection imaging, with a wide field of view, and is suitable for observing the surface structure of the retina and the morphology of blood vessels.

[0003] Sweep-frequency optical coherence tomography (SS-OCT): This technique uses swept-frequency laser technology to achieve high-speed, high-sensitivity three-dimensional tomographic imaging, which can obtain microscopic structural information of each layer of the retina and is suitable for observing interlaminar retinal lesions.

[0004] Electroretinography (ERG): By recording the electrophysiological signals generated by the retina under light stimulation, it reflects the functional status of the retina and is particularly suitable for assessing the functional integrity of retinal nerve cells and photoreceptors.

[0005] However, the above three types of devices typically operate independently in traditional applications, which leads to the following significant problems: (1) Timing asynchrony: Each system adopts independent triggering and control timing, which relies on software timing coordination. There are trigger delays and time jitters, which cause the acquired structural images and functional signals to be unable to be strictly aligned in time.

[0006] (2) Spatial misregistration: Since each modal scanning mechanism and coordinate system is independent, even data collected in similar time periods are difficult to achieve accurate spatial correspondence due to factors such as scanning trajectory, field of view, and patient eye movement.

[0007] (3) Difficulty in data fusion: The separation of structural imaging and functional testing data makes it difficult for clinicians or researchers to accurately locate functional abnormalities in specific anatomical structures, affecting the accuracy of disease diagnosis and the depth of research.

[0008] To overcome the aforementioned problems, some existing technologies have developed systems that integrate the two modalities, such as combining SLO with OCT, or attempting to integrate ERG functionality into an OCT system. However, these integration solutions mostly employ time-sharing multiplexing, i.e., alternating operation or software registration after acquisition. Time-sharing multiplexing is inherently asynchronous and cannot capture structural and functional information at the same instant; while software post-processing registration is limited by the aforementioned temporal jitter and spatial coordinate differences, resulting in large registration errors and poor real-time performance. It cannot provide structural-functional fusion results in real time during the examination, affecting the convenience and efficiency of clinical diagnosis.

[0009] Furthermore, in existing integration solutions, there is a trade-off between the degree of system integration and the core performance of each modality. In the pursuit of physical integration, system design is often constrained by spatial, optical, or electrical interference, forcing compromises on key performance indicators such as the ultra-wide field of view of SLO, the high-speed and high-resolution scanning capability of SS-OCT, or the high-sensitivity acquisition of ERG. This results in the overall performance of the integrated system being a compromise between the top-level performance of each independent modality, making it difficult to maintain the optimal operating state of each modality while achieving a high degree of integration.

[0010] In summary, the existing technology still lacks an integrated ophthalmic multimodal imaging system that can achieve strict synchronous triggering and data acquisition of SLO, SS-OCT and ERG at the hardware level, achieve high integration while ensuring independent high performance of each modality, and complete high-precision spatiotemporal registration and fusion visualization in real time during the examination. Summary of the Invention

[0011] [Technical Issues] The technical problem to be solved by this invention is: how to achieve spatiotemporal synchronization, high-precision registration and real-time fusion of three modes in a highly integrated system that maintains independent high performance of each mode of SLO, SS-OCT and ERG through hardware-level synchronous control.

[0012] [Technical Solution] To address the aforementioned technical problems, this invention provides an ophthalmic multimodal imaging system that synchronizes SLO, SS-OCT, and ERG signals. This system achieves nanosecond-level synchronous acquisition and registration of SLO, SS-OCT, and ERG signals via an FPGA hardware clock synchronization control bus.

[0013] The system includes a scanning laser ophthalmoscope (SLO) module, a swept-frequency optical coherence tomography (SS-OCT) module, an electroretinography (ERG) module, an FPGA main control board, an image processing unit, and a beam combining and scanning optical path. The output beam from the scanning laser ophthalmoscope (SLO) module and the output beam from the sweep frequency optical coherence tomography (SS-OCT) module are combined and then enter the fundus through the same scanning lens group and objective lens. The electroretinography (ERG) module is used to collect electrophysiological signals generated by light stimulation of the retina through contact electrodes. The FPGA main control board is used to generate a unified hardware clock and a global synchronization trigger signal to control the scanning laser ophthalmoscope SLO module, the swept-frequency optical coherence tomography (SS-OCT) module, and the electroretinography (ERG) module to work synchronously based on the same time reference, and to add a unified timestamp based on the unified hardware clock to the data collected by each module. The image processing unit is used to match feature points extracted from the image generated by the scanning laser ophthalmoscope SLO module with a pre-stored template to establish a coordinate mapping relationship between the scanning laser ophthalmoscope SLO module and the swept-frequency optical coherence tomography (SS-OCT) module, and to establish a temporal correlation between the electrophysiological signal generated by the electroretinography (ERG) module and the scanning position of the swept-frequency optical coherence tomography (SS-OCT) module based on the unified timestamp.

[0014] Optionally, the beam combining is achieved through an optical fiber coupling unit; and the scanning laser ophthalmoscope (SLO) module and the swept-frequency optical coherence tomography (SS-OCT) module each have independent galvanometer groups, lasers, and detectors.

[0015] Optionally, the laser of the scanning laser ophthalmoscope SLO module is a four-color laser, and the laser of the swept-frequency optical coherence tomography (SS-OCT) module is a high-speed swept-frequency laser; the detector of the scanning laser ophthalmoscope SLO module is a silicon photomultiplier tube, and the detector of the swept-frequency optical coherence tomography (SS-OCT) module is a balanced detector.

[0016] Optionally, the FPGA main control board includes: a clock management module for generating multiple derived clocks of the same source and phase; a trigger pulse generation module, which includes a dedicated timing state machine based on a reference clock count; and a trigger distribution and compensation module for fine-tuning each trigger signal through a programmable delay unit.

[0017] Optionally, the workflow of the dedicated timing state machine includes: IDLE state: Waiting for the horizontal synchronization signal output from the galvanometer control circuit of the scanning laser ophthalmoscope SLO module; DELAY_CNT state: After detecting the rising edge of the line synchronization signal, the internal counter is started to perform delay counting based on the reference clock; TRIG_GEN state: When the delay count is complete, a global hardware trigger pulse with a width of M reference clock cycles is output; HOLD state: After the trigger pulse is output, this state is maintained until the current row scan cycle ends or the conditions for entering the next cycle are met, and then it returns to the IDLE state.

[0018] Optionally, the unified timestamp is generated by a 64-bit monotonically increasing counter inside the FPGA, which is driven by the unified hardware clock; the FPGA main control board is provided with interface timing logic, which is used to latch the current value of the counter as a timestamp when sending acquisition trigger signals to each module, and store the timestamp together with the data acquired by the corresponding module.

[0019] Optionally, the image processing unit uses a dedicated image processing IP core integrated within the FPGA to perform differential Gaussian filtering, key point detection, and feature descriptor generation on the image generated by the scanning laser ophthalmoscope SLO module, thereby achieving hardware acceleration of feature extraction.

[0020] Optionally, the coordinate mapping relationship is obtained through calibration, which includes: simultaneously acquiring images generated by the scanning laser ophthalmoscope SLO module and volume data from the swept-frequency optical coherence tomography (SS-OCT) module using a standard model eye, identifying the same set of feature points and calculating coordinate mapping parameters; the coordinate mapping parameters are used to establish a mapping from the scanning coordinates of the swept-frequency optical coherence tomography (SS-OCT) module to the image coordinates generated by the scanning laser ophthalmoscope SLO module.

[0021] Optionally, to address large field-of-view nonlinear distortion, the coordinate mapping parameters are a set of transformation matrices based on partition mapping, or a set of coefficients of a high-order polynomial model.

[0022] Optionally, the synchronization delay between the scanning laser ophthalmoscope (SLO) module, the swept-frequency optical coherence tomography (SS-OCT) module, and the electroretinography (ERG) module is less than or equal to 50 nanoseconds.

[0023] [Beneficial Effects] This invention's system achieves nanosecond-level hardware synchronization of SLO, SS-OCT, and ERG through a unified hardware clock and global trigger signal generated by the FPGA main control board, and adds a unified timestamp to all data. This fundamentally eliminates the timing drift and jitter caused by independent clocks and software triggers in traditional systems, providing a unique and accurate time alignment reference for multi-source data. Based on the unified timestamp and the spatial mapping relationship between SLO images and SS-OCT scan coordinates established through a feature matching algorithm, the system can automatically and accurately map SS-OCT tomographic data to the two-dimensional navigation coordinate system of SLO, and associate ERG electrophysiological signals with specific scan anatomical locations through a timing matching table. This achieves point-to-point precise correspondence between retinal microstructure and physiological function, greatly improving the accuracy of lesion localization and joint diagnosis.

[0024] This invention employs an innovative design that features a shared core optical path and unified control, while key hardware is configured independently. SLO and SS-OCT share a scanning optical path to improve integration and coaxiality, while each is equipped with independent lasers, galvanometer groups, and detectors to ensure optimal imaging performance. This design effectively addresses the technical pain points of traditional multimodal devices that sacrifice modal performance for high integration, or struggle to achieve high-precision synchronization when pursuing independent performance.

[0025] This invention integrates three examination modalities into a single device, enabling simultaneous acquisition of multidimensional information in a single examination. This avoids the cumbersome operation, patient discomfort, and data correlation errors associated with switching between multiple devices, significantly improving clinical examination efficiency. The system has a compact structure and good scalability, making it suitable not only for ophthalmology outpatient and bedside examinations but also providing a powerful structure-function integrated research platform for scientific research and animal experiments on diseases such as glaucoma and diabetic retinopathy, enhancing the overall applicability and reliability of the equipment. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 The system structure block diagram provided by the present invention.

[0028] Figure 2 This is a schematic diagram of the optical path provided by the present invention.

[0029] Figure 3 The timing synchronization logic diagram provided for this invention.

[0030] Figure 4 The flowchart of the registration algorithm provided by this invention.

[0031] Figure 5 This is an imaging schematic diagram provided by the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be further described in detail below with reference to the accompanying drawings. In the specific implementation of this invention, the core circuit board is custom-designed according to the system functions and synchronization timing requirements, and its main control chip is a Xilinx 7020 series FPGA chip. The optical lenses, including the objective lens and the lenses in the scanning lens group, are customized according to the imaging field of view, resolution, and aberration control requirements. The main design parameters include radius of curvature, diameter, and surface tolerance. Other electronic components, such as lasers, detectors, galvanometer drivers, analog-to-digital converters, etc., and mechanical structural parts are all commercially available general-purpose products, and their selection is based on meeting the performance indicators of each module.

[0033] This embodiment provides an ophthalmic multimodal imaging system that synchronizes SLO, SS-OCT, and ERG. This system deeply integrates ultra-wide-angle scanning laser ophthalmoscope (SLO), high-speed swept-frequency optical coherence tomography (SS-OCT), and electroretinography (ERG). Through a unified hardware synchronization and precise registration mechanism, it achieves precise spatiotemporal correlation and joint analysis of retinal anatomical and physiological information. This system is particularly suitable for the early screening and accurate diagnosis of fundus diseases such as glaucoma and diabetic retinopathy.

[0034] Figure 1 The overall structural framework of the system in this embodiment is illustrated. The system mainly includes an optical path and acquisition module, a core control module, and an image processing unit. Physically, the system in this embodiment also includes a laser generation unit, an optical fiber coupling unit, a galvanometer scanning unit, a detector unit, and a visual stimulation and electrode acquisition unit. Each unit works collaboratively through an FPGA main control board, which will be described in detail below.

[0035] The optical path and acquisition module generates and guides probe light to the fundus, and acquires optical signals returning from the fundus and electrophysiological signals generated by the retina. Its core components include the Scanning Laser Ophthalmoscopy (SLO) module, the Scanned Frequency Optical Coherence Tomography (SS-OCT) module, and the Electroretinography (ERG) module. The core control module is an FPGA main control board, responsible for generating a unified hardware timing reference within the system and distributing synchronization trigger signals to each module to achieve strict hardware-level synchronization. The image processing unit reconstructs and analyzes the acquired multi-source data, and performs precise registration and fusion based on the unified spatiotemporal reference.

[0036] The system's integrated optical path design, such as Figure 2As shown, the system employs a multi-source integrated design. The system also includes a laser generation unit, which comprises two independent parts: a four-color laser providing illumination for the SLO module, and a high-speed sweeping laser providing a swept-frequency light source for the SS-OCT module. The four-color laser can emit wavelengths of 488nm, 520nm, 660nm, and 820nm, utilizing the different reflection characteristics of fundus tissues at different wavelengths to acquire high-contrast, wide-viewing-angle two-dimensional fundus reflection images. This multi-wavelength design aims to enhance image features: the 488nm wavelength is sensitive to the nerve fiber layer, aiding in the observation of superficial structures; the 520nm wavelength provides good contrast for blood vessels, facilitating angiography; the 660nm wavelength is advantageous for choroidal imaging; and 820nm serves as a reference wavelength, providing information on deep tissues. The high-speed sweeping laser has a center wavelength of 1060nm, a bandwidth of 100nm, and a sweep rate of up to 100kHz.

[0037] The SLO module includes a first galvanometer group, X1 and Y1, for beam scanning. This galvanometer group controls the illumination beam to perform a rapid two-dimensional scan on the retina, achieving a field of view of up to [missing information]. Ultra-wide-angle fundus imaging. The SLO signal light reflected from the fundus returns via the original optical path. The SS-OCT module contains independent second galvanometer groups, X2 and Y2 galvanometers. The swept laser is split into a reference arm and a sample arm via an optical fiber coupling unit. The sample arm light is guided by the second galvanometer group for scanning. The light returning after illuminating the retina interferes with the light returning from the reference arm in the coupler.

[0038] The beams from the SLO module and the SS-OCT module are combined in the fiber optic coupling unit and share subsequent scanning lens groups and objectives, thereby ensuring that the scanning areas of the fundus by both modules completely overlap in the physical optical path, laying the physical foundation for subsequent spatial registration. Preferably, in this embodiment, the objective lens is connected to a motor drive unit for automatic or manual focusing based on the refractive state of the fundus, in order to optimize the imaging quality of each modality.

[0039] The system also includes a detector unit for receiving and converting the aforementioned optical signals. This detector unit includes a silicon photomultiplier tube for detecting SLO reflected light and a balanced detector for detecting SS-OCT interference signals. The balanced detector suppresses common-mode noise and improves the system's signal-to-noise ratio through differential amplification.

[0040] The ERG module acquires weak electrophysiological signals generated by the retina under light stimulation via contact electrodes attached to the subject's cornea or eyelid. The light stimulation is emitted by the visual stimulation and electrode acquisition unit within the system. This unit uses a high-brightness LED array as the stimulation source, generating broadband white light with a wavelength range of 400–700 nm, and a peak brightness of [missing information]. The stimulation pulse width is adjustable within a range of 1 ms to 100 ms, and the repetition frequency is adjustable between 1 and 60 Hz. The intensity, duration, and frequency of the stimulation are set by the host computer and a synchronous trigger signal is distributed through the FPGA. The driving circuit of this LED array directly receives the synchronous trigger signal from the FPGA to precisely control the on and off times of the stimulation, thereby ensuring that the stimulation event is strictly synchronized in time with the scanning actions of other modalities.

[0041] Figure 3 The timing logic for multimodal synchronization of the system is demonstrated, with the FPGA main control board serving as the core of the synchronization mechanism. The FPGA main control board internally sets up a master reference clock source and generates a reference high-frequency clock signal (e.g., 200MHz, clock period 5ns) via a phase-locked loop (PLL), which serves as the system's sole time reference. Through its internal clock management unit, three in-phase and phase-locked clock signals are precisely derived from this reference clock, serving as the pixel sampling clock for the SLO module, the A-scan interference signal sampling clock for the SS-OCT module, and the sampling clock for the analog-to-digital converter (ADC) of the ERG module, respectively.

[0042] Furthermore, the FPGA main control board generates a global synchronization trigger signal. This signal is simultaneously distributed to: The galvanometer driver for the SLO module; External trigger input terminal (A-TRIG) of the SS-OCT module sweep frequency light source; The sampling enable pin of the ERG module. This ensures that the start of one line scan of SLO, one B-scan (i.e., one lateral depth scan) of SS-OCT, and the opening of one sampling window of ERG are all precisely coordinated by the same hardware trigger signal, keeping the synchronization delay between modules to less than or equal to 50 nanoseconds.

[0043] To achieve a synchronization accuracy of less than or equal to 50 nanoseconds, this embodiment incorporates a dedicated synchronization control logic circuit within the FPGA main control board. This circuit includes: Clock management module: It adopts a PLL and a clock management unit to generate multiple derived clocks with the same source, in phase and low jitter, based on the reference clock signal, which are used as the sampling clocks of each module. Trigger pulse generation module: Contains a dedicated timing state machine based on a reference clock count, used to generate a global hardware trigger pulse with precise width and delay at a specific phase point of the SLO row scan synchronization signal; the workflow of the dedicated timing state machine includes: IDLE state: Waiting for the horizontal synchronization signal output from the SLO module mirror control circuit; DELAY_CNT state: After detecting the rising edge of the SLO line synchronization signal, the internal counter is started to count the delay based on the reference clock. The number of clock cycles of the delay is a preset value N to achieve a fixed time delay. For example, under a 200 MHz clock, N=4 corresponds to a 20 ns delay. TRIG_GEN state: When the delay count is completed, a global hardware trigger pulse with a width of M reference clock cycles is output. For example, M=2 corresponds to a 10 ns pulse width. This pulse is simultaneously distributed to the SS-OCT module and the ERG module to trigger the A-scan sampling of SS-OCT and the visual stimulation of ERG. HOLD state: After the trigger pulse is output, this state is maintained until the current SLO line scan cycle ends or the conditions for entering the next cycle are met, and then it returns to the IDLE state to wait for the next line synchronization signal.

[0044] Trigger Distribution and Compensation Module: The trigger pulse is distributed to each module through the FPGA's global clock network and dedicated high-speed I / O pins. The on-chip programmable delay unit is used to fine-tune and compensate for the transmission delay of each trigger signal to ensure that the startup times of the SLO module, SS-OCT module and ERG module are strictly aligned.

[0045] The initial delay value of the programmable delay unit is determined through the following steps: During the system factory calibration or initial installation phase, a high-speed oscilloscope is used to measure the fixed time delay difference between the FPGA output trigger signal and the actual effective response generated by each peripheral module (SLO galvanometer driver, SS-OCT laser external trigger input terminal, ERG stimulus source drive circuit); based on the measurement results, a corresponding delay step value is configured for each trigger signal path and written into the FPGA's delay control register to complete static delay compensation.

[0046] Furthermore, the system also supports online delay compensation. During system operation, by monitoring and comparing the timestamps attached to the synchronization status flag signals returned by each module, the configuration parameters of the programmable delay unit are dynamically fine-tuned to achieve sub-clock cycle precision delay correction, in order to cope with minor drifts caused by component aging or environmental changes.

[0047] During data acquisition, a 64-bit monotonically increasing counter driven by a reference clock (200MHz) inside the FPGA appends a uniform timestamp to all acquired data. The minimum resolution of the timestamp is 5ns. Whether it is pixel data from SLO, A-scan data from SS-OCT, or sampling point data from ERG, all data are automatically marked with a global timestamp of the acquisition time before being stored in the shared data buffer inside the FPGA. All data is arranged in timestamp order, providing a precise time alignment basis for subsequent fusion.

[0048] In this embodiment, the operation of the 64-bit monotonically increasing counter is guaranteed by a reset mechanism: the system supports automatic power-on reset and host computer instruction reset to ensure that the counter starts from zero when the system starts or is recalibrated; in addition, the counter logic is linked with the status of the data acquisition module, and automatic reset can be triggered when an acquisition abnormality is detected to prevent the accumulation of invalid timestamp data.

[0049] To achieve precise binding between data and timestamps, the FPGA main control board has dedicated interface timing logic: when a data acquisition trigger signal is sent to any module within the same clock cycle, the current value of the counter is immediately latched as the timestamp of that batch of data; the acquired data is transmitted to the buffer within the FPGA via a high-speed interface and stored in the format of data block + timestamp. To eliminate timing uncertainties caused by data transmission, a clock of the same origin as the counter clock is used to manage data transmission, and an asynchronous FIFO buffer is used for temporary data storage to ensure a strict one-to-one correspondence between timestamps and data. In this embodiment, the preferred FIFO depth configuration is 4096×32bit to meet the temporary storage requirements under high sampling rates.

[0050] Figure 4 The core process of the registration algorithm is outlined. The task of the image processing unit is to achieve accurate spatiotemporal correlation and fusion of multimodal data based on a unified timestamp and acquired data, mainly including three levels: spatial registration, temporal alignment, and spatiotemporal fusion.

[0051] (1) Spatial registration The system uses real-time acquired ultra-wide-angle SLO fundus images as the spatial navigation map and reference coordinate system. The typical resolution of SLO images is 1024×1024 pixels. First, to establish coordinate mapping and compensate for eye movement, the image processing unit extracts stable feature points of the retina from the SLO images, such as vascular bifurcation points. Specifically, a feature extraction method based on Gaussian pyramids and Differential Gaussian (DoG) scale space is used to detect and accurately locate key points, generating 128-dimensional feature descriptors.

[0052] To meet real-time requirements, the feature extraction algorithm is hardware accelerated by using a dedicated image processing IP core designed within the FPGA, combined with an efficient parallel computing process and data flow optimization.

[0053] Image data is transmitted in real-time from the acquisition buffer to the feature processing pipeline within the FPGA via a high-speed interface, such as a data stream based on the AXI-Stream protocol. Specifically, the FPGA integrates dedicated IP cores for implementing Differential Gaussian (DoG) filtering, keypoint detection, and feature descriptor generation. Each IP core adopts a modular, pipelined architecture design. DoG filter IP core: Implemented based on programmable logic resources, it supports multi-scale Gaussian convolution such as 5 scales, with a kernel size of 3×3 or 5×5, adopts a fully pipelined architecture, and can achieve a processing speed of 2 pixels per clock cycle; Keypoint Detection IP Core: Performs extreme point detection on multi-scale images after DoG filtering. By comparing the current pixel with its 8 neighbors and the corresponding positions of the adjacent scales above and below, a total of 26 pixels, and combining non-maximum suppression (NMS) to quickly remove redundant points, the IP core processing latency is no more than 100 nanoseconds / pixel. Feature descriptor generation IP core: For detected key points, extract the gradient direction and magnitude information in their 16×16 pixel neighborhood to generate a 128-dimensional feature vector; This IP core uses a parallel multiplier array to accelerate gradient calculation in hardware, and the generation time of a single descriptor does not exceed 1 microsecond.

[0054] To fully leverage the parallel computing capabilities of the FPGA, this embodiment employs an image block-based parallel processing mechanism: each 1024×1024 pixel SLO image frame is divided into multiple 64×64 pixel sub-blocks, which are then distributed to multiple independent parallel processing units (PEs) within the FPGA. Each PE possesses complete filtering capabilities. Detection The descriptor generation pipeline can process an image sub-block independently, thereby achieving true parallel processing of different regions within the same frame of an image.

[0055] In terms of timing scheduling, the system minimizes pipeline idle time and improves overall throughput by overlapping data reading, processing, and storage operations. For example, while processing a sub-block of the current frame image is underway, data for the next frame sub-block has already begun loading, and the processing results of the previous frame are being output. The feature descriptors generated by each PE are ultimately collected and integrated by the aggregation module to form the complete feature set of the frame image.

[0056] Through the above-mentioned dedicated IP core design, block parallel architecture and pipeline scheduling optimization, the system can complete the real-time feature extraction of a single frame of 1024×1024 pixel SLO image in no more than 5 milliseconds, thereby ensuring the real-time performance of multimodal registration.

[0057] Subsequently, the feature descriptors extracted from the real-time SLO images are matched with a pre-stored standard scan region reference template, and the coordinate transformation relationship between the two is robustly estimated using the Random Sample Consensus (RANSAC) algorithm. This relationship describes the relative positional change between the real-time fundus and the calibration template, and can be described by a transformation matrix H (3×3 matrix), as shown in the following equation:

[0058] Where u, v are the pixel coordinates of the SLO image; x, y are the mirror drive coordinates of the SS-OCT scan; and the inverse matrix H... - ¹Used to convert SLO image coordinates into SS-OCT scan commands.

[0059] This transformation relationship, combined with the system's pre-stored calibration mapping parameters, enables the real-time and accurate mapping of the SS-OCT scanning position to the SLO navigation image coordinate system.

[0060] To achieve the aforementioned high-precision coordinate mapping, the system needs to be calibrated before use. The calibration process aims to establish a stable and accurate basic mapping relationship between SLO image coordinates and SS-OCT scan coordinates, and mainly includes the following steps: SLO images and SS-OCT volume data were acquired simultaneously using a standard model eye. Software-automated identification was employed to obtain the same set of feature points from both images. Specifically, the methods included: performing vascular enhancement preprocessing on the SLO images; extracting the vascular centerline using a skeletonization algorithm; automatically identifying vascular bifurcation and intersection points as stable feature points through node degree analysis; and performing sub-pixel-level localization on the detected feature points to generate a precise coordinate set.

[0061] The coordinate mapping relationship is constructed through multi-point calibration to address the nonlinear distortions that may be introduced by large field-of-view optical systems, forming a mapping model that covers the entire field of view and has consistent accuracy. Specifically, this involves: selecting multiple pairs of feature points identified in the above steps across the entire field of view; using the least squares method or the Random Sample Consensus (RANSAC) algorithm to fit a coordinate transformation model based on the feature point pairs, such as a two-dimensional affine or perspective transformation model; and further optimizing the accuracy across the entire field of view by employing more refined model strategies, such as a partitioned mapping strategy that divides the field of view into multiple regions and establishes transformation models for each region, or a higher-order polynomial model for local fitting and optimization. Ultimately, a complete set of coordinate mapping parameters is formed.

[0062] The coordinate mapping parameter set obtained from the calibration is stored as the basic calibration parameters of the system. At the same time, based on the standard model eye SLO image data collected during the calibration phase, feature descriptors of stable regions are extracted, and generated and stored as reference templates for subsequent real-time feature matching of the patient's fundus.

[0063] In addition, during the calibration process, time synchronization verification must be performed simultaneously: by inputting test signals into the ERG module and verifying the consistency between the timestamp of its returned signal and the system scan trigger signal, the accuracy of the multimodal timing reference must be ensured.

[0064] (2) Time alignment Since all data carries FPGA timestamps originating from the same reference, the core of time alignment is sorting and matching the timestamps. After receiving the data, the host computer sorts the SLO, SS-OCT, and ERG data streams in ascending order of their timestamps.

[0065] For each A-scan data packet of SS-OCT that includes its acquisition timestamp and SLO image coordinates obtained through spatial registration mapping, the sampling point with the smallest timestamp difference is found in the ERG data stream after sorting by time, and the potential value of the ERG sampling point is associated with this A-scan and its corresponding spatial location.

[0066] (3) Spatiotemporal fusion After completing spatial registration and temporal alignment, the system performs final spatiotemporal fusion. Based on the results of the preceding steps, the host computer constructs a spatial coordinate-structural data-functional response timing matching table. This table uses the physical location on the retina, i.e., the SLO image coordinates, as an index, and associates it with the corresponding SS-OCT deep structural data (A-scan or B-scan) and the temporally strictly aligned ERG functional response values ​​(potentials).

[0067] By using a lookup table, the system can directly and accurately correlate the anatomical structure represented by any scanned tiny area on the retina with its electrophysiological response to light stimulation. Ultimately, the system outputs a comprehensive dataset that deeply integrates ultra-wide-angle two-dimensional fundus structure, high-resolution three-dimensional tomographic structure, and local electrophysiological function information.

[0068] Through the aforementioned hardware synchronization and algorithm registration, the system ultimately outputs a comprehensive dataset that deeply integrates ultra-wide-angle two-dimensional fundus structure, high-resolution three-dimensional tomographic structure, and local electrophysiological function information. For example... Figure 5 As shown, during application, users can intuitively browse the entire fundus on the ultra-wide-angle SLO navigation map presented in the system software interface and click on any point. The system then displays the corresponding high-resolution SS-OCT tomographic image at the associated location on the interface, clearly showing the structure of each layer of the retina, and simultaneously providing the ERG functional response waveform for that specific local area.

[0069] The system in this embodiment achieves true integrated "structure-function" diagnosis, enabling doctors or researchers to directly and accurately correlate and analyze subtle structural abnormalities, such as thinning of the retinal nerve fiber layer, with specific functional declines recorded from the same location, such as reduced electrical signal amplitude. This system significantly improves the sensitivity and accuracy of detecting early, subtle pathological changes in fundus diseases such as glaucoma and diabetic retinopathy, providing a powerful tool for early intervention and precision treatment.

[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A SLO, SS-OCT and ERG synchronized ophthalmic multimodal imaging system, characterized in that, The scanning laser ophthalmoscope SLO module, the swept-source optical coherence tomography SS-OCT module, the electroretinogram ERG module, the FPGA master control board, the image processing unit, and the beam combining and scanning optical path are included. The outgoing beams of the scanning laser ophthalmoscope SLO module and the swept-source optical coherence tomography SS-OCT module are combined and then enter the fundus through the same scanning mirror group and the objective lens. The electroretinogram ERG module is configured to collect the electrical physiological signals generated by the retina under light stimulation through a contact electrode. The FPGA master control board is configured to generate a unified hardware clock and a global synchronization trigger signal, control the scanning laser ophthalmoscope SLO module, the swept-source optical coherence tomography SS-OCT module, and the electroretinogram ERG module to work synchronously based on the same time reference, and attach a unified timestamp based on the unified hardware clock to the data collected by each module. The image processing unit is configured to match the feature points extracted from the images generated by the scanning laser ophthalmoscope SLO module with a pre-stored template, establish the coordinate mapping relationship between the scanning laser ophthalmoscope SLO module and the swept-source optical coherence tomography SS-OCT module, and establish the time sequence correlation between the electrical physiological signals generated by the electroretinogram ERG module and the scanning positions of the swept-source optical coherence tomography SS-OCT module based on the unified timestamp.

2. The system of claim 1, wherein, The beam combining is achieved through a fiber coupling unit, and the scanning laser ophthalmoscope SLO module and the swept-source optical coherence tomography SS-OCT module each have an independent galvanometer mirror group, a laser, and a detector.

3. The system of claim 2, wherein, The laser of the scanning laser ophthalmoscope SLO module is a four-color laser, and the laser of the swept-source optical coherence tomography SS-OCT module is a high-speed swept laser; the detector of the scanning laser ophthalmoscope SLO module is a silicon photomultiplier, and the detector of the swept-source optical coherence tomography SS-OCT module is a balanced detector.

4. The system of claim 1, wherein, The FPGA master control board includes a clock management module for generating multiple derived clocks with the same source and phase, a trigger pulse generation module including a dedicated timing state machine based on reference clock counting, and a trigger distribution and compensation module for fine-tuning each trigger signal through a programmable delay unit.

5. The system of claim 4, wherein, The working process of the dedicated timing state machine includes: IDLE state: waiting for a line synchronization signal output by a galvanometer control circuit of the scanning laser ophthalmoscope SLO module; DELAY_CNT state: after detecting the rising edge of the line synchronization signal, starting an internal counter to perform delay counting based on the reference clock; TRIG_GEN state: when the delay counting is completed, outputting a global hardware trigger pulse with a width of M reference clock periods; HOLD state: after the trigger pulse is output, maintaining this state until the current line scanning period ends or the condition for entering the next period is met, and then returning to the IDLE state.

6. The system of claim 1, wherein, The unified timestamp is generated by a 64-bit monotonically increasing counter inside the FPGA, which is driven by the unified hardware clock; the FPGA main control board is provided with interface timing logic, which is used to latch the current value of the counter as a timestamp when sending a collection trigger signal to each module, and store the timestamp together with the data collected by the corresponding module.

7. The system of claim 1, wherein, The image processing unit performs differential Gaussian filtering, key point detection and feature descriptor generation on the image generated by the scanning laser ophthalmoscope (SLO) module through a special image processing IP core integrated in the FPGA, so as to realize hardware acceleration of feature extraction.

8. The system of claim 1, wherein, The coordinate mapping relationship is obtained through calibration, and the calibration includes: using a standard model eye to simultaneously collect the image generated by the scanning laser ophthalmoscope (SLO) module and the volume data of the swept-source optical coherence tomography (SS-OCT) module, identifying the same group of feature points and calculating the coordinate mapping parameters; the coordinate mapping parameters are used to establish the mapping from the scanning coordinates of the swept-source optical coherence tomography (SS-OCT) module to the image coordinates of the scanning laser ophthalmoscope (SLO) module.

9. The system of claim 8, wherein, In order to cope with the large field of view nonlinear distortion, the coordinate mapping parameters are a set of transformation matrices based on partition mapping, or a set of coefficients of a high-order polynomial model.

10. The system of claim 1, wherein, The synchronization delay between the scanning laser ophthalmoscope (SLO) module, the swept-source optical coherence tomography (SS-OCT) module and the electroretinogram (ERG) module is less than or equal to 50 nanoseconds.

Images