A multi-modal stimulation and neural signal processing system and method based on biological eyes

Abstract

This invention discloses a multimodal stimulation and neural signal processing system and method based on the biological eye, comprising: a unified master clock module for generating a global synchronization clock signal; a multimodal stimulation module for synchronously generating optical and electrical stimulation signals under the control of the synchronization clock signal to apply combined photoelectric stimulation to biological samples; a high-frequency acquisition and preprocessing module for synchronously acquiring the neurophysiological signals generated after stimulation of biological samples, and performing artifact removal, filtering, and spike potential extraction processing to obtain preprocessed neural pulse data; and an information decoding module for establishing a mapping model between stimulation parameters and neural responses through deep learning and inverse correlation techniques, and outputting a standardized decoded information stream. This invention achieves closed-loop integration of photoelectric multimodal stimulation and neural signal processing, significantly improving the real-time performance and accuracy of neural information interaction, and can be used for research on neural coding in the biological eye.

Description

Technical Field

[0001] This invention belongs to the field of neural interface and biomedical engineering, and particularly relates to a multimodal stimulation and neural signal processing system and method based on biological eyes. Background Technology

[0002] Currently, research in the fields of neural interfaces and biomedical engineering regarding the interaction between the biological eye and the nervous system primarily relies on microelectrode arrays (MEAs) technology. Existing common methods involve applying single electrical pulses to biological tissues such as the retina, optic nerve, or visual cortex via MEAs to induce spike potential firing in biological neurons. These signals are then collected and analyzed using recording electrodes. This approach is mainly based on the physical mechanisms of electrical stimulation, using parameters such as pulse amplitude, pulse width, and frequency to intervene in neuronal activity. Combined with spike potential signal classification and decoding algorithms, it attempts to establish a mapping relationship between artificial stimulation and neural responses, providing a fundamental technical means for visual function restoration and neural coding research.

[0003] However, existing technologies still face the following technical challenges in practical applications: First, the stimulation modality is limited, and relying solely on electrical pulse stimulation makes it difficult to reproduce the inherent light-sensing input patterns of the biological visual system, resulting in insufficient physiological fidelity in reproducing real visual perception. Second, electrical stimulation has low activation specificity for neurons and is prone to introducing significant electrical artifacts during signal acquisition, severely interfering with the accurate recording of weak neural response signals. Third, there is a logical gap between artificially synthesized electrical pulse sequences and the natural physiological coding mechanisms of organisms, making it difficult for the system to establish a precise and stable stimulus-response mapping relationship, posing challenges when dealing with dynamic and complex neural responses. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multimodal stimulation and neural signal processing system and method based on biological eyes.

[0005] One of them, a multimodal stimulation and neural signal processing system based on a biological eye, includes: The unified master clock module is used to generate a global synchronization clock signal, providing a unified time base reference for the entire system; A multimodal stimulation module, connected to the unified master clock module, is used to synchronously generate optical stimulation signals and electrical stimulation signals under the control of the global synchronization clock signal according to preset stimulation parameters, and apply photoelectric composite stimulation to the biological sample to obtain the stimulated response of the biological sample. The high-frequency acquisition and preprocessing module is connected to the unified master clock module and is used to synchronously acquire the neurophysiological signals generated after the biological sample is stimulated under the control of the global synchronization clock signal, and to perform artifact removal, filtering and spike potential signal extraction processing on the neurophysiological signals to obtain preprocessed neural pulse data. The information decoding module, connected to the high-frequency acquisition and preprocessing module, is used to perform feature association analysis based on the preprocessed neural impulse data using deep learning and reverse correlation techniques, establish a mapping model between stimulus parameters and neural responses, and output a standardized decoded information stream.

[0006] Preferably, the multimodal stimulation module includes: The photostimulation submodule is used to generate photostimulation patterns based on spatial location parameters, physical light intensity parameters, and time series parameters. The electrical stimulation submodule is used to generate multi-channel electrical pulse sequences based on stimulation mode, spatial location selection parameters, and waveform timing parameters. The optocoupler scheduling submodule is connected to the photostimulation submodule, the electrical stimulation submodule, and the unified master clock module, respectively. It is used to pre-compensate and synchronize the timing of the photostimulation signal and the electrical stimulation signal to ensure that the light spot projection and the starting phase of the electrical pulse are strictly aligned within the millisecond error range.

[0007] Preferably, the photostimulation submodule includes: The spatial coding unit, with a built-in affine transformation algorithm, is used to map the vector graphics of the virtual stimulus canvas to the hardware address of the physical light source array in real time, and to compensate for optical path geometric distortions to generate a pixel-level mask matrix. The light intensity control unit is connected to the spatial coding unit and has a built-in nonlinear brightness mapping lookup table, which is used to accurately map the preset light intensity command into the driving current or pulse width of the light source through Gamma correction, and linearly control the light power density. The timing scheduling unit is connected to the spatial coding unit and the optical intensity control unit respectively. It has a built-in ping-pong buffer scheduling algorithm, which is used to achieve strict continuity and temporal determinism of dynamic stimuli on the time axis through microsecond-level pulse width modulation sequence and parallel frame preloading mechanism.

[0008] Preferably, the electrical stimulation submodule includes: The drive mode mapping unit has a built-in impedance monitoring feedback algorithm, which is used to dynamically adjust the output range of the digital-to-analog converter according to the constant current or constant voltage physical amplitude set by the user, and automatically calculate the injected compensation charge to maintain the electrochemical balance of the tissue interface. The spatial addressing gating unit is connected to the driving mode mapping unit and has built-in electrode gating matrix mapping logic. It is used to activate specific independent channels or channel combinations in MEAs in real time according to the geometric distribution of biological samples to generate localized stimulation topology. The waveform timing encoding unit, connected to the spatial addressing gating unit, has a built-in nested loop counting scheduling logic, which is used to define the pulse width, pulse interval and loop count with microsecond-level precision to generate highly repeatable custom stimulus waveforms.

[0009] Preferably, the high-frequency acquisition and preprocessing module includes: The adaptive filtering unit is used to filter the acquired raw electrophysiological signals through an adaptive bandpass filter, filtering out baseline drift and high-frequency environmental noise to obtain the filtered signal. The real-time artifact removal unit is connected to the adaptive filtering unit. It is used to calculate the average artifact waveform characteristics at the moment of electrical stimulation in real time according to the stimulation trigger time stamp provided by the unified master clock module, and to use the anti-phase superposition technique to remove artifacts from the filtered signal to obtain a pure neural signal. The spike potential detection and classification unit is connected to the real-time cancellation unit for electrical artifacts. It is used to extract spike potential waveforms from the pure neural signals using an adaptive threshold method, perform principal component analysis and spatial consistency checks on the spike potential waveforms, apply a spatial penalty factor based on the physical distance between electrodes, and classify and obtain neuronal unit signals with spatial authenticity. A unified time stamp injection unit is connected to the multimodal stimulation module and the unified master clock module, respectively. It is used to insert light stimulation trigger events and electrical stimulation trigger events as synchronous metadata into the data stream in real time while recording neural signals, and automatically mark key signal segments before, during and after stimulation through a timestamp association algorithm.

[0010] Preferably, the information decoding module includes: The deep learning correlation analysis unit is used to perform deep correlation analysis on light stimulation parameters, electrical stimulation parameters and classified neuronal firing characteristics through reverse correlation technology, and decode to obtain the nonlinear mapping model between stimulus input and target neural response; The information flow standardization output unit, connected to the deep learning association analysis unit, is used to convert the real-time acquired neural responses into a standardized digital information flow according to the nonlinear mapping model, and to provide a data output interface for connection with external devices.

[0011] This invention also provides a method for multimodal stimulation and neural signal processing based on a biological eye, comprising: Based on a global synchronization clock signal generated by a unified master clock, optical stimulation signals and electrical stimulation signals are generated synchronously, and photoelectric composite stimulation is applied to the biological sample to obtain the stimulated response of the biological sample. Based on the unified master clock signal, the neurophysiological signals generated after the biological sample is stimulated are synchronously acquired, and the neurophysiological signals are processed by artifact removal, filtering and spike potential signal extraction to obtain preprocessed neural pulse data. Based on the preprocessed neural impulse data, feature association analysis is performed using deep learning and reverse correlation techniques to establish a mapping model between stimulus parameters and neural responses, and a standardized decoded information stream is output.

[0012] Preferably, the process of simultaneously generating optical stimulation signals and electrical stimulation signals includes: A light stimulation pattern is generated using the light stimulation submodule; A multi-channel electrical pulse sequence is generated through the electrical stimulation submodule; The firing timing of the light stimulation pattern and the electrical pulse sequence is pre-compensated and synchronized to achieve strict alignment between the light spot projection and the starting phase of the electrical pulse within millisecond-level error. The process of generating a light-stimulated pattern includes: The vector graphics of the virtual stimulus canvas are mapped to the hardware address of the physical light source array in real time based on the affine transformation algorithm, and the optical path geometric distortion is compensated to generate a pixel-level mask matrix. Based on a nonlinear brightness mapping lookup table, the preset light intensity command is accurately mapped to the driving current or pulse width of the light source through Gamma correction, thereby achieving linear control of the light power density. The ping-pong buffer scheduling algorithm ensures the strict continuity and temporal determinism of dynamic stimuli on the time axis through microsecond-level pulse width modulation sequences and parallel frame preloading mechanisms.

[0013] Preferably, the process of generating a multi-channel electrical pulse sequence includes: Based on the impedance monitoring feedback algorithm, the output range of the digital-to-analog converter is dynamically adjusted according to the constant current or constant voltage physical amplitude set by the user, and the injected compensation charge is automatically calculated to maintain the electrochemical balance of the tissue interface. Based on the electrode gating matrix mapping logic, specific independent channels or channel combinations in MEAs are activated in real time according to the geometric distribution of biological samples to generate localized stimulation topology. Based on nested loop counting scheduling logic, pulse width, pulse interval and loop count are defined with microsecond-level precision to generate highly repeatable custom stimulation waveforms.

[0014] Preferably, the process of performing artifact removal, filtering, and spike potential signal extraction on the neurophysiological signals includes: The acquired raw electrophysiological signals are filtered by an adaptive bandpass filter to remove baseline drift and high-frequency environmental noise, and the filtered signal is obtained. Based on the stimulation trigger time stamp provided by the unified master clock signal, the average artifact waveform characteristics at the moment of electrical stimulation are calculated in real time, and the filtered signal is stripped of artifacts using the inverse superposition technique to obtain a pure neural signal. The spike potential waveform is extracted from the pure neural signal using an adaptive thresholding method. Principal component analysis and spatial consistency checks are performed on the spike potential waveform. A spatial penalty factor is applied based on the physical distance between electrodes to classify and obtain neuronal unit signals with spatial authenticity. Light stimulation triggering events and electrical stimulation triggering events are inserted into the data stream in real time as synchronous metadata, and key signal segments before, during and after stimulation are automatically labeled using a timestamp association algorithm.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention effectively compensates for the lack of physiological realism in single electrical stimulation by integrating a multimodal collaborative control architecture of optical stimulation, electrical stimulation, and photoelectric composite stimulation, achieving a more accurate simulation of the natural coding mechanism of the biological visual system. Simultaneously, through a hardware synchronization mechanism driven by a unified master clock and real-time cancellation technology for electrical artifacts, it significantly suppresses acquisition noise introduced by electrical stimulation, improving the purity and reliability of neural signal recordings. Furthermore, by combining deep learning and inverse correlation techniques to construct a stimulus-response mapping model, it overcomes the logical gaps in traditional manual coding, achieving a closed-loop interaction from stimulus command issuance to neural signal decoding, providing a highly real-time and highly stable integrated system solution for biological eye neural coding research. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the system structure according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the photostimulation submodule structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the electrical stimulation submodule structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the signal decoding module structure according to an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0019] Example 1 like Figure 1 As shown, this embodiment provides a multimodal stimulation and neural signal processing system based on a biological eye, including: The unified master clock module is used to generate a global synchronization clock signal, providing a unified time base reference for the entire system; The multimodal stimulation module, connected to the unified master clock module, is used to synchronously generate optical stimulation signals and electrical stimulation signals under the control of a global synchronization clock signal according to preset stimulation parameters, and apply photoelectric composite stimulation to biological samples to obtain the stimulated response of biological samples. The high-frequency acquisition and preprocessing module is connected to the unified master clock module. It is used to synchronously acquire the neurophysiological signals generated after the biological sample is stimulated under the control of the global synchronization clock signal, and to perform artifact removal, filtering and spike potential signal extraction processing on the neurophysiological signals to obtain preprocessed neural pulse data. The information decoding module, connected to the high-frequency acquisition and preprocessing module, is used to perform feature association analysis on the preprocessed neural impulse data through deep learning and reverse correlation technology, establish a mapping model between stimulus parameters and neural responses, and output a standardized decoded information stream.

[0020] Furthermore, the multimodal stimulation module includes: The photostimulation submodule is used to generate photostimulation patterns based on spatial location parameters, physical light intensity parameters, and time series parameters. The electrical stimulation submodule is used to generate multi-channel electrical pulse sequences based on stimulation mode, spatial location selection parameters, and waveform timing parameters. The optocoupler scheduling submodule is connected to the photostimulation submodule, the electrical stimulation submodule, and the unified master clock module, respectively. It is used to pre-compensate and synchronize the timing of the photostimulation signal and the electrical stimulation signal, ensuring that the light spot projection and the starting phase of the electrical pulse are strictly aligned within the millisecond error range.

[0021] Furthermore, the photostimulation submodule includes: The spatial coding unit, with a built-in affine transformation algorithm, is used to map the vector graphics of the virtual stimulus canvas to the hardware address of the physical light source array in real time, and to compensate for optical path geometric distortions to generate a pixel-level mask matrix. The light intensity control unit, connected to the spatial coding unit, has a built-in nonlinear brightness mapping lookup table, which is used to accurately map the preset light intensity command into the driving current or pulse width of the light source through Gamma correction, and linearly control the light power density. The timing scheduling unit is connected to the spatial coding unit and the optical intensity control unit respectively. It has a built-in ping-pong buffer scheduling algorithm, which is used to achieve strict continuity and temporal determinism of dynamic stimuli on the time axis through microsecond-level pulse width modulation sequence and parallel frame preloading mechanism.

[0022] Furthermore, such as Figure 2 As shown, the photostimulation submodule involved in this embodiment aims to achieve high-resolution, physiological encoding of biological tissues (such as the retina and optic nerve organoids) by simulating natural light-sensing input. The system achieves precise control through the following methods: Spatial Position Parameter Encoding: The system backend employs a sophisticated coordinate transformation and image processing algorithm to accurately project the stimulus pattern. First, the software utilizes an affine transformation algorithm to establish a real-time mapping between the logical coordinate system of the virtual stimulus canvas and the physical addresses of the physical light source array (such as a DMD or Micro-LED array). The algorithm automatically compensates for tangential and radial geometric distortions introduced by the optical path system, thereby locking the projection position accuracy at the micrometer level. When the backend receives high-level vector description commands such as "circle," "grid," or "moving target," the system calls a scanline transformation algorithm to discretize the vector graphics in real time, generating a pixel-level mask matrix that can be directly recognized by the hardware. Based on this, the system constructs multi-region independent gating logic using a bitmap indexing algorithm, supporting parallel and independent regional control of the light-emitting array. This enables precise, spatially differentiated stimulation of different anatomical regions of biological samples (such as the retina, neural tissue, or organoids).

[0023] Physical characteristic adjustment: The platform incorporates a light source response characteristic model and utilizes a nonlinear brightness mapping and lookup table algorithm to precisely map the software-preset 0-255 level light intensity commands to the underlying hardware driving current or pulse width. This process eliminates nonlinear response errors in the light source hardware through Gamma correction, ensuring that the physical output light power density remains strictly linear with the experimental set value, thereby achieving precise micro-control of physiological brightness. For multi-wavelength light source systems, the backend employs a multi-channel spectral fusion algorithm, synchronously controlling the driving power of different wavelength channels such as red, green, blue, and violet through weight allocation logic. This not only achieves pure monochromatic stimulation but also simulates specific spectral power distributions through color mixing to achieve specific activation of different photosensitive structures within biological tissues. Furthermore, the system integrates real-time contrast modulation logic. The algorithm can calculate the intensity difference between the background light and the target stimulus signal in real time and dynamically compensate for light intensity fluctuations using an adaptive gain control mechanism, ensuring that the visual contrast of the stimulus remains constant under different reference brightness levels.

[0024] Time-series encoding: A high-precision timing scheduling algorithm ensures strict execution of stimulus signals along the timeline. The software first utilizes a timer interrupt algorithm to convert preset intensity parameters into a high-frequency pulse-width modulation (PWM) sequence in real time. By controlling the duty cycle of the drive signal at the microsecond level, precise temporal adjustment of light intensity is achieved while maintaining stable spectral characteristics. For dynamic video streams or high-speed flickering stimuli, a ping-pong buffer scheduling algorithm is used for adaptive management in the background. While the hardware executes the stimulus task for the current frame, the timing parameters for the next frame are simultaneously pre-calculated and loaded. This parallel processing mechanism effectively eliminates timing jumps or frame drops during complex dynamic switching. Furthermore, by optimizing the execution priority and logic gating of underlying instructions, the system ensures that the duration and trigger interval of each stimulus sequence have extremely high temporal determinism, providing a high-precision time reference for accurately recording the response delay of biological samples.

[0025] Furthermore, the electrical stimulation submodule includes: The drive mode mapping unit has a built-in impedance monitoring feedback algorithm, which is used to dynamically adjust the output range of the digital-to-analog converter according to the constant current or constant voltage physical amplitude set by the user, and automatically calculate the injected compensation charge to maintain the electrochemical balance of the tissue interface. The spatial addressing gating unit is connected to the driving mode mapping unit and has built-in electrode gating matrix mapping logic. It is used to activate specific independent channels or channel combinations in MEAs in real time according to the geometric distribution of biological samples to generate localized stimulation topology. The waveform timing encoding unit, connected to the spatial addressing gating unit, has a built-in nested loop counting scheduling logic, which is used to define the pulse width, pulse interval and loop count with microsecond-level precision to generate highly repeatable custom stimulus waveforms.

[0026] Furthermore, such as Figure 3 As shown, the electrical stimulation involved in this embodiment specifically includes: Stimulation Mode and Intensity: Supports switching and precise mapping between constant current and constant voltage driving modes. Through a built-in impedance monitoring and feedback algorithm, the output range of the digital-to-analog converter (DAC) is dynamically adjusted to accurately convert the user-defined physical amplitude (microampere-level current or millivolt-level voltage) into a hardware-level step instruction. Furthermore, the algorithm integrates charge balance mapping logic, automatically calculating the algebraic sum of positive and negative phase charges when each frame of stimulation signal is delivered. By adjusting the amount of compensation charge injected, it ensures the electrochemical balance of the biological tissue interface, preventing electrode polarization or tissue damage caused by long-term stimulation.

[0027] Spatial site selection and topology scheduling algorithm (location selection): For the selection of stimulation sites, the background control software constructs an electrode gating matrix mapping logic to achieve precise control over any independent channel or combination of channels in MEAs. Based on the geometric distribution of biological samples on the electrode array, the algorithm uses a spatial addressing algorithm to activate specific electrode units in real time, thereby achieving localized stimulation of target neuronal clusters or specific anatomical structures. Simultaneously, the system supports multi-channel parallel scheduling and can generate complex and variable stimulation topologies through software definition to simulate multi-point spatial interactions within neural circuits.

[0028] Stimulus waveform and timing logic encoding: Allows users to define various stimulus waveforms such as monophasic, symmetrical / asymmetrical biphasic, sine waves, or random white noise, and define pulse width and pulse interval with microsecond-level precision. The algorithm uses a nested loop counting and scheduling logic to precisely manage the number of loops of stimulus instructions and the delay time between sequences, ensuring high repeatability and rigor of stimulus patterns in the time domain, and providing a defined timing framework for inducing specific neurodynamic features.

[0029] In this embodiment, multimodal stimulation achieves deep coupling and precise alignment of optical and electrical stimulation in the temporal domain through a core algorithm. To eliminate physical delay differences between different hardware links, the platform constructs a unified master clock scheduling mechanism based on the Hardware Abstraction Layer (HAL). This algorithm pre-calculates and compensates for the pattern refresh delay of optical stimulation and the pulse firing delay of electrical stimulation, ensuring that the synchronization error of photoelectric commands at the underlying actuator is controlled within milliseconds. During the command execution phase, the system adopts a multi-threaded deterministic synchronous triggering protocol, replacing traditional software serial commands with hard triggering logic. When the background generates a photoelectric composite stimulation sequence, the algorithm broadcasts a synchronous trigger packet containing time stamp information in parallel to the optical modulator and microelectrode driver via a high-speed bus. The rising edge triggering mechanism of the physical layer ensures that the instant the light spot is projected is strictly aligned with the starting phase of the electrical pulse, achieving a combination of physiological encoding and high spatiotemporal response characteristics. In addition, to address artifact interference caused by electrical stimulation, the synchronization algorithm integrates stimulation gap blanking and dynamic alignment logic. The software adjusts the sampling alignment point of the optical stimulation signal by accurately calculating the duration of the electrical pulse, ensuring that the data acquisition module can effectively avoid electrical noise peaks during composite stimulation, thereby achieving seamless mapping of the entire "stimulation command - physiological response - signal decoding" chain on the time axis.

[0030] Furthermore, the high-frequency acquisition and preprocessing module includes: The adaptive filtering unit is used to filter the acquired raw electrophysiological signals through an adaptive bandpass filter, filtering out baseline drift and high-frequency environmental noise to obtain the filtered signal. The real-time artifact cancellation unit, connected to the adaptive filtering unit, is used to calculate the average artifact waveform characteristics at the moment of electrical stimulation in real time based on the stimulation trigger time stamp provided by the unified master clock module, and to use the anti-phase superposition technique to remove artifacts from the filtered signal to obtain a pure neural signal. The spike potential detection and classification unit, connected to the real-time cancellation unit for electrical artifacts, is used to extract spike potential waveforms from pure neural signals using an adaptive threshold method, perform principal component analysis and spatial consistency checks on the spike potential waveforms, apply a spatial penalty factor based on the physical distance between electrodes, and classify and obtain neuronal unit signals with spatial authenticity. The unified time stamp injection unit is connected to the multimodal stimulation module and the unified master clock module, respectively. It is used to insert light stimulation trigger events and electrical stimulation trigger events as synchronous metadata into the data stream in real time while recording neural signals, and automatically mark key signal segments before, during and after stimulation through a timestamp association algorithm.

[0031] Furthermore, the data preprocessing module in this embodiment employs a software-defined adaptive bandpass filter to initially filter out baseline drift and high-frequency environmental noise. Simultaneously, to address the strong artifact interference introduced by electrical stimulation, the algorithm calculates the average artifact waveform characteristics at the moment of stimulation in real time and uses an anti-superposition technique for physical-level noise removal. Then, an adaptive thresholding method is used to extract effective spike potential signals from the background recording, supporting unidirectional upward or downward detection and simultaneous bidirectional upward and downward detection. This process identifies level transitions in real time and extracts key waveform segments containing neuronal firing characteristics. Principal component analysis is performed on the identified Spike waveforms, projecting them into a high-dimensional feature space for preliminary mathematical clustering. A spatial consistency check is forcibly introduced into the clustering logic. The algorithm reads the hardware mapping table of MEAs to calculate the physical distance between the electrodes capturing the target signal. If the distance between two electrodes exceeds a preset physiological threshold (e.g., >200µm), even if their waveforms are highly similar in mathematical characteristics, the algorithm will classify them as different neuronal units using a spatial penalty factor. This logic effectively prevents misclassification caused by long-distance signal crosstalk, ensuring that each "Unit" is authentic both physically and biologically. For real-time alignment and labeling of signals and stimuli, the system utilizes a unified timestamp injection technique, simultaneously recording processed neural signals and inserting light and electrical stimulation trigger events as synchronous metadata into the data stream in real time. Through a backend-integrated timestamp association algorithm, the system can automatically identify and label key signal segments such as "Pre-stimulus," "During-stimulus," and "Post-stimulus," thereby ensuring that each electrophysiological response can be accurately traced back to its corresponding stimulation parameters.

[0032] Furthermore, the information decoding module includes: The deep learning correlation analysis unit is used to perform deep correlation analysis on light stimulation parameters, electrical stimulation parameters and classified neuronal firing characteristics through reverse correlation technology, and decode to obtain the nonlinear mapping model between stimulus input and target neural response; The information flow standardization output unit, connected to the deep learning correlation analysis unit, is used to convert real-time acquired neural responses into standardized digital information flows based on a nonlinear mapping model, and provides a data output interface for connection with external devices.

[0033] Furthermore, such as Figure 4As shown, the information decoding module in this embodiment aims to parse and establish the correlation logic between multimodal stimulus sequences and the electrophysiological response of the optic nerve. Through deep learning and reverse correlation techniques, the system can perform deep correlation analysis on optical and electrical stimulation parameters and classified neuronal firing characteristics, decoding the mapping model between specific stimulus inputs and target neural responses. Based on this mapping model, the module can transform complex neural responses into standardized information flows and provide an open data output interface for easy connection with external devices and reverse engineering.

[0034] This embodiment provides an integrated interactive platform for multimodal stimulation and neural signal processing based on the biological eye. It coordinates four core modules—optical / electrical stimulation, information acquisition and preprocessing, and information decoding—through a unified software system. This platform overcomes the limitations of traditional independent hardware, achieving end-to-end integration from stimulus command issuance and high-frequency acquisition of electrophysiological responses to real-time decoding of neural signals. It ensures that all modules operate under a unified clock reference, controlling the system's synchronization error to the millisecond level, and constructing a complete closed-loop real-time interactive chain of "stimulation-response-decoding-output".

[0035] The system platform in this embodiment possesses highly flexible multimodal stimulation capabilities. Through the optical stimulation module, wavelength, intensity, geometry, and dynamic video stream are adjusted to simulate the natural light-sensing input at the biological visual level, compensating for the non-physiological limitations of single electrical stimulation. Simultaneously, the electrical stimulation module applies adjustable artificial pulses to simulate information transmission between neurons, achieving highly specific and precise activation. Furthermore, the platform supports simultaneous photoelectric stimulation, using software to achieve spatiotemporal synchronous control of the dual-modal physical field. This deeply integrates the physiological coding advantages of optical stimulation with the high spatiotemporal response characteristics of electrical stimulation, achieving more natural and precise control of the biological eye.

[0036] This embodiment utilizes deep learning algorithms and inverse correlation technology to achieve in-depth analysis of the correlation logic between multimodal stimulus sequences and the electrophysiological response of the optic nerve. This module can correlate the physical characteristics of optical and electrical stimuli with classified neuronal firing patterns, automatically decoding and constructing a nonlinear mapping model between specific stimulus inputs and target neural responses. Using this model, the platform can quantify the impact of artificial intervention signals on optic nerve activity in real time, providing core algorithmic support for the system to achieve millisecond-level precise "stimulus-response" monitoring.

[0037] Example 2 Based on the same inventive concept, this embodiment also provides a method for multimodal stimulation and neural signal processing based on a biological eye, including: Based on the global synchronization clock signal generated by the unified master clock, optical stimulation signals and electrical stimulation signals are generated synchronously, and photoelectric composite stimulation is applied to biological samples to obtain the stimulated response of biological samples. Based on a unified master clock signal, the neurophysiological signals generated after biological samples are stimulated are synchronously acquired, and the neurophysiological signals are processed by artifact removal, filtering and spike potential signal extraction to obtain preprocessed neural pulse data. Based on the preprocessed neural impulse data, feature association analysis is performed using deep learning and reverse correlation techniques to establish a mapping model between stimulus parameters and neural responses, and a standardized decoded information stream is output.

[0038] Furthermore, the process of simultaneously generating optical stimulation signals and electrical stimulation signals includes: A light stimulation pattern is generated using the light stimulation submodule; A multi-channel electrical pulse sequence is generated through the electrical stimulation submodule; Pre-compensation and synchronization alignment are performed on the firing timing of the light stimulation pattern and the electrical pulse sequence to achieve strict alignment between the light spot projection and the starting phase of the electrical pulse within millisecond-level error; The process of generating a light-stimulated pattern includes: The vector graphics of the virtual stimulus canvas are mapped to the hardware address of the physical light source array in real time based on the affine transformation algorithm, and the optical path geometric distortion is compensated to generate a pixel-level mask matrix. Based on a nonlinear brightness mapping lookup table, the preset light intensity command is accurately mapped to the driving current or pulse width of the light source through Gamma correction, thereby achieving linear control of the light power density. The ping-pong buffer scheduling algorithm ensures the strict continuity and temporal determinism of dynamic stimuli on the time axis through microsecond-level pulse width modulation sequences and parallel frame preloading mechanisms.

[0039] Furthermore, the process of generating a multi-channel electrical pulse sequence includes: Based on the impedance monitoring feedback algorithm, the output range of the digital-to-analog converter is dynamically adjusted according to the constant current or constant voltage physical amplitude set by the user, and the injected compensation charge is automatically calculated to maintain the electrochemical balance of the tissue interface. Based on the electrode gating matrix mapping logic, specific independent channels or channel combinations in MEAs are activated in real time according to the geometric distribution of biological samples to generate localized stimulation topology. Based on nested loop counting scheduling logic, pulse width, pulse interval and loop count are defined with microsecond-level precision to generate highly repeatable custom stimulation waveforms.

[0040] Furthermore, the process of artifact removal, filtering, and spike potential signal extraction of neurophysiological signals includes: The acquired raw electrophysiological signals are filtered by an adaptive bandpass filter to remove baseline drift and high-frequency environmental noise, and the filtered signal is obtained. Based on the stimulation trigger time stamp provided by the unified master clock signal, the average artifact waveform characteristics at the moment of electrical stimulation are calculated in real time, and the artifact removal of the filtered signal is performed by the inverse superposition technique to obtain a pure neural signal. Spike potential waveforms were extracted from pure neural signals using an adaptive thresholding method. Principal component analysis and spatial consistency checks were performed on the spike potential waveforms. A spatial penalty factor was applied based on the physical distance between electrodes, and neuronal unit signals with spatial authenticity were obtained by classification. Light stimulation triggering events and electrical stimulation triggering events are inserted into the data stream in real time as synchronous metadata, and key signal segments before, during and after stimulation are automatically labeled using a timestamp association algorithm.

[0041] The multimodal stimulation and neural signal processing method based on biological eyes provided in this embodiment has all the advantages of the multimodal stimulation and neural signal processing system based on biological eyes provided in Embodiment 1.

[0042] Example 3 This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0043] Example 4 This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0044] Example 5 This embodiment also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0045] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal stimulation and neural signal processing system based on a biological eye, characterized in that, include: The unified master clock module is used to generate a global synchronization clock signal, providing a unified time base reference for the entire system; A multimodal stimulation module, connected to the unified master clock module, is used to synchronously generate optical stimulation signals and electrical stimulation signals under the control of the global synchronization clock signal according to preset stimulation parameters, and apply photoelectric composite stimulation to the biological sample to obtain the stimulated response of the biological sample. The high-frequency acquisition and preprocessing module is connected to the unified master clock module and is used to synchronously acquire the neurophysiological signals generated after the biological sample is stimulated under the control of the global synchronization clock signal, and to perform artifact removal, filtering and spike potential signal extraction processing on the neurophysiological signals to obtain preprocessed neural pulse data. The information decoding module, connected to the high-frequency acquisition and preprocessing module, is used to perform feature association analysis based on the preprocessed neural impulse data using deep learning and reverse correlation techniques, establish a mapping model between stimulus parameters and neural responses, and output a standardized decoded information stream.

2. The system according to claim 1, characterized in that, The multimodal stimulation module includes: The photostimulation submodule is used to generate photostimulation patterns based on spatial location parameters, physical light intensity parameters, and time series parameters. The electrical stimulation submodule is used to generate multi-channel electrical pulse sequences based on stimulation mode, spatial location selection parameters, and waveform timing parameters. The optocoupler scheduling submodule is connected to the photostimulation submodule, the electrical stimulation submodule, and the unified master clock module, respectively. It is used to pre-compensate and synchronize the timing of the photostimulation signal and the electrical stimulation signal to ensure that the light spot projection and the starting phase of the electrical pulse are strictly aligned within the millisecond error range.

3. The system according to claim 2, characterized in that, The photostimulation submodule includes: The spatial coding unit, with a built-in affine transformation algorithm, is used to map the vector graphics of the virtual stimulus canvas to the hardware address of the physical light source array in real time, and to compensate for optical path geometric distortions to generate a pixel-level mask matrix. The light intensity control unit is connected to the spatial coding unit and has a built-in nonlinear brightness mapping lookup table, which is used to accurately map the preset light intensity command into the driving current or pulse width of the light source through Gamma correction, and linearly control the light power density. The timing scheduling unit is connected to the spatial coding unit and the optical intensity control unit respectively. It has a built-in ping-pong buffer scheduling algorithm, which is used to achieve strict continuity and temporal determinism of dynamic stimuli on the time axis through microsecond-level pulse width modulation sequence and parallel frame preloading mechanism.

4. The system according to claim 2, characterized in that, The electrical stimulation submodule includes: The drive mode mapping unit has a built-in impedance monitoring feedback algorithm, which is used to dynamically adjust the output range of the digital-to-analog converter according to the constant current or constant voltage physical amplitude set by the user, and automatically calculate the injected compensation charge to maintain the electrochemical balance of the tissue interface. The spatial addressing gating unit is connected to the driving mode mapping unit and has built-in electrode gating matrix mapping logic. It is used to activate specific independent channels or channel combinations in MEAs in real time according to the geometric distribution of biological samples to generate localized stimulation topology. The waveform timing encoding unit, connected to the spatial addressing gating unit, has a built-in nested loop counting scheduling logic, which is used to define the pulse width, pulse interval and loop count with microsecond-level precision to generate highly repeatable custom stimulus waveforms.

5. The system according to claim 1, characterized in that, The high-frequency acquisition and preprocessing module includes: The adaptive filtering unit is used to filter the acquired raw electrophysiological signals through an adaptive bandpass filter, filtering out baseline drift and high-frequency environmental noise to obtain the filtered signal. The real-time artifact removal unit is connected to the adaptive filtering unit. It is used to calculate the average artifact waveform characteristics at the moment of electrical stimulation in real time according to the stimulation trigger time stamp provided by the unified master clock module, and to use the anti-phase superposition technique to remove artifacts from the filtered signal to obtain a pure neural signal. The spike potential detection and classification unit is connected to the real-time cancellation unit for electrical artifacts. It is used to extract spike potential waveforms from the pure neural signals using an adaptive threshold method, perform principal component analysis and spatial consistency checks on the spike potential waveforms, apply a spatial penalty factor based on the physical distance between electrodes, and classify and obtain neuronal unit signals with spatial authenticity. A unified time stamp injection unit is connected to the multimodal stimulation module and the unified master clock module, respectively. It is used to insert light stimulation trigger events and electrical stimulation trigger events as synchronous metadata into the data stream in real time while recording neural signals, and automatically mark key signal segments before, during and after stimulation through a timestamp association algorithm.

6. The system according to claim 1, characterized in that, The information decoding module includes: The deep learning correlation analysis unit is used to perform deep correlation analysis on light stimulation parameters, electrical stimulation parameters and classified neuronal firing characteristics through reverse correlation technology, and decode to obtain the nonlinear mapping model between stimulus input and target neural response; The information flow standardization output unit, connected to the deep learning association analysis unit, is used to convert the real-time acquired neural responses into a standardized digital information flow according to the nonlinear mapping model, and to provide a data output interface for connection with external devices.

7. A method for multimodal stimulation and neural signal processing based on biological eyes, characterized in that, include: Based on a global synchronization clock signal generated by a unified master clock, optical stimulation signals and electrical stimulation signals are generated synchronously, and photoelectric composite stimulation is applied to the biological sample to obtain the stimulated response of the biological sample. Based on the unified master clock signal, the neurophysiological signals generated after the biological sample is stimulated are synchronously acquired, and the neurophysiological signals are processed by artifact removal, filtering and spike potential signal extraction to obtain preprocessed neural pulse data. Based on the preprocessed neural impulse data, feature association analysis is performed using deep learning and reverse correlation techniques to establish a mapping model between stimulus parameters and neural responses, and a standardized decoded information stream is output.

8. The method according to claim 7, characterized in that, The process of simultaneously generating optical and electrical stimulation signals includes: A light stimulation pattern is generated using the light stimulation submodule; A multi-channel electrical pulse sequence is generated through the electrical stimulation submodule; The firing timing of the light stimulation pattern and the electrical pulse sequence is pre-compensated and synchronized to achieve strict alignment between the light spot projection and the starting phase of the electrical pulse within millisecond-level error. The process of generating a light-stimulated pattern includes: The vector graphics of the virtual stimulus canvas are mapped to the hardware address of the physical light source array in real time based on the affine transformation algorithm, and the optical path geometric distortion is compensated to generate a pixel-level mask matrix. Based on a nonlinear brightness mapping lookup table, the preset light intensity command is accurately mapped to the driving current or pulse width of the light source through Gamma correction, thereby achieving linear control of the light power density. The ping-pong buffer scheduling algorithm ensures the strict continuity and temporal determinism of dynamic stimuli on the time axis through microsecond-level pulse width modulation sequences and parallel frame preloading mechanisms.

9. The method according to claim 8, characterized in that, The process of generating a multi-channel electrical pulse sequence includes: Based on the impedance monitoring feedback algorithm, the output range of the digital-to-analog converter is dynamically adjusted according to the constant current or constant voltage physical amplitude set by the user, and the injected compensation charge is automatically calculated to maintain the electrochemical balance of the tissue interface. Based on the electrode gating matrix mapping logic, specific independent channels or channel combinations in MEAs are activated in real time according to the geometric distribution of biological samples to generate localized stimulation topology. Based on nested loop counting scheduling logic, pulse width, pulse interval and loop count are defined with microsecond-level precision to generate highly repeatable custom stimulation waveforms.

10. The method according to claim 7, characterized in that, The process of artifact removal, filtering, and spike potential signal extraction of the aforementioned neurophysiological signals includes: The acquired raw electrophysiological signals are filtered by an adaptive bandpass filter to remove baseline drift and high-frequency environmental noise, and the filtered signal is obtained. Based on the stimulation trigger time stamp provided by the unified master clock signal, the average artifact waveform characteristics at the moment of electrical stimulation are calculated in real time, and the filtered signal is stripped of artifacts using the inverse superposition technique to obtain a pure neural signal. The spike potential waveform is extracted from the pure neural signal using an adaptive thresholding method. Principal component analysis and spatial consistency checks are performed on the spike potential waveform. A spatial penalty factor is applied based on the physical distance between electrodes to classify and obtain neuronal unit signals with spatial authenticity. Light stimulation triggering events and electrical stimulation triggering events are inserted into the data stream in real time as synchronous metadata, and key signal segments before, during and after stimulation are automatically labeled using a timestamp association algorithm.

Images