Multi-modal neural signal synchronous acquisition method and system for Chinese speech brain-computer interface, electronic device and storage medium

Abstract

The application discloses a kind of multi-modal neural signal synchronous acquisition methods and systems for Chinese speech brain-computer interface, electronic equipment and storage medium.The method comprises: configuring multiple heterogeneous data sources, each data source realizes unified data source interface;For each data source, create an independent data acquisition pipeline;Multi-data source is realized by global synchronization controller Three-stage synchronous start;When data arrives, use host high-precision timer to mark microsecond-level timestamp, unify heterogeneous data to the same time reference, and execute clock drift compensation algorithm;Data is distributed to consumer by lock-free queue.The application realizes sub-millisecond cross-modal synchronization accuracy by three-stage synchronous start protocol, host high-precision timestamp marking, clock drift compensation algorithm;By supporting the lock-free queue of multiple consumer subscriptions, high-throughput and low-latency data transmission is realized;Through unified data source interface, good system scalability is realized.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface data acquisition technology, specifically relating to a method and system for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces, as well as electronic devices and storage media. Background Technology

[0002] Brain-computer interfaces (BCIs) enable direct communication between the human brain and external devices by collecting signals of brain neural activity, providing new communication and control pathways for patients with movement disorders. For research on speech-based BCIs in tonal languages ​​such as Mandarin Chinese, in order to accurately depict the entire process from "vocal intention generation → speech organ movement → acoustic signal output," it is necessary to simultaneously collect temporal data from multiple modalities, including cortical neural electrical signals, laryngeal and perioral surface electromyographic signals, facial and lip-shape video, and speech acoustic signals.

[0003] In practical system setup, researchers typically use dedicated neural signal acquisition devices (such as the Blackrock Neurotech Cerebus system) to record neural signals and mark them between the amplifier and the stimulation / recording device using TTL (Transistor-Transistor Logic) trigger lines. Simultaneously, they utilize multimodal middleware based on Network Time Protocol (NTP) (such as Lab Streaming Layer, LSL) to perform time alignment of video, audio, and other data streams across different computing nodes. However, when facing applications like Chinese speech decoding, which are highly sensitive to cross-modal temporal relationships and often require long-term continuous acquisition, existing BCI data acquisition platforms still have significant shortcomings in areas such as multi-source data synchronization accuracy, long-term clock drift suppression, and unified management of heterogeneous devices.

[0004] (1) Insufficient multimodal support

[0005] Existing neural signal acquisition systems are primarily designed for neurophysiological signals and lack native support for multimedia signals such as video and audio. Researchers need to use multiple independent acquisition systems to collect various signals separately and perform offline alignment after data acquisition. This not only increases the complexity of experimental operations but may also introduce time alignment errors that are difficult to eliminate.

[0006] (2) Limited synchronization accuracy

[0007] Existing synchronization schemes typically rely on hardware triggering (TTL) or Network Time Protocol (NTP). While hardware triggering can provide microsecond-level initial alignment, it cannot address the cumulative clock drift caused by frequency differences among devices (such as neural amplifier crystal oscillators, USB camera clocks, and sound card clocks) during long-term acquisition, leading to severe data misalignment in the later stages of acquisition. Furthermore, the LSL multimodal data stream synchronization scheme based on NTP typically has a synchronization error of 1-5 milliseconds and is susceptible to network load and system scheduling jitter. For Chinese speech BCI, speech acoustic features (such as tone and formants) are updated with frame shifts of 10-20 milliseconds, and the temporal relationship between neural activity and muscle movement is also on the order of milliseconds. Existing synchronization accuracy is insufficient to meet the requirements of high-precision neural-speech correlation analysis.

[0008] (3) Real-time performance bottleneck

[0009] With the introduction of multimodal sensors, the system's data throughput has jumped from less than 1 MB / s (megabytes per second) in traditional pure neural signal acquisition to tens of MB / s, increasing the data bandwidth by one to two orders of magnitude. At this throughput, the system needs to process tens of thousands of data packets per second, leaving only tens of microseconds for processing each packet. Traditional architectures use mutexes to protect shared buffers and involve multiple memory copies (user-mode / kernel-mode switching, serialization copying). In high-throughput scenarios with multiple consumers accessing concurrently, frequent lock contention and memory bus occupancy can cause severe latency spikes, leading to data packet backlog or even loss, compromising data integrity.

[0010] (4) Insufficient scalability

[0011] Existing data acquisition software is often hard-coded for specific hardware models, lacking a universal Hardware Abstraction Layer (HAL). When experimental requirements change and new sensors (such as eye trackers or microphone arrays) need to be introduced, problems such as incompatible driver interfaces and inconsistent data formats are encountered. This usually requires refactoring the underlying code, resulting in high system maintenance costs and poor reusability. Summary of the Invention

[0012] To address the problems of asynchronous clocks between heterogeneous devices, large cumulative drift during long-term acquisition, severe lock contention during high-throughput acquisition, and poor multimodal scalability in existing technologies, this invention provides a method and system for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces, as well as an electronic device and storage medium.

[0013] This invention achieves sub-millisecond-level synchronous acquisition of neural signals, electromyographic signals, video, and audio through a three-stage spin-synchronous startup protocol, a drift compensation algorithm based on a host unified time base, and a high-performance concurrent pipeline that supports zero copy. It is particularly suitable for speech brain-computer interfaces with extremely high timing accuracy requirements.

[0014] The technical solution adopted in this invention is as follows:

[0015] A method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces includes the following steps:

[0016] S1: Configure neural signal data sources, electromyographic signal data sources, video data sources, and audio data sources;

[0017] S2: Create an independent data acquisition pipeline for each data source in step S1. Each pipeline contains an acquisition thread and an output queue.

[0018] S3: The data acquisition pipeline in step S2 is started synchronously in three stages through a global synchronization mechanism;

[0019] S4: After the three-stage synchronous startup, a timer is used to mark the arrival timestamp of the host when the data from each data source arrives at the host;

[0020] S5: Utilizes an online linear regression algorithm, combined with the underlying hardware's native timestamp, to perform real-time correction of the host's arrival timestamp, generating a unified time base and marking it on data packets from various data sources, thus obtaining data packets containing the unified time base;

[0021] S6: Distribute data packets with a unified time base to multiple consumer modules in parallel through a lock-free output queue;

[0022] S7: Each consumer module processes the received data packets independently and sends the processed data packets to the Chinese speech brain-computer interface.

[0023] This invention achieves sub-millisecond cross-modal synchronization accuracy through a three-stage synchronous startup protocol, high-precision host timestamp marking, and a clock drift compensation algorithm; it achieves high-throughput and low-latency data transmission through a lock-free output queue supporting multiple consumer subscriptions; and it achieves good system scalability through a unified data source interface. This invention is applicable to application scenarios requiring simultaneous acquisition of multimodal neural signals, such as Chinese speech brain-computer interfaces.

[0024] In step S3, the data acquisition pipeline of step S2 is started synchronously in three stages through a global synchronization mechanism, specifically including:

[0025] Phase 1, Registration and Readiness: Waiting for all acquisition threads in the data acquisition pipeline to complete resource allocation and initialization and report readiness;

[0026] The second stage is synchronous triggering: the global synchronizer broadcasts a start signal, each acquisition thread adopts a spin-wait mode, and simultaneously triggers the underlying hardware corresponding to each data source to start acquisition.

[0027] The third stage is the start confirmation: each acquisition thread resynchronizes after confirming that the underlying hardware has actually started transmitting data.

[0028] The purpose of this invention's three-stage synchronous startup mechanism is to eliminate timing uncertainties during the startup process: In traditional direct startup methods, the creation and initialization times of each acquisition thread are unpredictable (typically on the order of milliseconds to tens of milliseconds), leading to significant deviations in the actual start times of acquisition from different data sources. The three-stage synchronous startup separates thread creation from data acquisition startup, ensuring that all data sources begin acquisition at the same time: The first stage completes all time-consuming resource allocation and hardware connection operations, and each acquisition thread reports readiness after initialization, pre-processing uncertain initialization overhead; In the second stage, after all acquisition threads report readiness, the global synchronizer broadcasts a startup signal, and each thread uses spin-wait to achieve microsecond-level synchronous triggering; The third stage confirms that the underlying hardware has actually started transmitting data, avoiding data loss or timestamp errors due to hardware response delays. Through this three-stage mechanism, the startup time difference between data sources can be controlled within 1 millisecond, providing reliable initial alignment for subsequent unified time benchmark calibration.

[0029] In step S4, when data from each data source arrives at the host, a timer is used to mark the host's arrival timestamp, specifically including:

[0030] Record the native timestamps returned by the underlying hardware corresponding to each data source, and use a timer to mark the host arrival timestamps to form a sequence of timestamp pairs.

[0031] The timestamp pair sequence of this invention establishes a mapping relationship between the device clock domain and the host clock domain, providing a data foundation for subsequent clock calibration; by continuously recording timestamp pairs, the cumulative clock drift caused by crystal oscillator frequency differences can be detected and quantified in real time; based on the timestamp pair sequence, online linear regression can accurately estimate the drift coefficient and initial offset, improving the time alignment accuracy of multimodal data streams to the sub-millisecond level.

[0032] In step S5, an online linear regression algorithm is used, combined with the underlying hardware's native timestamp, to perform real-time correction of the host's arrival timestamp, generating a unified time base, including:

[0033] An online linear regression algorithm for updating sequences using timestamps is employed to obtain the clock drift coefficients and initial offsets of each underlying hardware relative to the host. Then, the original timestamp of the current data packet is dynamically mapped to the calibrated host timestamp to generate a unified time base.

[0034] Each underlying hardware device uses an independent crystal oscillator clock, which drifts with the host clock. Simultaneously, the host arrival timestamp is affected by transmission delay jitter, which is non-negative but exhibits abnormal peaks, making a single host timestamp unsuitable as a unified time reference. To address these issues, we assume the hardware device clock... With host clock There is a linear mapping relationship between them: ,in This is the native timestamp of the hardware device, in microseconds. δ is the host timestamp in microseconds; k is the drift coefficient, a dimensionless linear scaling factor representing the frequency ratio of the hardware device clock to the host clock; b is the initial offset in microseconds, representing the difference in start time between the hardware device clock and the host clock; δ is the non-negative delay jitter in microseconds, representing the random positive deviation introduced during data transmission due to factors such as system scheduling and network latency. The online linear regression algorithm uses the timestamp from step S4 to estimate the parameters of the sequence in real time. and By fitting a large number of timestamp pairs, delay jitter can be effectively smoothed. By analyzing the impact of the clock mapping, a stable clock mapping relationship is extracted, and the native timestamps of hardware devices are mapped to calibrated unified timestamps. ,in It serves as a unified time reference after calibration, measured in microseconds, thereby compensating for clock drift, filtering out delay jitter, and achieving sub-millisecond cross-modal time alignment.

[0035] In step S5, the online linear regression algorithm is either the online recursive least squares method or the sliding window linear regression algorithm.

[0036] Online recursive least squares method maintains cumulative statistics, including: sample count n (the number of timestamp pairs processed), and the sum of device timestamps. (in represent ), host timestamp accumulation (in represent ), Sum of squared device timestamps and cross product and Upon arrival of each new data point, the above statistics are updated incrementally in O(1) time complexity, and the regression parameters are calculated using the least squares formula. and : , When each new data point arrives... The algorithm incrementally updates regression parameters with constant time complexity (i.e., execution time is independent of data size), eliminating the need to store historical data sequence pairs and maintaining a fixed memory footprint. Its advantages include: support for real-time online correction; memory usage is independent of acquisition duration, making it suitable for long-term continuous acquisition; and the ability to obtain statistically optimal estimates using all historical data under stable drift coefficient conditions.

[0037] The sliding window linear regression algorithm maintains a fixed-size buffer of timestamp pairs, where N is the sliding window size and represents the number of timestamp pairs stored in the buffer; N is a positive integer. When new data arrives, the oldest data is removed and the new data is added, and regression parameters are calculated only based on the most recent N timestamp pairs. Its effects include: quickly adapting to non-stationary changes in the drift coefficient; reducing the long-term impact of early outliers; and adjusting the window size... It allows for a flexible trade-off between estimation smoothness and response speed.

[0038] The two algorithms complement each other: the online recursive least squares method provides a globally optimal and stable estimate, while the sliding window algorithm provides fast adaptive capability. The system can flexibly select or combine these algorithms based on the stability of the acquisition environment.

[0039] In step S5, the data packet containing the unified time base adopts a zero-copy serialization format.

[0040] Zero-copy serialization formats, through a specially designed binary layout, allow the serialized byte stream to be directly interpreted and accessed in memory without data decoding or memory copying. Its benefits include: significantly reduced CPU overhead, enabling systems to support higher data throughput under the same hardware conditions; support for on-demand access, allowing consumers to read only the fields of interest without fully parsing the entire data structure; maintaining data format consistency throughout storage and transmission; and supporting cross-language data exchange, facilitating system integration.

[0041] In step S7, independent processing is performed using data storage or real-time neural signal feature extraction.

[0042] A system for implementing the aforementioned method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces includes:

[0043] The data source module includes a neural signal acquisition unit, an electromyography signal acquisition unit, a video acquisition unit, and an audio acquisition unit. Each acquisition unit implements a unified data source interface.

[0044] The pipeline module creates an independent acquisition pipeline for each data source. Each pipeline includes an acquisition thread, a timestamp marker component, and an output queue that supports multiple consumer subscriptions.

[0045] The global synchronization module manages the registration and synchronous startup of all data acquisition threads, and uses a three-stage synchronization protocol to ensure that all data sources start acquiring data at the same time.

[0046] The Stream Manager module manages the lifecycle of all pipelines in a unified manner, providing batch open, batch start, batch stop, and batch close operations;

[0047] The consumer module reads data from the output queue of the acquisition pipeline, including storage consumers, network distribution consumers, and real-time feature extraction consumers.

[0048] An electronic device includes: a processor and a memory, wherein the memory stores a computer program;

[0049] When the computer program runs, it executes the method described above.

[0050] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0051] Specifically, a method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces includes the following steps: S1: Configure multiple heterogeneous data sources (including but not limited to neural signals, electromyography, video, and audio), with each data source implementing a unified data source abstraction interface. This interface defines standard device control and data acquisition methods to shield underlying driver differences; S2: Build an independent high-concurrency acquisition pipeline for each data source. Each pipeline includes a dedicated acquisition thread, a timestamp marker component, and a lock-free output queue based on a Single Producer Multiple Consumer (SPMC) model.

[0052] S3: Executes the three-phase synchronization startup protocol via the global synchronization controller:

[0053] Phase 1 (Registration and Readiness): All acquisition threads complete resource allocation and register with the global synchronization controller, and report their readiness status after initialization is complete;

[0054] Second stage (spin trigger): The global synchronization controller broadcasts a start command, and each acquisition thread uses the CPU spin wait mechanism to trigger the underlying hardware at a microsecond-level response speed.

[0055] Phase 3 (Startup Confirmation): Confirm that all hardware has actually started transmitting data, ensuring that the physical starting point of the multimodal data stream is aligned on the timeline.

[0056] S4: When the data arrives in the host memory, use the host's high-precision timer to uniformly mark the microsecond-level original host arrival timestamp;

[0057] S5: Perform an online clock drift compensation step, using an online linear regression algorithm and combining the device's native timestamp to perform real-time correction on the original host arrival timestamp, generating a unified time reference to eliminate jitter and accumulated drift.

[0058] S6: The lock-free output queue is used to distribute data packets containing a unified time base to multiple parallel consumer modules in parallel. The data packets adopt a serialization format that supports zero copy (such as FlatBuffers, an efficient cross-platform serialization library that supports direct access to serialized data without parsing and copying), and support simultaneous disk storage, network distribution and real-time feature extraction, avoiding the latency caused by repeated memory copying.

[0059] S7: Each consumer module processes the received data independently:

[0060] Storage consumer: directly writes raw data packets to disk, including a unified time base;

[0061] Real-time consumer analysis: Performs real-time neural signal feature extraction.

[0062] Network transport consumers: Distribute serialized data packets to different Ethernet terminals via message queues.

[0063] Compared with the prior art, the present invention has the following advantages:

[0064] (1) Sub-millisecond synchronization accuracy and long-term stability

[0065] Existing technologies (such as LSL) rely on network clock synchronization, and the synchronization accuracy is limited by network jitter, typically ranging from 1 to 5 ms. Furthermore, they cannot solve the problem of accumulated clock drift during long-term acquisition. This invention reduces the startup time error to the microsecond level through a three-stage spin synchronization startup mechanism. Combined with an online linear drift compensation algorithm, it effectively suppresses accumulated clock errors within a 24-hour acquisition cycle, achieving sub-millisecond cross-modal synchronization accuracy and meeting the requirement for precise alignment between neural activity and speech frames (10-20 ms).

[0066] (2) High throughput and low latency concurrent architecture

[0067] For data streams of tens of MB / s generated by high-definition video and high-frequency neural signals, this invention abandons the traditional mutex lock mechanism and adopts a lock-free output queue combined with zero-copy serialization technology, effectively avoiding lock contention and repeated memory copying. This architecture significantly reduces CPU utilization while ensuring high throughput and controls end-to-end processing latency to the millisecond level, meeting the real-time requirements of closed-loop BCI.

[0068] (3) Flexible decoupling and scalability

[0069] By defining a unified data source abstraction interface, this invention completely decouples the upper-layer acquisition logic from the lower-layer hardware driver. New heterogeneous devices (such as eye trackers and depth cameras) only need to implement the standard interface to connect to the system, without modifying the core pipeline code, greatly reducing system maintenance costs and expansion difficulty.

[0070] (4) Built-in real-time neural decoding support

[0071] This invention incorporates a real-time feature extraction module, which can calculate decoding features such as firing rate and local field potential (LFP) band energy online and directly output them to downstream decoders, effectively supporting closed-loop speech synthesis and brain-computer interaction experiments.

[0072] (5) Multi-consumer parallel processing capability

[0073] This invention employs a single producer, multiple consumers (SPMC) lock-free output queue architecture, supporting multiple consumer modules to simultaneously subscribe to the same data stream. Tasks such as data storage, network distribution, and real-time feature extraction can be executed in parallel without blocking each other. Consumers can dynamically join or leave without affecting the acquisition process, significantly improving the system's flexibility and resource utilization efficiency.

[0074] (6) Cross-language data exchange capability

[0075] The zero-copy serialization format used in this invention supports direct reading by multiple programming languages ​​(C++, Python, Java, etc.). The data maintains a consistent format throughout the acquisition, storage, and transmission stages, eliminating the need for an additional format conversion layer and facilitating integration with third-party analysis tools and experimental control programs. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating the multimodal neural signal synchronous acquisition method of the present invention.

[0077] Figure 2 This is a connection topology diagram of the hardware system of the present invention.

[0078] Figure 3 This is a diagram showing the overall architecture of the software system of this invention.

[0079] Figure 4 This is a timing diagram of the three-stage synchronous startup protocol of the present invention.

[0080] Figure 5 This is a schematic diagram of the clock drift compensation principle of the present invention.

[0081] Figure 6 This is a schematic diagram of the multi-consumer output queue structure of the present invention.

[0082] Figure 7 This is a schematic diagram of the data packet format of the present invention.

[0083] Figure 8 This is an architecture diagram of the real-time feature extraction module of the present invention.

[0084] Figure 9 This is a performance comparison chart between the present invention and existing solutions. Detailed Implementation

[0085] To describe the present invention more specifically, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in various embodiments of the present invention can be combined accordingly without mutual conflict.

[0086] The process of synchronous acquisition of multimodal neural signals. (e.g., ...) Figure 1 As shown, a method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces includes the following steps:

[0087] S1: Configure multiple heterogeneous data sources. These data sources include, but are not limited to, neural signal data sources (such as the Blackrock Neurotech Cerebus system), electromyography signal data sources (such as the Intan system), video data sources (such as industrial cameras), and audio data sources (such as professional sound cards). Each data source implements a unified IDataSource abstract interface to shield the differences in underlying drivers.

[0088] S2: Build independent data acquisition pipelines. Create an independent data acquisition pipeline for each data source. Each pipeline includes: an acquisition thread, bound to a specific CPU core, responsible for reading data from the hardware; and an output queue, supporting a lock-free circular buffer for single producer multiple consumers (SPMC).

[0089] S3: Three-stage synchronous startup. Registration and readiness: After the acquisition thread completes initialization, it reports readiness to the global synchronization controller; Synchronization trigger: The global synchronization controller broadcasts a startup signal, and all threads use the CPU spin-wait mechanism to simultaneously call the hardware startup interface within microseconds; Startup confirmation: Confirm that the hardware actually generates data flow to ensure the alignment of the physical acquisition starting point.

[0090] S4: Unified Time Base Stamping. Upon arrival of data from each data source in the host memory, a microsecond-level arrival timestamp is immediately created using a high-precision host timer (QueryPerformanceCounter, QPC) to serve as a unified time base for aligning heterogeneous data.

[0091] S5: Online clock drift compensation. Using a built-in online drift compensation algorithm (such as online recursive least squares or sliding window linear regression), combined with the device hardware timestamp, the original arrival timestamp obtained in step S4 is corrected in real time to generate a unified time base after eliminating jitter and drift.

[0092] S6: Parallel Data Distribution. Data packets containing a unified time base are distributed in parallel to multiple consumer modules through the lock-free output queue of the SPMC. The data packets use a zero-copy serialization format, such as FlatBuffers (an efficient cross-platform serialization library that supports direct access to serialized data without parsing and copying), to improve distribution efficiency.

[0093] S7: Independent Consumer Processing. Each consumer module processes the received data independently. The storage consumer directly saves the data (including the unified time base) to disk. The real-time processing consumer directly extracts real-time neural signal features (such as calculating the pulse firing rate) based on the unified time base and sends the extracted features to the downstream decoder.

[0094] Hardware system environment setup, such as Figure 2 As shown, the hardware system includes heterogeneous data sources and connection methods.

[0095] The host is a high-performance workstation, equipped with an Intel Core i9-13900K processor (which uses its high clock speed to reduce spin wait error), 64GB DDR5 memory, and a 2TB NVMe SSD (to ensure high throughput writes).

[0096] Neural signal source: Blackrock Neurotech Cerebus system, connected via gigabit Ethernet, sampling rate set at 30kHz, acquiring 128 channels of cortical EEG.

[0097] Electromyography (EMG) signal source: Intan RHD2000 system, connected via USB 3.0, sampling rate 2kHz, acquiring 32 channels of surface EMG.

[0098] Video signal source: A Logitech C920 Pro webcam (or a similar USB camera) was used, connected via USB 2.0 interface, with a resolution of 1920x1080 and a frame rate of 30fps (frames per second) to capture lip movements.

[0099] Audio signal source: Use a Focusrite Scarlett 2i2 USB audio interface (or equivalent sound card), connected via USB 2.0 interface, with the sampling rate set to 48kHz, to acquire voice signals.

[0100] Software architecture and synchronization mechanisms. For example... Figure 3 As shown, the software system of this invention adopts a modular layered architecture design, consisting of a data source layer, a pipeline layer, a flow management layer, and an application layer, arranged from bottom to top. This layered design achieves decoupling of hardware abstraction, concurrent data acquisition, and business logic.

[0101] S1: Data Source Layer. This layer is located at the bottom of the architecture and is responsible for direct interaction with heterogeneous hardware devices. Hardware Abstraction: Defines a unified IDataSource interface, including lifecycle methods such as open(), start(), stop(), and close(), as well as the get_packet() data acquisition method. Driver Adaptation: Implements specific driver adapters for different hardware (such as Cerebus, Intan, industrial cameras, and sound cards). The adapters shield the differences in the underlying Software Development Kit (SDK) (such as Socket communication, USB transmission, and shared memory mapping), providing a unified data packet format to the upper layers.

[0102] S2: Pipeline Layer. This layer is the core execution unit for data acquisition, maintaining an independent high-performance acquisition pipeline for each data source. Acquisition Thread: Each pipeline has an independent acquisition thread, which is bound to a specific CPU core to reduce context switching. Immediately after acquiring a data packet from the data source layer, the acquisition thread calls the host high-precision timer (QPC) to add the original host timestamp to the data packet. Subsequently, the thread executes an online drift compensation algorithm to smooth and calibrate the current timestamp based on historical timestamp data, generating a unified time base that eliminates jitter and drift. Finally, the processed data packet containing the precise timestamp is pushed into a lock-free circular buffer in SPMC mode for upper-layer modules to subscribe to.

[0103] S3: Stream Management Layer. This layer acts as a bridge between the upper and lower layers of the system, with the Stream Manager as its core component. The Stream Manager is responsible for the lifecycle management of all pipelines, including creation, configuration, destruction, and status monitoring. It provides a unified control interface, allowing the application layer to start or stop the acquisition of all devices in batches with a single click. In addition, the management layer also includes a global synchronization controller, used to implement the core "three-stage synchronous startup protocol" of this invention. It maintains a global atomic flag, where all acquisition threads in the "ready" state spin and wait until they receive a unified trigger signal, thereby achieving microsecond-level cross-device startup alignment.

[0104] S4: Application Layer. This layer is located at the top of the architecture and consists of various consumer modules that subscribe to the pipeline layer's output queues and process data. The storage consumer is responsible for writing high-throughput raw data streams (including precise timestamps) to disk in real time, supporting multi-volume storage. The network distribution consumer is built on the high-performance ZeroMQ (Zero Message Queue) message library and adopts a publish-subscriber (PUB-SUB) model. This module encapsulates FlatBuffers data packets in its internal lock-free queue into ZMQ message frames and broadcasts them in real time via TCP (Transmission Control Protocol) (for remote monitoring on a local area network) or IPC (Inter-Process Communication) (for cross-process analysis on the local machine). ZeroMQ's background I / O thread automatically handles connection management and flow control, ensuring high throughput and low latency when multiple clients are accessing the system. The real-time analytics consumer directly utilizes the precise timestamps provided by the pipeline layer to filter neural signals, perform spike sorting, or extract frequency band energy, directly outputting feature vectors usable by the decoder.

[0105] S5: Three-stage spin-synchronous startup protocol. To address millisecond-level jitter caused by operating system scheduling, this invention disables interrupts and employs a CPU spin mechanism during the startup phase. For example... Figure 4As shown, the specific implementation is as follows: First, after completing initialization, all acquisition threads report readiness to the global controller and enter a spin loop, continuously checking the global start flag. Second, after confirming that all threads are ready, the global control thread sets the global start flag (global_start_flag). Finally, due to the cache coherence protocol, each CPU core detects the flag change within nanoseconds, and all acquisition threads almost simultaneously exit the loop and call the hardware trigger interface, achieving almost absolutely synchronous startup.

[0106] S6: Clock drift compensation algorithm. For long-term time drift caused by different hardware crystal oscillator frequency deviations (e.g., a sound card 48kHz might actually be 48.002kHz), this invention uses an online linear regression algorithm for compensation (e.g., ...). Figure 5 (As shown). This step is integrated into the acquisition thread of the pipeline layer, ensuring that all downstream consumers receive a consistent and accurate time reference. The algorithm input is the real-time acquired timestamp pairs, i.e., the device's native timestamps. and host arrival timestamp The algorithm uses online recursive least squares or sliding window linear regression to update the slope (representing the drift coefficient) and intercept (representing the initial offset) of the linear equation in real time. ,in The drift coefficient, This is the initial offset. The algorithm output is the corrected host timestamp. This timestamp compensates for long-term drift and latency jitter while preserving the high precision of hardware sampling, ensuring data alignment over long periods of time.

[0107] Real-time feature extraction and data stream mechanism.

[0108] S1: Lock-free zero-copy transmission mechanism based on a ring buffer. To solve the problem of uncontrollable latency caused by lock contention in traditional data acquisition systems under high load, the lock-free output queue of this invention is implemented using a lock-free ring buffer in a single producer-multiple consumer (SPMC) mode, such as... Figure 6As shown. In this mechanism, only one copy of the data packet is allocated and stored in memory, avoiding repeated data copying. The specific workflow is as follows: (1) The producer (i.e., the data source acquisition thread) writes the newly acquired data packet into the buffer and updates the write index (write_index) through an atomic operation. This operation is atomic, meaning that it is immediately visible between multi-core CPUs and does not require the intervention of the operating system's mutex lock. (2) Multiple consumers (such as storage modules, network modules, and analysis modules) independently maintain their own read index (read_index). Consumers can determine whether new data has arrived by comparing their own read index with the global write index. (3) Data packets adopt zero-copy serialization formats that support "in-place access" (such as FlatBuffers, etc.) Figure 7 (As shown). The packet structure includes fields such as source identifier, sequence number, unified timestamp, and data payload. This means that consumers can directly read binary fields in memory as object properties without first decoding and deserializing them as required by JSON (JavaScript Object Notation, a text-based data exchange format) or Protocol Buffers (a binary serialization format). This design eliminates the computational overhead of "kernel-to-user space memory copying" and "serialization / deserialization," which is key to achieving microsecond-level transmission latency.

[0109] S2: Real-time feature extraction pipeline for neural signals. Based on the data flow, this invention incorporates a real-time feature extraction module specifically designed for brain-computer interface applications. This module, acting as a consumer, subscribes to and processes neural signal data streams in real time (e.g., neural signal data streams). Figure 8 (1) Pulse signal processing path: For the high-frequency raw neural signal with a sampling rate of 30kHz, the module first performs threshold detection to extract the neuronal action potential (Spike Event). Then, according to the preset time window (e.g., 20ms, corresponding to the speech frame shift), the number of pulses fired in each channel within the window is counted, and the firing rate feature vector is calculated. (2) Field potential signal processing path: At the same time, the raw signal is subjected to anti-aliasing low-pass filtering and downsampled to 1kHz to obtain the local field potential (LFP) signal. Then, the LFP signal is subjected to short-time Fourier transform (STFT) to extract the energy spectral density of the Gamma band (30-80Hz). These two features (firing rate and Gamma energy) are packaged in real time and sent to the downstream decoder to drive speech synthesis or external device control, thereby realizing the real-time closed loop of "acquisition-processing-feedback".

[0110] System performance evaluation and comparative verification, such as Figure 9 As shown. To verify the effectiveness of the present invention, a benchmark test platform including photoelectric sensors, a high-precision function generator, and a multimodal acquisition device was built. In this embodiment, the LSL middleware and BCI2000 platform, which are widely used in the field of brain-computer interfaces, were selected as the core comparison objects, and in-depth evaluations were conducted on synchronization accuracy, startup alignment, real-time latency, and system stability.

[0111] S1: Cross-modal synchronization error assessment. This test uses a function generator to send synchronization pulses to the analog inputs of all acquisition devices, and evaluates the synchronization accuracy by calculating the time difference between the recorded signals of each modality (e.g., ...). Figure 9 (As shown in the upper left). The benchmark LSL scheme uses an NTP-based network time synchronization mechanism. According to existing research and experimental data, its accuracy can reach the 1ms level in an ideal LAN environment; however, in high-load network environments involving high-definition video streaming, due to the failure of the network latency symmetry assumption, its average synchronization residual degrades to 3-5ms, accompanied by latency jitter. In contrast, the scheme of this invention abandons the network estimation mechanism and uses a host high-precision timer (QPC) to uniformly stamp the data when it arrives in memory, and performs long-term drift correction. Experimental results show that under the same high load conditions, the average synchronization error of this invention is stable within 0.2ms (standard deviation less than 0.05ms), completely eliminating the impact of network load on synchronization accuracy, and significantly improving accuracy.

[0112] S2: 24-hour clock drift comparison. Evaluate the long-term clock stability of the system under a 24-hour continuous data acquisition scenario (e.g., ...). Figure 9 (As shown in the upper right corner). Due to the crystal oscillator error of consumer-grade electronic components (approximately 50 ppm, ppm being one part per million), under physical benchmarks without algorithmic intervention, 24 hours of operation will result in a cumulative time deviation of approximately 4.3 seconds. This will lead to severe audio-visual desynchronization or neural signal alignment misalignment in long-term recorded data. This invention employs an online linear drift compensation algorithm for correction. Experimental results show that this algorithm can effectively eliminate instantaneous jitter caused by system scheduling, controlling the residual time drift after 24 hours to within 1 ms, ensuring accurate data alignment in long-term neuroscience experiments.

[0113] S3: End-to-End Transmission Latency Comparison. This test measures the time difference from the generation of the physical signal to the availability of the data packet for downstream algorithms, i.e., the software transmission latency. The BCI2000, due to its block-based processing mechanism (typical block size of 32-64 sampling points), inherently has a latency typically in the range of 40-60ms. LSL, to smooth network jitter, usually requires application-layer buffering, resulting in a typical software transmission latency of 10-15ms. This invention's solution benefits from lock-free output queues and a zero-copy architecture, eliminating the need for data to pass through the network protocol stack and undergo multiple memory copies. The measured average transmission latency is only 2.5ms, significantly better than mainstream platforms, and can meet the real-time requirements of high-frequency closed-loop neural modulation.

[0114] S4: Full-load packet loss rate comparison. Under full-load conditions of approximately 195 MB / s high throughput (128-channel neural signals + 32-channel electromyography + 1080p@30fps video + 48kHz audio), the system stability (e.g., packet loss rate) is evaluated. Figure 9 (As shown in the lower right corner). When processing high-bandwidth video streams, the LSL scheme suffers from a packet loss rate of approximately 0.5% under high load due to multi-threaded lock contention and memory copy overhead. In contrast, the single-producer-multiple-consumer (SPMC) lock-free architecture employed in this invention eliminates lock contention between threads. Experimental results show that, under the same full load conditions, the packet loss rate of this invention is less than 0.2%, verifying its stability in complex multimodal large data stream scenarios.

[0115] S5: Heterogeneous Device Setup Offset Analysis. This test evaluates the "physical startup time difference" caused by the different initialization times of heterogeneous devices. Existing technologies such as LSL and BCI2000 typically employ asynchronous startup strategies, which cannot overcome the inherent initialization latency of the hardware. Publicly available technical documents and test data show that the typical startup offset between heterogeneous devices ranges from 7ms to 36ms. This results in physical misalignment of the acquired data in the initial stage, usually requiring complex offline calibration using external hardware trigger signals. This invention introduces a "three-stage spin-synchronous startup protocol," separating hardware initialization from triggering, and utilizing the atomic spin locks of multi-core CPUs to achieve microsecond-level simultaneous triggering. Experimental results show that the physical startup time deviation of each mode is compressed to within 1ms, achieving true "hard synchronous" startup and avoiding post-calibration.

[0116] In summary, compared with existing mainstream solutions such as LSL and BCI2000, this invention overcomes core pain points such as network jitter sensitivity, startup misalignment, and high latency through architectural innovation, providing reliable technical support for high-precision, multimodal speech brain-computer interface research.

[0117] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the division of steps or modules in the systems and methods described above is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0118] The embodiments described above are only some preferred embodiments of the present invention, and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces, characterized in that, Includes the following steps: S1: Configure neural signal data sources, electromyographic signal data sources, video data sources, and audio data sources; S2: Create an independent data acquisition pipeline for each data source in step S1. Each pipeline contains an acquisition thread and an output queue. S3: The data acquisition pipeline in step S2 is started synchronously in three stages through a global synchronization mechanism, specifically including: Phase 1, Registration and Readiness: Waiting for all acquisition threads in the data acquisition pipeline to complete resource allocation and initialization and report readiness; The second stage is synchronous triggering: the global synchronizer broadcasts a start signal, each acquisition thread adopts a spin-wait mode, and simultaneously triggers the underlying hardware corresponding to each data source to start acquisition. The third stage, start confirmation: each acquisition thread synchronizes again after confirming that the underlying hardware has actually started transmitting data. S4: After the three-stage synchronous startup, a timer is used to mark the arrival timestamp of the host when the data from each data source arrives at the host; S5: Utilizes an online linear regression algorithm, combined with the underlying hardware's native timestamp, to perform real-time correction of the host's arrival timestamp, generating a unified time base and marking it on data packets from various data sources, thus obtaining data packets containing the unified time base; S6: Distribute data packets with a unified time base to multiple consumer modules in parallel through the output queue; S7: Each consumer module processes the received data packets independently and sends the processed data packets to the Chinese speech brain-computer interface.

2. The method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces according to claim 1, characterized in that, In step S4, when data from each data source arrives at the host, a timer is used to mark the host arrival timestamp, including: Record the native timestamps returned by the underlying hardware corresponding to each data source, and use a timer to mark the host arrival timestamps to form a sequence of timestamp pairs.

3. The method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces according to claim 2, characterized in that, In step S5, an online linear regression algorithm is used, combined with the underlying hardware's native timestamp, to perform real-time correction of the host's arrival timestamp, generating a unified time base, including: An online linear regression algorithm for updating sequences using timestamps is employed to obtain the clock drift coefficients and initial offsets of each underlying hardware relative to the host. Then, the original timestamp of the current data packet is dynamically mapped to the calibrated host timestamp to generate a unified time base.

4. The method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces according to claim 3, characterized in that, In step S5, the online linear regression algorithm is either the online recursive least squares method or the sliding window linear regression algorithm.

5. The method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces according to claim 1, characterized in that, In step S5, the data packet containing the unified time base adopts a zero-copy serialization format.

6. The method for synchronous acquisition of multimodal neural signals for Chinese speech brain-computer interfaces according to claim 1, characterized in that, In step S7, independent processing is performed using data storage or real-time neural signal feature extraction.

7. A system for implementing the multimodal neural signal synchronous acquisition method for Chinese speech brain-computer interface as described in any one of claims 1 to 6, characterized in that, include: The data source module includes a neural signal acquisition unit, an electromyography signal acquisition unit, a video acquisition unit, and an audio acquisition unit. Each acquisition unit implements a unified data source interface. The pipeline module creates an independent acquisition pipeline for each data source. Each pipeline includes an acquisition thread, a timestamp marker component, and an output queue that supports multiple consumer subscriptions. The global synchronization module manages the registration and synchronous startup of all data acquisition threads, and uses a three-stage synchronization protocol to ensure that all data sources start acquiring data at the same time. The Stream Manager module manages the lifecycle of all pipelines in a unified manner, providing batch open, batch start, batch stop, and batch close operations; The consumer module reads data from the output queue of the acquisition pipeline, including storage consumers, network distribution consumers, and real-time feature extraction consumers.

8. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores a computer program; When the computer program is executed, it performs the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images