Dynamic fusion-based neural signal real-time decoding and video synchronization system and method

Abstract

The invention discloses a neural signal real-time decoding and video synchronization system and method based on dynamic fusion, and the system comprises a neural signal collection module which is used for obtaining a neural signal data stream; the video acquisition module is used for synchronously acquiring a video data stream; the real-time decoding and dynamic fusion module is used for receiving the neural signal data stream and the video data stream, executing a decoding algorithm in real time to extract neural features, fusing current and historical neural features and video context information, and generating a fused real-time decoding result; the video synchronous generation module is used for instantly generating or selecting corresponding video content according to the real-time decoding result, and ensuring that the video content and the neural signal decoding state at the current moment are strictly and synchronously output; and the synchronous output module is used for outputting and displaying the video content in real time. The problems of high delay, cross-modal mismatch and emergency behavior adaptation of neural signal decoding and video feedback are solved, and a high-reliability technical scheme is provided for brain-computer interface and neural behavioristics research.

Description

Technical Field

[0001] This invention belongs to the field of biomedical engineering and multimedia data processing technology, specifically relating to a system and method for real-time decoding of neural signals and video synchronization based on dynamic fusion. Background Technology

[0002] Neural signal decoding, a core component of brain-computer interfaces (BCIs), aims to convert acquired neural electrical signals (such as EEG, fMRI, or implanted electrode signals) into executable commands or state identifiers. Current technologies primarily rely on multi-channel high-frequency sampling devices (≥1kHz) to acquire signals, and then use methods such as adaptive filtering and independent component analysis (ICA) to reduce noise. These are then combined with time-frequency analysis (such as wavelet transform) or deep learning models (such as CNN / LSTM) to extract features such as motor intent and cognitive state. However, traditional software-based decoding processes suffer from significant latency (often >100ms), making it difficult to meet real-time interaction requirements. For example, in motor intent decoding tasks, Kalman filters and MLP models suffer from high computational complexity and cannot effectively capture temporal dependencies, resulting in insufficient real-time processing of high-throughput neural signals (e.g., 500+ channels), often requiring sacrifices in resolution or sampling rate to compress latency.

[0003] In the field of neural signal and video synchronization, existing systems face the core challenge of cross-modal temporal mismatch. Traditional solutions rely on system clock alignment (such as the NTP protocol), but the mismatch between the neural signal sampling rate (kHz level) and the video frame rate (typically 30-60fps) leads to temporal drift errors >10ms. Furthermore, video content generation is often based on pre-rendered template libraries, making it difficult to adapt to non-predefined states in neural decoding output (such as sudden behaviors in animal experiments). While Generative Adversarial Networks (GANs) can dynamically synthesize videos, their model inference latency is >50ms, and they lack a tightly coupled architecture with the neural decoder. For example, while pre-stored video libraries can achieve millisecond-level access by matching decoding states through hash mapping, they lack adaptability to non-predefined actions; and while real-time GAN synthesis can generate target action trajectories, its high computational load causes output lag, disrupting cross-modal synchronization.

[0004] Existing solutions have three major limitations: (1) Although the timestamp synchronization scheme achieves neural-behavior alignment through offline indexing, it does not solve the problem of matching real-time video generation with dynamic neural states; (2) Multi-screen video synchronization technology relies on pre-trained neural networks to correct deviations, but it is not optimized for the high-throughput characteristics of neural signals and does not integrate a content generation module; (3) Dynamic rendering technologies (such as GANs) lack hardware acceleration support, and in complex scenes, the latency is easily aggravated by the competition for computing resources. For example, although Neuralink's invasive BCI enables thought typing (20 characters per minute), the video feedback latency is still >80ms, which cannot meet the requirements of high-precision interaction.

[0005] In summary, the current technological ecosystem faces three major contradictions: high-precision decoding versus real-time requirements, predefined templates versus dynamic behavior adaptation, and independent processing versus cross-modal collaboration. There is an urgent need for a dynamic architecture that integrates algorithm and hardware co-optimization to compress end-to-end latency (<5ms), build a seamless mapping mechanism from neural states to video content, and establish a global clock reference (such as the PTP protocol) to achieve microsecond-level synchronization error control. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a system and method for real-time decoding and video synchronization of neural signals based on dynamic fusion. Through an end-to-end closed-loop architecture, it realizes millisecond-level collaborative control of intracranial neural signals and external behavioral videos of experimental animals, providing a high-precision tool for brain-computer interface and neurobehavioral research, and is especially suitable for high-precision real-time collaborative control of neural activity and external visual feedback in brain-computer interface applications.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a real-time neural signal decoding and video synchronization system based on dynamic fusion, comprising: a neural signal acquisition module, a video acquisition module, a real-time decoding and dynamic fusion module, a video synchronization generation module, and a synchronization output module; The neural signal acquisition module is used to acquire intracranial neural signals of experimental animals and generate neural signal data streams after preprocessing. The video acquisition module is used to synchronously acquire video data of the external environment or behavior and generate a video data stream after preprocessing. The real-time decoding and dynamic fusion module is used to receive neural signal data streams and video data streams, execute decoding algorithms in real time to extract neural features, and fuse current neural features with historical neural features and video context information to generate fused real-time decoding results. The video synchronization generation module is used to generate or select corresponding video content in real time based on the real-time decoding results, and ensure that the video content is strictly synchronized with the current neural signal decoding state. The synchronization output module is used to output and display strictly synchronized video content in real time.

[0008] Preferably, the neural signal acquisition module supports a dual-mode transmission mechanism, which enables high-bandwidth wired data transmission via Ethernet protocol and encrypted wireless transmission via low-power Bluetooth or WiFi module. In addition, the dual-mode transmission mechanism also supports a dynamic switching strategy, which automatically activates Ethernet mode when a wired connection is detected, while wireless mode is activated as a mobile experimental scenario or redundant link to ensure stable synchronization and secure transmission of neural signal data.

[0009] Preferably, the computation in the real-time decoding and dynamic fusion module includes three stages: feature extraction and preliminary decoding stage, multimodal dynamic fusion stage, and final decoding decision stage. Each stage is designed and computational resources are scheduled through a pipeline control module.

[0010] Preferably, in the real-time decoding and dynamic fusion module, the feature extraction and preliminary decoding stage is performed by the feature extraction unit, the multimodal dynamic fusion stage is performed by the matrix operation unit, and the final decoding decision stage is performed by the neural network inference unit. Parallel processing of cross-stage computing tasks and dynamic resource scheduling are achieved by reusing the same computing units used in each stage that consume a lot of resources.

[0011] Preferably, the feature extraction and preliminary decoding stage is used to extract neural signal features within the current time window from the neural signal data stream, and to initially generate the decoding state at the current moment, including decoding probability or potential state, based on the pre-trained decoder. The multimodal dynamic fusion stage is used to receive the neural signal features at the current moment, the decoding probability or potential state, the stored historical neural signal features and decoding state, and the contextual information in the video data stream at the current or nearby moment. The dynamic fusion strategy is applied to integrate the multimodal information and generate the fused state representation. The final decoding decision stage is used to output the final real-time decoding result, including motion instructions, cognitive states, or control commands, based on the fused state representation, and caches the decoding result, neural signal features, and decoding state for subsequent fusion.

[0012] Preferably, the pipeline control module is used to control the input and execution of the feature extraction and preliminary decoding stage, the multimodal dynamic fusion stage, and the final decoding decision stage, and to cache the output of each stage; when processing the fusion stage of the neural signal at the current time point, the neural signal at the next time point is input in parallel for feature extraction and preliminary decoding; when processing the final decoding decision of the neural signal at the current time point, the multimodal dynamic fusion of the neural signal at the next time point is performed in parallel to realize the pipeline design; different computing units are scheduled for different stages to achieve resource reuse.

[0013] Preferably, in the video synchronization generation module, a timestamp alignment mechanism including global clock calibration and frame rate adaptation is used to ensure that the video content is strictly synchronized with the current neural signal decoding state. At the same time, a pre-stored action template library is set up to support the rapid retrieval of video content corresponding to regular behaviors based on hash mapping, and a generative adversarial network is triggered in real time for sudden behaviors to dynamically synthesize the corresponding action trajectory video content.

[0014] Secondly, embodiments of the present invention also provide a method for real-time decoding and video synchronization of neural signals based on dynamic fusion, implemented using the aforementioned system for real-time decoding and video synchronization of neural signals based on dynamic fusion, comprising the following steps: The neural signal acquisition module was used to acquire intracranial neural signals from experimental animals and preprocessed them to generate a neural signal data stream. The video acquisition module synchronously collects video data of the external environment or behavior, and generates a video data stream after preprocessing. The real-time decoding and dynamic fusion module receives neural signal data streams and video data streams, executes decoding algorithms in real time to extract neural features, and fuses current neural features with historical neural features and video context information to generate a fused real-time decoding result. The video synchronization generation module generates or selects corresponding video content in real time based on the real-time decoding results, and ensures that the video content is strictly synchronized with the current neural signal decoding state. The synchronous output module is used to output and display strictly synchronized video content in real time.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described method for real-time decoding and video synchronization of neural signals based on dynamic fusion when executing the computer program.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for real-time decoding and video synchronization of neural signals based on dynamic fusion.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: The core innovation of this invention lies in overcoming the three technical contradictions in the fields of neural signal decoding and video synchronization: 1) By adopting a pipelined three-stage computing architecture and computing unit reuse strategy in the real-time decoding and dynamic fusion modules, high-precision decoding and millisecond-level real-time performance can be achieved simultaneously; 2) By dynamically aggregating neural features and video context information in the multimodal dynamic fusion stage, decoding accuracy can be effectively improved, and further, by achieving seamless switching of dynamic synthesis based on a pre-stored action template library and generative adversarial networks in the video synchronization module, the adaptability problem in sudden behavioral scenarios can be solved; 3) Relying on global clock calibration and frame rate adaptive technology, strict synchronization between kHz-level neural data and 30–100fps video streams can be achieved with an error ≤5ms. In animal neurobehavioral experiments, this system can accurately correlate paw-raising actions with neural peak potential sequences, and in brain-controlled virtual reality, it can generate intention-driven virtual limb movement feedback within 15ms, thus providing reliable technical support for neurorehabilitation and brain-computer interaction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of a real-time neural signal decoding and video synchronization system based on dynamic fusion provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the detailed processing flow of the video acquisition module provided in this embodiment of the invention; Figure 3 This is a flowchart illustrating the method for real-time decoding and video synchronization of neural signals based on dynamic fusion provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0021] The inventive concept of this invention is as follows: In view of the problems of cross-modal mismatch and delayed response to sudden behavior in existing neural signal decoding and video synchronization, this invention achieves millisecond-level collaborative control of neural signal decoding and behavioral video through a three-level pipeline dynamic fusion architecture and global clock calibration technology, so as to meet the real-time interaction needs of brain-computer interface and animal neurobehavioral research.

[0022] Figure 1 This is a schematic diagram of the structure of a real-time neural signal decoding and video synchronization system based on dynamic fusion provided in an embodiment of the present invention. Figure 1 As shown, the embodiment provides a real-time neural signal decoding and video synchronization system based on dynamic fusion, including: a neural signal acquisition module, a video acquisition module, a real-time decoding and dynamic fusion module, a video synchronization generation module, and a synchronization output module.

[0023] (1) Nerve signal acquisition module: used to acquire intracranial nerve signals of experimental animals and generate nerve signal data stream after preprocessing.

[0024] In this embodiment, a multi-channel microelectrode array is implanted into the intracranial cortex of experimental animals to capture extracellular spike potential signals. The signal-to-noise ratio is improved through multi-stage amplification, and power frequency interference is filtered out using an adaptive bandpass filter. The signal converted by a high-precision ADC is transmitted in two modes: a wired mode uses Gigabit Ethernet, while the wireless mode uses Bluetooth Low Energy or WiFi modules for encrypted transmission. Simultaneously, a dynamic switching strategy prioritizes the wired link, activating the wireless redundant channel only during mobile experiments to ensure transmission stability and security.

[0025] (2) Video acquisition module: synchronously acquires video data of external environment or behavior and generates video data stream after preprocessing.

[0026] In the embodiments, such as Figure 2 As shown, the video acquisition module process includes: the camera outputs the raw video stream; it is converted into parallel data via the MIPI interface, and then the effective image frames are extracted by CSI-2 unpacking; the PTP protocol is used to achieve clock synchronization with the neural signal; the RAW data is converted into RGB format by color interpolation, and the white balance module calibrates the color before writing it into the DDR storage unit; the YOLOv5 model is started synchronously to identify key targets in real time and output the motion trajectory coordinates and timestamp.

[0027] (3) Real-time decoding and dynamic fusion module: Receives neural signal data stream and video data stream, executes decoding algorithm in real time to extract neural features, and fuses current neural features with historical neural features and video context information to generate fused real-time decoding results.

[0028] In this embodiment, the real-time decoding and dynamic fusion module serves as the central processing engine of the system. Its computation comprises a three-stage pipeline architecture to reduce processing latency and achieve millisecond-level intent decoding and cross-modal synchronous control: a feature extraction and preliminary decoding stage (executed by the feature extraction unit), a multimodal dynamic fusion stage (executed by the matrix operation unit), and a final decoding decision stage (executed by the neural network inference unit). Each stage is controlled by a pipeline control module. During the fusion stage of processing the neural signal at the current time point, the neural signal at the next time point is input in parallel for feature extraction and preliminary decoding. During the final decoding decision of the neural signal at the current time point, the multimodal dynamic fusion of the neural signal at the next time point is performed in parallel, compressing the end-to-end latency to <10ms. By reusing the same and resource-intensive computing units used in each stage, parallel processing of cross-stage computing tasks and dynamic resource scheduling are achieved, significantly reducing hardware resource overhead.

[0029] Specifically, in the feature extraction and preliminary decoding stage, after the neural signal is filtered to remove power frequency interference through adaptive filtering, two types of core neural signal features are extracted in parallel by a dilated convolutional network: Time-domain pulse rate (SR): statistically analyzed within a specific time window ( The number of all neural peak potentials detected within () Then, divide the number of spike potentials by the size of the time window, i.e. .

[0030] γ-band energy: Quantization of 30-80Hz frequency band energy using Fast Fourier Transform (FFT) ),in, Indicates time The changing primitive neural signals, It indicates frequency, specifically referring to the range of the signal after Fourier transform.

[0031] During decoding, the pre-trained decoder generates the decoding state, including decoding probability or potential state, for the current moment based on the time-domain pulse rate and γ-band energy.

[0032] Assume there is There are 1 neuron channel, so for each time step... Construct a multidimensional feature vector As input to the decoder, This vector It integrates neural information from the time and frequency domains. Next, this vector is input into a pre-trained decoder to initially generate the decoding state.

[0033] a) Decoder architecture: For each moment The decoder receives the current feature vector. and its previous hidden state And update the current hidden state. Taking a gated loop unit as an example, its calculation process is as follows: , , , , in, , These represent updating the door and resetting the door, respectively. Indicates the candidate activation value. This represents the Sigmoid activation function. The symbol represents the hyperbolic tangent function, and ⊙ represents element-wise multiplication. , , , , , This represents the parameters that the decoder needs to learn.

[0034] b) Decoding state generation: Updated hidden status This constitutes the latent state at the current moment, encoding all historical neural information up to the current moment; it is the model's internal abstract representation of the intent. To obtain the final interpretable output, the latent state... Through a fully connected output layer, and normalized by the Softmax function, a target is generated for... Decoding probability distribution of each instruction : , , in, Represent a A dimensional probability vector, each element of which This indicates that the decoder predicts the current intent as the [number]. The probability of an instruction. , This represents the parameters that the decoder needs to learn.

[0035] c) Model training and inference: The decoder needs to be pre-trained on a large number of samples, and all parameters are optimized by minimizing the cross-entropy loss between the predicted probability and the true instruction label: , in, This represents the one-hot encoding of the actual label.

[0036] Specifically, the multimodal dynamic fusion stage receives the neural signal features at the current moment, the decoding probability or latent state, the stored historical neural signal features and decoding states, and contextual information (such as motion state, key objects, and scene semantics) from the video data stream at the current or nearby moments. The synchronized video data stream extracts joint motion trajectories and HOG gradient features using YOLOv5 and relies on the PTP protocol to ensure μs-level temporal alignment with the neural signals. The above multimodal information is integrated through the following three-level dynamic fusion strategy to generate the fused state representation: a) Attention-weighted: , in, The attention weights represent the neural-video features and are used to adaptively adjust the contribution weights of video semantics to the neural signal. This represents a query vector based on neural signal features. This represents a key vector based on contextual information in the video data stream. The superscript indicates the dimension of the key vector. This indicates transposition; when a sudden action occurs (such as jumping), the neural weights are automatically increased. Up to 120%.

[0037] b) Temporal context enhancement: 500ms of historical state is cached using an LSTM (Long Short Time Memory) network. and merge vectors , This is the state representation after fusion.

[0038] c) Fault tolerance mechanism: A transition frame is generated when the timing deviation is >5ms. ,in, Represents the weight parameters. Indicates the first Frame video.

[0039] Attention weighting and temporal context enhancement are responsible for the intelligent understanding and prediction of intentions, while the fault-tolerance mechanism monitors the brain's output and ensures the stability and robustness of the final action execution. It makes up for the shortcomings of the first two steps in extreme cases, and the three together ensure efficient, smooth, and reliable closed-loop execution from neural instructions to physical actions.

[0040] Specifically, the final decoding and decision-making stage is used to transform the fused state representation into high-precision motion commands, cognitive states, or control commands (such as joint control quantities) through kinematic calculations and PID control algorithms. Finally, it is output through multi-protocol interfaces such as EtherCAT / Modbus to realize the closed-loop execution of neural commands into physical actions.

[0041] (4) Video Synchronization Generation Module: Generates or selects corresponding video content in real time based on the real-time decoding results, and ensures that the video content is strictly synchronized with the current neural signal decoding state.

[0042] In this embodiment, the module generates strictly synchronized video feedback based on neural decoding instructions and achieves content generation through a dual-path rendering mechanism. A pre-stored action template library supports fast matching and invocation of regular behaviors (such as grabbing, stretching, and other predefined action segments) via hash mapping, with a response latency of ≤2ms. When no corresponding template is found in the decoding state, sudden behaviors (such as jumping) trigger a GAN (Generative Adversarial Network) to dynamically synthesize a video stream of the target action trajectory, with a generation latency of ≤15ms.

[0043] All video content is semantically bound to the current neural state and is strictly synchronized through a timestamp alignment mechanism: a global clock reference for neural signals and video streams is established using the PTP protocol to achieve microsecond-level system clock synchronization; the video output frame rate is dynamically adjusted according to the neural signal sampling rate to avoid timing drift caused by a fixed frame rate; when the synchronization deviation exceeds the 5ms threshold, the motion compensation interpolation algorithm automatically reconstructs the transition frame or discards the lagging frame to ensure real-time alignment across modalities and ensure the continuity of motion in the generated images.

[0044] (5) Synchronous output module: Outputs and displays strictly synchronized video content in real time.

[0045] In this embodiment, the module is used to push the video content that is strictly synchronized with the neural signal decoding state to the display terminal in real time, so as to realize real-time interaction and visual feedback between neural and video modes.

[0046] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides a method for real-time decoding and video synchronization of neural signals based on dynamic fusion, including the following steps: S1: The neural signal acquisition module is used to acquire intracranial neural signals of experimental animals and generate neural signal data streams after preprocessing.

[0047] S2 uses a video acquisition module to synchronously acquire video data of the external environment or behavior and generates a video data stream after preprocessing.

[0048] S3 uses a real-time decoding and dynamic fusion module to receive neural signal data streams and video data streams, executes decoding algorithms in real time to extract neural features, and fuses current neural features with historical neural features and video context information to generate a fused real-time decoding result.

[0049] S4 utilizes the video synchronization generation module to instantly generate or select corresponding video content based on the real-time decoding results, and ensures that the video content is strictly synchronized with the current neural signal decoding state.

[0050] S5 utilizes the synchronous output module to output and display strictly synchronized video content in real time.

[0051] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described method for real-time decoding and video synchronization of neural signals based on dynamic fusion when executing the computer program.

[0052] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for real-time decoding and video synchronization of neural signals based on dynamic fusion.

[0053] It should be noted that the method, electronic device, and computer-readable storage medium for real-time decoding and video synchronization of neural signals based on dynamic fusion provided in the above embodiments all belong to the same inventive concept as the system for real-time decoding and video synchronization of neural signals based on dynamic fusion. For details of their implementation process, please refer to the embodiments of the system for real-time decoding and video synchronization of neural signals based on dynamic fusion, which will not be repeated here.

[0054] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time neural signal decoding and video synchronization system based on dynamic fusion, characterized in that, include: The system includes a neural signal acquisition module, a video acquisition module, a real-time decoding and dynamic fusion module, a video synchronous generation module, and a synchronous output module. The neural signal acquisition module is used to acquire intracranial neural signals of experimental animals and generate neural signal data streams after preprocessing. The video acquisition module is used to synchronously acquire video data of the external environment or behavior and generate a video data stream after preprocessing. The real-time decoding and dynamic fusion module is used to receive neural signal data streams and video data streams, execute decoding algorithms in real time to extract neural features, and fuse current neural features with historical neural features and video context information to generate fused real-time decoding results. The video synchronization generation module is used to generate or select corresponding video content in real time based on the real-time decoding results, and ensure that the video content is strictly synchronized with the current neural signal decoding state. The synchronization output module is used to output and display strictly synchronized video content in real time.

2. The real-time neural signal decoding and video synchronization system based on dynamic fusion according to claim 1, characterized in that, The neural signal acquisition module supports a dual-mode transmission mechanism, enabling high-bandwidth wired data transmission via Ethernet protocol, while simultaneously achieving encrypted wireless transmission via low-power Bluetooth or WiFi modules. Furthermore, the dual-mode transmission mechanism supports a dynamic switching strategy: when a wired connection is detected, Ethernet mode is automatically activated, while wireless mode is used for mobile experimental scenarios or as a redundant link, ensuring stable synchronization and secure transmission of neural signal data.

3. The real-time neural signal decoding and video synchronization system based on dynamic fusion according to claim 1, characterized in that, The computation in the real-time decoding and dynamic fusion module includes three stages: feature extraction and preliminary decoding, multimodal dynamic fusion, and final decoding decision. Each stage is designed and computational resources are scheduled through a pipeline control module.

4. The real-time neural signal decoding and video synchronization system based on dynamic fusion according to claim 3, characterized in that, In the real-time decoding and dynamic fusion module, the feature extraction and preliminary decoding stages are performed by the feature extraction unit, the multimodal dynamic fusion stage is performed by the matrix operation unit, and the final decoding decision stage is performed by the neural network inference unit. By reusing the same computing units used in each stage that consume a lot of resources, parallel processing of cross-stage computing tasks and dynamic scheduling of resources are achieved.

5. The real-time neural signal decoding and video synchronization system based on dynamic fusion according to claim 3, characterized in that, The feature extraction and preliminary decoding stage is used to extract neural signal features within the current time window from the neural signal data stream, and to initially generate the decoding state at the current moment, including decoding probability or latent state, based on the pre-trained decoder. The multimodal dynamic fusion stage is used to receive the neural signal features at the current moment, the decoding probability or potential state, the stored historical neural signal features and decoding state, and the contextual information in the video data stream at the current or nearby moment. The dynamic fusion strategy is applied to integrate the multimodal information and generate the fused state representation. The final decoding decision stage is used to output the final real-time decoding result, including motion instructions, cognitive states, or control commands, based on the fused state representation, and caches the decoding result, neural signal features, and decoding state for subsequent fusion.

6. The real-time neural signal decoding and video synchronization system based on dynamic fusion according to claim 4, characterized in that, The pipeline control module controls the input and execution of the feature extraction and preliminary decoding stage, the multimodal dynamic fusion stage, and the final decoding decision stage, and caches the output of each stage. When processing the fusion stage of the neural signal at the current time point, the neural signal at the next time point is input in parallel for feature extraction and preliminary decoding. When processing the final decoding decision of the neural signal at the current time point, the multimodal dynamic fusion of the neural signal at the next time point is performed in parallel to realize the pipeline design. Different computing units are scheduled for different stages to achieve resource reuse.

7. The real-time neural signal decoding and video synchronization system based on dynamic fusion according to claim 1, characterized in that, In the video synchronization generation module, a timestamp alignment mechanism including global clock calibration and frame rate adaptation ensures that the video content is strictly synchronized with the current neural signal decoding state. At the same time, a pre-stored action template library is set up to support the rapid retrieval of video content corresponding to regular behaviors based on hash mapping. In response to sudden behaviors, a generative adversarial network is triggered in real time to dynamically synthesize the corresponding action trajectory video content.

8. A method for real-time decoding and video synchronization of neural signals based on dynamic fusion, implemented using the aforementioned system for real-time decoding and video synchronization of neural signals based on dynamic fusion, characterized in that... Includes the following steps: The neural signal acquisition module was used to acquire intracranial neural signals from experimental animals and preprocessed them to generate a neural signal data stream. The video acquisition module synchronously collects video data of the external environment or behavior, and generates a video data stream after preprocessing. The real-time decoding and dynamic fusion module receives neural signal data streams and video data streams, executes decoding algorithms in real time to extract neural features, and fuses current neural features with historical neural features and video context information to generate a fused real-time decoding result. The video synchronization generation module generates or selects corresponding video content in real time based on the real-time decoding results, and ensures that the video content is strictly synchronized with the current neural signal decoding state. The synchronous output module is used to output and display strictly synchronized video content in real time.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the method for real-time decoding and video synchronization of neural signals based on dynamic fusion as described in claim 8 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the method for real-time decoding of neural signals and video synchronization based on dynamic fusion as described in claim 8.

Images