A multi-modal data closed-loop processing method, device, equipment, medium and product for brain-computer interface

Abstract

The disclosure provides a brain-computer interface-oriented multi-modal data closed-loop processing method, device, equipment, medium and product, which comprises the following steps: acquiring multi-modal brain signals, and performing time alignment and parallel preprocessing on the multi-modal brain signals to obtain preprocessed multi-modal brain signals; performing mapping processing on the preprocessed multi-modal brain signals to obtain multi-modal brain region-level signals under a unified spatial reference; performing independent feature extraction on the multi-modal brain region-level signals under the unified spatial reference respectively to obtain feature data of each modality; performing feature fusion on the multi-modal feature data to obtain cross-modality brain region-level joint features; performing online decoding processing on the cross-modality brain region-level joint features, and generating a regulation and control instruction based on a brain state analysis result to drive a regulation and control execution module to execute, and the brain function state change caused by the regulation and control is perceived again through multi-modal signal acquisition, thereby realizing the whole process of multi-modal brain signal online real-time collaborative processing, fusion decoding and closed-loop control.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, device, medium, and product for multimodal data closed-loop processing oriented towards brain-computer interfaces. Background Technology

[0002] Brain-computer interfaces (BCIs) are direct communication channels connecting the brain to external devices and have become a frontier of multidisciplinary collaboration, with broad application prospects in fields such as medical rehabilitation. With the advancement of global brain science strategies, BCIs are developing towards higher precision, real-time capabilities, and closed-loop systems. The online real-time perception, decoding, and regulation of the target subject's brain functional state have become key requirements supporting closed-loop applications.

[0003] In brain-computer interface scenarios, it is necessary to collect and analyze brain function images of the target object. To address the obvious shortcomings of single-modal data analysis such as magnetic resonance imaging or optical imaging, multimodal data is collected and processed collaboratively.

[0004] However, current multimodal data processing usually relies on the acquisition of complete experimental data. Cross-modal registration, feature extraction, and signal modeling all depend on centralized calculations after the experiment, resulting in a long time delay and making online processing difficult. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, medium, and product for closed-loop processing of multimodal data for brain-computer interfaces, so as to realize online analysis of multimodal data and improve the analysis efficiency of multimodal data.

[0006] According to one aspect of this disclosure, a method for closed-loop processing of multimodal data for brain-computer interfaces is provided, comprising: Simultaneously acquire multimodal brain signals of the target object, and perform time alignment and parallel preprocessing on the multimodal brain signals to obtain preprocessed multimodal brain signals; Based on a predetermined cross-modal spatial mapping relationship, the preprocessed multimodal brain signals are mapped and processed to obtain multimodal brain region-level signals under a unified spatial reference. Independent feature extraction is performed on the multimodal brain region-level signals under the unified spatial benchmark to obtain feature data for each modality; Feature fusion is performed on multimodal feature data that have corresponding relationships on a time scale to obtain cross-modal brain region-level joint features; The cross-modal brain region-level joint features are decoded online to obtain the brain state analysis results of the target object; Based on the brain state analysis results, a control command is generated and transmitted to the control execution module for execution. The changes in brain functional state caused by the control command are perceived again through the synchronous acquisition of the multimodal brain signals to form a closed-loop operation.

[0007] According to another aspect of this disclosure, a multimodal data closed-loop processing device for brain-computer interfaces is provided, comprising: The data acquisition module is used to synchronously acquire multimodal brain signals of the target object, and perform time alignment and parallel preprocessing on the multimodal brain signals to obtain preprocessed multimodal brain signals. The spatial mapping module is used to perform mapping processing on the preprocessed multimodal brain signals based on a pre-determined cross-modal spatial mapping relationship, so as to obtain multimodal brain region-level signals under a unified spatial reference. The feature extraction module is used to independently extract features from the multimodal brain region-level signals under the unified spatial benchmark to obtain feature data for each modality. The feature fusion module is used to fuse feature data from multiple modalities that have corresponding relationships on a time scale to obtain cross-modal brain region-level joint features; The decoding module is used to perform online decoding processing on the cross-modal brain region-level joint features to obtain the brain state analysis results of the target object; The regulation module is used to generate regulation instructions based on the brain state analysis results and transmit them to the regulation execution module for execution. The changes in brain functional state caused by the regulation instructions are sensed again by the data acquisition module to form a closed-loop operation.

[0008] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multimodal data closed-loop processing method for brain-computer interfaces according to any embodiment of this disclosure.

[0009] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement the multimodal data closed-loop processing method for brain-computer interfaces as described in any embodiment of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, which, when executed by a processor, implements a multimodal data closed-loop processing method for brain-computer interfaces as described in any of the embodiments of this disclosure.

[0011] The technical solution provided in this disclosure ensures the temporal consistency of multimodal brain signals by synchronously acquiring and time-aligning the multimodal brain signals of the target object. By mapping the multimodal brain signals to a unified spatial reference, multimodal brain region-level signals located under the unified spatial reference are obtained, ensuring the spatial consistency of the multimodal brain signals. Spatial mapping is achieved based on a pre-determined cross-modal spatial mapping relationship, accelerating spatial mapping efficiency and providing a foundation for online processing of multimodal brain signals, ensuring the real-time performance of online processing. By performing feature extraction and feature fusion on the multimodal brain region-level signals respectively, different modalities of brain functional signals are collaboratively involved in brain functional state representation within the same decoding framework, thereby improving the direct usability of fused features in online decoding and control scenarios. Online decoding of the cross-modal brain region-level joint features obtained through feature fusion yields the brain state analysis results of the target object, enabling brain state analysis of the target object. Based on the brain state analysis results, control commands are generated to drive the control execution module. The changes in brain functional state caused by the control are again sensed through multimodal signal acquisition, realizing the entire process of online real-time collaborative processing, fusion decoding, and closed-loop control of multimodal brain signals.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a general flowchart of a multimodal data closed-loop processing method for brain-computer interfaces according to an embodiment of this disclosure; Figure 2 This is a schematic diagram of the hardware synchronization control and data acquisition structure of the optomagnetic multimodal brain functional imaging system provided in this embodiment of the disclosure; Figure 3 This is a timing diagram illustrating the synchronous acquisition and time alignment of optical and magnetic dual-modal brain signals provided in an embodiment of this disclosure; Figure 4This is a schematic diagram illustrating the construction of a cross-modal unified spatial reference provided in an embodiment of this disclosure; Figure 5 This is a detailed flowchart of a multimodal data closed-loop processing method for brain-computer interfaces provided in this embodiment of the disclosure; Figure 6 This is a schematic diagram of the structure of a multimodal data closed-loop processing device for brain-computer interfaces according to an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0018] This embodiment provides a multimodal data closed-loop processing method for brain-computer interfaces. This method is applicable to application scenarios of online perception, decoding and feedback of brain functional state. For example, the above application scenarios include, but are not limited to, brain-computer interface scenarios, neural modulation and neural feedback system scenarios, animal behavior research and neural mechanism research scenarios, and brain functional state detection and dynamic evaluation scenarios.

[0019] Figure 1This is a flowchart illustrating a multimodal data closed-loop processing method for brain-computer interfaces provided in this embodiment. This embodiment is applicable to situations where, in the aforementioned application scenarios, multimodal data of a target object is acquired online, and online analysis and decoding are performed to determine the brain state analysis results of the target object. This method can be executed by the multimodal data closed-loop processing device for brain-computer interfaces in this embodiment. This device can be implemented using software and / or hardware and can be integrated into electronic devices such as terminal devices, servers, computer equipment, and brain-computer interface devices. Figure 1 As shown, the method specifically includes the following steps: S110, synchronously acquire the multimodal brain signals of the target object, and perform time alignment and parallel preprocessing on the multimodal brain signals to obtain preprocessed multimodal brain signals.

[0020] S120, based on a pre-determined cross-modal spatial mapping relationship, the preprocessed multimodal brain signals are mapped to obtain multimodal brain region-level signals under a unified spatial reference.

[0021] S130, perform independent feature extraction on the multimodal brain region-level signals under the unified spatial benchmark to obtain feature data for each modality.

[0022] S140 performs feature fusion on multimodal feature data that have corresponding relationships on a time scale to obtain cross-modal brain region-level joint features.

[0023] S150, perform online decoding processing on the cross-modal brain region-level joint features to obtain the brain state analysis results of the target object; S160, Based on the brain state analysis results, a control instruction is generated and transmitted to the control execution module for execution. The changes in brain functional state caused by the control instruction are perceived again through the synchronous acquisition of the multimodal brain signals to form a closed-loop operation.

[0024] In this embodiment, the multimodal brain signals of the target object can be multimodal brain functional signals acquired online from the target object, and may include at least two modalities of brain signals. For example, multimodal brain signals may include, but are not limited to, at least two of magnetic resonance imaging (MRI) brain signals, optical brain signals, ultrasound brain signals, and electroencephalogram (EEG) brain signals. The target object here can be an animal. Optionally, the electronic device in this embodiment can establish connections with acquisition devices for multiple modal signals, which can be wired or wireless, and each modal signal can be acquired through the above connections. The input of a certain modal signal can be dynamically determined to be activated or deactivated according to the needs of different modal signals during data analysis, so as to obtain the multimodal brain signals required for the data analysis process. For example, different modal signal requirements may exist in different application scenarios.

[0025] Optionally, multimodal brain signals include magnetic resonance imaging (MRI) brain signals and optical brain signals. In some embodiments, MRI brain signals may include MRI functional brain signals, which characterize brain functional activity information at the whole-brain scale and can be acquired using MRI equipment based on blood oxygen level-dependent signals. In some embodiments, MRI brain signals may also include, but are not limited to, functional magnetic resonance imaging signals based on different contrast mechanisms, blood flow, blood volume, or metabolic-related signals acquired based on MRI, and other time-series signals related to brain functional states acquired based on MRI sequences. In some embodiments, optical brain signals may include optical functional brain signals, used to characterize changes in neural activity at the mesoscale, which can be obtained through mesoscale calcium imaging. In some embodiments, optical modal brain signals may also include, but are not limited to, optical imaging signals based on genetically encoded voltage indicator proteins for characterizing neuronal membrane potential changes; optical imaging signals based on endogenous or exogenous fluorescent labels for reflecting neural activity or related physiological processes; brain function-related signals acquired based on wide-field optical imaging, scanning optical imaging, or other forms of optical detection; and blood flow, blood oxygenation, or metabolic-related signals acquired optically.

[0026] To ensure the temporal consistency of multimodal brain signals, the multimodal brain signals of the target object are acquired synchronously. Optionally, a timestamp synchronization mechanism based on a unified high-precision clock source is used to synchronously acquire the multimodal brain signals of the target object. Specifically, the acquisition devices for multimodal brain signals are time-synchronized based on this unified clock source, and timed tasks are sent to the acquisition devices. This allows the acquisition devices to execute their respective timed tasks under time synchronization, acquiring multimodal brain signals corresponding to the same timestamp, thus achieving synchronous acquisition of multimodal brain signals. Optionally, the multimodal brain signals of the target object are acquired synchronously based on synchronization pulses or reference signals generated by an external synchronization control device. Specifically, the external synchronization control device establishes a connection with each of the multimodal brain signal acquisition devices and synchronously sends synchronization pulses or reference signals to them. Each acquisition device acquires signals upon receiving the synchronization pulses or reference signals, achieving synchronous acquisition of multimodal brain signals. Optionally, the multimodal brain signals of the target object are acquired synchronously based on a software-layer timestamp alignment and calibration mechanism. Specifically, a unified system clock allocates a global reference timestamp to the acquisition devices for multimodal brain signals. Different acquisition devices for different modalities construct their own time sequences according to the timestamp order, setting a timestamp for each frame. The time point of the modality with a lower sampling rate and larger time interval (e.g., the time point of the magnetic resonance modality) is used as the reference time anchor. Based on the reference time anchor, other modal signals are matched using timestamps to select the temporally nearest other modal signals. These selected modal signals are then calibrated to the reference time anchor on the time axis using linear interpolation or nearest neighbor interpolation. For time offsets caused by device startup delays, transmission delays, or sampling asynchrony, an online time offset compensation table is maintained, and dynamic corrections are performed based on the statistical deviation of timestamps over multiple consecutive frames to align other modal signals with the reference modality brain signal on the time axis. Optionally, the magnetic resonance modality is used as the reference modality. While acquiring brain signals from the magnetic resonance modality, a trigger signal is sent to the acquisition devices for brain signals from other modalities. The acquisition devices for brain signals from other modalities respond to the trigger signal and acquire brain signals, thereby achieving synchronous acquisition of multimodal brain signals.

[0027] In some embodiments, taking multimodal brain signals including magnetic resonance imaging (MRI) brain signals and optical imaging brain signals as an example, synchronously acquiring multimodal brain signals of a target object includes: sequentially acquiring the MRI brain signals at each time point in a preset time series using an MRI scanner; sending a trigger signal to a multimodal synchronization control module at the start time point of the preset time series; and sending an acquisition start command to an optical imaging device through the multimodal synchronization control module, so as to synchronously acquire the optical imaging brain signals through the optical imaging device.

[0028] Figure 2This is a schematic diagram of the hardware synchronization control and data acquisition structure of the opto-magnetic multimodal brain functional imaging system provided in this embodiment. A multimodal synchronization control module is configured in the electronic device and connected to both the magnetic resonance imaging device and the optical imaging device to coordinate the working timing of the two acquisition devices. According to... Figure 2 It is known that the electronic device is also equipped with a data processing and control module, which is used to realize the time alignment of multimodal brain signals. Correspondingly, the multimodal synchronization control module provides a unified timestamp signal to the data processing and control module, providing a high-precision timestamp as the basis for the time alignment of multimodal brain signals.

[0029] A preset time series corresponding to each magnetic resonance imaging (MRI) modality is established. This preset time series includes multiple time points, each of which serves as the trigger point for the MRI modality's brain signals. In other words, each time point in the preset time series triggers the MRI scanner to perform brain signal acquisition. The MRI scanner sequentially acquires MRI modality brain signals according to each time point in the preset time series and sets a timestamp for each MRI modality's brain signal. The MRI scanner transmits the MRI modality brain signals to the data processing and control module via a high-speed network interface. When the MRI scanner detects that the current moment meets the acquisition start time point in the preset time series, it triggers the acquisition of MRI modality brain signals and simultaneously sends a trigger signal to the multimodal synchronization control module. The multimodal synchronization control module sends an acquisition start command to the optical imaging device based on this trigger signal. The optical imaging device acquires signals according to this acquisition start command, obtains the optical modality brain signals, and sets a timestamp for each optical modality brain signal. The optical imaging device transmits the optical modality brain functional signal data to the data processing and control module via a wired data interface. After receiving the bimodal brain signals and their corresponding unified timestamp signals, the data processing and control module performs convergence processing and time alignment on the received bimodal brain signals, and sends the processed data to the data cache and storage module to support the online operation of the system and subsequent data backtracking and analysis. The data cache and storage module is used to cache and persistently store the multimodal brain signals and intermediate results during the acquisition and processing process, ensuring data integrity during continuous system operation.

[0030] It is understood that in other embodiments, the multimodal synchronization control module can also be connected to other modal brain signal acquisition devices. Upon receiving a trigger signal from the magnetic resonance imaging device, it synchronously sends acquisition start commands to other modal brain signal acquisition devices to achieve synchronous acquisition of multimodal brain signals. Correspondingly, the data processing and control module is connected to other modal brain signal acquisition devices to perform time synchronization of the received multimodal brain signals.

[0031] Figure 3This is a timing diagram illustrating the synchronous acquisition and time alignment of optical and magnetic dual-modal brain signals provided in an embodiment of this disclosure. Figure 3 The magnetic resonance data in the image are magnetic resonance modal brain signals, the optical image data are optical modal brain signals, and the data processing and control workstation is the data processing and control module in the above embodiment.

[0032] The data processing and control module receives magnetic resonance imaging (MRI) modal brain signals and optical modal brain signals, each carrying a timestamp. Correspondingly, time alignment of the multimodal brain signals is performed, including: mapping the MRI and optical modal brain signals to the same time axis based on a unified timestamp signal to establish a time correspondence between the MRI and optical modal brain signals; and buffering the time-aligned MRI and optical modal brain signals respectively. The unified timestamp signal can be provided by the multimodal synchronization control module to the data processing and control module. The time correspondence is established according to a preset time window. Specifically, based on the timestamp corresponding to the MRI modal brain signal, the MRI modal brain signal is mapped onto the time axis, and based on the timestamp corresponding to the optical modal brain signal, the optical modal brain signal is mapped onto the time axis, where the time axis is the same. The preset time window is then slid along the aforementioned time axis to establish a time correspondence between MRI and optical modal brain signals located within the same time window. The preset time window here corresponds to the allowable error duration. By establishing a time correspondence between the magnetic resonance modal brain signals and the optical modal brain signals that are within the allowable error duration, the temporal consistency of the multimodal brain signals is ensured.

[0033] The time-synchronized multimodal brain signals are buffered separately and preprocessed in parallel. It is understood that the preprocessing methods for different modalities can be the same or different, depending on the preprocessing requirements of each modality. Optionally, a preprocessing flow for each modality can be pre-defined, and preprocessing of different modalities can be performed in parallel based on this flow, thereby accelerating preprocessing efficiency.

[0034] In some embodiments, to decouple signal acquisition and signal processing and adapt to the differences in time update rates of different modalities of brain signals, the data processing and control module introduces a cache-based scheduling mechanism to collaboratively organize the online preprocessing of multimodal brain signals. Taking multimodal brain signals including magnetic resonance imaging (MRI) brain signals and optical brain signals as an example, the parallel preprocessing performed by the data processing and control module includes: sequentially performing preprocessing on cached MRI brain signals based on the preprocessing process of the MRI brain signals, and sequentially performing preprocessing on cached optical brain signals based on the preprocessing process of the optical brain signals. The preprocessing processes corresponding to the MRI brain signals and the optical brain signals are executed in parallel according to the data availability status. MRI brain signals can be understood as three-dimensional volume data output point-by-point. When the MRI imaging device completes signal acquisition corresponding to a time point, an MRI brain signal is obtained. Correspondingly, the MRI brain signal is in a data-available state, triggering the preprocessing of the MRI brain signal. Optical modal brain signals are processed in fixed-length time windows. When the optical image frames required to fill a time window are acquired, the data in that time window is determined to be complete. Accordingly, the optical modal brain signals are in a data-available state, triggering the preprocessing of the optical modal brain signals.

[0035] Optionally, the preprocessing of the magnetic resonance modal brain signal includes at least one of the following: estimating the rigid body motion parameters of the magnetic resonance imaging (MRI) sensor, applying motion correction to align to a reference space, resampling to the same voxel grid, spatial filtering to suppress high-frequency noise, and online updating of the GLM and extraction of statistical features. Specifically, the preprocessing of the MRI modal brain signal includes: after the MRI modal brain signal is input at each time point, using the MRI modal brain signal at a preset reference time point as a spatial reference, the head motion parameters reflecting the translation and rotation relationship are estimated by performing spatial registration calculations between the MRI modal brain signal at the current time point and the MRI modal brain signal at the preset reference time point. Based on these head motion parameters, a rigid body spatial transformation matrix is ​​constructed, and this transformation matrix is ​​applied to the MRI modal brain signal data at the current time point to achieve position correction of the volume data in three-dimensional space. After completing the rigid body spatial correction, a spatial resampling operation is performed on the MRI modal brain signal, mapping the corrected MRI modal brain signal to a spatial grid with a preset voxel resolution, so that the MRI modal brain signals at different time points maintain consistency in the spatial sampling structure. The resampled MRI modal brain signals are smoothed spatially to reduce high-frequency spatial noise and improve signal stability. The processed MRI modal brain signals are then input into an online generalized linear model (GLM). The GLM uses MRI modal brain signal data from the current and historical time points as regression inputs, updating model parameters through matrix recursion. This allows for the acquisition of MRI modal feature outputs reflecting changes in brain activity without relying on the complete time series.

[0036] Optionally, preprocessing of optical modal brain signals includes at least one of the following: classifying multi-band optical time series, estimating and correcting optical translational motion, removing slow drift caused by photobleaching, regressing blood flow components to suppress blood flow interference, and extracting neural activity features through online regression. Specifically, taking continuously acquired mesoscopic wide-field calcium fluorescence imaging data as an example, the preprocessing process for optical modal brain signals includes: during the acquisition of optical modal brain signals, optical images are generated sequentially according to alternating illumination. The continuously acquired optical images enter the online processing link according to the acquisition order. After the optical images enter the online processing link, frame classification processing is performed on the continuous optical images according to the preset alternating illumination order, dividing the optical images into two time-consistent signal sequences. One type of signal sequence mainly reflects hemodynamic changes, while the other type contains calcium-related signals. These two types of signal sequences are maintained in memory using fixed-length time windows for online calculation on a time-window basis. After signal classification and time window establishment, a preset fixed reference frame is used as a spatial reference. The spatial displacement between the optical image and the reference frame within the current time window is estimated to obtain motion parameters reflecting the rigid body translation relationship. Based on these motion parameters, spatial correction processing is performed on the optical image within the time window. This motion correction process only involves the estimation and application of translation parameters. After motion correction, online correction processing for photobleaching effect is performed on the optical modality brain signal. This processing estimates the variation trend of pixel-level signals within a fixed time window, removes low-frequency slowly changing components in real time, and performs local mean compensation on the signal after trend removal to maintain the stability of the signal amplitude scale. After photobleaching correction, a dual-channel linear regression correction method is introduced into the online processing link to reduce the interference of hemodynamic changes on calcium-related signals. Within each fixed time window, the reference signal channel is used as a hemodynamic reference quantity, and the main signal channel is used as the signal to be corrected. Online linear regression is performed on the two-channel time series of corresponding pixels within the window to obtain the regression coefficients of the reference signal to the main signal. The blood flow-related component is calculated based on the regression coefficient and subtracted from the main signal in real time to obtain the hemodynamically corrected calcium-related fluorescence signal. The regression coefficient is continuously updated over time windows using a recursive update method. After obtaining the hemodynamically corrected calcium-related fluorescence signal, its relative fluorescence change is calculated online. Within each fixed time window, the local baseline fluorescence intensity is estimated for the pixel-level fluorescence signal, and the current fluorescence change is normalized relative to this baseline to form a standardized characterization of relative fluorescence change.

[0037] Based on the above embodiments, in order to ensure the spatial consistency of multimodal brain signals, spatial mapping processing is performed on the multimodal brain signals to obtain multimodal brain region-level signals located under a unified spatial reference, thereby achieving accurate brain region-level correspondence of multimodal brain signals and providing a foundation for subsequent feature fusion of multimodal brain signals.

[0038] By establishing pre-defined cross-modal spatial mapping relationships, preprocessed multimodal brain signals are spatially mapped to a unified spatial reference. These cross-modal spatial mapping relationships include spatial mapping relationships corresponding to each modality. Each modality's spatial mapping relationship can be understood as a spatial transformation relationship from the original acquisition space corresponding to that modality to the unified spatial reference, and this spatial mapping relationship can be in the form of a spatial transformation matrix. Taking multimodal brain signals including magnetic resonance imaging (MRI) and optical imaging (OI) brain signals as an example, the pre-defined cross-modal spatial mapping relationships include spatial mapping relationships corresponding to the MRI modality and those corresponding to the OI. The MRI modality spatial mapping relationship is used to map the MRI brain signals to the unified spatial reference, and the OI spatial mapping relationship is used to map the OI brain signals to the unified spatial reference.

[0039] Based on a predetermined cross-modal spatial mapping relationship, the preprocessed multimodal brain signals are mapped to obtain multimodal brain region-level signals under a unified spatial reference. This includes: mapping the preprocessed magnetic resonance modal brain signals based on the spatial mapping relationship corresponding to the magnetic resonance modality to obtain magnetic resonance modal brain region-level signals under a unified spatial reference; and mapping the preprocessed optical modal brain signals based on the spatial mapping relationship corresponding to the optical modality to obtain optical modal brain region-level signals under a unified spatial reference.

[0040] In this embodiment, before online acquisition and processing of multimodal brain signals of the target object, a cross-modal spatial mapping relationship is predetermined, and this cross-modal spatial mapping relationship remains unchanged during the online processing. The spatial consistency of multimodal signals can be achieved by directly calling the cross-modal spatial mapping relationship, providing a spatial transformation basis for the online processing of multimodal brain signals and improving processing efficiency.

[0041] Optionally, the process of determining the spatial mapping relationship corresponding to the magnetic resonance modality includes: acquiring structural magnetic resonance images, standard brain atlas images, and functional magnetic resonance reference images; registering the functional magnetic resonance reference images and the standard brain atlas images to the space where the structural magnetic resonance images are located; and determining the spatial mapping relationship corresponding to the magnetic resonance modality based on the registration relationship. Here, the structural magnetic resonance image can be understood as a three-dimensional image reflecting the structural morphology of the brain, acquired through magnetic resonance imaging of a target object. For example, it can be a T1-weighted magnetic resonance image or a T2-weighted magnetic resonance image, and the space where the structural magnetic resonance image is located is used as a unified spatial reference. The functional magnetic resonance reference image can be understood as a single-frame image selected from functional magnetic resonance time-series data. For example, it can be a randomly selected single-frame functional magnetic resonance image, or a pre-defined representative single-frame functional magnetic resonance image, such as the first frame or average frame in the functional magnetic resonance time-series data, typically acquired using echo-planar imaging sequences, to reflect the spatial distribution of brain function corresponding to blood oxygenation level-dependent signals. Functional magnetic resonance (fMRI) reference images, serving as spatial representatives of fMRI signals, are used for registration with structural magnetic resonance (SMR) images to establish a spatial mapping relationship between functional signals and a unified space. Standard brain atlas images can be understood as standardized MRI template images with brain region segmentation labels, including anatomical divisions and spatial annotations of various brain regions, such as, but not limited to, the motor cortex and visual cortex. Optionally, the brain region segmentation scheme in standard brain atlas images can be constructed based on a large amount of population brain data. Optionally, the brain region segmentation scheme in standard brain atlas images can be determined according to different resolutions or granularities, and can be determined according to brain region segmentation requirements. Optionally, the brain region segmentation scheme in standard brain atlas images can also be implemented based on regions of interest, functional networks, or custom spatial partitioning methods.

[0042] Using the space of the structural MRI image as the target space, the functional MRI reference image is aligned with the structural MRI image through spatial registration. This yields a spatial transformation matrix from the original space of the functional MRI reference image to the target space of the structural MRI image, representing the spatial mapping relationship corresponding to the MRI modes. Using the space of the structural MRI image as the target space, a standard brain atlas image is configured within the space of the structural MRI image, and brain region segmentation labels from the standard brain atlas image are registered within the target space of the structural MRI image, ensuring that brain region segmentation labels are configured within a unified spatial reference.

[0043] By mapping the magnetic resonance modal brain signals according to the aforementioned spatial mapping relationship, the original magnetic resonance space is mapped to a unified spatial reference. Based on the brain region segmentation labels in the unified spatial reference, the mapped brain signals are then mapped to corresponding brain regions, yielding the magnetic resonance modal brain region-level signals located under the unified spatial reference. These brain region-level signals under the unified spatial reference can be understood as time-series signals representing the overall activity level of each brain region, obtained by aggregating the magnetic resonance signals of all voxels within each brain region, divided according to standard brain atlases, under the unified spatial reference.

[0044] Optionally, the process of determining the spatial mapping relationship corresponding to the optical mode includes: projecting the structural magnetic resonance image to obtain a two-dimensional projected image, wherein the projection angle of the two-dimensional projected image corresponds to the optical imaging angle of the optical imaging device; acquiring an optical reference image; and performing image registration between the optical reference image and the two-dimensional projected image based on multimodal annotations to obtain the spatial mapping relationship corresponding to the optical mode. The optical reference image can be understood as a standard reference image acquired by the optical imaging device to represent the original acquisition spatial angle and field of view of the optical mode, typically a wide-field fluorescence imaging image of the cortical surface.

[0045] By projecting a structural magnetic resonance (SMR) image from a top-view perspective, a two-dimensional (2D) projected image is obtained. The projection angle of this 2D projected image corresponds to the optical imaging angle of the optical imaging device, achieving spectral alignment between the 2D projected image in the magnetic resonance mode and the optical reference image. Multimodal markers simultaneously identified in both the magnetic resonance and optical modes are used to perform image registration between the optical reference image and the 2D projected image, obtaining the spatial mapping relationship corresponding to the optical mode. This spatial mapping relationship can be understood as a spatial transformation relationship from the original optical acquisition plane to the unified spatial reference of the structural magnetic resonance.

[0046] Optionally, the multimodal marker can be a short segment of PE tube filled with fluorescent solution embedded in an agarose gel above the cortex during simultaneous imaging. The location of this PE tube can be acquired by both magnetic resonance and optical imaging. Understandably, the multimodal marker can also be other different anatomical or functional landmarks, as long as they can be used for cross-modal alignment.

[0047] Figure 4 This is a schematic diagram illustrating the construction of a cross-modal unified spatial reference provided in an embodiment of this disclosure.

[0048] Given the spatial mapping of multimodal brain signals, and considering the differences in time update rate and signal dynamic characteristics among the multimodal brain signals, a cross-modal joint modeling method based on branch processing and feature fusion is constructed while maintaining the time update rhythm of each multimodal signal. This method is used to achieve online decoding of brain states. The online decoding process disclosed herein can be understood as real-time decoding process with high processing efficiency.

[0049] Each modality has a corresponding feature processing branch. Each feature processing branch is used to extract features from brain region-level signals of a specific modality. Optionally, a feature processing branch can be understood as a feature processing module, which is configured with rules or algorithms for feature extraction from brain region-level signals of a specific modality. Inputting a brain region-level signal of a specific modality into the corresponding feature processing branch enables feature extraction, yielding feature data for that modality. For example, each feature processing branch may include a feature statistical model, a machine learning model, etc., which are trained models with feature extraction capabilities matching the modality features. For example, each feature processing branch may include a feature extraction algorithm matching the modality features, such as a region of interest (ROI) identification algorithm or a brain region feature extraction algorithm. It is understood that feature data for different modalities may differ within the same application scenario, and feature data for the same modality may also differ across different application scenarios. Taking brain-computer interface scenarios as an example, the characteristic data of magnetic resonance modality may include, but are not limited to, at least one of the following: the time series mean characteristics of blood oxygenation level dependent on signal in each brain region, used to characterize the overall activation level of each brain region at the current time point; the regression coefficient characteristics of each brain region based on the output of the online generalized linear model, used to characterize the response intensity of each brain region to the task paradigm; the statistical characteristics of each brain region based on the output of the online generalized linear model, such as t-values ​​or z-values, used to characterize the statistical significance of activation in each brain region; and the functional connectivity characteristics between brain regions, such as the signal correlation coefficient between brain regions calculated based on a sliding time window, used to characterize the cooperative activity patterns between brain regions. The characteristic data of the optical modality may include, but are not limited to, at least one of the following: the time-series mean characteristics of the relative fluorescence changes of each brain region, used to characterize the overall level of neural activity in each brain region within the current time window; the peak characteristics or peak frequency characteristics of the relative fluorescence changes of each brain region, used to characterize the instantaneous intensity and temporal dynamics of neural activity in each brain region; the time-frequency domain characteristics of the relative fluorescence changes of each brain region, such as specific frequency band energy characteristics extracted based on short-time Fourier transform or wavelet transform, used to characterize the frequency distribution characteristics of neural activity in each brain region; and the spatial correlation characteristics between brain regions, used to characterize the coordinated neural activity patterns between adjacent or related brain regions at the mesoscale.

[0050] Taking multimodal brain signals, including magnetic resonance imaging (MRI) and optical imaging (OI) brain signals, as an example, feature extraction branches are set up for MRI and OI respectively. Feature extraction is performed on the MRI modality brain region-level signals through the feature extraction branch corresponding to the MRI modality, yielding MRI modality feature data. Similarly, feature extraction is performed on the OI brain region-level signals through the feature extraction branch corresponding to the OI modality, yielding OI modality feature data. The feature extraction processes for different modalities are independent and executed in parallel, accelerating feature extraction efficiency.

[0051] Feature fusion of different modalities avoids the shortcomings of single-modal feature data. The complementary nature of multimodal feature data improves the comprehensiveness of the data, providing a data foundation for improving the accuracy of brain state analysis results. To ensure temporal consistency during multimodal feature data fusion, feature fusion is performed on multimodal feature data that have corresponding relationships on a time scale. Specifically, for multimodal brain signals with temporal correspondence, the extracted multimodal feature data also exhibit temporal correspondence.

[0052] Optionally, feature fusion can be achieved by inputting multimodal feature data with corresponding relationships on a time scale into the cross-modal fusion module to obtain cross-modal brain region-level joint features. For example, the cross-modal fusion module can be used to splice or integrate the input multimodal feature data to obtain cross-modal brain region-level joint features; or the cross-modal fusion module can be used to perform weighted fusion calculation on the input multimodal feature data to obtain cross-modal brain region-level joint features.

[0053] Feature fusion is performed on multimodal feature data that have a temporal correspondence to obtain cross-modal brain region-level joint features. This includes: after a feature update of the magnetic resonance modal feature data, determining optical modal feature data that corresponds to the magnetic resonance modal feature data; and fusing the corresponding magnetic resonance modal feature data and the optical modal feature data using a cross-modal fusion module to obtain cross-modal brain region-level joint features. For example, based on the timestamp corresponding to the updated magnetic resonance modal feature data, a fixed time window corresponding to that timestamp is determined. This fixed time window can have either a starting timestamp or a center timestamp. The optical modal feature data corresponding to this fixed time window is determined as the optical modal feature data that has a temporal correspondence to the updated magnetic resonance modal feature data, and feature fusion is then performed.

[0054] Based on the feature extraction described in the above embodiments, the cross-modal brain region-level joint features obtained through feature fusion are decoded online to obtain the brain state analysis results of the target object. Optionally, the cross-modal brain region-level joint features are input into a pre-set decoding module to obtain the brain state analysis results of the target object. This decoding module can be, for example, a model such as a support vector machine, logistic regression, shallow neural network, or lightweight recurrent network, which can learn the mapping relationship between the cross-modal brain region-level joint features and brain state labels.

[0055] In some embodiments, the brain state analysis results of the target object are output; in other embodiments, a control instruction is generated based on the brain state analysis results, and the control instruction is transmitted to the control execution module so that the control execution module executes the control instruction. A pre-set correspondence between brain state types and control instructions can be established, and the control instruction corresponding to the brain state analysis results can be determined based on this correspondence. It is understood that the control execution module may differ in different application scenarios, and correspondingly, different correspondences between brain state types and control instructions can be set in different application scenarios to achieve different controls on different control execution modules. The control execution module may include, but is not limited to, reading and writing devices, visual devices, and neuromodulation devices. For example, a neuromodulation device may be an epilepsy control device. When the brain state analysis results indicate that the target object is in an epileptic state, a corresponding control instruction is generated to adjust the parameters of the epilepsy control device, thereby achieving neuromodulation of the target object to relieve the epileptic state.

[0056] By converting brain state analysis results into regulatory commands and regulating the regulatory execution module, a closed-loop control process is realized, which includes online acquisition of multimodal data, online decoding of brain states, and regulatory operations based on the decoding results.

[0057] Based on the above embodiments, Figure 5 This is a flowchart of a multimodal data closed-loop processing method for brain-computer interfaces provided in this disclosure. The process specifically includes: synchronously acquiring magnetic resonance imaging (MRI) modal brain signals and optical modal brain signals, and performing time synchronization and frame-level alignment processing on the data streams of multiple modalities through a unified trigger signal or timestamp mechanism, thereby ensuring the consistency of brain signals of different modalities on the time axis and providing a foundation for subsequent parallel processing and fusion modeling.

[0058] After time alignment, independent online preprocessing pipelines were constructed for magnetic resonance imaging (MRI) modal brain signals and optical modal brain signals, respectively, to perform parallel preprocessing on both modalities. The preprocessing process, constrained by online operation and continuous updates, was used to perform quality correction and feature preparation on the raw brain signals to meet the requirements of subsequent unified spatial registration and feature extraction. The preprocessing methods for MRI and optical modal brain signals are as follows: Figure 5 As shown.

[0059] After completing the single-modal online preprocessing, the magnetic resonance imaging (MRI) modality brain signals and the optical modality brain signals are mapped to a unified spatial reference. Based on preset brain regions or regions of interest, signals within the corresponding brain regions are extracted and aggregated to form a spatially consistent brain region-level brain functional signal representation. This step provides a unified spatial interface for the fusion modeling of brain signals from different modalities.

[0060] Based on the aforementioned multimodal brain region-level brain signals, magnetic resonance modal features and optical modal features are jointly modeled to construct a cross-modal fusion feature representation, enabling brain functional information under different modalities and time scales to collaboratively reflect changes in brain functional state in the same feature space.

[0061] The fused photomagnetic multimodal feature data is input into the decoding model to perform online decoding of the target brain functional state and continuously output the decoding results to support state recognition, event detection, or control decisions in the brain-computer interface system. Based on the online decoding results of the brain functional state, combined with the pre-set mapping relationship between the brain functional state and the control instructions (control parameters), corresponding control instructions and parameters are generated, and the control instructions are applied to the target object to guide or intervene in the target object's brain functional state.

[0062] In subsequent data acquisition and processing, the changes in brain function caused by the regulatory effect will be sensed again by the multimodal brain function signal acquisition and decoding module, thus forming a closed-loop operation mode of decoding-regulation-re-sensing.

[0063] Figure 6 This is a schematic diagram of a multimodal data closed-loop processing device for brain-computer interfaces provided in an embodiment of this disclosure. The device can be implemented in software and / or hardware, and specifically includes: a data acquisition module 210, a spatial mapping module 220, a feature extraction module 230, a feature fusion module 240, a decoding module 250, and a control module 260.

[0064] The data acquisition module 210 is used to synchronously acquire the multimodal brain signals of the target object, and perform time alignment and parallel preprocessing on the multimodal brain signals to obtain preprocessed multimodal brain signals. The spatial mapping module 220 is used to perform mapping processing on the preprocessed multimodal brain signals based on a predetermined cross-modal spatial mapping relationship, so as to obtain multimodal brain region-level signals under a unified spatial reference. Feature extraction module 230 is used to independently extract features from the multimodal brain region-level signals under the unified spatial benchmark to obtain feature data for each modality; The feature fusion module 240 is used to fuse feature data of multimodal features that have corresponding relationships on the time scale to obtain cross-modal brain region-level joint features. Decoding module 250 is used to perform online decoding processing on the cross-modal brain region-level joint features to obtain the brain state analysis results of the target object; The regulation module 260 is used to generate regulation instructions based on the brain state analysis results and transmit them to the regulation execution module for execution. The changes in brain functional state caused by the regulation instructions are sensed again by the data acquisition module to form a closed-loop operation.

[0065] Based on the above embodiments, optionally, the multimodal brain signals include magnetic resonance modal brain signals and optical modal brain signals; The data acquisition module 210 is used to: sequentially acquire the magnetic resonance modal brain signals based on each time point in a preset time series using a magnetic resonance imaging device; and send a trigger signal to the multimodal synchronization control module according to the start time point of the preset time series; and send an acquisition start command to the optical imaging device through the multimodal synchronization control module so as to synchronously acquire the optical modal brain signals through the optical imaging device.

[0066] Optionally, the magnetic resonance modal brain signal and the optical modal brain signal each carry a timestamp; The data acquisition module 210 is further configured to: map the magnetic resonance modal brain signal and the optical modal brain signal to the same time axis based on a unified timestamp signal, so as to establish a time correspondence between the magnetic resonance modal brain signal and the optical modal brain signal, and cache the time-aligned magnetic resonance modal brain signal and the optical modal brain signal respectively; perform preprocessing on the cached magnetic resonance modal brain signal sequentially based on the preprocessing process of the magnetic resonance modal brain signal, and perform preprocessing on the cached optical modal brain signal sequentially based on the preprocessing process of the optical modal brain signal, wherein the preprocessing processes corresponding to the magnetic resonance modal brain signal and the optical modal brain signal are executed in parallel according to the data availability status.

[0067] Based on the above embodiments, optionally, the cross-modal spatial mapping relationship includes the spatial mapping relationship corresponding to each mode; The spatial mapping module 220 is used to: map the preprocessed magnetic resonance modal brain signals based on the spatial mapping relationship corresponding to the magnetic resonance modal to obtain the magnetic resonance modal brain region-level signals under a unified spatial reference; and map the preprocessed optical modal brain signals based on the spatial mapping relationship corresponding to the optical modal to obtain the optical modal brain region-level signals under a unified spatial reference.

[0068] The spatial mapping module 220 is also used to: acquire structural magnetic resonance images, standard brain atlas images and functional magnetic resonance reference images, register the functional magnetic resonance reference images and the standard brain atlas images to the space where the structural magnetic resonance images are located, and determine the spatial mapping relationship corresponding to the magnetic resonance modes based on the registration relationship; The structural magnetic resonance image is projected to obtain a two-dimensional projected image, the projection angle of which corresponds to the optical imaging angle of the optical imaging device; an optical reference image is obtained, and the optical reference image and the two-dimensional projected image are image registered based on multimodal annotations to obtain the spatial mapping relationship corresponding to the optical mode.

[0069] Based on the above embodiments, optionally, the feature fusion module 240 is used to determine optical modal feature data that has a corresponding relationship with the magnetic resonance modal feature data after the magnetic resonance modal feature data has completed a feature update; and to perform fusion processing on the magnetic resonance modal feature data and the optical modal feature data that have a corresponding relationship based on the cross-modal fusion module to obtain cross-modal brain region-level joint features.

[0070] The above-described products can perform the methods provided in any embodiment of this disclosure, and have the corresponding functional modules and beneficial effects for performing the methods.

[0071] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0072] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0073] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as multimodal data closed-loop processing methods for brain-computer interfaces.

[0075] In some embodiments, the multimodal data closure processing method for brain-computer interfaces can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multimodal data closure processing method for brain-computer interfaces described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multimodal data closure processing method for brain-computer interfaces by any other suitable means (e.g., by means of firmware).

[0076] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] Computer programs used to implement the methods of this disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of this disclosure, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0080] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0081] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0082] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0083] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal data closed-loop processing method for brain-computer interfaces according to any embodiment of this disclosure.

[0084] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for closed-loop processing of multimodal data for brain-computer interfaces, characterized in that, include: Simultaneously acquire multimodal brain signals of the target object, and perform time alignment and parallel preprocessing on the multimodal brain signals to obtain preprocessed multimodal brain signals; Based on a predetermined cross-modal spatial mapping relationship, the preprocessed multimodal brain signals are mapped and processed to obtain multimodal brain region-level signals under a unified spatial reference. Independent feature extraction is performed on the multimodal brain region-level signals under the unified spatial benchmark to obtain feature data for each modality; Feature fusion is performed on multimodal feature data that have corresponding relationships on a time scale to obtain cross-modal brain region-level joint features; The cross-modal brain region-level joint features are decoded online to obtain the brain state analysis results of the target object; Based on the brain state analysis results, a control command is generated and transmitted to the control execution module for execution. The changes in brain functional state caused by the control command are perceived again through the synchronous acquisition of the multimodal brain signals to form a closed-loop operation.

2. The method according to claim 1, characterized in that, The multimodal brain signals include magnetic resonance modal brain signals and optical modal brain signals; Simultaneously acquire multimodal brain signals from the target subject, including: The magnetic resonance imaging (MRI) device sequentially acquires the magnetic resonance modal brain signals at each time point in a preset time series; and a trigger signal is sent to the multimodal synchronization control module at the start time point of the preset time series. The multimodal synchronization control module sends a data acquisition start command to the optical imaging device to synchronously acquire the optical modal brain signals through the optical imaging device.

3. The method according to claim 2, characterized in that, The magnetic resonance modal brain signal and the optical modal brain signal each carry a timestamp; The multimodal brain signals are time-aligned and preprocessed in parallel to obtain preprocessed multimodal brain signals, including: Based on a unified timestamp signal, the magnetic resonance modal brain signal and the optical modal brain signal are mapped to the same time axis to establish a time correspondence between the magnetic resonance modal brain signal and the optical modal brain signal, and the time-aligned magnetic resonance modal brain signal and the optical modal brain signal are cached respectively. The preprocessing process based on the magnetic resonance modal brain signal sequentially performs preprocessing on the cached magnetic resonance modal brain signal, and the preprocessing process based on the optical modal brain signal sequentially performs preprocessing on the cached optical modal brain signal, wherein the preprocessing processes corresponding to the magnetic resonance modal brain signal and the optical modal brain signal are executed in parallel according to the data availability status.

4. The method according to claim 2, characterized in that, The cross-modal spatial mapping relationship includes the spatial mapping relationship corresponding to each mode; Based on a pre-determined cross-modal spatial mapping relationship, the preprocessed multimodal brain signals are mapped to obtain multimodal brain region-level signals located under a unified spatial reference, including: Based on the spatial mapping relationship corresponding to the magnetic resonance modalities, the preprocessed magnetic resonance modal brain signals are mapped to obtain magnetic resonance modal brain region-level signals under a unified spatial reference. Based on the spatial mapping relationship corresponding to the optical modalities, the preprocessed optical modal brain signals are mapped to obtain optical modal brain region-level signals under a unified spatial reference.

5. The method according to claim 4, characterized in that, The process of determining the cross-modal space mapping relationship includes: Acquire structural magnetic resonance images, standard brain atlas images, and functional magnetic resonance reference images; register the functional magnetic resonance reference images and the standard brain atlas images to the space where the structural magnetic resonance images are located; and determine the spatial mapping relationship corresponding to the magnetic resonance modes based on the registration relationship. The structural magnetic resonance image is projected to obtain a two-dimensional projection image, and the projection angle of the two-dimensional projection image corresponds to the optical imaging angle of the optical imaging device. An optical reference image is acquired, and image registration is performed on the optical reference image and the two-dimensional projection image based on multimodal annotations to obtain the spatial mapping relationship corresponding to the optical mode.

6. The method according to claim 2, characterized in that, Feature fusion is performed on multimodal feature data that have corresponding relationships on a time scale to obtain cross-modal brain region-level joint features, including: Once the magnetic resonance modal feature data has been updated once, optical modal feature data that corresponds to the magnetic resonance modal feature data is determined. The cross-modal fusion module fuses the corresponding magnetic resonance modal feature data and optical modal feature data to obtain cross-modal brain region-level joint features.

7. A multimodal data closed-loop processing device for brain-computer interfaces, characterized in that, include: The data acquisition module is used to synchronously acquire multimodal brain signals of the target object, and perform time alignment and parallel preprocessing on the multimodal brain signals to obtain preprocessed multimodal brain signals. The spatial mapping module is used to perform mapping processing on the preprocessed multimodal brain signals based on a pre-determined cross-modal spatial mapping relationship, so as to obtain multimodal brain region-level signals under a unified spatial reference. The feature extraction module is used to independently extract features from the multimodal brain region-level signals under the unified spatial benchmark to obtain feature data for each modality. The feature fusion module is used to fuse feature data from multiple modalities that have corresponding relationships on a time scale to obtain cross-modal brain region-level joint features; The decoding module is used to perform online decoding processing on the cross-modal brain region-level joint features to obtain the brain state analysis results of the target object; The regulation module is used to generate regulation instructions based on the brain state analysis results and transmit them to the regulation execution module for execution. The changes in brain functional state caused by the regulation instructions are sensed again by the data acquisition module to form a closed-loop operation.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multimodal data closed-loop processing method for brain-computer interfaces according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the multimodal data closed-loop processing method for brain-computer interfaces as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the multimodal data closed-loop processing method for brain-computer interfaces according to any one of claims 1-6.

Images