Neural decoding system and method, electronic equipment and storage medium

Abstract

The invention provides a neural decoding system and method, electronic equipment and a storage medium, and the neural decoding system comprises a preprocessing module which carries out the preprocessing of an input original electroencephalogram signal, and obtains a preprocessing feature map; the multi-stage feature extraction module is configured to extract high-dimensional features of the preprocessed feature map; the multi-stage feature extraction module comprises at least two stages of feature extraction modules, and each stage of feature extraction module comprises a feature integration module and a convolution module; and the decoding module is configured to map the high-dimensional features extracted by the multi-stage feature extraction module into spatial distribution of a target decoding category to obtain predicted decoding category output. According to the invention, through structure optimization and module design of the decoding framework, the decoding accuracy and reasoning speed of neural decoding are significantly improved.

Description

Technical Field

[0001] This application belongs to the field of brain-computer interface technology and relates to a neural decoding system, method, electronic device and storage medium. Background Technology

[0002] Existing technologies have made significant progress in improving EEG decoding performance through signal processing and pre-training techniques, but they have neglected in-depth exploration of the macroscopic framework. In fact, the macroscopic framework of EEG decoding plays a crucial role in improving task performance. For example, in the time series domain, ModernTCN, by employing large-kernel convolutions and the ConvFFN module, achieved a 13.9% reduction in mean squared error (MSE) on the ETTm2 dataset compared to existing convolution-based models. In the computer vision domain, ConvNeXt-T, by optimizing both macroscopic architecture and microscopic components, outperformed the traditional ResNet-50 by 3.2% on the ImageNet-1K dataset. Therefore, improving the framework functionality in the EEG decoding field holds promise for achieving substantial performance gains. Summary of the Invention

[0003] This application provides a neural decoding system, method, electronic device, and storage medium to provide a macroscopic framework for neural decoding and improve the framework performance gain of EEG decoding.

[0004] In a first aspect, this application provides a neural decoding system, comprising: a preprocessing module for preprocessing input raw EEG signals to obtain a preprocessed feature map; a multi-stage feature extraction module configured to extract high-dimensional features from the preprocessed feature map; the multi-stage feature extraction module includes at least two stage feature extraction modules, each stage feature extraction module including a feature integration module and a convolution module; the feature integration module is configured as the first processing layer of each stage feature extraction module, realizing feature transformation or spatial scale alignment between the current feature processing stage and the previous feature processing stage; the convolution module is configured as the second processing layer of each stage feature extraction module, realizing intra-group feature interaction through multi-head group convolution, introducing pointwise convolution to realize cross-group feature fusion, and outputting the high-dimensional features; and a decoding module configured to map the high-dimensional features extracted by the multi-stage feature extraction module to the spatial distribution of the target decoding category, and obtain the predicted decoding category output.

[0005] In one implementation of the first aspect, the preprocessing module includes: a data preprocessing unit configured to normalize and augment the original EEG signal to generate diverse EEG signal training samples; a feature map reshaping unit configured to convert the EEG signal training samples into two-dimensional feature maps; and a feature map preprocessing unit configured to extract low-level local representations from the two-dimensional feature map to obtain a preprocessed feature map.

[0006] In one implementation of the first aspect, the feature map reshaping unit takes a one-dimensional input of the electroencephalogram (EEG) signal. Reconstructed into a two-dimensional feature map ,in Indicates batch size. Indicates the number of channels. Indicates the number of time steps.

[0007] In one implementation of the first aspect, the feature map preprocessing unit performs preliminary feature extraction on the two-dimensional feature map through four consecutive feature preprocessing units, each of which includes a two-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation function.

[0008] In one implementation of the first aspect, the feature integration module includes: a feature transformation unit configured at the beginning of each stage of the feature extraction module to increase the number of feature maps to initialize higher-level semantic representation information; the feature transformation units of each stage of the feature extraction module gradually increase the number of channels of the feature maps in different feature extraction stages using a step-by-step strategy, so that the feature extraction process progresses from low-level features to high-level semantic features layer by layer; a spatial scale alignment unit configured at the beginning of each stage of the feature extraction module to compress the size of the feature maps to initialize coarse-grained semantic representation; the spatial scale alignment units of each stage of the feature extraction module achieve significant compression of the size of early feature maps by centrally setting downsampling units in the second stage of the feature extraction module.

[0009] In one implementation of the first aspect, the convolution module includes: a multi-head grouped convolution unit configured to divide the input feature map into multiple groups according to the channel dimension, and to perform convolution operations independently within each group to achieve effective interaction of features within the group; and a pointwise convolution unit configured to perform fusion processing on cross-group features of multiple groups, and to integrate feature information from different groups through a 1×1 convolution kernel to enhance the global correlation between cross-group features.

[0010] In one implementation of the first aspect, the multi-head grouped convolutional unit divides the input feature map into multiple heads in the channel dimension, each head containing a fixed number of channels, and performs convolution operations independently within each head.

[0011] In one implementation of the first aspect, the decoding module includes: a generalized mean pooling unit configured to perform spatial compression on the input high-dimensional feature map, and to obtain pooled features by controlling different forms of pooling operations through a trainable power exponent; and a linear mapping unit configured to perform dimensional transformation on the pooled features, and to map the dimensionally transformed features to the target decoding category space through a fully connected layer, and output a predicted classification.

[0012] Secondly, this application provides a neural decoding method, comprising: preprocessing the input raw EEG signal to obtain a preprocessed feature map; performing multi-stage feature extraction on the preprocessed feature map to obtain high-dimensional features of the preprocessed feature map; the multi-stage feature extraction includes at least two stages of feature extraction; wherein each stage of feature extraction includes: using a feature integration module as the first processing layer of the feature extraction at each stage to realize feature transformation and spatial scale alignment between the current feature processing stage and the previous feature processing stage; using a convolution module as the second processing layer of the feature extraction at each stage to realize intra-group feature interaction through multi-head group convolution, introducing pointwise convolution to realize cross-group feature fusion, and outputting the high-dimensional features; mapping the high-dimensional features obtained by the multi-stage feature extraction to the spatial distribution of the target decoding category to obtain the predicted decoding category output.

[0013] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described neural decoding method.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described neural decoding method.

[0015] As described above, the neural decoding system, method, electronic device, and storage medium of this application have the following beneficial effects:

[0016] The neural decoding system provided in this application is a lightweight and efficient neural decoding framework. A preprocessing module normalizes and augments the raw one-dimensional EEG signal, converting it into a two-dimensional feature map. Low-level local representations are extracted from the two-dimensional feature map. A multi-stage feature extraction module achieves feature connections and scale alignment between different semantic levels. Each feature extraction stage employs a step-by-step feature connection strategy, combined with a pyramid structure for scale transitions. The convolution module achieves intra-group feature interaction through multi-head group convolution and integrates cross-group features using pointwise convolution. Finally, the decoding module maps the output features to the corresponding decoding category. This invention significantly improves the decoding accuracy and inference speed of neural decoding through structural optimization and module design of the decoding framework.

[0017] This application employs a preprocessing module to normalize and augment the raw one-dimensional EEG signal, converting the processed one-dimensional signal into a two-dimensional feature map. The scheme of extracting low-level local representations from the two-dimensional feature map is consistent with the transient temporal dynamics of brain signals in neural decoding tasks. It fully utilizes the effectiveness of short-range temporal information for neural decoding and is best suited for capturing temporal patterns in brain signals.

[0018] In this application, grouping the input and output channels of convolution in neural decoding tasks does not degrade performance; channel grouping is more suitable for aggregating local information while improving computational efficiency. This application uses a two-dimensional convolutional neural network as its basic architecture, fully leveraging the effectiveness of spatial locality in neural decoding, making it ideally suited for capturing spatial patterns in brain signals.

[0019] This application adopts a feature integration module as its basic architecture, making full use of the step-by-step method to change the number of feature maps, and realizing the effectiveness of multi-scale feature extraction for neural decoding.

[0020] This application improves computational efficiency and performance by applying downsampling early in the network process, which allows the model to focus on the most relevant features and reduce the impact of noise.

[0021] This application develops a highly efficient and powerful neural decoding (system) framework, NeuroSketch. Its overall architecture is similar to a 2D convolutional neural network (CNN 2D), employing a 2D input representation, which is then processed by a backbone layer to initiate feature extraction. The subsequent forward propagation process is divided into four stages, progressively capturing features from low to high levels using a step-by-step approach. In each stage, the initial component is responsible for increasing the number of feature maps. If the component is a downsampling layer, it also reduces the spatial resolution of the feature maps by half. This application uses a pyramid method to allocate downsampling layers to the second stage to improve performance and computational efficiency. Subsequently, a convolutional block is applied to extract features, including grouped convolutions, batch normalization, ReLU activation, and pointwise convolutions. After the four stages, the resulting features are passed through a generalized mean (GeM) pooling layer, which aggregates features along the channel and time dimensions. The pooled representation is then input into a linear layer for final decoding and classification. Attached Figure Description

[0022] Figure 1 The diagram shown is a schematic representation of an implementation framework of the neural decoding system described in this application.

[0023] Figure 2 The diagram shown is an exemplary implementation framework of the neural decoding system described in this application.

[0024] Figure 3 A schematic diagram of an exemplary overall architecture of the NeuroSketch neural decoding system described in this application is shown.

[0025] Figure 4 This is a schematic diagram of multi-head grouped convolution.

[0026] Figure 5The diagram shown is an exemplary implementation framework of the neural decoding system described in this application.

[0027] Figure 6 This diagram illustrates the characteristic changes of the raw electroencephalogram (EEG) signals in a neural decoding system.

[0028] Figure 7 The diagram shown is a schematic representation of an implementation flow of the neural decoding method described in an embodiment of this application.

[0029] Figure 8 The diagram shown is a schematic representation of an implementation structure of the electronic device described in this application.

[0030] Figure 9 A comparative diagram of different patching methods is shown.

[0031] Figure 10 A comparative diagram of different potential spatial transformation methods is shown.

[0032] Figure 11 This is a diagram showing the comparison of model performance.

[0033] Figure 12 The image is displayed as a scatter plot of NeuroSketch's scaling behavior on the DUIN dataset. Detailed Implementation

[0034] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0035] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0036] This application proposes an effective framework for neural decoding through system architecture optimization, optimizing the architecture from macro to micro levels, with performance improvements achieved at each step.

[0037] Neural decoding involves inferring external stimuli, cognitive states, or intentions from brain signals. These signals are typically recorded using methods such as electroencephalography (EEG), stereoelectroencephalography (SEEG), and electrocorticography (ECoG), which can be categorized as non-invasive or invasive techniques. EEG, as a non-invasive method, records electrical activity on the scalp and is widely used in cutting-edge scientific research and clinical settings due to its practicality and low cost. In contrast, invasive methods such as SEEG and ECoG capture signals from deeper brain structures, offering higher temporal and spatial resolution compared to non-invasive methods, and enabling more precise identification of neural activity associated with specific cognitive functions.

[0038] In neural decoding, recorded brain signals possess transient temporal dynamics and spatial locality, presenting unique modeling requirements. Traditional time-series data, such as weather, electricity consumption, and traffic flow, are sampled at relatively low frequencies (e.g., minutes or hours) and typically exhibit significant periodicity or trends. However, brain signals require higher temporal resolution to capture the brain's rapidly changing states. Even within brain signals themselves, their characteristics can vary considerably across different scenarios.

[0039] From a temporal perspective, compared to other scenarios (such as sleep staging), brain signals in neural decoding tasks typically contain key information within a shorter time frame. In sleep staging tasks, brain signals are recorded throughout the night, with each sleep stage labeled at a resolution of 30 seconds. In contrast, in the RapidSerial Visual Presentation (RSVP) paradigm used for decoding image stimuli, relevant information is encoded within only a few hundred milliseconds. This temporal characteristic requires the model to be able to recognize immediate and non-stationary neural responses to stimuli.

[0040] From a spatial perspective, task-driven neural activity in the decoding paradigm differs fundamentally from neural activity in pathological neurological disorders. Pathological conditions such as generalized epilepsy are characterized by widespread brain activity, leading to nonspecific channel interactions. In neural decoding tasks, cortical activation triggered by external stimuli is typically confined to functionally specialized areas. For example, speech stimuli primarily activate the left inferior frontal gyrus, while visual stimuli involve the visual cortex. These activations are recorded by physically adjacent electrodes, highlighting the need for models to capture the spatial locality of channels, rather than simply modeling whole-brain states.

[0041] Previous work on neural decoding can be broadly categorized into two approaches. The first is signal processing, which focuses on improving the signal-to-noise ratio and extracting task-relevant features. These methods rely on manually defined features, which are both time-consuming and highly experience-dependent. The second approach involves deep learning methods. However, current methods largely neglect in-depth exploration and modeling of the inherent temporal dynamics and spatial localities in neural decoding. Therefore, this application aims to provide a framework more suitable for neural decoding tasks through system architecture optimization.

[0042] This application aims to systematically provide an effective neural decoding framework to address how to best capture temporal and spatial patterns in brain signals (i.e., Problem 1), and how to improve neural decoding performance through macroscopic to microscopic architectural optimization based on a suitable architecture (i.e., Problem 2).

[0043] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, this embodiment provides a neural decoding system 100, including: a preprocessing module 110, a multi-stage feature extraction module 120, and a decoding module 130.

[0045] The preprocessing module 110 preprocesses the input raw EEG signal to obtain a preprocessed feature map.

[0046] The multi-stage feature extraction module 120 is configured to extract high-dimensional features from the preprocessed feature map. The multi-stage feature extraction module 120 includes at least two stage feature extraction modules 121, each stage including a feature integration module 1211 and a convolution module 1212. The feature integration module 1211 is configured as the first processing layer of each stage feature extraction module, realizing feature transformation or spatial scale alignment between the current feature processing stage and the previous feature processing stage. The convolution module 1212 is configured as the second processing layer of each stage feature extraction module, achieving intra-group feature interaction through multi-head group convolution and introducing pointwise convolution to achieve cross-group feature fusion, outputting the high-dimensional features. The feature integration module is located at the beginning of each feature processing stage, realizing feature transformation and spatial scale alignment between the current feature processing stage and the previous feature processing stage, thereby ensuring efficient transmission of multi-level information. The feature transformation adopts a step-by-step strategy, and the scale alignment adopts a pyramid structure design. The convolution module achieves intra-group feature interaction through multi-head group convolution and introduces pointwise convolution for cross-group feature fusion, effectively integrating feature information from different groups and enhancing the global correlation between features.

[0047] The decoding module 130 is configured to map the high-dimensional features extracted by the multi-stage feature extraction module to the spatial distribution of the target decoding category, thereby obtaining the predicted decoding category output.

[0048] The neural decoding system provided in this application is a lightweight and efficient neural decoding framework. A preprocessing module normalizes and augments the raw one-dimensional EEG signal, converting it into a two-dimensional feature map. Low-level local representations are extracted from the two-dimensional feature map. A multi-stage feature extraction module achieves feature connections and scale alignment between different semantic levels. Each feature extraction stage employs a step-by-step feature connection strategy, combined with a pyramid structure for scale transitions. The convolution module achieves intra-group feature interaction through multi-head group convolution and integrates cross-group features using pointwise convolution. Finally, the decoding module maps the output features to the corresponding decoding category. This invention significantly improves the decoding accuracy and inference speed of neural decoding through structural optimization and module design of the decoding framework.

[0049] In one embodiment of this application, as Figure 2 As shown, the preprocessing module 110 includes: a data preprocessing unit 111, a feature map reshaping unit 112, and a feature map preprocessing unit 113.

[0050] The data preprocessing unit 111 is configured to normalize and augment the raw EEG signals to generate diverse EEG signal training samples. The data preprocessing module normalizes the raw EEG signals and applies various data augmentation methods to the normalized EEG signals to generate diverse training samples.

[0051] For example, the data preprocessing unit 111 includes a normalization unit and a data augmentation unit; the normalization unit is configured to perform z-score normalization on the input raw EEG signal according to each channel; the data augmentation unit is configured to sample the normalized EEG signal according to a preset probability using multiple different data augmentation paths; the sampling types of the data augmentation paths include adding Gaussian noise, random time shifting, channel random masking, time period random masking, and sample mixing augmentation.

[0052] In one implementation, the addition of Gaussian noise generates noise from a standard normal distribution; the noise is scaled by a predefined standard deviation and added to the input to generate noisy data.

[0053] In one implementation, the random time shift defines a maximum shift range proportional to the length of the input sequence; for each sample, a shift step is randomly sampled from the maximum shift range; a positive value indicates that the sequence is shifted forward, and a negative value indicates that the sequence is shifted backward.

[0054] In one implementation, the channel random mask is applied to each channel with a specified probability, thereby setting the corresponding channel value to zero.

[0055] In one implementation, the time-segment random mask is applied along the time dimension to a time segment of random length to zero out the corresponding channel values.

[0056] In one implementation, the sample mixing enhancement samples a mixing coefficient from a beta distribution parameterized by hyperparameters; the mixing coefficient is used to linearly combine the original sample and the randomly selected sample, and the corresponding labels of the original sample and the randomly selected sample are mixed accordingly.

[0057] The feature map reshaping unit 112 is configured to convert the EEG signal training samples into two-dimensional feature maps. The feature map reshaping module converts the preprocessed EEG signals into two-dimensional feature maps to adapt to the input format of the subsequent two-dimensional convolutional neural network model.

[0058] For example, the feature map reshaping unit 112 inputs a one-dimensional EEG signal. Reconstructed into a two-dimensional feature map ,in Indicates batch size. Indicates the number of channels. Indicates the number of time steps.

[0059] The feature map preprocessing unit 113 is configured to extract low-level local representations from the two-dimensional feature map to obtain a preprocessed feature map.

[0060] For example: Figure 3 As shown, the feature map preprocessing unit 113 performs preliminary feature extraction on the two-dimensional feature map through four consecutive feature preprocessing units. Each feature preprocessing unit includes a two-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation function.

[0061] This application first normalizes and augments the original one-dimensional EEG signal through a preprocessing unit; the feature map reshaping unit converts the processed one-dimensional signal into a two-dimensional feature map; and the feature map preprocessing unit extracts low-level local representations from the two-dimensional feature map.

[0062] Furthermore, such as Figure 3 As shown, the feature integration module includes: a feature transformation unit (i.e., an embedding layer) or a spatial scale alignment unit (i.e., a downsampling layer).

[0063] The feature transformation unit is configured at the beginning of each stage of the feature extraction module to increase the number of feature maps in order to initialize higher semantic level representation information. The feature transformation unit of each stage of the feature extraction module gradually increases the number of channels of the feature map in different feature extraction stages with a step-by-step strategy, so that the feature extraction process progresses from low-level features to high-level semantic features layer by layer, thereby enhancing the network's ability to model multi-level semantic information and improving the overall expression effect.

[0064] The spatial scale alignment unit is configured at the starting position of each stage feature extraction module to compress the size of the feature map in order to initialize the coarse-grained semantic representation. The spatial scale alignment unit of each stage feature extraction module adopts a tower-style strategy by centrally setting the downsampling unit in the second stage feature extraction module, thereby significantly compressing the size of the early feature map, effectively reducing the overall computational cost, while keeping the decoding performance unaffected.

[0065] Furthermore, such as Figure 3 As shown, the convolution module (i.e., convolution block) includes: multi-head grouped convolution unit, batch normalization unit, ReLU activation unit, and pointwise convolution unit.

[0066] The multi-head grouped convolutional unit is configured to divide the input feature map into multiple groups according to the channel dimension, and perform convolution operations independently within each group to achieve effective interaction of features within the group.

[0067] Batch normalization units normalize the features of each channel, stabilize the feature distribution, make gradient updates more stable, and thus accelerate network convergence.

[0068] The ReLU activation unit introduces a nonlinear transformation into the features while filtering out redundant information.

[0069] The pointwise convolutional unit is configured to fuse cross-group features, integrating feature information from different groups through a 1×1 convolutional kernel to enhance the global correlation between cross-group features.

[0070] Specifically, the multi-head grouped convolutional unit is designed with reference to the multi-head self-attention mechanism. It divides the input feature map into multiple heads in the channel dimension. Each head contains a fixed number of channels and performs convolution operations independently within each head. As the network depth increases, the number of channels contained in each head in each feature extraction stage remains unchanged, while the number of heads gradually increases. This enables the network to extract feature information in parallel in multiple subspaces, improving the diversity of feature representation and decoding performance.

[0071] like Figure 4As shown, multi-head grouped convolution can divide the input features with 96 input channels into two groups of input features with a head dimension of 48 in the first feature extraction stage, and perform intra-group convolution and cross-group convolution processing. The number of channels of the output features in the first feature extraction stage is still 96. In the Nth feature extraction stage, the input features with 144 input channels are divided into three groups of input features with a head dimension of 48, and perform intra-group convolution and cross-group convolution processing. The number of channels of the output features in the Nth feature extraction stage is still 144.

[0072] In one embodiment of this application, as Figure 5 As shown, the decoding module 130 includes: a generalized mean pooling unit 131 and a linear mapping unit 132.

[0073] The generalized mean pooling unit 131 is configured to spatially compress the input high-dimensional feature map and control different forms of pooling operations through a trainable power exponent to obtain pooled features. Here, the power exponent refers to a trainable parameter p used in generalized mean pooling to control the feature aggregation method, and its value directly determines the specific form of the pooling operation. The generalized mean pooling unit spatially compresses the input high-dimensional feature map and controls different forms of pooling operations through a trainable power exponent, thereby enhancing the robustness of the model while preserving discriminative features.

[0074] The linear mapping unit 132 is configured to perform dimensionality transformation on the pooled features, and then map the dimensionally transformed features to the target decoding class space through a fully connected layer to output a predicted classification. The linear mapping unit performs dimensionality transformation on the pooled features and maps the features to the target decoding class space through a fully connected layer to achieve the final decoding classification prediction output.

[0075] like Figure 6 As shown, it provides a schematic diagram of the changes in feature dimensions of raw EEG signals at each processing stage during the processing of the neural decoding system described in this application. By comparing the step-by-step / pyramid-style feature extraction method used in this application with the traditional jump-style / pyramid-style feature processing method, the superiority of the neural decoding system described in this application in data processing can be better demonstrated.

[0076] like Figure 7 As shown in the figure, this application provides a neural decoding method, including steps S1 to S3.

[0077] S1, preprocess the input raw EEG signal to obtain a preprocessed feature map;

[0078] S2, perform multi-stage feature extraction on the preprocessed feature map to obtain high-dimensional features of the preprocessed feature map; the multi-stage feature extraction includes at least two stages of feature extraction; wherein, each stage of feature extraction includes steps S2A~S2B:

[0079] S2A utilizes the feature integration module as the first processing layer for feature extraction in each stage, realizing feature transformation and spatial scale alignment between the current feature processing stage and the previous feature processing stage.

[0080] S2B utilizes a convolutional module as the second processing layer for feature extraction at each stage. It achieves intra-group feature interaction through multi-head group convolution, introduces pointwise convolution to achieve cross-group feature fusion, and outputs the high-dimensional features.

[0081] S3, map the high-dimensional features obtained from the multi-stage feature extraction to the spatial distribution of the target decoding category, and obtain the predicted decoding category output.

[0082] The specific implementation details of the neural decoding method described in the embodiments of this application can be found in the relevant description of the neural decoding system described in the embodiments of this application, and will not be repeated here.

[0083] The scope of protection of the neural decoding method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0084] The neural decoding system described in this application can implement the neural decoding method described in this application. However, the implementation device of the neural decoding method described in this application includes, but is not limited to, the structure of the neural decoding system listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0085] This application also provides an electronic device, such as... Figure 8 As shown, the electronic device 800 includes a memory 810 and a processor 820.

[0086] The memory 810 is used to store computer programs. In some possible implementations, the memory may include a computer system readable medium in the form of volatile memory, such as RAM and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0087] The processor 820 is communicatively connected to the memory 810 and is used to execute the computer program stored in the storage module, so that the electronic device 800 can execute the neural decoding method provided in any embodiment of this application.

[0088] In some possible implementations, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0089] In some possible implementations, the electronic device 600 provided in this application embodiment may further include a display 630. The display is communicatively connected to a memory and a processor, and is used to display a graphical user interface (GUI) related to the data processing method provided in any embodiment of this application.

[0090] In this embodiment, the display may include a display screen (display panel). In some implementations, the display panel may be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or similar methods. Alternatively, the display may be a touch panel (touchscreen, touch screen), which may include a display screen and a touch-sensitive surface. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to a processor to determine the type of touch event. The processor then provides corresponding visual output on the display device based on the type of touch event.

[0091] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described neural decoding method.

[0092] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0093] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0094] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0095] The neural decoding system provided in this application is an effective framework for neural decoding. The decoding system described in this application matches the temporal and spatial characteristics of brain signals in neural decoding tasks, achieving leading performance in both tasks, as detailed below.

[0096] The goal of this architecture exploration approach is to find the most suitable model architecture for a specific task by exploring different network structures and connection methods. This process not only focuses on improving model performance but also needs to consider the limitations of computational resources, such as training time, memory consumption, and inference speed. Therefore, the architecture exploration method needs to strike a balance between performance and efficiency to meet the needs of practical applications.

[0097] For neural decoding tasks, this application categorizes them into three types based on the decoding modality: visual, auditory, and speech.

[0098] Visual decoding aims to reconstruct the visual stimuli seen by humans. Early decoding objects mainly focused on static images, and its core experimental method was the RSVP paradigm.

[0099] Auditory decoding can be further divided into auditory perception decoding and auditory imagery decoding. Auditory perception decoding mainly focuses on the analysis of neural signals of actual heard sounds, aiming to understand how the brain processes external auditory input stimuli, while auditory imagery decoding focuses on decoding the brain neural activity patterns of subjects when they imagine music or other sounds, aiming to analyze how the brain constructs and simulates auditory experiences.

[0100] The core objective of speech decoding is to directly convert the brain signals of a speaker or person intending to speak into speech signals or sentences. Speech decoding can be categorized according to the decoding target: phonemes, words, and sentences. Phonemes are the basic building blocks of speech pronunciation, and phoneme-oriented decoding focuses on analyzing human pronunciation patterns using brain activity. Direct sentence decoding is mostly conducted on strictly defined vocabulary sets, where words have significantly different pronunciations, which helps improve the accuracy and reliability of decoding. Speech decoding in open-vocabulary scenarios is the most challenging because the vocabulary is virtually unlimited, covering a rich variety of pronunciations and semantics.

[0101] To rigorously evaluate the framework-level performance gains of the neural decoding system proposed in this application, nearly 2000 experiments were conducted on six neural decoding datasets. These datasets cover three main sensory decoding categories: auditory, visual, and speech; three different types of brain signals: electroencephalography (EEG), electrocorticography (SEEG), and electrocorticography (ECoG); and eight mainstream neural decoding tasks. An overview of each dataset is shown in Table 1, and evaluation was performed on a total of six datasets. The evaluation process included task-based partitioning for Chisco and OpenMIIR, resulting in a total of eight different tasks. The diversity of the datasets ensured comprehensive evaluation across different sensory modalities, brain signal types, and decoding tasks, further validating the robustness and versatility of the framework. Since network complexity is closely related to final performance, this application maintained the model's parameter size at approximately 30 million by adjusting the model depth and embedding dimension.

[0102] Table 1: Overview of the datasets used in the experiments of this application

[0103]

[0104] 1. The effect analysis of the preprocessing module used in the neural decoding system described in this application is explained as follows:

[0105] This application analyzes the performance of nine commonly used architectures in neural decoding tasks. These architectures are selected from four categories: Convolutional Neural Networks (CNNs), Gated Recurrent Units (GRUs), Transformers, and hybrid architectures. The inputs of these models can be represented as... ,in Indicates batch size. Indicates the number of channels. Indicates the number of time steps.

[0106] This application analyzes two different modalities of CNNs: one-dimensional CNNs and two-dimensional CNNs. One-dimensional CNNs extract features by applying one-dimensional convolutional filters in the time dimension, with each filter combining features from all channels through an addition operation. Two-dimensional CNNs process the input... Remodeling Two-dimensional convolutional filters are applied in both the channel and time dimensions.

[0107] GRU processes input sequentially over time. At each time step, the current hidden state is updated using the current input and the hidden state of the previous time step.

[0108] Transformer uses a self-attention mechanism to process input in parallel. This is to capture the dependencies between all time steps.

[0109] This application also considers two Transformer variants: PatchTST and iTransformer. PatchTST uses a window size along the time dimension. and step length For input Divide into parts, and generate a set of parts. A length of A sequence of blocks, in which Then, it uses a Transformer layer to process the chunked input. This approach captures the relationships between different blocks while maintaining channel independence. In contrast, iTransformer treats all time steps of each channel as a holistic sequence and utilizes a self-attention mechanism to capture the interactions between channels.

[0110] In addition to single models, this application also analyzes hybrid architectures combining multiple methods, including CNN-GRU and two CNN-Transformer variants. One variant uses a CNN for feature extraction, followed by Transformer layers, while the other integrates a CNN and a Transformer in each module to jointly capture temporal information at different levels. Similar to the first CNN-Transformer variant, CNN-GRU uses a CNN for initial feature extraction, and then the GRU component processes the sequence data to capture temporal dependencies.

[0111] Table 2: Results of Infrastructure Analysis and Subsequent Analysis

[0112]

[0113] Here, v1 and v2 refer to the two hybrid methods mentioned above. The ratio shown in the middle of the table (e.g., 4:1) represents the ratio of Convolutional Neural Network (CNN) layers to Transformer layers in each hybrid model. The best result is indicated in bold, and the second best result is indicated by underline.

[0114] Table 2 (top section) illustrates the performance of the nine basic architectures mentioned above. Convolutional Neural Network (CNN)-based models, which excel in capturing local patterns, outperform hybrid models on most neural decoding tasks. Meanwhile, models based on Gated Recurrent Units (GRUs) and Transformers, better suited for modeling long-range dependencies, exhibit the lowest performance. This result is consistent with the transient temporal dynamics of brain signals in neural decoding tasks, indicating that short-range temporal information is more effective for neural decoding. Among CNN-based models, 2D CNNs (CNN 2D) that use convolutional kernels to extract local channel information outperform 1D CNNs (CNN 1D) that fuse all channel information additively. This performance difference is consistent with the spatial locality observed in neural decoding tasks. Transformer-based models perform poorly on some datasets, particularly the ThingsEEG dataset, where the highest accuracy is only 0.6%, just slightly above the 0.5% chance level.

[0115] As shown in Table 2 (top part), this application achieves normalization and data augmentation of the original one-dimensional EEG signal by using a preprocessing module, converts the processed one-dimensional signal into a two-dimensional feature map, and extracts low-level local representations from the two-dimensional feature map. This approach is consistent with the transient temporal dynamics of brain signals in neural decoding tasks, makes full use of the effectiveness of short-range temporal information for neural decoding, and is most suitable for capturing temporal patterns in brain signals.

[0116] 2. The effect analysis of the feature extraction module in the neural decoding system described in this application is explained below:

[0117] From a temporal perspective, this application analyzes how short-term and long-term temporal information affect model performance. This application controls short-term and long-term information by adjusting the ratio of convolutional neural network (CNN) and transformer layers in two hybrid models. As can be seen from the results in Table 2 (middle section), performance decreases as the proportion of short-term information decreases, demonstrating the effectiveness of short-term information for neural decoding.

[0118] From a spatial perspective, based on the effectiveness of 2D convolutional neural networks (CNN 2D), this application can similarly apply this finding to other models, such as gated recurrent units (GRUs) and transformers, to further verify its applicability. This application proposes a simple yet effective patching method to enhance local channel interactions in transformers and gated recurrent units. Figure 9 As shown, this paper compares different patching methods. First, a sliding window method is used to divide the input sequence into patches. Ordinary gated loop units and Transformers treat a single timestamp on all channels as a token; PatchTST treats multiple timestamps on a single channel as a token; iTransformer treats all timestamps of the entire channel as a token. Unlike the patching methods proposed in PatchTST and iTransformer, which aggregate information from multiple timestamps and channels, processing channels independently or focusing on global channel interactions, this application reshapes the patches by connecting the channel dimension and the patch dimension to preserve local channel relationships within the same time window.

[0119] The results in Table 2 (lower part) demonstrate that the method presented in this application exhibits highly competitive performance compared to other methods. By incorporating the token embedding method of this application, the accuracy of the transformer on the DUIN dataset is significantly improved from 8.5% to 23.4%. Furthermore, the accuracy of the gated recurrent unit on the perception and imagination tasks of the OpenMIIR dataset is substantially improved from the random level to 95.8% and 95.7%, respectively, further demonstrating the effectiveness of spatial locality in neural decoding.

[0120] As shown in Table 2 (lower part), the application uses a two-dimensional convolutional neural network (CNN 2D) as the basic architecture to make full use of the effectiveness of spatial locality in neural decoding, which is best suited for capturing spatial patterns in brain signals.

[0121] 3. The effect (macro-optimization) of the feature integration module (embedding layer) used in the feature extraction module of the neural decoding system described in this application is analyzed and explained as follows:

[0122] This application analyzes the transformation of the latent space dimension during forward propagation. For a two-dimensional convolutional neural network, its latent space representation refers to... ,in Indicates the number of feature maps. and These are the height and width of the feature maps, respectively. During forward propagation, the network increases the number of feature maps to capture higher-level information, while simultaneously reducing the size of the feature maps to reduce redundancy and improve computational efficiency. Based on this overall trend, this application will analyze the number of feature maps (…). ) and size ( and Transformation of ).

[0123] In neural network design, the appropriate allocation of feature maps across each layer is a crucial issue. This can typically be achieved through two widely adopted methods: step-by-step and skip-by-step. For example... Figure 10 As shown, this illustrates a comparison of different latent space transformation methods. Regarding the number of feature maps, the stepping method gradually increases them, while the skipping method rapidly increases them in the early stages. The stepping method uniformly increases the number of feature maps during the forward propagation, allowing feature extraction to progress gradually from low to high levels, helping the network focus on features at different levels. In contrast, the skipping method rapidly increases the number of feature maps in the early stages to reach the embedding dimension, and then fixes the number of feature maps in later stages, allowing the model to focus on refining and extracting higher-level features, but this usually incurs higher computational costs.

[0124] Table 3: Results of Latent Space Transformation and Computational Optimization

[0125]

[0126] In Table 3, lower GFLOPs indicate lower computational cost. Best results are shown in bold. ACC and F1 represent accuracy. Table 3 (top section) compares the performance and computational cost between the stepping and skipping methods. Compared to the skipping method, the stepping method achieves highly competitive performance while requiring 67.8% fewer floating-point operations. For example, the stepping method achieves 88.6% accuracy on the FacesHouses dataset, while the skipping method requires more than three times the computational cost and only achieves 85.7% accuracy to achieve similar accuracy. This indicates that neural decoding does not entirely rely on high-level neural representations, and multi-scale feature extraction is generally more efficient.

[0127] As shown in Table 3 (top part), this application uses a feature integration module as its basic architecture, making full use of the step-by-step method to change the number of feature maps, thus realizing the effectiveness of multi-scale feature extraction for neural decoding.

[0128] 4. The effect (macro-optimization) of the feature integration module (downsampling layer) used in the feature extraction module of the neural decoding system described in this application is analyzed and explained as follows:

[0129] During forward propagation, the transformation of feature map size is achieved through downsampling. Therefore, this application further analyzes how downsampling should be distributed throughout the forward propagation. If downsampling is only applied in the last few layers, the model must process high-resolution feature maps early on, which significantly increases computational complexity and defeats the purpose of downsampling to improve computational efficiency. Therefore, this application analyzes two common strategies: the pyramid method and the pagoda method. Figure 10 As shown, the pyramid method distributes the downsampling process evenly throughout the network, gradually reducing the size of the feature maps. This pyramid-like structure allows the network to retain more detailed features at the cost of increased computational complexity. The tower method takes a more aggressive approach, concentrating the downsampling modules in the early stages of the network. This results in a significant reduction in feature map size in the early stages, thereby lowering computational costs.

[0130] Table 3 (middle section) compares the performance and computational cost of the pyramid method and the tower method. The tower method achieves slightly higher accuracy than the pyramid method while requiring 41.1% fewer floating-point operations. Specifically, on the DUIN dataset, the tower method achieves an accuracy of 70.1%, while the pyramid method, with its higher computational cost, only achieves 64.7% accuracy. Furthermore, on the ThingsEEG dataset, the tower method improves accuracy by 2.0% to 20.4%, surpassing the 20.2% accuracy of one-dimensional CNNs, making it the best-performing architecture on the ThingsEEG dataset. These results indicate that over-extracting detailed information from the raw signal is unnecessary, as EEG signals typically have a low signal-to-noise ratio. Therefore, this application improves computational efficiency and performance by applying downsampling early in the network process, allowing the model to focus on the most relevant features and reduce the impact of noise.

[0131] 5. The effect (micro-optimization) of the convolution module used in the feature extraction module of the neural decoding system described in this application is analyzed and explained as follows:

[0132] This application analyzes the core computational method of convolutional neural networks: convolution operation. Given a height of... Width is The number of channels is Input Convolution operations use a height of Width is convolution kernel In the input Swipe up to calculate the output ,in , , These refer to the number of output channels, height, and width, respectively. The number of parameters involved in this convolution operation is... , representing the learnable weights within the convolutional kernel.

[0133] In addition to ordinary convolution, this application proposes various variants to improve computational efficiency and model performance, including grouped convolution and separable convolution.

[0134] Grouped convolution: Grouped convolution Input channel and output channels Divided into The convolution is then performed within each group, limiting the interaction between channels and effectively reducing the number of parameters compared to ordinary convolution. times.

[0135] Separable convolution: Separable convolution decomposes ordinary 2D convolution into two different operations, used for depthwise separable convolution in the spatial dimension. and pointwise convolution for channel dimensions Depthwise separable convolution is an extreme case of grouped convolution, where the number of groups is... In this case, each output feature map directly corresponds to a specific input feature map. To achieve inter-channel interaction, pointwise convolutions are typically applied afterwards, where the height and width of the convolution kernel are both set to 1. In general, the number of parameters in a separable convolution is [a fraction] that of a regular convolution. .

[0136] This application analyzes the performance changes when replacing ordinary convolutions with separable convolutions and grouped convolutions. As shown in Table 3 (lower part), the grouped convolution variant achieves slightly better performance compared to other convolution methods while minimizing computational cost. Furthermore, the separable convolution variant also exhibits competitive performance with lower computational cost than ordinary convolutions. For example, ordinary convolutions achieve 5.9% accuracy on the SEED-DV dataset with 758 GFLOPs of computation, while separable convolutions achieve 6.1% accuracy with a 35% reduction in computational cost. Meanwhile, with almost the same computational cost, grouped convolutions outperform separable convolutions, improving accuracy by 0.8% to 6.9%. Experimental results indicate that grouping the input and output channels of convolutions does not degrade performance in neural decoding tasks. Channel grouping may be more suitable for aggregating local information while improving computational efficiency.

[0137] Therefore, this application adopts grouped convolution as the core computation method as the final framework of the neural decoding system described in this application, which has the advantages of being suitable for aggregating local information while improving computational efficiency.

[0138] In summary, this application provides a neural decoding system, which is an efficient and powerful framework called NeuroSketch.

[0139] like Figure 3 As shown in one embodiment of this application, an exemplary overall architecture diagram of a neural decoding system (NeuroSketch) is provided. Similar to a two-dimensional convolutional neural network, NeuroSketch takes a two-dimensional representation of the raw EEG signal as input, and then performs initial feature extraction through a preprocessing stage. The subsequent forward propagation process is divided into four feature extraction stages, progressively capturing features from low to high levels. In each stage, the initial component is used to increase the number of feature maps, the downsampling layer reduces the size of the feature maps by half, and the embedding layer remains unchanged. Based on the pyramid method, this application allocates the downsampling layer to the second stage to improve performance and computational efficiency. Subsequently, feature extraction is performed using convolutional blocks, each convolutional block including grouped convolution, batch normalization, ReLU activation function, and pointwise convolution. The final feature vector will be further used for multiple neural decoding tasks. After the four stages, the obtained features are passed through a generalized mean (GeM) pooling layer, which aggregates features along the channel and time dimensions. The pooled representation is then input into a linear layer for final decoding and classification.

[0140] This application uses datasets to evaluate the NeuroSketch neural decoding system, as detailed below. The diverse selection of datasets in the evaluation ensures a comprehensive assessment of different neural decoding tasks.

[0141] 1) Specifications of the scale and version of the NeuroSketch neural decoding system

[0142] This application implements two versions of NeuroSketch at different scales: NeuroSketch-Base (1.4 million parameters) and NeuroSketch-Large (4.2 million parameters). For the baseline models, this application selects representative state-of-the-art models from various fields, including time series models: ModernTCN, MedFormer; computer vision backbone networks: ConvFormer, CAFormer; well-known brain models: DeepConvNet, EEGNet; recent brain models: SpaRCNet; basic intracranial EEG models: seegnificant; and basic EEG models: CBraMod. This application uses officially pre-trained weights for all base models.

[0143] Overall, the neural decoding system described in this application (i.e., NeuroSketch) consistently maintains leading performance in neural decoding tasks. Figure 11 As shown, NeuroSketch outperforms all selected baselines, highlighting its powerful ability to model brain signals across various decoding scenarios. Furthermore, this application notes that, except for the DUIN dataset, NeuroSketch-Base performs similarly to NeuroSketch-Large on all datasets, while on the DUIN dataset, NeuroSketch-Large significantly outperforms NeuroSketch-Base. Figure 11 The diagram shows a comparison of model performance. Specifically, compared to the second-best baseline model, Conformer, the large version of Neural Sketch achieves a 22.1% improvement in accuracy on Chisco-R, while the basic version of Neural Sketch achieves a 9.2% improvement in accuracy on Chisco-I.

[0144] 2) Description of the dataset for the NeuroSketch neural decoding system

[0145] Table 4 shows the average classification accuracy (mean ± standard deviation of three folds) of the models on multiple neural decoding datasets. Higher throughput indicates faster inference speed. Best results are shown in bold. Entries not applicable or not evaluated are indicated by "-". Due to the significant differences between intracranial electroencephalography (iEEG) and electroencephalography (EEG), each pre-trained model was evaluated only on the pre-trained modality, therefore the corresponding cross-modal cells are marked with "-". Furthermore, CBraMod uses a patch size of 200, while ThingsEEG has an input length of 100, therefore CBraMod cannot run on ThingsEEG and is marked with "-". Table 4 reports the excellent performance of NeuroSketch on three speech decoding tasks.

[0146] Table 4: Average classification accuracy of the model on multiple neural decoding datasets (mean ± standard deviation of three folds)

[0147]

[0148] As shown in Table 4, on the DUIN dataset, the large version of Neural Sketch outperforms the second-best baseline model, ConvFormer, by 65.5% in accuracy, highlighting its ability to extract discriminative features from neural signals. As a base model for intracranial EEG, Seegnificant performs poorly on DUIN (accuracy of 5.3%), likely due to its single-layer transformer lacking sufficient capability to handle this challenging neural decoding task. Classifying semantic categories while subjects read or imagine sentences is even more difficult. However, Neural Sketch achieves the best results, demonstrating its robustness in challenging speech decoding scenarios.

[0149] As shown in Table 4, when decoding static images on the FacesHouses dataset, NeuroSketch-Large outperforms the second-best baseline model, ModernTCN, with a 5.2% improvement in accuracy. Furthermore, on the more challenging ThingsEEG dataset, NeuroSketch-Base is competitive with the recent brain model, SPaRCNet. Con-vFormer and CAFormer also demonstrate strong visual decoding performance on the ThingsEEG dataset. For the highly challenging video decoding task, most baseline models perform close to random levels (2.5% accuracy), while NeuroSketch-Large shows a significant advantage with 4.7% accuracy. This result highlights its ability to capture complex neural representations associated with dynamic visual stimuli.

[0150] The results in Table 4 demonstrate that NeuroSketch achieves state-of-the-art performance on both auditory decoding tasks. On the OpenMIIR-P dataset, the EEG-based model CBraMod achieves the highest baseline performance (97.5% accuracy) among brain domain models, highlighting the effectiveness of pre-training and its transferability to auditory decoding.

[0151] In Table 4, to ensure consistency and comparability, this application follows the preprocessing procedures described in the original publications for each dataset.

[0152] For the DUIN dataset, this application follows the preprocessing setup described in the original work. For each subject, 10 SEEG channels were selected according to the original configuration. The signal was downsampled to 1000 Hz, and each trial lasted 2.5 seconds. The classification target corresponds to 61 predefined Chinese words presented to the participants, with the aim of directly decoding spoken content from neural activity.

[0153] For the Chisco dataset, this application analyzed two tasks: silent reading (Chisco-R) and imagined speaking (Chisco-I). Following the approach of the original work, this application removed three noisy channels, retaining the remaining 122 valid EEG channels. All signals were downsampled to 500Hz. The input duration for Chisco-R was set to 5 seconds, and the input duration for Chisco-I was set to 3.3 seconds, aligned with their respective task lengths. For the classification objective, both datasets were labeled with 39 predefined semantic categories.

[0154] For the FacesHouses dataset, this application segments continuous electrocorticography (ECG) recordings into time periods based on stimulus labels, retaining only face and house trials, and applying channel-wise z-score normalization to reduce inter-channel variability. Input data includes 31 to 102 channels per subject, a sampling frequency of 1000 Hz, and a trial duration of 400 ms. Classification labels are defined as binary distinctions between face and house.

[0155] For the ThingsEEG dataset, this application used the preprocessed data published by the original authors and defined a visual stimulus classification task. For each subject, this application loaded a test set from the original Things-EEG dataset, which contained images from 200 different categories, each presented 80 times. This produced a recorded signal of shape [200, 80, 17, 100], where 17 is the number of channels and 100 is the number of time steps. These 17 channels were selected from the occipital and parietal cortex (O1, Oz, O2, PO7, PO3, POz, PO4, PO8, P7, P5, P3, P1, Pz, P2, P4, P6, P8) to focus on the regions most relevant to visual processing. This application assigned labels to the corresponding trials and then reshaped the data to [16000, 17, 100] (i.e., 200 categories × 80 trials), with the corresponding labels reshaped to

[16000] . The data is then randomly shuffled and divided into training, validation, and test sets. Specifically, 20% of the data is used as the test set, while the remaining 80% is divided into three folds; in each round, one fold is used for validation, and the other two folds are used for training. The model input is an EEG segment of shape [17, 100], and the target is a label from one of 200 image categories. This decoding task is performed separately for each subject.

[0156] For the SEED-DV dataset, this application employs the first benchmark task from the original SEED-DV dataset: 40-class classification of fine-grained video concepts to evaluate dynamic visual stimulus decoding. Following the preprocessing protocol described in the original work, this application downsamples the EEG signals to 200Hz, with each trial lasting 2 seconds and containing 62 channels. Classification labels correspond to 40 predefined video concepts. To standardize the evaluation process, this application merges all blocks of data for each subject, randomly shuffles the trials, and divides them into training, validation, and test sets.

[0157] For the OpenMIIR dataset, this application focuses on two tasks: perception (denoted as OpenMIIR-P) and imagination (denoted as OpenMIIR-I). Following the preprocessing protocol of the original work, this application used raw EEG signals sampled at 512Hz without further downsampling. The data were bandpass filtered between 0.5 and 30Hz, and the experiment was segmented according to the audio initiation event, defining a time period for each musical stimulus under the corresponding conditions. For the perception task, this application used condition 1, while for the imagination task, this application summarized the data from conditions 2, 3, and 4. After segmentation, the continuous recordings for each stimulus were further divided into non-overlapping windows of 600 samples each across 64 channels. Then, this application applied channel-wise z-score normalization to each window. Classification labels were defined as 12 different musical stimuli.

[0158] 3) Analysis of Model Depth and Embedding Dimensions of the NeuroSketch Neural Decoding System

[0159] Table 5 shows the model depth and embedding dimension during the infrastructure analysis of this application. In both the one-dimensional convolutional neural network (CNN 1D) and two-dimensional convolutional neural network (CNN 2D) architectures, the kernel size was set to 3. In CNN 1D, the number of channels in the backbone layer increased from 1 to 64, followed by four stages, each consisting of five layers. The number of channels in these stages gradually increased from 128 to 256, 512, and finally to 1024. In CNN 2D, the channel configuration in the backbone layer also increased from 1 to 64. This was followed by four stages, each containing four layers, with the number of channels increasing from 96 to 192, 384, and 768, respectively. In the CNN-GRU architecture, the ratio of convolutional layers to GRU layers was set to 3:1. In the CNN-Transformer architecture, the initial ratio of convolutional layers to Transformer modules was set to 4:1. To fairly analyze how short-term and long-term temporal information affect model performance, this application fixes the total number of layers at 20 and systematically adjusts the proportion of Transformer modules. To systematically evaluate the effectiveness of the proposed block-segmentation method, this application only changes the data segmentation method while keeping the embedding and subsequent self-attention-based feature extraction pipeline unchanged. All other hyperparameters are fixed to ensure that observed performance differences are attributable to the effectiveness of the block-segmentation method.

[0160] Table 5: Model Configuration for Infrastructure Analysis

[0161]

[0162] From a macroscopic perspective, the neural decoding system described in this application optimizes the transformation of the latent space during forward propagation. Specifically, this application analyzes two strategies for increasing the number of feature maps: a stepping method and a skipping method. In the stepping method, the number of feature maps gradually increases: starting from 1, increasing to 64 through the backbone layer, and then progressively increasing to 96, 192, 384, and 768 in four stages, each stage containing four layers. In contrast, the skipping method increases the number of feature maps from 1 to 64 in the backbone layer, and then directly increases it to 384 using an embedding layer. Subsequently, 16 layers are used to refine high-level features. This application also analyzes two strategies for reducing the size of feature maps: a pyramid method and a pagoda method. In the pyramid method, a downsampling layer is placed in each of the last three stages, with a downsampling rate of 2 in each stage. In contrast, the pagoda method places three identical downsampling layers consecutively in the second stage.

[0163] From a microscopic perspective, the neural decoding system described in this application optimizes the computational method. Specifically, this application analyzes two variants of ordinary convolution: grouped convolution and depthwise separable convolution. For grouped convolution, the number of groups is set to 4. For depthwise separable convolution, the number of groups is set to 1, followed by pointwise convolution with a kernel size of 1 to aggregate channel-dimensional information.

[0164] This application provides a detailed description of the final neural decoding architecture to ensure transparency and reproducibility. Given a shape of... Input tensor ,in It refers to the batch size. It is the number of channels. It refers to the length of time, which this application first reshapes into This forms a 2D representation. The reshaping operation aligns with the subsequent convolutional kernel size and ensures that the temporal structure of the data remains intact in the first backbone layer. The reshaped 2D representation then passes through a backbone stage consisting of four Conv2D-batch normalized-ReLU blocks with kernel size [3,3,3,3], padding of [1,1,1,1], and stride of [2,1,1,2]. The input / output channel counts of these four blocks are [1→64], [64→256], [256→64], and [64→96]. Following the preprocessing stage, there are four feature extraction stages. The input / output channel dimensions of these stages are [96→Dstage 1], [Dstage 1→Dstage 2], [Dstage 2→Dstage 3], and [Dstage 3→Dstage 4]. Each stage contains d blocks and has two key components:

[0165] 1. Patch Embedding: In the first module of each stage, a 2D convolution with a kernel size of 3 and a stride of 1 maps the input channels to the output channels, followed by batch normalization. For the first three modules of the second stage, this application uses a stride of 2 to downsample the input; in other cases, when no resolution change is required, the module is simplified to an identity mapping.

[0166] 2. Convolution Module: This component applies grouped 3×3 convolutions (number of groups equal to G) to effectively capture local inter-channel dependencies. This is followed by batch normalization, ReLU activation, and 1×1 convolutions for feature fusion. The output is fed back into the patch embedding input via residual connections.

[0167] After four stages of feature extraction, this application obtained a shape of The tensor, in which and These represent the channel dimension and time dimension after downsampling, respectively. Then, this application applies generalized mean pooling (GeM pooling) to aggregate them into a single representation. Finally, a linear layer is used to generate the class probabilities. This application implements two variants of this architecture: NeuroSketch-Base and NeuroSketch-Large. Their configurations are summarized in Table 6.

[0168] Table 6: Detailed model configurations for NeuroSketch-Base and NeuroSketch-Large

[0169]

[0170] To further enhance data diversity, this application employs a powerful data augmentation strategy, which includes the following techniques:

[0171] Random Shift: This application defines a maximum shift range based on the length of the input sequence. For each training instance, a shift step is randomly sampled from this range. Positive values ​​indicate a forward shift of the sequence, while negative values ​​indicate a backward shift. This approach enhances the model's robustness to uncertainties in stimulus initiation time.

[0172] Noise: This application generates noise from a standard normal distribution that matches the shape of the original data. The noise is scaled using a predefined standard deviation and added to the input to generate noisy data. This method improves the model's ability to recover from signal perturbations.

[0173] Channel masking: During training, a mask is applied to each channel with a specified probability, setting the corresponding channel value to zero. This technique reduces over-reliance on specific channels and promotes better integration of multi-channel information.

[0174] Temporal masking: Similar to channel masking, this application applies a mask along the time dimension. This encourages the model to robustly extract features across different time periods and enhances generalization ability.

[0175] Mixing: Mixing coefficients λ are sampled from a beta distribution parameterized by hyperparameter α = 0.4. λ is used to linearly combine the original samples and randomly selected samples, and their corresponding labels are mixed accordingly. This augmentation method exposes the model to a wider range of data combinations, thereby improving generalization ability.

[0176] Table 7: Training Hyperparameters

[0177]

[0178] Detailed hyperparameters are shown in Table 7. On multiple datasets, this application observed that decoding performance generally improves with increasing training epochs. Based on this observation, this application sets the number of training epochs to 500 in most experiments to fully optimize model performance. However, for the Chisco dataset, the accuracy plateaus at approximately 80 epochs and does not improve further. Therefore, this application limits training on the Chisco dataset to 100 epochs.

[0179] 4) Scaling behavior analysis of the NeuroSketch neural decoding system

[0180] This application analyzes the scaling behavior of NeuroSketch. As shown in Table 8, on the DUIN dataset, NeuroSketch-Large achieves a significant performance improvement over NeuroSketch-Base, while the two models perform similarly on other datasets. To further analyze this phenomenon, this application examines the subject-level performance on the DUIN dataset, and the results are summarized in Table 8.

[0181] Table 8: Scaling behavior of NeuroSketch on the DUIN dataset

[0182]

[0183] This application calculates the mean accuracy and standard deviation across three different cross-validation folds. Furthermore, it demonstrates the relative improvement in accuracy between NeuroSketch-Large and NeuroSketch-Base. NeuroSketch-Large outperforms NeuroSketch-Base across all subjects, although the degree of improvement varies. To visually analyze the relationship between the degree of improvement and sample difficulty, a scatter plot of accuracy versus improvement is presented, as shown below. Figure 12 As shown, this is a scatter plot of NeuroSketch's scaling behavior on the DUIN dataset. The x-axis represents the NeuroSketch-Base accuracy for each subject, and the y-axis represents the relative improvement achieved by NeuroSketch-Large. The scatter plot reveals an overall negative correlation between accuracy and improvement; in other words, the more difficult the sample classification, the more significant the scaling effect. Notably, for subjects with NeuroSketch-Base accuracy below 5%, NeuroSketch-Large produced a relative improvement of over 400%. This indicates that scaling NeuroSketch is effective in solving more challenging tasks.

[0184] 5) Hyperparameter analysis of the NeuroSketch neural decoding system

[0185] Since the architecture of the Neural Decoding System (NeuroSketch) described in this application is based on a CNN-2D backbone, convolutional operations play a central role in its performance. To better understand their impact, this application conducts a hyperparameter analysis, focusing on two key factors: kernel size and the number of groups in grouped convolutions.

[0186] The size of the convolutional kernel directly affects the receptive field, thus controlling the extent to which the model can capture local spatiotemporal dependencies. In NeuroSketch-Large, this application uses a kernel size of 3 as the default setting because smaller kernels are generally more effective at capturing fine-grained dynamics while maintaining lower parameter overhead. To further analyze the impact of kernel size, this application additionally evaluated larger kernels of 5 and 7 on the DUIN dataset. The results are summarized in Table 9, expressed as mean and standard deviation, calculated over three different cross-validation folds.

[0187] Table 9: Results of NeuroSketch on the DUIN dataset with different kernel sizes

[0188]

[0189] These results indicate that larger kernel sizes lead to lower decoding accuracy, meaning that neural decoded signals exhibit transient temporal dynamics. Larger kernels tend to capture a longer range of temporal features, which may dilute short-term patterns that are crucial for accurate decoding in this context. Embodiments of this application provide an example of auditory decoding OpenMIIR, including: "Imperial March" (Star Wars theme), "Serenade," "Hedwig's Theme," "Emperor Waltz," "Mary Had a Little Lamb," "Jingle Bells," "Take Me to a Baseball Game," "The Amazing Chimneyman," "Mary Had a Little Lamb" (instrumental), "Jingle Bells" (instrumental), "Take Me to a Baseball Game" (instrumental), and "The Amazing Chimneyman" (instrumental).

[0190] This application provides an example of visual decoding ThingsEEG, including:

[0191] 1. Aircraft carrier 2. Antelope 3. Back scratcher 4. Balance beam 5. Banana 6. Baseball bat 7. Basil 8. Basketball 9. Bassoon tube 10. Conductor's bat (4) 11. Batter 12. Beaver 13. Bench 14. Bicycle 15. Birthday cake 16. Blowtorch 17. Boat 18. Bok choy 19. Hood; soft cap 20. Bottle opener 21. Stand; baby carrier 22. Bread 23. Breadbox 24. Insect 25. Stroller; off-road vehicle 26. Bullet 27. Round loaf 28. Bush 29. Fried squid 30. Candlestick 31. Handcart 32. Cashew 33. Cat 34. Caterpillar 35. CD player 36. Chain 37. Leather leggings 38. Cheese 39. Cheetah 40. Box (2) 41. Wind chime 42. Chopsticks 43. Anti-slip nails; pry bar 44. Kitchen knife 45. Jacket 46. Cobra 47. Coconut 48. Coffee beans 49. Coffee machine 50. Cookies 51. Le Cordon Bleu (a culinary term referring to fine dining) 52. Onesie 53. Crab 54. Caramel pudding 55. Crepe 56. Crib 57. Croissant 58. Crow 59. Cruise ship 60. Breadcrumbs 61. Cupcakes 62. Dagger 63. Dalmatian 64. Dessert 65. Dragonfly 66. Spinning top (Jewish holiday toy) 67. Drum 68. Duffel bag 69. Eagle 70. Eel 71. Egg 72. Elephant 73. Espresso 74. Face mask 75. Ferry 76. Flamingo 77. Folder 78. 79. Fork 80. Freezer 81. French horn 82. Fruit 83. Garlic 84. Gloves 85. Golf cart 86. Gondola 87. Goose 88. Gopher 89. Gorilla 90. Grasshopper 91. Hand grenade 92. Hamburger 93. Hammer 94. Handbrake 95. Headscarf 96. High chair 97. Hoodie 98. Hummingbird 99. Ice cubes 100. Ice pack 101. Jeep 102. Jelly beans 103. Jukebox 104. Water bottle 105. Knee pads 106. Long-handled spoon 107. Lamb 108. Lampshade 109. Laundry basket 110. Lettuce 111. Firefly 112. Manatee 113. Marijuana 114. Meatloaf 115. Metal detector; 116. Minivan; 117. Modem; 118. Mosquito; 119. Leather flashlight; 120. Music box; 121. Mussels; 122. Bedside table; 123. Okra; 124. Omelet; 125. Onion.126. Orange 127. Orchid 128. Ostrich 129. Pajamas 130. Black Panther 131. Paperweight 132. Pear 133. Chili (1) 134. Pheasant 135. Pickaxe 136. Pie 137. Pigeon 138. Pocket 139. Small Folding Knife 140. Popcorn 141. Popsicle 142. Opossum 143. Pretzel 144. Pug 145. Punch (2) 146. Wallet 147. Radish 148. Raspberry 149. Recorder 150. Rhinoceros 151. Robot 152. Rooster 153. Small Rug 154. Sailboat 155. Sandals 156. Sandpaper 157. Sausage 158. Green Onion 159. Scallop 160. 161. Scooter 162. Seagull 163. Seaweed 164. Seed 165. Skateboard 166. Sled 167. Sleeping bag 168. Slide 169. Slingshot 170. Snowshoes 171. Scraper; Spatula 172. Spoon 173. Touring car 174. Stethoscope 175. Strawberry 176. Submarine 177. Suit 178. T-shirt 179. Table 180. Taillight 181. Tape recorder 182. Television 183. Crown 184. Tick 185. Ketchup 186. Pliers 187. Tool 188. Top hat 189. Treadmill 190. Bandeau top 191. Turkey 192. Unicycle 193. Bench vise 194. Volleyball 195. Wallpaper 196. Walnut Wheat 197. Wheelchair 198. Windshield 199. Wine 200. Wok.

[0192] This application provides a language decoding Du-IN example, as shown in Table 10 below. Table 10: Chinese characters and their corresponding English translations dataset.

[0193]

[0194] In summary, this invention discloses a lightweight and efficient neural decoding framework, belonging to the field of brain-computer interface and neural decoding technology. The framework first normalizes and augments the original one-dimensional EEG signal through a data preprocessing module; a feature map reshaping module converts the processed one-dimensional signal into a two-dimensional feature map; a feature map preprocessing module extracts low-level local representations from the two-dimensional feature map; and an embedding module implements feature connections and scale alignment between different semantic levels. Each feature extraction stage employs a step-by-step feature connection strategy, combined with a pyramid structure for scale alignment; the convolution module achieves intra-group feature interaction through multi-head group convolution and integrates cross-group features using pointwise convolution; finally, the decoding module maps the output features to the corresponding decoding category. This invention significantly improves the decoding accuracy and inference speed of neural decoding through structural optimization and module design of the decoding framework.

[0195] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0196] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0197] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0198] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A neural decoding system, comprising: The method comprises the following steps: a preprocessing module is configured to preprocess an input raw electroencephalogram signal to obtain a preprocessed feature map; a multi-stage feature extraction module is configured to extract high-dimensional features of the preprocessed feature map; the multi-stage feature extraction module comprises at least two-stage feature extraction modules, and each stage feature extraction module comprises a feature integration module and a convolution module; the feature integration module is configured as a first processing layer of each stage feature extraction module, and is configured to realize feature conversion or spatial scale alignment between a current feature processing stage and a previous feature processing stage; the convolution module is configured as a second processing layer of each stage feature extraction module, and is configured to realize group feature interaction through multi-head grouped convolution and realize cross-group feature fusion through point-by-point convolution, and output the high-dimensional features; a decoding module is configured to map the high-dimensional features extracted by the multi-stage feature extraction module to a spatial distribution of a target decoding category to obtain a predicted decoding category output.

2. The neural decoding system of claim 1, wherein, The preprocessing module comprises: a data preprocessing unit configured to normalize and enhance data of the raw electroencephalogram signal to generate diversified electroencephalogram signal training samples; a feature map remodeling unit configured to convert the electroencephalogram signal training samples into a two-dimensional feature map; a feature map preprocessing unit configured to extract low-level local representations from the two-dimensional feature map to obtain preprocessed feature maps.

3. The neural decoding system of claim 2, wherein, The feature map remodeling unit remodels one-dimensional input of electroencephalogram signals into two-dimensional feature maps wherein denotes batch size, denotes number of channels, denotes number of time steps.

4. The neural decoding system of claim 2, wherein, The feature map preprocessing unit performs preliminary feature extraction on the two-dimensional feature map through four consecutive feature preprocessing units, and each feature preprocessing unit comprises a two-dimensional convolution layer, a batch normalization layer and a nonlinear activation function.

5. The neural decoding system of claim 1, wherein, The feature integration module comprises: a feature conversion unit configured at the starting position of each stage feature extraction module to increase the number of feature maps to initialize higher semantic level representation information; the feature conversion units of each stage feature extraction module gradually increase the channel number of the feature maps in different feature extraction stages in a step-by-step strategy, so that the feature extraction process gradually progresses from low-level features to high-level semantic features layer by layer; a spatial scale alignment unit configured at the starting position of each stage feature extraction module to compress the size of the feature maps to initialize coarse-grained semantic representation; the spatial scale alignment units of each stage feature extraction module realize significant compression of the size of the early feature maps by concentrating the downsampling units in the second stage feature extraction module.

6. The neural decoding system of claim 1, wherein, The convolution module comprises: a multi-head grouped convolution unit configured to divide the input feature map into multiple groups according to the channel dimension, and independently perform convolution operation in each group to realize effective interaction of the group features; a point-by-point convolution unit configured to fuse the cross-group features of the multiple groups, and integrate feature information from different groups through a 1×1 convolution kernel to enhance the global correlation between the cross-group features.

7. The neural decoding system of claim 6, wherein, The multi-head grouped convolution unit divides the input feature map into multiple heads in the channel dimension, each head contains a fixed number of channels, and convolution operation is independently performed in each head.

8. The neural decoding system of claim 1, wherein, The decoding module comprises: The generalized mean pooling unit is configured to perform spatial compression on the input high-dimensional feature map, control different forms of pooling operations through a trainable power index, and obtain a pooled feature; The linear mapping unit is configured to perform dimension transformation on the pooled feature, map the dimension-transformed feature to a target decoding category space through a fully connected layer, and output a predicted classification.

9. A method of neural decoding, comprising: The method comprises: performing preprocessing on an input raw electroencephalogram signal to obtain a preprocessed feature map; performing multi-stage feature extraction on the preprocessed feature map to obtain a high-dimensional feature of the preprocessed feature map; The multi-stage feature extraction comprises at least two stages of feature extraction; wherein each stage of feature extraction comprises: using a feature integration module as a first processing layer of each stage of feature extraction to realize feature conversion and spatial scale alignment between the current feature processing stage and the previous feature processing stage; using a convolution module as a second processing layer of each stage of feature extraction to realize intra-group feature interaction through multi-head grouped convolution and cross-group feature fusion through point-by-point convolution, and output the high-dimensional feature; mapping the high-dimensional feature obtained through the multi-stage feature extraction to a spatial distribution of a target decoding category to obtain a predicted decoding category output.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 9. The processor executes the computer program to implement the neural decoding method of claim 9.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the neural decoding method of claim 9.

Images