Method, product adapted to reconstruct continuous long sequence imagined speech

Abstract

This invention discloses a method and product adapted for reconstructing continuous long sequences of imagined speech, relating to the field of brain-computer interface technology. The invention designs a novel reconstruction model, which first extracts the memory of an EEG signal that meets the model input requirements from a subject through a feature embedding unit, and fuses and embeds the subject's identity information to obtain fused memory features. Then, a feature preprocessing layer captures multi-dimensional and multi-scale characteristics from the fused memory features and simultaneously performs adaptive weight calibration on the channel dimension. Subsequently, preprocessed features are generated through feature fusion. Next, a fine-grained sequence modeling unit performs deep temporal analysis on the preprocessed features to obtain a high-level feature representation. Finally, an output regression layer predicts based on the high-level feature representation to obtain the subject's corresponding Mel spectrum. Compared to existing patents, this invention achieves a significant performance improvement.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and more specifically, to: 1. a method adapted for reconstructing continuous long sequences of imagined speech; 2. a computer program product. Background Technology

[0002] Brain-computer interface-based imagination speech decoding technology has become a research hotspot in the fields of neuroengineering and human-computer interaction. The core logic of this technology is to bypass the damaged peripheral speech motor pathways, directly capture the neural electrical activity signals generated by the human brain during silent recitation, decode and reconstruct the speech content that the patient wants to express through algorithms, and finally convert it into speech output that can be understood by the outside world, providing these patients with an alternative to traditional oral communication.

[0003] Existing continuous reconstruction methods still face challenges in long-sequence imagined speech tasks, primarily due to the lack of adaptive mechanisms for dual dynamic changes. Firstly, while most existing research has addressed the significant subject-specific differences in EEG signals by using static embedding to extract subject-specific features for different individuals, these methods generally lack dynamic adaptation mechanisms, making it difficult to maintain stable feature tracking performance amidst complex signal variations. Although subsequent studies have achieved dynamic extraction of subject information, improving model flexibility to some extent, these methods are still insufficient in handling the non-stationary dynamic features of the same subject over time, resulting in insufficient long-term memory capacity in long-sequence decoding and an inability to simultaneously address inter-subject differences and intra-subject dynamic non-stationarity. Secondly, for the lateral sparse dynamic changes in acoustic features, traditional models typically rely on standard attention mechanisms. However, in high-noise environments, these methods struggle to distinguish key information from background interference, hindering the recognition of sparse but crucial neural representations and limiting the accurate extraction of fine acoustic structures under extremely low signal-to-noise ratios.

[0004] Although there are already patents (for example, Chinese invention patent No. 2024115262655 discloses a method for generating and reconstructing an imagined speech model, and the proposed imagined speech reconstruction model can be represented as DMF2Mel) that have been studied, there is still room for improvement in its performance. Summary of the Invention

[0005] Based on this, it is necessary to address the performance limitations of DMF2Mel as proposed in existing patents by providing methods and products adapted for reconstructing continuous long sequences of imagined speech.

[0006] This invention is achieved using the following technical solution: In a first aspect, the present invention discloses a method adapted for reconstructing continuous long sequences of imagined speech, comprising: The EEG signals generated by the target subject during continuous verbal imagery live Process it to meet the specifications required by the model input and match it with its identity information ID. live Together, we input the trained and adapted reconstruction model of continuous long sequences of imagined speech to obtain the Mel spectrogram representing the corresponding imagined speech. live .

[0007] The reconstruction model for adapting to continuous long sequences of imagined speech includes: a feature embedding unit, a feature preprocessing unit, a fine-grained sequence modeling unit, and an output regression unit; the feature embedding unit is used for the EEG signal (EGG) of a subject that meets the model input requirements. res Memory retrieval is performed, and the subject's identity information (ID) is fused and embedded to obtain fused memory features F. ENH The feature preprocessing layer is used to extract features from F. ENH The process captures multi-dimensional and multi-scale characteristics and simultaneously performs adaptive weight calibration along the channel dimension, thereby generating preprocessed features F through feature fusion. fusion Fine-grained sequence modeling is used for F fusion Perform deep temporal analysis to obtain high-level feature representation F High The output regression layer is used to represent F based on high-level feature representations. High A prediction is made to obtain the corresponding Mel spectrum for the subject.

[0008] This method adapted for reconstructing continuous long sequences of imagined speech implements the method or process according to embodiments of this disclosure.

[0009] Secondly, the present invention discloses a computer program product, comprising a computer program. When executed by a processor, the computer program implements the steps of the method adapted to reconstructing a continuous long sequence of imagined speech as disclosed in the first aspect.

[0010] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention designs a novel reconstruction model for the task of reconstructing continuous long-sequence imagined speech. First, a feature embedding unit extracts the EEG signal of a subject that meets the model input requirements and fuses and embeds the subject's identity information to obtain fused memory features, achieving personalized feature mapping. Then, a feature preprocessing layer captures multi-dimensional and multi-scale characteristics from the fused memory features and simultaneously performs adaptive weight calibration on the channel dimension. Preprocessed features are then generated through feature fusion, providing highly stable input data for the subsequent Mel spectrogram reconstruction process. Next, a fine-grained sequence modeling unit performs deep temporal analysis on the preprocessed features to obtain high-level feature representations, achieving temporal dependency capture and fine-grained speech feature modeling. Finally, an output regression layer predicts based on the high-level feature representations to obtain the subject's corresponding Mel spectrogram. Compared to existing patents, this invention achieves significant performance improvements, enhancing the reconstruction accuracy and robustness of EEG signals to Mel spectrograms, making it highly suitable for reconstructing continuous long-sequence imagined speech.

[0011] 2. The feature embedding part in the reconstruction model of this invention forms an adaptive closed loop through identity-aware memory and momentum update, updates and integrates subject-specific information in real time, and adaptively adjusts the model response according to individual characteristics, effectively suppressing error accumulation in the long sequence decoding process.

[0012] 3. In the reconstruction model of this invention, the feature preprocessing unit constructs a robust global acoustic skeleton by parallel fusion of channel dimension calibration and multi-scale spatiotemporal perception, thus bridging the representation gap between global temporal trends and local fine-grained acoustic details.

[0013] 4. The fine-grained sequence modeling part of the reconstructed model of this invention introduces a feedforward layer and an adaptive sparse attention layer. The former can achieve joint optimization of feedforward feature enhancement and temporal local modeling, while the latter can maintain linear complexity and significantly reduce computational overhead and memory usage while ensuring the ability to model long sequence contexts, thereby improving the accuracy of feature focusing and the overall generalization ability of the model. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 An architecture diagram of the reconstruction model for adapting continuous long sequence imagined speech provided in Embodiment 1 of the present invention; Figure 2 for Figure 1Structure diagram of the memory retrieval layer; Figure 3 for Figure 1 Structure diagram of the mid-temporal coding layer; Figure 4 for Figure 1 Structure diagram of the middle channel coding layer; Figure 5 for Figure 1 Structure diagram of the feature processing unit; Figure 6 for Figure 5 Structure diagram of the feature processing sublayer; Figure 7 for Figure 6 Structure diagram of the feedforward layer; Figure 8 for Figure 6 Structure diagram of the adaptive sparse attention layer. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that when a component is said to be "installed on" another component, it can be directly on the other component or it may be in a component that is centered on it. When a component is said to be "set on" another component, it can be directly set on the other component or it may also be in a component that is centered on it. When a component is said to be "fixed to" another component, it can be directly fixed to the other component or it may also be in a component that is centered on it.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0019] Example 1 See Figure 1 This demonstrates the structural design of the reconstruction model (referred to as DA2Mel) for adapting continuous long sequence imagined speech provided in this embodiment 1. It can be divided into four parts according to function: feature embedding part, feature preprocessing part, fine-grained sequence modeling part, and output regression part.

[0020] The following is a detailed introduction to each of these four parts: I. The feature embedding unit is used to process the EEG signals of a subject that meet the model input requirements. res Memory retrieval is performed, and the subject's identity information (ID) is fused and embedded to obtain fused memory features F. ENH .

[0021] The core of the feature embedding design lies in addressing significant individual differences in EEG signals and the non-stationary changes in neural activity of the same subject at different time points.

[0022] On the one hand, identity information is highly distinctive and used to distinguish different subjects. Introducing it can enhance the model's adaptability and personalization to individual differences in EEG signals among different subjects, thereby improving the accuracy of imagined speech reconstruction. On the other hand, because neural representations are affected by the dynamic fluctuations of physiological and psychological states and drift over time, the signal distribution undergoes continuous changes. This multi-scale change across individuals and time presents a severe challenge to model building—the static features extracted by conventional pre-training will quickly mismatch with the dynamically changing neural states, making it difficult for the model to maintain stable decoding performance. Therefore, the feature embedding part introduces a real-time tracking mechanism—which accurately captures the dynamic fluctuations in the time dimension through memory retrieval, enabling the model to continuously adapt to evolving neural representations.

[0023] Specifically, the feature embedding unit can be designed to include: a one-hot encoding layer, a linear projection layer, a convolutional processing layer, a memory extraction layer, and a memory fusion layer.

[0024] Then, in the feature embedding part: 101. The one-hot coding layer performs one-hot coding on the subject's identity information ID.

[0025] 102. The linear projection layer performs a linear projection transformation on the output of the one-hot encoding layer (which identifies the subject's identity) to obtain the embedded feature F. EMB .

[0026] 103. The convolutional processing layer receives the EEG signals from the subject that meet the model input requirements. res Extract the basic convolutional features F basic .

[0027] 104. Memory retrieval layer combined with embedded features F EMB From the basic convolutional feature F basic Extract personalized memory features F PER .

[0028] For more details, see Figure 2The memory retrieval layer can be divided into: query construction sublayer, local memory retrieval sublayer, global memory retrieval sublayer, and memory update sublayer.

[0029] Among them, the query constructs sub-layers for the basic convolutional features F basic Perform mean pooling to generate query features F Q The local memory extraction sublayer first relies on the embedded features F EMB From the memory bank Mem bank The local memory vector Mem corresponding to the index local Then, the query feature F is applied through an attention mechanism. Q With local memory vector Mem local Perform association modeling to obtain local similarity weights Sim local and with the local memory vector Mem local Perform a weighted summation to obtain the corresponding local context feature. local The global memory retrieval sublayer first retrieves the memory bank Mem. bank Flattening is performed to obtain the global memory vector Mem global Then, the query feature F is applied through an attention mechanism. Q With global memory vector Mem global Perform association modeling to obtain global similarity weights Sim global and the global memory vector Mem global Perform a weighted summation to obtain the corresponding global context feature. global The memory update sublayer only works during the model training phase and not during the model inference phase.

[0030] It should be noted that the memory bank Mem bank The model is configured with a capacity of Z subjects, which are divided into Z regions. At the start of model training, a series of memory vectors are randomly generated—that is, the memory vector of the z-th subject is randomly generated in the z-th region; z∈[1,Z]. Therefore, for the subject with the identity information ID, its memory vector can be represented as Mem. bank [Sub ID ].

[0031] Therefore, when the memory update sublayer works, it uses the basic convolutional features F basic The memory bank Mem is updated iteratively using a preset fixed momentum coefficient β as the basis for updates. bank This ensures the model can identify and value signal changes relevant to the subjects—specifically, the memory update sublayer first processes the base convolutional features F... basic Mean pooling is performed along the time dimension to obtain the feature mean F. ave Then, according to the fixed momentum coefficient β, the characteristic mean Fave Iterative updates to the memory bank Mem bank The memory bank vector corresponding to the subject is used to realize the memory bank Mem. bank The dynamic update. Of course, the above iterative update can be expressed mathematically as: ; In the formula, New_Mem bank [Sub ID [] represents the updated memory vector; Mem bank [Sub ID ] represents the memory vector before the update.

[0032] This method of iteratively updating the memory bank Mem using a fixed momentum coefficient β enables the memory bank Mem to be updated more efficiently. bank Maintain stability and smoothness during the update process, avoiding drastic fluctuations in the memory vectors and improving the memory bank Mem. bank The convergence stability and robustness of the characteristic representation.

[0033] The value of the fixed momentum coefficient β can be selected according to the actual situation. In this embodiment 1, the fixed momentum coefficient β is set to 0.2.

[0034] Furthermore, it should be noted that the model training phase is divided into two stages according to the sequence of rounds; in the first stage, personalized memory features F... PER For local context features local In the second stage, personalized memory features F PER Includes local context features. local and global context feature Context global .

[0035] 105. Memory fusion layer on basic convolutional features F basic Embedded features F EMB Personalized memory characteristics F PER Multi-source feature fusion is performed to obtain fused memory feature F. ENH —It enhances subject-specific information, making it easier for the model to identify and emphasize signal components relevant to the subject.

[0036] Specifically: In the first stage, the memory fusion layer combines the basic convolutional features F basic Embedded features F EMB Personalized memory characteristics F PER (In reality, only the local context feature is present) local After unifying the dimensions, the features are then superimposed to obtain the fused memory feature F corresponding to this stage. ENH .

[0037] In the second stage, the memory fusion layer applies local context features. local Global Context Features global Perform learnable weighted processing (i.e., the memory fusion layer converts local context features) local Global Context Features global (Assigning appropriate learnable weights and weighting), and then combining them with the basic convolutional features F. basic Embedded features F EMB After unifying the dimensions, the features are superimposed to obtain the fused memory feature F corresponding to this stage. ENH .

[0038] In this way, by integrating the different stages, we can focus on modeling the individual characteristics of the subjects in the early stage to ensure the accurate representation of individual differences, and introduce global common features in the later stage to enhance the generalization ability of the model. This approach balances personalized adaptation and global modeling, effectively improving the stability and accuracy of long-sequence imagined speech reconstruction.

[0039] In summary, the feature embedding unit forms an adaptive closed loop through the identity-aware memory bank and momentum update, updates and integrates subject-specific information in real time, and adaptively adjusts the model response based on individual characteristics, effectively suppressing error accumulation during long sequence decoding.

[0040] II. Feature preprocessing layer is used to extract fused memory features F ENH The process captures multi-dimensional and multi-scale characteristics and simultaneously performs adaptive weight calibration along the channel dimension, thereby generating preprocessed features F through feature fusion. fusion .

[0041] In existing technologies, the lack of early structural guidance weakens the global modeling ability of late-stage attention mechanisms. Simultaneously, when processing features after deep transformation, data distortion or redundancy can easily lead to biased attention allocation, resulting in decreased reconstruction performance. These factors can cause related models to get stuck in local optima, failing to effectively construct a global semantic path and ultimately affecting the overall speech reconstruction effect. Therefore, the feature preprocessing layer proposes a dual-dynamic adaptive network—emphasizing a coarse-to-fine modeling strategy. Lightweight structural guidance is introduced in the early stages of the network, explicitly modeling the global spatial structure and channel importance as prior knowledge. This enables more efficient and accurate processing of complex spatiotemporal features in EEG signals, providing highly stable input data for subsequent Mel spectrogram reconstruction, ensuring the accuracy and reliability of the reconstruction results.

[0042] Specifically, the feature preprocessing layer can be designed to include: a spatiotemporal coding layer, a channel coding layer, and a feature fusion layer.

[0043] Therefore, in the feature preprocessing layer: 201. Spatiotemporal coding layer for fusion memory features F ENH Multi-scale spatial and temporal feature extraction is performed, and then the spatiotemporally enhanced feature F is integrated and output. ST .

[0044] For details, please refer to Figure 3 The spatiotemporal coding layer can be divided into: temporal convolution branch, spatial convolution branch, multi-scale dilated convolution branch, average pooling layer, and splicing and fusion layer.

[0045] The temporal convolution branch employs parallel temporal convolutions with different kernel sizes (in this embodiment 1, two parallel 1D convolutions with kernel sizes of 3 and 5 are used) to process the fused memory features F. ENH Appropriate convolution processing is performed to capture its short-term temporal scale characteristics; the spatial convolution branch employs parallel spatial convolutions with different kernel sizes (in this embodiment 1, two parallel 1D convolutions with kernel sizes of 7 and 9 are used) to process the fused memory features F. ENH Appropriate convolution processing is performed to capture its spatial distribution characteristics; the multi-scale dilated convolution branch employs parallel dilated convolutions with different dilation rates (in this embodiment 1, three parallel dilated convolutions with dilation rates of 6, 12, and 18 are used) to process the fused memory feature F. ENH Appropriate convolutional processing is performed to capture its multi-scale contextual characteristics; average pooling layers are used to process the fused memory features F. ENH Average pooling is performed; the concatenation and fusion layer concatenates the outputs of the temporal convolutional branch, spatial convolutional branch, multi-scale dilated convolutional branch, and average pooling layer along the channel dimension, and then performs convolution to fuse features, resulting in the spatiotemporal enhanced feature F. ST .

[0046] The spatiotemporal coding layer designed above can enhance the model's ability to perceive the local structure and global context of the signal, and improve the richness and robustness of feature representation.

[0047] 202. Channel coding layer for fused memory features F ENH Adaptive weight calibration is performed along the channel dimension to generate channel-enhanced features F. Channel .

[0048] The channel coding layer is designed based on the SE channel attention mechanism. See [link / reference] Figure 4 The channel coding layer can be divided into: a global average pooling layer, a dimension flattening layer, a ReLU activation function, a Sigmoid activation function, a dot product calculation layer, and two fully connected layers.

[0049] Among them, the global average pooling layer is used to fuse memory features FENH Perform global average pooling along the sequence length dimension to obtain the global compressed feature F. zip ; Dimensional flattening layer compresses global features F zip The process involves: dimensional flattening; compression of the channel dimension by the output of the first fully connected layer according to a preset compression ratio (16 in this embodiment); a non-linear transformation of the output of the first fully connected layer using the ReLU activation function; restoration of the channel dimension to its original state by the output of the ReLU activation function in the second fully connected layer; mapping of the output of the second fully connected layer using the Sigmoid activation function (generally to the [0,1] interval) to obtain the channel attention weights; and a dot product layer fusing the memory features F. ENH The channel-enhanced feature F is obtained by performing a channel-wise, element-wise multiplication with the channel attention weights. Channel .

[0050] The channel coding layer designed above analyzes the correlation between channels and adaptively learns the importance of different channel features. It assigns greater weight to feature channels with rich information and less weight to redundant or noisy feature channels, thereby achieving effective feature enhancement and suppression of redundant information, and improving the model's ability to focus on key channel features and its anti-interference performance.

[0051] 203. Feature fusion layer enhances spatiotemporal features F ST and channel enhancement features F Channel We perform learnable weighted processing with regularization to obtain the preprocessed features F. fusion .

[0052] Specifically, the feature fusion layer enhances the spatiotemporal features F ST Channel enhancement feature F Channel Assign appropriate learnable weights and perform weighted summation, then sum the weighted results and perform regularization to obtain the preprocessed features F. fusion .

[0053] In summary, the feature preprocessing unit constructs a robust global acoustic skeleton by merging channel dimension calibration and multi-scale spatiotemporal perception in parallel, thus bridging the representation gap between global temporal trends and local fine-grained acoustic details.

[0054] III. The fine-grained sequence modeling unit is used to process the preprocessed features F fusion Perform deep temporal analysis to obtain high-level feature representation F High .

[0055] Specifically, the fine-grained sequence modeling unit can be designed to include: a dimension transpose layer, a feature processing unit, a feature splicing layer, a linear mapping layer, and a residual fusion layer.

[0056] Therefore, in the fine-grained sequence modeling section: 301. Dimensional transpose layer on preprocessed features F fusion Perform dimension transpose processing.

[0057] 302. The feature processing unit performs continuous fine-grained optimization of the output of the dimension transpose layer across the entire depth range.

[0058] More specifically, the feature processing unit includes: N feature processing sub-layers, N feature splicing layers, and N fully connected layers.

[0059] See Figure 5 The input to the (n+1)th feature processing sublayer is the output of the dimension transpose layer, and the input to the nth fully connected layer is the output of the nth fully connected layer. The output is first concatenated with the output of the dimension transpose layer in the feature dimension by the (n+1)th feature concatenation layer, and then transformed in dimension by the (n+1)th fully connected layer; n∈[0,N-1]; where the output of the dimension transpose layer is also used as the output of the 0th fully connected layer.

[0060] It should be noted that, considering the balance between model size and performance, N is generally set to 4.

[0061] The feature processing sublayer is the core of the fine-grained sequence modeling unit. Since each feature processing sublayer has the same structural design, see [link to relevant documentation]. Figure 6 Taking a feature processing sub-layer as an example, it includes: an adaptive sparse attention layer, a convolutional processing layer, a linear projection layer, a regularization layer, and two feedforward layers.

[0062] In this feature processing sublayer: The first feedforward layer performs feedforward feature extraction on the input of the feature processing sub-layer; the adaptive sparse attention layer performs adaptive sparse attention computation on the output of the first feedforward layer to focus on high attention score features; the convolutional processing layer performs convolution processing on the output of the adaptive sparse attention layer; the linear projection layer performs linear projection on the output of the convolutional processing layer; the regularization layer regularizes the output of the linear projection layer; the second feedforward layer performs feedforward feature extraction on the output of the regularization layer to obtain the output of the feature processing sub-layer.

[0063] In this embodiment 1, both the feedforward layer and the adaptive sparse attention layer employ special designs: I. See Figure 7 The feedforward layer includes: a layer normalization layer, two linear transformation layers, two GELU activation functions, two dimension transpose layers, a depthwise separable convolutional layer, a regularization layer, and a residual connection layer.

[0064] So, in a certain feedforward layer: The input of the feedforward layer is first normalized by a layer normalization layer, then expanded by the first linear transformation layer, followed by the first GELU activation function to enhance the nonlinear expression of the features. After that, it is adapted to the convolutional input format by the first dimension transpose layer, and then the long-distance contextual interaction of the sequence features is realized by the depthwise separable convolutional layer. Then, the nonlinear expression of the features is further enhanced by the second GELU activation function. Then, the format is restored by the second dimension transpose layer, and the feature dimension is restored by the second linear transformation layer. Finally, the feature regularization is performed by the regularization layer, and the output of the feedforward layer is obtained by element-wise addition with the input of the feedforward layer through the residual connection layer.

[0065] The feedforward layer of the above design introduces a hole depth separable convolution to achieve joint optimization of feedforward feature enhancement and temporal local modeling.

[0066] II. See Figure 8 The adaptive sparse attention layer includes: a feature projection layer, a normalization layer, a feature compression layer, an adaptive span prediction layer, a sparse weighted computation layer, and a residual connection layer.

[0067] Therefore, in a certain adaptive sparse attention layer: The input to the adaptive sparse attention layer is first normalized by a normalization layer, and then the query feature Q, key feature K, and value feature V are generated by a feature projection layer. Next, the query feature Q is passed through an adaptive span prediction layer to generate an adaptive mask M. Finally, the key feature K and value feature V are compressed by a feature compression layer to obtain the compressed key feature K. cmp Value characteristics V cmp Next, the query feature Q, adaptive mask M, and compressed key feature K are used. cmp Value characteristics V cmp Sparse attention is calculated and weighted through a sparse weighted computation layer to obtain attention feature O; finally, attention feature O is superimposed with the input of the adaptive sparse attention layer to obtain the output of the adaptive sparse attention layer.

[0068] More specifically: ① The adaptive span prediction layer is divided into: mean aggregation layer, two linear layers, ReLU activation function, normalization layer, and mask generation layer.

[0069] Therefore, the query vector Q is first averaged along the time dimension by a mean aggregation layer, then undergoes an initial linear transformation by the first linear layer, followed by a nonlinear transformation by the ReLU activation function, then a second linear transformation by the second linear layer, then normalized by a normalization layer, then rounded up by a mask generation layer, and then processed according to the Top-k method (i.e., selecting the k positions with the largest values ​​after rounding up and setting them to 1, and setting the remaining positions to 0) to obtain the adaptive mask M.

[0070] ② The feature compression layer is divided into: key compression branch and value compression branch. The two branches have the same structure and both include: block reshaping layer and multilayer perceptron.

[0071] Therefore, in the key compression branch, the key feature K is first segmented by the block reshaping layer according to the preset block size, and then compressed by the multilayer perceptron to obtain the compressed key feature K. cmp In the value compression branch, the value feature V is first segmented by the block reshaping layer according to a preset block size, and then compressed by the multilayer perceptron to obtain the compressed value feature V. cmp .

[0072] ③ The calculation formula for the sparse weighted computing layer is: ; In the formula, softmax(.) represents the softmax function; d k Represents the compressed bond feature K cmp Dimensions.

[0073] It's important to note that conventional global attention requires calculating attention weights across the entire sequence, resulting in computational complexity that increases quadratically with sequence length, leading to low inference efficiency for long sequences. Conventional fixed sparse attention uses a manually set fixed span, failing to adapt to the differences in feature distribution across various signals. The aforementioned adaptive sparse attention layer, however, abandons both of these conventional attention methods. It constrains feature distribution through a global feature anchoring strategy, fundamentally suppressing irrelevant random background noise. This targeted modeling capability of acoustic sparsity allows the model to keenly capture dynamic transients in acoustic states even amidst complex neurophysiological fluctuations.

[0074] The aforementioned adaptive sparse attention layer reduces computation by compressing key-value features into blocks and adaptively learns the effective attention span of each sequence based on query features. While ensuring the ability to model long sequence contexts, it maintains linear complexity and significantly reduces computational overhead and memory usage. Furthermore, the adaptive span can better match the dynamic changes in time-series data such as EEG signals, improving the accuracy of feature focusing and the overall generalization ability of the model.

[0075] 303. The feature splicing layer splices the output of the dimension transpose layer and the outputs of N feature processing sub-layers within the feature processing unit.

[0076] 304. The linear mapping layer performs linear transformation and dimension mapping on the output of the feature concatenation layer to unify it to the same dimension.

[0077] 305. The residual fusion layer superimposes the output of the linear mapping layer with the output of the dimension transpose layer to obtain the high-level feature representation F. High .

[0078] IV. Output regression layer is used for high-level feature representation F High A prediction is made to obtain the corresponding Mel spectrum for the subject.

[0079] Specifically, the output regression layer can be designed to include: a linear layer.

[0080] Therefore, in the output regression layer: the linear layer represents the high-level feature representation F. High A linear transformation is performed to reduce the dimensionality and output the Mel spectrum.

[0081] In summary, the reconstruction model for continuous long-sequence imagined speech presented above employs a dual-dynamic adaptive network, which effectively captures local features and short-term dependencies in EEG signals. It also utilizes an adaptive sparse attention mechanism, which efficiently models long-term signals, particularly suitable for long-sequence imagined speech data. Furthermore, it employs feature embedding to integrate individual information from specific subjects, enabling the model to provide personalized responses based on the characteristics of different participants. These novel structural designs and introductions facilitate accurate reconstruction of continuous imagined speech features, significantly improving the accuracy of long-term imagined speech reconstruction and effectively solving the technical challenge of accurately reconstructing continuous speech in existing imagined speech reconstruction methods.

[0082] Example 2 This embodiment 2 provides a specific application of the reconstruction model adapted for continuous long sequence imagined speech in embodiment 1—used in the model inference stage to reconstruct continuous long sequence imagined speech. In other words, this embodiment 2 discloses a method adapted for reconstructing continuous long sequence imagined speech, which includes the following steps: The EEG signals generated by the target subject during continuous verbal imagery live Process it to meet the specifications required by the model input and match it with its identity information ID. live Together, we input the trained and adapted reconstruction model of continuous long sequences of imagined speech to obtain the Mel spectrogram representing the corresponding imagined speech. live .

[0083] Among them, electroencephalogram (EEG) signals liveIf the data is acquired at the first frequency (generally 8192Hz), the processing procedure is as follows: EEG signals are processed... live First, artifacts are removed (a multi-channel Wiener filter is recommended, as it minimizes the mean square error between the output signal and the desired signal). Then, the signal is resampled to a common average value and downsampled to a second frequency (typically 64Hz) to meet the model input specifications. This process is necessary because the acquired EEG signals may contain noise and fluctuating or non-stationary signal components—these interferences can negatively impact the model's processing quality. Furthermore, the high frequency of the acquired EEG signals is detrimental to model processing.

[0084] Of course, if EEG live If the specifications already meet the model input requirements, no further processing is needed; the model can be directly input.

[0085] It is important to note that: The reconstruction model adapted to continuous long sequence imagined speech is trained and validated based on a sample EEG dataset (divided into training and validation sets according to a certain ratio), and the model with the best validation performance is retained as the trained reconstruction model adapted to continuous long sequence imagined speech.

[0086] The sample EEG dataset can use either labeled EEG data from the target subjects or labeled EEG data from non-target subjects. Models trained using the former perform better, while models trained using the latter show slightly lower performance but are still acceptable. If the sample EEG dataset is known, it can be used directly; otherwise, it is recommended to construct it yourself: provide the subject with some known continuous speech (with its corresponding Mel spectrogram as the true label), and have the subject mentally imagine these continuous speech sounds while simultaneously collecting the subject's EEG signals using an EEG signal acquisition device. This constitutes the subject's sample EEG dataset. Furthermore, it should be noted that when using the subject's sample EEG dataset for model training, it should be processed to meet the model input requirements—the processing procedure is similar to the loading process described above and will not be repeated here.

[0087] It should be noted that deep learning gradient descent is recommended for training, and a loss function should be constructed for backpropagation to update model parameters. The loss function is constructed using a linear combination of weights—it can be designed as follows: ; In the formula, Loss represents the loss function; Loss1 represents the Pearson correlation coefficient loss between the model prediction and the actual Mel spectrogram; Loss2 represents the L1 loss between the model prediction and the actual Mel spectrogram; α represents the weighting coefficient; 0 < α < 1.

[0088] Example 3 To verify the effectiveness and superiority of the reconstruction model for adapting continuous long sequences of imagined speech provided in the above embodiments, this embodiment 3 is based on the SparrKULee EEG dataset (involving 85 Dutch-speaking subjects with normal hearing – sequentially labeled as subject 1 to subject 85; memory bank Mem) bank Simulation and ablation experiments were conducted using a configuration with a capacity of 85 subjects, and the results were compared with several existing advanced deep learning methods (including VLAAI, HappyQuokka, SSM2Mel, and DMF2Mel).

[0089] To fully verify the robustness of the model, a control experiment was set up in Example 3: 1. Preserving the story paradigm: The experiment was conducted using EEG data from the 1st to the 70th subjects throughout the process to evaluate the model's ability to generalize to speech content not seen during the training phase.

[0090] 2. Preserving the subject paradigm: The experiment was conducted using EEG data from all 85 subjects—the model was trained and validated based on the EEG data of subjects 1 to 70, and tested on the EEG data of subjects 71 to 85, in order to verify the model’s ability to adapt to the neural activity patterns of new and unknown subjects.

[0091] The Pearson correlation coefficient was used to quantitatively evaluate the control experimental protocol, and the weighted average of the Pearson correlation coefficients under the two scenarios was used as the overall quantitative evaluation.

[0092] First, a comparison was made with several existing advanced deep learning methods, and the results are shown in Table 1.

[0093] Table 1 Comparison Results

[0094] As shown in Table 1, under the story-preserving paradigm, DA2Mel exhibits excellent generalization performance, with a Pearson correlation coefficient of 0.081, significantly outperforming existing comparison methods—nearly 9.5% higher than DMF2Mel—fully demonstrating DA2Mel's outstanding advantage in handling unknown stories. Under the subject-preserving paradigm, DA2Mel also performs excellently, with a Pearson correlation coefficient of 0.050, still outperforming existing comparison methods—nearly 4.2% higher than DMF2Mel, demonstrating good cross-individual adaptive ability. In overall quantitative assessment, DA2Mel continues to comprehensively outperform existing comparison methods—nearly 8.3% higher than DMF2Mel.

[0095] In summary, DA2Mel offers a significant performance improvement over DMF2Mel.

[0096] Second, the core part of DA2Mel was systematically ablated, and its performance was compared under the story paradigm. The results are shown in Table 2.

[0097] Table 2 Comparison Results

[0098] As shown in Table 2, removing the fine-grained sequence modeling unit resulted in a 22.22% decrease in model performance, confirming its crucial role in capturing temporal dependencies and fine-grained speech features. Removing the feature preprocessing layer resulted in an 18.52% decrease in model performance, indicating its indispensable role in noise reduction and feature enhancement of EEG signals. Removing the feature embedding unit resulted in an 8.64% decrease in model performance, verifying its core importance in mapping EEG neural activity features to the speech feature space.

[0099] In summary, DA2Mel significantly outperforms any model variant lacking a single core module in overall performance. Through the coordinated efforts of its core components, DA2Mel effectively ensures the robustness and prediction accuracy of EEG signal speech modeling tasks.

[0100] Third, the internal structure and parameters of DA2Mel were ablated, and its performance was compared under the story-preserving paradigm. The results are shown in Table 3.

[0101] Table 3 Comparison Results

[0102] As shown in Table 3, in the feature preprocessing layer: after replacing the dual-path parallel architecture with a serial connection, the Pearson correlation coefficient of the model dropped to 0.076, indicating a significant performance degradation; when only a single path was retained, the Pearson correlation coefficient of the model dropped to 0.078 and 0.075 respectively, neither reaching the performance level of the complete model, confirming the advantages of the parallel structure in efficiently fusing spatial topology and channel spectral features, as well as the complementarity of the two types of features; in the fine-grained sequence modeling layer: after replacing the adaptive sparse attention with a self-attention mechanism, the Pearson correlation coefficient of the model dropped to 0.074, indicating a performance decline, verifying the superiority of the adaptive masking mechanism in suppressing noise and focusing on speech-related features; after removing the concatenation of the outputs of the feature processing sublayer and the dimension transpose layer, making the feature processing layer a single serial data processing process, the Pearson correlation coefficient of the model dropped to 0.076, indicating a performance decline, suggesting that continuous fine-grained optimization that always incorporates the output of the dimension transpose layer is necessary throughout the entire depth of the model.

[0103] Based on the experimental results in Tables 2-3, it is demonstrated that DA2Mel achieves superior performance compared to existing methods through the collaborative design of the layer structure.

[0104] Example 4 This embodiment 4 discloses a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method for reconstructing continuous long sequence imagined speech disclosed in embodiment 2.

[0105] The computer equipment can be either a mobile terminal or a fixed terminal. Examples of the former include mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (such as in-vehicle navigation terminals); examples of the latter include digital TVs and desktop computers.

[0106] This embodiment 4 also discloses a readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the steps of the method adapted to reconstructing a continuous long sequence of imagined speech disclosed in embodiment 2 are performed.

[0107] The readable storage medium may include, but is not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0108] This embodiment 4 also discloses a computer program product, including a computer program. When executed by a processor, the computer program implements the steps of the method for reconstructing continuous long sequences of imagined speech disclosed in embodiment 2.

[0109] It should be noted that the computer program used to execute the above can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as C or similar languages. The computer program can be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer through any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN).

[0110] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method adapted for reconstructing continuous long sequences of imagined speech, characterized in that, include: The EEG signals generated by the target subject during continuous verbal imagery live Process it to meet the specifications required by the model input and match it with its identity information ID. live Together, we input the trained and adapted reconstruction model of continuous long sequences of imagined speech to obtain the Mel spectrogram representing the corresponding imagined speech. live ; The reconstruction model for adapting to continuous long sequences of imagined speech includes: a feature embedding unit, a feature preprocessing unit, a fine-grained sequence modeling unit, and an output regression unit; the feature embedding unit is used for the EEG signal (EGG) of a subject that meets the model input requirements. res Memory retrieval is performed, and the subject's identity information (ID) is fused and embedded to obtain fused memory features F. ENH The feature preprocessing layer is used to extract features from F. ENH The process captures multi-dimensional and multi-scale characteristics and simultaneously performs adaptive weight calibration along the channel dimension, thereby generating preprocessed features F through feature fusion. fusion Fine-grained sequence modeling is used for F fusion Perform deep temporal analysis to obtain high-level feature representation F High The output regression layer is used to represent F based on high-level feature representations. High A prediction is made to obtain the corresponding Mel spectrum for the subject.

2. The method for reconstructing continuous long sequences of imagined speech according to claim 1, characterized in that, The feature embedding unit includes: a one-hot encoding layer, a linear projection layer, a convolutional processing layer, a memory extraction layer, and a memory fusion layer; In the feature embedding section: the one-hot encoding layer performs one-hot encoding on the ID; the linear projection layer performs a linear projection transformation on the output of the one-hot encoding layer to obtain the embedded feature F. EMB Convolutional processing layers from EGG res Extract the basic convolutional features F basic ; Memory retrieval layer combined with F EMB From F basic Extract personalized memory features F PER ; Memory fusion layer for F basic F EMB F PER F is obtained by multi-source feature fusion. ENH .

3. The method for reconstructing continuous long sequences of imagined speech according to claim 2, characterized in that, The memory retrieval layer includes: query construction sublayer, local memory retrieval sublayer, global memory retrieval sublayer, and memory update sublayer; In the memory retrieval layer: query constructs sublayers for F basic Perform mean pooling to generate query features F Q The local memory retrieval sublayer first relies on F EMB From the memory bank Mem bank The local memory vector Mem corresponding to the index local Then, through the attention mechanism, F Q With Mem local Perform association modeling to obtain local similarity weights Sim local and with Mem local Perform a weighted summation to obtain the corresponding local context feature. local The global memory retrieval sublayer first retrieves Mem... bank Flattening is performed to obtain the global memory vector Mem global Then, through the attention mechanism, F Q With Mem global Perform association modeling to obtain global similarity weights Sim global and with Mem global Perform a weighted summation to obtain the corresponding global context feature. global The memory update sublayer only works during the model training phase and not during the model inference phase. Among them, when the memory update sublayer is working, it uses F basic Mem is updated iteratively using a preset fixed momentum coefficient β as the basis for updates. bank ; Or / and, the model training phase is divided into the first phase and the second phase according to the order of rounds; In the first stage, F PER For Context local The memory fusion layer will F basic F EMB F PER After unifying the dimensions, the layers are superimposed to obtain the F corresponding to that stage. ENH ; In the second stage, F PER Including Context local and Context global The memory fusion layer for Context local Context global Perform learnable weighted processing and compare it with F basic F EMB After unifying the dimensions, the layers are superimposed to obtain the F corresponding to that stage. ENH .

4. The method for reconstructing continuous long sequences of imagined speech according to claim 1, characterized in that, The feature preprocessing layer includes: a spatiotemporal coding layer, a channel coding layer, and a feature fusion layer; In the feature preprocessing layer: the spatiotemporal coding layer for F ENH Multi-scale spatial and temporal feature extraction is performed, and then the spatiotemporally enhanced feature F is integrated and output. ST ; Channel coding layer for F ENH Adaptive weight calibration is performed along the channel dimension to generate channel-enhanced features F. Channel Feature fusion layer for F ST and F Channel Perform learnable weighted processing with regularization to obtain F. fusion .

5. The method for reconstructing continuous long sequences of imagined speech according to claim 4, characterized in that, The spatiotemporal coding layer includes: temporal convolutional branch, spatial convolutional branch, multi-scale dilated convolutional branch, average pooling layer, and splicing and fusion layer; In the spatiotemporal coding layer: the temporal convolution branch employs parallel temporal convolutions with different kernel sizes to process F... ENH Appropriate convolution processing is performed to capture its short-time scale characteristics; the spatial convolution branch employs parallel spatial convolutions with different kernel sizes to process F... ENH Appropriate convolution processing is performed to capture its spatial distribution characteristics; the multi-scale dilated convolution branch employs parallel dilated convolutions with different dilation rates to process F. ENH Appropriate convolutional processing is performed to capture its multi-scale contextual characteristics; average pooling layers are used to process F... ENH Average pooling is performed; the concatenation and fusion layer is used to concatenate the outputs of the temporal convolutional branch, spatial convolutional branch, multi-scale dilated convolutional branch, and average pooling layer along the channel dimension, and then perform convolution to fuse features, obtaining F. ST ; Or / and, the channel coding layer includes: a global average pooling layer, a dimension flattening layer, a ReLU activation function, a Sigmoid activation function, a dot product calculation layer, and two fully connected layers; In the channel coding layer: the global average pooling layer for F ENH Perform global average pooling along the sequence length dimension to obtain the global compressed feature F. zip ; Dimensional flattening layer for F zip The process involves: dimensional flattening; compression of the channel dimensions by the output of the first fully connected layer according to a preset compression ratio; a non-linear transformation of the output of the first fully connected layer using the ReLU activation function; restoration of the channel dimensions to their original state by the output of the ReLU activation function in the second fully connected layer; mapping of the output of the second fully connected layer to the channel attention weights using the Sigmoid activation function; and finally, a dot product layer that calculates the F... ENH Perform a channel-wise, element-wise dot product with the channel attention weights to obtain F. Channel ; Or / and, feature fusion layer to F ST F Channel Assign appropriate learnable weights and perform weighted summation, then sum the weighted results and perform regularization to obtain F. fusion .

6. The method for reconstructing continuous long sequences of imagined speech according to claim 1, characterized in that, The fine-grained sequence modeling unit includes: a dimension transpose layer, a feature processing unit, a feature concatenation layer, a linear mapping layer, and a residual fusion layer; In the fine-grained sequence modeling section: the dimension transpose layer pairs F fusion The process involves: performing dimension transposition; continuously fine-grained optimization of the output of the dimension transposition layer across the entire depth range; concatenating the output of the dimension transposition layer and the outputs of N feature processing sub-layers within the feature processing unit; applying linear transformation and dimension mapping to the output of the feature concatenation layer to unify it to the same dimension; and superimposing the output of the linear mapping layer with the output of the dimension transposition layer to obtain F. High .

7. The method for reconstructing continuous long sequences of imagined speech according to claim 6, characterized in that, The feature processing unit includes: N feature processing sub-layers, N feature splicing layers, and N fully connected layers; The input to the (n+1)th feature processing sublayer is the output of the dimension transpose layer, and the input to the (n)th fully connected layer is the output of the nth fully connected layer. The output is first concatenated with the output of the dimension transpose layer in the feature dimension by the (n+1)th feature concatenation layer, and then transformed in dimension by the (n+1)th fully connected layer; n∈[0,N-1]; the output of the dimension transpose layer is used as the output of the 0th fully connected layer; The feature processing sublayer includes: an adaptive sparse attention layer, a convolutional processing layer, a linear projection layer, a regularization layer, and two feedforward layers. In a certain feature processing sub-layer: the first feedforward layer performs feedforward feature extraction on the input of the feature processing sub-layer; the adaptive sparse attention layer performs adaptive sparse attention calculation on the output of the first feedforward layer to focus on high attention score features; the convolutional processing layer performs convolution processing on the output of the adaptive sparse attention layer; the linear projection layer performs linear projection on the output of the convolutional processing layer; the regularization layer performs regularization on the output of the linear projection layer; the second feedforward layer performs feedforward feature extraction on the output of the regularization layer to obtain the output of the feature processing sub-layer.

8. The method for reconstructing continuous long sequences of imagined speech according to claim 7, characterized in that, The feedforward layer includes: a layer normalization layer, two linear transformation layers, two GELU activation functions, two dimension transpose layers, a depthwise separable convolutional layer, a regularization layer, and a residual connection layer; In a certain feedforward layer: the input of the feedforward layer is first processed by a layer normalization layer, then by the first linear transformation layer to expand the feature dimension, then by the first GELU activation function to enhance the non-linear expression of the feature, then by the first dimension transpose layer to adapt the convolution input format, then by the depthwise separable convolution layer to realize long-distance contextual interaction of sequence features, then by the second GELU activation function to further enhance the non-linear expression of the feature, then by the second dimension transpose layer to restore the format, then by the second linear transformation layer to restore the feature dimension, then by the regularization layer to perform feature regularization processing, and finally by adding the input of the feedforward layer element-wise through the residual connection layer to obtain the output of the feedforward layer; Or / and, the adaptive sparse attention layer includes: feature projection layer, normalization layer, feature compression layer, adaptive span prediction layer, sparse weighted computation layer, and residual connection layer; In a certain adaptive sparse attention layer: the input of this adaptive sparse attention layer is first normalized by a normalization layer, and then the query feature Q, key feature K, and value feature V are generated by a feature projection layer; then Q is passed through an adaptive span prediction layer to generate an adaptive mask M, and K and V are compressed by a feature compression layer to obtain the compressed key feature K. cmp Value characteristics V cmp Next, Q, M, and K cmp V cmp Sparse attention features O are obtained by performing sparse attention computation and weighting through a sparse weighted computation layer; finally, O is superimposed with the input of the adaptive sparse attention layer to obtain the output of the adaptive sparse attention layer. The adaptive span prediction layer includes: a mean aggregation layer, two linear layers, a ReLU activation function, a normalization layer, and a mask generation layer. In the adaptive span prediction layer: Q is first averaged along the time dimension by the mean aggregation layer, then undergoes an initial linear transformation by the first linear layer, followed by a nonlinear transformation by the ReLU activation function, then a second linear transformation by the second linear layer, then normalized by the normalization layer, then rounded up by the mask generation layer, and finally processed according to the Top-k method to obtain the adaptive mask M. The calculation formula for the sparse weighted computing layer is: ； In the formula, softmax(.) represents the softmax function; d k Represents the compressed bond feature K cmp Dimensions.

9. The method for reconstructing continuous long sequences of imagined speech according to claim 1, characterized in that, The output regression layer includes: a linear layer; In the output regression layer: the linear layer represents the high-level feature representation F. High A linear transformation is performed to reduce the dimensionality and output the Mel spectrum.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method adapted to reconstructing a continuous long sequence of imagined speech as described in any one of claims 1-9.

Images