Method for generating imagined speech reconstruction model, imagined speech reconstruction method

Abstract

The present application relates to a method for generating an imagined speech reconstruction model and an imagined speech reconstruction method, wherein the generating method comprises: obtaining a sample EEG signal, wherein the sample EEG signal is an EEG signal generated by a subject when he or she is imagining continuous speech; conducting supervised training on the imagined speech reconstruction model through the sample EEG signal, wherein the imagined speech reconstruction model comprises a feature embedding module, a feature preprocessing module and a state space module; the feature preprocessing module comprises a first U network module and an external attention mechanism block connected in sequence, wherein the first U network module comprises a first convolutional layer, a second convolutional layer, a first target module, a third convolutional layer, a second target module, a third target module and a fourth convolutional layer connected in sequence, wherein the target module comprises a plurality of structural state space modules connected in sequence. The present invention solves the problem that the imagined speech reconstruction model generated by the current method for generating the imagined speech reconstruction model cannot reconstruct continuous speech more accurately.

Description

Technical Field

[0001] The present application relates to the field of brain-computer interface, and in particular to a method for generating an imagined speech reconstruction model and an imagined speech reconstruction method. Background Art

[0002] Imagined speech is an emerging research direction in the field of brain interfaces, which aims to synthesize the information of stimulus audio directly from brain signals, that is, imagined speech reconstruction. Imagined speech reconstruction may be the key to opening a new era of human communication, from the current voice- or text-based communication to brain signal-based communication. In addition, imagined speech reconstruction technology can help patients who cannot speak, or those who may lose their voice in the future. Among a series of brain signal recording technologies, invasive technologies provide a detailed view of neural activity, but are limited by the necessity of surgical implantation. In contrast, electroencephalography, as a non-invasive technology, is safer and more acceptable. However, due to the non-stationary nature and high signal-to-noise ratio of EEG signals, as well as human physiological specificity, the EEG signals of different subjects performing the same imagined speech task may be very different, and the EEG signals of the same subject for the same imagined speech task at different times may also be very different. Accurate decoding of imagined speech EEG signals remains a challenging problem.

[0003] Most current methods use convolutional neural networks and recurrent neural networks (as imagined speech reconstruction models) to process and classify EEG signals generated during imagined speech. Due to the limitations of the current methods for generating imagined speech reconstruction models, although the imagined speech reconstruction models generated by them have made progress in reconstructing speech features at the word or letter level, they still have low accuracy in accurately reconstructing continuous speech features, especially in long time series at the minute level.

[0004] There is currently no effective solution to the problem that the imagined speech reconstruction model generated by the current imagined speech reconstruction model generation method cannot reconstruct continuous speech more accurately. Summary of the invention

[0005] The present invention provides a method for generating an imaginary speech reconstruction model and an imaginary speech reconstruction method to solve the problem that the imaginary speech reconstruction model generated by the current method for generating an imaginary speech reconstruction model cannot reconstruct continuous speech more accurately.

[0006] The present invention provides a method for generating an imaginary speech reconstruction model, comprising:

[0007] Acquire sample EEG signals, where the sample EEG signals are EEG signals generated by the subject when the subject is performing continuous speech imagination; perform supervised training on the imagined speech reconstruction model using the sample EEG signals to obtain a trained imagined speech reconstruction model;

[0008] Wherein, the imagined speech reconstruction model includes a feature embedding module, a feature preprocessing module and a state space module;

[0009] The feature embedding module includes a first normalization layer, a first multi-head attention mechanism block, a second normalization layer and a first feedforward module connected in sequence, the input of the first normalization layer is the identity information of the subject, and the input of the first multi-head attention mechanism block also includes the sample EEG signal;

[0010] The feature preprocessing module includes a first U network module and an external attention mechanism block connected in sequence, the input of the first U network module is the output of the first feedforward module, the first U network module includes a first convolutional layer, a second convolutional layer, a first target module, a third convolutional layer, a second target module, a third target module and a fourth convolutional layer connected in sequence, the input of the third convolutional layer also includes the output of the second convolutional layer, the input of the second target module also includes the output of the first convolutional layer, the input of the third target module also includes the output of the first convolutional layer, and the first target module, the second target module and the third target module each include a plurality of structural state space modules connected in sequence;

[0011] The state space module includes a fourth target module, a first selective state space module, a fifth target module and a second selective state space module. The input of the fourth target module includes the output of the external attention mechanism block and the output of the first feedforward module. The input of the first selective state space module includes the output of the fourth target module and the output of the first feedforward module. The input of the fifth target module includes the output of the first selective state space module and the output of the first feedforward module. The input of the second selective state space module includes the output of the fifth target module and the output of the first feedforward module. The fourth target module and the fifth target module both include a plurality of multi-head attention modules connected in sequence. The output of the second selective state space module generates a mel-spectrogram after passing through a linear layer.

[0012] In a second aspect, the present invention provides an imaginary speech reconstruction method, comprising:

[0013] Inputting the EEG signal generated by the user when performing continuous speech imagination into the imagined speech reconstruction model, and obtaining the mel-spectrogram of the imagined speech of the user through the imagined speech reconstruction model;

[0014] Wherein, the imagined speech reconstruction model is obtained by the method for generating the imagined speech reconstruction model described in the first aspect.

[0015] Compared with the related art, the present invention applies the U-Net network structure based on the structural state space model to EEG signal processing to capture the local features of the EEG signal and the short-term dependencies of these features, and proposes a new state space module that combines a selective state space module with a multi-head attention module for efficiently modeling long sequences of EEG signals, which is particularly effective for long sequence data of imagined speech. In addition, due to the subject correlation of EEG signals, we also introduced a feature embedding module and an external attention mechanism block to integrate the information of specific subjects and customize the model response according to the characteristics of individual subjects. The proposal and introduction of these new structures help to accurately reconstruct the features of continuous imagined speech and improve the accuracy of imagined speech reconstruction at the long sequence level. It solves the problem that the imagined speech reconstruction model generated by the current method for generating the imagined speech reconstruction model cannot reconstruct continuous speech more accurately.

[0016] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a structural diagram of an imagined speech reconstruction model in some embodiments of the present invention;

[0018] Figure 2 is a structural diagram of a U network module in some embodiments of the present invention;

[0019] Figure 3 is a structural diagram of a structural state space module in some embodiments of the present invention;

[0020] Figure 4 is a structural diagram of a multi-head attention module in some embodiments of the present invention;

[0021] Figure 5 is a block diagram of a selective state space module in some embodiments of the present invention;

[0022] Figure 6 is a structural diagram of a feedforward module in some embodiments of the present invention;

[0023] Figure 7 is a structural diagram of a convolution module in some embodiments of the present invention;

[0024] Figure 8 It is a flow chart of a method for generating an imagined speech reconstruction model in some embodiments of the present invention. DETAILED DESCRIPTION

[0025] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0026] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish objects of the same type and do not represent a specific ordering of the objects.

[0027] In an embodiment of the present invention, a method for generating an imaginary speech reconstruction model is provided, and the method flow comprises the following steps:

[0028] Step S110, obtaining a sample EEG signal, where the sample EEG signal is an EEG signal generated when the subject is performing continuous speech imagination.

[0029] Specifically, some known continuous speech can be given to the subject, and the subject can be asked to silently imagine these continuous speech, and the EEG signal of the subject is collected by using an EEG signal collection device to obtain sample EEG signals. In the subsequent model training process, the corresponding training labels are determined according to these known continuous speech.

[0030] There may be some artifacts in the EEG signal, and removing the artifacts can improve the model training effect. In some embodiments, obtaining the sample EEG signal includes: collecting the EEG signal generated by the subject when performing continuous speech imagination at a first frequency; removing the artifacts in the EEG signal through a multi-channel Wiener filter, and then re-referencing the EEG signal to a common average, and finally downsampling the EEG signal to a second frequency to obtain a sample EEG signal. Among them, the first frequency can be 8192 Hz, and the second frequency can be 64 Hz. The following is a specific example to specifically illustrate the process of collecting and removing artifacts from the sample EEG signal.

[0031] In one embodiment, the original EEG signal generated by the subject when perceiving or imagining continuous speech is first acquired at a sampling rate of 8192 Hz, and then a multi-channel Wiener filter is used to remove artifacts in the original EEG signal. The processed EEG signal is then re-referenced to a common average, and finally the processed EEG signal is downsampled to 64 Hz to obtain a sample EEG signal.

[0032] Specifically, the various types of EEG signals collected may contain noise information and contain fluctuating and non-stationary signal components. These interference information may affect the learning quality of the network model. Therefore, this implementation preprocesses the original EEG data.

[0033] The design of the multi-channel Wiener filter can be achieved by solving an optimization problem, that is, finding a filter that minimizes the mean square error between the output signal and the desired signal. Let x(n) be the multi-channel signal vector, d(n) be the desired signal vector, and v(n) be the noise vector. The goal of the Wiener filter is to find a filter H(z) that makes the output signal y(n) close to the desired signal d(n):

[0034]

[0035] in represents the convolution operation and h(n) is the impulse response of the filter.

[0036] The mean square error (MSE) is defined as:

[0037]

[0038] The Wiener filter solves the filter coefficients by minimizing the MSE:

[0039]

[0040] Next, re-referencing the EEG signal to a common mean is a common EEG signal processing step. For all recorded EEG channels, the mean of the signal at each time point is calculated:

[0041]

[0042] In the above formula, CAR(n) is the common average reference value at time point n, x i (n) is the signal value of the ith channel at time point n, and N is the total number of channels. Then, for each channel signal, the calculated common average value is subtracted point by point. The formula is:

[0043]

[0044] In the above formula, is the signal value of the i-th channel at time point n after processing.

[0045] Step S120, supervised training is performed on the imagined speech reconstruction model using sample EEG signals to obtain a trained imagined speech reconstruction model.

[0046] In some embodiments, the loss function used in the training of the imagined speech reconstruction model is:

[0047]

[0048] Among them, Loss represents the total loss, Loss1 represents the first loss, and Loss2 represents the second loss. represents the weight coefficient, x i and i All represent the model prediction values. represents the true label value, and Represents x i and i The sample mean of . i is an index, indicating the sample number, and n is the total number of samples.

[0049] Reference Figure 1 ,The imagined speech reconstruction model includes a feature embedding module, a feature preprocessing module and a state space module.

[0050] The feature embedding module includes a first normalization layer, a first multi-head attention mechanism block, a second normalization layer and a first feedforward module connected in sequence. The input of the first normalization layer is the identity information of the subject, and the input of the first multi-head attention mechanism block also includes a sample EEG signal.

[0051] It should be noted that, in the present invention, the output of each object connected sequentially is the input of the latter according to the connection order. For example, the output of the first normalization layer is the input of the first multi-head attention mechanism block, and the output of the second normalization layer is the input of the first feedforward module.

[0052] Based on the above structure, the feature processing steps of the feature embedding module include:

[0053] The subject's identity information is layer-normalized to obtain the first feature, and the first feature and the sample EEG signal are fused through a multi-head attention mechanism to obtain the second feature. The second feature is layer-normalized to obtain the third feature, and finally the third feature is processed through a feedforward module to obtain the fourth feature. The fourth feature is the output of the feature embedding module.

[0054] The subject's identity information can be identifiable information such as ID, which is mainly used to distinguish different subjects. During training, the sample EEG signal and the subject's identity information are input into the feature embedding module together. Its main purpose is to enhance the adaptability and personalization of the model to the individual differences of EEG signals of different subjects. By considering each subject's unique ID as part of the model input, the model can better capture these individual differences, thereby improving the accuracy of imagined speech reconstruction.

[0055] Specifically, the feature embedding module receives the preprocessed EEG signal and integrates it with the layer-normalized subject identity information embedding through multi-head attention, thereby enhancing the feature representation of subject-specific information, and the model can identify and attach importance to the signal parts related to each subject. The features containing the subject identity information will be again normalized and fed-forwarded to maintain the stability and consistency of the data, providing the model with richer information for subsequent processing tasks.

[0056] Reference Figure 1 and Figure 2 The feature preprocessing module includes a first U network module and an external attention mechanism block connected in sequence. The input of the first U network module is the output of the first feedforward module. The first U network module includes a first convolutional layer, a second convolutional layer, a first target module, a third convolutional layer, a second target module, a third target module and a fourth convolutional layer connected in sequence. The input of the third convolutional layer also includes the output of the second convolutional layer. The input of the second target module also includes the output of the first convolutional layer. The input of the third target module also includes the output of the first convolutional layer. The first target module, the second target module and the third target module all include multiple structural state space modules connected in sequence, that is, the output of the previous structural state space module is the input of the next structural state space module.

[0057] Among them, the first U network module is obtained by improving the traditional U network, mainly by replacing some of the convolutional layers with target modules, and each target module includes multiple structural state space modules connected in sequence. The input of the first U network module is the input of the first convolutional layer, the output of the fourth convolutional layer is the output of the first U network module, the input of the first structural state space module in the first target module is the input of the first target module, and the output of the last structural state space module in the first target module is the output of the first target module. Similarly, the second target module and the third target module are the same.

[0058] In the first U network module, in addition to the sequential connection of each part, there are also some residual connections. The output of the first convolutional layer is connected to the output of the second target module through residual connection as the input of the third target module, and the output of the first convolutional layer is connected to the output of the third convolutional layer through residual connection as the input of the second target module, and the output of the second convolutional layer is connected to the output of the first target module through residual connection as the input of the third convolutional layer.

[0059] In some embodiments, the first to fourth convolutional layers all use 1 1 convolution.

[0060] Reference Figure 3 In some embodiments, the structural state space module includes a first linear layer, a structural state space model, a first Dropout layer and a third normalization layer connected in sequence, and the input of the first Dropout layer also includes the output of the first linear layer. The input of the first linear layer is the input of the structural state space module, and the output of the third normalization layer is the output of the structural space module.

[0061] The expression of the structural state space model is:

[0062]

[0063] Among them, x t represents the input of the structural state space model at time t, h t and h t-1 They represent the hidden states at time t and time t-1 respectively, , and C Both represent the dynamic matrices of the structural state space model, y t Represents the output of the structural state-space model at time t.

[0064] In the structural state-space module, in addition to connecting each part in sequence, the output of the linear layer is connected to the output residual of the structural state-space model and then used as the input of the first Dropout layer.

[0065] Since a structural state space model is adopted in the U network, in the above embodiment, the first U network module can also be called a U-Net module of the structural state space model.

[0066] In these embodiments, the first U network module can capture local features and short-term dependencies in EEG signals by combining a structural state space model with a U-Net architecture, which is crucial for understanding subtle changes in the signal. The features processed and outputted contain rich spatiotemporal information, which can provide effective input for subsequent Mel-spectrogram reconstruction. At the same time, external attention is an attention mechanism that aims to enhance the model's sensitivity to individual differences and improve the effect of feature extraction. By introducing an external attention mechanism, the model can better capture features related to individual differences in EEG signals. This is particularly important for processing EEG signals of different subjects. The attention maintains linear complexity, making the calculation efficient and suitable for processing long sequence data. The design of the feature preprocessing module enables the model to process EEG signals more effectively, capture complex spatiotemporal features and individual differences, thereby improving the accuracy of Mel-spectrogram reconstruction. The design of this module fully considers the characteristics of EEG signals and provides strong support for the decoding of imagined speech.

[0067] Among them, the feature processing steps of the external attention mechanism block include:

[0068] The block input (the output of the first U network module) is queried to obtain the first feature, the first extrinsic attention weight is multiplied by the first feature to obtain the second feature, the second feature is normalized to obtain the third feature, and finally the second extrinsic attention weight is multiplied by the third feature to obtain the output of the extrinsic attention mechanism block.

[0069] Reference Figure 1 The state space module includes a fourth target module, a first selective state space module, a fifth target module and a second selective state space module. The input of the fourth target module includes the output of the external attention mechanism block and the output of the first feedforward module. The input of the first selective state space module includes the output of the fourth target module and the output of the first feedforward module. The input of the fifth target module includes the output of the first selective state space module and the output of the first feedforward module. The input of the second selective state space module includes the output of the fifth target module and the output of the first feedforward module. The fourth target module and the fifth target module both include a plurality of multi-head attention modules connected in sequence. The output of the second selective state space module generates a mel-spectrogram after passing through a linear layer.

[0070] Among them, the output of the external attention mechanism block and the output of the first feedforward module are residually connected as the input of the fourth target module, the output of the fourth target module and the output of the first feedforward module are residually connected as the input of the first selective state space module, the output of the first selective state space module and the output of the first feedforward module are residually connected as the input of the fifth target module, and the output of the fifth target module and the output of the first feedforward module are residually connected as the input of the second selective state space module. The fourth and fifth target modules both include multiple multi-head attention modules, wherein the input of the first multi-head attention module and the output of the last multi-head attention module are the input and output of the corresponding target module, respectively.

[0071] Reference Figure 4 In some embodiments, the multi-head attention module includes a first pre-convolutional layer, a second U network module, a second feedforward module, a second multi-head attention mechanism block, a first convolutional module and a third feedforward module connected in sequence, the second U network module is the same as the first U network module, the input of the second multi-head attention mechanism block also includes the output of the second U network module, the input of the first convolutional module also includes the input of the second multi-head attention mechanism block, the input of the third feedforward module also includes the input of the first convolutional module, and the output of the third feedforward module is connected with the input residual as the output of the multi-head attention module.

[0072] The second U network module has the same structure as the first U network module, and in some embodiments, it can be a U-Net module based on a structural state space model.

[0073] It should be noted that in other embodiments, the multi-head attention module can also directly adopt the multi-head attention mechanism block, but adopting the module structure in the above embodiment has better effect.

[0074] Specifically, the expressions of the first multi-head attention mechanism block and the second multi-head attention mechanism block are:

[0075]

[0076] in, Q , K and V denote the query vector, key vector and value vector respectively, Indicates the preset zoom ratio. Attention ( Q , K , V ) represents the weighted attention score.

[0077] Reference Figure 5In some embodiments, the first selective state space module and the second selective state space module each include a second pre-convolution layer, a third U network module, a fourth feedforward module, a selective state space model, a second convolution module and a fifth feedforward module connected in sequence, the third U network module is the same as the first U network module, the input of the selective state space model also includes the output of the third U network module, the input of the second convolution module also includes the input of the selective state space model, the input of the fifth feedforward module also includes the input of the second convolution module, and the output of the fifth feedforward module is connected with the input residual as the output of the selective state space module;

[0078] The selective state-space model has the same expression as the structural state-space model, except that the selective state-space model introduces a parameterized matrix that allows , and C The matrix changes depending on the input.

[0079] In the above embodiment, the overall architecture of the selective state space module and the multi-head attention module is the same, the difference is that the multi-head attention module is used before the convolution module, and the structural state space model is used before the convolution module. The other parts, such as pre-convolution, U network module, feedforward module and convolution module are the same.

[0080] Specifically, each feedforward module in the present invention can adopt the same structure, referring to Figure 6 , exemplarily, the feedforward module includes a normalization layer, a linear layer, a Swish activation layer, a Dropout layer, a linear layer, and a Dropout layer connected in sequence. Figure 7 , and the first convolution module and the second convolution module include a layer normalization layer, a gradual convolution, a Glu activation layer, a depth convolution, a batch normalization, a Swish activation layer, a point-by-point convolution and a Dropout layer connected in sequence, and the convolution module input is connected to the output residual of the Dropout layer as the convolution module output.

[0081] In summary, the present invention applies the U-Net network structure based on the structural state space model to EEG signal processing to capture the local features of the EEG signal and the short-term dependencies of these features, and proposes a new state space module that combines a selective state space module with a multi-head attention module for efficiently modeling long sequences of EEG signals, which is particularly effective for long sequence data of imagined speech. In addition, due to the subject correlation of EEG signals, we also introduced a feature embedding module and an external attention mechanism block to integrate the information of specific subjects and customize the model response according to the characteristics of individual subjects. The proposal and introduction of these new structures help to accurately reconstruct the features of continuous imagined speech and improve the accuracy of imagined speech reconstruction at the long sequence level. It solves the problem that the imagined speech reconstruction model generated by the current method for generating the imagined speech reconstruction model cannot reconstruct continuous speech more accurately.

[0082] As described above, at least one embodiment of each part of the method for generating an imaginary speech reconstruction model provided by the present invention has been described respectively. It should be noted that the various parts of the method for generating an imaginary speech reconstruction model can be combined using different embodiments to form different embodiments of the method for generating an imaginary speech reconstruction model.

[0083] Reference Figure 8 In a specific overall embodiment, the method for generating an imagined speech reconstruction model includes:

[0084] S1: With a sampling rate of 8192 Hz, the EEG signals generated by the subjects when they perceive or imagine continuous speech are obtained.

[0085] This embodiment obtains the original EEG signal containing the imagined speech at the same sampling rate. The format of the data is (B, C, T), where B represents the batch size, C represents the number of leads, and T represents the time sample point size.

[0086] S2: Multi-channel Wiener filters are used to remove artifacts. Then, the EEG signal is re-referenced to a common mean, and finally, the EEG signal is down-sampled to 64 Hz.

[0087] The various EEG signals collected in step 1 may contain noise information and contain signal components with volatility and non-stationarity. These interference information may affect the learning quality of the network model. Therefore, the original EEG data in this implementation are all preprocessed.

[0088] The design of the multi-channel Wiener filter can be achieved by solving an optimization problem, that is, finding a filter that minimizes the mean square error between the output signal and the desired signal. Let x(n) be the multi-channel signal vector, d(n) be the desired signal vector, and v(n) be the noise vector. The goal of the Wiener filter is to find a filter H(z) that makes the output signal y(n) close to the desired signal d(n):

[0089]

[0090] in represents the convolution operation and h(n) is the impulse response of the filter.

[0091] The mean square error (MSE) is defined as:

[0092]

[0093] The Wiener filter solves the filter coefficients by minimizing the MSE:

[0094]

[0095] Next, re-referencing the EEG signal to a common mean is a common EEG signal processing step. For all recorded EEG channels, the mean of the signal at each time point is calculated:

[0096]

[0097] In the above formula, CAR(n) is the common average reference value at time point n, x i (n) is the signal value of the ith channel at time point n, and N is the total number of channels. Then, for each channel signal, the calculated common average value is subtracted point by point. The formula is:

[0098]

[0099] In the above formula, is the signal value of the i-th channel at time point n after processing.

[0100] S3: Build an imagined speech reconstruction network based on the state-space model, such as Figure 2 As shown in Figure 1, its input is the EEG signal of imagined speech, and its output is the mel spectrogram of reconstructed stimulus audio. The imagined speech reconstruction framework includes a subject feature embedding module, a feature preprocessing module, a state space module, and an output regression module.

[0101] The subject feature embedding module includes a layer normalization layer, a multi-head attention layer, a layer normalization, and a feedforward module. Its main purpose is to enhance the adaptability and personalization of the model to the individual differences of EEG signals of different subjects. By considering each subject's unique ID as part of the model input, the model can better capture these individual differences, thereby improving the accuracy of imagined speech reconstruction.

[0102] The module receives the preprocessed EEG signal and integrates it with the layer-normalized one-hot encoded subject embedding based on the subject identity through multi-head attention, thereby enhancing the feature representation of subject-specific information. The model can identify and attach importance to the signal parts related to each subject. The features containing subject information will be layer-normalized and fed-forward network again, maintaining the stability and consistency of the data, providing the model with richer information for subsequent processing tasks.

[0103] The feature preprocessing module uses a feature preprocessing module, which adopts a U-Net module based on a structured state-space model and an external attention mechanism block based on two external, learnable, shared linear layers.

[0104] The expression of the structural state space model is:

[0105]

[0106] Among them, x t represents the input of the structural state space model at time t, h t and h t-1 They represent the hidden states at time t and time t-1 respectively, , and C Both represent the dynamic matrices of the structural state space model, y t Represents the output of the structural state-space model at time t.

[0107] By combining the structured state space model with the U-Net architecture, it is possible to capture local features and short-term dependencies in EEG signals. Figure 3As shown in the U-Net module of the structured state space model in . This is crucial for understanding subtle changes in the signal. The features output after its processing contain rich spatiotemporal information, which can provide effective input for the subsequent reconstruction of the Mel spectrogram. At the same time, external attention is an attention mechanism that aims to enhance the model's sensitivity to individual differences and improve the effect of feature extraction. By introducing the external attention mechanism, the model can better capture the features related to individual differences in the EEG signal. This is especially important for processing EEG signals from different subjects. The attention maintains linear complexity, making it computationally efficient and suitable for processing long sequence data. The design of the feature preprocessing module enables the model to process EEG signals more effectively, capture complex spatiotemporal features and individual differences, and thus improve the accuracy of Mel spectrogram reconstruction. The design of this module fully considers the characteristics of EEG signals and provides strong support for the decoding of imagined speech.

[0108] The state space module adopts a selective state space model and a multi-head self-attention mechanism. This module combines the feedforward mechanism, multi-head attention and convolution operations to enhance the model's sensitivity to local features and its own convolution capabilities. The specific structure is as follows: Figure 3 As shown in the multi-head attention module / selective state space model module in, the feedforward module and convolution module used in it are also shown in detail. The network receives pre-extracted features, integrates the original features processed separately by the ESM module, learns a wider range of data representations, and improves training stability through diversified inputs. The selective state space model is improved on the basis of the structured state space model, so that the A, B, and C matrices change according to the input. In the attention mechanism, the spatiotemporal features extracted by the feature extraction module are first linearly transformed to obtain the query vector, key vector, and value vector. Then the query vector is dot-producted with the key vector and input into the weight matrix obtained by Softmax. The weight matrix is ​​dot-producted with the value vector again and input into Softmax to obtain the weighted attention score. The expression of the attention mechanism is as follows:

[0109]

[0110] In the above formula, Attention(Q , K , V） represents the final weighted attention score; Q, K, V They represent the query vector, key vector and value vector obtained by linear transformation of the spatiotemporal features extracted from EEG signals; Indicates the preset zoom ratio.

[0111] The state-space module combines the selective state-space model and the multi-head self-attention mechanism to form an efficient processing pipeline. This combination takes advantage of the selective state-space model's advantages in processing time series data and the multi-head self-attention mechanism's ability to capture contextual information. It is responsible for processing and integrating the features extracted from the EEG signals and outputting high-level representations for reconstructing the mel-spectrogram. Finally, the output regression module uses a linear layer to predict the mel-spectrogram of the imagined speech.

[0112] S4: Use imagined speech EEG signals to train the imagined speech reconstruction network. The process is as follows:

[0113] S41: Divide the dataset into training set, validation set and test set according to the quantity ratio of 8:1:1.

[0114] S42: The Pearson correlation coefficient loss that measures the correlation between the predicted value of the stimulus audio and the actual mel-spectrogram value is used as the first loss function loss1, and the L1 loss that measures the absolute error between the predicted value and the true value is used as the second loss function loss2; the final loss Loss obtained by linearly combining loss1 and loss2 according to the weights is used as the optimization function. The expression of the optimization function Loss is as follows:

[0115]

[0116] In the above formula, Represents the weight coefficient of Pearson correlation coefficient loss and L1 loss, 0< <1;x i Represents the true label value; i and y i Represents the predicted probability value. and Represents x i and i The sample mean of . i is an index, indicating the sequence number of each observation in the sample, and n is the total number of sample data.

[0117] S43: Use the training set and validation set to train and validate the imagined speech reconstruction network.

[0118] S44: Save the model parameters of the network with the best prediction results on the validation set and test it using the test set.

[0119] S5: Use the tested imagined speech reconstruction network as the reconstruction model to reconstruct the stimulus audio of the EEG signal and output the mel-spectrogram of the reconstructed stimulus audio.

[0120] The imagined speech reconstruction method based on the state-space model provided in this embodiment uses the EEG signal that is passed through a multi-channel Wiener filter to remove artifacts, re-referenced to a common average, and finally down-sampled as input features.

[0121] Due to the strong subject correlation of EEG signals, the initial input EEG signals will pass through the subject feature embedding module, which can integrate the original signal with the subject's unique features to obtain subject-related feature representations. This feature will then be preprocessed by the U-Net module based on the structured state space model and the external attention mechanism block composed of two external, learnable, shared linear layers. The preprocessed features will have rich spatiotemporal information and be sensitive to individual differences, providing strong support for the decoding of imagined speech.

[0122] The state-space module fully combines the advantages of the selective state-space model in processing time series data and the ability of the multi-head self-attention mechanism in capturing contextual information.

[0123] This embodiment divides the collected imagined speech EEG data into a training set, a validation set, and a test set, and uses a deep learning gradient descent method to train the model, thereby obtaining an imagined speech reconstruction network with higher reconstruction accuracy than existing solutions.

[0124] In order to verify the effectiveness of the method for generating the imagined speech reconstruction model in the above embodiment, an imagined speech reconstruction model was constructed according to the method in the above embodiment in a simulation experiment, and was trained using the collected data set to test the performance of the trained imagined speech reconstruction model.

[0125] The experiment uses the SparrKULee EEG dataset, which includes EEG data from 85 Dutch-speaking participants with normal hearing. Each subject went through 8 to 10 trials, each lasting about 15 minutes, listening to a story told in fluent Flemish (Belgian Dutch). We perform multi-channel Wiener filtering on the raw data to remove artifacts, then re-reference the EEG signal to the common mean, and downsample the EEG signal to 64 Hz. The dataset is mainly divided into three parts: training (80%), validation (10%), and testing (10%), which are used to optimize model training and evaluate performance.

[0126] The present invention conducted experiments on public datasets and compared them with some advanced deep learning methods proposed previously. In order to quantitatively evaluate the restoration results of the audio mel spectrogram, the Pearson correlation coefficient was used as an evaluation indicator. In order to further compare the impact of the application of each module in the model of the present invention on the final prediction performance of the trained network model, this experiment also formulated an ablation experiment. In the ablation experiment, the U-Net network based on the structured state space model, the imagined speech reconstruction network with embedded strength regulator or external attention mechanism blocks were removed as control groups for subsequent experiments.

[0127] In the training phase of different network models, Adam was used as the optimizer with a learning rate of 0.0005. The StepLR scheduler was also used, which automatically reduced the learning rate by 0.9 every 50 epochs. The original EEG signal was preprocessed as the input of the deep neural network, and then a certain number of training rounds were set for the deep neural network to train. Finally, the best model in the training was selected to test the data in the test set, and the test results were used as the corresponding prediction results.

[0128] In the ablation experiment, the prediction accuracy of different network models in the test data set is shown in Table 1 below:

[0129] Table 1: Prediction accuracy of the network model of the present invention and the control group

[0130]

[0131] Table 2: Prediction accuracy of the present invention and the control group of the ablation experiment

[0132]

[0133] From the analysis of the data in Table 1 and Table 2, it can be seen that the Pearson correlation coefficient of the method for reconstructing mel-spectrogram from EEG signals based on the state-space model provided by the present invention for reconstructing the stimulation audio reaches 0.069. Compared with deep learning methods such as VLAAI, HappyQuokka, ConvConcatnet and ConvConcatnet based on ensemble learning in Table 1, the Pearson correlation coefficient of the method of the present invention is increased by 38.0%, 32.7%, 19.0% and 9.5%, respectively. In Table 2, it can be seen by comparison that the introduction of the U-Net network based on the structured state-space model improves the effect by 21.0%, the introduction of the external attention mechanism block improves it by 13.1%, and the introduction of the embedded strength regulator improves it by 9.5%. It can be concluded from the table that the introduction of the U-Net network based on the structured state-space model has a relatively obvious improvement in the reconstruction effect.

[0134] In an embodiment of the present invention, there is also provided an imaginary speech reconstruction method, comprising:

[0135] The EEG signal generated by the user when performing continuous speech imagination is input into the imagined speech reconstruction model, and the mel-spectrogram of the user's imagined speech is obtained through the imagined speech reconstruction model; wherein the imagined speech reconstruction model is obtained by the generation method of the imagined speech reconstruction model provided by the present invention. In actual use, since the user information cannot be seen in advance, the feature embedding module is not available. When actually reconstructing the mel-spectrogram from the user's EEG signal, the user's EEG signal is directly input into the feature preprocessing module and the state space model, and the reconstructed mel-spectrogram is output.

[0136] In the embodiments of the present invention, there is also provided a device for generating an imaginary speech reconstruction model, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0137] The generating device of the imagined speech reconstruction model comprises:

[0138] The signal acquisition module is used to acquire sample EEG signals, where the sample EEG signals are EEG signals generated by the subject when performing continuous speech imagination.

[0139] The model training module is used to supervise the training of the imagined speech reconstruction model through sample EEG signals to obtain the trained imagined speech reconstruction model.

[0140] Among them, the imagined speech reconstruction model includes a feature embedding module, a feature preprocessing module and a state space module.

[0141] The feature embedding module includes a first normalization layer, a first multi-head attention mechanism block, a second normalization layer and a first feedforward module connected in sequence. The input of the first normalization layer is the identity information of the subject, and the input of the first multi-head attention mechanism block also includes a sample EEG signal.

[0142] The feature preprocessing module includes a first U network module and an external attention mechanism block connected in sequence. The input of the first U network module is the output of the first feedforward module. The first U network module includes a first convolutional layer, a second convolutional layer, a first target module, a third convolutional layer, a second target module, a third target module and a fourth convolutional layer connected in sequence. The input of the third convolutional layer also includes the output of the second convolutional layer. The input of the second target module also includes the output of the first convolutional layer. The input of the third target module also includes the output of the first convolutional layer. The first target module, the second target module and the third target module each include a plurality of structural state space modules connected in sequence.

[0143] The state space module includes a fourth target module, a first selective state space module, a fifth target module and a second selective state space module. The input of the fourth target module includes the output of the external attention mechanism block and the output of the first feedforward module. The input of the first selective state space module includes the output of the fourth target module and the output of the first feedforward module. The input of the fifth target module includes the output of the first selective state space module and the output of the first feedforward module. The input of the second selective state space module includes the output of the fifth target module and the output of the first feedforward module. The fourth target module and the fifth target module both include a plurality of multi-head attention modules connected in sequence. The output of the second selective state space module passes through a linear layer to generate a mel-spectrogram.

[0144] For the description of the device for generating the imagined speech reconstruction model, specific reference may be made to the introduction of the method for generating the imagined speech reconstruction model in the present invention.

[0145] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0146] In an embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method for generating an imaginary speech reconstruction model or the imaginary speech reconstruction method provided by the present invention.

[0147] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0148] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

Claims

1. A method for generating an imaginary speech reconstruction model, characterized in that: include: Acquiring a sample EEG signal, wherein the sample EEG signal is an EEG signal generated by a subject when performing continuous speech imagination; Performing supervised training on the imagined speech reconstruction model through the sample EEG signal to obtain a trained imagined speech reconstruction model; Wherein, the imagined speech reconstruction model includes a feature embedding module, a feature preprocessing module and a state space module; The feature embedding module includes a first normalization layer, a first multi-head attention mechanism block, a second normalization layer and a first feedforward module connected in sequence, the input of the first normalization layer is the identity information of the subject, and the input of the first multi-head attention mechanism block also includes the sample EEG signal; The feature preprocessing module includes a first U network module and an external attention mechanism block connected in sequence, the input of the first U network module is the output of the first feedforward module, the first U network module includes a first convolutional layer, a second convolutional layer, a first target module, a third convolutional layer, a second target module, a third target module and a fourth convolutional layer connected in sequence, the input of the third convolutional layer also includes the output of the second convolutional layer, the input of the second target module also includes the output of the first convolutional layer, the input of the third target module also includes the output of the first convolutional layer, and the first target module, the second target module and the third target module each include a plurality of structural state space modules connected in sequence; The state space module includes a fourth target module, a first selective state space module, a fifth target module and a second selective state space module. The input of the fourth target module includes the output of the external attention mechanism block and the output of the first feedforward module. The input of the first selective state space module includes the output of the fourth target module and the output of the first feedforward module. The input of the fifth target module includes the output of the first selective state space module and the output of the first feedforward module. The input of the second selective state space module includes the output of the fifth target module and the output of the first feedforward module. The fourth target module and the fifth target module both include a plurality of multi-head attention modules connected in sequence. The output of the second selective state space module generates a mel-spectrogram after passing through a linear layer.

2. The method for generating an imaginary speech reconstruction model according to claim 1, characterized in that: The structural state space module includes a first linear layer, a structural state space model, a first Dropout layer and a third normalization layer connected in sequence, and the input of the first Dropout layer also includes the output of the first linear layer.

3. The method for generating an imaginary speech reconstruction model according to claim 2, characterized in that: The multi-head attention module includes a first pre-convolutional layer, a second U network module, a second feedforward module, a second multi-head attention mechanism block, a first convolution module and a third feedforward module connected in sequence. The second U network module is the same as the first U network module. The input of the second multi-head attention mechanism block also includes the output of the second U network module. The input of the first convolution module also includes the input of the second multi-head attention mechanism block. The input of the third feedforward module also includes the input of the first convolution module. The output of the third feedforward module is connected with the input residual as the output of the multi-head attention module.

4. The method for generating an imaginary speech reconstruction model according to claim 3, characterized in that: The expressions of the first multi-head attention mechanism block and the second multi-head attention mechanism block are both: ， in, Q , K and V denote the query vector, key vector and value vector respectively, Indicates the preset zoom ratio. Attention ( Q , K , V ) represents the weighted attention score.

5. The method for generating an imaginary speech reconstruction model according to claim 1, characterized in that: The first selective state space module and the second selective state space module both include a second pre-convolution layer, a third U network module, a fourth feedforward module, a selective state space model, a second convolution module and a fifth feedforward module connected in sequence. The third U network module is the same as the first U network module. The input of the selective state space model also includes the output of the third U network module. The input of the second convolution module also includes the input of the selective state space model. The input of the fifth feedforward module also includes the input of the second convolution module. The output of the fifth feedforward module is connected with the input residual and serves as the output of the selective state space module.

6. The method for generating an imaginary speech reconstruction model according to claim 1, characterized in that: Obtaining sample EEG signals includes: Collecting the EEG signal generated by the subject when performing continuous speech imagination through a first frequency; Artifacts in the EEG signal are removed by a multi-channel Wiener filter, and then the EEG signal is re-referenced to a common average value, and finally the EEG signal is down-sampled to a second frequency to obtain the sample EEG signal.

7. The method for generating an imaginary speech reconstruction model according to claim 5, characterized in that: The loss function used in the training of the imagined speech reconstruction model is: ， Among them, Loss represents the total loss, Loss1 represents the first loss, and Loss2 represents the second loss. represents the weight coefficient, x i and i All represent the model prediction values. represents the true label value, and Represents x i and i The sample mean of , i is an index indicating the sample number, and n is the total number of samples.

8. A method for reconstructing imagined speech, characterized in that: include: Inputting the EEG signal generated by the user when performing continuous speech imagination into the imagined speech reconstruction model, and obtaining the mel-spectrogram of the imagined speech of the user through the imagined speech reconstruction model; Wherein, the imagined speech reconstruction model is obtained by the method for generating the imagined speech reconstruction model according to any one of claims 1-7.

9. A device for generating an imaginary speech reconstruction model, characterized in that: include: A signal acquisition module, used to acquire sample EEG signals, wherein the sample EEG signals are EEG signals generated by the subject when performing continuous speech imagination; A model training module, used for performing supervised training on the imagined speech reconstruction model through the sample EEG signal to obtain a trained imagined speech reconstruction model; Wherein, the imagined speech reconstruction model includes a feature embedding module, a feature preprocessing module and a state space module; The feature embedding module includes a first normalization layer, a first multi-head attention mechanism block, a second normalization layer and a first feedforward module connected in sequence, the input of the first normalization layer is the identity information of the subject, and the input of the first multi-head attention mechanism block also includes the sample EEG signal; The feature preprocessing module includes a first U network module and an external attention mechanism block connected in sequence, the input of the first U network module is the output of the first feedforward module, the first U network module includes a first convolutional layer, a second convolutional layer, a first target module, a third convolutional layer, a second target module, a third target module and a fourth convolutional layer connected in sequence, the input of the third convolutional layer also includes the output of the second convolutional layer, the input of the second target module also includes the output of the first convolutional layer, the input of the third target module also includes the output of the first convolutional layer, and the first target module, the second target module and the third target module each include a plurality of structural state space modules connected in sequence; The state space module includes a fourth target module, a first selective state space module, a fifth target module and a second selective state space module. The input of the fourth target module includes the output of the external attention mechanism block and the output of the first feedforward module. The input of the first selective state space module includes the output of the fourth target module and the output of the first feedforward module. The input of the fifth target module includes the output of the first selective state space module and the output of the first feedforward module. The input of the second selective state space module includes the output of the fifth target module and the output of the first feedforward module. The fourth target module and the fifth target module both include a plurality of multi-head attention modules connected in sequence. The output of the second selective state space module generates a mel-spectrogram after passing through a linear layer.

10. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the method for generating an imaginary speech reconstruction model according to any one of claims 1 to 7 or the imaginary speech reconstruction method according to claim 8.

Images