A method and device for decoding EEG text based on semantic alignment and sentence retrieval

Abstract

The application discloses a kind of electroencephalogram text decoding method and device based on semantic alignment and sentence retrieval, it is related to natural language understanding and cognitive neuroscientific technical field.Method includes: the electroencephalogram signal training data is input initial electroencephalogram signal encoding model, text true value is input BERT model, and then calculate contrast learning loss function;Predictive electroencephalogram signal encoding feature is input initial classifier, and calculate classification loss function;Contrast learning loss function and classification loss function are combined, and initial electroencephalogram signal encoding model and initial classifier are trained;To be generated electroencephalogram signal is input to trained electroencephalogram signal encoding model, and coding feature is input classifier, and word prediction probability is input beam search module, obtain keyword set, and according to AhoCorasick algorithm retrieve the text sentence with highest similarity in corpus with keyword set.Can realize efficient, accurate electroencephalogram text generation using the application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language understanding and cognitive neuroscience, and particularly discloses an electroencephalogram text decoding method and device based on semantic alignment and sentence retrieval. BACKGROUND

[0002] The relationship between human language and brain activity has long been an important topic in neuroscience and cognitive science research. The brain, as the core of cognitive, emotional and behavioral functions, is responsible for language production and understanding. With the rapid development of neuroscience and computing technology, researchers have made breakthroughs in the exploration of brain decoding methods, promoting the development of brain-computer interface technology, making it possible to communicate and control external devices through brain signals.

[0003] In this field, decoding semantic information in brain activity, especially converting it into words or sentences, has received widespread attention. This technology not only extracts language information from brain signals, but also opens up new prospects for the development of brain-computer communication systems. In particular, for patients with severe language disorders, BCI technology can convert brain signals into complete text or speech, providing them with an alternative way of communication, which has great clinical significance and social value.

[0004] With the rapid development of natural language processing (NLP) technology, researchers have fully utilized the powerful capabilities of pre-trained large language models in language understanding and generation, making it possible to accurately decode text from non-invasive electroencephalogram signals. For example, Wang et al. from the University of Illinois first proposed a technical solution that uses pre-trained large language models to decode electroencephalogram features and generate open text. This method opened up a new way of decoding text from electroencephalogram, and subsequently many researchers have further explored by combining the powerful capabilities of language models to improve the accuracy of electroencephalogram signal decoding and the quality of text generation.

[0005] Existing technologies generally use multi-layer convolutional neural networks or deep learning structures such as Transformers to encode electroencephalogram signals, extract electroencephalogram signal features through preprocessing and feature extraction modules, and then input these features into pre-trained large language models to generate text. This method has made significant progress in the task of generating text from electroencephalogram, and has gradually been applied in the field of brain-computer interfaces.

[0006] However, although these methods have achieved encouraging results in experiments, some technical bottlenecks still exist. In particular, existing EEG decoding methods generally rely on teacher-forcing generation strategies. Specifically, during the generation of each token, the model relies on the previously predicted real text as input to guide the generation of subsequent text. Although this strategy can improve the stability of the model during training, it fails to effectively reflect the performance of the model in actual applications. Because in actual applications, the model usually needs to make predictions based on the previously generated token and cannot rely on the real label, the difference between this training strategy and the actual reasoning process leads to generally poor decoding effects of the model in the absence of teacher forcing, and low decoding quality and robustness. Summary of the Invention

[0007] In order to solve the technical problems of poor decoding effect, low decoding quality and robustness of existing models, the present invention provides a method and device for EEG text decoding based on semantic alignment and sentence retrieval. The technical solution is as follows:

[0008] On the one hand, a method for decoding EEG text based on semantic alignment and sentence retrieval is provided. The method is implemented by an EEG text decoding device based on semantic alignment and sentence retrieval, and the method includes:

[0009] S1. Obtain EEG signal training data and corresponding text truth data;

[0010] S2. Randomly mask the EEG signal training data, and input the masked EEG signal training data into the initial EEG signal encoding model to obtain the predicted EEG signal encoding features;

[0011] S3. Input the text truth data into the frozen parameter BERT model to obtain the truth text encoding features;

[0012] S4. Calculating a contrastive learning loss function based on the predicted EEG signal encoding features and the true text encoding features;

[0013] S5. Inputting the predicted EEG signal coding features into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal coding features, and calculating a classification loss function based on the predicted EEG signal coding features, the corresponding predicted classification text, and the text truth data;

[0014] S6. Combining the contrastive learning loss function with the classification loss function, the initial EEG signal encoding model and the initial classifier are trained to obtain a trained EEG signal encoding model;

[0015] S7, acquire a to-be-generated electroencephalogram signal, input the to-be-generated electroencephalogram signal to the trained electroencephalogram signal coding model to obtain to-be-generated electroencephalogram signal coding features, input the to-be-generated electroencephalogram signal coding features into a classifier to obtain a word prediction probability corresponding to the to-be-generated electroencephalogram signal;

[0016] S8, input the word prediction probability into a beam search module to obtain a keyword set, retrieve a text sentence with the highest similarity to the keyword set in a corpus according to the Aho Corasick algorithm, and determine the text sentence as a text corresponding to the to-be-generated electroencephalogram signal.

[0017] In another aspect, a device for electroencephalogram text decoding based on semantic alignment and sentence retrieval is provided. The device is applied to an electroencephalogram text decoding method based on semantic alignment and sentence retrieval. The device comprises:

[0018] An acquisition unit is configured to acquire electroencephalogram signal training data and corresponding text true value data.

[0019] A first training unit is configured to perform random masking on the electroencephalogram signal training data, input the masked electroencephalogram signal training data into an initial electroencephalogram signal coding model, and obtain predicted electroencephalogram signal coding features.

[0020] A second training unit is configured to input the text true value data into a BERT model with frozen parameters to obtain true value text coding features.

[0021] A contrast learning unit is configured to calculate a contrast learning loss function according to the predicted electroencephalogram signal coding features and the true value text coding features.

[0022] A classification unit is configured to input the predicted electroencephalogram signal coding features into an initial classifier to obtain predicted classification texts corresponding to the predicted electroencephalogram signal coding features, and calculate a classification loss function according to the predicted electroencephalogram signal coding features, the corresponding predicted classification texts, and the text true value data.

[0023] A third training unit is configured to train the initial electroencephalogram signal coding model and the initial classifier in combination with the contrast learning loss function and the classification loss function to obtain a trained electroencephalogram signal coding model.

[0024] An encoding unit is configured to acquire a to-be-generated electroencephalogram signal, input the to-be-generated electroencephalogram signal to the trained electroencephalogram signal coding model to obtain to-be-generated electroencephalogram signal coding features, input the to-be-generated electroencephalogram signal coding features into a classifier to obtain a word prediction probability corresponding to the to-be-generated electroencephalogram signal.

[0025] The retrieval unit is configured to input the word prediction probability into a beam search module to obtain a keyword set, retrieve a text sentence with the highest similarity to the keyword set from a corpus according to an Aho Corasick algorithm, and determine the text sentence as the text corresponding to the brain electrical signal to be generated.

[0026] In another aspect, a brain electrical text decoding device based on semantic alignment and sentence retrieval is provided, which comprises a processor and a memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement any one of the above brain electrical text decoding methods based on semantic alignment and sentence retrieval.

[0027] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, wherein the at least one instruction is loaded and executed by a processor to implement any one of the above brain electrical text decoding methods based on semantic alignment and sentence retrieval.

[0028] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0029] By combining the deep feature extraction of brain electrical signals, the contrast learning mechanism and the efficient retrieval strategy, the method can effectively decode the text that meets the semantic requirements from the brain electrical signals. Specifically, first, the brain electrical data and the text data are preprocessed, and a closed word list is constructed by removing stop words and restoring word forms, etc., to provide high-quality input for the subsequent decoding process. Then, the Conformer model is used to encode the brain electrical signals, and the Subject layer is introduced to adapt to the individual differences of different subjects to enhance the robustness of the brain electrical encoder. Through the mask contrast learning mechanism, the model can better capture the potential semantic information in the brain electrical signals, and the contrast learning loss function optimizes the representation ability of the encoder.

[0030] In the candidate word generation stage, the closed word classification module generates a candidate word set through a beam retrieval strategy, and then uses the Aho Corasick algorithm to match the retrieved candidate words to determine the text that best matches the original brain electrical signals. Finally, through the combination of beam search and string matching, the sentence with the highest semantic similarity to the brain electrical signals is retrieved from the corpus, realizing efficient and accurate brain electrical text generation. The innovation of the present embodiment lies in the combination of semantic alignment and sentence retrieval strategies, which overcomes the high computational overhead and generation accuracy problems that may be encountered by traditional generation models when processing brain electrical signals, thereby improving the effectiveness and usability of brain electrical signal decoding. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0032] Figure 1 is a flow chart of an electroencephalogram text decoding method based on semantic alignment and sentence retrieval provided by an embodiment of the present application;

[0033] Figure 2 is a framework diagram of an electroencephalogram encoder model based on mask contrast learning provided by an embodiment of the present application;

[0034] Figure 3 is a process schematic diagram of retrieval input and beam retrieval provided by an embodiment of the present application;

[0035] Figure 4 is a block diagram of an electroencephalogram text decoding device based on semantic alignment and sentence retrieval provided by an embodiment of the present application;

[0036] Figure 5 is a structural schematic diagram of an electroencephalogram text decoding device based on semantic alignment and sentence retrieval provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the present application will be described below with reference to the drawings.

[0038] In the embodiments of the present application, the words such as "example", "for example" and the like are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0039] In the embodiments of the present application, "image" and "picture" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent. "Of", "corresponding" and "corresponding" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent.

[0040] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1. When the distinction is not emphasized, the meanings expressed are consistent.

[0041] To make the technical problems, technical solutions and advantages to be solved by the present application clearer, specific embodiments will be described in detail below with reference to the drawings.

[0042] The embodiment of the present application provides a brain electric text decoding method based on semantic alignment and sentence retrieval. The method can be implemented by a brain electric text decoding device based on semantic alignment and sentence retrieval. The device can be a terminal or a server. Figure 1 As shown in the flowchart of the brain electric text decoding method based on semantic alignment and sentence retrieval, the processing procedure of the method can include the following steps:

[0043] S1, acquiring brain electric signal training data and corresponding text true value data.

[0044] In a feasible implementation, the brain electric signal training data and the text true value data are one-to-one corresponding according to sequence positions. That is, the brain electric signal training data is a brain electric signal data sequence, the text true value data is a text data sequence, and the semantic text corresponding to the nth brain electric signal in the brain electric signal data sequence is the nth text word in the text data sequence.

[0045] Optionally, the specific operation steps of S1 can include the following S11-S15:

[0046] S11, acquiring initial brain electric data and initial text data in a data set.

[0047] In a feasible implementation, the data set can be a ZuCo data set. The text and brain electric data contained in the ZuCo data set need to be preprocessed respectively, so that the brain electric data segments and the words in the text are one-to-one corresponding.

[0048] S12, performing noise removal operation and filtering processing on the initial brain electric data. The noise removal operation includes eye movement and muscle artifact removal, and the filtering processing includes high-pass filtering and notch filtering.

[0049] S13, dividing the brain electric signals after the noise removal operation and the filtering processing according to the corresponding relationship with the words to obtain intermediate brain electric signals corresponding to the words.

[0050] In a feasible implementation, the original brain electrical signal is subjected to noise removal and filtering processing. The steps include high-pass filtering, notch filtering, and artifact removal, etc. The eye movement and muscle artifacts are removed by an EEG (Electroencephalogram) cleaning tool, which can be a MARA algorithm (Maximum Alternative Routing Algorithm). After the signal processing, for each brain electrical segment, the eye tracking data is used for alignment, and the features are segmented and labeled according to the reading time of the word and the brain electrical response, and the brain electrical data is divided into brain electrical signal segments corresponding to the words.

[0051] S14, filter the stop words in the initial text data, and screen the filtered words according to the frequency and semantic intensity to obtain screened words.

[0052] S15, compare the screened words with the words corresponding to the intermediate brain electrical signal, select the words and the intermediate brain electrical signal with consistent semantics, determine the brain electrical signal training data and the corresponding text true value data, and the brain electrical signal training data and the corresponding text true value data completely correspond in position.

[0053] The text true value data can also be referred to as a closed word set.

[0054] In a feasible implementation, the preprocessing of the text data first needs to construct a closed word set. There are a large number of stop words (such as "the", "a", "an", and "is") in English that have no actual semantics. These words frequently appear in training and test samples, which may cause the brain electrical encoder or classifier to pay excessive attention to irrelevant information. Therefore, the embodiment of the present application first filters out these stop words to focus on words with semantics. The embodiment of the present application counts the brain electrical-text pairs, screens the frequently appearing and semantically strong words, and ensures that the training samples have sufficient representativeness and cover key semantics. When constructing the closed word set, the embodiment of the present application also restores the parts of speech of the words (such as restoring verbs to their original forms and restoring nouns to their singular forms), reduces the redundancy of synonymous or variant words, ensures that each word is a unique semantic unit, and improves the accuracy and representativeness of the statistical results.

[0055] S2, randomly mask the brain electrical signal training data, input the masked brain electrical signal training data into the initial brain electrical signal encoding model, and obtain the predicted brain electrical signal encoding features.

[0056] In a feasible implementation, before the brain electrical signal is input into the encoder, in order to improve the sample efficiency and enhance the robustness to signal changes, the embodiment of the present application also adopts a masking mechanism in the frequency dimension of the brain electrical signal. The brain electrical signal is masked with a probability of 10%, for example,Figure 2 Specifically, a certain row of data in the frequency domain dimension of the electroencephalogram signal data input into the electroencephalogram encoder is removed, and the remaining data of other frequency domains is retained, and the electroencephalogram signal is processed in this random removal manner.

[0057] Optionally, the electroencephalogram signal encoding model can include a spatial encoding module, a Conformer module, a Subject layer, a maximum pooling layer, and a first full connection layer.

[0058] Optionally, as shown in FIG. 2, the specific operation steps of S2 can include the following S21-S25: Figure 2

[0059] S21, input the masked electroencephalogram signal training data into the spatial encoding module to obtain spatial filtering features, and fuse the frequency band positioning position embedding in the spatial filtering features to obtain the labeled electroencephalogram signal.

[0060] In a feasible implementation, the masked electroencephalogram signal training data is used as input, the spatial encoding module constructed by multiple convolution layers is used to perform spatial filtering in the spatial dimension, and the frequency band positioning position embedding is fused to complete the labeling of the electroencephalogram frequency. The process can be represented as Spatial(EEG)+pos_embed(EEG), where EEG is the input electroencephalogram signal training data.

[0061] S22, input the labeled electroencephalogram signal into the Conformer module to obtain electroencephalogram signal initial encoding features.

[0062] In a feasible implementation, in order to capture global information across frequency bands, the Conformer model is further used to encode the electroencephalogram frequency domain features in the embodiment of the application. The Conformer is a neural network model for capturing global dependencies between EEG frequency bands and spatial connectivity between channels. The main parameters of the embodiment are as follows: the feedforward size is 1024, the number of layers is 3, and the number of self-attention heads is 8. The process can be represented as Conformer(x1), where x1 is the output of step S21.

[0063] S23, input the electroencephalogram signal initial encoding features into the Subject layer to process individual differences and generate corresponding electroencephalogram signal modeling data.

[0064] ​In a feasible implementation, after the encoding in step S22, the embodiment of the present application designs a Subject layer to enhance the modeling of individual characteristics by introducing individual information, thereby improving the robustness of the model. Specifically, the embodiment of the present application creates a dedicated mapping function for each subject, which is trained according to the electroencephalogram characteristics of the subject to extract the potential representation related to the individual. The mapping ensures that the model can better adapt to the individual differences of different subjects while capturing semantic information, and the process can be represented as Subject(x2), where x2 is the output of step S22. The size of the Subject layer used by the embodiment of the present application is 64.

[0065] S24, input the electroencephalogram signal modeling data into the max-pooling layer to obtain the pooling feature data of the electroencephalogram signal.

[0066] In a feasible implementation, after the Subject layer in S23, a global max-pooling layer is used to pool the output of the Subject layer to obtain the final electroencephalogram representation corresponding to each word. The process can be represented as Maxpool(x3), where x3 is the output of S23.

[0067] S25, input the pooling feature data into the first fully connected layer to obtain the predicted electroencephalogram signal encoding feature.

[0068] S3, input the text true value data into the BERT model with frozen parameters to obtain the true value text encoding feature.

[0069] In a feasible implementation, for the text feature, the embodiment of the present application uses the BERT model to encode the entire sentence and extracts the feature of each word corresponding to the electroencephalogram signal from it. BERT is a commonly used pre-training language model in the field of natural language processing.

[0070] It should be noted that before inputting the text data into BERT, the words need to be segmented and converted into text labels that can be recognized by BERT. If a word is split into multiple segmented words in the segmentation process, the average feature of these segmented words is used to represent the feature of the word.

[0071] S4, calculate the contrastive learning loss function according to the predicted electroencephalogram signal encoding feature and the true value text encoding feature.

[0072] In a feasible implementation, in order to capture semantic information from the potential features of the electroencephalogram signal, the embodiment of the present application uses a contrastive learning mechanism to promote the alignment of the electroencephalogram signal and the corresponding language representation in the feature space, thereby enhancing the representation ability of the encoder.

[0073] Optionally, the specific operation mode of S4 can be as follows:

[0074] According to the predicted electroencephalogram signal coding feature, the true value text coding feature, and the following formula (1), a contrast learning loss function is calculated:

[0075]

[0076] wherein, represents the contrast learning loss function, represents the i-th predicted electroencephalogram signal coding feature, represents the i-th true value text coding feature, represents the j-th true value text coding feature, the i-th predicted electroencephalogram signal coding feature corresponds to the i-th true value text coding feature, and the i-th predicted electroencephalogram signal coding feature does not correspond to the j-th true value text coding feature; l represents a preset temperature coefficient, and M represents the number of predicted electroencephalogram signal coding features.

[0077] In an available implementation, the similarity between the electroencephalogram feature and the text feature is calculated, and the formula aims to maximize the similarity of matched feature pairs (i.e., the electroencephalogram signal and the text feature of the same instance) and minimize the similarity of non-matched feature pairs (i.e., the similarity between the electroencephalogram and the text feature of different instances). The temperature coefficient τ controls the sensitivity of the similarity calculation, and the embodiment of the present application can set it to 0.007. The exponential function is used to enhance the weight of similar feature pairs, and finally through the logarithmic and normalization operations, it is ensured that the model can learn the effective cross-modal feature alignment, so as to realize the effective docking of the electroencephalogram signal and the text feature.

[0078] The contrast learning loss function used by the embodiment of the present application in the training process quantizes the similarity between positive and negative samples as a constant, and optimizes the electroencephalogram encoder by minimizing the loss function shown in the following formula (1)

[0079] S5, input the predicted electroencephalogram signal coding feature into the initial classifier to obtain the predicted classification text corresponding to the predicted electroencephalogram signal coding feature, and calculate a classification loss function according to the predicted electroencephalogram signal coding feature, the corresponding predicted classification text, and the text true value data.

[0080] In an available implementation, a candidate word set needs to be constructed, that is, the features of the electroencephalogram encoder need to be predicted as words, and the word prediction probability is obtained. In the embodiment of the present application, a closed set word classification module is designed. In this module, the classification ability of the electroencephalogram encoder is further enhanced by combining supervised learning and contrast learning.

[0081] Optionally, the classifier comprises a second full connection layer and a Softmax layer.​

[0082] The specific operation mode of S5 can include S51-S53:

[0083] S51, input the predicted electroencephalogram signal coding feature into an initial second full connection layer to obtain an electroencephalogram connection feature;

[0084] S52, input the electroencephalogram connection feature into a Softmax layer to obtain a word probability corresponding to the predicted electroencephalogram signal coding feature, and determine a predicted classification text corresponding to the predicted electroencephalogram signal coding feature according to the word probability corresponding to the predicted electroencephalogram signal coding feature;

[0085] S53, calculate a classification loss function according to the predicted electroencephalogram signal coding feature, the predicted classification text corresponding thereto, text true value data, and the following formula (2):

[0086]

[0087] wherein, represents the classification loss function, is one-hot encoding of the i-th predicted electroencephalogram signal coding feature, i is the i-th predicted electroencephalogram signal coding feature, i represents one-hot encoding of the real classification text, represents a probability of predicting the i-th electroencephalogram signal coding feature as the i-th classification text.

[0088] In a feasible implementation, a second full connection layer with a Softmax activation function is used to directly classify the electroencephalogram features output by the electroencephalogram encoder in S2, so as to optimize the relationship between the electroencephalogram features generated by the encoder and the actual words.

[0089] S6, combine the contrast learning loss function and the classification loss function to train the initial electroencephalogram signal coding model and the initial classifier to obtain a trained electroencephalogram signal coding model.

[0090] In a feasible implementation, the contrast loss and the classification loss are simultaneously used for model training of the electroencephalogram encoder, and optionally, the specific operation steps of S6 can include S61-S62:

[0091] S61, calculate a total loss function according to the contrast learning loss function, the classification loss function, and the following formula (3):

[0092]

[0093] wherein, Total loss function is represented, and alpha represents the weight coefficient of the contrast learning loss function, beta represents the weight coefficient of the classification loss function, alpha+beta=1;

[0094] In a feasible implementation, alpha=0.5 and beta=0.5 can be set in the embodiment.

[0095] S62, according to the total loss function, the initial electroencephalogram coding model and the initial classifier are iteratively trained until the total loss function converges, the training is stopped, and the trained electroencephalogram coding model and the classifier are obtained.

[0096] S7, obtaining the to-be-generated electroencephalogram, inputting the to-be-generated electroencephalogram into the trained electroencephalogram coding model to obtain the to-be-generated electroencephalogram coding feature, inputting the to-be-generated electroencephalogram coding feature into the classifier to obtain the word prediction probability corresponding to the to-be-generated electroencephalogram.

[0097] The trained electroencephalogram coding model includes a spatial coding module, a Conformer module, a Subject layer, a maximum pooling layer, and a first full connection layer.

[0098] In a feasible implementation, after inputting the to-be-generated electroencephalogram into the trained electroencephalogram coding model, the process of generating the to-be-generated electroencephalogram coding feature can include the following S71-S75:

[0099] S71, inputting the to-be-generated electroencephalogram into the spatial coding module to obtain spatial filtering features, and fusing a frequency band positioning position embedding in the spatial filtering features to obtain a marked electroencephalogram;

[0100] S72, inputting the marked electroencephalogram into the Conformer module, and using the Conformer model to encode the electroencephalogram frequency domain feature to obtain an initial electroencephalogram coding feature;

[0101] S73, inputting the initial electroencephalogram coding feature into the Subject layer to process individual differences and generate corresponding electroencephalogram modeling data;

[0102] S74, inputting the electroencephalogram modeling data into the maximum pooling layer to obtain the pooling feature data of the electroencephalogram;

[0103] S75, inputting the pooling feature data into the first full connection layer to obtain the predicted electroencephalogram coding feature.

[0104] The process of inputting the to-be-generated electroencephalogram coding feature into the classifier to generate the word prediction probability corresponding to the to-be-generated electroencephalogram can include the following S76-S77:

[0105] S76, input the to-be-generated electroencephalogram signal coding feature into a second full connection layer to obtain an electroencephalogram connection feature;

[0106] S77, input the electroencephalogram connection feature into a Softmax layer to obtain a word prediction probability corresponding to the to-be-generated electroencephalogram signal.

[0107] S8, input the word prediction probability into a beam search module to obtain a keyword set, and search for a text sentence with the highest similarity to the keyword set in a corpus according to an Aho Corasick algorithm, so as to determine the text sentence as a text corresponding to the to-be-generated electroencephalogram signal.

[0108] In a feasible implementation, in the embodiment of the application, the text generation process is not directly completed by a generation model, but is implemented by a search strategy. Specifically, for each electroencephalogram segment, first, the word prediction probability obtained by the closed-set word classification module (i.e., the classifier described above) is used as input, and a beam search method is used to obtain a word candidate set. The beam search is a heuristic search algorithm, which gradually generates a text by retaining a certain number of optimal candidate solutions (referred to as beam width) at each step. By setting a suitable beam width, the beam search can explore more possibilities while ensuring computational efficiency, and finally select words with higher probabilities as the candidate set. Figure 3 For example, set the beam width to 10, and the word prediction set obtained is as shown in the input of the searcher in Figure 3 {may, during, serve, augeust, november}, {become, island, first, go, time}, {star, much, work, come, make}……, and the like.

[0109] In the beam search process, the word prediction probability corresponding to each electroencephalogram segment is used as the initial input for the search, and the algorithm searches for the most likely word sequence in the vocabulary according to these probabilities. Figure 3 A feasible case of beam search is shown in the following figure, and the words

may

during

[0110] In the first step of beam search, the algorithm selects a number of words with high probabilities from the vocabulary as initial candidate words based on the predicted probability distribution. For example, the words

may

during

may

may-become

may-island

may-first

during

during-become

during-island

during-time

may

may-become

may-star

during

during-star

during-time

[0111] This process takes into account the context information and semantic association of words, ensuring that the retrieved candidate words match the semantics of the original EEG signal as much as possible. Then, through further screening and sorting of the candidate set, the words that best meet the context and semantic requirements are selected to generate the text. The formula is defined as follows:

[0112]

[0113] where, represents the set of all sentences in the data set; q' represents the candidate query set; k represents the maximum length of the candidate set.

[0114] After finding the keyword set through beam search, the embodiment of the application further retrieves the most suitable sentence from the corpus by string matching the keyword combination with the original sentence. Specifically, the embodiment of the application adopts the Aho Corasick algorithm as a training-free scoring method for query retrieval sentences to calculate the similarity between the query keywords and the corpus sentences. The Aho Corasick algorithm is a high-efficiency multi-mode string matching algorithm used to find the occurrence position of multiple patterns (keywords) in a text. It realizes fast matching by constructing a finite state machine (FSM). The algorithm first constructs a Trie tree (i.e. a dictionary tree) to store all the patterns to be matched, and then constructs a failure pointer to realize the jump from one pattern failure to the next possible matching pattern, thereby accelerating the matching process. In the embodiment of the application, the Aho Corasick algorithm is used to calculate the matching degree between the query keywords and the corpus sentences. It processes the keyword set obtained through beam search to quickly calculate the number of occurrences of the keywords in the sentence, thereby providing a basis for the similarity score of the candidate sentence. The formula for calculating the matching degree is as follows:

[0115]

[0116] Wherein, |q ∩ c| represents the intersection size between the query keyword set q and the corpus sentence c, and the max function is used to select the optimal matching. Finally, the sentence with the highest similarity obtained through matching is taken as the text corresponding to the generated electroencephalogram signal.

[0117] The experimental process and results are described as follows:

[0118] Performance verification was performed on a public electroencephalogram text dataset containing 309 cases of data, and by combining contrastive learning, closed-set word classification, mask mechanism and efficient retrieval strategy, significant effects were achieved in the electroencephalogram signal decoding text task. Through multi-level feature extraction of electroencephalogram signals and contrastive learning mechanism, the method can better capture the semantic information in the electroencephalogram signals, and further improve the robustness and accuracy of the decoding model. By using the Conformer model combined with the Subject layer, individual differences can be processed, and the electroencephalogram features of different subjects can be adapted, further enhancing the generalization ability of the model. In the word prediction results, a Top20 prediction accuracy of 60.21%, a Top15 prediction accuracy of 54.66%, a Top10 prediction accuracy of 37.45%, a Top5 prediction accuracy of 25.27%, and a Top1 word prediction accuracy of 5.37% were achieved. The TopN accuracy represents the probability distribution of the correct words in the TopN words predicted by the electroencephalogram signal segment. In the generated text results, for sentences with greater than or equal to 3 valid electroencephalogram text pairs, the Recall value was 27.02%, the Precision value was 28.51%, and the BLEU-4 value was 17.30%. For sentences with greater than or equal to 3 valid electroencephalogram text pairs, the Recall value was 30.21%, the Precision value was 33.18%, and the BLEU-4 value was 21.38%. The above results prove that the method of the embodiment of the application can effectively improve the accuracy and generation quality of electroencephalogram decoding text. This technology not only provides a new approach for decoding and understanding electroencephalogram signals, but also has broad practical application potential. In particular, in the fields of brain-computer interface, emotion recognition, auxiliary communication devices and human-computer interaction, it can achieve more natural and accurate conversion of electroencephalogram signals to text. This electroencephalogram decoding technology is expected to help intelligent analysis of brain waves, bringing new breakthroughs in brain disease diagnosis, emotion monitoring and even the construction of barrier-free communication systems. With the continuous maturation and optimization of technology, this method will have broader application prospects in intelligent medical treatment, psychological research, smart home and other industries.

[0119] In the embodiment of the present application, the electroencephalogram signal training data and the corresponding text true value data are obtained; the electroencephalogram signal training data is randomly masked, the masked electroencephalogram signal training data is input into an initial electroencephalogram signal coding model to obtain predicted electroencephalogram signal coding features; the text true value data is input into a BERT model with frozen parameters to obtain true value text coding features; a contrast learning loss function is calculated according to the predicted electroencephalogram signal coding features and the true value text coding features; the predicted electroencephalogram signal coding features are input into an initial classifier to obtain predicted classification texts corresponding to the predicted electroencephalogram signal coding features, and a classification loss function is calculated according to the predicted electroencephalogram signal coding features, the corresponding predicted classification texts and the text true value data; the initial electroencephalogram signal coding model and the initial classifier are trained in combination with the contrast learning loss function and the classification loss function to obtain a trained electroencephalogram signal coding model; the to-be-generated electroencephalogram signal is obtained, the to-be-generated electroencephalogram signal is input into the trained electroencephalogram signal coding model to obtain to-be-generated electroencephalogram signal coding features, the to-be-generated electroencephalogram signal coding features are input into the classifier to obtain word prediction probabilities corresponding to the to-be-generated electroencephalogram signal; the word prediction probabilities are input into a beam search module to obtain a keyword set, and a text sentence with the highest similarity to the keyword set is retrieved in a corpus according to the AhoCorasick algorithm, and the text sentence is determined as a text corresponding to the to-be-generated electroencephalogram signal.

[0120] In the embodiment of the present application, the deep feature extraction of the electroencephalogram signal, the contrast learning mechanism and the efficient retrieval strategy are combined, and the method can effectively decode the text meeting the semantic requirements from the electroencephalogram signal. Specifically, the electroencephalogram data and the text data are preprocessed first, and a closed set of words is constructed by removing stop words and restoring word forms, providing high-quality input for the subsequent decoding process. Then, the Conformer model is used to encode the electroencephalogram signal, and the Subject layer is introduced to adapt to the individual differences of different subjects, to enhance the robustness of the electroencephalogram encoder. Through the mask contrast learning mechanism, the model can better capture the potential semantic information in the electroencephalogram signal, and the contrast learning loss function optimizes the representation ability of the encoder.

[0121] In the candidate word generation stage, the closed set word classification module generates a candidate word set through the beam search strategy, and then uses the AhoCorasick algorithm to match the retrieved candidate words to determine the text most consistent with the original electroencephalogram signal. Finally, through the combination of beam search and string matching, the sentence with the highest semantic matching to the electroencephalogram signal is retrieved from the corpus, realizing efficient and accurate electroencephalogram text generation. The innovation of the present embodiment lies in the combination of semantic alignment and sentence retrieval strategy, which overcomes the high computational overhead and generation accuracy problems that may be encountered by traditional generation models in processing electroencephalogram signals, thereby improving the effectiveness and usability of electroencephalogram signal decoding.

[0122] Figure 4 is a device block diagram 400 of electroencephalogram text decoding based on semantic alignment and sentence retrieval according to an exemplary embodiment, which is used for an electroencephalogram text decoding method based on semantic alignment and sentence retrieval. Referring to Figure 4 , the device comprises an acquisition unit 410, a first training unit 420, a second training unit 430, a contrast learning unit 440, a classification unit 450, a third training unit 460, an encoding unit 470, and a retrieval unit 480. Among them:

[0123] The acquisition unit 410 is configured to acquire electroencephalogram signal training data and corresponding text true value data.

[0124] The first training unit 420 is configured to randomly mask the electroencephalogram signal training data, input the masked electroencephalogram signal training data into an initial electroencephalogram signal encoding model, and obtain predicted electroencephalogram signal encoding features.

[0125] The second training unit 430 is configured to input the text true value data into a BERT model with frozen parameters to obtain true value text encoding features.

[0126] The contrast learning unit 440 is configured to calculate a contrast learning loss function according to the predicted electroencephalogram signal encoding features and the true value text encoding features.

[0127] The classification unit 450 is configured to input the predicted electroencephalogram signal encoding features into an initial classifier to obtain predicted classification text corresponding to the predicted electroencephalogram signal encoding features, and calculate a classification loss function according to the predicted electroencephalogram signal encoding features, the corresponding predicted classification text, and the text true value data.

[0128] The third training unit 460 is configured to train the initial electroencephalogram signal encoding model and the initial classifier in combination with the contrast learning loss function and the classification loss function to obtain a trained electroencephalogram signal encoding model.

[0129] The encoding unit 470 is configured to acquire a to-be-generated electroencephalogram signal, input the to-be-generated electroencephalogram signal into the trained electroencephalogram signal encoding model to obtain to-be-generated electroencephalogram signal encoding features, and input the to-be-generated electroencephalogram signal encoding features into the classifier to obtain word prediction probabilities corresponding to the to-be-generated electroencephalogram signal.

[0130] The retrieval unit 480 is configured to input the word prediction probabilities into a beam search module to obtain a keyword set, retrieve a text sentence with the highest similarity to the keyword set in a corpus according to an AhoCorasick algorithm, and determine the text sentence as a text corresponding to the to-be-generated electroencephalogram signal.

[0131] Optionally, the acquisition unit 410 is configured to:

[0132] S11, acquire initial electroencephalogram data and initial text data in a data set;

[0133] S12, perform noise removal operation and filtering processing on the initial electroencephalogram data, wherein the noise removal operation includes eye movement and muscle artifact removal, and the filtering processing includes high-pass filtering and notch filtering;

[0134] S13, divide the electroencephalogram signal after the noise removal operation and the filtering processing according to the relationship corresponding to the words to obtain intermediate electroencephalogram signals corresponding to the words;

[0135] S14, filter stop words in the initial text data, and screen the filtered words according to frequency and semantic intensity to obtain screened words;

[0136] S15, compare the screened words with the words corresponding to the intermediate electroencephalogram signals, select words and intermediate electroencephalogram signals with consistent semantics, determine electroencephalogram signal training data and corresponding text true value data, and the electroencephalogram signal training data and the corresponding text true value data completely correspond in position.

[0137] Optionally, the electroencephalogram signal encoding model comprises a spatial encoding module, a Conformer module, a Subject layer, a maximum pooling layer, and a first full connection layer.

[0138] Optionally, the first training unit 420 is configured to:

[0139] S21, input the masked electroencephalogram signal training data into the spatial encoding module to obtain spatial filtering features, fuse band positioning position embedding in the spatial filtering features, and obtain labeled electroencephalogram signals;

[0140] S22, input the labeled electroencephalogram signals into the Conformer module to obtain initial electroencephalogram signal encoding features;

[0141] S23, input the initial electroencephalogram signal encoding features into the Subject layer to process individual differences and generate corresponding electroencephalogram signal modeling data;

[0142] S24, input the electroencephalogram signal modeling data into the maximum pooling layer to obtain pooling feature data of the electroencephalogram signal;

[0143] S25, input the pooling feature data into the first full connection layer to obtain predicted electroencephalogram signal encoding features.

[0144] Optionally, the contrast learning unit 440 is configured to:

[0145] According to the predicted electroencephalogram signal coding feature, the true value text coding feature, and the following formula (1), a contrastive learning loss function is calculated:

[0146]

[0147] wherein, denotes the contrastive learning loss function, denotes the i th predicted electroencephalogram signal coding feature, denotes the i th true value text coding feature, denotes the j th true value text coding feature, the i th predicted electroencephalogram signal coding feature corresponds to the i th true value text coding feature, and the i th predicted electroencephalogram signal coding feature does not correspond to the j th true value text coding feature; τ denotes a preset temperature coefficient, and M denotes the number of predicted electroencephalogram signal coding features.

[0148] Optionally, the classifier comprises a second fully connected layer and a Softmax layer.

[0149] The classification unit 450 is configured to:

[0150] S51, input the predicted electroencephalogram signal coding feature into an initial second fully connected layer to obtain an electroencephalogram connection feature;

[0151] S52, input the electroencephalogram connection feature into a Softmax layer to obtain a word probability corresponding to the predicted electroencephalogram signal coding feature, and determine a predicted classification text corresponding to the predicted electroencephalogram signal coding feature according to the word probability corresponding to the predicted electroencephalogram signal coding feature.

[0152] S53, according to the predicted electroencephalogram signal coding feature, the corresponding predicted classification text, text true value data, and the following formula (2), a classification loss function is calculated:

[0153]

[0154] wherein, denotes the classification loss function, is one-hot coding of the i th predicted classification text corresponding to the i th predicted electroencephalogram signal coding feature, e i is the i th predicted electroencephalogram signal coding feature, w i denotes one-hot coding of the real classification text, denotes a probability of predicting the i th electroencephalogram signal coding feature as the i th classification text.

[0155] Optionally, the third training unit 460 is configured to:

[0156] S61, according to the contrastive learning loss function, the classification loss function, and the following formula (3), a total loss function is calculated:

[0157]

[0158] wherein, represents a total loss function, a represents a weight coefficient of a contrast learning loss function, and β represents a weight coefficient of a classification loss function, and a+β=1;

[0159] S62, according to the total loss function, the initial electroencephalogram signal coding model and the initial classifier are iteratively trained until the total loss function converges, the training is stopped, and the trained electroencephalogram signal coding model and the classifier are obtained.

[0160] The embodiment of the present application can effectively decode the text meeting the semantic requirements from the electroencephalogram signal by combining the deep feature extraction of the electroencephalogram signal, the contrast learning mechanism and the efficient retrieval strategy. Specifically, first, the electroencephalogram data and the text data are preprocessed, and a closed word list is constructed by removing stop words and restoring word forms, to provide high-quality input for the subsequent decoding process. Then, the Conformer model is used to encode the electroencephalogram signal, and the Subject layer is introduced to adapt to the individual differences of different subjects, to enhance the robustness of the electroencephalogram encoder. Through the mask contrast learning mechanism, the model can better capture the potential semantic information in the electroencephalogram signal, and the contrast learning loss function optimizes the representation ability of the encoder.

[0161] In the candidate word generation stage, the closed word classification module generates a candidate word set through the beam retrieval strategy, and then uses the AhoCorasick algorithm to match the retrieved candidate words, to determine the text most consistent with the original electroencephalogram signal. Finally, through the combination of beam retrieval and string matching, the sentence most matched with the semantic of the electroencephalogram signal is retrieved from the corpus, to realize efficient and accurate electroencephalogram text generation. The innovation of the embodiment lies in combining the semantic alignment and sentence retrieval strategy, which overcomes the high computational overhead and generation accuracy problems that the traditional generation model may encounter when processing the electroencephalogram signal, thereby improving the effect and usability of the electroencephalogram signal decoding.

[0162] Figure 5 is a structural schematic diagram of an electroencephalogram text decoding device based on semantic alignment and sentence retrieval provided by the embodiment of the present application, as Figure 5 shown, the electroencephalogram text decoding device based on semantic alignment and sentence retrieval can include the electroencephalogram text decoding apparatus based on semantic alignment and sentence retrieval shown in the above Figure 4 . Optionally, the electroencephalogram text decoding device based on semantic alignment and sentence retrieval 510 can include a first processor 2001.

[0163] Optionally, the electroencephalogram text decoding device 510 based on semantic alignment and sentence retrieval can further include a memory 2002 and a transceiver 2003.

[0164] The first processor 2001 is connected with the memory 2002 and the transceiver 2003, for example, through a communication bus.

[0165] The following will be specifically introduced Figure 5 The various constituent components of the electroencephalogram text decoding device 510 based on semantic alignment and sentence retrieval will be specifically introduced as follows:

[0166] The first processor 2001 is the control center of the electroencephalogram text decoding device 510 based on semantic alignment and sentence retrieval, and can be one processor or a collective term of multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0167] Optionally, the first processor 2001 can execute various functions of the electroencephalogram text decoding device 510 based on semantic alignment and sentence retrieval by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0168] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, such as the CPU0 and the CPU1 shown in FIG. Figure 5

[0169] In a specific implementation, as an embodiment, the electroencephalogram text decoding device 510 based on semantic alignment and sentence retrieval can also include multiple processors, such as the first processor 2001 and the second processor 2004 shown in FIG. Figure 5

[0170] ​​The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0171] Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 can be integrated with the first processor 2001 or exist independently and access the data through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0172] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0173] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0174] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or can exist independently and can be connected to the EEG text decoding device 510 based on semantic alignment and sentence retrieval through an interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0175] It should be noted that ​ The structure of the EEG text decoding device 510 based on semantic alignment and sentence retrieval shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0176] In addition, the technical effects of the electroencephalogram text decoding device 510 based on semantic alignment and sentence retrieval can refer to the technical effects of the electroencephalogram text decoding method based on semantic alignment and sentence retrieval described in the above method embodiments, which will not be repeated here.

[0177] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0178] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRAM).

[0179] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0180] It should be understood that the term "and / or" herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects, but can also represent an "and / or" relationship, which can be understood in the context before and after.

[0181] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0182] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0183] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0185] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0186] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0187] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0188] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0189] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for decoding EEG text based on semantic alignment and sentence retrieval, characterized in that: The method comprises: S1. Obtain EEG signal training data and corresponding text truth data; S2. Randomly mask the EEG signal training data, and input the masked EEG signal training data into the initial EEG signal encoding model to obtain the predicted EEG signal encoding features; Wherein, the S2 includes: S21, inputting the masked EEG signal training data into the spatial encoding module to obtain spatial filtering features, fusing the frequency band positioning position embedding into the spatial filtering features to obtain the labeled EEG signal; S22, input the labeled EEG signal into the Conformer module to obtain the initial coding features of the EEG signal; S23, input the initial coding features of the EEG signal into the Subject layer to process individual differences and generate corresponding EEG signal modeling data; S24, inputting the EEG signal modeling data into the maximum pooling layer to obtain pooled feature data of the EEG signal; S25, inputting the pooled feature data into the first fully connected layer to obtain the predicted EEG signal encoding features; S3. Input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features; S4. Calculating a contrastive learning loss function based on the predicted EEG signal encoding features and the true text encoding features; S5. Inputting the predicted EEG signal coding features into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal coding features, and calculating a classification loss function based on the predicted EEG signal coding features, the corresponding predicted classification text, and the text truth data; S6. Combining the contrastive learning loss function with the classification loss function, the initial EEG signal encoding model and the initial classifier are trained to obtain a trained EEG signal encoding model; S7. Obtain an EEG signal to be generated, input the EEG signal to be generated into a trained EEG signal coding model to obtain coding features of the EEG signal to be generated, input the coding features of the EEG signal to be generated into a classifier, and obtain a word prediction probability corresponding to the EEG signal to be generated; S8. Input the word prediction probability into a beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the EEG signal to be generated.

2. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 1 is characterized in that: The step S1 of acquiring EEG signal training data and corresponding text truth data includes: S11, obtaining initial EEG data and initial text data in the dataset; S12, performing noise removal and filtering on the initial EEG data, wherein the noise removal includes removing eye movement and muscle artifacts, and the filtering includes high-pass filtering and notch filtering; S13, segmenting the EEG signal after noise removal and filtering according to the relationship corresponding to the words to obtain intermediate EEG signals corresponding to the words; S14, filtering the stop words in the initial text data, and screening the filtered words according to frequency and semantic strength to obtain filtered words; S15. Compare the filtered words with the words corresponding to the intermediate EEG signals, select semantically consistent words and intermediate EEG signals, determine the EEG signal training data and the corresponding text true value data, and the EEG signal training data and the corresponding text true value data are completely corresponding in position.

3. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 1 is characterized in that: The step S4 calculates the contrastive learning loss function based on the predicted EEG signal encoding features and the true text encoding features, including: According to the predicted EEG signal encoding features, the true text encoding features and the following formula (1), the contrastive learning loss function is calculated: （1） in, represents the contrastive learning loss function, represents the i-th predicted EEG signal encoding feature, represents the i-th true value text encoding feature, represents the jth true value text encoding feature, the i-th predicted EEG signal encoding feature corresponds to the i-th true value text encoding feature, and the i-th predicted EEG signal encoding feature does not correspond to the j-th true value text encoding feature; represents the preset temperature coefficient, and M represents the number of predicted EEG signal encoding features.

4. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 1 is characterized in that: The classifier includes a second fully connected layer and a Softmax layer; The step S5 inputs the predicted EEG signal encoding feature into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal encoding feature, and calculates a classification loss function based on the predicted EEG signal encoding feature, the corresponding predicted classification text, and the text truth data, including: S51, inputting the predicted EEG signal encoding features into the initial second fully connected layer to obtain EEG connection features; S52, inputting the EEG connection feature into the Softmax layer to obtain the word probability corresponding to the predicted EEG signal coding feature, and determining the predicted classification text corresponding to the predicted EEG signal coding feature based on the word probability corresponding to the predicted EEG signal coding feature; S53. Calculate the classification loss function based on the predicted EEG signal encoding features, the corresponding predicted classification text, the text truth data, and the following formula (2): （2） in, represents the classification loss function, is the one-hot encoding of the predicted classification text corresponding to the i-th predicted EEG signal encoding feature, is the i-th predicted EEG signal encoding feature, Represents the one-hot encoding of the real classification text, Represents the probability of predicting the i-th EEG signal encoding feature as the i-th classification text.

5. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 1 is characterized in that: The S6 combines the contrastive learning loss function and the classification loss function to train the initial EEG signal encoding model and the initial classifier to obtain a trained EEG signal encoding model, including: S61. Calculate the total loss function based on the contrastive learning loss function, the classification loss function, and the following formula (3): （3） in, represents the total loss function, represents the weight coefficient of the contrastive learning loss function, represents the weight coefficient of the classification loss function, ; S62. According to the total loss function, the initial EEG signal encoding model and the initial classifier are iteratively trained until the total loss function converges, and the training is stopped to obtain the trained EEG signal encoding model and classifier.

6. An EEG text decoding device based on semantic alignment and sentence retrieval, wherein the EEG text decoding device based on semantic alignment and sentence retrieval is used to implement the EEG text decoding method based on semantic alignment and sentence retrieval as described in any one of claims 1 to 5, characterized in that: The device comprises: An acquisition unit, used to acquire EEG signal training data and corresponding text truth data; The first training unit is used to randomly mask the EEG signal training data and input the masked EEG signal training data into the initial EEG signal coding model to obtain the predicted EEG signal coding features; The second training unit is used to input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features; A contrastive learning unit, configured to calculate a contrastive learning loss function based on the predicted EEG signal encoding features and the true text encoding features; A classification unit, configured to input the predicted EEG signal encoding features into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal encoding features, and calculate a classification loss function based on the predicted EEG signal encoding features, the corresponding predicted classification text, and the text truth data; The third training unit is used to train the initial EEG signal encoding model and the initial classifier by combining the contrastive learning loss function and the classification loss function to obtain a trained EEG signal encoding model; An encoding unit is used to obtain an EEG signal to be generated, input the EEG signal to be generated into a trained EEG signal encoding model to obtain encoding features of the EEG signal to be generated, input the encoding features of the EEG signal to be generated into a classifier to obtain a word prediction probability corresponding to the EEG signal to be generated; A retrieval unit is used to input the word prediction probability into a beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the EEG signal to be generated.

7. An EEG text decoding device based on semantic alignment and sentence retrieval, characterized in that: The EEG text decoding device based on semantic alignment and sentence retrieval includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images