Electroencephalogram text decoding method and device based on semantic alignment and sentence retrieval

Abstract

The invention discloses an electroencephalogram text decoding method and device based on semantic alignment and sentence retrieval, and relates to the technical field of natural language understanding and cognitive neuroscience. The method comprises the following steps: inputting electroencephalogram signal training data into an initial electroencephalogram signal coding model, inputting a text truth value into a BERT model, and further calculating a contrast learning loss function; the predicted electroencephalogram signal coding features are input into an initial classifier, and a classification loss function is calculated; in combination with a contrast learning loss function and a classification loss function, training the initial electroencephalogram signal coding model and the initial classifier; and inputting a to-be-generated electroencephalogram signal into the trained electroencephalogram signal coding model, inputting coding features into the classifier, inputting the word prediction probability into a beam search module to obtain a keyword set, and retrieving a text statement with the highest similarity with the keyword set in a corpus according to an AhoCorasick algorithm. By adopting the method and the device, efficient and accurate electroencephalogram text generation can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language understanding and cognitive neuroscience, and in particular to an electroencephalogram text decoding method and device based on semantic alignment and sentence retrieval. Background Art

[0002] Human language and its relationship with brain activity have long been important research topics in neuroscience and cognitive science. As the core of cognitive, emotional and behavioral functions, the brain is responsible for the generation and understanding of language. With the rapid development of neuroscience and computing technology, researchers have made breakthrough progress in the exploration of brain decoding methods, which has promoted the development of brain-computer interface technology and made it possible to communicate and control external devices through brain signals.

[0003] In this field, the study of decoding semantic information in brain activity, especially converting it into words or sentences, has received widespread attention. This technology can not only extract language information from brain signals, but also open up new prospects for the development of brain-computer communication systems. In particular, for patients with severe language disorders, BCI technology can convert brain signals into complete text or speech, providing them with an alternative way of communication. This breakthrough has great clinical significance and social value.

[0004] With the rapid development of natural language processing (NLP) technology, researchers have made full use of the powerful capabilities of pre-trained large language models in language understanding and generation, thus realizing the possibility of accurately decoding text from non-invasive EEG signals. For example, Wang et al. from the University of Illinois first proposed a technical solution to use pre-trained large language models to decode EEG features and generate open text. This method has opened up a new idea for EEG decoding text. Subsequently, many researchers have further explored this approach, combining the powerful capabilities of language models to improve the accuracy and generation quality of EEG signal decoding.

[0005] Existing technologies generally encode EEG signals through multi-layer convolutional neural networks or deep learning structures such as Transformer, extract EEG signal features through preprocessing and feature extraction modules, and then input these features into a pre-trained large language model to generate text. This type of method has made significant progress in the task of generating EEG decoded text and has gradually been applied in the field of brain-computer interfaces.

[0006] However, despite the encouraging experimental results of these methods, some technical bottlenecks still exist. In particular, existing EEG decoding methods generally rely on a teacher-forcing generation strategy. Specifically, during the generation of each token, the model relies on the previously predicted real text as input to guide the generation of subsequent text. Although this strategy can improve the stability of the model during training, it fails to effectively reflect the performance of the model in practical applications. Because in practical applications, the model usually needs to make predictions based on the previous generated token and cannot rely on the real label, the difference between this training strategy and the actual reasoning process leads to generally poor decoding effects, low decoding quality and robustness in the absence of teacher forcing. Summary of the invention

[0007] In order to solve the technical problems that the decoding effect of the model in the prior art is generally poor, and the decoding quality and robustness are low, the embodiment of the present invention provides an EEG text decoding method and device based on semantic alignment and sentence retrieval. The technical solution is as follows:

[0008] On the one hand, a method for decoding electroencephalogram text based on semantic alignment and sentence retrieval is provided, the method is implemented by an electroencephalogram text decoding device based on semantic alignment and sentence retrieval, and the method comprises:

[0009] S1, obtain EEG signal training data and corresponding text truth data;

[0010] S2, randomly masking the EEG signal training data, and inputting the masked EEG signal training data into the initial EEG signal encoding model to obtain the predicted EEG signal encoding features;

[0011] S3, input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features;

[0012] S4, calculating a contrastive learning loss function according to the predicted EEG signal encoding features and the true value text encoding features;

[0013] S5, inputting the predicted EEG signal coding feature into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal coding feature, and calculating a classification loss function according to the predicted EEG signal coding feature, the corresponding predicted classification text and the text truth data;

[0014] S6. Combining the contrastive learning loss function with the classification loss function, the initial EEG signal encoding model and the initial classifier are trained to obtain a trained EEG signal encoding model;

[0015] S7, obtaining the EEG signal to be generated, inputting the EEG signal to be generated into a trained EEG signal coding model to obtain coding features of the EEG signal to be generated, inputting the coding features of the EEG signal to be generated into a classifier, and obtaining a word prediction probability corresponding to the EEG signal to be generated;

[0016] S8. Input the word prediction probability into a beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the EEG signal to be generated.

[0017] On the other hand, a device for decoding EEG text based on semantic alignment and sentence retrieval is provided, and the device is applied to the method for decoding EEG text based on semantic alignment and sentence retrieval, and the device comprises:

[0018] An acquisition unit, used to acquire EEG signal training data and corresponding text truth data;

[0019] The first training unit is used to randomly mask the EEG signal training data, and input the masked EEG signal training data into the initial EEG signal coding model to obtain the predicted EEG signal coding features;

[0020] The second training unit is used to input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features;

[0021] A contrastive learning unit, used for calculating a contrastive learning loss function according to the predicted EEG signal encoding features and the true value text encoding features;

[0022] A classification unit, used for inputting the predicted EEG signal coding features into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal coding features, and calculating a classification loss function according to the predicted EEG signal coding features, the corresponding predicted classification text and the text truth data;

[0023] The third training unit is used to train the initial EEG signal encoding model and the initial classifier by combining the contrastive learning loss function and the classification loss function to obtain a trained EEG signal encoding model;

[0024] The encoding unit is used to obtain the EEG signal to be generated, input the EEG signal to be generated into a trained EEG signal encoding model to obtain the encoding features of the EEG signal to be generated, input the encoding features of the EEG signal to be generated into a classifier, and obtain the word prediction probability corresponding to the EEG signal to be generated;

[0025] The retrieval unit is used to input the word prediction probability into a beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the electroencephalogram signal to be generated.

[0026] On the other hand, a device for decoding EEG text based on semantic alignment and sentence retrieval is provided, and the device for decoding EEG text based on semantic alignment and sentence retrieval comprises: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for decoding EEG text based on semantic alignment and sentence retrieval is implemented.

[0027] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned EEG text decoding methods based on semantic alignment and sentence retrieval.

[0028] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0029] By combining deep feature extraction of EEG signals, contrastive learning mechanism and efficient retrieval strategy, this method can effectively decode semantically qualified text from EEG signals. Specifically, the EEG data and text data are first preprocessed, and a closed word list is constructed by removing stop words and restoring parts of speech, so as to provide high-quality input for the subsequent decoding process. Then, the Conformer model is used to encode the EEG signals, and the Subject layer is introduced to adapt to the individual differences of different subjects to enhance the robustness of the EEG encoder. Through the mask contrastive learning mechanism, the model can better capture the potential semantic information in the EEG signals, and the contrastive learning loss function optimizes the representation ability of the encoder.

[0030] In the candidate word generation stage, the closed-set word classification module generates a candidate word set through a beam retrieval strategy, and then uses the AhoCorasick algorithm to match the retrieved candidate words to determine the text that best matches the original EEG signal. Finally, by combining beam retrieval and string matching, the sentence that best matches the semantics of the EEG signal is retrieved from the corpus to achieve efficient and accurate EEG text generation. The innovation of this embodiment lies in that by combining the strategy of semantic alignment and sentence retrieval, it overcomes the high computational overhead and generation accuracy problems that traditional generation models may encounter when processing EEG signals, thereby improving the effect and usability of EEG signal decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a flow chart of an EEG text decoding method based on semantic alignment and sentence retrieval provided by an embodiment of the present invention;

[0033] Figure 2 This is a framework diagram of an EEG encoder model based on mask contrast learning provided by an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of a search input and beam search process provided by an embodiment of the present invention;

[0035] Figure 4 It is a block diagram of an EEG text decoding device based on semantic alignment and sentence retrieval provided by an embodiment of the present invention;

[0036] Figure 5 It is a structural schematic diagram of an EEG text decoding device based on semantic alignment and sentence retrieval provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0038] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0039] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0040] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0041] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0042] The embodiment of the present invention provides an EEG text decoding method based on semantic alignment and sentence retrieval, which can be implemented by an EEG text decoding device based on semantic alignment and sentence retrieval, and the EEG text decoding device based on semantic alignment and sentence retrieval can be a terminal or a server. Figure 1 The flowchart of the EEG text decoding method based on semantic alignment and sentence retrieval is shown. The processing flow of the method may include the following steps:

[0043] S1. Obtain EEG signal training data and corresponding text truth data.

[0044] In a feasible implementation, the EEG signal training data and the text truth data correspond one-to-one according to the sequence position, that is, the EEG signal training data is an EEG signal data sequence, the text truth data is a text data sequence, and the semantic text corresponding to the nth EEG signal in the EEG signal data sequence is the nth text word in the text data sequence.

[0045] Optionally, the specific operation steps of S1 may include the following S11-S15:

[0046] S11. Obtain initial EEG data and initial text data in the data set.

[0047] In a feasible implementation, the data set may be the ZuCo data set. The text and EEG data contained in the ZuCo data set need to be preprocessed separately so that the EEG data segments correspond to the words in the text one by one.

[0048] S12, performing noise removal and filtering processing on the initial EEG data, the noise removal operation includes removing eye movement and muscle artifacts, and the filtering processing includes high-pass filtering and notch filtering.

[0049] S13, segmenting the EEG signal after the noise removal and filtering process according to the relationship corresponding to the words to obtain the intermediate EEG signal corresponding to the words.

[0050] In a feasible implementation, the original EEG signal is subjected to noise removal and filtering. This includes steps such as high-pass filtering, notch filtering, and artifact removal. Eye movement and muscle artifacts are removed by an EEG (Electroencephalogram) cleaning tool, which may be a MARA algorithm (Maximum Alternative Routing Algorithm). After signal processing, eye tracking data is used to align each EEG segment, and features are segmented and annotated based on the reading time and EEG response of the word, and the EEG data is divided into EEG signal segments corresponding to the word.

[0051] S14, filtering the stop words in the initial text data, and filtering the filtered words according to frequency and semantic strength to obtain filtered words.

[0052] S15. Compare the filtered words with the words corresponding to the intermediate EEG signals, select the semantically consistent words and intermediate EEG signals, determine the EEG signal training data and the corresponding text truth data, and the EEG signal training data and the corresponding text truth data are completely corresponding in position.

[0053] Among them, the text truth data can also be called a closed word list.

[0054] In a feasible implementation manner, preprocessing text data first requires building a closed word set. There are a large number of stop words without actual semantics in English (such as "the", "a", "an" and "is", etc.). These words frequently appear in training and test samples, which may cause the EEG encoder or classifier to over-focus on irrelevant information. Therefore, the embodiment of the present invention first filters out these stop words to focus on semantic words. This embodiment performs statistics on EEG-text pairs, screens out frequently occurring words with strong semantics, and ensures that the training samples are sufficiently representative and cover key semantics. When constructing a closed word list, this embodiment also performs part-of-speech restoration on words (such as restoring verbs to their original form and nouns to their singular form), reduces the redundancy of synonymous or deformed words, ensures that each word is a unique semantic unit, and improves the accuracy and representativeness of the statistical results.

[0055] S2. Randomly mask the EEG signal training data, and input the masked EEG signal training data into the initial EEG signal encoding model to obtain predicted EEG signal encoding features.

[0056] In a feasible implementation, before the EEG signal is input to the encoder, in order to improve sample efficiency and enhance robustness to signal changes, this embodiment also uses a masking mechanism in the frequency dimension of the EEG signal, and masks the EEG signal with a probability of 10%, such as Figure 2 Specifically, a row of data in the frequency domain dimension of the EEG signal data input into the EEG encoder is removed, and the remaining data in other frequency domains is retained, and the EEG signal is processed in this random removal manner.

[0057] Optionally, the EEG signal encoding model may include a spatial encoding module, a Conformer module, a Subject layer, a maximum pooling layer and a first fully connected layer.

[0058] Alternatively, if Figure 2 As shown, the specific operation steps of S2 may include the following S21-S25:

[0059] S21. Input the masked EEG signal training data into the spatial coding module to obtain spatial filtering features, fuse the frequency band positioning position embedding into the spatial filtering features, and obtain the marked EEG signal.

[0060] In a feasible implementation, the masked EEG signal training data is used as input, and a spatial encoding module constructed using multiple convolutional layers is used to perform spatial filtering in the spatial dimension, and the frequency band positioning position embedding is integrated to complete the labeling of the EEG frequency. The process can be expressed as Spatial(EEG)+pos_embed(EEG), where EEG is the input EEG signal training data.

[0061] S22. Input the labeled EEG signal into the Conformer module to obtain the initial coding features of the EEG signal.

[0062] In a feasible implementation, in order to capture global information across frequency bands, the embodiment of the present invention further uses a Conformer model to encode EEG frequency domain features. Conformer is a neural network model used to capture global dependencies between EEG frequency bands and spatial connectivity between channels. The main parameters of this embodiment are: feedforward size is 1024, number of layers is 3, and number of self-attention heads is 8. The process can be expressed as Conformer(x1), where x1 is the output of step S21.

[0063] S23. Input the initial coding features of the EEG signal into the Subject layer to process individual differences and generate corresponding EEG signal modeling data.

[0064] In a feasible implementation, after encoding in step S22, an embodiment of the present invention designs a Subject layer, which enhances the modeling of individual characteristics by introducing individual information, thereby improving the robustness of the model. Specifically, the embodiment of the present invention creates an exclusive mapping function for each subject, which is trained according to the EEG characteristics of the subject to extract potential representations related to the individual. This mapping ensures that the model can better adapt to individual differences between different subjects while capturing semantic information. The process can be expressed as Subject(x2), where x2 is the output of step S22. The size of the Subject layer used in the embodiment of the present invention is 64.

[0065] S24. Input the EEG signal modeling data into the maximum pooling layer to obtain pooled feature data of the EEG signal.

[0066] In a feasible implementation, after passing through the Subject layer in S23, the output of the Subject layer is pooled using a global maximum pooling layer to obtain the final EEG representation corresponding to each word. The process can be expressed as Maxpool(x3), where x3 is the output of S23.

[0067] S25. Input the pooled feature data into the first fully connected layer to obtain the predicted EEG signal encoding features.

[0068] S3. Input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features.

[0069] In a feasible implementation, for text features, an embodiment of the present invention uses a BERT model to encode the entire sentence and extract features of the words corresponding to each EEG signal. BERT is a commonly used pre-trained language model in the field of natural language processing.

[0070] It should be noted that before inputting text data into BERT, the words need to be segmented and converted into text labels that BERT can recognize. If a word is split into multiple words during the word segmentation process, the average features of these words are used to represent the features of the word.

[0071] S4. Calculate the contrastive learning loss function based on the predicted EEG signal encoding features and the true text encoding features.

[0072] In a feasible implementation, in order to capture semantic information from the latent features of EEG signals, the embodiment of the present invention uses a contrastive learning mechanism to promote the alignment of EEG signals and corresponding language representations in the feature space, thereby enhancing the representation ability of the encoder.

[0073] Optionally, the specific operation mode of S4 may be as follows:

[0074] According to the predicted EEG signal encoding features, the true value text encoding features and the following formula (1), the contrastive learning loss function is calculated:

[0075]

[0076] in, represents the contrastive learning loss function, represents the i-th predicted EEG signal encoding feature, represents the i-th true value text encoding feature, represents the jth true value text encoding feature, the ith predicted EEG signal encoding feature corresponds to the ith true value text encoding feature, and the ith predicted EEG signal encoding feature does not correspond to the jth true value text encoding feature; l represents the preset temperature coefficient, and M represents the number of predicted EEG signal encoding features.

[0077] In one feasible implementation, by calculating the EEG features and text features The formula aims to maximize the similarity of matching feature pairs (i.e., EEG signals and text features of the same instance) and minimize the similarity of non-matching feature pairs (i.e., the similarity between EEG and text features of different instances). Among them, the temperature coefficient τ controls the sensitivity of the similarity calculation, and the embodiment of the present invention can be set to 0.007. The exponential function is used to enhance the weight of similar feature pairs, and finally through logarithmic and normalization operations, it is ensured that the model can learn effective cross-modal feature alignment, thereby achieving effective docking of EEG signals and text features.

[0078] The contrastive learning loss function used in the training process of the embodiment of the present invention is to quantify the similarity between positive and negative samples and minimize the loss function shown in the above formula (1): To optimize the EEG encoder.

[0079] S5. Input the predicted EEG signal encoding features into the initial classifier to obtain the predicted classification text corresponding to the predicted EEG signal encoding features, and calculate the classification loss function according to the predicted EEG signal encoding features, the corresponding predicted classification text and the text true value data.

[0080] In a feasible implementation, it is necessary to first construct a candidate word set, that is, it is necessary to predict the features of the EEG encoder as words and obtain the word prediction probability. A closed set word classification module is designed in the embodiment of the present invention. In this module, the classification ability of the EEG encoder is further enhanced by combining supervised learning and contrastive learning.

[0081] Optionally, the classifier includes a second fully connected layer and a Softmax layer;

[0082] The specific operation mode of S5 may include S51-S53:

[0083] S51, inputting the predicted EEG signal encoding features into the initial second fully connected layer to obtain EEG connection features;

[0084] S52, inputting the EEG connection feature into the Softmax layer to obtain the probability of a word corresponding to the predicted EEG signal coding feature, and determining the predicted classification text corresponding to the predicted EEG signal coding feature according to the probability of a word corresponding to the predicted EEG signal coding feature;

[0085] S53, according to the predicted EEG signal encoding features, the corresponding predicted classification text, the text truth data and the following formula (2), calculate the classification loss function:

[0086]

[0087] in, represents the classification loss function, is the one-hot encoding of the predicted classification text corresponding to the i-th predicted EEG signal encoding feature, e i is the i-th predicted EEG signal encoding feature, w i Represents the one-hot encoding of the real classification text, Represents the probability of predicting the i-th EEG signal encoding feature as the i-th classified text.

[0088] In a feasible implementation, a second fully connected layer with a Softmax activation function is used to directly classify the EEG features output by the EEG encoder in S2, thereby optimizing the relationship between the EEG features generated by the encoder and the actual vocabulary.

[0089] S6. Combine the contrastive learning loss function and the classification loss function to train the initial EEG signal encoding model and the initial classifier to obtain a trained EEG signal encoding model.

[0090] In a feasible implementation, for the EEG encoder, it is necessary to use contrast loss and classification loss simultaneously for model training. Optionally, the specific operation steps of S6 may include S61-S62:

[0091] S61. Calculate the total loss function according to the contrastive learning loss function, the classification loss function and the following formula (3):

[0092]

[0093] in, represents the total loss function, α represents the weight coefficient of the contrastive learning loss function, β represents the weight coefficient of the classification loss function, α+β=1;

[0094] In a feasible implementation manner, in the embodiment of the present invention, α=0.5 and β=0.5 may be set.

[0095] S62. According to the total loss function, the initial EEG signal encoding model and the initial classifier are iteratively trained until the total loss function converges, and the training is stopped to obtain the trained EEG signal encoding model and classifier.

[0096] S7. Obtain the EEG signal to be generated, input the EEG signal to be generated into a trained EEG signal coding model, obtain coding features of the EEG signal to be generated, input the coding features of the EEG signal to be generated into a classifier, and obtain the word prediction probability corresponding to the EEG signal to be generated.

[0097] Among them, the trained EEG signal encoding model includes a spatial encoding module, a Conformer module, a Subject layer, a maximum pooling layer, and the first fully connected layer.

[0098] In a feasible implementation manner, after the EEG signal to be generated is input into a trained EEG signal encoding model, the process of generating the encoding features of the EEG signal to be generated may include the following S71-S75:

[0099] S71, inputting the EEG signal to be generated into a spatial coding module to obtain a spatial filtering feature, fusing the frequency band positioning position embedding into the spatial filtering feature, and obtaining a marked EEG signal;

[0100] S72, inputting the marked EEG signal into the Conformer module, using the Conformer model to encode the EEG frequency domain features to obtain the initial coding features of the EEG signal;

[0101] S73, inputting the initial coding features of the EEG signal into the Subject layer to process individual differences and generate corresponding EEG signal modeling data;

[0102] S74, inputting the EEG signal modeling data into the maximum pooling layer to obtain pooled feature data of the EEG signal;

[0103] S75. Input the pooled feature data into the first fully connected layer to obtain the predicted EEG signal encoding features.

[0104] The process of inputting the coding features of the to-be-generated EEG signal into the classifier and generating the word prediction probability corresponding to the to-be-generated EEG signal may include the following S76-S77:

[0105] S76, inputting the encoded features of the EEG signal to be generated into the second fully connected layer to obtain EEG connection features;

[0106] S77. Input the EEG connection features into the Softmax layer to obtain the word prediction probability corresponding to the EEG signal to be generated.

[0107] S8. Input the word prediction probability into the beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the EEG signal to be generated.

[0108] In a feasible implementation, in an embodiment of the present invention, the text generation process is not completed directly through the generation model, but is implemented through a retrieval strategy. Specifically, for each EEG segment, the word prediction probability obtained by the closed-set word classification module (i.e., the above-mentioned classifier) ​​is first used as input, and a beam search method is used to obtain a word candidate set. Beam search is a heuristic search algorithm that gradually generates text by retaining a certain number of optimal candidate solutions (called beam width) at each step. By setting an appropriate beam width, beam search can explore more possibilities while ensuring computational efficiency, and ultimately select words with higher probabilities as candidate sets. Reference Figure 3 , let’s take a feasible example: set the beam width to 10, and the word prediction set obtained is as follows Figure 3 The search engine input is as follows: {may,during,serve,augeust,november}, {become,island,first,go,time}, {star,much,work,come,make}...etc.

[0109] During the beam search process, the predicted probabilities of the words corresponding to each EEG segment are used as the initial input to the search, and the algorithm retrieves the most likely word sequence in the vocabulary based on these probabilities. Figure 3 The following is a feasible example of beam search. In the beam search process, the words [may] and [during] are used as examples to illustrate:

[0110] In the first step of beam retrieval, the algorithm selects several words with higher probabilities from the vocabulary as the initial candidate word set according to the predicted probability distribution. For example, the words [may] and [during] are selected as the starting nodes in the figure to enter the next step of the search process. In the subsequent steps, beam retrieval gradually expands and generates new word candidate sequences based on the current candidate words and the probability distribution of subsequent EEG segments. For example, starting from [may], the possible sequences to be generated include [may-become], [may-island], and [may-first], etc.; and starting from [during], [during-become], [during-island], and [during-time], etc. may be generated. The total probability of each sequence is calculated by multiplying the predicted probability of each word. After each round of expansion, the algorithm will sort all the generated candidate sequences according to the total probability, and retain the sequences with the highest probability as the starting point for the next round of search. For example, starting from [may], the possible sequences to be retained are [may-become] and [may-star]; starting from [during], [during-star] and [during-time] may be retained. This process will continue to recursively until all words corresponding to the EEG segments have been retrieved, and finally several complete candidate word sequences are generated.

[0111] This process takes into account the contextual information and the semantic association of words, ensuring that the retrieved candidate words match the semantics of the original EEG signal as closely as possible. Then, by further screening and sorting the candidate set, the words that best meet the contextual and semantic requirements are finally selected to generate text. The formula is defined as follows:

[0112]

[0113] in, represents the set of all sentences in the dataset; q′ represents the candidate query set; k represents the maximum length of the candidate set.

[0114] After finding the keyword set through beam retrieval, the embodiment of the present invention further retrieves the most suitable sentence from the corpus by string matching the keyword combination with the original sentence. Specifically, the embodiment of the present invention adopts the AhoCorasick algorithm as a training-free scoring method for query retrieval sentences to calculate the similarity between the query keyword and the corpus sentence. The AhoCorasick algorithm is an efficient multi-mode string matching algorithm for finding the occurrence positions of multiple patterns (keywords) in a text. It achieves fast matching by constructing a finite state automaton (FSM). The algorithm first constructs a Trie tree (i.e., a dictionary tree) to store all the patterns to be matched, and then jumps from a pattern failure to the next possible matching pattern by constructing a failure pointer, thereby accelerating the matching process. In this embodiment of the invention, the AhoCorasick algorithm is used to calculate the matching degree between the query keyword and the corpus sentence. It processes the keyword set obtained by beam retrieval and quickly calculates the number of occurrences of the keyword in the sentence, thereby providing a basis for the similarity scoring of the candidate sentences. The calculation formula for the matching degree is as follows:

[0115]

[0116] Among them, |q ∩ c| represents the size of the intersection between the query keyword set q and the corpus sentence c, and the max function is used to select the best match. Finally, the sentence with the highest similarity obtained by matching is used as the text corresponding to the EEG signal to be generated.

[0117] The following is a description of the experimental process and results:

[0118] The performance was verified on a public EEG text dataset containing 309 data. By combining contrastive learning, closed-set word classification, masking mechanism and efficient retrieval strategy, significant results were achieved in the EEG signal decoding text task. This method can better capture the semantic information in the EEG signal through multi-level feature extraction and contrastive learning mechanism of EEG signals, thereby improving the robustness and accuracy of the decoding model. The Conformer model combined with the Subject layer can handle individual differences, adapt to the EEG characteristics of different subjects, and further enhance the generalization ability of the model. In the word prediction results, 60.21% Top20 prediction accuracy, 54.66% Top15 prediction accuracy, 37.45% Top10 prediction accuracy, 25.27% Top5 prediction accuracy, and 5.37% Top1 word prediction accuracy were achieved, where the TopN accuracy represents the probability distribution of EEG signal fragments predicted as words. The correct word is included in the first N words. In the results of generating text, for sentences with more than or equal to 3 valid EEG text pairs, the Recall value is 27.02%, the Precision value is 28.51%, and the BLEU-4 value is 17.30%. For sentences with more than or equal to 3 valid EEG text pairs, the Recall value is 30.21%, the Precision value is 33.18%, and the BLEU-4 value is 21.38%. The above results prove that the method of the embodiment of the present invention can effectively improve the accuracy and generation quality of EEG decoding text. This technology not only provides new ideas for the decoding and understanding of EEG signals, but also has a wide range of practical application potential. In particular, in the fields of brain-computer interface, emotion recognition, auxiliary communication equipment and human-computer interaction, a more natural and accurate conversion of EEG signals to text can be achieved. This EEG decoding technology is expected to help the intelligent analysis of brain waves, and bring new breakthroughs to the diagnosis of brain diseases, emotion monitoring, and even the construction of barrier-free communication systems. With the continuous maturity and optimization of technology, this method will show a broader application prospect in multiple industries such as smart medical care, psychological research, and smart homes.

[0119] In the embodiment of the present invention, EEG signal training data and corresponding text truth data are obtained; the EEG signal training data are randomly masked, and the masked EEG signal training data are input into the initial EEG signal coding model to obtain predicted EEG signal coding features; the text truth data is input into the BERT model with frozen parameters to obtain the truth text coding features; according to the predicted EEG signal coding features and the truth text coding features, the contrastive learning loss function is calculated; the predicted EEG signal coding features are input into the initial classifier to obtain the predicted classification text corresponding to the predicted EEG signal coding features; according to the predicted EEG signal coding features, the corresponding predicted classification text and the text truth data, the classification loss function is calculated. loss function; combining the contrastive learning loss function and the classification loss function, training the initial EEG signal coding model and the initial classifier to obtain the trained EEG signal coding model; obtaining the EEG signal to be generated, inputting the EEG signal to be generated into the trained EEG signal coding model to obtain the coding features of the EEG signal to be generated, inputting the coding features of the EEG signal to be generated into the classifier to obtain the word prediction probability corresponding to the EEG signal to be generated; inputting the word prediction probability into the beam search module to obtain a keyword set, retrieving the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determining the text sentence as the text corresponding to the EEG signal to be generated.

[0120] The embodiment of the present invention combines deep feature extraction of EEG signals, contrastive learning mechanism and efficient retrieval strategy, and the method can effectively decode text that meets semantic requirements from EEG signals. Specifically, the EEG data and text data are first preprocessed, and a closed word list is constructed by removing stop words and restoring parts of speech, so as to provide high-quality input for the subsequent decoding process. Then, the Conformer model is used to encode the EEG signals, and the Subject layer is introduced to adapt to the individual differences of different subjects to enhance the robustness of the EEG encoder. Through the mask contrast learning mechanism, the model can better capture the potential semantic information in the EEG signals, and the contrast learning loss function optimizes the representation ability of the encoder.

[0121] In the candidate word generation stage, the closed-set word classification module generates a candidate word set through a beam retrieval strategy, and then uses the AhoCorasick algorithm to match the retrieved candidate words to determine the text that best matches the original EEG signal. Finally, by combining beam retrieval and string matching, the sentence that best matches the semantics of the EEG signal is retrieved from the corpus to achieve efficient and accurate EEG text generation. The innovation of this embodiment lies in that by combining the strategy of semantic alignment and sentence retrieval, it overcomes the high computational overhead and generation accuracy problems that traditional generation models may encounter when processing EEG signals, thereby improving the effect and usability of EEG signal decoding.

[0122] Figure 4 400 is a block diagram of an EEG text decoding device based on semantic alignment and sentence retrieval according to an exemplary embodiment. The device 400 is used for an EEG text decoding method based on semantic alignment and sentence retrieval. Figure 4 The device includes an acquisition unit 410, a first training unit 420, a second training unit 430, a comparative learning unit 440, a classification unit 450, a third training unit 460, an encoding unit 470 and a retrieval unit 480. Among them:

[0123] An acquisition unit 410 is used to acquire EEG signal training data and corresponding text truth data;

[0124] The first training unit 420 is used to randomly mask the EEG signal training data, and input the masked EEG signal training data into the initial EEG signal coding model to obtain the predicted EEG signal coding features;

[0125] A second training unit 430 is used to input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features;

[0126] A contrastive learning unit 440, configured to calculate a contrastive learning loss function according to the predicted EEG signal encoding features and the true value text encoding features;

[0127] The classification unit 450 is used to input the predicted EEG signal coding feature into an initial classifier to obtain the predicted classification text corresponding to the predicted EEG signal coding feature, and calculate the classification loss function according to the predicted EEG signal coding feature, the corresponding predicted classification text and the text truth data;

[0128] The third training unit 460 is used to train the initial EEG signal encoding model and the initial classifier by combining the contrastive learning loss function and the classification loss function to obtain a trained EEG signal encoding model;

[0129] The encoding unit 470 is used to obtain the EEG signal to be generated, input the EEG signal to be generated into a trained EEG signal encoding model to obtain encoding features of the EEG signal to be generated, input the encoding features of the EEG signal to be generated into a classifier, and obtain a word prediction probability corresponding to the EEG signal to be generated;

[0130] The retrieval unit 480 is used to input the word prediction probability into the beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the EEG signal to be generated.

[0131] Optionally, the acquiring unit 410 is configured to:

[0132] S11, obtaining initial EEG data and initial text data in the data set;

[0133] S12, performing noise removal and filtering processing on the initial EEG data, wherein the noise removal operation includes removing eye movement and muscle artifacts, and the filtering processing includes high-pass filtering and notch filtering;

[0134] S13, segmenting the EEG signal after noise removal and filtering according to the relationship corresponding to the words to obtain the intermediate EEG signal corresponding to the words;

[0135] S14, filtering the stop words in the initial text data, and screening the filtered words according to frequency and semantic strength to obtain screened words;

[0136] S15. Compare the filtered words with the words corresponding to the intermediate EEG signals, select the semantically consistent words and intermediate EEG signals, determine the EEG signal training data and the corresponding text truth data, and the EEG signal training data and the corresponding text truth data are completely corresponding in position.

[0137] Optionally, the EEG signal encoding model includes a spatial encoding module, a Conformer module, a Subject layer, a maximum pooling layer and a first fully connected layer.

[0138] Optionally, the first training unit 420 is configured to:

[0139] S21, inputting the masked EEG signal training data into the spatial coding module to obtain spatial filtering features, integrating the frequency band positioning position embedding into the spatial filtering features, and obtaining the marked EEG signal;

[0140] S22, inputting the labeled EEG signal into the Conformer module to obtain the initial coding features of the EEG signal;

[0141] S23, inputting the initial coding features of the EEG signal into the Subject layer to process individual differences and generate corresponding EEG signal modeling data;

[0142] S24, inputting the EEG signal modeling data into the maximum pooling layer to obtain pooled feature data of the EEG signal;

[0143] S25. Input the pooled feature data into the first fully connected layer to obtain the predicted EEG signal encoding features.

[0144] Optionally, the comparative learning unit 440 is used to:

[0145] According to the predicted EEG signal encoding features, the true value text encoding features and the following formula (1), the contrastive learning loss function is calculated:

[0146]

[0147] in, represents the contrastive learning loss function, represents the i-th predicted EEG signal encoding feature, represents the i-th true value text encoding feature, represents the j-th true value text encoding feature, the i-th predicted EEG signal encoding feature corresponds to the i-th true value text encoding feature, and the i-th predicted EEG signal encoding feature does not correspond to the j-th true value text encoding feature; τ represents the preset temperature coefficient, and M represents the number of predicted EEG signal encoding features.

[0148] Optionally, the classifier includes a second fully connected layer and a Softmax layer;

[0149] The classification unit 450 is used to:

[0150] S51, inputting the predicted EEG signal encoding features into the initial second fully connected layer to obtain EEG connection features;

[0151] S52, inputting the EEG connection feature into the Softmax layer to obtain the probability of a word corresponding to the predicted EEG signal coding feature, and determining the predicted classification text corresponding to the predicted EEG signal coding feature according to the probability of a word corresponding to the predicted EEG signal coding feature;

[0152] S53, calculating the classification loss function according to the predicted EEG signal encoding features, the corresponding predicted classification text, the text truth data and the following formula (2):

[0153]

[0154] in, represents the classification loss function, is the one-hot encoding of the predicted classification text corresponding to the i-th predicted EEG signal encoding feature, e i is the i-th predicted EEG signal encoding feature, w i Represents the one-hot encoding of the real classification text, Represents the probability of predicting the i-th EEG signal encoding feature as the i-th classified text.

[0155] Optionally, the third training unit 460 is configured to:

[0156] S61. Calculate the total loss function according to the contrastive learning loss function, the classification loss function and the following formula (3):

[0157]

[0158] in, represents the total loss function, α represents the weight coefficient of the contrastive learning loss function, β represents the weight coefficient of the classification loss function, α+β=1;

[0159] S62. According to the total loss function, the initial EEG signal encoding model and the initial classifier are iteratively trained until the total loss function converges, and the training is stopped to obtain the trained EEG signal encoding model and classifier.

[0160] The embodiment of the present invention combines deep feature extraction of EEG signals, contrastive learning mechanism and efficient retrieval strategy, and the method can effectively decode text that meets semantic requirements from EEG signals. Specifically, the EEG data and text data are first preprocessed, and a closed word list is constructed by removing stop words and restoring parts of speech, so as to provide high-quality input for the subsequent decoding process. Then, the Conformer model is used to encode the EEG signals, and the Subject layer is introduced to adapt to the individual differences of different subjects to enhance the robustness of the EEG encoder. Through the mask contrast learning mechanism, the model can better capture the potential semantic information in the EEG signals, and the contrast learning loss function optimizes the representation ability of the encoder.

[0161] In the candidate word generation stage, the closed-set word classification module generates a candidate word set through a beam retrieval strategy, and then uses the AhoCorasick algorithm to match the retrieved candidate words to determine the text that best matches the original EEG signal. Finally, by combining beam retrieval and string matching, the sentence that best matches the semantics of the EEG signal is retrieved from the corpus to achieve efficient and accurate EEG text generation. The innovation of this embodiment lies in that by combining the strategy of semantic alignment and sentence retrieval, it overcomes the high computational overhead and generation accuracy problems that traditional generation models may encounter when processing EEG signals, thereby improving the effect and usability of EEG signal decoding.

[0162] Figure 5 is a structural diagram of an EEG text decoding device based on semantic alignment and sentence retrieval provided by an embodiment of the present invention, such as Figure 5 As shown, the EEG text decoding device based on semantic alignment and sentence retrieval may include the above Figure 4 The EEG text decoding device based on semantic alignment and sentence retrieval is shown. Optionally, the EEG text decoding device based on semantic alignment and sentence retrieval 510 may include a first processor 2001 .

[0163] Optionally, the EEG text decoding device 510 based on semantic alignment and sentence retrieval may also include a memory 2002 and a transceiver 2003 .

[0164] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0165] Combine the following Figure 5 The components of the EEG text decoding device 510 based on semantic alignment and sentence retrieval are specifically introduced:

[0166] The first processor 2001 is the control center of the EEG text decoding device 510 based on semantic alignment and sentence retrieval, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0167] Optionally, the first processor 2001 can perform various functions of the EEG text decoding device 510 based on semantic alignment and sentence retrieval by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0168] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0169] In a specific implementation, as an embodiment, the EEG text decoding device 510 based on semantic alignment and sentence retrieval may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0170] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0171] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0172] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0173] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0174] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently, and can be transmitted through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0175] It should be noted that Figure 5 The structure of the EEG text decoding device 510 based on semantic alignment and sentence retrieval shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0176] In addition, the technical effects of the EEG text decoding device 510 based on semantic alignment and sentence retrieval can refer to the technical effects of the EEG text decoding method based on semantic alignment and sentence retrieval described in the above method embodiment, and will not be repeated here.

[0177] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0178] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0179] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0180] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0181] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0182] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0183] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0185] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0188] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0189] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An EEG text decoding method based on semantic alignment and sentence retrieval, characterized in that: The method comprises: S1, obtain EEG signal training data and corresponding text truth data; S2, randomly masking the EEG signal training data, and inputting the masked EEG signal training data into the initial EEG signal encoding model to obtain the predicted EEG signal encoding features; S3, input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features; S4, calculating a contrastive learning loss function according to the predicted EEG signal encoding features and the true value text encoding features; S5, inputting the predicted EEG signal coding feature into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal coding feature, and calculating a classification loss function according to the predicted EEG signal coding feature, the corresponding predicted classification text and the text truth data; S6. Combining the contrastive learning loss function with the classification loss function, the initial EEG signal encoding model and the initial classifier are trained to obtain a trained EEG signal encoding model; S7, obtaining the EEG signal to be generated, inputting the EEG signal to be generated into a trained EEG signal coding model to obtain coding features of the EEG signal to be generated, inputting the coding features of the EEG signal to be generated into a classifier, and obtaining a word prediction probability corresponding to the EEG signal to be generated; S8. Input the word prediction probability into a beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the EEG signal to be generated.

2. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 1 is characterized in that: The step S1 of acquiring EEG signal training data and corresponding text truth data includes: S11, obtaining initial EEG data and initial text data in the data set; S12, performing noise removal and filtering processing on the initial EEG data, wherein the noise removal operation includes removing eye movement and muscle artifacts, and the filtering processing includes high-pass filtering and notch filtering; S13, segmenting the EEG signal after noise removal and filtering according to the relationship corresponding to the words to obtain the intermediate EEG signal corresponding to the words; S14, filtering the stop words in the initial text data, and screening the filtered words according to frequency and semantic strength to obtain screened words; S15. Compare the filtered words with the words corresponding to the intermediate EEG signals, select the semantically consistent words and intermediate EEG signals, determine the EEG signal training data and the corresponding text truth data, and the EEG signal training data and the corresponding text truth data are completely corresponding in position.

3. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 1 is characterized in that: The EEG signal encoding model includes a spatial encoding module, a Conformer module, a Subject layer, a maximum pooling layer and a first fully connected layer.

4. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 3 is characterized in that: The step S2 inputs the masked EEG signal training data into the initial EEG signal coding model to obtain the predicted EEG signal coding features, including: S21, inputting the masked EEG signal training data into the spatial coding module to obtain spatial filtering features, integrating the frequency band positioning position embedding into the spatial filtering features, and obtaining the marked EEG signal; S22, inputting the labeled EEG signal into the Conformer module to obtain the initial coding features of the EEG signal; S23, inputting the initial coding features of the EEG signal into the Subject layer to process individual differences and generate corresponding EEG signal modeling data; S24, inputting the EEG signal modeling data into the maximum pooling layer to obtain pooled feature data of the EEG signal; S25. Input the pooled feature data into the first fully connected layer to obtain the predicted EEG signal encoding features.

5. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 4 is characterized in that: The step S4 calculates the contrastive learning loss function according to the predicted EEG signal encoding features and the true text encoding features, including: According to the predicted EEG signal encoding features, the true value text encoding features and the following formula (1), the contrastive learning loss function is calculated: in, represents the contrastive learning loss function, represents the i-th predicted EEG signal encoding feature, represents the i-th true value text encoding feature, represents the j-th true value text encoding feature, the i-th predicted EEG signal encoding feature corresponds to the i-th true value text encoding feature, and the i-th predicted EEG signal encoding feature does not correspond to the j-th true value text encoding feature; τ represents the preset temperature coefficient, and M represents the number of predicted EEG signal encoding features.

6. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 4 is characterized in that: The classifier includes a second fully connected layer and a Softmax layer; The step S5 inputs the predicted EEG signal coding feature into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal coding feature, and calculates a classification loss function according to the predicted EEG signal coding feature, the corresponding predicted classification text, and the text truth data, including: S51, inputting the predicted EEG signal encoding features into the initial second fully connected layer to obtain EEG connection features; S52, inputting the EEG connection feature into the Softmax layer to obtain the probability of a word corresponding to the predicted EEG signal coding feature, and determining the predicted classification text corresponding to the predicted EEG signal coding feature according to the probability of a word corresponding to the predicted EEG signal coding feature; S53, calculating the classification loss function according to the predicted EEG signal encoding features, the corresponding predicted classification text, the text truth data and the following formula (2): in, represents the classification loss function, is the one-hot encoding of the predicted classification text corresponding to the i-th predicted EEG signal encoding feature, e i is the i-th predicted EEG signal encoding feature, w i Represents the one-hot encoding of the real classification text, Represents the probability of predicting the i-th EEG signal encoding feature as the i-th classified text.

7. The EEG text decoding method based on semantic alignment and sentence retrieval according to claim 4 is characterized in that: The S6 combines the contrastive learning loss function with the classification loss function to train the initial EEG signal encoding model and the initial classifier to obtain a trained EEG signal encoding model, including: S61. Calculate the total loss function according to the contrastive learning loss function, the classification loss function and the following formula (3): in, represents the total loss function, α represents the weight coefficient of the contrastive learning loss function, β represents the weight coefficient of the classification loss function, α+β=1; S62. According to the total loss function, the initial EEG signal encoding model and the initial classifier are iteratively trained until the total loss function converges, and the training is stopped to obtain the trained EEG signal encoding model and classifier.

8. An EEG text decoding device based on semantic alignment and sentence retrieval, the EEG text decoding device based on semantic alignment and sentence retrieval is used to implement the EEG text decoding method based on semantic alignment and sentence retrieval as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition unit, used to acquire EEG signal training data and corresponding text truth data; The first training unit is used to randomly mask the EEG signal training data, and input the masked EEG signal training data into the initial EEG signal coding model to obtain the predicted EEG signal coding features; The second training unit is used to input the text truth data into the BERT model with frozen parameters to obtain the truth text encoding features; A contrastive learning unit, used for calculating a contrastive learning loss function according to the predicted EEG signal encoding features and the true value text encoding features; A classification unit, used for inputting the predicted EEG signal coding features into an initial classifier to obtain a predicted classification text corresponding to the predicted EEG signal coding features, and calculating a classification loss function according to the predicted EEG signal coding features, the corresponding predicted classification text and the text truth data; The third training unit is used to train the initial EEG signal encoding model and the initial classifier by combining the contrastive learning loss function and the classification loss function to obtain a trained EEG signal encoding model; The encoding unit is used to obtain the EEG signal to be generated, input the EEG signal to be generated into a trained EEG signal encoding model to obtain the encoding features of the EEG signal to be generated, input the encoding features of the EEG signal to be generated into a classifier, and obtain the word prediction probability corresponding to the EEG signal to be generated; The retrieval unit is used to input the word prediction probability into a beam search module to obtain a keyword set, retrieve the text sentence with the highest similarity to the keyword set in the corpus according to the AhoCorasick algorithm, and determine the text sentence as the text corresponding to the electroencephalogram signal to be generated.

9. An EEG text decoding device based on semantic alignment and sentence retrieval, characterized in that: The EEG text decoding device based on semantic alignment and sentence retrieval includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images