A method and device for Chinese decoding based on brain magnetic signals

Abstract

The application relates to the technical field of brain-computer interfaces, and provides a Chinese decoding method and device based on magnetoencephalogram signals, which comprises the following steps: constructing a fixed Chinese sentence library comprising a plurality of samples, wherein the samples comprise a Chinese target sentence and a standard full-pinyin sequence; collecting magnetoencephalogram signals and key events when a user inputs the Chinese target sentence in full pinyin; performing normalization processing on the key events to obtain a character-level key label sequence; inputting the magnetoencephalogram signals into a neural decoding model to obtain a time sequence representation and an output character probability; training the neural decoding model according to the output character probability, the character-level key label sequence, the standard full-pinyin sequence and the time sequence representation; obtaining target magnetoencephalogram signals of the user and inputting the target magnetoencephalogram signals into the trained neural decoding model to obtain a target sequence; and matching the normalized target sequence with the fixed Chinese sentence library or a pinyin Chinese mapping table to obtain a Chinese decoding sentence. The application can stably and explainably recover a Chinese sentence based on magnetoencephalogram signals.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of brain-computer interfaces, and in particular to a Chinese decoding method and apparatus based on magnetoencephalographic signals. Background

[0002] Brain-computer interfaces aim to restore human motor intention, language intention or text input intention through neural signals. For typing tasks, when a user inputs characters on a keyboard, their brain activities are temporally associated with key preparation, motor execution, visual feedback and language processing processes. OPM-MEG (referring to magnetoencephalographic signals collected using the optically pumped magnetometer OPM technology), as a non-invasive magnetoencephalography acquisition technology, can record brain activities related to typing behavior with high temporal resolution, and thus is suitable for character-level or key-level neural decoding.

[0003] For English typing tasks, existing methods usually decode magnetoencephalographic signals into QWERTY character sequences through a model, and use sequence learning methods to handle the non-strict synchronization problem between brain signals and key sequences. However, there are obvious differences between Chinese input tasks and English input tasks. The same pinyin sequence in Chinese may correspond to multiple Chinese characters or words. For example, "shijie" can correspond to "世界" (world), "十届" (tenth session), etc. Therefore, Chinese full-pinyin input usually includes two stages: pinyin key sequence input and input method candidate selection. At this time, if the magnetoencephalographic signals are directly aligned to the Chinese character sequence, it is easy to mix input method candidate selection, homophone disambiguation and language model decision-making into the neural decoding labels, thereby reducing the stability and interpretability of model training. In a fixed sentence library scenario, even if the corresponding standard full-pinyin representation can be pre-established for Chinese sentences to provide a priori conditions for limiting the output space of the neural decoding model, it is still difficult to distinguish neural decoding errors, homophone ambiguities and input method post-processing errors.

[0004] Therefore, the technical problem to be solved by the present application is, under the conditions of a fixed Chinese sentence library and full-pinyin input, how to avoid the ambiguities of homophones and input method candidate selection caused by directly aligning magnetoencephalographic signals to Chinese character labels, so as to realize stable and interpretable restoration of Chinese sentences based on magnetoencephalographic signals. Summary of the Invention

[0005] The embodiments of the present application provide a Chinese decoding method and apparatus based on magnetoencephalographic signals, which can avoid the ambiguities of homophones and input method candidate selection caused by directly aligning magnetoencephalographic signals to Chinese character labels, and realize stable and interpretable restoration of Chinese sentences based on magnetoencephalographic signals.

[0006] In a first aspect, the embodiments of the present application provide a Chinese decoding method based on magnetoencephalographic signals, comprising:

[0007] Construct a fixed Chinese sentence database consisting of multiple samples, including the target Chinese sentence and the corresponding standard full-spell sequence and sentence identifier;

[0008] Collect brainwave signals and key press events when users input Chinese target sentences using full Pinyin input;

[0009] Normalize the key events to obtain a character-level key label sequence;

[0010] The brain magnetic signal is input into the neural decoding model to obtain the time series representation of the sample and the output character probability. The output character probability is either the probability of Pinyin character or the probability of key character.

[0011] The CTC loss is obtained based on the output character probability and the character-level key label sequence. The alignment auxiliary loss is obtained based on the output character probability, the standard full-spelling sequence, and the time series representation. The neural decoding model is updated based on the CTC loss and the alignment auxiliary loss.

[0012] The target brainwave signal of the user is acquired and input into a trained neural decoding model to obtain the target character probability, and the target sequence is obtained based on the target character probability.

[0013] The normalized target sequence is matched with a fixed Chinese sentence database or a Pinyin-Chinese mapping table generated based on the fixed Chinese sentence database to obtain the Chinese decoded sentence.

[0014] Optionally, the character-level key label sequence includes Latin letter tokens, space tokens, and separator tokens; the above normalization processing of key events yields a character-level key label sequence, including:

[0015] Map the space key in the key event to a space token;

[0016] Convert the tone characters to ASCII characters without tone to obtain the Latin letter token;

[0017] Convert Chinese punctuation marks or full-width punctuation marks to space tokens or separator tokens.

[0018] Optionally, when the output character probability is the Pinyin character probability, the alignment auxiliary loss obtained above based on the output character probability, the standard full Pinyin sequence, and the time series representation includes:

[0019] The pinyin sequence is obtained based on the probability of the pinyin characters.

[0020] The time-series representation is segmented based on the pinyin sequence to obtain the pinyin syllable-level neural representation;

[0021] Generate corresponding pinyin syllable-level text representations based on standard full pinyin sequences;

[0022] Alignment-assisted loss is derived from the syllable-level neural representation and the syllable-level text representation of Pinyin.

[0023] Optionally, when the output character probability is the keystroke character probability, the alignment auxiliary loss obtained above based on the output character probability, the standard full-spelling sequence, and the time series representation includes:

[0024] The key sequence is obtained based on the probability of the key characters.

[0025] The time-series representation is segmented based on the key sequence to obtain word-level neural representations;

[0026] Generate corresponding word-level text representations based on standard full-spelling sequences;

[0027] Alignment-assisted loss is derived from word-level neural representations and word-level text representations.

[0028] Optionally, the neural decoding model includes a temporal convolution module, a Conformer encoding module, and a CTC output head connected in sequence; the time series representation is obtained through the output of the Conformer encoding module.

[0029] Optionally, the method further includes:

[0030] If the normalized target sequence fails to match in the fixed Chinese sentence database or the Pinyin Chinese mapping table, the edit distance between the normalized target sequence and each standard full Pinyin sequence is calculated. The Chinese target sentences corresponding to the standard full Pinyin sequences whose edit distance is less than or equal to the preset edit threshold are selected as candidate Chinese sentences. The candidate Chinese sentence with the smallest edit distance is selected as the Chinese decoding sentence.

[0031] Optionally, the method further includes:

[0032] If there are multiple candidate Chinese sentences with the smallest edit distance, the Chinese decoding statement is determined based on the frequency of occurrence of the candidate Chinese sentences in the fixed Chinese sentence database, the preset priority, or the lexicographical order rule.

[0033] Secondly, another embodiment of this application provides a Chinese decoding device based on magnetoencephalography (MEG) signals, comprising:

[0034] The building module is used to construct a fixed Chinese sentence library containing multiple samples. The samples include the target Chinese sentence and the standard full-pinyin sequence and sentence identifier corresponding to the target Chinese sentence.

[0035] The acquisition module is used to collect the brain magnetic signals and key press events when the user inputs the target Chinese sentence in full Pinyin.

[0036] The label module is used to normalize key events and obtain a character-level key label sequence;

[0037] The input module is used to input the magnetoencephalogram (MEG) signal into the neural decoding model to obtain the time series representation of the sample and the output character probability. The output character probability is either the probability of a pinyin character or the probability of a key character.

[0038] The training module is used to obtain the CTC loss based on the output character probability and the character-level key label sequence, obtain the alignment auxiliary loss based on the output character probability, the standard full spelling sequence and the time series representation, and update the neural decoding model based on the CTC loss and the alignment auxiliary loss.

[0039] The inference module is used to acquire the user's target brain magnetic signal and input it into a trained neural decoding model to obtain the target character probability, and then obtain the target sequence based on the target character probability.

[0040] The mapping module is used to match the normalized target sequence with a fixed Chinese sentence database or a Pinyin-Chinese mapping table generated based on the fixed Chinese sentence database to obtain the Chinese decoded sentence.

[0041] In another aspect, this application provides a computer device including at least one connected processor, memory, and transceiver, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the methods described in the above aspects.

[0042] In another aspect, this application provides a computer storage medium including instructions that, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0043] Compared with existing technologies, the Chinese decoding method based on magnetoencephalography (MEG) signals provided in this application has the following beneficial effects: First, this application collects MEG signals and key events when a user inputs a Chinese target sentence using full pinyin input, and inputs the MEG signals into a neural decoding model, outputting a time series representation and pinyin / keyword character probabilities. That is, this application sets the MEG signal decoding target as pinyin / keyword character probabilities, rather than Chinese character sequences, thereby reducing the interference of homophones and input method candidate selection on model training. Second, after obtaining the target sequence in the inference stage, this application matches it with a fixed Chinese sentence library or a pinyin-Chinese mapping table generated based on a fixed Chinese sentence library to obtain the Chinese decoded sentence of the target sequence. Since the version of the fixed Chinese sentence library is determined, the same target sequence input will inevitably produce the same result, thereby reducing the uncertainty brought about by the general input method candidate ranking and the random generation of open-domain language models, and improving the stability and interpretability of Chinese sentence decoding. Attached Figure Description

[0044] Figure 1 This is a flowchart of a Chinese decoding method based on magnetoencephalography (MEG) signals according to this application;

[0045] Figure 2This is a schematic diagram of the neural decoding model training process in this application;

[0046] Figure 3 This is a structural diagram of a Chinese decoding device based on magnetoencephalography (MEG) signals according to this application. Detailed Implementation

[0047] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0048] Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The module divisions appearing in the embodiments of this application are merely logical divisions; in actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces, and the indirect couplings or communication connections between modules may be electrical or other similar forms, none of which are limited in the embodiments of this application. Moreover, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0049] Brain-computer interfaces aim to recover a person's motor intentions, verbal intentions, or text input intentions through neural signals. For typing tasks, when a user inputs characters on a keyboard, their brain activity is temporally correlated with key preparation, motor execution, visual feedback, and language processing. OPM-MEG (referring to magnetoencephalography signals acquired using optically pumped magnetometer (OPM) technology), as a non-invasive magnetoencephalography acquisition technique, can record brain activity related to typing behavior with high temporal resolution, making it suitable for character-level or key-level neural decoding.

[0050] For English typing tasks, existing methods usually decode magnetoencephalography signals into QWERTY character sequences through a model, and use sequence learning methods to handle the non-strict synchronization problem between brain signals and keystroke sequences. However, there are obvious differences between Chinese input tasks and English input tasks. The same pinyin sequence in Chinese may correspond to multiple Chinese characters or words. For example, "shijie" can correspond to "shijie (world)", "shijie (tenth session)", etc. Therefore, Chinese full-pinyin input usually includes two stages: pinyin keystroke sequence entry and input method candidate selection. At this time, if the magnetoencephalography signals are directly aligned to the Chinese character sequence, it is easy to mix the input method candidate selection, homophone disambiguation and language model decision-making into the neural decoding labels, thereby reducing the stability and interpretability of model training.

[0051] In the scenarios of controlled experiments and fixed sentence libraries, experimental materials are usually composed of pre-set Chinese sentences. The pre-set Chinese sentences can objectively establish corresponding standard full-pinyin representations, which provides a priori condition for limiting the output space of the model. However, without an independent decoding layer from magnetoencephalography signals to pinyin sequences, and a pinyin-to-Chinese character mapping layer with fixed rules and traceable processes, it is still difficult to distinguish neural decoding errors, homophone ambiguities and input method post-processing errors. Therefore, there is a need for a magnetoencephalography signal to Chinese sentence decoding method suitable for Chinese full-pinyin input experiments with a fixed sentence library, so that the model first recovers the pinyin or keystroke sequences that are more directly related to the magnetoencephalography timing, and then outputs the final Chinese character sentence based on the pinyin or keystroke sequences.

[0052] To solve the above problems, please refer to Figure 1 , the embodiments of the present application provide a Chinese decoding method based on magnetoencephalography signals. This method can be used in non-invasive brain-computer interface Chinese input experiments, Chinese typing intention decoding, auxiliary communication systems, magnetoencephalography language cognition research, and controlled Chinese input evaluation systems for fixed corpora, comprising:

[0053] Step S1: Construct a fixed Chinese sentence library including a plurality of samples, where each sample includes a target Chinese sentence, as well as a standard full-pinyin sequence and a sentence identifier corresponding to the target Chinese sentence. For example:

[0054]

[0055] The standard full-spell sequence is a full-spell sequence without tones, and adjacent pinyin syllables are separated by spaces. Pinyin containing "ü" can be uniformly converted to "v"; diacritics can be retained as diacritics tokens; Chinese punctuation can be converted to space tokens or separator tokens. In the specific implementation process, the fixed Chinese sentence database can be divided into training set, validation set, and test set. The division can be based on independent trials, conversations, or subjects. The same sentence can appear in different sets, but the same sampling trial in step S2 cannot cross sets. Division is only based on sentence identifiers when no sentence generalization is observed. Sentences that cross conversations are given priority to enter the validation set. During the model training phase, only samples from the training set are used, and samples from the validation set are used when verifying whether the model has converged.

[0056] Step S2: Collect the brain magnetoencephalogram (MEG) signals and key press events when the user inputs the target Chinese sentence in full Pinyin.

[0057] Specifically, the system displays a target Chinese sentence from a fixed Chinese sentence database on an electronic device, and the user inputs the target Chinese sentence using full pinyin based on the on-screen prompts.

[0058] During user input, EEG signals and key press events are collected simultaneously. Each key press event includes the key value pressed by the user, the start time, the sentence identifier, and the user identifier.

[0059] For each Chinese target sentence, a continuous segment of magnetoencephalography (MEG) signal is extracted based on the sentence start event and sentence end event; furthermore, the MEG signal may also include MEG signal segments collected within a preset time window before the Chinese target sentence is displayed and a preset tail time window after the sentence input is completed.

[0060] Step S3: Normalize the key events to obtain a character-level key label sequence.

[0061] The character-level key label sequence includes one or more of the following: Latin letter tokens, space tokens, and separator tokens. The above normalization process for key events yields the character-level key label sequence, including:

[0062] Step S31: Map the space key in the key event to a space token.

[0063] Step S32: Convert the tone characters to ASCII characters without tone to obtain the Latin letter token.

[0064] Step S33: Convert Chinese punctuation or full-width punctuation into space tokens or separator tokens.

[0065] In addition, non-target key events or control key events can be removed from key events before normalization, and uppercase letters can be converted to lowercase letters. For example, if the user inputs:

[0066] ni hao shi jie

[0067] The above key events can be converted into a character-level key label sequence:

[0068] ni & hao & shi & jie

[0069] In this context, "&" represents a space (token).

[0070] Step S4: Input the brain magnetic signal into the neural decoding model to obtain the time series representation of the sample and the output character probability. The output character probability is either the probability of a pinyin character or the probability of a keystroke character.

[0071] Please see Figure 2 The neural decoding model may include a temporal convolutional module, a Conformer encoding module, and a CTC output head connected in sequence; the time series representation is obtained through the output of the Conformer encoding module, and the CTC output head outputs the probability of the pinyin character or the probability of the key character corresponding to each time step. In some embodiments, a spatial mixing layer may also be added between the temporal convolutional module and the Conformer encoding module.

[0072] Step S5: Obtain the CTC loss based on the output character probability and the character-level key label sequence; obtain the alignment auxiliary loss based on the output character probability, the standard full-spelling sequence, and the time series representation; and update the neural decoding model based on the CTC loss and the alignment auxiliary loss.

[0073] Optionally, when the output character probability is the Pinyin character probability, the alignment auxiliary loss obtained above based on the output character probability, the standard full Pinyin sequence, and the time series representation includes:

[0074] Step S511: Obtain the pinyin sequence based on the probability of the pinyin characters.

[0075] By performing CTC decoding on the probabilities of the pinyin characters, a pinyin sequence can be obtained.

[0076] Step S512: Segment the time-series representation according to the pinyin sequence to obtain the pinyin syllable-level neural representation. Specifically, based on the position of the space token in the pinyin sequence output by the CTC output head, the time-series representation of the neural decoding model is divided into multiple segments, each segment corresponding to a pinyin syllable. Then, consecutive pinyin syllables are merged into multiple pinyin syllable-level segments to obtain the pinyin syllable-level neural representation.

[0077] Step S513: Generate the corresponding pinyin syllable-level text representation based on the standard full pinyin sequence.

[0078] For example, the standard full-pinyin sequence ni hao shi jie is divided into four pinyin syllables: ni, hao, shi, and jie. Text embeddings are generated for each syllable to obtain a pinyin syllable-level text representation.

[0079] Step S514: Obtain the alignment assistance loss based on the syllable-level neural representation and the syllable-level text representation of the pinyin. Alignment training employs contrastive learning loss, which increases the similarity between corresponding neural representations and text representations within the same sentence, and decreases the similarity between non-corresponding neural representations and text representations. Alignment training does not directly introduce Chinese character labels, thus reducing the impact of homophone ambiguity on the neural decoding stage.

[0080] Optionally, when the output character probability is the keystroke character probability, the alignment auxiliary loss obtained above based on the output character probability, the standard full-spelling sequence, and the time series representation includes:

[0081] Step S521: Obtain the key sequence based on the probability of the key characters.

[0082] By performing CTC decoding on the probability of the key characters, the key sequence can be obtained.

[0083] Step S522: Segment the time series representation according to the key sequence to obtain word-level neural representation.

[0084] Specifically, based on the position of the space token in the key sequence output by the CTC output head, the time series representation of the neural decoding model is divided into multiple segments, each segment corresponding to a pinyin word. Then, consecutive pinyin words are merged into multiple word-level segments to obtain word-level neural representations.

[0085] Step S523: Generate the corresponding word-level text representation based on the standard full-spell sequence.

[0086] Step S524: Obtain alignment-assisted loss based on word-level neural representation and word-level text representation.

[0087] Assuming the brain magnetic signal is X, and the character-level key label sequence is... The neural decoding model outputs the probability of the corresponding pinyin character at each time step based on the magnetoencephalogram signal X. .

[0088] At this point, the CTC loss calculated based on the Pinyin character probability and the character-level key label sequence is:

[0089] .

[0090] Here, π represents the candidate alignment path containing blanks and repeating characters, and B denotes the operation of removing blanks and merging consecutive repeating characters. The CTC loss is backpropagated to the temporal convolutional module and the Conformer encoding module. The contrastive auxiliary loss from alignment training serves as the auxiliary training objective, and the shared neural encoder and neural projection layer in the Conformer encoding module are backpropagated.

[0091] In the specific implementation process, steps S1-S5 are the model training phase. During the training process, steps S2-S5 are repeated until the neural decoding model meets the convergence condition, and the trained neural decoding model is obtained and applied to the inference phase. The convergence condition for model training can be the Pinyin character error rate index.

[0092] .

[0093] Where S, D, and I represent the number of replacements, deletions, and insertions, respectively, and N represents the total number of pinyin syllables. In addition, options include pinyin sequence accuracy, alignment retrieval accuracy or positive / negative sample similarity interval, fixed sentence database candidate Top-1 accuracy, Chinese decoded sentence accuracy, rejection rate, and false acceptance rate.

[0094] When the error rate of the neural decoding model in the validation set reaches a preset threshold for pinyin characters, and the improvement over K consecutive training rounds is less than δ, and the complete accuracy of the Chinese decoding sentence reaches a preset value, the training is considered to have converged, training can be stopped, and the model checkpoint with the lowest pinyin character error rate in the validation set is selected for inference.

[0095] In one embodiment, it can be set , As a parameter for early stopping.

[0096] Step S6: Obtain the user's target brain magnetic signal and input it into the trained neural decoding model to obtain the target character probability, and obtain the target sequence based on the target character probability.

[0097] During the reasoning phase, the target sequence can be in the form of a pinyin sequence or a key sequence.

[0098] Step S7: Match the normalized target sequence with a fixed Chinese sentence database or a Pinyin-Chinese mapping table generated based on the fixed Chinese sentence database to obtain the Chinese decoded sentence.

[0099] The Pinyin-Chinese mapping table uses the standard full Pinyin sequence as the key and the target Chinese sentence as the value. For example:

[0100] ni hao shi jie -> Hello World

[0101] I want to drink water.

[0102] Specifically, the target sequence output by the neural decoding model is first normalized. If the normalized target sequence matches exactly in the mapping table, the corresponding Chinese target sentence is output as the Chinese decoding statement.

[0103] Steps S6-S7 above constitute the inference phase. Key events and standard full-spelling sequences are only used for training label generation or offline quality analysis. Key names, key characters, or key press times are not input during the inference phase.

[0104] Compared with existing technologies, the Chinese decoding method based on magnetoencephalography (MEG) signals provided in this application has the following beneficial effects: First, this application collects MEG signals and key events when a user inputs a Chinese target sentence using full pinyin input, and inputs the MEG signals into a neural decoding model, outputting a time series representation and pinyin / keyword character probabilities. That is, this application sets the MEG signal decoding target as pinyin / keyword character probabilities, rather than Chinese character sequences, thereby reducing the interference of homophones and input method candidate selection on model training. Second, after obtaining the target sequence in the inference stage, this application matches it with a fixed Chinese sentence library or a pinyin-Chinese mapping table generated based on a fixed Chinese sentence library to obtain the Chinese decoded sentence of the target sequence. Since the version of the fixed Chinese sentence library is determined, the same target sequence input will inevitably produce the same result, thereby reducing the uncertainty brought about by the general input method candidate ranking and the random generation of open-domain language models, and improving the stability and interpretability of Chinese sentence decoding.

[0105] In some embodiments, the method further includes:

[0106] Step S8: If the normalized target sequence fails to match in the fixed Chinese sentence database or the Pinyin Chinese mapping table, calculate the edit distance between the normalized target sequence and each standard full Pinyin sequence, take the Chinese target sentence corresponding to the standard full Pinyin sequence whose edit distance is less than or equal to the preset edit threshold as the candidate Chinese sentence, and select the candidate Chinese sentence with the smallest edit distance as the Chinese decoding sentence.

[0107] Specifically, if an exact match fails, the edit distance between the normalized target sequence and each standard full-spell sequence is calculated, and the best candidate is selected from the candidate Chinese sentences whose edit distance is less than or equal to a preset edit threshold. Assume the fixed Chinese sentence database is as follows:

[0108] .

[0109] in, It is a standard full spelling sequence. This corresponds to the Chinese target sentence. The neural decoding model outputs a normalized target sequence. Then, an exact match is performed first; if an exact match fails, you can choose:

[0110] .

[0111] Herein, is a predefined character-level or syllable-level weighted edit distance. Only when , a candidate Chinese sentence is output , τ is a preset edit threshold, otherwise a rejection mark is output.

[0112] For example, if the model outputs "ni hao shi ji", and the phrase table contains "ni hao shi jie -> 你好世界" and the edit distance between the two does not exceed the preset edit threshold, "你好世界" can be output as the Chinese decoded sentence.

[0113] Optionally, the method further comprises:

[0114] Step S9: if there are a plurality of candidate Chinese sentences with the minimum edit distance, determining the Chinese decoded sentence according to the occurrence frequency of the candidate Chinese sentences in the fixed Chinese sentence library, a preset priority or a lexicographical order rule.

[0115] The final output at least comprises the candidate Chinese sentence, the target sequence, the Chinese decoded sentence, the user identifier and the sentence identifier corresponding to the Chinese decoded sentence. In practical applications, an external Chinese input method, an n-gram language model, a neural language model or a candidate reordering model may be used to perform steps S7 to S9, so as to expand to a larger sentence library or an open-domain Chinese input scenario while keeping the neural decoding model unchanged.

[0116] It can be understood that, since the version of the fixed Chinese sentence library is determined, the edit distance calculation method is fixed, the preset edit threshold is fixed, the candidate sorting and same-score processing rules are fixed, and rejection is performed when the preset edit threshold is not reached instead of free generation, the same target sequence, fixed Chinese sentence library and parameters will inevitably produce the same result, thereby reducing the uncertainty caused by candidate sorting of a general input method and random generation of an open-domain language model.

[0117] Referring to Figure 3 , another embodiment of the present application provides a Chinese decoding apparatus based on magnetoencephalographic signals, comprising:

[0118] A construction module 101, configured to construct a fixed Chinese sentence library including a plurality of samples, where the samples include a Chinese target sentence, and a standard full-pinyin sequence and a sentence identifier corresponding to the Chinese target sentence.

[0119] An acquisition module 102, configured to acquire magnetoencephalographic signals and key events when a user inputs the Chinese target sentence in full pinyin.

[0120] A labeling module 103, configured to perform normalization processing on the key events to obtain a character-level key label sequence.

[0121] The input module 104 is used to input the magnetoencephalogram (MEG) signal into the neural decoding model to obtain the time series representation of the sample and the output character probability. The output character probability is either the probability of a pinyin character or the probability of a keystroke character.

[0122] Training module 105 is used to obtain CTC loss based on output character probability and character-level key label sequence, obtain alignment auxiliary loss based on output character probability, standard full spelling sequence and time series representation, and update neural decoding model based on CTC loss and alignment auxiliary loss.

[0123] The inference module 106 is used to acquire the user's target brain magnetic signal and input it into the trained neural decoding model to obtain the target character probability, and obtain the target sequence based on the target character probability.

[0124] The mapping module 107 is used to match the normalized target sequence with a fixed Chinese sentence database or a Pinyin-Chinese mapping table generated based on the fixed Chinese sentence database to obtain the Chinese decoded sentence.

[0125] In some embodiments, the device further includes an editing module, which is used to calculate the edit distance between the normalized target sequence and each standard full-spell sequence when the mapping module 107 fails to match, and to select the Chinese target sentence corresponding to the standard full-spell sequence whose edit distance is less than or equal to a preset edit threshold as a candidate Chinese sentence, and to select the candidate Chinese sentence with the smallest edit distance as the Chinese decoding sentence.

[0126] In some embodiments, the editing module is further configured to determine the Chinese decoding statement based on the frequency of occurrence of the candidate Chinese sentences in a fixed Chinese sentence database, a preset priority, or a lexicographical order rule when there are multiple candidate Chinese sentences with the smallest editing distance.

[0127] The specific limitations of the Chinese decoding device based on magnetoencephalography (MEG) signals provided in this embodiment can be found in the embodiment of the Chinese decoding method based on MEG signals described above, and will not be repeated here. Each module in the above-described Chinese decoding device based on MEG signals can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0128] This application provides a computer device that may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it causes the processor to perform the steps of a Chinese character decoding method based on brainwave signals as described in any of the above embodiments.

[0129] The working process, working details, and technical effects of the computer device provided in this embodiment can be found in the embodiment of a Chinese decoding method based on brain magnetic signals described above, and will not be repeated here.

[0130] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of a Chinese character decoding method based on magnetoencephalography (MEG) signals as described in any of the above embodiments. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0131] The working process, working details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiment of a Chinese decoding method based on magnetoencephalography signals described above, and will not be repeated here.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0133] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A Chinese decoding method based on magnetoencephalography (MEG) signals, characterized in that, include: A fixed Chinese sentence database is constructed, comprising multiple samples, wherein the samples include Chinese target sentences and standard full-spelling sequences and sentence identifiers corresponding to the Chinese target sentences; Collect brainwave signals and key press events when users input Chinese target sentences using full Pinyin input; The key events are normalized to obtain a character-level key label sequence; The brain magnetic signal is input into a neural decoding model to obtain the time series representation of the sample and the output character probability, wherein the output character probability is the probability of pinyin characters or the probability of key characters. The CTC loss is obtained based on the output character probability and the character-level key label sequence. The alignment auxiliary loss is obtained based on the output character probability, the standard full-spelling sequence, and the time series representation. The neural decoding model is updated based on the CTC loss and the alignment auxiliary loss. The target brain magnetic signal of the user is acquired and input into a trained neural decoding model to obtain the target character probability, and the target sequence is obtained based on the target character probability. The normalized target sequence is matched with the fixed Chinese sentence database or the Pinyin-Chinese mapping table generated based on the fixed Chinese sentence database to obtain the Chinese decoded sentence.

2. The Chinese decoding method based on magnetoencephalography (MEG) signals according to claim 1, characterized in that, The character-level key label sequence includes Latin letter tokens, space tokens, and separator tokens; the normalization process of the key events yields the character-level key label sequence, including: Map the space key in the key event to a space token; Convert the tone characters to ASCII characters without tone to obtain the Latin letter token; Convert Chinese punctuation marks or full-width punctuation marks to space tokens or separator tokens.

3. The Chinese decoding method based on magnetoencephalography (MEG) signals according to claim 1, characterized in that, When the output character probability is a Pinyin character probability, the step of obtaining the alignment auxiliary loss based on the output character probability, the standard full Pinyin sequence, and the time series representation includes: The pinyin sequence is obtained based on the probability of the pinyin characters. The time-series representation is segmented based on the pinyin sequence to obtain the pinyin syllable-level neural representation; Generate the corresponding pinyin syllable-level text representation based on the standard full pinyin sequence; Alignment-aided loss is obtained based on the syllable-level neural representation and the syllable-level text representation of the pinyin.

4. The Chinese decoding method based on magnetoencephalography (MEG) signals according to claim 1, characterized in that, When the output character probability is the keystroke character probability, the alignment auxiliary loss obtained based on the output character probability, the standard full-character sequence, and the time series representation includes: The key sequence is obtained based on the probability of the key characters. The time-series representation is segmented based on the key sequence to obtain word-level neural representations; Generate corresponding word-level text representations based on the standard full-spelling sequence; Alignment-assisted loss is obtained based on the word-level neural representation and the word-level text representation.

5. The Chinese decoding method based on magnetoencephalography (MEG) signals according to claim 3, characterized in that, The neural decoding model includes a temporal convolution module, a Conformer encoding module, and a CTC output head connected in sequence; the time series representation is obtained through the output of the Conformer encoding module.

6. The Chinese decoding method based on magnetoencephalography (MEG) signals according to claim 1, characterized in that, Also includes: If the normalized target sequence fails to match in the fixed Chinese sentence database or the Pinyin Chinese mapping table, the edit distance between the normalized target sequence and each standard full Pinyin sequence is calculated. The Chinese target sentence corresponding to the standard full Pinyin sequence whose edit distance is less than or equal to the preset edit threshold is taken as the candidate Chinese sentence, and the candidate Chinese sentence with the smallest edit distance is selected as the Chinese decoding statement.

7. The Chinese decoding method based on magnetoencephalography (MEG) signals according to claim 6, characterized in that, Also includes: If there are multiple candidate Chinese sentences with the smallest edit distance, the Chinese decoding statement is determined based on the frequency of occurrence of the candidate Chinese sentences in the fixed Chinese sentence database, the preset priority, or the lexicographical order rule.

8. A Chinese decoding device based on magnetoencephalography (MEG) signals, characterized in that, include: A construction module is used to construct a fixed Chinese sentence library including multiple samples, wherein the samples include Chinese target sentences and standard full-spelling sequences and sentence identifiers corresponding to the Chinese target sentences; The acquisition module is used to collect the brain magnetic signals and key press events when the user inputs the target Chinese sentence in full Pinyin. The tag module is used to normalize the key events and obtain a character-level key tag sequence; The input module is used to input the magnetoencephalogram (MEG) signal into the neural decoding model to obtain the time series representation of the sample and the output character probability, wherein the output character probability is the probability of a pinyin character or the probability of a key character. The training module is used to obtain the CTC loss based on the output character probability and the character-level key label sequence, obtain the alignment auxiliary loss based on the output character probability, the standard full-spelling sequence and the time series representation, and update the neural decoding model based on the CTC loss and the alignment auxiliary loss. The inference module is used to acquire the user's target brain magnetic signal and input it into a trained neural decoding model to obtain the target character probability, and obtain the target sequence based on the target character probability. The mapping module is used to match the normalized target sequence with the fixed Chinese sentence database or the Pinyin-Chinese mapping table generated based on the fixed Chinese sentence database to obtain the Chinese decoded sentence.

9. A computer device, characterized in that, It includes at least one processor, a memory, and a transceiver; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to perform the method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, It includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images