Chinese character writing decoding method for invasive brain-machine interface

Abstract

Disclosed in the present application is a Chinese character writing decoding method for an invasive brain-machine interface. The method comprises: acquiring original motor neural signals, and processing same to obtain ESA neural signals; standardizing the ESA neural signals, and performing division to obtain a training set and a test set; dividing the training set into a handwriting stroke data set and a handwriting stroke pause data set; training a filter for decoding strokes and a filter for decoding stroke pauses, and training a state discriminator for discriminating between stroke writing and stroke writing pauses; inputting data of the test set into the state discriminator, inputting discriminated stroke writing state neural data into the filter for decoding strokes, and inputting discriminated stroke writing pause state neural data into the filter for decoding stroke pauses; and concatenating decoded stroke data and decoded stroke pause data, so as to obtain a final decoded Chinese character. In the present application, the inconsistency between stroke writing and stroke writing pauses during Chinese character writing is taken into consideration, corresponding motor neural signals are divided into two states, i.e., stroke writing and stroke writing pauses, and different filters are trained, thereby effectively reducing the impact of the differences in neural data in different states on a decoder, and improving the performance and robustness of the decoder.

Description

A Chinese character writing decoding method for invasive brain-computer interface Technical Field

[0001] The present invention relates to the field of brain-computer interface signal decoding, and in particular to a Chinese character writing decoding method for invasive brain-computer interfaces. Background Art

[0002] Brain-computer interface (BMI) systems aim to restore some motor function and special senses in paralyzed patients. They do this primarily by recording neuronal activity in the brain and translating it into external motion control commands, such as those for prosthetic limbs and computer cursors. Through BMI technology, humans can express their thoughts and manipulate devices through their brains, effectively improving the ability of people with physical disabilities to communicate with the outside world and control their external environment, significantly enhancing their quality of life.

[0003] For example, Chinese patent document CN102309380A discloses an intelligent wheelchair based on a multimodal brain-computer interface, which converts the subject's control intention into instructions and sends them to the communication unit of the control module, and then controls the wheelchair through the controller.

[0004] A Chinese patent document with publication number CN114366122A discloses a method for analyzing motor imagery based on an EEG brain-computer interface. The method uses an EEG brain-computer interface to obtain raw EEG data, estimates the subject's imagined limb movements through real-time analysis of the data, and helps the subject complete the imagined limb movements through traction using motion-assistive equipment.

[0005] Writing is a vital form of human communication, yet some diseases can render people with disabilities unable to write. Recent research on brain-computer interfaces (BCIs) offers great hope for restoring writing ability. In 2021, Willett's team pioneered a study on handwriting English using an invasive BCI, achieving remarkable results. In their experiment, a stroke patient who had lost the ability to write imagined writing English letters and achieved a spelling speed of 90 characters per minute with an accuracy rate of 98.78%. However, their work was based on classifying English letters, which have simple structures and few strokes. Their proposed decoding framework is difficult to directly apply to languages ​​like Chinese, which have complex structures and many strokes. When writing Chinese characters, the writing of a stroke and the breaking of a stroke are two distinct states. Due to neuronal plasticity, brain activity patterns change depending on the behavioral state. These changes can make the mapping function from neural signals to motor signals unstable, leading to inaccurate decoding results.

[0006] Summary of the Invention

[0007] The present invention provides a Chinese character writing decoding method for invasive brain-computer interfaces. Based on the two different neural representations of writing strokes and writing broken strokes during the writing process, a state discriminator is designed to decode neural data in different states separately, greatly reducing the instability of neural signals when performing different tasks and improving the robustness of the decoder.

[0008] A Chinese character writing decoding method for an invasive brain-computer interface comprises the following steps:

[0009] (1) Obtain the original motor nerve signal, filter it to obtain the neural signal of the specified frequency band, and further process it to obtain the ESA neural signal;

[0010] (2) Normalize the ESA neural signal, extract the motor neural signal of the data segment with a set window length, and construct a training data set;

[0011] (3) In the training data set, the motor nerve signals correspond to Chinese character writing motor signals. According to the Chinese character stroke motion signals and the Chinese character broken stroke motion signals of the Chinese character writing motion signals, the corresponding motor nerve signals are divided into writing stroke motor nerve signals and writing broken stroke motor nerve signals;

[0012] (4) Using the motor neural signals and the Chinese character writing motor signals as input, the following three models are trained: 1) Using the motor neural signals of writing strokes and the corresponding Chinese character stroke motor signals to train a predictor for decoding Chinese character strokes; 2) Using the motor neural signals of writing broken strokes and the corresponding Chinese character broken stroke motor signals to train a predictor for decoding Chinese character broken strokes; 3) Using the motor neural signals of writing strokes and the motor neural signals of writing broken strokes to train a state discriminator for writing strokes and broken strokes in the process of writing Chinese characters;

[0013] (5) During the application process, the motor neural signal to be decoded is input into the state discriminator, the neural signal of the Chinese character strokes discriminated is input into the predictor of the Chinese character strokes decoded, and the neural signal of the Chinese character broken strokes discriminated is input into the predictor of the Chinese character broken strokes decoded; the decoded stroke motion signal and the broken stroke motion signal are spliced ​​together to obtain a complete decoded Chinese character motion signal.

[0014] The specific process of step (1) is:

[0015] After being amplified and digitized by the Neuroport system, neural activity is recorded at a frequency of 30 kHz. The recorded neural signals are first filtered by a first-order Butterworth filter with a cutoff frequency set at 300 Hz to obtain neural signals above 300 Hz. They are then full-wave rectified and filtered again by a first-order Butterworth filter with a cutoff frequency set at 12 Hz. They are then downsampled at 1000 Hz to obtain ESA neural signals.

[0016] When collecting the Chinese character dataset in step (1), the Chinese character writing paradigm used was in MP4 format. The experimental process could be played in full screen, with a pure black background and green regular script in the center of the screen. The size of the regular script could be set to 600*600. The paradigm included four states: "Prepare", "Reaction", "Go", and "Delay". The "Go" state was the handwriting state. In this state, a right hand holding a chalk would trace the trajectory of the Chinese character, guiding the volunteers to imagine writing the Chinese character.

[0017] The data in step (2) are the ESA-type neural signals obtained after processing in step (1). Preferably, the data in the preparation and return phases of the paradigm can be removed, and the actual operation phase can be selected for analysis.

[0018] For standardization and smoothing operations, the z-score function and movmean function in Matlab can be used to smooth the neuronal activity of each channel of the neural signal. The specific smoothing window size can be selected according to actual needs.

[0019] In step (3), the corresponding motor nerve signals are divided into writing stroke motor nerve signals and writing pen break motor nerve signals. The specific process is as follows:

[0020] The Chinese character writing trajectory data (Chinese character writing motion signal) includes the speed data in the x-direction and y-direction, as well as the length of each stroke and broken stroke of each Chinese character, where the stroke state is represented by 1 and the broken stroke state is represented by 0; according to the stroke state and the broken stroke state, the corresponding motor nerve signal is divided into the writing stroke motor nerve signal and the writing broken stroke motor nerve signal, and the corresponding writing stroke motor nerve signal and the corresponding writing broken stroke motor nerve signal are spliced ​​together in sequence to obtain two arrays Spike s and Spike b ;

[0021] Among them, Spike s Spike is the motor nerve signal array corresponding to the writing strokes. b is the motor nerve signal array corresponding to the broken pen.

[0022] In step (4), the input format of the motor nerve signal is: Trial*Timebin*Electrode

[0023] Among them, Trial is the number of corresponding Chinese characters, Timebin is the time length of the neural signal corresponding to each Chinese character, and Electrode is the number of signal acquisition electrode channels.

[0024] The predictors for decoding Chinese character strokes and the predictors for decoding broken strokes both use Kalman filters; the state discriminators for writing strokes and broken strokes use HMM decoders and use the Viterbi algorithm to solve the state sequence.

[0025] The input of the stroke writing and broken stroke state discriminator is the complete neural signal during the Chinese character writing process, and the output is the state sequence of stroke writing and broken stroke writing. The specific format is [1111….0000….1111….], where 1 represents the stroke writing state and 0 represents the broken stroke writing state.

[0026] The specific process of step (5) is:

[0027] The motor nerve signal to be decoded is input into the state discriminator of writing strokes and writing broken strokes, and the length of each stroke or broken stroke is obtained according to the predicted writing stroke and broken stroke state sequence, thereby obtaining the writing stroke and broken stroke length sequence;

[0028] The corresponding motor nerve signals are input into a predictor for decoding Chinese character strokes and a predictor for decoding Chinese character broken strokes, respectively, to obtain a stroke prediction value and a broken stroke prediction value;

[0029] According to the written stroke and stroke length sequence, the stroke and stroke break prediction values ​​are spliced ​​as [stroke, stroke break, stroke, ..., stroke break, stroke] to obtain a complete decoded Chinese character motion speed signal. The motion speed signal is integrated to obtain the Chinese character motion position signal.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. Based on the two different neural representations of writing strokes and writing broken strokes during the writing process, the present invention designs a state discriminator to decode neural data in different states separately, greatly reducing the instability of neural signals when performing different tasks and improving the robustness of the decoder.

[0032] 2. Based on the traditional Kalman filter and HMM decoder, this paper proposes a Chinese character writing decoding method for invasive brain-computer interface, which reduces the impact of the instability of neural signals when performing different writing tasks to a certain extent. In the test, it is better than the single Kalman filter algorithm, proving the effectiveness of this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a flow chart of a Chinese character writing decoding method for an invasive brain-computer interface provided by an embodiment of the present invention;

[0034] FIG2 is a schematic diagram of a paradigm used in the method of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0036] This study used a single clinical dataset. All clinical and experimental procedures in this study were approved by the Medical Ethics Committee of the Second Affiliated Hospital of Zhejiang University (Ethics Review No. 2019-158). Informed consent was obtained verbally from the participants and their immediate family members and signed by their legal representatives. The volunteer was a 75-year-old male who was involved in a car accident four years ago and became quadriplegic after a C4 cervical spine trauma. The volunteer could only move the part above his neck and had normal language ability and comprehension of all tasks. For limb motor behavior, the patient's skeletal muscle strength score was 0 / 5, and he had completely lost the ability to control limb movements.

[0037] Two 96-channel Utah intracortical microelectrode arrays (4 mm × 4 mm, 1.4 mm long Utah arrays, Blackrock Microsystems, Salt Lake City, UT, USA) were implanted in the left primary motor cortex of the volunteers, one located in the central hand knot area and the other 2 mm away. Computed tomography and functional magnetic resonance imaging (fMRI) were used to guide implantation. During the implantation procedure, participants were asked to perform hand and elbow flexion and extension exercises, and functional MRI scans were used to confirm the activation area in the motor cortex. A robotic arm was used to assist in electrode placement during the surgery. Participants were given a week to recover before commencing neural signal recordings and BCI training tasks.

[0038] Participants underwent BCI training three days per week, lasting approximately three hours per day. These training sessions included preparing for signal recording, impedance testing, spike classification, and paradigm tasks. The experiment was terminated if a participant became fatigued or developed any physical abnormalities (e.g., fever or urinary tract infection). The entire experiment lasted six months, and the data used in this paper encompasses five days of training.

[0039] The Chinese character writing task was conducted on a computer monitor 1.5 meters in front of the volunteer. The task paradigm required the participant to imagine their right upper arm holding chalk and writing on a blackboard. The paradigm was in MP4 format and occupied the entire monitor screen, with the target character written in the center. The paradigm had a pure black background and the target character was in green regular script, which could be set to 600*600 pixels. The paradigm consisted of four states: "Prepare," "Reaction," "Go," and "Delay." The "Prepare" state lasted 1 second, during which the character to be written appeared on the screen, reminding the participant to prepare mentally. The "Reaction" state lasted 0.3 seconds, during which a ding sound played, reminding the participant that writing was about to begin. The "Go" state lasted 3-12 seconds, depending on the complexity of the character. In this state, a right hand holding chalk traced the character's trajectory, guiding the participant to imagine writing the character. The "Delay" phase lasted 0.8 seconds, during which the screen turned black, indicating the beginning of a new character. During the experiment, target words were displayed in a random order. Task parameters, including target word size, distance from the top of the screen, and maximum writing time, were configurable in the task settings. For each trial, participants were instructed to imagine exerting force on their right upper arm while holding chalk in their right hand and writing on a blackboard.

[0040] The present invention provides a Chinese character writing decoding method for invasive brain-computer interfaces. Based on an HMM decoder and a traditional Kalman filter, it determines the different states existing in the writing process and trains filters according to the different states of writing strokes and pen breaks. This reduces to a certain extent the impact of the instability of neural signals when performing different writing tasks, greatly improving the performance of the decoder.

[0041] A Chinese character writing decoding method for an invasive brain-computer interface comprises the following steps:

[0042] (1) Motor nerve signal preprocessing: The original motor nerve signal is obtained from the hardware device, filtered to obtain the specified frequency band of the data to be processed, and processed to obtain the ESA (Entire spiking activity) neural signal.

[0043] Specifically, this example uses the Neuroport system (NSP, Blackrock Microsystems) to record neural signals. After the neural activity is amplified and digitized, it is recorded at a frequency of 30KHz. The recorded neural signal first passes through a first-order Butterworth filter with a cutoff frequency of 300 to obtain a neural signal above 300Hz. It is then full-wave rectified and passed through a first-order Butterworth filter again with a cutoff frequency of 12Hz, and then downsampled at 1000Hz to obtain the ESA neural signal.

[0044] Specifically, in a general experiment, one experimental session collects 30 Chinese characters of different categories, and each Chinese character is written 3 times. The training set data is 29 Chinese characters * 3, and the test set data is 1 * 1.

[0045] Specifically, as shown in the schematic diagram of the paradigm employed in the method of the present invention in Figure 1, the task paradigm requires the participant to imagine holding a piece of chalk in their right arm and writing on a blackboard. The Chinese character writing paradigm used is in MP4 format and can be played full-screen during the experiment. The background is pure black, with green regular script in the center of the screen. The regular script size can be set to 600*600. The paradigm consists of four states: "Prepare," "Reaction," "Go," and "Delay." The "Prepare" state lasts for 1 second. During this state, the Chinese character to be written appears on the screen, reminding the participant to prepare mentally. The "Reaction" state lasts for 0.3 seconds. During this state, a ding sound plays, reminding the participant that writing is about to begin. The "Go" state lasts for 3-12 seconds, depending on the complexity of the character. In this state, a right hand holding chalk traces the character's trajectory, guiding the participant to imagine writing the character. The "Delay" phase lasts for 0.8 seconds, during which the screen turns black, indicating the beginning of a new Chinese character.

[0046] (2) Standardize the data, extract data segments with appropriate window lengths, obtain preprocessed neural data, and divide the data into training sets and test sets according to reasonable proportions, including:

[0047] Specifically, in addition to the data from the paradigm preparation and return stages, the actual operation stage was selected for analysis. In this example, the start time of the actual writing stage was 3100ms, and the end time of the writing stage was determined by the length of the Chinese character, which was between 8100-12100ms. That is, the length of the Chinese character in this example was 5s-9s.

[0048] Specifically, the format of the Chinese character writing motor nerve signal is as follows: Trial*Timebin*Electrode

[0049] Among them, Trial is the number of corresponding Chinese characters, Timebin is the length of the neural signal time corresponding to each Chinese character (ms), and Electrode is the number of signal acquisition electrode channels.

[0050] Specifically, in this example, 90 Chinese characters are collected every day, that is, Trial = 90. Timebin is the time required for the Chinese character with the longest writing time among the Chinese characters collected on that day. For example, if the Chinese character with the longest writing time requires 9s, then Timebin = 3100 + 9000 = 12100. In actual calculation, for Chinese characters with less writing time, the corresponding writing time is taken from the long array. For example, if a Chinese character requires 5s of writing time, then the time period from 3100 - 8100 in the array can be taken. In this example, Electrode = 192, that is, the neural signals corresponding to 192 electrodes are collected.

[0051] Specifically, for the standardization and smoothing operations, in this example, the z-score function in Matlab is used to smooth the neural activities of each channel of the neural signal. The specific smoothing window is set to 5 in this example.

[0052] Specifically, for the binned operation on the standardized neural data, a binned pattern with overlap is used, and the window size is set to 200ms and the step size is set to 50ms.

[0053] Specifically, for the division of data, the writing signals of only one type of Chinese character category can be used as the test set, and the remaining category of Chinese characters can be used as the training set. The "leave-one-out" method can be used for actual testing.

[0054] (3) According to the Chinese character movement data, the corresponding movement neural signals are divided into corresponding writing stroke movement neural signals and writing break stroke movement neural signals.

[0055] Specifically, the Chinese character movement signal includes the movement data in the x and y directions and the length of each stroke and break stroke of each Chinese character. For example, the total length of the movement signal of the Chinese character "大" is 80, and the stroke and break stroke sequence is [15, 13, 20, 14, 18], where the odd terms are the stroke lengths and the even terms are the break stroke lengths.

[0056] Specifically, the corresponding Chinese character writing movement neural signal is divided into corresponding stroke and break stroke neural signals according to the stroke and break stroke sequence, and the corresponding stroke movement neural signals and the corresponding break stroke movement neural signals are respectively spliced together in order to obtain two arrays Spike s and Spike b .

[0057] Among them, Spike s is the array of movement neural signals corresponding to the writing strokes, Spikeb is the motor nerve signal array corresponding to the broken pen.

[0058] (4) The training process takes the neural signals and the Chinese character writing trajectory signals as input to train the following three models: 1) Use the writing stroke movement neural signals and the corresponding Chinese character stroke movement signals to train a predictor for decoding Chinese character strokes; 2) Use the writing stroke movement neural signals and the corresponding Chinese character stroke movement signals to train a predictor for decoding Chinese character stroke breaks; 3) Use the writing stroke neural signals and the writing stroke break neural signals to train a state discriminator for writing strokes and writing stroke breaks during the Chinese character writing process.

[0059] 1) Using the handwriting stroke motor signals and the corresponding Chinese character stroke motor signals to train a predictor for decoding Chinese character strokes;

[0060] Preferably, a Kalman filter can be used as a filter for decoding Chinese character stroke motion.

[0061] Specifically, a Kalman filter is used to decode handwriting stroke motion based on the writing stroke motor neural signal, and the discharge pattern is defined as: k =H k x k +q k #(1)

[0062] in, The C neurons representing the strokes of the Chinese character observed in time Δt at time t k The discharge frequency at the moment, is a matrix that linearly relates the state of the hand to the firing of neurons.

[0063] Specifically, the hand movement state is equated with the Chinese character copying trajectory state, that is, Indicates that at time t k =kΔt time, the position in the x direction, the position in the y direction, the speed in the x direction, the speed in the y direction, the acceleration in the x direction, and the acceleration in the y direction of the Chinese character copying trajectory.

[0064] Specifically, it is assumed that the mean of the noise in the observations is 0 and conforms to the normal distribution, that is,

[0065] The state update equation is:

[0066] x k+1 =A k x k +W k #(2)

[0067] in, is the coefficient matrix, the noise term This formula shows that the motion state (speed) of the hand at time k+1 is linearly related to the state at time k. Assuming that these estimates are normally distributed and A can be learned from the data k and W k In fact, A k ,H k ,W k ,Q k They may change with the number of time steps k, but to simplify the calculation, they are considered constants. Therefore, their values ​​can be estimated from the training data using the least squares method.

[0068] Because A k ,H k ,W k ,Q k Regarding k being independent of each other, ignoring the subscripts, they are expressed as A, H, W, Q, and the coefficient matrices A and H are estimated using the least squares method:

[0069] where ||·|| is the regular L 2 paradigm.

[0070] Solving equations (3) and (4) yields: H=ZX T (XX T ) -1 #(6)

[0071] in:

[0072] Using the estimated A and H, we can further obtain W and Q: W=(X2-AX1)(X2-AX1) T / (M-1)#(9) Q=(Z-HX)(Z-HX) T / M #(10)

[0073] After obtaining the parameters A, H, W, and Q, the neuronal spike firing frequency and hand movement can be encoded according to equations (1) and (2).

[0074] To reconstruct hand movements based on hand neuron discharges, the steps are as follows:

[0075] For each x k , reconstruction using the Kalman filter consists of the following two steps:

[0076] i): (A priori step) predict x from the state of formula (2) k The predicted values ​​are marked as

[0077] ii): (Posteriori step) using t k Discharge frequency information update at the moment Update the value marked as

[0078] To evaluate the estimation ability, the a priori error and the posterior error are defined as follows:

[0079] Assumptions If both are unbiased estimates, then the a priori and posterior errors have the characteristics of a covariance matrix (in one dimension, the covariance matrix is ​​just the square of the Euclidean distance between the true value and the estimate). The covariance matrices of the a priori and posterior error estimates are defined as follows:

[0080] The posterior estimator is the final estimate of the state. To simplify the estimation process, we assume that the estimators are all linear. Therefore, we can express the posterior estimate as a linear combination of the prior estimate and the weighted difference between the actual measurement value and the measurement prediction value, as shown below:

[0081] Among them, K k is the gain matrix, which is defined as follows:

[0082] Where Q is the measurement error matrix.

[0083] Combining all the above formulas, the Kalman filter algorithm is used to reconstruct the hand's motion state of writing Chinese characters based on the given hand neuron discharge frequency:

[0084] I. Discrete Kalman filter time update equation:

[0085] In each t k time, from the original t k-1 Get the prior estimate at every moment and calculate the error covariance matrix:

[0086] II. Measurement value update formula:

[0087] Leveraging Valuation and discharge frequency z k Update the estimated value according to formula (11) and calculate the covariance matrix. This process can be described as:

[0088] At each moment, the Kalman filter is iteratively updated according to the above two steps to obtain the estimated value of the writing stroke motion.

[0089] 2) Using the written stroke-breaking motor signals and the corresponding Chinese character stroke-breaking motor signals to train a predictor for decoding Chinese character stroke-breaking;

[0090] Specifically, this step is the same as step 1), except that the writing stroke motion neural signal in step 1) is replaced with the writing stroke-breaking motion neural signal, and the corresponding Chinese character stroke motion signal is replaced with the corresponding Chinese character stroke-breaking motion signal.

[0091] 3) Using the neural signals of writing strokes and the neural signals of writing broken strokes to train a state discriminator for writing strokes and broken strokes in the process of writing Chinese characters;

[0092] Specifically, HMM is used as a state discriminator for writing strokes and broken strokes in the process of writing Chinese characters.

[0093] Specifically, the Gaussian mixture model (GMM) algorithm is used to solve the HMM algorithm emission probability matrix.

[0094] Specifically, the four states in the HMM algorithm are set as follows:

[0095] Hidden state is set to writing strokes (s) and writing breaks (b)

[0096] The observed state corresponds to the writing motor nerve signal.

[0097] The transition probability matrix is ​​a 2*2 square matrix:

[0098] Among them, a ss is the probability of the stroke state changing to the stroke state, a sb is the probability of the stroke state turning into the broken stroke state; a bs is the probability of the broken pen state turning into the stroke state, a bb The probability of the broken pen state turning into the broken pen state.

[0099] Specifically, the transition probability matrix is ​​set as:

[0100] Specifically, the emission probability is solved using the GMM model:

[0101] In the problem of writing Chinese characters, the classification of strokes and broken strokes is solved using the Gaussian mixture model (GMM), which defines the following information:

[0102] x j Represents the jth observation data, j=1,2,…,N

[0103] K represents the number of sub-Gaussian models in the mixture model. In this problem, K=2

[0104] α kis the probability that the observed data belongs to the kth sub-model,

[0105] is the Gaussian distribution function of the k-th sub-model, The expanded form is as follows:

[0106] γ jk It represents the probability that the j-th observation data belongs to the k-th sub-model.

[0107] The probability distribution of the Gaussian mixture model is:

[0108] Among them, the parameters Represents the expectation, variance, and probability of each sub-model occurring in the mixed model.

[0109] Among them, the observation data x j is the motor nerve signal corresponding to writing Chinese characters. In the two Gaussian distribution sub-models, μ is the mean motor nerve signal corresponding to the writing stroke and the mean motor nerve signal corresponding to the writing stroke interruption. The solution process of μ is as follows:

[0110] The corresponding Chinese character writing motor neural signal is divided into corresponding stroke and stroke-breaking neural signals according to the stroke and stroke-breaking sequence, and the corresponding stroke motor neural signal and the corresponding stroke-breaking motor neural signal are spliced ​​together to obtain two arrays Spike s and Spike b .

[0111] Among them, Spike s Spike is the motor nerve signal array corresponding to the writing strokes. b The array of motor nerve signals corresponding to the broken pen is in two-dimensional format: bin*Electrode

[0112] Specifically, bin is the time length of the neural signal corresponding to all strokes or pen breaks in this experiment (50ms), and Electrode is the number of signal acquisition electrode channels, which is 192 in this case.

[0113] Specifically, the Num and Timebin dimensions are averaged to obtain the Spike in 1*Electrode format. ms and Spike mb The motor nerve signal array is used as the mean μ of the two Gaussian sub-models.

[0114] Specifically, for the above Gaussian mixture model, the EM expectation maximization method is used to solve:

[0115] E-step: Based on the current parameters, calculate the probability that each observation data j comes from sub-model k:

[0116] M-step: Calculate the model parameters for the next iteration:

[0117] Iterate the E-step and M-step until convergence conditions are reached.

[0118] At this point, the emission probability matrix of the HMM process is obtained:

[0119] Among them, b 11 is the probability that the observation point is a written stroke, b 21 is the probability of writing a broken pen at the observation point.

[0120] Specifically, the Viterbi algorithm can be used to solve the writing stroke and writing break state sequence corresponding to the motor nerve signal:

[0121] Input: Model λ=(A,B,∏) and observation sequence O=(o1,o2,…,o T ).

[0122] Where A and B are the probability matrix and transition matrix for writing and breaking strokes, respectively. ∏ is the initial state value, which is set to |1 0|. By default, the first stroke is a writing stroke.

[0123] Solution: Writing motor nerve signals corresponding to writing strokes and pen break state sequences

[0124] 1) Initialize the local state: δ1(i) = ∏ i b i (o1),i=1,2,…,N#(28)

[0125] 2) Perform dynamic programming to recursively calculate the local state at time t = 2, 3, ..., T:

[0126] 3) Calculate the maximum δ at time T T (i) is the probability of the most likely hidden state sequence occurring at time T

[0127] Calculate the maximum time T That is the most likely hidden state sequence at time T

[0128] 4) Exploiting local state Start backtracking, for t=T-1,T-2,…,1:

[0129] Finally, the most likely stroke sequence is obtained in

[0130] (5) The test process inputs neural signals to predict the writing trajectory of Chinese characters, which specifically includes four steps: 1) inputting the test set neural data into the state discriminator of writing strokes and writing broken strokes; 2) inputting the neural signals of writing Chinese character strokes identified by the state discriminator of writing strokes and writing broken strokes into the predictor of decoding Chinese character strokes; 3) inputting the neural signals of writing Chinese character broken strokes identified by the state discriminator of writing strokes and writing broken strokes into the predictor of decoding Chinese character broken strokes; 4) splicing the decoded stroke motion signal and the broken stroke motion signal together to obtain a complete decoded Chinese character motion signal.

[0131] 1) Input the test set neural data into the state discriminator of writing strokes and writing broken strokes;

[0132] Specifically, the test set data is input into the HMM state discriminator to obtain the stroke state and the stroke break state sequence [1110000…..001001] in the process of writing Chinese characters.

[0133] Where 1 represents a written stroke and 0 represents a broken stroke. The length of each stroke or broken stroke is obtained according to the predicted written stroke and broken stroke state sequence, and the written stroke and broken stroke length sequence is obtained.

[0134] 2) Inputting the neural signal of the written Chinese character strokes identified by the stroke-writing and broken-stroke-writing state discriminator into the predictor for decoding the Chinese character strokes;

[0135] Specifically, the motor nerve signal and motion signal corresponding to the writing stroke state are obtained according to the writing stroke state sequence obtained in step 1), and are input into the filter for decoding the Chinese character strokes.

[0136] 3) inputting the neural signal of the broken stroke of the Chinese character identified by the state discriminator of writing strokes and broken strokes into the predictor of the broken stroke of the Chinese character decoding;

[0137] Specifically, the motor nerve signal and the motion signal corresponding to the broken stroke state are obtained according to the writing stroke state sequence obtained in step 1), and are input into the filter for decoding the broken stroke of Chinese characters.

[0138] 4) The decoded stroke motion signal and the broken stroke motion signal are spliced ​​together to obtain a complete decoded Chinese character motion signal.

[0139] Specifically, the length of each stroke or stroke break is obtained based on the written stroke and stroke break state sequence predicted in step 1), thereby obtaining a written stroke and stroke break length sequence. The corresponding motor nerve signal is input into a corresponding filter to obtain a stroke prediction output and a stroke break prediction output. According to the written stroke and stroke break length sequence, the stroke and stroke break prediction values ​​are concatenated in the order of [stroke, stroke break, stroke, ..., stroke break, stroke] to obtain a complete decoded Chinese character motion velocity signal. The motion velocity signal is then integrated to obtain a Chinese character motion position signal.

[0140] (6) Performance evaluation of the Chinese character writing decoding method for invasive brain-computer interfaces: This method is compared with other methods in the test set data to evaluate the performance and effectiveness of this method.

[0141] This example uses a clinical handwritten Chinese character dataset, which consists of data from nine sessions over five days. Each session contains 90 trials, including 30 Chinese characters, each handwritten three times. Figure 2 shows the handwriting paradigm used during experimental acquisition. This task requires the participant to imagine holding a piece of chalk on a blackboard with their upper right arm. The paradigm is in MP4 format and occupies the entire monitor screen, with the target character written in the center. The paradigm has a pure black background and the target character is in green regular script. The regular script size can be set to 600*600. The paradigm consists of four states: "Prepare," "Reaction," "Go," and "Delay." The "Prepare" state lasts for 1 second. During this state, the character to be handwritten appears on the screen, reminding the participant to prepare mentally. The "Reaction" state lasts for 0.3 seconds. During this state, a ding sound plays, reminding the participant that writing is about to begin. The "Go" state lasts 3-12 seconds, depending on the complexity of the character. During this state, a right hand holding a piece of chalk traces the character's trajectory, guiding the volunteer to imagine writing the character. The "Delay" phase lasts 0.8 seconds, during which the screen turns black, indicating the beginning of a new character.

[0142] Table 1 shows a comparison of the present invention's invasive brain-computer interface-based Chinese character decoding method and a standard Kalman filter decoding algorithm on a clinical handwritten Chinese character dataset. The primary evaluation metric is the correlation coefficient between the decoded trajectory and the actual trajectory. As shown in the table, the average CC of Chinese character decoding performance for each experimental session is shown. This method outperforms the standard Kalman filter decoding algorithm, demonstrating its effectiveness.

[0143] Table 1

[0144] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Chinese character writing decoding method for invasive brain-computer interface, characterized in that: The following steps are involved: (1) Obtaining the original motor nerve signal, filtering to obtain the nerve signal of the specified frequency band, and further processing to obtain the ESA nerve signal; (2) Standardize the ESA neural signal, extract the motor neural signal of the data segment with a set window length, and construct a training data set; (3) In the training data set, the motor nerve signal corresponds to a Chinese character writing motor signal, and according to the Chinese character stroke motion signal and the Chinese character broken stroke motion signal of the Chinese character writing motion signal, the corresponding motor nerve signal is divided into a writing stroke motor nerve signal and a writing broken stroke motor nerve signal; (4) Using the motor neural signals and the Chinese character writing motor signals as input, the following three models are trained: 1) Using the motor neural signals of writing strokes and the corresponding Chinese character stroke motion signals to train a predictor for decoding Chinese character strokes; 2) Using the motor neural signals of writing broken strokes and the corresponding Chinese character broken stroke motion signals to train a predictor for decoding Chinese character broken strokes; 3) Using the motor neural signals of writing strokes and the motor neural signals of writing broken strokes to train a state discriminator for writing strokes and broken strokes in the process of writing Chinese characters; (5) During the application process, the motor nerve signal to be decoded is input into the state discriminator, the discriminated motor nerve signal of the written Chinese character stroke is input into the predictor of the Chinese character stroke decoding, and the discriminated motor nerve signal of the broken stroke of the written Chinese character is input into the predictor of the broken stroke of the Chinese character decoding; the decoded stroke motion signal and the broken stroke motion signal are spliced ​​together to obtain a complete decoded Chinese character motion signal.

2. The Chinese character writing decoding method for invasive brain-computer interface according to claim 1 is characterized in that: The specific process of step (1) is as follows: After the neural activity is amplified and digitized by the Neuroport system, it is recorded at a frequency of 30KHz. The recorded neural signal first passes through a first-order Butterworth filter with a cutoff frequency set at 300Hz to obtain neural signals above 300Hz. It is then full-wave rectified and again passed through the The ESA neural signal was obtained by passing it through a 1st-order Butterworth filter with the cutoff frequency set to 12 Hz and then down-sampling it at 1000 Hz.

3. The Chinese character writing decoding method for invasive brain-computer interface according to claim 1 is characterized in that: In step (3), the corresponding motor nerve signal is divided into a writing stroke motor nerve signal and a writing pen break motor nerve signal. The specific process is as follows: The Chinese character writing motion signal includes speed data in the x-direction and y-direction, as well as the length of each stroke and broken stroke of each Chinese character, where the stroke state is represented by 1 and the broken stroke state is represented by 0; according to the stroke state and the broken stroke state, the corresponding motor nerve signal is divided into a writing stroke motor nerve signal and a writing broken stroke motor nerve signal, and the corresponding writing stroke motor nerve signal and the corresponding writing broken stroke motor nerve signal are spliced ​​together in sequence to obtain two arrays Spike s and Spike b ; Among them, Spike s Spike is an array of motor nerve signals corresponding to writing strokes. b It is the motor nerve signal array corresponding to the broken pen.

4. The Chinese character writing decoding method for invasive brain-computer interface according to claim 1 is characterized in that: In step (4), the input format of the motor nerve signal is: Trial*Timebin*Electrode Among them, Trial is the number of corresponding Chinese characters, Timebin is the time length of the neural signal corresponding to each Chinese character, and Electrode is the number of signal acquisition electrode channels.

5. The Chinese character writing decoding method for invasive brain-computer interface according to claim 1 is characterized in that: In step (4), the predictor for decoding Chinese character strokes and the predictor for decoding Chinese character broken strokes both use Kalman filters.

6. The Chinese character writing decoding method for invasive brain-computer interface according to claim 1 is characterized in that: In step (4), the state discriminator for writing strokes and writing broken strokes adopts an HMM decoder and uses the Viterbi algorithm to solve the state sequence.

7. The Chinese character writing decoding method for invasive brain-computer interface according to claim 1 is characterized in that: In step (4), the input of the stroke writing and broken stroke state discriminator is the complete motor nerve signal in the process of Chinese character writing, and the output is the stroke writing and broken stroke state sequence, the specific format is [1111….0000….1111….], where 1 represents the stroke writing state and 0 represents the broken stroke state.

8. The Chinese character writing decoding method for invasive brain-computer interface according to claim 1 is characterized in that: The specific process of step (5) is as follows: The motor nerve signal to be decoded is input into the state discriminator of writing strokes and broken strokes, and the length of each stroke or broken stroke is obtained according to the predicted writing stroke and broken stroke state sequence, so as to obtain the writing stroke and broken stroke length sequence; The corresponding motor nerve signals are respectively input into a predictor for decoding Chinese character strokes and a predictor for decoding Chinese character broken strokes to obtain a stroke prediction value and a broken stroke prediction value; According to the written stroke and stroke length sequence, the stroke and stroke break prediction values ​​are spliced ​​as [stroke, stroke break, stroke, ..., stroke break, stroke] to obtain a complete decoded Chinese character motion speed signal, and the motion speed signal is integrated to obtain a Chinese character motion position signal.

Images