Brain-computer interface decoding method based on multi-dimensional label data

Abstract

The invention discloses a brain-computer interface decoding method based on multi-dimensional label data, which comprises the following steps of: obtaining a neural signal when a patient imaginates to execute a specified task and average data of multi-dimensional motion data when a plurality of healthy testees actually execute a first task, aligning the average data with the neural signal in time, taking the average data after time alignment as labels of neural signals, namely multi-dimensional labels, and constructing a standard library by a plurality of multi-dimensional labels; training a decoder based on the neural signal and the label so as to obtain a mapping relation between the neural signal and the multi-dimensional label; and inputting a neural signal of a patient into the trained decoder to obtain predicted multi-dimensional motion data, searching a multi-dimensional label most similar to the predicted multi-dimensional motion data from the standard library, and executing a task corresponding to the searched multi-dimensional label. According to the method, the decoder is trained by using abundant multi-dimensional labels, so that the decoding accuracy of the trained decoder is obviously improved, and a patient is prevented from being interfered by external information.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer interfaces, and particularly relates to a brain-computer interface decoding method based on multi-dimensional tag data. Background Art

[0002] Brain-computer interface technology, as a cutting-edge technology, has the ability to directly decode the neural signals of patients with motor injuries, bringing hope for patients to restore their interaction ability with the outside world. Its principle is to capture the neural signals emitted by the patient's brain and convert them into instructions that can be recognized by the device, thereby realizing the control of external devices. However, in the actual application of clinical brain-computer interfaces, severe challenges are faced. Due to the impaired motor function of patients, a large amount of motion data is missing, resulting in a lack of effective motion tag data to train an efficient decoder. This greatly limits the popularization and effect of brain-computer interfaces in clinical treatment.

[0003] To break through this dilemma, numerous studies have actively explored innovative methods. Among them, external guidance means such as videos have become the key direction to solve the problem. Specifically, researchers use videos to show specific experimental tasks to patients with motor injuries, guide the patients to imagine performing corresponding actions, and synchronously record the neural signals generated by the patients during this process. Subsequently, the recorded neural signals are accurately aligned with the guidance information in the time dimension, and then decoding work is carried out, ultimately realizing the effective conversion from neural signals to motion control output.

[0004] Chinese Patent Application with Publication No. CN116880691A discloses a brain-computer interface interaction method based on handwriting trajectory decoding. This method guides a paralyzed subject to imagine writing specific Chinese characters under the guidance of a video and synchronously records the neural signals of the paralyzed subject during this process. By constructing a non-linear decoding model, this method decodes the neural signals into the writing trajectory of Chinese characters, and then identifies them as standard Chinese characters through methods such as optical character recognition, realizing a brain-computer interface system from neural signals to text output, providing a convenient communication means for aphasic and disabled patients.

[0005] Chinese Patent Application with Publication No. CN118819292A discloses a brain-computer interface paradigm and decoding method based on rhythm imagination, including: guiding the limbs in the guiding video to move rhythmically at a fixed frequency, and the subject imagines the rhythm guided by the observed movement. This method uses the task discriminant component analysis method to decode the side of the hand performing the task and the movement frequency using electroencephalogram signals, and non-invasively and efficiently realizes the movement of the exoskeleton at a specific frequency, and is expected to be popularized to actual applications and clinical scenarios.

[0006] However, the method of guiding the subjects to complete the imagination experiment through external information disclosed above has certain defects. First, the dimension of the external information for guidance is limited, resulting in insufficient decoded information volume and affecting the decoding accuracy. For example, the video used to guide the subjects to imagine handwriting only contains the trajectories of characters on a two-dimensional plane, lacking higher-dimensional motion information related to handwriting, which may be an important reason affecting the accuracy of character decoding and recognition. Second, in the imagination experiment, the subjects may overly rely on external guiding information, deviating from the natural state of actually performing the task and affecting the quality of neural signals.

[0007] Therefore, a method for solving the missing label data is needed to replace the external information for guiding the subjects to conduct the imagination experiment and avoid the interference of external information on the patients. Summary of the Invention

[0008] The present invention provides a brain-computer interface decoding method based on multi-dimensional label data. This method uses rich multi-dimensional labels to train the decoder, significantly improving the decoding accuracy of the trained decoder and avoiding the interference of external information on the patients.

[0009] The present invention provides a brain-computer interface decoding method based on multi-dimensional label data, including:

[0010] Obtain the neural signals when the patient imagines performing the first task, and the average data of the multi-dimensional motion data when multiple healthy subjects actually perform the first task. Align the average data with the neural signals in time, and use the time-aligned average data as the label of the neural signals, that is, the multi-dimensional label. Multiple multi-dimensional labels construct a standard library;

[0011] Train the decoder based on the neural signals and labels, and obtain the mapping relationship between the neural signals and multi-dimensional labels through the trained decoder;

[0012] During application, input the neural signals of the patient into the trained decoder to obtain the predicted multi-dimensional motion data, find the multi-dimensional label most similar to the predicted multi-dimensional motion data from the standard library, and execute the task corresponding to the found multi-dimensional label.

[0013] Preferably, when the specified task is writing characters, the multi-dimensional motion data when the healthy subject performs the specified task includes three-dimensional pen tip speed, the grip force on the pen, and the pressure of the pen tip on the paper.

[0014] Preferably, when the healthy subject performs the writing task, record the spatial trajectories of multiple marked points on the pen through an optical motion capture system and deduce the pen tip speed, and record the grip force on the pen and the pressure of the pen tip on the paper through a thin film pressure sensor.

[0015] Preferably, the method for obtaining the multi-dimensional label most similar to the predicted multi-dimensional motion data from the standard library includes:

[0016] Match the predicted multi-dimensional motion data with each multi-dimensional label in the label library by the correlation coefficient matching method or the dynamic time warping matching method, and use the multi-dimensional label with the highest matching degree as the most similar multi-dimensional label.

[0017] Preferably, matching the predicted multi-dimensional motion data with each label in the standard library by the dynamic time warping matching method to obtain the label with the highest matching degree includes: calculating the alignment path length between the predicted multi-dimensional motion data and each multi-dimensional label, and using the multi-dimensional label corresponding to the shortest path as the multi-dimensional label with the highest matching degree.

[0018] Preferably, perform upsampling on each dimension of the multi-dimensional motion data first, and then perform downsampling, so that the multi-dimensional motion data is time-aligned with the neural signal.

[0019] Preferably, train a decoder based on the neural signal and the label, and the decoder includes a linear decoder such as a Kalman filter or a nonlinear decoder such as an artificial neural network.

[0020] Preferably, the multi-dimensional motion data includes speed, acceleration, force, and angle data.

[0021] Preferably, the patient imagines performing a specified task based on visual or auditory cues, and the specified task includes limb movement, object manipulation, and language expression.

[0022] Preferably, record the neural signal of the patient imagining performing the specified task through a microelectrode array.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] The present invention solves the problem of lack of label data in the prior art by converting the multi-dimensional motion data obtained when a healthy subject performs a specified task into a multi-dimensional label. Moreover, the label provided by the present invention is obtained from a healthy subject and has a higher dimension. The decoder obtained after training has better decoding accuracy, that is, it can decode the neural signal into a more dimensional output completely, and is no longer limited to the dimension of external cue information, which helps to improve the decoding and recognition accuracy of the imagination task in the field of brain-computer interface. Description of the Drawings

[0025] Figure 1 It is a flowchart of the brain-computer interface decoding method based on multi-dimensional label data provided by a specific embodiment of the present invention;

[0026] Figure 2Schematic diagram of handwritten motor imagery and neural signal recording provided by an embodiment of the present invention;

[0027] Figure 3 Schematic diagram of multi-dimensional handwritten data recording provided by an embodiment of the present invention;

[0028] Figure 4 Schematic diagram of the decoding result of label data provided by an embodiment of the present invention;

[0029] Figure 5 Schematic diagram of the decoding result being recognized as a standard Chinese character and output provided by an embodiment of the present invention. Detailed implementation manners

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0031] In a specific embodiment of the present invention, the multi-dimensional label data is used as the decoding target, and a mapping relationship is directly established between the neural signal and the label data. This can help the subject generate neural signals of imagined movements based on their own intentions and cognitive processes without external information interference, so as to decode them into corresponding experimental tasks more accurately and efficiently, and achieve more natural and precise interaction with the outside world.

[0032] A specific embodiment of the present invention provides a brain-computer interface decoding method based on multi-dimensional label data, as Figure 1 shown, including:

[0033] When a patient with a motor injury imagines performing a first task in a specific embodiment of the present invention, neural signals are recorded through a microelectrode array. At the same time, multiple healthy subjects perform the same first task, and multi-dimensional motion data of the healthy subjects is recorded. The average data is obtained by averaging the recorded multi-dimensional motion data. The average data is aligned with the time of the neural signal, and the time-aligned average data is used as the multi-dimensional label. A standard library is constructed with multiple multi-dimensional labels. By converting the multi-dimensional motion data of healthy subjects into multi-dimensional labels in a specific embodiment of the present invention, the dimension of the labels is enriched, and the problem of insufficient labels in the prior art is solved.

[0034] In a specific embodiment, the multi-dimensional motion data provided by a specific embodiment of the present invention includes data such as speed, acceleration, force, and angle.

[0035] In a specific embodiment, each dimension of the multi-dimensional motion data is first upsampled to improve the resolution, and then downsampled so that the multi-dimensional motion data is time-aligned with the neural signal.

[0036] In a specific embodiment, the patient provided in this embodiment imagines performing a specified task based on visual or auditory cues, and the specified task includes limb movement, object manipulation, and language expression.

[0037] In a specific embodiment of the present invention, a decoder is trained through the constructed multi-dimensional tags and corresponding neural signals. A mapping relationship between the neural signals and the multi-dimensional tags is constructed through the trained decoder. Since a suitable amount of multi-dimensional tags are provided in the specific embodiment of the present invention, it is beneficial to the training of the decoder, thereby improving the decoding accuracy of the decoder.

[0038] During the decoding process, the new neural signals of the patient are input into the trained decoder to obtain a decoding result, that is, predicted multi-dimensional motion data. Then, the multi-dimensional tag most similar to the decoding result is identified in the standard library, and the task corresponding to the identified multi-dimensional tag is executed to obtain an action recognition result.

[0039] In a specific embodiment, this embodiment decodes and recognizes the Chinese characters imagined by a paralyzed subject as the output of standard Chinese characters. During the process of the paralyzed subject imagining writing Chinese characters, two microelectrode arrays are implanted into their brain to synchronously record neural signals. Then, multi-dimensional data such as the three-dimensional nib speed, the grip force on the pen, and the pressure of the nib on the paper when multiple healthy subjects write the same Chinese characters are recorded. After aligning with the neural signals of the paralyzed subject, they are averaged among different healthy subjects as label data. A decoder is trained to decode the neural signals of the paralyzed subject into label data, and the decoding result is compared with the label data of each Chinese character. The decoding result is recognized as a standard Chinese character through a handwritten character recognition algorithm.

[0040] When the specified task is writing Chinese characters, the brain-computer interface decoding method based on multi-dimensional label data provided in this embodiment includes the following steps:

[0041] Step 1, a patient with a motor injury imagines performing an experimental task: The patient with a motor injury imagines performing an experimental task of a specific action according to the cue and synchronously records their neural signals. In the embodiment of the present invention, a writing video of Chinese characters is played on the screen in front of the paralyzed subject, and the paralyzed subject imagines writing the Chinese character in stroke order according to the guidance of the video. As Figure 2 shown, during this process, two microelectrode arrays implanted into the brain of the paralyzed subject are used to synchronously record their neural signals. The characters imagined by the paralyzed subject writing by hand can be any form of characters such as Chinese characters or English letters.

[0042] Step 2, healthy subjects actually perform the same task and obtain labeled data: Healthy subjects actually perform the experimental tasks imagined by patients with sports injuries, synchronously record multi-dimensional motion data during this process, and use it as the labeled data for the imagination task after synchronizing with the neural signals of patients with sports injuries. In the embodiment of the present invention, multiple healthy subjects are asked to actually write the Chinese characters imagined by the paralyzed subjects, and record their three-dimensional pen tip speed, grip force on the pen, and pressure of the pen tip on the paper when writing, as Figure 3 shown. Among them, the pen tip speed is recorded by an optical motion capture system, and the pen tip speed is derived by recording the spatial trajectories of multiple marker points on the pen; the grip force on the pen and the pressure of the pen tip on the paper are both recorded by a thin film pressure sensor. The data of different dimensions of each Chinese character are synchronized with the neural signals through timestamps, and then the data of each dimension when multiple subjects write the same Chinese character are averaged as the labeled data, and a standard character library is established with the labeled data of multiple Chinese characters.

[0043] Step 3, decoding labeled data from neural signals: The decoding model includes linear decoders such as Kalman filters, non-linear decoders such as artificial neural networks, etc., which can be trained to achieve the mapping from neural signals to labeled data. In the embodiment of the present invention, the decoding model is a long short-term memory neural network. The trained decoding model can decode the neural signals of paralyzed subjects imagining writing other Chinese characters into corresponding multi-dimensional handwritten data.

[0044] Figure 4 shows the multi-dimensional labeled data of a Chinese character ("Pin") and the corresponding decoding results in the embodiment of the present invention. The left sub-graph is the spatial trajectory obtained by integrating the three-dimensional speed of the labeled data of this Chinese character, the middle sub-graph is the labeled data of each dimension of this Chinese character and the corresponding decoding results, and the right sub-graph is the Chinese character trajectory obtained by integrating the decoding results of the three-dimensional speed, which can be visually recognized by the naked eye as which Chinese character.

[0045] Step 4, recognition of motion tasks: The decoding results need to be matched with the labeled data in the standard library, so as to recognize the decoding results as action outputs. The matching methods include the correlation coefficient matching method and the dynamic time warping (DTW) method, etc. Figure 5 shows a schematic diagram of recognizing the decoding result of a Chinese character ("Chuan") as a standard Chinese character in the embodiment of the present invention. The recognition method adopted in the embodiment of the present invention is the dynamic time warping algorithm. Specifically, the alignment path length between the decoding result and the labeled data of each Chinese character in the standard character library is calculated, and the Chinese character corresponding to the shortest path is used as the final recognition result, so as to obtain the output of the standard Chinese character.

[0046] Based on the above steps, the embodiments of the present invention utilize multi-dimensional handwritten movement data of healthy subjects as labeled data to achieve the decoding and recognition of imagined handwritten movements. Moreover, the decoding method for missing labeled data proposed by the present invention can be extended to brain-computer interface experiments for various imagined tasks, breaking the limitations of missing labeled data in the traditional paradigm, more fully mining the information in neural signals, helping to improve the decoding and recognition accuracy of imagined tasks, and promoting the improvement of the performance of clinical brain-computer interface systems.

[0047] The above-described embodiments have elaborated in detail on the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, supplements, and equivalent replacements made within the scope of the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A brain-computer interface decoding method based on multi-dimensional label data, characterized in that: include: Obtaining neural signals when the patient imagines performing the first task, and average data of multi-dimensional motion data when multiple healthy subjects actually perform the first task, aligning the average data with the neural signal in time, and using the average data after time alignment as a label of the neural signal, i.e., a multi-dimensional label, and constructing a standard library with multiple multi-dimensional labels; Training a decoder based on the neural signal and the label, and obtaining a mapping relationship between the neural signal and the multi-dimensional label through the trained decoder; When applied, the patient's neural signal is input into the trained decoder to obtain predicted multi-dimensional motion data, a multi-dimensional label most similar to the predicted multi-dimensional motion data is found from the standard library, and the task corresponding to the found multi-dimensional label is executed.

2. The brain-computer interface decoding method based on multi-dimensional label data according to claim 1, characterized in that: When the designated task is writing characters, the multi-dimensional motion data of the healthy subjects when performing the designated task include three-dimensional pen tip speed, pen grip strength and pen tip pressure on paper.

3. The brain-computer interface decoding method based on multi-dimensional label data according to claim 2 is characterized in that: When healthy subjects performed a writing task, the spatial trajectories of multiple marking points on the pen were recorded by an optical motion capture system and the pen tip velocity was derived, and the pen grip force and the pressure of the pen tip on the paper were recorded by a thin film pressure sensor.

4. The brain-computer interface decoding method based on multi-dimensional label data according to claim 1, characterized in that: The method of obtaining a multi-dimensional label most similar to the predicted multi-dimensional motion data from a standard library comprises: The predicted multi-dimensional motion data is matched with each multi-dimensional label in the label library through the correlation coefficient matching method or the dynamic time warping matching method, and the multi-dimensional label with the highest matching degree is taken as the most similar multi-dimensional label.

5. The brain-computer interface decoding method based on multi-dimensional label data according to claim 4 is characterized in that: The predicted multidimensional motion data is matched with each label in the standard library through the dynamic time warping matching method to obtain the label with the highest matching degree, including: calculating the alignment path length of the predicted multidimensional motion data and each multidimensional label, and taking the multidimensional label corresponding to the shortest path as the multidimensional label with the highest matching degree.

6. The brain-computer interface decoding method based on multi-dimensional label data according to claim 1, characterized in that: Each dimension of the multi-dimensional motion data is first up-sampled and then down-sampled, so that the multi-dimensional motion data is time-aligned with the neural signal.

7. The brain-computer interface decoding method based on multi-dimensional label data according to claim 1, characterized in that: A decoder is trained based on the neural signal and the label, wherein the decoder includes a linear decoder such as a Kalman filter or a non-linear decoder such as an artificial neural network.

8. The brain-computer interface decoding method based on multi-dimensional label data according to claim 1, characterized in that: The multi-dimensional motion data includes velocity, acceleration, force and angle data.

9. The brain-computer interface decoding method based on multi-dimensional label data according to claim 1, characterized in that: Patients imagine performing designated tasks based on visual or auditory cues, which include limb movements, object manipulation, and language expression.

10. The brain-computer interface decoding method based on multi-dimensional label data according to claim 1, characterized in that: Neural signals were recorded using microelectrode arrays while the patient imagined performing a specified task.

Images