A method for issuing continuous motion trajectory control instructions for brain-computer interface based on magnetic brain signals

By using the magnetic brain signal and motion trajectory prediction model, the problem of low decoding accuracy of scalp EEG signals is solved, high-precision prediction and control of three-dimensional motion trajectory is achieved, and the decoding capability of the MI-BCI system is improved.

CN118568420BActive Publication Date: 2024-11-08BEIJING UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411037108.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-08
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The existing motion trajectory decoding accuracy based on scalp EEG signals is low, which limits the decoding ability of the MI-BCI system to three-dimensional degree of freedom motion trajectory tasks.

Method used

Using a method based on brain magnetic signal, the accuracy of motion trajectory prediction is improved by obtaining the brain magnetic signal of the user during motion imagination, using the motion trajectory prediction model, and controlling the multi-degree of freedom robotic arm for three-dimensional motion.

Benefits of technology

Through high spatial resolution decoding of the brain magnetic signal, accurate prediction and control of the motion trajectory is achieved, and the decoding accuracy and multi-degree-of-free movement of the MI-BCI system are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118568420B_ABST
    Figure CN118568420B_ABST
Patent Text Reader

Abstract

The present invention provides a method for issuing continuous motion trajectory control instructions of a brain-computer interface based on brain magnetic signals, which relates to the field of surgical technology, including: obtaining brain magnetic signals of a user during motor imagination; calling a motion trajectory prediction model to perform trajectory prediction based on brain magnetic signals to obtain target motion trajectory prediction data; controlling a multi-degree-of-freedom manipulator to perform three-dimensional motion based on the target motion trajectory prediction data; wherein the motion trajectory prediction model is obtained based on brain magnetic signal sample training, the brain magnetic signal sample is generated when the subject performs motor imagination according to the motion trajectory video, and the motion trajectory video is generated according to the three-dimensional motion trajectory data sample generated when the subject's hand moves. The present invention uses brain magnetic signals with higher spatial resolution to efficiently decode motor imagination tasks, thereby achieving accurate prediction of motion trajectories, and then constructing continuous motion trajectory control instructions of a brain-computer interface based on brain magnetic signals to control the manipulator to perform precise movements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of surgical technology, and in particular to a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals. Background Art

[0002] Brain-Computer Interface (BCI) collects and analyzes brain activity, allowing the brain and computer to communicate directly without relying on traditional neuromuscular pathways. It has broad application prospects in rehabilitation medicine, assistive technology, daily entertainment and other fields. At present, brain-computer interface systems can be divided into event-related potential brain-computer interface systems (ERP-BCI), steady-state visual evoked potential (SSVEP) brain-computer interface systems (SSVEP-BCI), and motor imagery (MI) brain-computer interface systems (MI-BCI) based on the main paradigm. Among them, MI-BCI is an active BCI system that can collect physiological signals generated by the brain imagining a certain limb movement without actual movement, providing a relatively natural experience and reducing the fatigue of the subjects. The MI-BCI system can decode the user's movement intention and convert it into a control signal for the external device, which can be directly related to the recovery of motor function, reflecting the user's ability of autonomous movement consciousness, and can promote the effective recovery of damaged motor function areas in the brain. In addition, MI-BCI can provide a more natural way of control because it simulates the neural process of natural movement and can achieve multi-degree-of-freedom control, that is, users can complete multiple different actions or functions by imagining different limb movements. Therefore, MI-BCI technology is particularly suitable for people who have lost their limb movement ability due to disease or injury. It can be applied to the field of controlling assistive devices for people with disabilities, such as prostheses, wheelchairs, etc., or to help stroke or brain injury patients recover their motor functions through rehabilitation training through mind-controlled games and virtual reality environments.

[0003] At present, scalp electroencephalography (EEG) has the advantages of high security, low cost and high real-time, and is a commonly used signal form in MI-BCI systems. After decades of development, multiple EEG-based MI paradigms have been successfully developed, and gradually aim to achieve more refined limb task classification, that is, MI tasks of different joints of the same limb or different degrees of freedom of a single joint, and have achieved many achievements.

[0004] MI-BCI research has great application potential, and most non-invasive MI-BCI systems are built based on EEG signals. Although EEG signals have the advantage of high temporal resolution, since they are measured on the scalp, the neuronal electrical signals attenuate when passing through the skull, resulting in a low signal-to-noise ratio for the EEG signal, which restricts the improvement of MI-BCI decoding accuracy. Especially for motion trajectory prediction research, the relatively low spatial resolution of EEG signals limits the decoding research of continuous motion trajectory tasks with three-dimensional freedom to a certain extent. Summary of the invention

[0005] The present invention provides a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals, so as to solve the defect of low accuracy in motion trajectory decoding based on scalp EEG in the prior art, and realize efficient decoding of motor imagery tasks based on magnetic brain signals with higher spatial resolution, thereby effectively improving the accuracy of motion trajectory prediction.

[0006] The present invention provides a method for issuing continuous motion trajectory control instructions of a brain-computer interface based on magnetic brain signals, comprising the following steps.

[0007] Obtain the user's brain magnetic signals during motor imagery;

[0008] Calling a motion trajectory prediction model to perform trajectory prediction processing based on the magnetic brain signal to obtain target motion trajectory prediction data;

[0009] Controlling a multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data;

[0010] Among them, the motion trajectory prediction model is obtained by training based on magnetoencephalogram signal samples, and the magnetoencephalogram signal samples are generated when the subjects imagine movements based on motion trajectory videos, and the motion trajectory videos are generated based on three-dimensional motion trajectory data samples generated when the subjects move their hands.

[0011] According to a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals provided by the present invention, the training method of the motion trajectory prediction model includes:

[0012] Acquire a three-dimensional motion trajectory data sample generated when the subject's hand moves within a preset time, and generate a motion trajectory video based on the three-dimensional motion trajectory data sample;

[0013] Obtaining a sample of the magnetic brain signal of the subject when performing motor imagery according to the motion trajectory video;

[0014] Preprocessing the magnetoencephalogram signal samples to obtain preprocessed magnetoencephalogram signal samples;

[0015] Calling the initial motion trajectory prediction model to perform trajectory prediction processing based on the preprocessed magnetoencephalogram signal samples to obtain motion trajectory prediction data;

[0016] The initial motion trajectory prediction model is iteratively trained based on the three-dimensional motion trajectory data samples and the motion trajectory prediction data to obtain the motion trajectory prediction model.

[0017] According to a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on a brain magnetic signal provided by the present invention, the method of preprocessing the brain magnetic signal sample to obtain the preprocessed brain magnetic signal sample includes:

[0018] Performing baseline drift removal and downsampling processing on the EEG signal samples to obtain downsampled EEG signal samples;

[0019] Performing bandpass filtering on the downsampled magnetoencephalogram signal samples to obtain filtered magnetoencephalogram signal samples;

[0020] The filtered magnetoencephalogram signal samples are subjected to independent component analysis to obtain preprocessed magnetoencephalogram signal samples.

[0021] According to a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on a brain magnetic signal provided by the present invention, the brain magnetic signal sample includes multiple channels, and the method further includes:

[0022] Constructing an undirected graph representation according to the preprocessed magnetoencephalogram signal samples, wherein the nodes of the undirected graph representation correspond one-to-one to the channels of the magnetoencephalogram signal samples;

[0023] Respectively calculating two differential entropy features of the leads corresponding to the nodes represented by the undirected graph under two preset frequency bands, and combining the two differential entropy features into a dual-frequency differential entropy feature;

[0024] A preset number of continuous undirected graph representations in the time dimension are spliced ​​to obtain spliced ​​samples.

[0025] According to a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals provided by the present invention, the initial motion trajectory prediction model includes a spatiotemporal attention module, an adaptive graph structure learning module and a spatiotemporal convolution module;

[0026] The calling of the initial motion trajectory prediction model to perform trajectory prediction processing based on the preprocessed magnetoencephalogram signal samples to obtain motion trajectory prediction data includes:

[0027] Inputting the preprocessed magnetoencephalogram signal samples into the spatiotemporal attention module for attention learning to obtain a temporal attention weight and a spatial attention weight;

[0028] Inputting the preprocessed magnetoencephalogram signal samples into the adaptive graph structure learning module for adaptive learning to obtain an adjacency matrix corresponding to the preprocessed magnetoencephalogram signal samples;

[0029] Inputting the temporal attention weight, the spatial attention weight and the adjacency matrix into the spatiotemporal convolution module for graph convolution and time domain convolution processing to obtain the convolution features of the preprocessed magnetoencephalographic signal samples in three-dimensional space;

[0030] Trajectory prediction processing is performed based on the convolution features to obtain motion trajectory prediction data.

[0031] According to a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals provided by the present invention, the initial motion trajectory prediction model also includes a multi-task interactive network, and the trajectory prediction processing based on the convolutional features to obtain motion trajectory prediction data includes:

[0032] Inputting the convolutional features in the three-dimensional space into the multi-task interactive network for soft attention mask processing respectively, to obtain the convolutional features in the three-dimensional space after interactive fusion;

[0033] The convolutional features in the three-dimensional space after interactive fusion are respectively input into the fully connected layer to obtain the position information in three dimensions corresponding to the three-dimensional space, and the position information in the three dimensions is used as the motion trajectory prediction data.

[0034] According to a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals provided by the present invention, the initial motion trajectory prediction model is iteratively trained based on the three-dimensional motion trajectory data sample and the motion trajectory prediction data to obtain the motion trajectory prediction model, including:

[0035] Calculating the error between the three-dimensional motion trajectory data sample and the motion trajectory prediction data to obtain a loss function;

[0036] Based on the loss function, the initial motion trajectory prediction model is iteratively trained through a back propagation algorithm to obtain the motion trajectory prediction model.

[0037] The present invention also provides a brain-computer interface continuous motion trajectory control instruction issuing system based on brain magnetic signals, including the following brain magnetic acquisition equipment, computing equipment and mechanical arm equipment:

[0038] The brain magnetic acquisition device is used to collect the user's brain magnetic signals and send the brain magnetic signals to the computing device, wherein the brain magnetic signals are generated when the user watches the three-dimensional motion trajectory video and performs motion imagination;

[0039] The computing device is used to receive the brain magnetic signal, and call the motion trajectory prediction model to perform motion trajectory prediction processing on the brain magnetic signal to obtain target motion trajectory prediction data corresponding to the motor imagination; and send the target motion trajectory prediction data to the robotic arm device;

[0040] The robotic arm device is used to receive the target motion trajectory prediction data and control the multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-mentioned methods for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals as described in any of the above-mentioned methods.

[0043] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals.

[0044] The method for issuing continuous motion trajectory control instructions for a brain-computer interface based on brain magnetic signals provided by the present invention obtains the brain magnetic signals of the user during motor imagination, then calls a motion trajectory prediction model to perform trajectory prediction processing on the brain magnetic signals to obtain target motion trajectory prediction data, and finally controls a multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data. Since the motor imagination task is efficiently decoded using brain magnetic signals with higher spatial resolution, accurate prediction of the motion trajectory is achieved, and then continuous motion trajectory control instructions for a brain-computer interface based on brain magnetic signals are constructed to control the robotic arm to perform precise movements. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a flow chart of a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals provided by the present invention.

[0047] Figure 2 This is a first-stage experimental flow chart of three-dimensional motion trajectory acquisition provided by the present invention.

[0048] Figure 3 This is a second-stage experimental flow chart of the three-dimensional motion trajectory acquisition provided by the present invention.

[0049] Figure 4 It is a schematic diagram of a motion trajectory prediction network based on spatiotemporal graph convolution provided by the present invention.

[0050] Figure 5 It is a structural schematic diagram of a brain-computer interface continuous motion trajectory control instruction issuing system based on brain magnetic signals provided by the present invention.

[0051] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Figure 1 : is a flow chart of a method for issuing continuous motion trajectory control instructions of a brain-computer interface based on magnetic brain signals provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 103.

[0054] Step 101: Acquire the user's brain magnetic signal during motor imagery.

[0055] In some embodiments, the user's brain magnetic signals can be collected by a high-density magnetometer and gradiometer system, a whole-head brain magnetic system, or a ring coil array. Motor imagery refers to a psychological process of replaying some felt movements and behaviors in the mind without actual muscle activity. This psychological activity can be identified and applied through brain-computer interface (BCI) technology, providing an interface for patients with movement disorders or patients who are completely locked in to interact with the environment by controlling robotic prostheses, wheelchairs, and other equipment.

[0056] Magnetoencephalography (MEG) signals record the relevant magnetic field signals generated by the current formed by the postsynaptic potential of neurons to indirectly project the neural electrical activity in the brain. This signal is not easily affected by the medium and has high temporal resolution and high spatial resolution. In addition, MEG signals are less sensitive to artifacts generated by muscle activity, reducing signal loss caused by filtering and making the signal more holistic.

[0057] Step 102: Calling a motion trajectory prediction model to perform trajectory prediction processing based on the magnetic brain signal to obtain target motion trajectory prediction data.

[0058] In some embodiments, the target motion trajectory prediction data is motion vector data of the x-axis, y-axis and z-axis obtained by decoding the motion trajectory prediction model based on the magnetoencephalogram signal.

[0059] Step 103: Control the multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data.

[0060] In some embodiments, the motion trajectory prediction model is trained based on magnetoencephalogram signal samples, the magnetoencephalogram signal samples are generated when the subject imagines motion based on a motion trajectory video, and the motion trajectory video is generated based on three-dimensional motion trajectory data samples generated when the subject moves his hand.

[0061] In some embodiments, the multi-degree-of-freedom robotic arm can move according to the motion vector data of the x-axis, y-axis and z-axis obtained in step 102, thereby helping the user to achieve the conversion of motion intentions, allowing the user to independently complete some basic movements.

[0062] In some embodiments, the training method of the motion trajectory prediction model includes: obtaining a three-dimensional motion trajectory data sample generated when the subject's hand moves within a preset time, and generating a motion trajectory video based on the three-dimensional motion trajectory data sample; obtaining a brain magnetic signal sample when the subject performs motion imagination according to the motion trajectory video; preprocessing the brain magnetic signal sample to obtain a preprocessed brain magnetic signal sample; calling the initial motion trajectory prediction model to perform trajectory prediction processing based on the preprocessed brain magnetic signal sample to obtain motion trajectory prediction data; iteratively training the initial motion trajectory prediction model based on the three-dimensional motion trajectory data sample and the motion trajectory prediction data to obtain the motion trajectory prediction model.

[0063] Specifically, the present invention uses continuous imaginary movement of the upper limb movement trajectory to iteratively train the initial movement trajectory prediction model, and finally realizes the continuous movement trajectory prediction of the upper limb movement imagination task in three-dimensional space based on the trained movement trajectory prediction model. The experiment includes a total of 10 experiments, each experiment includes 40 trials, and there are two stages in total.

[0064] In the first stage, for the upper limb continuous movement imagination task involved in the present invention, a healthy subject with regular exercise habits was selected to perform continuous trajectory movement in three-dimensional space in accordance with the experimental paradigm requirements with the sampling points of the motion acquisition device attached. The subject remained relaxed during the entire experiment, was not allowed to open his mouth, swallow or chew, and the participant's head movement was controlled to be limited to a maximum range of 5 mm, so that only the upper limbs could perform continuous movements. Figure 2 , Figure 2 This is the first stage experimental flow chart of the three-dimensional motion trajectory acquisition provided by the present invention, such as Figure 2 As shown, each movement of the subject is performed in an 8-second cycle, including 2 seconds of rest (to remind attention), 4 seconds of movement, and 2 seconds of rest as one cycle, and 4 seconds of movement as one trial. The motion acquisition device is used to record the motion trajectory of the entire cycle to construct the corresponding three-dimensional motion trajectory data sample, and the three-dimensional trajectory data sample is imported into the computer to produce a three-dimensional motion trajectory video.

[0065] In the second stage, see Figure 3 , Figure 3 is a second-stage experimental flow chart of the three-dimensional motion trajectory acquisition provided by the present invention, such as Figure 3 As shown in the figure, the subjects performed the synchronous motor imagery task according to the video process displayed on the screen. In the 2-second resting stage (reminder), a static red ball appeared on the screen to prompt the subjects to prepare for imagination. In the 2-second motor imagery preparation stage, the video presented a three-dimensional motion trajectory and was in a static state, so that the subjects could prepare. In the 4-second motor imagery stage, a red dot appeared at the beginning of the three-dimensional trajectory, and a dynamic induction prompt was given within 4 seconds through the three-dimensional motion trajectory. The subjects performed motor imagery in a visual imagination manner according to the prompt. The red dot disappeared after reaching the end of the trajectory, and then entered the 2-second resting stage. During the rest stage, a white ball appeared in the center of the screen and lasted for 2 seconds.

[0066] In an embodiment of the present invention, an OptiTrack motion acquisition device is used to collect three-dimensional motion trajectory data of the subject when performing actual trajectory motion as a sample, and a 275-channel CTF MEGTM magnetoencephalogram signal acquisition device with a sampling rate of 1200 Hz is used to collect the whole-brain magnetoencephalogram signals of the subject when performing motor imagery as a sample.

[0067] In some embodiments, the EMG signal samples are preprocessed to obtain preprocessed EMG signal samples, including: performing baseline drift removal and downsampling processing on the EMG signal samples to obtain downsampled EMG signal samples; performing bandpass filtering on the downsampled EMG signal samples to obtain filtered EMG signal samples; and performing independent component analysis on the filtered EMG signal samples to obtain preprocessed EMG signal samples.

[0068] Specifically, the EEG signal samples were first processed to remove baseline drift, that is, a reference signal was subtracted from the EEG signal data to more accurately analyze the neural activity of interest. The EEG signal was then downsampled to 200 Hz to reduce the amount of calculation. The downsampled data was bandpass filtered at 1-40 Hz to filter out high-frequency and low-frequency noise. Finally, independent component analysis (ICA) was used to remove interference from electrooculogram signals such as eye movements and blinks, and the preprocessed EEG signal samples were obtained.

[0069] In some embodiments, the magnetic brain signal sample includes multiple channels, and the method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals provided by the present invention also includes: constructing an undirected graph representation based on the preprocessed magnetic brain signal samples, wherein the nodes of the undirected graph representation correspond one-to-one to the channels of the magnetic brain signal samples; respectively calculating two differential entropy features of the leads corresponding to the nodes of the undirected graph representation in two preset frequency bands, and combining the two differential entropy features into a dual-frequency differential entropy feature; splicing a preset number of continuous undirected graph representations in the time dimension to obtain a spliced ​​sample.

[0070] In some embodiments, see Figure 4 , Figure 4It is a schematic diagram of a motion trajectory prediction network based on spatiotemporal graph convolution provided by the present invention. The motion trajectory prediction network based on spatiotemporal graph convolution is the initial motion trajectory prediction model in the present invention, which includes three parts: a spatiotemporal graph convolution feature learning module, a multi-task interactive fusion module, and a continuous action prediction module. For hundreds of channels of EEG signals, the present invention utilizes the learning ability of the graph structure, considers that the time domain and frequency domain information of the EEG signals are both specific in the process of continuous motion intention, extracts the dual-frequency differential entropy (DE) features, and organizes them into the input of the graph structure; constructs a spatiotemporal graph convolutional network (STGCN) learning method, and learns and fuses the deep information inside and between nodes by adaptively learning the weights of each node and adjacent nodes, so as to realize the in-depth mining of hidden layer features; based on a multi-task interactive fusion network oriented to the correlation and dependency between dimensions, the EEG signal features extracted in three dimensions are interactively learned and fused to learn a comprehensive understanding and comprehensive prediction of the position, so as to improve the expression ability of high-dimensional features; finally, the three branches output the three-dimensional spatial coordinates of the current moment respectively, so as to realize continuous action prediction.

[0071] Specifically, in decoding of EEG signals, mining the functional connectivity information between channels is an important means of capturing the spatial structure of the brain, which can effectively improve the expressive power of the extracted features, thereby improving the decoding accuracy of EEG signals. Therefore, the present invention organizes EEG signal samples into a graph representation as the input of the subsequent model. First, the multi-channel EEG data is Divided into 400 ms windows (100 ms step size), constructed as an undirected graph representation , V express N A collection of graph nodes, where the nodes of the graph correspond one by one to the channels of the magnetoencephalographic signal samples; represents the set of edges, is the adjacency matrix, corresponding to the connection relationship between different channels. Then calculate the frequency bands of 1-3 Hz and 6-32 Hz N The DE features of the leads corresponding to the nodes constitute the dual-frequency DE features as the node features of the graph representation. Afterwards, the M (M>=2) graph representations with continuous time dimension are concatenated into a sample with the time dimension as the axis as the input of the model.

[0072] In some embodiments, see Figure 4The initial motion trajectory prediction model includes a spatiotemporal attention module, an adaptive graph structure learning module and a spatiotemporal convolution module. The calling of the initial motion trajectory prediction model performs trajectory prediction processing based on the preprocessed EEG signal samples to obtain motion trajectory prediction data, including: inputting the preprocessed EEG signal samples into the spatiotemporal attention module for attention learning to obtain time attention weights and space attention weights; inputting the preprocessed EEG signal samples into the adaptive graph structure learning module for adaptive learning to obtain an adjacency matrix corresponding to the preprocessed EEG signal samples; inputting the time attention weights, space attention weights and the adjacency matrix into the spatiotemporal convolution module for graph convolution and time domain convolution processing to obtain the convolution features of the preprocessed EEG signal samples in three-dimensional space; performing trajectory prediction processing based on the convolution features to obtain motion trajectory prediction data.

[0073] Specifically, in order to realize the motion prediction of three-dimensional space motion, the present invention decomposes the position of the motion into three dimensions of x, y and z, and takes the initial position before the start of the motion as the coordinate origin, and takes the position of the three dimensions relative to the initial point during the motion as the prediction target. In the position prediction of each dimension, the STGCN architecture is used to extract the hidden layer features of the brain magnetic signal sample. The designed STGCN network uses a graph convolutional network to explore the dynamic and intrinsic connection information between different brain magnetic channels, and uses time domain convolution to extract the time context information of the brain magnetic sample. It mainly includes three parts: spatiotemporal attention module, adaptive graph structure learning module and spatiotemporal convolution module.

[0074] Among them, the spatiotemporal attention module includes a temporal attention mechanism and a spatial attention mechanism. The spatial attention mechanism is used to dynamically focus on the brain regions and channels that are strongly associated with the action prediction task in space, and the temporal attention mechanism is used to dynamically capture the key data in the time series that are recognizable and reliable for action prediction. The input of this part is the EEG signal sample. Through attention learning, the temporal attention weight and spatial attention weight are obtained. The adaptive graph structure learning module can capture the dynamic functional connection relationship between EEG signal channels, construct different graph structures (adjacency matrices) for different EEG signal samples, and can adaptively learn the importance between nodes. The input of this part is the EEG signal sample. Through adaptive learning, the adjacency matrix corresponding to the sample is obtained. The spatiotemporal convolution module is a combination of graph convolution and time domain convolution, which is used to simultaneously extract the spatial dimension and temporal dimension features of the EEG signal sample. Through the series connection of multiple layers of spatiotemporal convolution, the convolution feature representation of the EEG signal sample from primary to advanced level, that is, the convolution feature in three-dimensional space, is extracted layer by layer.

[0075] In some embodiments, the initial motion trajectory prediction model also includes a multi-task interactive network, and the trajectory prediction processing is performed based on the convolutional features to obtain motion trajectory prediction data, including: inputting the convolutional features in the three-dimensional space into the multi-task interactive network for soft attention mask processing to obtain the convolutional features in the three-dimensional space after interactive fusion; inputting the convolutional features in the three-dimensional space after interactive fusion into the fully connected layer to obtain the position information in the three dimensions corresponding to the three-dimensional space, and using the position information in the three dimensions as the motion trajectory prediction data.

[0076] Specifically, the three dimensions of x, y and z can be regarded as three prediction tasks. Since the coordinate prediction tasks in the three dimensions are similar and strongly correlated, the present invention adopts a multi-task learning framework based on the feature interaction mode, that is, the spatiotemporal graph convolutional feature learning and the task interaction fusion network (position prediction of a single dimension) are connected in series. The spatiotemporal graph convolutional feature learning specifically extracts relevant features of a single dimension, and the subsequent multi-task interaction network uses a soft attention mask to selectively fuse features of other dimensions, thereby improving the comprehensive understanding of spatial information in each dimension and realizing joint optimization in three-dimensional space.

[0077] In some embodiments, the iterative training of the initial motion trajectory prediction model based on the three-dimensional motion trajectory data samples and the motion trajectory prediction data to obtain the motion trajectory prediction model includes: calculating the error between the three-dimensional motion trajectory data samples and the motion trajectory prediction data to obtain a loss function; based on the loss function, iteratively training the initial motion trajectory prediction model through a back propagation algorithm to obtain the motion trajectory prediction model.

[0078] Specifically, the interactively fused features enter the fully connected layer (Dense), output the position information of the current dimension, and use the Euclidean distance as an indicator to calculate the predicted position. With actual location Error :

[0079]

[0080] in, Indicates the calculation of mean square error.

[0081] At the same time, calculate the differential of the position at this moment and the previous moment ( ) Get the motion speed information and calculate the error with the actual speed as a constraint :

[0082]

[0083] The sum of the position error and the velocity error constitutes the overall error in the network training process, and the model parameters are optimized through the back propagation algorithm. At the same time, the same operation is performed on the position and velocity information of the y-axis and z-axis respectively to obtain a three-dimensional spatial decoding model.

[0084] In a specific embodiment, a limited space is first set, a target position is randomly given, and the subject is asked to move the right hand to the target location autonomously within a specified time. In this process, motion capture markers are attached to the subject's hand, and professional motion capture equipment such as OptiTrack is used to record the three-dimensional motion trajectory information of the subject; then, based on the motion trajectory information, a motion trajectory video is constructed, and the subject is asked to watch the video and imagine the arm motion trajectory while collecting brain magnetic signals. Taking into account the real-time requirements and prediction difficulties of three-dimensional motion trajectory prediction, the present invention uses professional motion capture equipment to record the actual motion trajectory data of the subject's hand and make it into a three-dimensional animation form to guide the subject to imagine the movement and improve the correlation between the subject's imagination and the trajectory.

[0085] The collected EEG signals were then preprocessed. Each time the process of imagining the motion trajectory (4 seconds) was regarded as a trial, it was subdivided into a sliding window with a window length of 400 ms and a step length of 100 ms. The motion trajectory data and EEG data collected by the corresponding motion capture device were organized into a trial of continuous imagined motion trajectory samples and EEG signal samples.

[0086] Finally, the magnetoencephalogram signal samples are used as input to the initial motion trajectory prediction model to output the corresponding xyz axis three-dimensional motion vector. The output motion trajectory prediction data is then fit calculated with the three-dimensional motion trajectory data samples collected by the motion capture device to constrain the initial motion trajectory prediction model for learning, thereby optimizing the prediction accuracy of the subject's motion trajectory and obtaining the motion trajectory prediction model.

[0087] Based on the trained motion trajectory prediction model, a motor imagery brain-computer interface system based on brain magnetic signals is constructed. The system collects the user's current brain magnetic data, inputs it into the motion trajectory prediction model for motion trajectory prediction, and sends the predicted xyz motion vector data to the robotic arm to complete the continuous control of the robotic arm.

[0088] The following is a description of the system for issuing continuous motion trajectory control instructions for the brain-computer interface based on magnetic brain signals provided by the present invention. The system for issuing continuous motion trajectory control instructions for the brain-computer interface based on magnetic brain signals described below and the method for issuing continuous motion trajectory control instructions for the brain-computer interface based on magnetic brain signals described above can be referenced to each other.

[0089] See also Figure 5 , Figure 5It is a structural schematic diagram of a brain-computer interface continuous motion trajectory control instruction issuing system based on brain magnetic signals provided by the present invention, comprising a brain magnetic acquisition device, a computing device and a robotic arm device, wherein the brain magnetic acquisition device is used to acquire the user's brain magnetic signals and send the brain magnetic signals to the computing device, wherein the brain magnetic signals are generated when the user watches a three-dimensional motion trajectory video and performs motion imagination; the computing device is used to receive the brain magnetic signals, and call a motion trajectory prediction model to perform motion trajectory prediction processing on the brain magnetic signals to obtain target motion trajectory prediction data corresponding to the motion imagination; the target motion trajectory prediction data is sent to the robotic arm device; the robotic arm device is used to receive the target motion trajectory prediction data, and control a multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data.

[0090] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a method for issuing continuous motion trajectory control instructions for a brain-computer interface based on brain magnetic signals, the method comprising: obtaining the brain magnetic signals of the user during motion imagination; calling the motion trajectory prediction model to perform trajectory prediction processing based on the brain magnetic signals to obtain target motion trajectory prediction data; and controlling the multi-degree-of-freedom manipulator to perform three-dimensional motion based on the target motion trajectory prediction data.

[0091] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for issuing continuous motion trajectory control instructions for a brain-computer interface based on brain magnetic signals provided by the above-mentioned methods. The method includes: obtaining the brain magnetic signals of the user during motor imagination; calling a motion trajectory prediction model to perform trajectory prediction processing based on the brain magnetic signals to obtain target motion trajectory prediction data; and controlling a multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals provided by the above-mentioned methods. The method includes: obtaining the magnetic brain signals of the user during motor imagination; calling a motion trajectory prediction model to perform trajectory prediction processing based on the magnetic brain signals to obtain target motion trajectory prediction data; and controlling a multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data.

[0094] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals, characterized in that: include: Obtain the user's brain magnetic signals during motor imagery; Calling a motion trajectory prediction model to perform trajectory prediction processing based on the magnetic brain signal to obtain target motion trajectory prediction data; Controlling a multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data; The motion trajectory prediction model is obtained by training based on magnetoencephalogram signal samples, the magnetoencephalogram signal samples are generated when the subject performs motion imagination according to the motion trajectory video, and the motion trajectory video is generated according to the three-dimensional motion trajectory data samples generated when the subject's hand moves; The training method of the motion trajectory prediction model includes: Acquire the three-dimensional motion trajectory data sample generated when the subject's hand moves within a preset time, and generate the motion trajectory video based on the three-dimensional motion trajectory data sample; Acquiring the electroencephalographic signal sample when the subject performs motor imagery according to the motion trajectory video; Preprocessing the magnetoencephalogram signal samples to obtain preprocessed magnetoencephalogram signal samples; Calling the initial motion trajectory prediction model to perform trajectory prediction processing based on the preprocessed magnetoencephalogram signal samples to obtain motion trajectory prediction data; Iteratively training the initial motion trajectory prediction model based on the three-dimensional motion trajectory data sample and the motion trajectory prediction data to obtain the motion trajectory prediction model; The initial motion trajectory prediction model includes a spatiotemporal attention module, an adaptive graph structure learning module and a spatiotemporal convolution module. The calling of the initial motion trajectory prediction model to perform trajectory prediction processing based on the preprocessed magnetoencephalogram signal samples to obtain motion trajectory prediction data includes: Inputting the preprocessed magnetoencephalogram signal samples into the spatiotemporal attention module for attention learning to obtain a temporal attention weight and a spatial attention weight; Inputting the preprocessed magnetoencephalogram signal samples into the adaptive graph structure learning module for adaptive learning to obtain an adjacency matrix corresponding to the preprocessed magnetoencephalogram signal samples; Inputting the temporal attention weight, the spatial attention weight and the adjacency matrix into the spatiotemporal convolution module for graph convolution and time domain convolution processing to obtain the convolution features of the preprocessed magnetoencephalographic signal samples in three-dimensional space; Trajectory prediction processing is performed based on the convolution features to obtain motion trajectory prediction data.

2. The method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals according to claim 1 is characterized in that: The preprocessing of the magnetoencephalogram signal samples to obtain the preprocessed magnetoencephalogram signal samples includes: Performing baseline drift removal and downsampling processing on the EEG signal samples to obtain downsampled EEG signal samples; Performing bandpass filtering on the downsampled magnetoencephalogram signal samples to obtain filtered magnetoencephalogram signal samples; The filtered magnetoencephalogram signal samples are subjected to independent component analysis to obtain preprocessed magnetoencephalogram signal samples.

3. The method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals according to claim 1 is characterized in that: The magnetoencephalogram signal sample includes a plurality of channels, and the method further includes: Constructing an undirected graph representation according to the preprocessed magnetoencephalogram signal samples, wherein the nodes of the undirected graph representation correspond one-to-one to the channels of the magnetoencephalogram signal samples; Respectively calculating two differential entropy features of the leads corresponding to the nodes represented by the undirected graph under two preset frequency bands, and combining the two differential entropy features into a dual-frequency differential entropy feature; A preset number of continuous undirected graph representations in the time dimension are spliced ​​to obtain spliced ​​samples.

4. The method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals according to claim 1 is characterized in that: The initial motion trajectory prediction model also includes a multi-task interactive network, and the trajectory prediction processing is performed based on the convolution feature to obtain motion trajectory prediction data, including: Inputting the convolutional features in the three-dimensional space into the multi-task interactive network for soft attention mask processing respectively, to obtain the convolutional features in the three-dimensional space after interactive fusion; The convolutional features in the three-dimensional space after interactive fusion are respectively input into the fully connected layer to obtain the position information in three dimensions corresponding to the three-dimensional space, and the position information in the three dimensions is used as the motion trajectory prediction data.

5. The method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals according to claim 1, characterized in that: The iterative training of the initial motion trajectory prediction model based on the three-dimensional motion trajectory data sample and the motion trajectory prediction data to obtain the motion trajectory prediction model includes: Calculating the error between the three-dimensional motion trajectory data sample and the motion trajectory prediction data to obtain a loss function; Based on the loss function, the initial motion trajectory prediction model is iteratively trained through a back propagation algorithm to obtain the motion trajectory prediction model.

6. A brain-computer interface continuous motion trajectory control instruction issuing system based on brain magnetic signals, characterized in that: Including brain magnetic collection equipment, computing equipment and robotic arm equipment: The brain magnetic acquisition device is used to collect the user's brain magnetic signals and send the brain magnetic signals to the computing device, wherein the brain magnetic signals are generated when the user watches the three-dimensional motion trajectory video and performs motion imagination; The computing device is used to receive the brain magnetic signal, and call the motion trajectory prediction model to perform motion trajectory prediction processing on the brain magnetic signal to obtain target motion trajectory prediction data corresponding to the motor imagination; and send the target motion trajectory prediction data to the robotic arm device; The robotic arm device is used to receive the target motion trajectory prediction data, and control the multi-degree-of-freedom robotic arm to perform three-dimensional motion based on the target motion trajectory prediction data; The training method of the motion trajectory prediction model includes: Acquire a three-dimensional motion trajectory data sample generated when the subject's hand moves within a preset time, and generate the motion trajectory video based on the three-dimensional motion trajectory data sample; Obtaining a sample of the magnetic brain signal of the subject when performing motor imagery according to the motion trajectory video; Preprocessing the magnetoencephalogram signal samples to obtain preprocessed magnetoencephalogram signal samples; Calling the initial motion trajectory prediction model to perform trajectory prediction processing based on the preprocessed magnetoencephalogram signal samples to obtain motion trajectory prediction data; Iteratively training the initial motion trajectory prediction model based on the three-dimensional motion trajectory data sample and the motion trajectory prediction data to obtain the motion trajectory prediction model; The initial motion trajectory prediction model includes a spatiotemporal attention module, an adaptive graph structure learning module and a spatiotemporal convolution module. The calling of the initial motion trajectory prediction model to perform trajectory prediction processing based on the preprocessed magnetoencephalogram signal samples to obtain motion trajectory prediction data includes: Inputting the preprocessed magnetoencephalogram signal samples into the spatiotemporal attention module for attention learning to obtain a temporal attention weight and a spatial attention weight; Inputting the preprocessed magnetoencephalogram signal samples into the adaptive graph structure learning module for adaptive learning to obtain an adjacency matrix corresponding to the preprocessed magnetoencephalogram signal samples; Inputting the temporal attention weight, the spatial attention weight and the adjacency matrix into the spatiotemporal convolution module for graph convolution and time domain convolution processing to obtain the convolution features of the preprocessed magnetoencephalographic signal samples in three-dimensional space; Trajectory prediction processing is performed based on the convolution features to obtain motion trajectory prediction data.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for issuing continuous motion trajectory control instructions for a brain-computer interface based on magnetic brain signals as described in any one of claims 1 to 5.