Biosignal processing device, biosignal processing method, and program
The biosignal processing device improves motor output decoding accuracy by integrating brain and muscle activity sensors with a state transition model, addressing the challenge of varying decoding precision in precise motor tasks.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- NIPPON TELEGRAPH & TELEPHONE CORP
- Filing Date
- 2024-11-12
- Publication Date
- 2026-05-22
AI Technical Summary
Existing brain activity-movement output decoders struggle with varying decoding accuracy, particularly in tasks requiring precise motor control like pinching, due to the lack of consideration for musculoskeletal dynamics.
A biosignal processing device that incorporates brain activity, muscle activity, and muscle strength sensors, along with signal processing units, constructs a state transition model using a Kalman filter to integrate EMG and ECoG data, accounting for musculoskeletal dynamics to improve decoding accuracy.
Enhances the accuracy of motor output estimation, especially in tasks like pinching, by simulating neurological aspects of motor control and incorporating spinal neural circuit feedback.
Smart Images
Figure 2026085135000001_ABST
Abstract
Description
Technical Field
[0001] One aspect of this invention relates to a technique for processing biological signals to construct a brain activity-movement output decoder.
Background Art
[0002] Movement decoding based on brain activity has played an important role in the progress of brain-machine interfaces (BMIs) such as rehabilitation and prosthetic control. Electro-corticogram (ECoG) is one of the biological signals representing brain activity, and a brain activity-movement output decoder that predicts movement output from its sensor values is known. Non-Patent Document 1 introduces that it is possible to decode movement outputs such as limb trajectories, wrist flexion / extension, and simple finger movements from brain activity.
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] The brain activity-movement output decoder estimates movement output from brain activity using a model constructed by machine learning. However, in existing technologies, the decoding accuracy varies greatly depending on the task. In particular, in tasks that require precise adjustment of movement (such as pinching), it is difficult to predict movement output with high accuracy.
[0005] This invention was made in view of the above circumstances and aims to provide a technology that can improve the decoding accuracy of motor output. [Means for solving the problem]
[0006] A biosignal processing device according to one aspect of this invention comprises a brain activity sensor, a muscle activity sensor, a muscle strength sensor, a brain activity signal processing unit, an electromyogram signal processing unit, a muscle strength signal processing unit, and a learning unit. The brain activity sensor measures a brain activity signal that reflects the brain activity of a subject given a task and outputs brain activity time-series data. The muscle activity sensor measures an electromyogram signal that reflects the muscle activity of the subject and outputs electromyogram time-series data. The muscle strength sensor measures the motor output related to the subject's task and outputs motor output time-series data. The brain activity signal processing unit calculates brain activity signal features from the brain activity time-series data. The electromyogram signal processing unit calculates electromyogram signal features from the electromyogram time-series data. The muscle strength signal processing unit calculates motor output signal features from the motor output time-series data. The learning unit constructs a trained model by training a state transition model, which uses time-series motor output and time-series muscle activity as state variables and time-series brain activity as an observed variable, using brain activity signal features, electromyogram signals, and motor output signal features. [Effects of the Invention]
[0007] According to one aspect of this invention, the decoding accuracy of motion output can be improved. [Brief explanation of the drawing]
[0008] [Figure 1] Figure 1 shows an example of a biosignal processing device according to an embodiment. [Figure 2] Figure 2 is a functional block diagram showing an example of the processor 1 and storage 5 shown in Figure 1. [Figure 3] Figure 3 is a diagram illustrating the procedure for acquiring training data in an embodiment. [Figure 4]Figure 4 is a diagram illustrating the generation of a trained model. [Figure 5] Figure 5 is a diagram illustrating the state estimation performed by the estimation unit 16. [Figure 6] Figure 6 shows a comparison of Force estimates over time for representative trials. [Figure 7] Figure 7 is a box plot showing the correlation coefficient (CC) and root mean square error (RMSE) between the true value and the given value for each trial. [Figure 8] Figure 8 is a graph showing the results of a detailed analysis for each different movement phase. [Figure 9] Figure 9 is a box plot showing the CC and RMSE for each trial, plotted for each of the four motor phases. [Modes for carrying out the invention]
[0009] <Structure> Figure 1 shows an example of a biosignal processing device according to an embodiment. The biosignal processing device 100 shown in Figure 1 is a computer comprising a processor 1 such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), memory 4, storage 5, and a sensor interface (I / F) unit 2 and an input / output interface (I / F) unit 3. These are connected via a bus 6.
[0010] Storage 5 includes, for example, a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and a non-volatile memory such as ROM (Read Only Memory). In addition to basic software such as an OS (Operating System), Storage 5 stores application programs necessary to perform various processes according to one embodiment of this invention. Memory 4 is a semiconductor storage device that includes, for example, non-volatile memory and volatile memory such as RAM (Random Access Memory).
[0011] The sensor interface (I / F) section 2 is, for example, a USB (Universal Serial Interface) slot or a serial interface such as RS-232C, and is connected to the ECoG sensor 21, EMG sensor 22, and Force sensor 23.
[0012] The ECoG sensor 21 is a brain activity sensor that measures brain activity signals reflecting the brain activity of a subject given a task and outputs brain activity time-series data. In this embodiment, the task is pinching an object with the fingertips, and the ECoG sensor 21 measures the cortical electroencephalogram (ECoG) during this action and outputs ECoG time-series data.
[0013] The EMG sensor 22 is a muscle activity sensor that measures electromyogram signals that reflect the muscle activity of a subject and outputs electromyogram time-series data. In this embodiment, the EMG sensor 22 measures the electromyogram (EMG) of a subject performing a task and outputs EMG time-series data.
[0014] The Force sensor 23 is a muscle strength sensor that measures the motor output of the subject in relation to the above task and outputs time-series data of the motor output. Examples of motor output that can be measured by the Force sensor 23 include fingertip force and fingertip position. In this embodiment, the Force sensor 23 is attached to the subject's fingertip to measure fingertip force and outputs Force time-series data. The input / output interface unit 3 is connected to the monitor 31 and the operation unit 32 to provide an operating environment using a GUI (Graphical User Interface).
[0015] FIG. 2 is a functional block diagram showing an example of the processor 1 and the storage 5 shown in FIG. 1. The processor 1 includes a data acquisition unit 11, an ECoG signal processing unit 12, an EMG signal processing unit 13, a Force signal processing unit 14, a learning unit 15, and an estimation unit 16 as functional blocks according to the embodiment. These functional blocks are processing functions realized by loading the program 51 stored in the storage into the memory 4 and executing it by the processor 1. That is, the program 51 includes instructions for causing a computer to function as the biological signal processing apparatus 100.
[0016] The data acquisition unit 11 acquires ECoG time series data from the ECoG sensor 21, acquires EMG time series data from the EMG sensor 22, and acquires Force time series data from the Force sensor 23. These data are stored in the storage 5 as ECoG time series data 52, EMG time series data 53, and Force time series data 54.
[0017] The ECoG signal processing unit 12 calculates brain activity signal features from the brain activity time series data. In the embodiment, the ECoG signal processing unit 12 calculates ECoG signal features from the ECoG time series data 52 acquired from the ECoG sensor 21. For example, notch filtering is used to remove the line noise of the commercial power supply, a band-pass filter (1.5 - 240 Hz) is applied, and then the Hilbert transform is used to extract the average power of two high-frequency bands (high-gamma 1, 60 - 120 Hz; high-gamma 2, 120 - 180 Hz) as features. Then, resampling (200 Hz) is performed to align the time points with the EMG time series data, and finally z-normalization is performed.
[0018] Note that the average power in the high gamma band is said to reflect neural activity. Therefore, instead of ECoG measurement, the Firing Rate of neurons can be directly measured and appropriate preprocessing can be performed to substitute for the ECoG signal features.
[0019] The EMG signal processing unit 13 calculates EMG signal features from EMG time-series data. In this embodiment, the EMG signal processing unit 13 calculates EMG signal features from EMG time-series data 53 acquired from the EMG sensor 22. For example, after removing commercial power line noise with a notch filter, the EMG signal processing unit 13 applies a bandpass filter (1.5-60 Hz) and resamples (200 Hz). Furthermore, the EMG signal processing unit 13 calculates the envelope of muscle activity by performing low-frequency filtering using full-wave rectification and moving average, and finally obtains EMG signal features by z-normalizing if necessary.
[0020] The Force signal processing unit 14 calculates motion output signal features from motion output time series data. In this embodiment, the Force signal processing unit 14 calculates Force signal features from Force time series data 54 acquired from the Force sensor 23. For the Force time series data, processing such as resampling (200Hz) to align the time (timestamp) with the ECoG time series data and EMG time series data may be sufficient. Alternatively, filtering may be performed as appropriate depending on the characteristics of the device and sensor.
[0021] The ECoG signal features, EMG signal features, and Force signal features are stored in storage 5 as training data 55. The processing procedures for ECoG and EMG can be applied to methods such as those disclosed in reference [1].
[0022] The learning unit 15 repeatedly provides the ECoG signal features, EMG signal features, and Force signal features to the state transition model to train it and construct a trained model. In this embodiment, a Kalman filter is used as the state transition model.
[0023] In particular, in this embodiment, a trained model is constructed by training a state transition model that uses EMG time series data as state variables in addition to Force time series data, and ECoG time series data as observed variables. The constructed trained model is stored in storage 5 (trained model 56).
[0024] In other words, in this embodiment, a state transition model is applied using brain activity time series data as the observed variable, motor output time series data as the target of estimation, and EMG (electromyography) time series data as the state variable. This makes it possible to construct a brain activity-motor output decoder that takes musculoskeletal dynamics into account.
[0025] The estimation unit 16 estimates the motor output by inputting ECoG signal features into the constructed trained model 56 and obtaining the motor output as its output.
[0026] Figure 3 is a diagram illustrating the procedure for acquiring training data in an embodiment. In this embodiment, monkeys are assumed to be the subjects and to perform pinching as the task. Below, we describe a method for acquiring ECoG time-series data in the motor cortex and EMG time-series data of the upper limbs when performing the task, and constructing a decoding model that estimates pinching fingertip force as motor output, taking musculoskeletal constraints into account. Here, the motor cortex may include the primary motor cortex and the premotor cortex.
[0027] In Figure 3, when subject 200 performs a task of pinching a force sensor, time-series ECoG data (e.g., 20 channels) from the motor cortex, time-series EMG (electromyogram) data from the upper limbs, and time-series force data are simultaneously collected by the ECoG sensor 21, EMG sensor 22, and Force sensor 23, respectively.
[0028] The ECoG signal processing unit 12 processes the ECoG time series data 52 to calculate ECoG signal features and stores them in the training data 55 of the storage 5. The EMG signal processing unit 13 processes the EMG time series data 53 to calculate EMG signal features and stores them in the training data 55 of the storage 5. The Force signal processing unit 14 processes the Force time series data 54 to calculate Force signal features and stores them in the training data 55 of the storage 5.
[0029] Here, as time-series EMG (electromyogram) data from the upper limbs, data from 14 channels are acquired, for example, from the anterior deltoideus (aDEL), triceps (TRI), biceps (BB), brachioradialis (BR), extensor carpi radialis (ECR), extensor digitorum communis (EDC), flexor carpi ulnaris (FCU), extensor digitorum-4, 5 (ED45), extensor carpi ulnaris (ECU), extensor digitorum-2, 3 (ED23), adductor pollicis (ADP), flexor digitorum profundus (FDP), flexor digitorum superficialis (FDS), palmaris longus (PL), and flexor carpi radialis (FCR). Note that the electrodes of the ECoG sensor 21 and the EMG sensor 22 should be positioned using surgical techniques.
[0030] Figure 4 is a diagram illustrating the generation of a pre-trained model. The learning unit 15 trains a Kalman filter 15a, which acts as a learner, using three types of training datasets to construct a pre-trained model 56.
[0031] <effect> Next, the operation in the embodiment will be explained in detail using mathematical formulas. For the purposes of this discussion, it is common practice to divide the training dataset into training and test sets in an arbitrary ratio. In this embodiment, a total of 690 trial data sets were divided in an 8:2 ratio (training:test).
[0032] The procedure for modeling the Kalman filter is explained using mathematical formulas (see, for example, reference [2]). <Definition of the equation of state> The state vector 《1》 is represented by equation (1).
number
[0033] The equation of state is given by equation (2).
number
[0034] In this embodiment, we propose to include (4) in (1). That is, the feedback of the EMG from the previous time point to the EMG from the next time point is modeled using a state equation to represent the dynamics of the musculoskeletal system.
[0035] <Definition of the observation equation> The observed vector 《7》 is expressed by equation (3).
number
[0036] In equation (3), 《8》 represents the high-gamma i feature of the j-th channel at time point 《2》. The observation vector 《7》 includes ECoG signal features of high-gamma 1 and 2 for each channel at time point 《2》 from several time points prior, and a constant term 1. In the embodiment, it includes ECoG signal features from 50 ms to 5 ms prior. This is because it is resampled at 200 Hz, meaning that the observation vector 《7》 in equation (3) includes ECoG signal features from 10 to 1 time point prior. Since motor command brain activity occurs prior to movement, we focus on signals from several ms prior. For how many time points prior to use the signals, we referred to reference [1].
[0037] The observation equation is expressed by equation (4).
number
[0038] <Method for deriving parameters> The derivation of parameter 《11》 corresponds to training the learner. This can be determined using general methods. Regarding (12), it can be found using the least squares method with equations (5) and (6).
number
[0039] Next, the residuals of each estimated value
[13] are given by equations (7) and (8).
number
[0040] Furthermore, the noise covariance matrix (14) can be obtained using equations (9) and (10).
number
[0041] Figure 5 is a diagram illustrating the state estimation performed by the estimation unit 16. In Figure 5, the estimation unit 16 performs state estimation at each point in time using a general Kalman filter algorithm. That is, the estimation unit 16 inputs the test dataset or ECoG time series data (15) acquired in real time into the trained model 56 (in which the parameters (11) are stored) and outputs a state vector (16) including the Force estimate.
[0042] Note that it is necessary to set an initial value (17). If using offline data (test dataset), you can set the actual initial value, and if using online data, you can set an appropriate initial value artificially. In this embodiment, the actual initial value of the test dataset was set.
number
[0043] The algorithm for one step at time point 《2》 is shown below. First, we obtain the pre-estimated value of the state vector from the state equation using equation (11). Next, we obtain the prior covariance matrix
[19] using equation (12).
number
[0044] Furthermore, the Kalman gain (20) is calculated using equation (13), and the posterior estimate of the state vector (21) is calculated using equation (14). Note that the parentheses in equation (13) + This is the generalized inverse matrix, but if the inverse matrix can be found, you can simply calculate the inverse matrix.
number
number
[0045] <Accuracy verification results of the embodiment> To verify the accuracy of the embodiment, estimation was performed on the test dataset and evaluated using the correlation coefficient (CC) and root mean square error (RMSE) for each trial. Furthermore, to demonstrate the effectiveness of modeling musculoskeletal dynamics, it was compared with a model that does not consider musculoskeletal dynamics (comparison model). Specifically, in the comparison model, EMG was not included in the state vector 《1》, and learning and estimation were performed using the same process with equation (17).
number
[0046] Figure 6 shows a comparison of Force estimates over time for representative trials. Figure 6 shows that the embodiment model (With EMG Model) can estimate motor output with higher accuracy than the comparison model (Without EMG Model).
[0047] Figure 7 is a box plot showing the correlation coefficient (CC) and root mean square error (RMSE) with the true value for each trial. From the perspective of CC and RMSE, it is shown that the model of the embodiment can estimate motor output with significantly higher accuracy than the comparison model (paired t-test, α=0.05).
[0048] Figure 8 is a graph showing the results of a detailed analysis for each different motion phase. The dotted line represents when Force is 0 (No Force), the solid line represents the rising phase, the dashed line represents the peak phase, and the dashed line represents the falling phase.
[0049] Figure 9 is a box plot showing the CC and RMSE for each trial, plotted for each of the four motor phases. Referring to Figures 8 and 9, it can be seen that at the peak, the model of the embodiment estimates motor output with significantly higher accuracy than the comparison model (paired t-test, Bonferroni correction, α=0.05 / 4).
[0050] <Effects> As described above, in this embodiment, a brain activity-motor output decoder that takes musculoskeletal dynamics into account was constructed by applying a state transition model, such as a Kalman filter, to time-series brain activity data (e.g., ECoG in the motor cortex and neuronal firing rate) as observed variables, and time-series motor output data (e.g., fingertip force and fingertip position) as the target of estimation, in addition to EMG (electromyography) time-series data as state variables. In this way, it becomes possible to simulate the neurological aspects of motor control, such as those represented by spinal neural circuits.
[0051] Existing technologies do not take into account the dynamic adjustment (dynamics) of the musculoskeletal system by spinal neural circuits. The spinal cord receives not only motor command inputs but also feedback (FB) inputs from the brain via musculoskeletal proprioceptors (organs that detect muscle tension and joint angles), and outputs action potentials derived from muscle activity via motor neurons. This FB-mediated adjustment is especially important for tasks that require precise motor control.
[0052] Therefore, by constructing a brain activity-motor output decoder that takes musculoskeletal dynamics into account using the model of the embodiment, it becomes possible to estimate motor output with higher accuracy. Accordingly, according to the embodiment, it is possible to improve the decoding accuracy of motor output. In particular, the proposed model was demonstrated to improve the accuracy of estimating fingertip force, a motor output, in tasks requiring precise motor control, such as pinching. This is expected to enhance its practical application in BMI applications such as rehabilitation and prosthetic limb control.
[0053] However, this invention is not limited to the embodiments described above. (1) For example, in this embodiment, a Kalman filter was adopted as the state transition model. However, other methods that can take into account the time-series transitions of motion can also be used.
[0054] (2) In this embodiment, the EMG FB input was treated as a pseudo-proprioceptive FB for calculation. Alternatively, by using a sensor capable of actually measuring biological signals from proprioceptors, it is also possible to input real values into the system to train the model and obtain estimated values.
[0055] (3) Figure 1 shows an example of implementing the functions of a biosignal processing device on a standalone computer. However, it is not limited to this, and the functions of the biosignal processing device can also be implemented on a stacked or clustered server, or on a virtual server on the web or cloud. In this way, it is possible to train the model on the cloud and download the trained model to on-premises, without excessively consuming on-premises resources.
[0056] (4) In addition, various modifications are possible to the functional configuration of the biosignal processing device, the content of the computational processing related to the construction of the trained model, and the configuration of various sensors such as electroencephalogram sensors, without departing from the spirit of this invention.
[0057] Embodiments of the present invention have been described in detail in the above disclosure. The above description is merely illustrative in all respects of the present invention. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations may be adopted as appropriate depending on the embodiment.
[0058] In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and materialized without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined.
[0059] <References> [1]Tatsuya Umeda, Tadashi Isa, and Yukio Nishimura. Temporal dynamics of the sensorimotor convergence underlying voluntary limb movement. Proceedings of the National Academy of Sciences, 119(48):e2208353119, November 2022. Publisher: Proceedings of the National Academy of Sciences. [2]W. Wu, MJ Black, Y. Gao, E. Bienenstock, M. Serruya, A. Shaikhoun, and JP Donoghue. 2002. Neural decoding of cursor motion using a Kalman filter. In Proceedings of the 15th International Conference on Neural Information Processing Systems (NIPS'02). MIT Press, Cambridge, MA, USA, 133-140. [Explanation of symbols]
[0060] 1… Processor 2…Sensor interface section 3… Input / Output Interface Section 4…Memory 5…Storage 6... Bus 11...Data acquisition unit 12…ECoG signal processing unit 13…EMG signal processing unit 14…Signal Processing Unit 15…Learning Department 15a... Kalman filter 16…Estimation part 21…ECoG sensor 22…EMG sensor 23...Sensor 31…Monitor 32...Operation unit 51…Program 52…ECoG time series data 53…EMG time series data 54…Force time series data 55...Training data 56…Trained Model 100... Biosignal processing device 200... Subjects.
Claims
1. A brain activity sensor that measures brain activity signals reflecting the brain activity of a subject given a task and outputs time-series data of brain activity, A muscle activity sensor that measures electromyogram signals reflecting the muscle activity of the subject and outputs electromyogram time-series data, A muscle strength sensor that measures the motor output of the subject in relation to the task and outputs time-series data of the motor output, A brain activity signal processing unit that calculates brain activity signal features from the aforementioned brain activity time-series data, An electromyogram signal processing unit that calculates electromyogram signal features from the aforementioned electromyogram time-series data, A muscle force signal processing unit that calculates motor output signal features from the aforementioned motor output time series data, A biosignal processing device comprising: a learning unit that constructs a trained model by training a state transition model, which models time-series motor output and time-series muscle activity as state variables and time-series brain activity as an observed variable, using the brain activity signal features, the electromyogram signal, and the motor output signal features.
2. Furthermore, the biosignal processing device according to claim 1 further comprises an estimation unit that inputs the brain activity signal features to the trained model to estimate motor output.
3. The biosignal processing apparatus according to claim 1, wherein the state transition model is a Kalman filter.
4. The biosignal processing device according to claim 1, wherein the brain activity sensor measures an ECoG (Electrocorticogram) in the motor cortex.
5. The biosignal processing device according to claim 1, wherein the brain activity sensor measures the firing rate of neurons in the motor cortex.
6. The process involves a computer acquiring brain activity time-series data from a brain activity sensor that measures brain activity signals reflecting the brain activity of a subject given a task and outputs brain activity time-series data, and The process by which the computer acquires electromyogram time-series data from a muscle activity sensor that measures electromyogram signals reflecting the muscle activity of the subject and outputs electromyogram time-series data, The process by which the computer obtains the time-series data of motor output from a muscle strength sensor that measures the motor output of the subject related to the task and outputs time-series data of motor output, The process by which the computer calculates brain activity signal features from the brain activity time series data, The process by which the computer calculates electromyogram signal features from the electromyogram time-series data, The process by which the computer calculates the motion output signal features from the motion output time series data, A biosignal processing method comprising the process of constructing a trained model by training a state transition model, which models time-series motor output and time-series muscle activity as state variables and time-series brain activity as an observed variable, using the brain activity signal features, the electromyogram signal, and the motor output signal features of the computer.
7. A program that includes instructions for causing a computer to function as a biosignal processing device according to any one of claims 1 to 5.