Reinforcement learning based adaptive state observation for brain computer interface

Abstract

An adaptive state-observation model based on reinforcement learning (RL) is presented for implementing a brain-machine interface (BMI) for decoding brain signals to determine motor actions and control a machine to perform the motor actions. In the model, the brain signals are processed by a neural network (NN) for applying a nonlinear mapping defined by NN weights to the brain signals to thereby produce transformed brain signals. The NN learns the nonlinear mapping by RL to allow the weights to be adaptively and continuously updated to follow the nonlinearities and nonstationarities of the brain signals. The transformed brain signals are processed by a Kalman filter (KF) to produce control signals for controlling the machine to perform the motor actions, thereby utilizing the KF to provide smooth generation of the control signals while reducing adverse effects of the nonlinearities and nonstationarities of the brain signals on the KF.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 220,994, filed July 12, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0003] List of abbreviations

[0004] AGREL Attention Gated Reinforcement Learning

[0005] BC brain control

[0006] BMI Brain-Computer Interface

[0007] EEG

[0008] fMRI (functional magnetic resonance imaging)

[0009] KF Kalman filter

[0010] NN Neural Network

[0011] MC manual control

[0012] RL (Reinforcement Learning)

[0013] SD Sprague Dowry Technical Field

[0014] This invention generally relates to a brain-machine interface (BMI) for decoding brain signals from a subject and controlling a machine to perform motor actions desired by the brain signals. The BMI can be used to control the movement of a neural prosthesis using brain signals from a paralyzed patient to restore motor function. Furthermore, this invention particularly relates to using a neural network with RL and KF to achieve smooth, adaptive state estimation of continuous control processes executed in nonlinear, non-stationary systems encountered in decoding the brain signals and generating control signals for controlling the machine. Background Technology

[0015] BMI establishes a direct communication pathway between the subject's brain and an external device. BMI collects noisy signals from neurons in the brain and estimates motor intentions or desired movements based on these signals. In one important practical application, this estimated motor intention can be used to control robots to perform mechanical actions, thereby assisting people with motor disabilities. In another application related to the gaming industry, the estimated movement intention can be used to control virtual actors in computer games.

[0016] Signal processing algorithms play a crucial role in BMI. As a commonly used state-observation model, Kalman filtering (KF) has been employed to decode motion intentions into states derived from high-dimensional observations of multiple neural firing activities. These motion states evolve over time, as described by a linear state model using Kalman filtering, which reflects how neuronal firing adjusts motion in the presence of Gaussian noise. The implementation of KF well accounts for the gradual changes in continuous brain states and is therefore particularly suitable for brain-controlled tasks where subjects continuously adjust their brain states to control an external robot. However, applying state-observation models to BMI is challenging due to the nonlinear and non-stationary nature of the nervous system.

[0017] State-observation models are useful tools for accurate and stable state estimation of a system under consideration. Kalman filtering (KF) is a commonly used linear model. In Kalman filtering, the state transition function shows how the state evolves. The observation function describes the linear mapping between the state and observations in the presence of Gaussian noise. KF exhibits optimal performance for stationary linear systems. However, if the system is nonlinear, Kalman filtering can only approximate the system using a linear mapping. Therefore, Kalman filtering may lead to poor performance. Furthermore, in nonstationary systems, if the state-observation mapping differs from the initial assumptions in the training data, KF cannot maintain good performance with fixed parameters. A recalibration process is required to collect the latest data, which is time-consuming.

[0018] RL is an alternative algorithm for adaptive state estimation. As an advantage, RL can non-linearly transform the observations to change the state. The transformation is treated as an action in RL. If the action drives the state closer to the goal, the observation-state mapping is reinforced by a reward signal; otherwise, the mapping is penalized. The reward signal can be continuously generated during use, allowing the RL model to adaptively update its parameters to follow non-stationary systems. However, one drawback of RL is that it does not include the relationship between the previous and current states, resulting in non-smooth state estimations. When using BMI to control neural prostheses in healthcare applications, non-smooth state estimations often lead to unnatural movements of the prosthesis.

[0019] There is a need in the art for an improved technique used in BMI for adaptive state estimation of nonlinear, nonstationary systems, while smoothing the state estimation. Summary of the Invention

[0020] The mathematical equations referenced in this "Summary of the Invention" section can be found in the "Detailed Description" section.

[0021] A first aspect of the present invention is to provide a computer-implemented method for controlling a machine to perform a motor action determined by brain signals from a subject.

[0022] In this method, a neural network (NN) processes the brain signal to apply a nonlinear mapping defined by multiple weights of the NN to the brain signal, thereby generating a transformed brain signal. Furthermore, multiple weights are updated via a reductive reasoning (RL) process, allowing the NN to learn the nonlinear mapping through RL, enabling adaptive and continuous updates of the nonlinear mapping to follow the nonlinearity and nonstationarity of the brain signal. The transformed brain signal is then processed by a kinematic flow processor (KF) to generate control signals for controlling the machine to perform motor actions. Advantageously, the KF is used to provide smooth generation of the control signals while reducing the adverse effects of the nonlinearity and nonstationarity of the brain signal on the KF during control signal generation.

[0023] In the RL process, the transformed brain signal is equated to multiple scores, each associated with a number of candidate motor actions competing to become a motor action. Each score for a corresponding candidate motor action indicates the probability that the corresponding candidate motor action is the motor action desired by the brain signal. Multiple weights are calculated based on at least the multiple scores. The neural network (NN) is then updated using these calculated weights to configure a nonlinear mapping. Advantageously, in calculating the multiple weights and updating the NN using them, "equating the transformed brain signal to multiple scores" guides the nonlinear mapping to follow the nonlinearity and non-stationarity of the brain signal, while allowing RL to be applied to NN learning.

[0024] In some embodiments, calculating multiple weights based on at least multiple scores includes: calculating multiple probabilities associated with multiple candidate motor actions from the multiple scores, wherein each probability associated with a corresponding candidate motor action is a probability that "the corresponding candidate motor action is the motor action desired by the brain signal"; selecting a winner from the multiple candidate motor actions that competes to become the desired motor action based on the multiple probabilities; determining a reward for selecting the winner as the motor action based on whether the winner is actually the desired motor action; and calculating multiple weights based at least on the reward.

[0025] In some embodiments, P(a) is calculated by equation (10). t =k′), P(a t =k′) is the probability that "the k′-th candidate movement among multiple candidate movement movements is the movement movement expected by the brain signal", where N is the total number of candidate movement movements among the multiple candidate movement movements, y t = [y1, ..., y N ] T It is the transformed brain signal, where y k For y tThe k-th component, and α≥0 is a control parameter.

[0026] In some embodiments, the winner is selected as the one in P(a t =k′), k′=1,...,N, all N probability values ​​from P(a t =k * The kth * Candidate movement actions.

[0027] In some embodiments, if the kth * If each candidate motor action is the motor action desired by the brain signal, then r t By r t =1 is given; if not, then r t By r t =0 is given, r t Because of the selection of the kth * The reward obtained by selecting one candidate movement action as the movement action.

[0028] In some embodiments, the neural network (NN) is a three-layer NN comprising an input layer, a hidden layer, and an output layer. The input layer has a D-type sensor for receiving brain signals. z Each node, and the brain signal also has D z The hidden layer has J hidden units. The output layer has N nodes for outputting the transformed brain signal. The transformed brain signal has N components. Furthermore, multiple weights are calculated using equations (13) and (14), where: w ij It is the weight from the i-th node of the input layer to the j-th hidden unit. From the j-th hidden unit to the k-th hidden unit of the output layer * The weights of each node; γ is the learning rate; δ is the weights of the nodes given by δ = r t -P(a t =k * The error function is calculated; f(δ) is the error spread function given by equation (12); z ti It is z t The i-th component, z t The brain signal is obtained at time step t; and h j It is the value of the j-th hidden unit, given by equation (8).

[0029] For example, the control signal is calculated as x given by equation (20). t|t , where: x t|t It is the posterior estimate of the mean of state x at time step t; x t|t-1 It is a priori estimate of the average value of state x at time step t; K is the Kalman gain given by equation (16), where P t|tIt is the posterior estimate of the covariance of state x at time step t, and R is the covariance of y. t The covariance matrix of the Gaussian noise term. (Term y) t It is the transformed brain signal generated by NN at time step t.

[0030] A second aspect of the invention provides a system for capturing brain signals of a subject and performing motor actions determined by the brain signals. The system includes a sensing device, a machine, and a computer. The sensing device is used to capture the subject's brain signals. The sensing device may be an EEG sensing device, an fMRI device, etc. The sensing device may even be multiple electrodes implanted in the subject's brain. The machine is used to perform the motor actions. The machine may be a prosthesis, a second computer configured to generate motor actions on virtual objects for virtual reality applications, etc. The computer is configured to perform a computational process according to any embodiment of the methods disclosed above, processing the brain signals to determine the motor actions and controlling the machine to perform the motor actions.

[0031] A third aspect of the present invention provides a BMI device for capturing brain signals of a subject and controlling a machine to perform a motor action determined by the brain signals. The BMI device includes a sensing device and a computer. The sensing device is used to capture the brain signals of the subject. The sensing device may be an EEG sensing device, an fMRI device, etc. The computer is configured to perform a calculation process according to any embodiment of the methods disclosed above, processing the brain signals to determine a motor action and controlling the machine to perform the motor action.

[0032] Other aspects of this disclosure are disclosed as shown in the following embodiments. Attached Figure Description

[0033] Figure 1 A schematic diagram is depicted of the disclosed algorithm for estimating the state of an RL-based adaptive state observation model, as considered and disclosed in this invention.

[0034] Figure 2 The rat lever pressing task used in the experiment to obtain the experimental results is shown, in which Figure 2 (a) shows the physical movement trajectory of the rat in MC mode. Figure 2 (b) Describes the transition motion state in BC mode.

[0035] Figure 3A The total number of successful trials on the same day, obtained from experiments used to test the following three algorithms, is plotted: Kalman filtering; algorithms with online updates as disclosed herein; and algorithms with fixed parameters.

[0036] Figure 3BThe response times for different dates were plotted in experiments conducted using the three algorithms.

[0037] Figure 3C The inter-trial time of brain-triggered responses to the three algorithms, as observed in experiments using the three algorithms, was plotted.

[0038] Figure 4 This is a schematic diagram of an exemplary system that uses the disclosed RL-based adaptive state observation model to decode brain signals and subsequently execute the motor actions desired by the brain signals, wherein the model utilizes a neural network (NN) and then a kinematic flow (KF) to process the brain signals, and multiple weights of the NN are updated by an RL process.

[0039] Figure 5 An exemplary embodiment of an RL process is described for updating multiple weights and thereby configuring a nonlinear mapping provided by a neural network when processing brain signals.

[0040] Figure 6 Some embodiments for calculating multiple weights are described. Detailed Implementation

[0041] Unless otherwise stated, “subject” in this article is to be interpreted as a living individual, whether human or animal.

[0042] As used in this article, "brain signal" is synonymous with neural signal. Essentially, brain signal comprises electrical potential recordings that may arise from a mixture of underlying components of brain activity. Typically, brain signals are multidimensional. For example, brain signals can be obtained via EEG or fMRI.

[0043] Unless otherwise stated, each of the following terms is understood to be an entity with any positive dimension: “signal,” “state,” “observation,” and “noise.” That is, the entity can be a multidimensional entity with multiple components, or it can be a one-dimensional entity with one component. This invention relates to decoding brain signals. In the context of decoding brain signals (such as EEG signals), brain signals are typically multidimensional.

[0044] In this specification and the appended claims, unless otherwise stated, signals, states, observations and noise are understood as vectors.

[0045] In this specification and the appended claims, "prior estimate" and "posterior estimate" (as verbs) are interpreted as technical terms used in Kalman filtering. In Kalman filtering, the prior estimate is the predictive estimate, and the posterior estimate is the update estimate. "Prior estimate" and "posterior estimate" (as nouns) are interpreted similarly.

[0046] As used herein, "motion action" is understood to be an action that causes a pre-selected object to move. In the specification and claims herein, the pre-selected object can be a physical object or a virtual object in a computer science context. In one example, the physical object can be a neural prosthesis, such that the action causes the prosthesis to enter into physical motion and move along a meaningful trajectory. In another example, the virtual object can be a computer-generated artificial arm presented to a computer game player via a virtual reality gaming station.

[0047] As used in this article, the symbol "←" is the assignment operator, which means that the variable on the left side of the operator is assigned the value calculated by the expression shown on the right side of the operator.

[0048] This paper discloses a method and system for decoding brain signals of a subject to determine a desired motor action and controlling a machine to perform that action. Advantageously, the disclosed method and system utilize an RL-based adaptive state observation model implemented by combining RL and Kalman filtering. Before detailing various embodiments of the invention, the theoretical development of the RL-based adaptive state observation model is provided.

[0049] A. Theoretical Development

[0050] The RL-based adaptive state observation model advantageously combines RL with a linear state transition model. A linear state transition function is used in the RL-based adaptive state observation model to provide a smooth estimate of state evolution. Simultaneously, based on current observations, RL is employed to nonlinearly refine the posterior state estimate. Refinement is considered an action in RL, and a reward signal is provided for this action. During continuous state estimation, the parameters used in RL are adaptively changed according to the reward signal. In this way, a timely updated nonlinear RL model is obtained to fine-tune the final state prediction, which can potentially follow changes in the state-observation mapping during use in nonlinear, non-stationary systems.

[0051] The linear state transition model used in the disclosed RL-based adaptive state observation model is developed by considering Kalman filtering. Kalman filtering is a commonly used linear state observation model. The detailed process of Kalman filtering is as follows: The state at the current time step t is represented as a vector. Where D x It is the dimension of the state. The observation is represented as Where D z This is the dimension of the observation. The state and observation function are given by the following equation.

[0052] x t =Fx t-1 +q (1)

[0053] and

[0054] z t =Hx t +p (2)

[0055] Where: F is the state transition matrix; H represents the linear mapping from the state to the observation; and q and p are the noise of the state transition and the noise of the observation model, respectively. Assume q and p are independent zero-mean Gaussian white noise with covariance matrices Q and R, respectively. and

[0056] At the current time t, the posterior mean and covariance of the states from the previous time step are respectively denoted as x. t-1|t-1 and P t-1|t-1 From the state transition equation given in equation (1), the prior update of the state can be obtained as:

[0057] x t|t-1 =Fx t-1|t-1 (3)

[0058] and

[0059] P t|t-1 =FP t-1|t-1 F T +Q, (4)

[0060] Where x t|t-1 and P t|t-1 These represent the prior estimates of the mean and covariance of the state at the current time, respectively.

[0061] After obtaining the prior estimate of the state, since the observations are also obtained at the same time t, the mean and covariance of the state need to be adjusted posteriorly. The calculation steps are shown in the following equation:

[0062] K = P t|t-1 H T HP t|t-1 H T +R) -1 (5)

[0063] x t|t =x t|t-1 +K(z t -Hx t|t-1 (6)

[0064] and

[0065] P t|t =(I-KH)P t|t-1 (7)

[0066] Where: K is the Kalman gain, which is a coefficient used to determine the degree of confidence in the observation; x t|t and P t|t These are the posterior estimates of the mean and covariance of the current state, respectively.

[0067] The entire process of Kalman filtering for state estimation is summarized as follows. First, initial conditions are set for the mean and covariance of the state, denoted as x0 and P0, respectively. At each time step, prior updates calculated by equations (3) and (4) are performed first. Then, the current observation z is used... t The posterior update is performed using equations (5)-(7). Posterior state x t|t The average value is the final output from the state described by KF.

[0068] In this invention, AGREL is used as the RL scheme. For an explanation of AGREL technology, please refer to, for example, PRROELFSEMA and A. VAN OOYEN's "Attention-Gated Reinforcement Learning of Internal Representations for Classification", Neural Computation, 17, 2176-2214 (2005). To illustrate, the RL scheme uses a three-layer artificial neural network to approximate any nonlinear mapping. Typically, neural networks with three or more layers can be used to approximate nonlinear functions. The input layer has D... z The system has J nodes, each representing a component of the observation. The hidden layer has J nodes, each with a sigmoid activation function. Each of the J nodes in the hidden layer is called a hidden unit. The output layer has N nodes, each representing a possible action for the state. Each of the N nodes in the output layer is called an output unit. The softmax policy probabilistically selects the final action based on the action value. If the action brings the state closer to the goal, a reward of r is given. t =1 is given by the algorithm, otherwise r is given. t =0. Then the reward signal is used to update the weights connected to the selected action. The mathematical explanation is given below.

[0069] For the current observation z t The action selection process involves the following equations:

[0070]

[0071]

[0072]

[0073] The value of the hidden unit is calculated by equation (8), where h j It is the output of the j-th hidden unit; w ij Z represents the weights from the i-th input unit to the j-th hidden unit; and z ti It is the current neural discharge vector z t The i-th component. The action value is calculated by equation (9), where y k It is the value of the k-th output unit. Quantity v jk It represents the weights from the j-th hidden unit to the k-th output unit. The quantity P(a) t =k′) represents the probability of selecting the k′-th action. The value α≥0 is a parameter used to control the probability concentration of the softmax policy. If α is very large, the probability is more concentrated on large action values. If α is very small, the probability for each action becomes similar, making the RL scheme more focused on exploring different actions. Finally, one action is selected from N possible actions. * Let it be an index, such that the k-th index is... * The action is the selected action. Typically, k * It is based on some predefined rules that the probability P(a) is calculated from N possible actions by equation (10) among N probabilities. t =k * The predefined rule is determined as follows. An example of the predefined rule is provided below. During the training phase, k... * The results of the Monte Carlo simulation are randomly determined based on the N probabilities. During the testing phase, k * It is the maximum likelihood estimate, such that P(a) t =k * It is the largest among the N probabilities.

[0074] After an action is selected, if the action moves the actuator toward the target, the instantaneous reward is determined by r. t =1 is given, otherwise r t =0. The error signal δ and the error spread function f(δ) are given by the following equation:

[0075] δ=r t -P(a t =k * (11)

[0076] and

[0077]

[0078] Quantity P(a) t =k* ) can be considered as taking action k * The expected reward. The error signal δ represents the difference between the actual reward and the expected reward. The error spread function f(δ) is used to amplify the effect of unexpected rewards. When the action probability is low and it leads to a reward, δ is close to 1 and f(δ) takes a large value. The error spread function f(δ) is then used in the weight update to improve the learning efficiency of unexpected reward actions in simulated animal learning. The relevant equations are given by the following:

[0079]

[0080] and

[0081]

[0082] Where γ is the learning rate, w ij It is the weight from the i-th node in the input layer to the j-th hidden unit, and From the j-th hidden unit to the k-th hidden unit of the output layer * The weights of each node are updated. Note that only the weights relevant to the selected action are updated.

[0083] The present invention, which utilizes a dynamic observation model to achieve adaptive state estimation via RL, is described below. In this invention, the state transition function is the same as in equation (1), such that smooth state prediction is inherited from the linear state transition. Specifically, the observation function in equation (2) is modified as follows. Instead of using a linear mapping from the observation to the state, the state is generated nonlinearly from the observation using the AGREL technique, which is represented as g t (z t The nonlinearity of the observation function is shown. The modified observation function is given by the following equation:

[0084] g t (z t )=x t +p (15)

[0085] Where p is the estimated noise of the RL model.

[0086] After the observation function changes, the posterior update process is also altered. Then, z in equations (5)-(7) is... t H and g are respectively replaced by g. t (z t And the identity matrix I. The final update equation is shown as:

[0087] K = P t|t-1 (P t|t-1 +R) -1 (16)

[0088] xt|t =x t|t-1 +K(g t (z t )-x t|t-1 (17)

[0089] and

[0090] P t|t =(IK)P t|t-1 (18).

[0091] Note that since an artificial neural network is used in the RL model to generate the nonlinear mapping, we can conclude that:

[0092] y t =g t (z t (19)

[0093] Where y t = [y1, ..., y N ] T It is the output of NN.

[0094] The algorithm disclosed above for estimating the state of an RL-based adaptive state observation model is as follows: Figure 1 The work shown, Figure 1 A schematic diagram illustrating the structure of the disclosed algorithm is provided. Initial values ​​for x0 and P0 are first set. For each time step, the posterior motion state x is calculated using equations (3), (4), and (16)-(18). t|t Then, a reward signal r can be obtained based on whether the state is closer to the target. t Then, the reward signal is used to update the parameters of AGREL as shown in equations (13) and (14). Nonlinearity g t (·) also changes accordingly. In this way, the disclosed algorithm may be able to follow the changes in the state-observation mapping during the control process of nonlinear, non-stationary systems.

[0095] The advantages of the disclosed algorithm are summarized as follows. First, the disclosed algorithm combines a nonlinear RL model with a linear state transition model, thereby providing smooth and adaptive state estimation for nonstationary systems. Second, this invention leverages the advantage of RL's online parameter adjustment, eliminating the need for recalibration steps on nonstationary systems. This improves time efficiency and enhances the user experience during the use of this invention.

[0096] B. Experimental Verification

[0097] The algorithms disclosed above were tested in BMI using the BC task. BMI is a framework that helps paralyzed individuals regain their motor function by translating their neural signals into motion from an external neural prosthesis. In BMI, a state consists of one or more components selected from the position, velocity, etc., of the neural prosthesis. Observations are neural signals extracted from the subject's brain. The human nervous system is typically nonlinear and, due to neural plasticity, is also nonstationary.

[0098] The published algorithm was tested using a lever pressing task on SD rats. Experimental examples show... Figure 2 middle. Figure 2 The rat control lever pressing task was described, in which Figure 2 (a) shows the physical movement trajectory of the rat in MC mode. Figure 2 (b) Describes the motion states transitioned in BC mode.

[0099] exist Figure 2 (a) illustrates the MC training process. The rat needs to wait in the starting position. When the rat hears a sound cue indicating the start of the trial (10 kHz, 0.9 s), the rat needs to approach the control lever, press the lever, and hold it for a period of time (0.5 s). The trial is then considered successful. A second sound cue (10 kHz, 0.09 s) is presented to the rat, and the rat returns to the starting position to receive a water reward. If the rat does not hold the lever for a sufficient amount of time, or does not press the lever within the specified time, the trial is considered a failure, and the rat does not receive a reward. When a trial ends, after a randomized interval between trials, another sound cue is given to the rat to indicate the start of the next trial.

[0100] While the rats were performing the task, their neural signals were simultaneously recorded. SD rats were implanted with two 16-channel microelectrodes, one in the primary motor cortex (M1) and the other in the medial prefrontal cortex (mPFC). Both electrodes were implanted in the left hemisphere, corresponding to the region opposite in orientation to the rat's forearm used to press the control lever during the task. During the experiment, neural signals were recorded by Plexon (Plexon Corporation, Dallas, Texas) at a sampling frequency of 40 kHz. The neural signals were then passed through a 4-pole Butterworth high-pass filter (500 Hz). Spikes were detected by setting a threshold at -4σ, where σ is the standard deviation of the signal amplitude. The applicant calculated spikes within a non-overlapping window of 100 milliseconds, and the applicant concatenated 7 historical spike windows, which increased the observation dimension D. z=256. Behavioral events in rats were recorded using a behavior system from Lafayette Instruments, Inc. These event signals were synchronized with neural signals from Plexon.

[0101] After rats became proficient in the MC task, they were trained in related BC tasks, which... Figure 2 As shown in (b). In the BC task, there is no physical control lever for the rat to press. The physical trajectory of the rat performing the MC task is transformed into a one-dimensional cursor state. At the start of the experiment, the cursor is assumed to be near 0. As the rat approaches the control lever, the cursor rises from 0 to 1.5, as shown in the first part of curve 210. During the hold on the control lever, the cursor remains near 1.5. When the experiment is successful and the rat returns to receive the reward, the cursor changes from 1.5 to 0, as shown in the second part of curve 220. The parameters of KF are trained based on the transformed cursor trajectory (state) and the simultaneous spike (observation). The rat then begins to use only its neural signals to brain-control the cursor. The details of the rat's BC training are as follows. When the cursor remains within the range (0, 0.75) for a predetermined time, an audio cue (same as in MC) is presented to the rat, indicating the start of the experiment. The rat will attempt to move the cursor into the region (0.75, 1.5). If the cursor is within this region for a given time period (same as in MC), the experiment is considered successful. The successful audio cue (same as in MC) is given to the rat, and the rat receives a water reward. If the cursor does not remain within (0.75, 1.5) for a sufficient period of time, or does not move into that area, the trial is considered a failure, and the rat does not receive a reward. Unlike the MC task, the next trial begins after a random time interval. To begin the next trial in the BC task, the rat must move the cursor to (0, 0.75) and remain there for a predefined period of time. Otherwise, the next trial cannot begin.

[0102] This experiment collected 7 days of neural (observational) and trajectory (state) data from rats as they became proficient in the BC task under the same KF parameters. The disclosed algorithm was then trained using data recorded sequentially from day 1 to day 7. The disclosed algorithm was also tested with fixed parameters after day 1. For reference, a linear KF was also tested. Experimental results are shown in... Figures 3A-3C middle.

[0103] Figure 3AThe total number of successful trials on the same day was plotted during the rat lever pressing task experiment. The horizontal axis represents different days. The vertical axis shows the number of successful trials. Results using KF, the published algorithm, and the published algorithm with fixed parameters are shown in left column 310, right column 330, and middle column 320, respectively. According to the experimental paradigm, KF had approximately 200 successful trials. On day 1, as it was the first day of training, the number of successful trials was similar for all three methods. After day 1, the middle column 320 consistently outperformed the left column 310, demonstrating the advantage of the nonlinear observation-state mapping from the RL model. Figure 3A Among the three columns 310, 320, and 330, the right column 330 has the highest number of successful trials, indicating that the published algorithm is the most effective in capturing non-stationary neural signals on a daily basis.

[0104] Figure 3B The response times observed in the experiments over different number of days are plotted. Response time is the time interval between the start cue and the success moment. A smaller response time means the algorithm can complete the task more efficiently. The bars and whiskers represent the mean and standard deviation of the response time, respectively. The difference between the middle bar 320 and the left bar 310 demonstrates the advantage of RL's nonlinear mapping from neural patterns to states. The significant difference in response time between KF and the filter with RL (fixed) indicates that neural patterns are non-stationary over several days. For the disclosed algorithm with continuous updates (right bar 330), the response time is smaller than the corresponding response times of the other two techniques. Furthermore, the response time is more stable over several days, indicating that the disclosed algorithm with online updates of RL successfully follows the changes in neural patterns.

[0105] Figure 3C The brain-triggered inter-trial times (BTs) for the three algorithms identified in the experiments were plotted. The brain-triggered inter-trial time is the time interval between the success time and the cue to start the next trial. A smaller brain-triggered inter-trial time means the algorithm can trigger the next trial more quickly, making it more likely to complete more trials within a given time. During training, both of the disclosed algorithms (online update / fixed parameters) exhibited smaller and more stable brain-triggered inter-trial times than KF. The brain-triggered inter-trial times for online update and fixed parameters were similar, indicating that the neural patterns between the success moment and the start of the next trial were relatively stable. The non-linear mapping of the RL trained from day one was sufficient to drive the cursor from a success start to trigger the next initiation.

[0106] C. Details of the embodiments of the present invention

[0107] This invention is developed based on the theoretical work and experimental verification disclosed above.

[0108] Figure 4 A schematic diagram of an exemplary system 400 is depicted, which employs an RL-based adaptive state observation model as disclosed above to decode a subject's brain signals 462 to determine the motor action desired by the subject, and the system then executes that motor action. At the core of system 400 is a computer 410 configured with an RL-based adaptive state observation model 418 for decoding brain signals 462 and generating control signals 468 to control a machine 450 to execute the motor action. System 400 acquires the subject's brain signals 462 by sensing the brain signals 462 using a sensing device 440. In practical implementation, the BMI can be formed solely by computer 410, or it can consist of computer 410 and sensing device 440, or it can simply be a software item encoded using the RL-based adaptive state observation model 418.

[0109] A first aspect of the invention is to provide a computer-implemented method for controlling a machine 450 to perform a motor action determined by brain signals 462. The disclosed method is essentially developed based on an RL-based adaptive state observation model 418.

[0110] In this method, brain signal 462 is processed by NN 412 to apply a nonlinear mapping to brain signal 462, thereby generating a transformed brain signal 464. The nonlinear mapping is defined by multiple weights 466 of NN 412. The multiple weights 466 are updated by RL process 413, such that NN 412 learns the nonlinear mapping through RL. Note that the multiple weights 466 are updated continuously over time because brain signal 462 is continuously generated by the subject over time, making brain signal 462 a time series of signal samples, each of which is multidimensional. Once a new sample of brain signal 462 is received, the multiple weights 466 are updated. As a result of the weight update, it allows the nonlinear mapping provided by NN 412 to be adaptively and continuously updated to follow the nonlinearity and nonstationarity of brain signal 462. The transformed brain signal 464, as the output of NN 412, is processed by KF 415 to generate control signals 468 for controlling the machine to perform motor actions. The cascaded connection of NN 412 and KF 415 advantageously utilizes KF 415 to provide smooth generation of control signal 468, while NN 412 prevents the nonlinearity and non-stationarity of brain signal 462 from adversely affecting KF 415 during the generation of control signal 468.

[0111] Figure 5 An exemplary embodiment of the RL process 413 is described.

[0112] Note that one function of the transformed brain signal 464 is to guide KF 415 to generate control signal 468, and another function is to be used by RL process 413 to provide feedback to NN 412 by updating multiple weights 466. Also note that, as mentioned above, the output layer of NN 412 has multiple nodes, each representing a possible action. In this regard, the motion action to be performed by machine 450 is selected from multiple candidate motion actions, and the candidate motion actions compete to become the motion action.

[0113] In step 510, the RL process 413 equates the transformed brain signal 464 to multiple scores, each associated with a plurality of candidate motor actions competing to become the motor action. Specifically, each score for a corresponding candidate motor action represents the probability that the corresponding candidate motor action is the motor action desired by brain signal 462. For illustration, each score corresponds to y k And the above probability corresponds to P(a) t =k′), as shown in Equation (10). In step 520, multiple weights 466 are calculated based on at least multiple scores. In step 530, RL process 413 updates NN 412 with the calculated multiple weights 466 to configure the nonlinear mapping.

[0114] It can be assumed that the brain signal 462 and the multiple scores (each score indicating the probability that a candidate motion is said motion) are two signals with different properties, and therefore, these two signals have very different statistical behaviors. By treating the transformed brain signal 464 as multiple scores in the RL process 413, and noting that the transformed brain signal 464 is obtained at the output of NN 412, it can be assumed that "updating NN412 using multiple weights 466 calculated based on multiple scores" potentially drives NN 412 to "absorb" the nonlinearity and nonstationarity of the incoming brain signal 462 in some way so as to generate multiple scores at the output of NN 412. Furthermore, "treating the transformed brain signal 464 as multiple scores" allows RL to be applied to NN 412. Therefore, "equating the transformed brain signal 464 with multiple scores when calculating multiple weights 466 and updating NN 412 using the calculated multiple weights" will guide the nonlinear mapping to follow the nonlinearity and nonstationarity of brain signal 462, and allow RL to be applied to NN learning.

[0115] Figure 6 Some embodiments of step 520 are described.

[0116] In step 610, multiple probabilities associated with multiple candidate motor actions are calculated from multiple scores. As used herein, the multiple probabilities associated with multiple candidate motor actions are understood as follows: each probability associated with a corresponding candidate motor action is the probability that the corresponding candidate motor action is the motor action desired by brain signal 462. In some embodiments, P(a t =k′) is calculated by equation (10), P(a t =k′) represents the probability that the k′-th candidate movement among multiple candidate movements is the movement expected by brain signal 462, where N is the total number of candidate movements among multiple candidate movements, y t = [y1, ..., y N ] T It is the transformed brain signal 464, where y k It is y t The k-th component, and α≥0 is a control parameter.

[0117] In step 620, a candidate motion is selected from a plurality of candidate motions as the winner in the competition to become the desired motion. The selection is based on a plurality of probabilities calculated in step 610. In some embodiments, the winner is selected as the k-th... * There are candidate motion actions such that P(a) t =k * ) is in "P(a t The largest of all N values ​​of “=k′),k′=1,...,N”.

[0118] After completing step 620, step 630 determines the reward for selecting the winner as the movement action. Specifically, the reward is determined based on whether the winner actually performed the desired movement action. In some embodiments, if the k-th... * If each candidate motor action is the motor action desired by the brain signal, then r t By r t =1 is given; if not, then r t =0, r t Because of the selection of the kth * The reward obtained by selecting one candidate movement action as the movement action.

[0119] In step 640, multiple weights 466 are calculated at least based on the reward determined in step 630. According to certain embodiments, the multiple weights 466 are calculated in the specific case where the NN 412 is a three-layer NN, as detailed below. The three-layer NN has an input layer, an output layer, and a hidden layer between the input and output layers. The input layer has a D for receiving brain signals 462. zEach node. Note that brain signal 462 has D in a signal sample obtained at a certain moment. z There are J components. The hidden layer has J hidden units. The output layer has N nodes for outputting the transformed brain signal 464. Therefore, the transformed brain signal 464 has N components from a signal sample obtained at a certain time. ij |i = 1, ..., D z ;j=1,...,J} and {v jk The multiple weights 466 composed of |j=1,...,J;k=1,...,N} are calculated by equations (13) and (14), where: w ij These are the weights from the i-th node in the input layer to the j-th hidden unit; From the j-th hidden unit to the k-th hidden unit of the output layer * The weights of the nodes, k * It is used to indicate the selection of the k-th motion from N candidate motion actions. * A candidate motor action serves as an indicator of the winner in the competition for the motor action expected by brain signal 462; γ is the learning rate; δ is the error function calculated by equation (11); f(δ) is the error spread function given by equation (12); z ti It is z t The i-th component, z t The brain signal 462 was obtained at time step t; and h i It is the value of the j-th hidden unit, given by equation (8).

[0120] Reference Figure 4 Control signal 468 is based on y by KF 415. t Generate, y t This is a signal sample of the transformed brain signal 464 obtained at time step t. For example, the control signal 468 is calculated as x given by the following formula. t|t :

[0121] x t|t =x t|t-1 +K(y t -x t|t-1 (20)

[0122] Where: x t|t It is the posterior estimate of the mean of state x at time step t; x t|t-1 It is a priori estimate of the average value of state x at time step t; and K is the Kalman gain given by equation (16), where P t|t It is the posterior estimate of the covariance of state x at time step t, and R is the covariance of y. tThe covariance matrix of the Gaussian noise term. Note that state x is the state of KF 415. Furthermore, KF 415 is used to model machine 450. Therefore, the state transition matrix F can be formulated based on the characteristics of machine 450.

[0123] A second aspect of the invention is a system 400 for capturing brain signals 462 of a subject and performing motor actions determined by the brain signals 462.

[0124] System 400 includes: a sensing device 440 for capturing brain signals 462 from a subject; a machine 450 for performing motor actions; and a computer 410 configured to perform a computational process of processing the brain signals 462 to determine a motor action and controlling the machine 450 to perform the motor action. Specifically, the computational process is implemented as any embodiment of the method disclosed above in the first aspect of the invention.

[0125] Note that the computer 410 is also configured to receive brain signals 462 from the sensing device 440. The computer 410 can be connected to the sensing device 440 wirelessly or via a wired connection.

[0126] The sensing device 440 can be an EEG sensing device, an fMRI device, etc. The sensing device 440 can even be multiple electrodes implanted in the subject's brain (e.g., if the subject has a severe, irreversible spinal cord injury).

[0127] Machine 450 can be a prosthesis or a second computer configured to generate motion on virtual objects for virtual reality applications.

[0128] A third aspect of the present invention is to provide a BMI device 405 for capturing brain signals 462 of a subject and controlling a machine 450 to perform a motor action determined by the brain signals 462.

[0129] BMI device 405 includes: a sensing device 440 for capturing brain signals 462 of a subject; and a computer 410 configured to perform a computational process of processing the brain signals 462 to determine a motor action and controlling a machine 450 to perform the motor action. The computational process is implemented as any embodiment of the method disclosed above in the first aspect of the invention.

[0130] Note that the computer 410 is also configured to receive brain signals 462 from the sensing device 440. The computer 410 can be connected to the sensing device 440 wirelessly or via a wired connection.

[0131] The sensing device 440 can be an EEG sensing device, an fMRI device, etc.

[0132] This disclosure may be embodied in other specific forms without departing from its spirit or essential characteristics. Therefore, these embodiments are to be considered illustrative rather than restrictive in all respects. The scope of the invention is indicated by the appended claims rather than by the foregoing description, and all variations falling within the equivalent meaning and scope of the claims are therefore intended to be included within the scope of the invention.

Claims

1. A computer-implemented method for controlling a machine to perform a motor action determined by brain signals from a subject, the method comprising: The brain signal is processed using a neural network (NN) to apply a nonlinear mapping defined by multiple weights of the NN to the brain signal, thereby producing a transformed brain signal. The multiple weights are updated through a reinforcement learning (RL) process, enabling the neural network to learn the nonlinear mapping through RL. and The transformed brain signal is processed using a Kalman filter (KF) to generate control signals for controlling the machine to perform the motor action, wherein... The control signal is calculated as given by the following formula. : in: In time step t status x The posterior estimate of the average value of the states, wherein the states x This refers to the state of the KF, which is used to model the machine; In time step t status x Prior estimates of the average; and K It is by The given Kalman gain, where In time step t status x Prior estimates of the covariance, R Is The covariance matrix of the Gaussian noise term in the equation, where In time step t The transformed brain signal generated by the NN.

2. The method according to claim 1, wherein, The RL process includes: The transformed brain signal is equated to multiple scores associated with multiple candidate motor actions, which compete to become the motor action. Each score of the corresponding candidate motor action indicates the probability that the corresponding candidate motor action is the motor action expected by the brain signal. The multiple weights are calculated based on at least the multiple scores; and The neural network is updated using the calculated weights to configure the nonlinear mapping. Thus, when calculating the weights and updating the neural network using the calculated weights, the equivalent step of equating the transformed brain signal with the multiple scores guides the nonlinear mapping to follow the nonlinearity and nonstationarity of the brain signal, while allowing RL to be applied to neural network learning.

3. The method according to claim 2, wherein, The steps of calculating the multiple weights based on at least the multiple scores include: From the multiple scores, multiple probabilities associated with the multiple candidate motor actions are calculated, wherein each probability associated with a corresponding candidate motor action is the probability that the corresponding candidate motor action is the motor action expected by the brain signal; Based on the multiple probabilities, the winner of the competition to become the desired movement action is selected from the multiple candidate movement actions; Determine the reward for selecting the winner as the movement action based on whether the winner is actually the desired movement action; and The multiple weights are calculated based on at least the rewards.

4. The method according to claim 3, wherein: Calculated by the following formula The probability that the k-th candidate motor action among the plurality of candidate motor actions is the motor action expected by the brain signal is: Where N is the total number of candidate motion actions among the plurality of candidate motion actions. It is the transformed brain signal, wherein yes The kth component, and These are control parameters; and The winner was selected as... From all N probability values The Candidate movement actions.

5. The method according to claim 4, wherein, If the first If a candidate motor action is the motor action expected by the brain signal, then Give it, if not, then Give, For the reason that the first was selected The reward obtained by selecting one candidate movement action as the movement action.

6. The method according to claim 5, wherein: The NN is a three-layer NN, including: Having for receiving the brain signals The input layer of each node, the brain signal has One component; have J The hidden layer of each hidden unit; and Having for outputting the transformed brain signal N The output layer of each node, the transformed brain signal has N One component; and The multiple weights are calculated by the following formula: and in: From the first of the input layer i The node to the first j The weights of each hidden unit; From the first j The first hidden unit to the output layer The weight of each node; It is an assignment operator that means the variable on the left side of the operator is assigned the value calculated by the expression shown on the right side of the operator; It is the learning rate; It is by The calculated error function; It is the error spread function, given by the following equation: yes The i One portion, In order to time step t The acquired brain signals; and It is the first j The value of each hidden unit is given by the following formula: 。 7. A system for capturing brain signals of a subject and performing motor actions determined by said brain signals, said system comprising: Sensing device for capturing brain signals from the subject; A machine used to perform the aforementioned motion; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 1.

8. The system according to claim 7, wherein, The sensing device is an electroencephalogram (EEG) sensing device.

9. The system according to claim 7, wherein, The machine in question is a prosthesis.

10. The system according to claim 7, wherein, The machine is a second computer configured to generate the motion actions on virtual objects for virtual reality applications.

11. A system for capturing brain signals of a subject and performing motor actions determined by said brain signals, said system comprising: Sensing device for capturing brain signals from the subject; A machine used to perform the aforementioned motion; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 2.

12. A system for capturing brain signals of a subject and performing motor actions determined by said brain signals, said system comprising: Sensing device for capturing brain signals from the subject; A machine used to perform the aforementioned motion; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 3.

13. A system for capturing brain signals of a subject and performing motor actions determined by said brain signals, the system comprising: Sensing device for capturing brain signals from the subject; A machine used to perform the aforementioned motion; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 4.

14. A system for capturing brain signals of a subject and performing motor actions determined by said brain signals, the system comprising: Sensing device for capturing brain signals from the subject; A machine used to perform the aforementioned motion; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 5.

15. A system for capturing brain signals of a subject and performing motor actions determined by said brain signals, the system comprising: Sensing device for capturing brain signals from the subject; A machine used to perform the aforementioned motion; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 6.

16. A system for capturing brain signals of a subject and performing motor actions determined by said brain signals, the system comprising: Sensing device for capturing brain signals from the subject; A machine used to perform the aforementioned motion; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 1.

17. A brain-computer interface (BMI) device, the BMI device being used to capture brain signals of a subject and control a machine to perform a motor action determined by the brain signals, the BMI device comprising: Sensing devices used to capture the brain signals from the subject; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 1.

18. The BMI device according to claim 17, wherein, The sensing device is an electroencephalogram (EEG) sensing device.

19. A brain-computer interface (BMI) device, the BMI device being used to capture brain signals of a subject and control a machine to perform a motor action determined by the brain signals, the BMI device comprising: Sensing device for capturing brain signals from the subject; and A computer configured to perform a computational process of processing the brain signals to determine the motor action and controlling the machine to perform the motor action according to the method of claim 2.

Images