An electroencephalogram signal decoding device and real-time learning method based on ECRAM array

Abstract

The present invention discloses an electroencephalogram signal decoding device and a real-time learning method based on an ECRAM array. The present invention adjusts the update accuracy of the current layer weight connected to the neuron in the next cycle interval in real time through the activation frequency change of the current layer neuron in the current cycle interval. That is, if the activation frequency rises, it means that the corresponding neuron has insufficient capacity, and it is necessary to improve the update accuracy of the corresponding weight update amount, that is, increase the number of writes. If the activation frequency drops, it means that the corresponding neuron has redundant capacity, and it is necessary to reduce the update accuracy of the corresponding weight update amount, that is, reduce the number of writes, thereby effectively reducing the number of write cycles of the ECRAM array and reducing power consumption. The decoding device proposed by the present invention has sustainable learning ability for the electroencephalogram signal of an individual subject and is suitable for use as a low-power portable terminal device.

Description

Technical Field

[0001] The present invention belongs to the technical field of signal processing of brain-computer interface devices, and specifically relates to an electroencephalogram (ECRAM) array-based electroencephalogram (EEG) signal decoding device and a real-time learning method. Background Art

[0002] The Brain Computer Interface (BCI) system is a method that directly connects the biological brain to external devices without going through the muscle system. The electrode array records the electrical activity of the cerebral cortex, decodes the relevant movement intention, and controls external devices (robotic arms, electric wheelchairs, etc.). Its original intention is to help disabled people interact better with the outside world.

[0003] EEG signals are high-dimensional, noisy, and time-varying, and have individual differences among different subjects. Traditional shallow model decoding methods (such as recursive Bayesian decoders, Kalman filters, etc.) can no longer meet the requirements of high accuracy in motion signal decoding, and more and more deep neural networks are being applied to motion signal processing.

[0004] The original EEG signals collected by the device are in the analog domain. The decoding device based on digital circuit chips needs to convert the signals into digital signals before processing, which consumes a lot of energy and time and cannot meet the requirements of low power consumption and low latency for portable terminal devices. In addition, since the motion signals of the same subject may vary over time during actual use and the EEG signals of different subjects vary from person to person, the decoding device still needs to have the ability to continuously learn the EEG signals of individual subjects after leaving the factory, so the requirements for low power consumption and real-time performance will be more stringent.

[0005] For example, the invention patent application with publication number CN117874524 discloses a learnable brain-computer interface device based on a memristor array, which first processes and decodes the acquired first EEG signal and feeds it back to the subject to generate a second EEG signal, and updates the model parameters in the memristor array according to whether there is an error-related potential signal in the second EEG signal. However, there is still a lack of low-power, accurate algorithms to support on-chip real-time learning of EEG signal decoding devices.

[0006] Publication No. CN114903499A discloses a brain-computer interface signal processing system and device, which is integrated on a chip. The system includes: a signal amplification module, a filtering module, a sampling and holding circuit module and a time-sharing multiplexing control module; the signal amplification module is used to amplify and process the neural signals collected by the electrodes of the brain-computer interface; the filtering module is used to obtain noise-reduced neural signals; the sampling and holding circuit module is used to obtain single-ended signals; the time-sharing multiplexing control module is used to time-sharing multiplex the single-ended signal and a pre-set reference signal, and output a differential analog signal; this patent application reduces the interference of external environment and human factors in the process of neural signal acquisition by integrating the signal amplification module, the filtering module, the sampling and holding circuit module and the time-sharing multiplexing control module on the chip, but the power consumption of the device is still large, and it is still difficult to support on-chip real-time learning. Summary of the invention

[0007] The present invention provides a real-time learning method for electroencephalogram signal decoding based on an ECRAM array. The method can effectively reduce the number of write cycles of the ECRAM array and reduce power consumption while ensuring good accuracy.

[0008] The present invention provides a real-time learning method for electroencephalogram signal decoding based on ECRAM array, comprising:

[0009] The weight matrix of the decoding model is mapped to the conductance matrix of the ECRAM array, and the initial update accuracy provided by the user is distributed to each layer based on the parameter amount of each layer of the decoding model to obtain the benchmark update accuracy of each layer;

[0010] In the real-time training process, a loss function is constructed based on the predicted value and the true intention label output by the decoding model based on the preprocessed EEG signal, and the loss function is used to back-propagate through the gradient descent method to obtain the update amount of the weights of each layer. The real-time training process includes multiple training cycle intervals. After training in the current training cycle interval, an adjustment amount of the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval is obtained based on the activation frequency change of the current layer neurons and the benchmark update accuracy. The adjustment amount is added to the update accuracy of the current layer weight update amount connected to the current layer neurons in the current training cycle interval to obtain the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval. The update accuracy of the weight update amount is the accuracy of writing the conductance update value mapped by the weight update amount into the ECRAM array;

[0011] When the real-time training is completed, the predicted value output by the trained decoding model is parsed through FPGA to obtain the control instruction, and the control instruction is used to control the external mechanical prosthesis to make corresponding actions, thereby realizing the decoding of EEG signals.

[0012] Preferably, the method for obtaining the change in activation frequency of neurons in the current layer in the current training cycle includes:

[0013] The current training cycle interval includes multiple training cycles. When the difference in activation frequency of the current layer neurons obtained in two consecutive training cycles does not exceed the set threshold, the training of one training cycle interval is completed, and the difference between the activation frequency of the current layer neurons obtained in the last training cycle in the training cycle interval and the activation frequency of the current layer neurons obtained in the first training cycle is used as the change in the activation frequency of the current layer neurons in the current training cycle interval.

[0014] Preferably, the activation frequency of the nth neuron in the lth layer during the training cycle is for:

[0015]

[0016] in, represents the activation value of the nth neuron in the lth layer after the activation function in the i-th iteration, M is the total number of iterations in a training cycle, is the indicator function, when The value is 1 when , otherwise it is 0. It represents the number of iterations in which the activation value of the n-th neuron in the l-th layer is not 0 during the training cycle.

[0017] Preferably, based on the connection relationship between the weights of the current layer and the postsynaptic neurons, the neurons of the current layer are grouped, and the average activation frequency of the neurons in each group is calculated. The update accuracy of the corresponding weight update amount is adjusted in the next training cycle interval based on the obtained activation frequency average value.

[0018] Preferably, when the batch gradient descent method is used for training, the neurons of the current layer of multiple samples are grouped based on the connection relationship between the weight of the current layer and the post-synaptic neurons of multiple samples, and the average activation frequency of the neurons in each group of each sample is calculated. The average activation frequency of multiple samples corresponding to each weight is averaged again to obtain the batch activation frequency average value, and the update accuracy of the update amount of the corresponding weight is adjusted according to the obtained batch activation frequency average value.

[0019] Preferably, after training in the current training cycle interval, the update accuracy adjustment amount of the current layer weight update amount connected to the current layer neurons in the next training cycle interval is obtained based on the activation frequency change amount of the current layer neurons and the benchmark update accuracy, and the update accuracy adjustment amount of the weight update amount of the lth layer for the pth or pth group of neurons is obtained based on the activation frequency change amount of the current layer neurons and the benchmark update accuracy. for:

[0020]

[0021] in, is the activation frequency change of the pth or pth group of neurons in the lth layer in the current training cycle, k l Update the accuracy of the baseline for layer l.

[0022] Preferably, based on the parameter amount of each layer of the decoding model, the overall update accuracy provided by the user is distributed to each layer to obtain the benchmark update accuracy of each layer, wherein the benchmark update accuracy k of the lth layer is l for:

[0023] k l =round((1-α l )×K)

[0024]

[0025] Among them, K is the initial update precision provided by the user, round(·) is the rounding function, and Param l is the parameter of the lth layer, count(Param l ) is the parameter count of the lth layer, is the sum of all layer parameters, L is the number of layers, and j is the index of the layer.

[0026] Preferably, the conductivity update value ΔG mapped to the weight update amount is:

[0027]

[0028] W2G max =G max -G min

[0029] W2G min =G min -G max

[0030] Among them, ΔW is the update amount of weight, W2G max and W2G min are the maximum and minimum values ​​that can be theoretically represented after mapping the weight matrix W to the two ECRAM arrays, respectively. max and W min are the maximum and minimum values ​​of the current layer weight, G max and G min are the maximum and minimum values ​​of the device within the conductivity range, respectively.

[0031] Preferably, the decoding model is a deep neural network model, and the ECRAM array is used to map a weight matrix of the decoding model.

[0032] The present invention also provides an electroencephalogram signal decoding device based on an ECRAM array, comprising:

[0033] A signal acquisition module, used for preprocessing the received EEG signals;

[0034] A signal processing module, used for mapping the weight matrix of the decoding model into the conductance matrix of the ECRAM array, and allocating the initial update precision provided by the user to each layer based on the parameter quantity of each layer of the decoding model to obtain the benchmark update precision of each layer;

[0035] In the real-time training process, a loss function is constructed based on the predicted value and the true intention label output by the decoding model based on the preprocessed EEG signal, and the loss function is used to back-propagate through the gradient descent method to obtain the update amount of the weights of each layer. The real-time training process includes multiple training cycle intervals. After training in the current training cycle interval, an adjustment amount of the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval is obtained based on the activation frequency change of the current layer neurons and the benchmark update accuracy. The adjustment amount is added to the update accuracy of the current layer weight update amount connected to the current layer neurons in the current training cycle interval to obtain the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval. The update accuracy of the weight update amount is the accuracy of writing the conductance update value mapped by the weight update amount into the ECRAM array;

[0036] When the real-time training is completed, the predicted value output by the trained decoding model is output to the FPGA;

[0037] The external device control module includes an FPGA, which obtains control instructions through FPGA analysis and controls the external mechanical prosthesis to perform corresponding actions through the control instructions.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention adjusts the update accuracy of the current layer weight update amount connected to the neuron in the next cycle interval in real time through the change in the activation frequency of the current layer neuron in the current cycle interval. That is, if the activation frequency increases, it means that the capacity of the corresponding neuron is insufficient, and it is necessary to improve the update accuracy of the corresponding weight update amount, that is, increase the number of writes; if the activation frequency decreases, it means that the capacity of the corresponding neuron is redundant, and it is necessary to reduce the update accuracy of the corresponding weight update amount, that is, reduce the number of writes, thereby effectively reducing the number of write cycles of the ECRAM array and reducing power consumption.

[0040] Due to the natural advantages of analog chips in processing analog EEG signals, combined with the EEG signal decoding method based on ECRAM array provided by the present invention, the decoding device proposed by the present invention has sustainable learning capabilities for the EEG signals of individual subjects and is suitable for use as a low-power portable terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of a training method provided for a specific embodiment of the present invention;

[0042] Figure 2 A schematic block diagram of an electroencephalogram signal decoding device provided in a specific embodiment of the present invention;

[0043] Figure 3 An example diagram of an application scenario provided for a specific embodiment of the present invention;

[0044] Figure 4 Energy efficiency comparison diagram of full precision, 8-bit fixed precision and mixed precision implemented by the method provided by the present invention provided in a specific embodiment of the present invention;

[0045] Figure 5 A speedup comparison diagram of full precision, 8-bit fixed precision and mixed precision implemented by the method provided by the present invention provided in a specific embodiment of the present invention;

[0046] Figure 6 This is a comparison chart of the accuracy of the full precision, 8-bit fixed precision and mixed precision achieved by the method provided by the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on it.

[0048] like Figure 1 As shown, a specific embodiment of the present invention provides a real-time learning method for EEG signal decoding based on an ECRAM array, comprising:

[0049] (1) Constructing the baseline update accuracy of each layer: Mapping the weight matrix of the decoding model to the conductance matrix of the ECRAM array, and distributing the initial update accuracy provided by the user to each layer based on the parameter quantity of each layer of the decoding model to obtain the baseline update accuracy of each layer.

[0050] The present invention uses an ECRAM array to deploy the weight matrix of the decoding model, which is a deep convolutional neural network, and receives the preprocessed simulated EEG signal as the input of the decoding model to learn or decode the finger movement intention of the individual subject. Among them, mapping the model weight matrix to the conductance value of the ECRAM array involves two situations: the first is initialized as a random network, at which time usually no operation is required; the second is mapping the pre-trained model, at which time the ECRAM device needs to be written. Since the device conductance value is always positive, a weight matrix is ​​mapped to two ECRAM arrays, representing positive weights and negative weights respectively, and the relationship is

[0051] W=W pos -W neg

[0052] Among them, W is an arbitrary weight matrix of the model, W pos is the matrix to be mapped into the ECRAM array representing positive weights, W neg is the matrix to be mapped into the ECRAM array representing negative weights, and W pos and W neg All elements in are positive. pos Mapping to the conductivity matrix as an example (W neg The process is as follows:

[0053]

[0054] Among them, W max and W min Represents the maximum and minimum values ​​of the weights of a specific layer of the model, which are usually determined by the initialization method. max and G min Represent the maximum and minimum values ​​of the device within the conductivity range, respectively.

[0055] The specific embodiment of the present invention distributes the overall update accuracy provided by the user to each layer based on the parameter amount of each layer of the decoding model to obtain the benchmark update accuracy of each layer, wherein the benchmark update accuracy k of the lth layer is l for:

[0056] k l =round((1-α l )×K)

[0057]

[0058] Among them, K is the initial update precision provided by the user, round(·) is rounding, Param l is the parameter of the lth layer, count(Param l ) is the parameter count of the lth layer, is the sum of all layer parameters, L is the number of layers, and j is the index of the layer.

[0059] The specific embodiment of the present invention provides l The larger the value, the more parameters the layer has. It is believed that the layer as a whole has better robustness to the write error of the device, so a relatively low update accuracy is pre-allocated. l The smaller it is, the fewer parameters the layer has. It is believed that the layer as a whole has worse robustness to device write errors, so a relatively high update accuracy is pre-allocated.

[0060] In a specific embodiment, the number of parameters in the third layer of the decoding model is 2359296, the total number of model parameters is 5672576, K=8, then α3=0.416, k3=5, in the ECRAM device selected in the specific embodiment of the present invention, the 5-bit update accuracy corresponds to the device needs to be repeatedly written 3 times. Therefore, the update accuracy of each layer of the model can be preliminarily pre-assigned, and the number of writes to each layer of the ECRAM device can be determined.

[0061] (2) During real-time training, the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle is adjusted based on the activation frequency change of the current layer neurons and the benchmark update accuracy:

[0062] In the real-time training process, a loss function is constructed based on the predicted value and the true intention label output by the decoding model based on the preprocessed EEG signal, and the loss function is used to back-propagate and update the weights through the gradient descent method to obtain the update amount of the weights of each layer. The real-time training process includes multiple training cycle intervals. After training in the current training cycle interval, an adjustment amount of the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval is obtained based on the activation frequency change of the current layer neurons and the benchmark update accuracy. The adjustment amount is added to the update accuracy of the current layer weight update amount connected to the current layer neurons in the current training cycle interval to obtain the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval. The update accuracy of the weight update amount is the accuracy of writing the conductance update value mapped by the weight update amount into the ECRAM array.

[0063] In the training process of the specific embodiment of the present invention, it is necessary to calculate the loss function of the model prediction output and the true intention label, and use the back propagation algorithm to calculate the update gradient of the neural network model parameters to complete an iteration of the ECRAM array conductance value. When updating the ECRAM array conductance value, the activation frequency change law of the postsynaptic neuron corresponding to the model weight is used to adaptively adjust the writing accuracy and writing times of each device.

[0064] Specifically, for the finger motion trajectory prediction task, the true intention label is a 2×5×4 binary tensor, where the first dimension (2) indicates the left hand and the right hand, the second dimension (5) represents the five fingers, and the third dimension (4) represents four motion trajectories (flexion, extension, abduction, and adduction). A value of 0 indicates that there is no motion intention for a certain action at this position, and a value of 1 indicates that there is a motion intention for a certain action at this position. Further, the back propagation algorithm is used to calculate the update gradient and update amount of the neural network model parameters.

[0065] Specifically, the conductance update value ΔG mapped to the weight update amount provided in the specific embodiment of the present invention is:

[0066]

[0067] W2G max =G max -G min

[0068] W2G min =G min -G max

[0069] Among them, ΔW is the update amount of weight, W2G max and W2G min are the maximum and minimum values ​​that can be theoretically represented after mapping the weight matrix W to the two ECRAM arrays, respectively. max and W min are the maximum and minimum values ​​of the current layer weight, G max and G min are the maximum and minimum values ​​of the device within the conductivity range, respectively.

[0070] A specific embodiment of the present invention groups the neurons of the current layer based on the connection relationship between the weights of the current layer and the post-synaptic neurons, calculates the average activation frequency of the neurons in each group, and adjusts the update accuracy of the corresponding weight update amount in the next training cycle interval based on the obtained activation frequency average value.

[0071] Specifically, for the convolutional layer, one convolution kernel corresponds to one channel of the output feature map, so all neurons in the channel are grouped together and the average activation frequency is calculated. For the fully connected layer, one row of the weight matrix corresponds to one output neuron, so each neuron is an independent group.

[0072] When using the batch gradient descent method to train a neural network, the activation frequencies inferred from all samples in the batch are averaged. For the convolutional layer, the average activation frequency over the feature map (batch, height, width) dimension is calculated as the activation frequency of the group of neurons; for the fully connected layer, the average activation frequency over the (batch) dimension is calculated as the activation frequency of the neuron.

[0073] In one embodiment, the average activation frequency is calculated by grouping neurons based on the weights of each layer of the neural network model and the connection relationship between postsynaptic neurons. For example, the third layer of the decoding model is a convolutional layer, and the dimension of the 124th convolution kernel of the filter of this layer is (256, 3, 3). Then the two-dimensional matrix of neurons of the 124th channel of the corresponding output feature map of the third layer should be divided into a group, and the activation frequencies of all neurons in the matrix should be averaged to regulate the update accuracy of the 2304 (=256×3×3) weights of the 124th convolution kernel.

[0074] For example, the 7th layer of the decoding model is a fully connected layer, and the 12th row of the weight matrix with dimension (100,5120) is connected to the 12th output neuron of the 7th layer. The activation frequency of this single neuron regulates the update accuracy of the 5120 weights in the 12th row.

[0075] When the batch gradient descent method is used to train the decoding model, the activation frequencies inferred from all samples in the batch are averaged: for example, there are 64 samples in a batch. As mentioned above, for the average activation frequency of the 124th channel of the output feature map of the third layer, the activation frequencies obtained on the 64 samples should be further averaged as the activation frequency of this group of neurons; for the 12th output neuron of the seventh layer, the activation frequencies obtained on the 64 samples should be further averaged as the activation frequency of the neuron.

[0076] In a specific embodiment, a method for obtaining a change in activation frequency of neurons in a current layer in a current training cycle includes:

[0077] The current training cycle interval includes multiple training cycles. When the difference in activation frequency of the current layer neurons obtained in two consecutive training cycles does not exceed the set threshold, the training of one training cycle interval is completed, and the difference between the activation frequency of the current layer neurons obtained in the last training cycle in the training cycle interval and the activation frequency of the current layer neurons obtained in the first training cycle is used as the change in the activation frequency of the current layer neurons in the current training cycle interval.

[0078] In one embodiment, when the activation frequency of the neuron group of the 124th channel of the output feature map of the 3rd layer of the decoding model is 0.52 in training cycle 1, 0.45 in training cycle 2, 0.59 in training cycle 3, 0.66 in training cycle 4, and 0.655 in training cycle 5, it is considered that the group of neurons is in a stable state in cycle 5 (the AF change between two consecutive cycles does not exceed the threshold value of 0.01), that is, cycles 1 to 5 are a training cycle interval, and the group of neurons shows an overall increase in this interval (0.655-0.52>0).

[0079] In a specific embodiment of the present invention, after training in a training cycle, the increase or decrease in the activation frequency of the neuron reflects the insufficiency or redundancy of the neuron's ability, so as an indicator for adjusting the update accuracy of its synaptic connection (also the device writing accuracy), the adjustment amount of the update accuracy of the weight update amount of the pth or pth group of neurons in the lth layer is for:

[0080]

[0081] in, is the activation frequency change of the pth or pth group of neurons in the lth layer in the current training cycle, k l is the baseline update accuracy of the lth layer. The obtained adjustment amount is added to the update accuracy of the current layer weight update amount connected to the current layer neurons in the current training cycle interval to obtain the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval.

[0082] In one embodiment, for the neuron group of the 124th channel of the output feature map of the third layer, its activation frequency in the first training cycle interval increased by 0.135, that is, And the update precision of the pre-allocated weights of the third layer is k3=5, then the update precision adjustment value of all weights of the 124th convolution kernel of the filter in this layer is That is, the 1-bit update accuracy needs to be improved.

[0083] In the next training cycle, the update precision of all weights of the 124th convolution kernel of the third layer filter is adjusted to 5+1=6-bit. In the process of updating the conductance value of the ECRAM array according to the conductance update amount ΔG iteratively calculated in this cycle, the number of writes needs to be increased from the original 3 times to 4 times. It can be understood that the update precision of the next weight update amount will be adjusted to 6+adjustment amount x, where the adjustment amount x is obtained by the adjustment amount calculation formula based on the baseline update precision 5 and the activation frequency change.

[0084] (3) After the real-time training is completed, the predicted value output by the trained decoding model is parsed through FPGA to obtain the control instruction, and the control instruction is used to control the external mechanical prosthesis to make corresponding actions, thereby realizing the decoding of EEG signals.

[0085] In one embodiment, when the predicted value output by the decoding model is a finger motion trajectory, the output is a binary tensor of dimension 2×5×4, where the first dimension (2) indicates the left hand and the right hand, the second dimension (5) represents five fingers, and the third dimension (4) represents four motion trajectories (flexion, extension, abduction, and adduction). A value of 0 indicates that there is no motion intention for a certain action at that position, and a value of 1 indicates that there is a motion intention for a certain action at that position.

[0086] A specific embodiment of the present invention sends the output tensor to an FPGA to parse the output tensor, clarify which fingers will do which actions to obtain motion trajectory information, and then encode the motion trajectory information in a specific way and package it into control instructions to send to an external mechanical prosthesis, thereby decoding the EEG signal and controlling the external mechanical fingers to make corresponding actions.

[0087] The specific embodiment of the present invention also provides an electroencephalogram signal decoding device based on ECRAM array, such as Figure 2 As shown, it includes a signal acquisition module, a signal processing module and an external device control module.

[0088] Among them, the signal acquisition module provided in the specific embodiment of the present invention is used to pre-process the received EEG signal.

[0089] The signal processing module provided in the specific embodiment of the present invention is used to map the weight matrix of the decoding model to the conductance matrix of the ECRAM array, and distribute the initial update accuracy provided by the user to each layer based on the parameter quantity of each layer of the decoding model to obtain the benchmark update accuracy of each layer.

[0090] In the real-time training process, a loss function is constructed based on the predicted value and the true intention label output by the decoding model based on the preprocessed EEG signal, and the loss function is used to back-propagate through the gradient descent method to obtain the updated amount of the weights of each layer. The real-time training process includes multiple training cycle intervals. After training in the current training cycle interval, an adjustment amount of the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval is obtained based on the activation frequency change of the current layer neurons and the benchmark update accuracy. The adjustment amount is added to the update accuracy of the current layer weight update amount connected to the current layer neurons in the current training cycle interval to obtain the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval. The update accuracy of the weight update amount is the accuracy of writing the conductance update value mapped by the weight update amount into the ECRAM array.

[0091] When the real-time training is completed, the predicted value output by the trained decoding model is output to the FPGA.

[0092] The external device control module provided in a specific embodiment of the present invention includes an FPGA, and a control instruction is obtained by parsing the FPGA, and the control instruction is used to control the external mechanical prosthesis to perform corresponding actions.

[0093] like Figure 3As shown, the example application scenarios of the present invention are rich. (1) Before the device leaves the factory, the EEG signal acquisition module acquires and integrates the EEG signals of multiple subjects to form a training data set, and performs preprocessing. The preprocessed data is sent to the EEG signal processing module based on the ECRAM array. The processing module performs offline learning based on the real motion trajectory label corresponding to each sample, and updates the conductance value of the ECRAM array accordingly. (2) After the device leaves the factory, the EEG signal of a specific subject at any time can be learned in real time online. The EEG signal acquisition module acquires the EEG signal of an individual in a real environment and performs preprocessing. The preprocessed data is sent to the EEG signal processing module based on the ECRAM array. The signal processing module performs online learning in combination with the real motion trajectory label learned this time, and updates the conductance value of the ECRAM array accordingly; (3) After the device leaves the factory, it can decode and predict the EEG signal intention of individual subjects in a real environment. The EEG signal acquisition module collects the EEG signals of individuals in a real environment and performs preprocessing, and sends the preprocessed data to the EEG signal processing module based on the ECRAM array. The processing module performs reasoning by running the neural network model and obtains the corresponding motion trajectory prediction results, and sends the prediction results to the external device controller. The external device controller analyzes the motion trajectory prediction results and controls the prosthetic finger to perform the corresponding action. Since EEG signals have individual differences and time variability, it is very important that the device still has the ability to learn after leaving the factory, so that the motion intention of a specific user can be predicted more accurately within a specific time period.

[0094] like Figure 4 and Figure 5 As shown, by applying the ECRAM array-based EEG signal decoding device provided by the present invention, that is, the mixed precision shown in the figure, in real-time learning scenarios or offline learning scenarios, the model training energy efficiency ratio and acceleration are better than full precision (usually CPU or GPU) and 8-bit fixed precision (usually digital decoding device); Figure 6 As shown, the ECRAM array-based EEG signal decoding device provided by a specific embodiment of the present invention, that is, the mixed precision shown in the figure, has no significant decrease in the test accuracy of EEG signals compared with the full-precision device in real-time learning scenarios or offline learning scenarios, thereby ensuring the accuracy of the learning method.

[0095] In summary, the EEG signal decoding method based on ECRAM array provided in the specific embodiment of the present invention has the characteristics of low power consumption, low latency and accuracy. The device has the function of on-chip learning for individual EEG signals, so the EEG signal decoding device proposed in the present invention is very suitable for use as a portable device.

[0096] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should be within the scope of protection of the present invention.

Claims

1. A real-time learning method for EEG signal decoding based on ECRAM array, characterized in that: include: The weight matrix of the decoding model is mapped to the conductance matrix of the ECRAM array, and the initial update accuracy provided by the user is distributed to each layer based on the parameter amount of each layer of the decoding model to obtain the benchmark update accuracy of each layer; In the real-time training process, a loss function is constructed based on the predicted value and the true intention label output by the decoding model based on the preprocessed EEG signal, and the loss function is used to back-propagate through the gradient descent method to obtain the update amount of the weights of each layer. The real-time training process includes multiple training cycle intervals. After training in the current training cycle interval, an adjustment amount of the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval is obtained based on the activation frequency change of the current layer neurons and the benchmark update accuracy. The adjustment amount is added to the update accuracy of the current layer weight update amount connected to the current layer neurons in the current training cycle interval to obtain the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval. The update accuracy of the weight update amount is the accuracy of writing the conductance update value mapped by the weight update amount into the ECRAM array; When the real-time training is completed, the predicted value output by the trained decoding model is parsed through FPGA to obtain the control instruction, and the control instruction is used to control the external mechanical prosthesis to make corresponding actions, thereby realizing the decoding of EEG signals.

2. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 1, characterized in that: The method for obtaining the change in activation frequency of neurons in the current layer during the current training cycle includes: The current training cycle interval includes multiple training cycles. When the difference in activation frequency of the current layer neurons obtained in two consecutive training cycles does not exceed the set threshold, the training of one training cycle interval is completed, and the difference between the activation frequency of the current layer neurons obtained in the last training cycle in the training cycle interval and the activation frequency of the current layer neurons obtained in the first training cycle is used as the change in the activation frequency of the current layer neurons in the current training cycle interval.

3. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 2, characterized in that: The activation frequency of the nth neuron in the lth layer during the training cycle for: in, represents the activation value of the nth neuron in the lth layer after the activation function in the i-th iteration, M is the total number of iterations in a training cycle, is the indicator function, when The value is 1 when , otherwise it is 0. It represents the number of iterations in which the activation value of the n-th neuron in the l-th layer is not 0 during the training cycle.

4. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 1, characterized in that: Based on the connection relationship between the weights of the current layer and the postsynaptic neurons, the neurons of the current layer are grouped, and the average activation frequency of the neurons in each group is calculated. The update accuracy of the corresponding weight update amount is adjusted in the next training cycle interval according to the obtained average activation frequency.

5. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 1, characterized in that: When the batch gradient descent method is used for training, the neurons of the current layer of multiple samples are grouped based on the connection relationship between the weights of the current layer and the post-synaptic neurons of multiple samples. The average activation frequency of the neurons in each group of each sample is calculated, and the average activation frequency of multiple samples corresponding to each weight is averaged again to obtain the batch activation frequency average. The update accuracy of the update amount of the corresponding weight is adjusted according to the obtained batch activation frequency average.

6. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 1, characterized in that: After training in the current training cycle, based on the activation frequency change of the neurons in the current layer and the benchmark update accuracy, the update accuracy adjustment of the weight update amount of the current layer connected to the neurons in the current layer in the next training cycle is obtained, and the update accuracy adjustment of the weight update amount of the pth or pth group of neurons in the lth layer is obtained. for: in, is the activation frequency change of the pth or pth group of neurons in the lth layer in the current training cycle, k l Update the accuracy of the baseline for layer l.

7. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 1, characterized in that: Based on the parameters of each layer of the decoding model, the overall update accuracy provided by the user is distributed to each layer to obtain the benchmark update accuracy of each layer, where the benchmark update accuracy k of the lth layer is l for: k l =round((1-α l )×K) Among them, K is the initial update precision provided by the user, round(·) is the rounding function, and Param l is the parameter of the lth layer, count(Param l ) is the parameter count of the lth layer, is the sum of all layer parameters, L is the number of layers, and j is the index of the layer.

8. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 1, characterized in that: The conductivity update value ΔG mapped by the weight update is: W2G max =G max -G min W2G min =G min -G max Among them, ΔW is the update amount of weight, W2G max and W2G min are the maximum and minimum values ​​that can be theoretically represented after mapping the weight matrix W to the two ECRAM arrays, respectively. max and W min are the maximum and minimum values ​​of the current layer weight, G max and G min are the maximum and minimum values ​​of the device within the conductivity range, respectively.

9. The real-time learning method for EEG signal decoding based on ECRAM array according to claim 1, characterized in that: The decoding model is a deep neural network model, and the ECRAM array is used to map the weight matrix of the decoding model.

10. An electroencephalogram signal decoding device based on an ECRAM array, characterized in that: include: A signal acquisition module, used for preprocessing the received EEG signals; A signal processing module, used for mapping the weight matrix of the decoding model into the conductance matrix of the ECRAM array, and allocating the initial update precision provided by the user to each layer based on the parameter quantity of each layer of the decoding model to obtain the benchmark update precision of each layer; In the real-time training process, a loss function is constructed based on the predicted value and the true intention label output by the decoding model based on the preprocessed EEG signal, and the loss function is used to back-propagate through the gradient descent method to obtain the update amount of the weights of each layer. The real-time training process includes multiple training cycle intervals. After training in the current training cycle interval, an adjustment amount of the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval is obtained based on the activation frequency change of the current layer neurons and the benchmark update accuracy. The adjustment amount is added to the update accuracy of the current layer weight update amount connected to the current layer neurons in the current training cycle interval to obtain the update accuracy of the current layer weight update amount connected to the current layer neurons in the next training cycle interval. The update accuracy of the weight update amount is the accuracy of writing the conductance update value mapped by the weight update amount into the ECRAM array; When the real-time training is completed, the predicted value output by the trained decoding model is output to the FPGA; The external device control module includes an FPGA, which obtains control instructions through FPGA analysis and controls the external mechanical prosthesis to perform corresponding actions through the control instructions.

Images