On-chip decoder and method for decoding neural activity, brain implant comprising the same and prosthetic system comprising the brain implant

Abstract

The present invention concerns an on-chip decoder (ocD), a brain implant, a prosthetic system and a method for decoding the task-related brain activity from multi-unit recordings of groups of neurons in a brain area and convert it into digital data usable by a classifier generating data labels corresponding to task-related information processed by said brain area, wherein said decoder uses multi-unit threshold crossing rates, called neural activity, subdivided into channels and time slots representing features of neural activity, in which it measures both the total magnitude and the uniformity of between-classes distances across all features to extract the most salient features and thereby generate a neural code (DNC) representing the highest class saliency values and reducing the neuronal state space into a low-dimensional DNC subspace, the parameters of the On-chip decoder (ocD) being computed offline during training and stored in an on-chip memory for online decoding.

Description

On-chip decoder and method for decoding neural activity, brain implant comprising the same and prosthetic system comprising the brain implant Field of the invention

[0001] The present invention relates to the field of Brain to Machine Interfaces and in particularto on-chip decoders, for example intended to be integrated into a Brain implant, for recording and decoding the neural activity, preferably to communicate with a digital machine (any device with computing resources) processing the decoded activity and possibly performing tasks accordingly. Such interfaces are for example used to restore communication or motor abilities of injured persons.

[0002] This invention is described in the present invention for intended (or attempted)handwriting recognition but can be used for intended speech or even in other brain modalities, such as the auditory modality, for example to enable a digital discrimination or recognition of sound frequencies or the visual modality, for example to enable adigital detection of salient features in a visual field. The functionalities (or tasks to be performed by the digital machine) enabled by the invention can be, for example speech recognition for the auditory modality, and the present description focuses in a non- limiting manner on intended handwriting, to obtain a digital representation of intended letters which can then be displayed, printed or sent through a network by the machine. Background of the invention

[0003] In the field of Brain to Machine Interfaces, many solutions have been developed todecode the brain activity, in various motor or sensory modalities and for performing various tasks accordingly. Recently, cutting-edge brain-machine interfaces have revealed the potential of decoders such as recurrent neural networks (RNNs) inpredicting attempted handwriting or speech, enabling rapid communication recovery after paralysis. RNNs have been a popular decoding model for their high accuracy through complex representations but they often are viewed as a black-box model and entail significant computational complexity and memory. In addition, most of known solutions rely on benchtop configurations with resource-intensive computing units, leading to bulkiness and excessive power demands. For clinical translation, the brain to machine interfaces must be realized in the form of miniaturized, implantable systems and achieve high decoding accuracy in a variety of prosthetic tasks. To date, only a handful of systems have reported on-chip decoding for conventional interface tasks such as finger movement. These solutions present the drawbacks of solely implementing specific decoder components on chip, consuming significant power, having too large dimensions, utilizing power-intensive commercial analog front-ends (AFEs) and / or lacking the high bandwidth necessary for more intricate interface tasks. There remains a gap for a high-channel-count, low-power brain to machine interfaces capable ofsimultaneous neural recording and decoding, in particular motor decoding, especially for rapid restoration of intricate movements like handwriting. Summary of the invention

[0004] One purpose of the present invention is to overcome at least some drawbacks of theprior art by proposing a decoder having a high bandwidth without requiring large processing resources and / or with reduced size and / or low power consumption.

[0005] This purpose is achieved by a decoder for decoding the task-related brain activity fromelectrophysiological multi-unit recordings of a plurality of groups of neurons in a brain area and convert it into digital data usable by a classifier for generating predictions, called data labels, corresponding to task-related information processed by said brain area, characterized in that said decoder uses as an input dataset, multi-unit thresholdcrossing rates, called neural activity, subdivided into channels and time slots representing features of neural activity, in which it measures both the total magnitude and the uniformity of between-class distances across all said features to extract the most salient features and thereby generate a neural code representing the highest classsaliency values and reducing the neuronal state space into a low-dimensional subspace.

[0006] Such decoder thereby can be implemented on a chip as defined in claim 1.

[0007] According to another feature, said the parameters of the On-chip decoder are computedoffline during training and stored in a memory of said On-chip decoder for online decoding.

[0008] According to another feature, said neural activity, formed by the input dataset of multi-unit threshold crossing rates, is priorly converted by a smoothing unit into a smoothed spike rate by a convolution of the spike rates with a Gaussian smoothing kernel to reduce noise sensitivity and temporal misalignment over time.

[0009] According to another feature, said On-chip decoder comprises an onset aligner forreducing power consumption, said onset aligner: detecting the periods of task-relevant activities by selecting a subset of said neural activities in which the activity level increases above a predefined threshold and combining said subset to create a mean class activity; detecting onsets and durations of task-relevant activities by detecting when both the magnitude and slope of said mean class activity cross and remain above thresholds for a predefined period and thereby generating an alignment block for aligning the neural activities with the onset and / or peak of said mean class activity, such that the On-chip decoder generates said neural code for each class in real time, according to the timing generated by the alignment block.

[0010] According to another feature, said On-chip decoder executes the following algorithm forgenerating the neural code to extract the most salient features: Selecting the input of neural activity, by; Forming the mean class activity, byOnset detection by thresholding and generating alignments block for aligning neural activities and defining idle periods and active periods, and During the active periods, computing the class saliency and extracting the neural code, by .

[0011] According to another feature, said On-chip decoder further comprises a shared memoryaddressed by an indexed approach, splitting said shared memory into variable-size segments across inputs and classes.

[0012] According to another feature, said On-chip decoder further comprises a classifierimplemented in the same chipset as the On-chip decoder, the parameters of said classifier being also computed offline during training and stored in an on-chip memory.

[0013] According to another feature, said classifier is a liner discriminant analysis classifierexecuting DNC-weight multiplications and predicting the class with the highest distance from classifier hyperplanes.

[0014] According to another feature, said smoothing unit is implemented on the On-chipdecoder itself or on a dedicated chip or on an analog front-end providing the input to the On-chip decoder.

[0015] According to another feature, said On-chip decoder further comprises an analog front-end providing the input to the On-chip decoder and wherein said analog front-end is implemented in the same chipset as the On-chip decoder.

[0016] According to another feature, said analog front-end comprises 192 channels and issubdivided in 24 front-end modules each comprising an 8-channel time-division multiplexor for multiplexing the electrophysiological recordings.

[0017] According to another feature, said analog front-end comprises a high-pass filter toextract spikes and / or a low-pass filtered to obtain local field potentials for enhanced decoding stability.

[0018] According to another feature, said On-chip decoder further comprises a movementtranslation module, implemented in the same chipset as the On-chip decoder, and converting the classes discriminated by said classifier into sequences of activation for effectors of a robotic limb, the parameters of said movement translation module beingalso computed offline during training and stored in an on-chip memory.

[0019] Another purpose of the present invention is also to overcome at least some drawbacks ofthe prior art by proposing a brain implant having a high bandwidth without requiring large processing resources and / or with reduced size and / or low power consumption.

[0020] This purpose is reached by a brain implant forming a brain to machine interface, fordecoding the task-related brain activity from electrophysiological multi-unit recordings of a plurality of groups of neurons in a brain area and convert it into digital data usable by a classifier for generating data labels corresponding to task-related information processed by said brain area, characterized in that said brain implant comprises an On- chip decoder according to any one of the embodiments described herein.

[0021] According to another feature, said Brain implant further comprises a movementtranslation module, implemented in another chipset of the brain implant, and converting the classes discriminated by said classifier into sequences of activation for effectors of arobotic limb, the parameters of said movement translation module being also computed offline during training and stored in an on-chip memory.

[0022] According to another feature, said On-chip decoder of said Brain implant comprises amovement translation module, implemented in the same chipset as the On-chip decoder, and converting the classes discriminated by said classifier into sequences of activation for effectors of a robotic limb, the parameters of said movement translation module being also computed offline during training and stored in an on-chip memory.

[0023] According to another feature, said Brain implant further comprises at least one array ofelectrodes for implantation in said brain area and recording said brain activity.

[0024] According to another feature, said Brain implant further comprises a connector forconnecting to a digital machine having computing resources for processing the decoded activity and possibly performing tasks accordingly.

[0025] Another purpose of the present invention is also to overcome at least some drawbacks ofthe prior art by proposing a prosthetic system having a high bandwidth without requiring large processing resources and / or with reduced size and / or low power consumption.

[0026] This purpose is reached by a prosthetic brain system Prosthetic brain system forrestoring functions of an injured subject, implantation in a brain area for recording and decoding said brain’s activity, characterized in that it comprises an array of electrodes for recording electrophysiological multi-unit signals from a brain area and at least one brain implant according to 13 or 14, converting said signals into digital data usable by a classifier, said brain implant comprising at least one on-chip decoder according to one of the claims 1 to 13, and outputting data labels, corresponding to task-related information processed by said brain area, to a digital machine having computing resources for processing the decoded activity and performing tasks accordingly or outputting saidneural code to said classifier generating said data labels and implemented in said digital machine for generating said data labels, processing the decoded activity and performing tasks accordingly.

[0027] According to another feature, said task is the output of letters imagined to be written bythe subject in which said brain implant is implanted, said system comprising a display or sound generator for outputting said letters.

[0028] According to another feature, said task is the execution of movements by a robotic limbwhen movements of a real limb is imagined by the subject in which said brain implant is implanted, said system comprising at least one movement translation module for converting said classes into activation sequences of effectors of said robotic limb for controlling various parts of said robotic limb according to the decoded movements.

[0029] Another purpose of the present invention is also to overcome at least some drawbacks ofthe prior art by proposing a method for decoding brain activity with a high bandwidth without requiring large processing resources.

[0030] This purpose is reached by a method a Method for decoding the task-related brainactivity from electrophysiological multi-unit recordings of a plurality of groups of neurons in a brain area and convert it into digital data usable by a classifier for generating data labels corresponding to task-related information processed by said brain area, characterized in that it is implemented by a decoder: -using multi-unit threshold crossing rates, called neural activity, as an input dataset,- subdividing the neural activity into channels and time slots representing features ofneural activity, -measuring both the total magnitude and the uniformity of between-classes distancesacross all features to extract the most salient features and thereby generate a neural code representing the highest class saliency values and reducing the neuronal state space into a low-dimensional DNC subspace.

[0031] According to another feature, said Method further comprises smoothing said dataset intoa smoothed spike rate, by a convolution of the spike rates with a Gaussian smoothing kernel to reduce noise sensitivity and temporal misalignment over time.

[0032] According to another feature, said Method further comprises a training phase tocompute the parameters of the decoder offline and store them in a memory readable by the decoder for online decoding.

[0033] According to another feature, said Method further reduces power consumption, by:detecting the periods of task-relevant activities by selecting a subset of said neural activities in which the activity level increases above a predefined threshold and combining said subset to create a mean class activity; detecting onsets and durations of task-relevant activities by detecting when both the magnitude and slope of said mean class activity cross and remain above thresholds for a predefined period and thereby generating an alignment block for aligning the neural activities with the onset and / or peak of said mean class activity, such that the On-chip decoder generates said neural code for each class in real time, according to the timing generated by the alignment block.

[0034] According to another feature, generating the neural code to extract the most salientfeatures is obtained by: Selecting the input of neural activity, by ; Forming the mean class activity, byOnset detection by thresholding and generating alignments block for aligning neural activities and defining idle periods and active periods, and During the active periods, computing the class saliency and extracting the neural code, by .

[0035] According to another feature, the decoder outputs said neural to a classifier generating,by classification, said data labels corresponding to task-related information processed by said brain area, the parameters of said classifier being also computed offline during training and stored in a memory for online decoding and classification. Brief description of the drawings

[0036] Various technical features and advantages of the present invention will appear moreclearly by reading the description of various examples of embodiments below, made in reference to the illustrative and non-limiting drawings, among which:

[0037] Figure 1 shows a conceptual diagram of a brain implant (BMI) according to in someembodiments;

[0038] Figure 2 shows an example of hardware architecture and timing diagram of a brainimplant (BMI) according to some embodiments;

[0039] Figure 3 shows the algorithm used for extracting the neural codes in someembodiments.

[0040] Figure 4 shows the detailed hardware architecture of a decoder according to someembodiments including an onset aligner (OA);

[0041] Figure 5 shows the detailed hardware architecture of a decoder according to someembodiments using memory sharing;

[0042] Figure 6 illustrates an example of chip micrographs and area / power breakdowns ina chipset forming the brain implant (BMI) according to some embodiments. Detailed description of embodiments

[0043] The present invention concerns an On-chip decoder (ocD) and a method for decodingneural activity, a brain implant (BMI) comprising such decoder (ocD) and a prosthetic system comprising such brain implant (BMI). Various embodiments are proposed, by focusing on the decoding method and the fact that it can be executed by a small chipwhich doesn’t need to have large processing resources.

[0044] The brain activity is generally recorded by an array of electrodes sensing the electricalsignals of groups of neurons (multi-unit activity), which are preferably mesoscale (extracellular and / or Local Field Potential - LFP) recording electrodes implanted insidethe skull, within the brain tissues for recording neurons around a few hundreds of micrometers. However, the invention may also be used macroscale electrodes either at the surface of the cortex (EcoG, for electrocorticography) or outside the skull (EEG, for electroencephalography), for tasks requiring a dialogue between more extended areas of the brain. In addition, the invention can be used for microscale single-unit activity recordings (extracellular recording of a single neurons with a higher impedance electrodes).

[0045] These electrophysiological signals are complex and dense information, thereforerequiring the use of decoder such as described herein, which can advantageously be an on-chip decoder because of its low power consumption and reduced need of computing resources (although it can also be contemplated as being implemented in other known ways). Such recordings can be used for decoding the task-related brain activity from electrophysiological multi-unit recordings of a plurality of groups of neurons in a brain area and convert it into digital data usable by a classifier for generating predictions,called data labels (or control data), corresponding to task-related information processedby said brain area (the label or information being for example a letter in a handwritingdecoding task). Then, the data labels can be used by a digital machine (a machine having at least one computing unit) for processing the data and performing the related task.

[0046] In a general manner, the decoder (ocD) uses as an input dataset, multi-unit thresholdcrossing rates, called neural activity, subdivided into channels and time slots representing features of neural activity, that are measured based on both the total magnitude and the uniformity of between-class distances across all features to extract the most salient features and thereby generate a neural code (DNC) representing the highest class saliency values and reducing the neuronal state space into a low- dimensional DNC subspace, the parameters of the On-chip decoder (ocD) being computed offline during training and stored in an on-chip memory for online decoding. Preferably, the input dataset of multi-unit threshold crossing rates is priorly converted by a smoothing unit into a smoothed spike rate representing the neural activity, by a convolution of the spike rates with a Gaussian smoothing kernel to reduce noise sensitivity and temporal misalignment over time. For example, the decoder receives up to 512 spike rates (for example with 10 milliseconds bins, but other bin sizes arepossible) as input and its output comprises 512 channels and 31 classes, which matchesvery well the requirements for recognition of letters handwriting, in particular. Preferably, said smoothed spike rate representing said neural activity is generated by the On-chip decoder (ocD) itself, but it may be implemented in a dedicated chip or in an analog front-end (AFE) providing the input to the On-chip decoder (ocD). Preferably, the decoder (ocD) is implemented in a chipset (or even a single chip) comprising a smoothing unit for such smoothing, an onset aligner (OA) and neural codes extractor, and generally a classifier.

[0047] In some embodiments, the On-chip decoder (ocD) further comprises an onset aligner(OA) for reducing power consumption. Such onset aligner (OA) detects the periods of task-relevant activities by selecting a subset of said neural activities in which the activity level increases above a predefined threshold and combining said subset to create a mean class activity (OC). The onset aligner (OA) further detects onsets and durations of task- relevant activities by detecting when both the magnitude and slope of said mean classactivity (OC) cross and remain above thresholds for a predefined period (TTC) and thereby generating an alignment block (tOD, tPD) for aligning the neural activities with the onset and / or peak of said mean class activity (OC), such that the On-chip decoder generates said neural code (DNC) for each class in real time, according to the timing generated by the alignment block (tOD, tPD).

[0048] In some embodiments, the On-chip decoder (ocD) the following algorithm forgenerating the neural code (DNC) to extract the most salient features: -Selecting the input of neural activity, bywhere the G function quantifies the activity escalation following an activity onset. According to these equations, a subset of inputs exhibiting the greatest activity escalation is chosen, only if it is observed in over half of the trials. -Forming the mean class activity (OC), bywhich is the weighted average of the selected subset of inputs. -Onset detection by thresholding and generating alignments block for aligning neuralactivities and defining idle periods and active periods, and -During the active periods, computing the class saliency and extracting the neuralcode (DNC), byin which ςc is the class saliency, and M() denotes the margin function. The first term in the saliency equation (ςc) denotes the geometric mean (GM) of the overall discrimination of desired class ‘c’ from all other classes. The second term assesses the homogeneity of the class ‘c’ discriminations from other individual classes by calculating the ratio of the geometric mean to the arithmetic mean (AM). By multiplying these two terms, saliency equation (ςc) concurrently evaluates the overall class discrimination and the homogeneity associated with class ‘c’. This value quantifies the discriminative information that each feature conveys about a class of interest. The margin function determines the minimum spacing between features in a DNC set, enhancing their independence and subsequently reducing mutual information between features. Finally, features with high-class saliency that maintain a marginal distance from other features will form the class-specific DNC set.

[0049] In some embodiments, the On-chip decoder (ocD) further comprises a shared memoryaddressed by an indexed approach, splitting said shared memory into variable-size segments across inputs and classes.

[0050] In some embodiments, the On-chip decoder (ocD) further comprises a classifier (LDA)implemented in the same chipset as the On-chip decoder (ocD), the parameters of said classifier (LDA) being also computed offline during training and stored in an on-chip memory. In some embodiments, said classifier (LDA) is a liner discriminant analysis classifier executing DNC-weight multiplications and predicting the class with the highest distance from classifier hyperplanes.

[0051] In some embodiments, the On-chip decoder (ocD) further comprises an analog front-end(AFE) providing the input to the On-chip decoder (ocD). Thanks to the size reduction of the present invention, the analog front-end (AFE) can be implemented on the same chipset or even the same chip as the On-chip decoder (ocD). In some embodiments, analog front-end (AFE) comprises 192 channels and is subdivided in 24 front-end modules each comprising an 8-channel time-division multiplexor for multiplexing the electrophysiological recordings. In some embodiments, the On-chip decoder (ocD) or the analog front-end (AFE) comprise a high-pass filter to extract spikes and / or a low- pass filtered to obtain local field potentials for enhanced decoding stability.

[0052] Some embodiments concern a brain implant (BMI) comprising an On-chip decoder(ocD) according to the invention, for forming a brain to machine interface (BMI), for decoding the task-related brain activity from electrophysiological multi-unit recordings of a plurality of groups of neurons in a brain area and converting it into digital data usable by a classifier for generating data labels (or control data) corresponding to task- related information processed by said brain area. In some embodiments, the brainimplant (BMI) further comprises at least one array of electrodes for implantation in said brain area and recording said brain activity and / or a connector for connecting to a digital machine having computing resources for processing the decoded activity and possiblyperforming tasks accordingly.

[0053]

[0054] Some embodiments also concern a prosthetic brain system for restoring functions of aninjured subject, by implantation in a brain area for recording and decoding said brain activity. This prosthetic system comprises an array of electrodes outputting the electrophysiological signals to a brain implant (BMI) of the invention which outputs data labels corresponding to task-related information processed by said brain area to a digital machine having computing resources for processing the decoded activity and performing tasks accordingly or outputting said neural code (DNC) to a classifier generating said data labels and implemented in said digital machine.

[0055] In preferred embodiments, the prosthetic brain system comprises an array of electrodesfor recording electrophysiological multi-unit signals from a brain area and at least one brain implant according to any one of the embodiments described herein, converting said signals into digital data usable by a classifier, said brain implant comprising at least one on-chip decoder according to any one of the embodiments described herein, and outputting data labels, corresponding to task-related information processed by said brain area, to a digital machine having computing resources for processing the decoded activity and performing tasks accordingly or outputting said neural code to said classifier generating said data labels and implemented in said digital machine for generating said data labels, processing the decoded activity and performing tasks accordingly.

[0056] In some embodiments, said task is the output of letters imagined to be written by thesubject in which said brain implant is implanted, said system comprising a display or sound generator for outputting said letters.

[0057] In some embodiments, said task is the execution of movements by a robotic limbwhen movements of a real limb is imagined by the subject in which said brain implant is implanted, said system comprising at least one movement translation module for converting said classes into activation sequences of effectors of saidrobotic limb for controlling various parts of said robotic limb according to the decoded movements.

[0058] Finally, some embodiments also concern a method for decoding the task-related brainactivities. Such method preferably comprises: -using multi-unit threshold crossing rates, called neural activity, as an inputdataset, -subdividing the neural activity into channels and time slots representing featuresof neural activity, -measuring both the total magnitude and the uniformity of between-classdistances across all features to extract the most salient features and therebygenerate a neural code (DNC) representing the highest class saliency values and reducing the neuronal state space into a low-dimensional DNC subspace.

[0059] In preferred embodiments, the method further comprises smoothing said dataset into asmoothed spike rate, by a convolution of the spike rates with a Gaussian smoothing kernel to reduce noise sensitivity and temporal misalignment over time.

[0060] In some embodiments, the method further comprises a training phase to compute theparameters of the decoder offline and store them in a memory readable by the decoder for online decoding. In general, the method will use the algorithm detailed in the present application and illustrated in figure 3.

[0061] In some embodiments the method further reduces power consumption, by:- detecting the periods of task-relevant activities by selecting a subset of saidneural activities in which the activity level increases above a predefined threshold and combining said subset to create a mean class activity (OC); -detecting onsets and durations of task-relevant activities by detecting when boththe magnitude and slope of said mean class activity (OC) cross and remain above thresholds for a predefined period (TTC) and thereby generating an alignment block (tOD, tPD) for aligning the neural activities with the onset and / or peak of said mean class activity (OC), such that the On-chip decoder generates said neural code (DNC) for each class in real time, according to the timing generated by the alignment block (tOD, tPD).

[0062] Some embodiments of the invention may also concern independently an onset aligner(OA) usable on any kind of decoder for limiting the power consumption. The onset aligner described herein should thus not be interpretated as being limited to its use in combination of the other elements of the present application and its features can be isolated for those of the other elements for providing a power saving solution (method and / or device) in other decoding systems or devices, for example without the generation of the neural code described herein. Such independent onset aligner (OA) is defined as being configured for reducing power consumption in the processing of a fluctuating signal by at least one processor (in a computer or a chip), said onset aligner: -detecting periods of activities by selecting periods in which the fluctuatingsignal increases above a predefined threshold and combining said periods to create a mean class activity (OC); -detecting onsets and durations of active relevant periods by detecting when boththe magnitude and slope of said mean class activity (OC) cross and remain above thresholds for a predefined period (TTC) and thereby generating an alignment block (tOD, tPD) for aligning the neural activities with the onset and / or peak of said mean class activity (OC), such that said processor processes said fluctuating signal, according to the timing generated by the alignment block (tOD, tPD).

[0063] In general, in the onset aligner, may it be independent or according to the method, thedecoder (ocD), the brain implant (BMI) or the system described herein, the alignmentblock can be designed according to the onset and / or the peak and the alignment block (tOD, tPD) can thus be defined by the timing after the onset (tOD) and / or by the timing after the peak (tPD).

[0064] The same possible independence applies to the analog front-end described herein.Hence, some embodiments of the invention may also concern independently an analog front-end using an area-efficient time-division multiplexing scheme as described herein and usable for other kind of recordings and decoders.

[0065] Further details of preferred embodiments will now be described in reference to theillustrating and non-limiting figures. It should be noted that the references using in thefigures are not implying any limitations either and are not always acronyms. For example, the reference (LDA) used for the classifier doesn’t imply that the classifier is necessarily a linear discriminant analysis classifier (for which LDA stand for) although the present invention advantageously allows the use of such a simple, low-complexity classifier (which is more easily implemented in the chipset).

[0066] Figure 1 illustrates the conceptual diagram of a brain implant (BMI) according to someembodiments. The notion of « neural code » (DNC) is introduced in the presentdescription to define a set of spatiotemporal features of neuronal activity that effectively discerns a specific class from others in complex classification tasks. Inspired by the brain’s mechanism for selecting visually salient objects, we employ class saliency—a measure of both the total magnitude and the uniformity of between-class distances across all features—to extract the most salient features (DNCs). Figure 3 shows the algorithm for extracting this neural code (DNC) by selecting the highest class saliency values. By such method, the neuronal state space is successfully reduced into a low- dimensional DNC subspace (as illustrated in Fig.1, center). This proposed model can effectively extract DNCs for class identification without the need for complex pre- processing steps such as time warping or spike sorting. Moreover, by leveraging content-rich low-dimensional DNCs, data structures can be accurately learned using a simple, low-complexity classifier such as linear discriminant analysis (for example with only 4000 parameters). The decoder and classifier parameters are computed offline during training and stored in an on-chip memory for online decoding. The brain implant (BMI) was tested on recordings of 192 channels performed during 10.7 hours composing an intracortical dataset acquired from a tetraplegic patient during attempted handwriting. This dataset contains multi-unit threshold crossing rates and can be seamlessly fed into the proposed decoder (ocD) for handwritten letter classification.

[0067] Fig. 2 illustrates an example of the hardware architecture and timing diagram of a brainimplant (BMI) according to some embodiments. Obtaining high-resolution motor information is essential to enhancing the decoding accuracy of intracortical brain implants (BMI) (i.e., recording neurons within the cortex of the brain). Therefore, in some embodiments, the Analog Front End (AFE) is designed to flexibly record up to 192 channels of broadband (10kHz) neural activity, compatible for example with two electrode arrays of 96 channels implanted (for example in the premotor cortex for amotor task such as handwriting decoding). To realize a compact chipset, an area- efficient time-division multiplexing scheme has been adopted for the Analog Front End (AFE) of 192 channels. A low multiplexing factor of 8 was chosen to alleviate noise folding from electrodes. This embodiment of Analog Front End (AFE) comprises 24 front-end modules (Fig.2, top-left), each multiplexing 8 channels of electrodes. In addition, a SAR-assisted DC servo loop (DSL) can be employed to suppress 8-chelectrode DC offsets (±50mV) to below ±1mV using 7-bit binary search. The forward path consists of a two-stage capacitively coupled low-noise amplifier (LNA) followed by an anti-aliasing Gm-C integrator. The complementary input pairs in the current-reuse LNA are biased in the subthreshold region (gm / ID ≈ 26) for improved noise efficiency. To minimize inter-channel crosstalk, kT / C-free resetting is realized using chopper stabilization. Upon brief resetting in each sampling period, an input capacitor (CIN) can sample kT / C noise that is confined within 10kHz per channel. With chopping at160kHz, the kT / C and 1 / f noise are up-converted to its integer multiples, where the notches of the integrator’s sinc response attenuate these unwanted signals. The broadband output of such Analog Front End (AFE) can be, for example externally, high-pass filtered to extract spikes (>300Hz), and / or low-pass filtered to obtain local field potentials (LFPs, <300Hz) for enhanced decoding stability. As explained, the on-chip decoder (ocD) can then extract task-relevant neural activities associated with the subject’s intention for movement. In some embodiments, the on-chip decoder (ocD) receives up to 512 spike rates (10ms bins) as input. Preferably, this input is convolved with a Gaussian smoothing kernel (σ=16, window width of 100) to reduce noise sensitivity and temporal misalignment over time. To reduce dimensionality, training time, and model complexity, brain-inspired neural codes (DNC) are used as a compact feature subset and fed to a classifier (for example an on-chip LDA classifier).

[0068] Figure 3 shows the algorithm used for extracting the neural code (DNC). Figure 4 showsdetailed hardware architecture of a decoder according to some embodiments including an onset aligner (OA). The smoothed spike rate represents the data content required for motor decoding, termed as neural activity in this work. A subset of neural activities is selected following the subject’s attempted movement, provided that the activity level increases above a predefined threshold. The selected subset is then linearly combined to create a mean class activity (OC) for onset detection. An onset is declared if both the magnitude and slope of mean class activity (OC) cross and remain above thresholds for a predefined period (TTC). Upon detection of a task-relevant activity onset, the decoder aligns the neural activity with the onset and / or peak of the Oc. The DNCs are subsequently extracted for each class in real time, according to the timing generated bythe alignment block (tOD, tPD). Unlike conventional decoders, such activity-driven model performs neural decoding only upon onset detection and goes idle during low- activity states, thus reducing power consumption with fewer MAC operations adaptively to the typing speed, in characters per seconds. Given the multi-label nature of neuralcodes (DNC), these class-specific features carry distinct information for each class withnotable separability from others. As a result, a simple linear classifier can accurately predict the output within the DNC subspace. The LDA executes DNC-weight multiplications and predicts the class with the highest distance from classifier hyperplanes. Representing neural activity in a compact DNC subspace (with maximum 128 dimensions) enables a reduction of more than 13 times of the dimensionality compared to original spike rates and a reduction of 51200 times at the output of the decoder (with 512 channels and 31 classes). A conventional 31-class classifier (LDA) would require approximately 1.6MB of weight memory to store all hyperplane parameters, resulting in 1.6 millions MAC operations for neural decoding. Alternatively, the decoder according to some embodiments selects a small subset of neural codes (DNC) to reduce memory 7.8 times and MAC requirements 320 times.

[0069] Figure 5 illustrates some embodiments in which the hardware efficiency of the decoder(ocD) is further enhanced by using memory sharing. Given the varying number of neural codes (DNC) across channels, an indexed addressing approach is adopted to split a shared memory into variable-size segments across inputs and classes, yielding an additional reduction of 12.9 times in on-chip decoding memory required.

[0070] As explained above, in some embodiments, the various components of the decoder areall on the same chipset which can be fabricated in 65nm low-power CMOS.

[0071] Fig. 6 illustrates an example of such chipset by providing chip micrographs andarea / power breakdowns. Compared to the existing decoders, the 31-class decoder of some embodiments can support more than 4 times higher channel count with more than 13 times area efficiency while performing more complex decoding tasks such as handwriting. Moreover, when integrating the Analog Front End, the miniaturized chipset obtained performs area-efficient broadband neural recording, obviating the need for off-the-shelf recording devices.

[0072] As mentioned above, in some embodiments, the invention is used for motor control,such as operating a robotic limb. The decoder, implant, and system described herein are well-suited for this task, as they can decode neural signals and classify them into 31 classes. In the future, this capability may be expanded to 100 or more classes. For instance, decoding letters requires only 31 classes (26 for the alphabet and 5 for punctuation), whereas effective limb movement control typically requires around 100 classes. Controlling a natural arm involves complex muscle synergies, whereas a roboticarm operates differently. Communication with a robotic limb (e.g., arm) can be morestraightforward. The chip generates movement commands for the shoulders, arms, elbows, wrists, and fingers, which are then transmitted to the robotic limb. A sequence of activation of these effectors is thus provided by a movement translation module in the present invention, which converts the classes decoded from the brain into orders usable by the robotic (e.g., prosthetic) arm (or limb). Typically, additional electronics on the robotic limb handle coordination and synergy, ensuring smooth execution of movements.

[0073] Regarding the brain region, it’s challenging to determine the optimal area for decodingimagined movements, as different BMI studies take different approaches. For example, a handwriting BMI study (High-performance brain-to-text communication via handwriting; Willett et al. Nature 593 (2021), pp. 249–254) implanted electrodes in the‘hand knob’ area of the premotor cortex. Meanwhile, another study placed electrodes in the motor cortex for robotic hand control, with additional implants in the somatosensory cortex for sensory feedback. Indeed, motor control based on neural activity has been shown to be feasible and improved by somatosensory feedback, as expected (“A brain- computer interface that evokes tactile sensations improves robotic arm control”, Flesher et al. Science 21 May 2021, Vol 372, Issue 6544; pp.831-836; DOI:10.1126 / science.abd0380). However, implementing the decoding and eventually the classification in a brain implant was not yet performed, at least for letters display or limb movements control, and the present invention allows to obtain an efficient and low- power consuming decoder and / or implant and / or system for performing various interpretation of the neural activity in motor tasks, but also in perception, like the somatosensory modality, in particular for the feedback of motor control as performed bythe brain.

[0074] Accordingly, some embodiments of said On-chip decoder further comprises amovement translation module, implemented in the same chipset as the On-chip decoder, and converting the classes discriminated by said classifier into sequences of activation for effectors of a robotic limb. Preferably, the parameters of said movement translation module are computed offline during training and stored in an on-chip memory. In some alternative embodiments, the movement translation module is implemented in another chip which can be integrated in a brain implant comprising the decoder as described herein or is implemented (as a software for example) in a digital machine such as anydevice comprising a processor (a smartphone or a computer for example).

[0075] In some embodiments Regarding the limb movements task, the prosthetic brain systemfunctions as follows:- The implanted brain-machine interface (BMI) records neural activity associatedwith imagined movements. -The on-chip decoder processes these signals in real time, extracting relevantfeatures and translating them into movement commands. -These decoded movement signals are then sent to a robotic limb controller,which adjusts the limb’s actuators accordingly.

[0076] In some embodiments, the system can integrate adaptive feedback mechanisms to refinethe control based on sensory inputs or user intent. This framework ensures precise,intuitive control of the robotic limb based on the subject’s imagined movements.

[0077] It will be understood by reading the present application that all the functional blocks ofthe decoder (the smoothing unit, the onset aligner, the neural code extractor and even the classifier) can advantageously be integrated in a single decoding chip, which can also integrate or be combined with the recording chip integrating the analog front end into asingle chipset as illustrated in figure 6, with a great reduction of size. However, it will also be understood that these functional blocks (of both the decoding chip and the recording chip) could be separated (and combined with each other) differently in several chips of a chipset (or even externally to the brain implant) and figure 6 is not limiting but shows a preferred embodiment.

[0078] The present application describes various technical features and advantages withreference to the figures and / or to various embodiments. Those skilled in the art will understand that the technical features of a given embodiment can indeed be combined with features of one or more other embodiment(s) unless the reverse is explicitly mentioned or these characteristics are incompatible or the combination does not work. In addition, the technical features described in a given embodiment can be isolated from the other features of this mode unless the reverse is explicitly mentioned, in particular because the functional considerations provided in the present application will provide a sufficient explanation so that the structural adaptations possibly necessary are within the reach of those skilled in the art. Therefore, the embodiments described in the present application should be considered by way of illustration and the invention should not be limited to the details given above.

Claims

Claims1. On-chip decoder (ocD) for decoding the task-related brain activity fromelectrophysiological multi-unit recordings of a plurality of groups of neurons in a brain areaand convert it into digital data usable by a classifier for generating predictions, called datalabels, corresponding to task-related information processed by said brain area, characterizedin that said decoder uses as an input dataset, multi-unit threshold crossing rates, called neuralactivity, subdivided into channels and time slots representing features of neural activity, in which it measures both the total magnitude and the uniformity of between-class distances across all said features to extract the most salient features and thereby generate a neural code (DNC) representing the highest class saliency values and reducing the neuronal state space into a low-dimensional DNC subspace.

2. On-chip decoder (ocD) according to claim 1, wherein said the parameters of the On-chip decoder (ocD) are computed offline during training and stored in a memory of said On- chip decoder (ocD) for online decoding.

3. On-chip decoder (ocD) according to claim 1 or 2, wherein said neural activity,formed by the input dataset of multi-unit threshold crossing rates, is priorly converted by a smoothing unit into a smoothed spike rate by a convolution of the spike rates with a Gaussian smoothing kernel to reduce noise sensitivity and temporal misalignment over time.

4. On-chip decoder (ocD) according to any one of the preceding claims, furthercomprising an onset aligner (OA) for reducing power consumption, said onset aligner: -detecting the periods of task-relevant activities by selecting a subset of said neuralactivities in which the activity level increases above a predefined threshold and combining said subset to create a mean class activity (OC); -detecting onsets and durations of task-relevant activities by detecting when both themagnitude and slope of said mean class activity (OC) cross and remain above thresholds for a predefined period (TTC) and thereby generating an alignment block (tOD, tPD) for aligning the neural activities with the onset and / or peak of said mean class activity (OC), such that the On-chip decoder generates said neural code (DNC) for each class in real time, according to the timing generated by the alignment block (tOD, tPD).

5. On-chip decoder (ocD) according to any one of the preceding claims, wherein theOn-chip decoder executes the following algorithm for generating the neural code (DNC) to extract the most salient features: -Selecting the input of neural activity, by;- Forming the mean class activity (OC), by-Onset detection by thresholding and generating alignments block for aligning neuralactivities and defining idle periods and active periods, and -During the active periods, computing the class saliency and extracting the neuralcode (DNC), by6. On-chip decoder (ocD) according to any one of the preceding claims, furthercomprising a shared memory addressed by an indexed approach, splitting said shared memory into variable-size segments across inputs and classes.

7. On-chip decoder (ocD) according to any one of the preceding claims, furthercomprising a classifier (LDA) implemented in the same chipset as the On-chip decoder (ocD), the parameters of said classifier (LDA) being also computed offline during training and stored in an on-chip memory.

8. On-chip decoder (ocD) according to any one of the preceding claim, wherein saidclassifier (LDA) is a liner discriminant analysis classifier executing DNC-weight multiplications and predicting the class with the highest distance from classifier hyperplanes.

9. On-chip decoder (ocD) according to any one of the claims 3 to 8, wherein saidsmoothing unit is implemented on the On-chip decoder (ocD) itself or on a dedicated chip or on an analog front-end (AFE) providing the input to the On-chip decoder (ocD).

10. On-chip decoder (ocD) according to any one of the preceding claims, furthercomprising an analog front-end (AFE) providing the input to the On-chip decoder (ocD) and wherein said analog front-end (AFE) is implemented in the same chipset as the On-chip decoder (ocD).

11. On-chip decoder (ocD) according to claims 9 or 10, wherein said analog front-end(AFE) comprises 192 channels and is subdivided in 24 front-end modules each comprisingan 8-channel time-division multiplexor for multiplexing the electrophysiological recordings.

12. On-chip decoder (ocD) according to any one of claims 9 to 11, wherein said analogfront-end (AFE) comprises a high-pass filter to extract spikes and / or a low-pass filtered to obtain local field potentials for enhanced decoding stability.

13. On-chip decoder (ocD) according to any one of claims 7 to 12, further comprising amovement translation module, implemented in the same chipset as the On-chip decoder (ocD), and converting the classes discriminated by said classifier into sequences of activation for effectors of a robotic limb, the parameters of said movement translationmodule being also computed offline during training and stored in an on-chip memory.

14. Brain implant (BMI) forming a brain to machine interface (BMI), for decoding thetask-related brain activity from electrophysiological multi-unit recordings of a plurality of groups of neurons in a brain area and convert it into digital data usable by a classifier for generating data labels corresponding to task-related information processed by said brain area, characterized in that said brain implant comprises an On-chip decoder (ocD) accordingto any one of the claims 1 to 12.

15. Brain implant (BMI) according to claim 14, further comprising a movementtranslation module, implemented in another chipset of the brain implant (BMI), and converting the classes discriminated by said classifier into sequences of activation foreffectors of a robotic limb, the parameters of said movement translation module being alsocomputed offline during training and stored in an on-chip memory.

16. Brain implant (BMI) according to claim 14, wherein said On-chip decoder (ocD) is adecoder according to claim 13.

17. Brain implant (BMI) according to any one of claims 14 to 16, further comprising atleast one array of electrodes for implantation in said brain area and recording said brain activity.

18. Brain implant (BMI) according to any one of claims 14 to 17, further comprising aconnector for connecting to a digital machine having computing resources for processing the decoded activity and possibly performing tasks accordingly.

19. Prosthetic brain system for restoring functions of an injured subject, implantation in abrain area for recording and decoding said brain’s activity, characterized in that it comprisesan array of electrodes for recording electrophysiological multi-unit signals from a brain area and at least one brain implant (BMI) according to 13 or 14, converting said signals into digital data usable by a classifier, said brain implant (BMI) comprising at least one on-chipdecoder (ocD) according to one of the claims 1 to 13, and outputting data labels,corresponding to task-related information processed by said brain area, to a digital machinehaving computing resources for processing the decoded activity and performing tasksaccordingly or outputting said neural code to said classifier generating said data labels andimplemented in said digital machine for generating said data labels, processing the decodedactivity and performing tasks accordingly..

20. Prosthetic brain system according to claim 19, wherein the task is the output of lettersimagined to be written by the subject in which said brain implant (BMI) is implanted, saidsystem comprising a display or sound generator for outputting said letters.

21. Prosthetic brain system according to claim 19, wherein the task is the execution ofmovements by a robotic limb when movements of a real limb is imagined by the subject in which said brain implant (BMI) is implanted, said system comprising at least one movementtranslation module for converting said classes into activation sequences of effectors of saidrobotic limb for controlling various parts of said robotic limb according to the decoded movements.

22. Method for decoding the task-related brain activity from electrophysiological multi-unit recordings of a plurality of groups of neurons in a brain area and convert it into digital data usable by a classifier for generating data labels corresponding to task-related information processed by said brain area, characterized in that it is implemented by a decoder: -using multi-unit threshold crossing rates, called neural activity, as an input dataset,- subdividing the neural activity into channels and time slots representing features ofneural activity, -measuring both the total magnitude and the uniformity of between-classes distancesacross all features to extract the most salient features and thereby generate a neural code (DNC) representing the highest class saliency values and reducing the neuronal state space into a low-dimensional DNC subspace.

23. Method according to claim 22, further comprising smoothing said dataset into asmoothed spike rate, by a convolution of the spike rates with a Gaussian smoothing kernel to reduce noise sensitivity and temporal misalignment over time,24. Method according to claim 22 or 23, further comprising a training phase to computethe parameters of the decoder offline and store them in a memory readable by the decoder for online decoding.

25. Method according to any one of claims 22 to 24, further reducing powerconsumption, by: -detecting the periods of task-relevant activities by selecting a subset of said neuralactivities in which the activity level increases above a predefined threshold and combining said subset to create a mean class activity (OC); -detecting onsets and durations of task-relevant activities by detecting when both themagnitude and slope of said mean class activity (OC) cross and remain above thresholds for a predefined period (TTC) and thereby generating an alignment block (tOD, tPD) for aligning the neural activities with the onset and / or peak of said mean class activity (OC), such that the On-chip decoder generates said neural code (DNC) for each class in real time, according to the timing generated by the alignment block (tOD, tPD).

26. Method according to any one of claims 22 to 25, wherein generating the neural code(DNC) to extract the most salient features is obtained by: -Selecting the input of neural activity, by; -Forming the mean class activity (OC), by- Onset detection by thresholding and generating alignments block for aligning neuralactivities and defining idle periods and active periods, and -During the active periods, computing the class saliency and extracting the neuralcode (DNC), by27. Method according to any one of claims 22 to 26, wherein the decoder outputs saidneural (DNC) to a classifier (LDA) generating, by classification, said data labels corresponding to task-related information processed by said brain area, the parameters of said classifier (LDA) being also computed offline during training and stored in a memory for online decoding and classification.

Images