Online learning method for a neural interface

Abstract

Method for learning a direct neural interface, the direct neural interface being connected to sensors (21…2I1) previously arranged around the brain of a user, the interface being configured to control an actuator (6) according to electrophysiological signals detected by each sensor, the learning method comprising the formation of a predictive model ( ), by regression between a learning tensor, formed ) from the detected electrophysiological signals and a control tensor, representative of mental tasks performed by the user during different time epochs, the predictive model allowing the estimation of a task, imagined by the user according to the electrophysiological signals detected at each epoch, the method being such that the formation of the direct model is carried out according to the weight respectively assigned to each epoch.

Description

Title of the invention: Online learning method for a neural interface. Technical field

[0001] The technical field of the invention relates to direct neural interfaces, usually referred to as "BCI" (Brain Computer Interface), intended to control an actuator from neurophysiological signals. EARLIER ART

[0002] The field of direct neural interfaces is rapidly developing and appears to be an attractive solution for enabling users with disabilities to control actuators by thought. This involves detecting and recording electrophysiological signals emitted by the cortex. These signals are processed by algorithms to generate a control signal, which then controls actuators. The control signal allows the actuator—an exoskeleton, a computer, or a robot—to be operated to provide assistance to the user. The algorithms used translate an instruction given by the user. This instruction is captured by electrodes in the form of electrophysiological signals, which are representative of the electrical activity of neurons. This electrical activity can be measured at the level of the cortex using cortical electrodes placed in the skull.It can also be measured using electroencephalography electrodes, which are less invasive as they are placed on the scalp, but also less effective, particularly in terms of spatial resolution. Another solution is to record electrophysiological signals using magnetoencephalography, which requires a dedicated setup.

[0003] The algorithms implemented are generally based on a predictive model. The predictive model uses input data, obtained by preprocessing recorded electrophysiological signals, to establish a control signal for the actuator or actuators. The control signal must correspond to an intention expressed by the user, whose electrophysiological signals are recorded. The user's expressed intention is manifested in the form of electrophysiological signals, which are recorded and transmitted to the direct neural interface, forming observational data. The electrophysiological signals are processed to obtain observational data, forming input data for the predictive model, which then generates a control signal. corresponding to the intention expressed by the user. The control signal allows the actuator to be controlled.

[0004] The measured signals are processed to form generally multidimensional observational data, and include: - a spatial component, representative of the spatial origin of the electrophysiological signal; - a frequency component, representative of the intensity of the electrophysiological signal in different frequency bands; - a temporal component.

[0005] Each observation data point is associated with an epoch, that is, a predetermined time interval, for example, on the order of 1 second after the user intends to perform the task. At each epoch, an observation tensor is formed, combining the observation data. A predictive model is fed into the observation tensor. The predictive model, applied to the observation tensor, allows the estimation of a control signal, enabling the control of the actuators. The control signal is generally expressed by a control vector.

[0006] The predictive model is established during a learning phase, in which the user performs predefined tasks for which the output of the predictive model is known. The objective is then, following each task, to determine task-specific components of the recorded electrophysiological signals. This may involve, in particular, determining correlations between components of the electrophysiological signals and the model output.

[0007] The development of predictive models has been extensively described. For example, US patent 9480583 describes the application of a multivariate partial least squares linear regression method for establishing a predictive model. Such a method is known by the English acronym "NPLS" or by the term "N-way Partial Least Squares." The application of such a method was also described in the publication Eliseyev A, Aksenova T (2013) "Recursive N-way Partial Least Squares for Brain Computer Interface" P1OS ONE July 2013, Volume 8 Issue 7 e69962. Such a method is also described in the document Yelisyeyev A "Brain-Computer Interface with cortical electrical activity recording" Human health and pathology. University of Grenoble, 2011.

[0008] However, using an NPLS-type method requires processing a large amount of training data, for example several hundred or several thousand for a model output corresponding to a specific task. This implies storing a large amount of information in memory, which is not suitable for performing online, i.e., real-time, learning. or in near real-time. By near real-time, we mean learning carried out in successive sequences, each sequence lasting a few seconds or a few minutes.

[0009] To reduce the amount of information to be memorized, a learning process implementing a REW-NPLS type method has been developed, REW standing for "Recursive Exponentially Weighted". The training of a predictive model, using REW-NPLS, applied to a BCI interface, is described in EP3563218. Such an approach is also justified by the fact that neural signals are not stationary, which necessitates regular updates to the predictive model.

[0010] The inventors propose an improvement to the method described in EP3563218, in order to improve the learning performance of the predictive model. Description of the invention

[0011] A first object of the invention is a method for learning a direct neural interface, the direct neural interface being connected to sensors previously arranged around the brain of a user, each sensor being configured to detect an electrophysiological signal dependent on the user's neural activity, the interface being configured to control an actuator based on the detected electrophysiological signals, the learning method comprising: - a) selection of a mental task to perform, chosen from a predetermined list of tasks; - b) execution, by the user, of the mental task selected in step a) and, during the execution of the task, • acquisition of electronic signals from the various sensors; • extraction of characteristics of electronic signals; • formation of an observation tensor from the characteristics of the extracted signals; - c) repetition of steps a) and b) during a predetermined number of time periods forming a sequence; - d) formation of a learning tensor from the observation tensors formed at each time epoch (n), and of a control tensor from the tasks selected at each epoch, each term of the learning tensor and of the control tensor being associated with an epoch of the sequence; - e) formation of a predictive model, by regression between the learning tensor and the control tensor, the predictive model allowing to estimate a task, chosen from the list of tasks, imagined by the user according to the observation tensor formed at each epoch;

[0012] steps b), d) and e) being implemented by a processing unit;

[0013] the process being characterized in that it comprises: - definition of a weighting criterion for each period; - assigning a weight to each epoch, the weight being defined according to the weighting criterion for said epoch, according to which two different epochs of the sequence, for which the weighting criterion is different, are assigned two different weights;

[0014] the process being such that the formation of the direct model is carried out according to the weight respectively assigned to each epoch.

[0015] The term epoch corresponds to the equivalent of the Anglo-Saxon term "epoch", and corresponds to a time range, of predetermined duration, for example 1 second, during which steps a) and b) are implemented.

[0016] The weight assigned to an epoch may depend on the task selected during said epoch.

[0017] According to one possibility: - steps a) to e) are implemented during several successive sequences; - the weighting criterion is a frequency of occurrence of each task, the weight of each period is higher the lower the number of occurrences of the task, following successive sequences carried out.

[0018] According to one possibility, after each new sequence, the process includes an update of a total weighted number of occurrences for each task, the update comprising, for each task: - determining the number of occurrences in the new sequence; - weighting of the number of occurrences, in the new sequence, by the weight respectively assigned to the task in the new sequence; - summation of the weighted number of occurrences of the task, in the new sequence, to the total weighted number for said task resulting from the previous sequence, the latter being multiplied by a forgetting factor.

[0019] According to one possibility, the weighting criterion is a learning performance, the process comprising: - determination of a learning performance indicator for each task following each epoch; - determining the weight of each task based on the task learning performance criterion.

[0020] According to one possibility, the weighting criterion is the quality of the signals collected at each sequence, the process comprising: - determination of a quality criterion for the signals collected at each sequence; - determination of the weight of each task according to the quality criterion of the signals collected respectively during the execution of each task.

[0021] According to one possibility, in step e), the predictive model is formed by multivariate regression, comprising a calculation of a cross covariance tensor between the training tensor and the control tensor, the cross covariance tensor of each sequence being established from a product: - of the observation tensor; - of the control tensor; - weights respectively assigned to each era.

[0022] According to one possibility: - step c) is repeated so as to form several successive sequences, each sequence being assigned a rank; - step d) is implemented for each sequence; - in step e), the predictive model is formed from two sequences successive, from a sum of the cross covariance tensor established for the higher rank sequence and the cross covariance tensor established for the lower rank sequence multiplied by a forgetting factor.

[0023] According to one possibility: - the learning tensor and the control tensor are formed from a matrix, one dimension of which is the number of epochs in the sequence; - the weights assigned to each epoch of the sequence form a diagonal matrix where each dimension is the number of epochs per sequence, each term of the diagonal matrix corresponding to the weight assigned to each epoch respectively executed during said sequence.

[0024] According to one possibility, for at least one specific task, the weight is determined so that the number of occurrences of said specific task, weighted by the weight assigned to the specific task, is greater than the number of occurrences of at least one other task, weighted by the weight assigned to said other task.

[0025] According to one possibility, the weight assigned to each task is bounded by a predefined maximum value.

[0026] Another object of the invention is a direct neural interface, the direct neural interface comprising sensors pre-arranged around the brain of a user, and configured to detect electrophysiological signals, representative of the user's neural activity, the interface being programmed to control an actuator, by implementing a predictive control model, the predictive model being configured to generate an actuator control signal from detected electrophysiological signals, the interface comprising a processing unit, configured to acquire the electronic signals at each step b), and to implement steps d) and e) of a method according to the first object of the invention.

[0027] The actuator can be a device external to the user or a device implanted in the user's body.

[0028] The invention will be better understood upon reading the description of the exemplary embodiments presented later in this description, in connection with the figures listed below. FIGURES

[0029] Fig. 1 schematically represents a neural interface connected to a user, and connected to a processor capable of implementing a process according to the invention.

[0030] Fig. 2 represents the main steps of a method for implementing the invention.

[0031] Figure 3 shows a geometric mean of class recall determined by an offline predictive model parameterized by NPLS or by SW-NPLS, as a function of a class imbalance ratio. The ordinate axis corresponds to the geometric mean and the abscissa axis corresponds to the class imbalance ratio.

[0032] Figure 4 shows a comparison of the geometric mean of class recall determined by a predictive model implementing different learning algorithms. The y-axis represents the geometric mean and the x-axis represents each learning algorithm.

[0033] Figures 5A to 5C represent an evolution of class size as a function of a number of iterations by implementing a predictive model trained respectively by REW-NPLS (which corresponds to the prior art), as well as by implementing the invention, taking into account a maximum weight per class equal to 2 and 10.

[0034] Figures 5D to 5F represent the class imbalance ratio as a function of a number of iterations by implementing a predictive model trained respectively by REW-NPLS (which corresponds to the prior art), as well as by implementing the invention, taking into account a maximum weight per class equal to 2 and 10. PRESENTATION OF SPECIFIC IMPLEMENTATION METHODS

[0035] Figure 1 shows the main elements of a neural interface 1 according to the invention. It is a device comprising sensors 2i.. .2L b allowing the acquisition of electrophysiological signals representative of neuronal activity. It is an integer corresponding to the number of sensors. The sensors 2i...2L i are For example, cortical electrodes, with the subscript E denoting the number of cortical electrodes. The sensors 2i...2n are connected to a processing unit 3, for example, a microprocessor, via a wired or wireless connection. Each sensor 2p...2i is configured to detect an electrophysiological signal emitted by a user 10. From each detected electrophysiological signal, each sensor 21...2n transmits an electronic signal Sp... to the processing unit. The processing unit 3 is capable of implementing algorithms, such as predictive models, to detect characteristics of electronic signals specific to a task performed by the user. The processing unit 3 can, for example, be a processor connected to a memory implementing instructions to perform decoding algorithms such as those described in the publications cited in connection with the prior art. These algorithms allow the detected physiological signals to be decoded in order to determine the characteristics of the correlated electrophysiological signals of the mental tasks performed by the user 10.

[0036] A mental task, hereinafter referred to as a task, is understood to be an action imagined by a user to whom the direct neural interface is connected. This action corresponds to an intention to perform a specific task, in particular a motor task. The specific task is instructed to the user by a third party or by a dedicated algorithm.

[0037] During the operational functioning of the direct neural interface 1, as mentioned in relation to the prior art, the user successively performs mental tasks. The processing unit 3 receives the electronic signals S1...S1 [ transmitted by the sensors 2i.. .2L i, representing the electrophysiological signals produced by the user and detected by the sensors. From the detected electronic signals, when a correlation with a task is detected, the microprocessor generates a control signal S c for an actuator 6. Thus, the direct neural interface decodes the electrophysiological signals produced by the user 10 in order to generate, using a predictive model, control signals for an actuator. The quality of the decoding is all the better when the decoding algorithm has undergone high-quality training.

[0038] During the learning process, the user has a list T of tasks T to perform. As described in relation to the prior art, during a learning phase, a supervisor, human or machine, can ask the user to successively perform tasks k chosen from the list of K tasks. The objective is to progressively determine the electrophysiological characteristics best correlated with the tasks. These characteristics then allow the establishment of the predictive model, implemented during decoding, by which user 10 can control actuator 6 connected to processing unit 3.

[0039] Each task is to be performed within a time frame, called an epoch, n. The number of epochs to be considered for training is very high, potentially reaching several hundred or several thousand. In EP3563218, cited in the prior art, the principles of REW-NPLS learning are described. According to such learning, observational data forming a three-dimensional tensor are available at each epoch n.

[0040] The recorded electrophysiological signals undergo preprocessing, whereby the signal from each electrode, during each epoch, is subjected to a frequency analysis. This can, for example, be a wavelet transform, such as a Morlet wavelet transform, or a CCWT (Continuous Complex Wavelet Transform) decomposition. The duration of each epoch n can be 1 second or 2 seconds, with a temporal overlap between two consecutive epochs. More precisely, during each epoch, a frequency analysis is performed at regular intervals, for example, every 100 ms. An epoch thus groups together several frequency analyses shifted in time. During an epoch, several frequency analyses are performed, temporally offset from one another.

[0041] At each epoch n, we can associate an observation tensor Xn, of which: - the first mode corresponds to the position of each electrode, of dimension II. - the second mode corresponds to the temporal positions of the wavelets, of dimension 12; - the third mode corresponds to the frequency bands resulting from the frequency analysis, of dimension 13;

[0042] A training sequence u comprises N epochs n, extending over a time range θ. θ is an integer index assigned chronologically to each sequence. Each training sequence corresponds to a learning tensor Xu, of dimension Nx11x12x13: the learning tensor Xu groups N normalized observation tensors Xn as described below. More generally, the learning tensor Xu is of dimension Nx1..x1h x..x1H, with θ <h<H, h étant un indice et H étant un entier positif. Dans cet exemple, H = 3.

[0043] Each epoch n corresponds to a control signal Yn, which can be represented by a control vector of dimension (K, 1). Each term of the control vector corresponds to a task T from the list T of predefined tasks. Over the N epochs forming the time range u, the different control signals form a matrix Yu of dimension (K^N).

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] Alternatively, the control signal Yn can be a matrix, or even a multidimensional tensor, in which case the different control signals form a tensor Ÿ of mode NxJ 1.. .x Jg x...JG. G > 1. In general, each task k corresponds to a specific control signal. When the interface is implemented, the predictive model allows the observation tensor Xn to be converted into a control signal Yn for each epoch n. The predictive model can notably be a multilinear model, learned by regression between Xu and Y, for example using a multivariate partial least squares (PLS) method. Such a model allows an estimation of the control signal according to an expression of the type: yn = where cst 'c predictive model. Ÿa = BXn + b+(b where - B is a prediction tensor, comprising prediction coefficients; - b is a bias tensor; The term tensor encompasses both a vector (1st order tensor), a matrix (2nd order tensor) or higher order tensors. In the example described below, which is not exhaustive, the predictive model is such that that : Ÿn = B Xn + b (1'), where B is a matrix of dimension (K,P), and XD and b are vectors of dimension (P, 1) and (K, 1). Xn is a vector resulting from a vectorization of the tensor Xn, with o — TT^ T Equation (1) can be used to establish an emission probability. By taking into account probabilities of state changes, the user's state at different successive times can be estimated using a hidden-state Markov model, in which the task performed by the user at different successive times is considered a state. By implementing an algorithm, for example a forward algorithm, one can estimate the different successive states taken by the user. We will describe, in connection with [Fig. 2], the main steps of a method enabling the implementation of the invention, so as to form a predictive model as described by (1) or (1'). The predictive model is developed online, i.e., in real time or near real time, with iterative updating, as described in EP3563218. The steps involving mathematical processing are implemented by the processing unit 3. The objective of the predictive model is to estimate, when using the interface, at a time n, the control vector y according to (1) or (1')

[0054] It is known that the objective of a model established by NPLS is to project the observation tensor into a low-dimensional latent space, maximizing the covariance between the observation and control tensors.

[0055] In EP3563218, the tensors Xu and Yu and pXY determined at ^u-1 ^u-1 are used. through consecutive learning sequences u and u-1. We form weighted covariance tensors such that:

[0056] X = Xu + 2 CXX (2)

[0057] and

[0058] CXY =XYYu + 2C^J (3)

[0059] Where 2 is a forgetting factor. 2 is a positive real number, between 0 and 1.

[0060] This makes it possible to establish a predictive model, as described in (1), by putting in It employs a recursive REW-NPLS type algorithm. The advantage is regular updates to the predictive model, and learning is performed using limited memory resources.

[0061] This allows us to form a predictive model such that y = BX + b + E and bu are updated at each sequence.

[0062] Step 100: The user imagines a task k, at a time t, which corresponds to a known control signal YD. The task associated with time t may, in particular, correspond to a movement of the actuator, chosen from among K possible tasks. At the same time, the electrophysiological signals resulting from the different sensors are recorded. More precisely, the electrophysiological signals are recorded over an epoch, n, extending over a duration 6t from time t.

[0063] Each term Yn(k) of the control signal corresponds to a task k. The value of Yn(k) is equal to 1 when the user imagines that they are executing task k and 0 otherwise. One of the tasks may be an inactive task, designated by the acronym IS (Inactive State).

[0064] Step 110: Pre-processing. At each epoch n, the signals undergo time-frequency analysis, as described above, in order to form an observation tensor Xn-

[0065] Steps 100 and 110 are repeated N times, so as to form a learning tensor Xu- N can for example be equal to 150. N is the number of epochs n forming the learning sequence u. Hereafter, u denotes a sequence, that is to say a succession of instants of epochs n. For example, the total duration of steps 100 to 110 can be 15 seconds, each epoch lasting a duration ôt of 1 second, with a shift of 100 ms between two successive epochs n, ^+1, which implies an overlap of 90% between two successive epochs.

[0066] Step 120: Assigning a weight to each epoch.

[0067] The inventors have observed that recursive learning according to the prior art, as described in EP3563218, can lead to class imbalance. Class imbalance refers to an imbalance in the occurrence of certain classes, corresponding respectively to certain terms of the control vector. Indeed, certain tasks, corresponding respectively to certain terms of the control vector, may be underrepresented and require a longer training time. For example, when the actuator is an exoskeleton, controlling hand translation may require more training than controlling wrist rotation.

[0068] Furthermore, the imbalance affecting the difficulty of learning between tasks can vary over time, between different successive learning sequences.

[0069] Furthermore, during the learning process, an additional task may be added, which leads to adding a term to the control vector.

[0070] During learning, the user, or the supervisor, cannot, by himself, compensate for the imbalance between tasks, because the fact that learning some tasks k is more difficult than others cannot be controlled by the user.

[0071] Thus, at each epoch n, a weight wn is assigned, the value of which varies depending on whether one wishes to over- or under-represent the observation at said epoch n. More precisely, the weight wn depends on the task k assigned at epoch n, among the K possible tasks. The task k assigned at epoch n corresponds to the non-zero term of the control signal Y n. During the same sequence u, the weights wn corresponding to the same task k, that is, the same task, have the same value. Thus, for the same task k, the weight assigned during the same sequence u is

[0072] Substep 121: Determination of weights

[0073] Following a sequence u, the weights forming the matrix dia^W^ are established as follows: - we determine the number of occurrences of the majority class following the sequence - : W. »ù :

[0074] j is the number of occurrences of each class k following the previous sequence u -1. During the first sequence, is initialized, for example equal to 0.

[0075] is the number of occurrences of each class k, during the sequence u, before weighting.

[0076] 3 is the forgetting factor described previously. - we determine the weight assigned to each class k, during the par

[0077] N™J-XN^ (5) Wi=--HT-- uu

[0078] and w^ = 0si^=0(6)

[0079] It is preferable not to assign excessively large weights to certain tasks, so as not to increase the noise level affecting the determination of the predictive model. This amounts to avoiding an overweighting of certain classes k. Thus, it can be imposed that a maximum value wmax be established. When (5) leads to a value such that > ​​Wmax, then = Wmax-

[0080] After the weight assigned to each class ka has been defined, the weight wn associated with epoch n is such that wn = k corresponding to the task associated with epoch n.

[0081] Substep 1 2 2: determination of pjk

[0082] We determine qUi corresponds to the number of weighted occurrences of class k, by :

[0083] Nk +wM(7)

[0084] jÿ* is intended to be used when implementing expressions (5) and (6) in the following sequence u + 1.

[0085] Besides the frequency of occurrence of the tasks, other criteria can be taken into account to assign a weight to each epoch n. - Learning performance: for example, tasks for which learning performance is considered low can be weighted less. Learning quality can be assessed using a recall performance indicator, which corresponds to a ratio between the number of occurrences of correctly classified tasks and the number of tasks imagined by the user. - the presence of an outlier (aberrant value) at the time considered, in which case the weight can be chosen to be zero: the aim here is to assign a weight based on the quality of the recorded signals, in order to minimize or cancel the influence of signals considered to be aberrant; - the occurrence of a task change, by underweighting the moments occurring just after a task change being underweighted compared to the subsequent moments: this involves taking into account a reaction time of the user, occurring at each task change, and during which the neurological response of the user is considered to be transient;

[0086] More generally, a weighting criterion is defined for each epoch. This may be a criterion of frequency of occurrence of the task performed at each epoch, or of learning performance of the task selected at each epoch. or a quality criterion for the signals recorded at each time period, or a temporal criterion following a change of task.

[0087] The weighting criteria can be combined: for example, the weight of an epoch can be defined both according to the frequency of occurrence and the learning performance.

[0088] In order for the invention to make sense, it is preferable that at least two different epochs of the same sequence be respectively assigned two different weights.

[0089] Step 1 3 0: training of the learning tensor and the control vector for the sequence u.

[0090] To each learning sequence 11 corresponds a learning tensor Xw of dimension Nxl 1 xI2xI3: the learning tensor Xu groups N observation tensors Xn corresponding respectively to N epochs 11. Each observation tensor Xn is formed of terms (Xn / j ), WHERE H is the number of modes of the observation tensor.

[0091]

[0092]

[0093]

[0094] The training of the learning tensor involves a normalization of the observation tensors Xn, then a grouping of each normalized observation tensor formed for each epoch n of the same sequence u. Each observation tensor Xn is normalized by the following operations: \rTot_ (8) u - u-1 + 2ÎJ=1 wn - N^ot is a normalization term for the sequence u; XfTot is the size of the training set accumulated since the start of training, taking into account the weights. - 2 is the forgetting factor described earlier; - wn is the weight associated with each epoch n of the sequence u; - n™ is the normalization term for the previous sequence U-1. During the first sequence (u = 1), we take XQOt = 0. We then calculate an average yXi for each term of the N observation tensors, forming the sequence u

[0095] ijXi — —1.

[0096]

[0097] is each term with coordinate 1 of the observation tensor Xn; We then calculate a quadratic sum: SS^ =

[0098] A standard deviation is then calculated (11) °u VN^-l

[0099] And we normalize each term of the observation tensor Xn by:

[0100] ^iu^' (12), means "is replaced by" xnÀ4 a /

[0101] The same procedure is followed for each control vector Yn. An average pYk is calculated for each term of the N control vectors Yn for the sequence u

[0102] V13) n™ uA Pn-1 J

[0103] Yn is a term with coordinate k of the control vector Yn;

[0104] We then calculate a quadratic sum • SS^ = (ASS) A w n y 2 Y14) n,k A /

[0105] A standard deviation is then calculated

[0106] I SS^-N^urk2 (15) / rYk = .—----L_1X_.

[0107] And we normalize each term of the control vector by:

[0108] yny^k (16) ^nk k /

[0109] Step 130 involves normalizing each observation tensor Xa and each control tensor Yn by taking into account the weight wn associated with each epoch n of the sequence 11. This consists of calculating a time mean and a time standard deviation, weighted by the weight assigned to each epoch, of each term of the observation tensors and the control vector. The time mean and the time standard deviation are calculated for terms with the same coordinates, taking into account each epoch n forming the sequence u.

[0110] The learning tensor Xu and the control matrix Yu are then formed for the sequence u. Each normalized observation tensor Xn can be expressed as an observation vector Xn, of dimension P, with P = 11 x 12 x 13, following the vectorization of the tensor XD, in which case the learning tensor Xu is a learning matrix Xu formed from the N observation vectors: Xu= (XX The learning matrix Xu is of dimension (N, P).

[0111] We also form a control matrix Yu from each normalized control vector, y = ( yy ) T- In this example, Yu is of dimension (N, K).

[0112] Step 1 4 0: Establishing the predictive model.

[0113] We now describe the establishment of a predictive model from the learning matrix Xu and the control matrix Yu resulting from step 130. It is first necessary to establish the predictive model described in (1) and (1'). Sub-step 141

[0114] From the learning matrix Xu and the control matrix Yu, the covariance and cross-covariance matrices are defined as follows:

[0115] cT'= xldiag[WjX:!+ACy'{ <20>

[0116] and

[0117] C^Y = xT,diag[Wl)Yu+XClrl 1211

[0118] diag(W^ is a diagonal matrix of dimension (N,N). Each term of dia^W^ is the weight wn assigned at epoch n, calculated during step 120.

[0119] Substep 1 42: In this substep, the matrix B u and the vector bu are determined from the covariance and cross covariance matrices Qxx and QXY resulting from the previous substep, as described in EP3563218. This corresponds in particular to step 140 of EP3563218. In EP3563218, the predictive control model is updated by multivariate partial least squares linear regression (PLLS), but other types of multivariate regressions may be used.

[0120] The predictive model can be used to estimate the most probable task ordered by the user, using a Hidden Markov Model (HMM) type algorithm, where each task is considered a user state. In this case, each epoch is assigned a user state, the user state corresponding to the execution of a task. Successive states are estimated by an HMM type algorithm, using the predictive model, as described in EP3789852.

[0121] Step 1 5 0: reiteration. Steps 100 to 140 are repeated for a subsequent sequence, which allows for regular, or even continuous, updating of the predictive model. Experimental trials

[0122] The method described above was implemented based on data collected during the clinical trial “Brain Computer Interface: Neuroprosthetic Control of a Motorized Exoskeleton,” reported on clinicaltrials.gov under reference NCT02550522. These sequences are described in AL Benabid et al., “An exoskeleton controlled by an epidural wireless brain-machine interface in a tetraplegia patient: a proof-of-concept demonstration,” The Lancet Neurology, vol. 18, no. 12, Art. no. 12, Dec. 2019, and in A. Moly et al., “An adaptive closed-loop ECoG decode for long-term and stable bimanual control of an exoskeleton by a quadriplegia,” J. Neural Eng., vol. 19, no. 2, Art. no. 2, Mar. 2022.

[0123] EcoG signals were recorded using a WIMAGINE wireless implant as described in Mestais C. et al “WIMAGINE: Wireless 64-Channel ECoG recording implant for long terni clinical applications”, IEEE Transactions on neural Systems and rehabilitation engineering, Vol. 23, Nol, January 2015.

[0124] An observation tensor was calculated every 100 ms, according to a sliding window. Frequency analysis was performed by continuous complex wavelet transform (CCWT) on the last second of the signal, with fifteen wavelets derived from the Morlet mother wavelet, and centered on fifteen frequencies equally spaced between 10 and 150 Hz.

[0125] Each sequence consisted of a buffer storing the data from each sensor for 15 seconds. This corresponds to a set of 150 observation data points per buffer. The duration of one epoch was 1 second.

[0126] During training, the user was given instructions to perform mental tasks designed to control a virtual avatar. The predictive model was trained to decode five different tasks: - translation of the right hand (AS RH); - translation of the left hand (AS LH); - rotation of the right wrist (AS RW); - rotation of the left wrist (AS LW); - inactivity (IS).

[0127] At each epoch, that is to say every 100ms, we had a control vector Yt, formed of 5 terms Y ^k)' Taking the value 1 for the task k executed, k being between 1 and 5.

[0128] Ten sequences were available, from which five sequences were randomly chosen for training the predictive model and five sequences for testing, during which the predictive model was applied. This was repeated 50 times, avoiding having the same sets of test and training sequences.

[0129] The respective durations of the sequences were respectively 37 min 26 s, 29 min 8 s, 48 ​​min 3 s, 9 min 35 s, 50 min 6 s, 53 min 2 s, 31 min 41 s, 37 min 58 s, 47 min 34 s, 22 min 12 s.

[0130] During the test trials, each model was evaluated by a performance criterion corresponding to a geometric mean of the recall, denoted Gm. The recall of a class k is defined by y = ^jy^ = j. The geometric mean is defined as : 101311 Gm = = =

[0132] Gm is equal to 1 if the data set is well classified. The recalls calculated for each class, as well as the geometric mean Gm, are independent of the size of each class.

[0133] For each predictive model, a class imbalance ratio (CIR) was quantified such that _ size of the majority class size of the minority class (23)

[0134] The greater the imbalance between the classes, the higher the CIR ratio. A CIR of 1 corresponds to a balance between each class.

[0135] During the tests, Ÿn was estimated by applying the predictive model to the data corresponding to each epoch, and YD corresponds to the control vector corresponding to the task requested.

[0136] Different values ​​of wmax were taken into account, respectively ranging from 2 to infinity. When wmax is equal to infinity, this means that there is no maximum value assigned to the weights.

[0137] Offline training was also performed, without taking into account the forgetting factor in the covariance and cross-covariance matrices. This amounts to defining the predictive model as follows: - either by NPLS; - either by implementing the invention, without taking into account the forgetting factor (X = 0), i.e. by weighted NPLS, or SW-NPLS (Sample Weighted NPLS - N partial least squares with weighted samples), in which case the tensors taken into account, at each sequence, are: [0B8] cxx = xTdiag^w)x^ (24) and cxy =

[0139] In this case, the weight matrix diag(W) is a diagonal matrix of size (N,N), where each term is associated with an epoch n, and whose value is the weight assigned to the class k corresponding to the epoch n.

[0140] Each class is assigned a weight whose value is N™aj (25)

[0141] XinaJ corresponds to the value of the majority class and corresponds to the value of class k.

[0142] Offline learning was carried out by combining the sessions used for learning, the duration of which was therefore between 130 and 240 minutes. As previously mentioned, offline learning is difficult to integrate into a compact device because it requires significant memory. given the amount of data processed, it is used here as a tool for comparing online learning outcomes.

[0143] Figure 3 shows the geometric mean Gm, as defined in (9), calculated by considering the predictive models established offline by NPLS or SW-NPLS, respectively. The geometric mean is plotted against the IRC. Each point in this figure corresponds to a test of a predictive model obtained by NPLS or SW-NPLS using training data from 5 sessions randomly selected from the first 10 sessions in the database. The predictive model was tested using the 5 other sessions not selected for training from the first 10 sessions. The experiments (training and testing) were repeated 50 times. It is observed that in the case of significant imbalance (high imbalance ratio), the geometric mean of the recalls Gm of the SW-NPLS algorithm far surpasses that obtained by NPLS.This shows that in the event of an imbalance between classes, the predictive model obtained by SW-NPLS is more accurate than the predictive model obtained by NPLS. This comparison confirms the value of weighting underrepresented classes.

[0144] Figure 4 shows a comparison of the geometric mean of class recall determined by different algorithms. It is a boxplot presentation. The results are represented by a box whose bounds are the first and third quartiles, and the line corresponds to the median. The extreme values, outside the box, show the minimum and maximum values. The x-axis corresponds to each implemented algorithm, with, from left to right: - NPLS (offline); - SW-NPLS (offline, with each sample weighted); - REW-NPLS (recursive online model); - the invention (RSW-NPLS - online recursive model, with weighting of each sample), considering respectively a maximum value wmax equal to 2, 4, 6, 8, 10 and infinity. Considering wmax equal to infinity is equivalent to not taking into account a maximum value.

[0145] On [Fig.4], a vertical dotted line has been drawn, separating offline and online learning.

[0146] It is observed that the maximum value of Gm is obtained using a predictive model developed offline by SW-NPLS. However, as previously stated, this is a model whose training is performed offline. This appears to correspond to an optimum to be achieved.

[0147] The performance of recursive models, whose training is carried out online, is better when class weighting is implemented (RSW- models NPLS). We also observe that the wmax value influences classification performance. Considering winax = 8 or wmax = 10, classification performance is close to that obtained with the model established offline by SW-NPLS (non-recursive approach).

[0148] In the absence of taking into account a value Wmax, that is to say by considering wmax equal to infinity, the classification performance is worse than when taking into account a value wmax.

[0149] Figures 5A to 5C illustrate the evolution of class size as a function of the number of iterations, implementing a predictive model trained respectively by REW-NPLS (a recursive NPLS approach without epoch weighting, which corresponds to the prior art), and by implementing the invention, taking into account a maximum weight per class of 2 and 10, respectively. Without weighting (with REW-NPLS), the IS class is over-represented by a factor of 2. The REW-NPLS algorithms improve the balance between the classes. When wmax = 2, only two tasks (SSLH and ASRH) are balanced with the IS class. When wmax = 10, the constraint imposed by the algorithm is sufficiently large that all classes are balanced after a sufficient number of iterations.

[0150] Figures 5D to 5F represent the class imbalance ratio as a function of a number of iterations by implementing a predictive model trained respectively by REW-NPLS (which corresponds to the prior art), as well as by implementing the invention (RSW-NPLS), taking into account a maximum weight per class equal to 2 and 10. It can be observed that the weighting reduces the class imbalance ratio, as explained in (8), and this more when wmax= 10.

[0151] In Figures 5A to 5F, each curve corresponds to a task, among the tasks previously listed. Variants

[0152] According to one possibility, it is advantageous, during training, to deliberately aim for a specific imbalance between the tasks performed by the user. This may notably concern a predetermined task, for example, resting. Insufficient training of the resting state can generate, during the implementation of the predictive model, false activations whereby the user is considered to be in an active state when the desired state is a resting state.

[0153] For example, one can aim for a higher proportion of occurrences for rest than for other active tasks. To do this, for each task k, a target proportion R^ is assigned.

[0154] In substep 121, the target proportion R^est is taken into account as follows: [°155] NTina^ — iv u — inax| r 1c )

[0156] and

[0157] (5') ™k =------ uii

[0158] The inventors believe that it is preferable for the resting task to be over-represented by a factor of 2 to 2.5 compared to the other active tasks. This improves the performance as well as the stability of the implementation of the direct model.

[0159] In the description of substeps 121, weights ^uji were calculated, corresponding respectively to each class, to achieve a balance between the different classes, so as to overweight the classes to the extent that they are in the minority. According to another possibility, the majority classes can be underweighted.

[0160] The invention enables online learning of a predictive model and can be implemented for any BCI-type system, including devices in which decoding is transmitted to the spinal nerves or muscles. In this case, the actuator is previously implanted in the user's body: for example, a stimulation device.

Claims

Demands

1. A method for learning a direct neural interface, the direct neural interface being connected to sensors (2i..2n) previously arranged around a user's brain, each sensor being configured to detect an electrophysiological signal dependent on the user's neural activity, the interface being configured to control an actuator (6) based on the detected electrophysiological signals, the learning method comprising: a) selection of a mental task (k) to be performed, chosen from a predetermined list of tasks (1, ..., ... K); b) execution, by the user, of the mental task selected in step a) and, during the execution of the task: • acquisition of electronic signals from the various sensors; • extraction of characteristics of electronic signals; • formation of an observation tensor (Xn) from the characteristics of the extracted signals; c) repetition of steps a) and b) during a predetermined number of time epochs (n), forming a sequence (u); d) formation of a learning tensor (Xu) from the observation tensors (Xn) formed at each time epoch (n), and of a control tensor (Ÿu) from the tasks selected at each epoch, each term of the learning tensor and the control tensor being associated with an epoch of the sequence; e) formation of a predictive model (F), by regression between the learning tensor and the control tensor, the predictive model allowing to estimate a task, chosen from the list of tasks, imagined by the user according to the observation tensor formed at each time; steps b), d) and e) being implemented by a processing unit; the process being characterized in that it comprises: - definition of a weighting criterion for each epoch; - assignment of a weight^a) to each epoch, the weight being defined according to the weighting criterion for said epoch, according to which two different epochs of the sequence, for which the weighting criterion is different, are assigned two different weights; the process being such that the formation of the direct model is carried out according to the weight respectively assigned to each epoch.

2. A method according to claim 1, wherein the weight assigned at a time depends on the task selected during said time.

3. A method according to any one of the preceding claims, wherein: - steps a) to e) are implemented during several successive sequences; - the weighting criterion is defined according to a frequency of occurrence of each task, the weight of each epoch (^) is higher the lower the number of occurrences of the task, following the successive sequences carried out.

4. A method according to claim 3 comprising, after each new sequence (u), an update of a total weighted number of occurrences for each task, the update comprising, for each task (i): - determination of a number of occurrences in the new sequence (”); - weighting of the number of occurrences, in the new sequence, by the weight (w^f) respectively assigned to the task in the new sequence; - summation of the weighted number of occurrences of the task, in the new sequence, to the total weighted number for said task J resulting from the previous sequence J' the latter being multiplied by a forgetting factor 0).

5. A method according to any one of the preceding claims, wherein the weighting criterion is defined according to a learning performance, the method comprising: - determining a learning performance indicator for each task following each epoch; - determining the weight of each task according to the learning performance criterion of the task.

6. A method according to any one of the preceding claims, wherein the weighting criterion is defined according to a quality of the signals collected at each sequence, the method comprising: - determination of a quality criterion of the signals collected at each sequence; - determination of the weight of each task according to the quality criterion of the signals collected respectively during the execution of each task.

7. A method according to any one of the preceding claims, wherein in step e), the predictive model is formed by multivariate regression, comprising a calculation of a cross covariance tensor between the learning tensor and the control tensor, the cross covariance tensor of each sequence being established from a product of: - the observation tensor; - the control tensor; - the weights (wn) respectively assigned to each epoch.

8. A method according to claim 7, wherein - step c) is repeated so as to form several successive sequences, to each sequence being assigned a rank (u); - step d) is implemented for each sequence; - during step e), the predictive model is formed from two successive sequences, from a sum of the cross covariance tensor established for the higher rank sequence (u) and the cross covariance tensor established for the lower rank sequence (ul) multiplied by a forgetting factor (A).

9. A method according to any one of claims 7 or 8, wherein: - the learning tensor and the control tensor are formed of a matrix, one dimension of which is the number of epochs in the sequence; - the weights (wn) form a diagonal matrix (diag(W^), each dimension of which is the number of epochs per sequence, each term of the diagonal matrix corresponding to the weight assigned to each epoch respectively executed during said sequence.

10. A method according to any one of the preceding claims, wherein for at least one specific task, the weight is determined so that the number of occurrences of said specific task, weighted by the weight assigned to the specific task, is greater than the number of occurrences of at least one other task, weighted by the weight assigned to said other task.

11. A method according to any one of the preceding claims, wherein the weight assigned to each task is bounded by a predefined maximum value.

12. Direct neural interface, the direct neural interface comprising sensors (2i...2n) pre-arranged around a user's brain and configured to detect electrophysiological signals representative of the user's neural activity, the interface being configured to control an actuator (6) by implementing a predictive control model, the predictive model being configured to generate

13. an actuator control signal from detected electrophysiological signals; the interface comprising a processing unit (3), configured to acquire electronic signals during each step b), and to implement steps d) and e) of a method according to any one of the preceding claims. Interface according to claim 12, wherein the actuator is a device external to the user or a device implanted in the user's body

Images