Method and device for automatically processing data on brain electrical activity

Abstract

The invention relates to a method for automatically processing data on the electrical activity of a brain of a user and comprising the following steps: receiving electrical data reflecting electrical activity of the brain of the user and measured by electroencephalography; in a time window, extracting one or more activity characteristics of the brain from the electrical data; in a time window, normalising said one or more activity characteristics to produce one or more normalised characteristics; generating one or more multi-dimensional characteristic images based on said one or more normalised characteristics; and processing said one or more multi-dimensional characteristic images by means of an algorithm, so as to produce a prediction of a mental state of the user, wherein said steps are repeated with a refresh rate at least equal to 1 Hz, preferably at least equal to 3 Hz, and preferably equal to 10 Hz.

Description

Method and device for the automatic processing of brain electrical activity data technical field

[0001] In general, the invention relates to the field of processing electrical activity data from the brain of a human or animal user. In particular, the invention relates to the automatic processing of electrical activity data from a user's brain, measured by electroencephalography, for the purpose of recognizing the user's mental state. State of the art

[0002] A user's mental state can be defined as a reflection of their state of mind. A mental state can include, for example, the emotions felt by the user, their intentions, such as their intentions to move, and other metrics such as concentration, fatigue, engagement, and the user's mental workload.

[0003] Recognizing a user's mental state can be beneficial in various fields, such as scientific research, mental health, security, and entertainment. A user's mental state can manifest in different ways, for example, through behavioral, vocal, facial, and physiological markers. Physiological markers include, for example, brain electrical activity, which can be measured by electroencephalography (EEG), muscle electrical activity, which can be measured by electromyography (EMG), eye movements, body temperature, and so on.Among physiological markers, brain electrical activity is of particular interest for recognizing a user's mental state because brain electrical activity is difficult for a user to manipulate, unlike other markers such as muscle electrical activity or eye movements.

[0004] Among the methods for recognizing a user's mental state, there is particular interest in automatic methods, i.e., those that can be entirely performed by a machine without human intervention, as opposed to manual methods, i.e., those that include at least one step requiring human intervention. Indeed, automatic methods can be easier to implement and / or deploy on a large scale than manual methods.

[0005] Among the methods for recognizing a user's mental state, there is particular interest in dynamic methods, i.e., where at least part of the In dynamic methods, the data being processed is not known at the start of processing, unlike static methods, where all the data to be processed is known at the start of processing. A special case of a dynamic method is real-time processing, where the data is processed at least as quickly as it is received.

[0006] The document "Li et al., "Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks", Applied Sciences (2017), vol. 7, no. 1060" (hereinafter "Li et al. (2017)"), published on October 13, 2017, describes a method for analyzing EEG signals to determine a user's emotion. This method is static. Therefore, it has the drawback of only allowing data analysis after the data acquisition period has ended.

[0007] The document "Wang et al., "EEG emotion recognition based on differential entropy feature matrix through 2D-CNN-LSTM network", EURASIP Journal on Advances in Signal Processing (2024), 2024:49" (hereinafter "Wang et al. (2024)"), published on April 8, 2024, describes another method for analyzing EEG signals to determine a user's emotion. This method is also static. Therefore, it also has the drawback of only allowing data analysis after the data acquisition period has ended.

[0008] In the field of recognizing a user's mental state, there is therefore a need for an automatic and dynamic method of recognizing a user's mental state based on a measurement of electrical activity in the user's brain. Description of the invention

[0009] One of the objects of the invention is to meet the aforementioned needs, in particular to provide a method for the automatic and dynamic recognition of a user's mental state. To this end, the invention proposes a method for the automatic processing of electrical activity data from the brain of a human or animal user in order to recognize the user's mental state, in cases where at least part of the brain electrical activity data is not known at the start of said data processing. The invention further provides a computer program implementing the method for the automatic processing of brain electrical activity data, as well as a device for the automatic processing of brain electrical activity data.

[0010] According to a first aspect, the invention proposes a method for the automatic processing of electrical activity data from a user's brain, comprising the following steps: a) receiving electrical data reflecting the electrical activity of the user's brain and measured by electroencephalography; d) within a time window, extracting one or more brain activity features from the electrical data of the previous step; e) within a time window, normalizing said activity feature(s) from step d) to produce one or more normalized features; f) generating one or more multidimensional characteristic images (MCIs) based on said normalized feature(s) from step e); g) processing said MCIs from step f) by an algorithm to produce a prediction of the user's mental state;where steps a) to g) are repeated with a refresh rate of at least 1 Hz, preferably at least 3 Hz, preferably 10 Hz.

[0011] As detailed below, optional steps, respectively denoted b) and c), for converting and processing electrical data from step a) may be carried out between step a) and step d). Where applicable, the data on which step d) is performed is data from the step directly preceding step d), i.e., step a), b), or c). Thus, the data in step d) is calculated directly or indirectly from data from step a).

[0012] Preferably, steps a) to g) of the method according to the invention are carried out in the order presented above, i.e. alphabetical order.

[0013] Within the framework of the present invention, a user can be a human person or an animal.

[0014] A time window used during a step of the method according to the invention at a given time contains the data preceding that time for a period equal to the duration of the window. In other words, if "t" denotes the current time at which the method according to the invention is executed, a time window of duration "T" will contain the data for the interval [tT,t]. During the initial executions of the method according to the invention, there may not be enough data to fill the entire time window. In this case, it is possible to replace the missing data with null values, for example, and then execute the method according to the invention.

[0015] Generally, the time windows corresponding to successive repetitions of the same step of the method according to the invention may overlap, at least partially. In particular, these time windows may overlap if they are longer than the refresh period. In this document, the refresh period is the inverse of the refresh rate. The refresh rate may also be referred to here as the "refresh frequency" or "refresh rate." For example, a refresh rate of 1 Hz corresponds to a refresh period of 1 s. As another example, a refresh rate of 3 Hz corresponds to a refresh period of 1 / 3 s. As yet another example, a refresh rate of 10 Hz corresponds to a refresh period of 0.1 s.

[0016] The method according to the invention works in the following manner. First, electrical data is received, said electrical data reflecting electrical activity of the user's brain and being measured by electroencephalography.

[0017] Next, one or more brain activity features are extracted from the received electrical data within a specified time window. This received electrical data may have undergone one or more processing steps between its reception and its use in extracting one or more brain activity features. For the purposes of this document, a brain activity feature is a value used to quantify the user's brain activity and calculated based on electrical data reflecting that brain activity.

[0018] Next, the brain activity feature(s) contained within a time window are normalized to produce one or more normalized features. For the purposes of this document, a "normalized feature" is an activity feature that has undergone a normalization step and whose value is therefore contained within a bounded interval whose limits do not vary from one repetition of the method according to the invention to another. The normalization of one or more brain activity features is necessary to enable the processing of said normalized feature(s), or of data calculated based on said normalized feature(s), by an algorithm in a subsequent step of the method according to the invention.Furthermore, the normalization of one or more brain activity characteristics makes it possible to compare values ​​produced within the framework of the method according to the invention and obtained under different conditions, such as different mental states in the same user within the same session. of measurements, different measurement sessions with the same user, and different measurement sessions with different users. Indeed, several factors that can affect the measurement of the user's brain electrical activity can vary between different conditions, such as environmental factors, like temperature or relative humidity, electrical factors, like the quality of contact between an electroencephalography electrode and the user's skull, or other factors.

[0019] Next, one or more multidimensional characteristic images (MCIs) are generated based on one or more normalized features. An MCI is an image calculated based on one or more normalized features and allows for the preservation of information on the spatial origin of each feature on which the MCI is based; i.e., for a given feature, information on the position on the user's skull of the electroencephalography electrode(s) used to measure the electrical data from which the feature is derived.

[0020] Alternatively, an ICM can also be a spectral image, such as a spectrogram, i.e., a representation of the frequency content, or spectrum, of one or more brain activity features, possibly normalized, as a function of time. The spectrum of one or more brain activity features can be obtained, for example, by means of a Fourier transform of said feature(s).

[0021] Finally, the aforementioned ICM(s) are processed by an algorithm to produce a prediction of the user's mental state. Such a prediction enables the recognition of the user's mental state.

[0022] The steps included in the method according to the invention of receiving electrical data, extracting one or more brain activity features, normalizing one or more brain activity features, generating one or more ICMs, and predicting a mental state of the user are performed in a loop repeatedly with a refresh rate, i.e. a cadence, of at least 1 Hz. At each repetition of said steps, a prediction of a mental state of the user is produced, giving rise to a sequence of mental states produced by the method according to the invention.

[0023] In the method according to the invention, step e) of normalization is carried out before step f) of generating one or more ICMs. Indeed, the said ICM(s) are generated from the said normalized characteristic(s) produced in step e).

[0024] The method according to the invention is automatic in the sense that all the steps included in the method can be entirely carried out by a machine without human intervention. In particular, the method according to the invention can be implemented by computer.

[0025] The method according to the invention is also dynamic in that it allows the recognition of a user's mental state based on a limited set of electrical brain activity data, and in particular without prior knowledge of all the data from a measurement session. Indeed, the steps in the method according to the invention are performed repeatedly in a loop. Thus, with each repetition of these steps, a prediction of the user's mental state is made without knowing future data and based on a limited amount of past data, the limit being determined by the duration of the longest time window used in the method according to the invention.

[0026] Preferably, the refresh rate can be at least 3 Hz to achieve a user mental state recognition rate faster than the threshold of consciousness, which is known to be approximately 300 ms, or about 3 Hz. Preferably, the refresh rate can be 10 Hz, which may represent a good compromise between a reasonable time to execute the method according to the invention, allowing it to be run using a reasonably complex computer device, and a limited latency, on the order of 100 ms.

[0027] The inventors propose several possible embodiments of the invention including optional features, some of which can be combined.

[0028] In one embodiment, the duration of the time window for step d) of extracting one or more brain activity features is between 1 s and 10 s, preferably between 1 s and 5 s, preferably between 1 s and 3 s, and preferably 2 s. Indeed, the duration of the time window used for step d) of extracting one or more brain activity features from the electrical data is at least 1 s in order to have enough data to calculate a frequency representation of the electrical data contained within said time window. Furthermore, the duration of said time window is at most 10 s in order to avoid having data corresponding to several overlapping mental states within said time window.Furthermore, the duration of said time window is preferably at most equal to 5s and even more preferably at most equal to 3s in order to limit the amount of data to be processed and thus limit the amount of calculations required to process said data. Finally, the duration of said time window is preferably equal to 2s because this duration is suitable for recognizing a mental state of the user such as an emotion or an intention to move.

[0029] In one embodiment, the duration of the time window for step e) of normalizing one or more brain activity characteristics is at most 10 seconds, preferably at most 5 seconds, and preferably 2 seconds. This is because the duration of the time window used for step e) of normalizing one or more brain activity characteristics is at most 10 seconds to avoid data corresponding to multiple overlapping mental states within said time window. Furthermore, the duration of said time window is preferably at most 5 seconds to limit the amount of data to be processed and thus limit the amount of computation required for processing said data. Finally, the duration of said time window is preferably 2 seconds because this duration is suitable for recognizing a user's mental state, such as an emotion or an intention to move.

[0030] In one embodiment, step e), which normalizes one or more brain activity characteristics, is performed using a sigmoid function parameterized by statistical parameters calculated based on the brain activity characteristics within the time window of step e), and a linear application of the image values ​​of the sigmoid function to a bounded interval, preferably the interval [0,1]. Indeed, the algorithm used in step g), which predicts the user's mental state based on MCIs, can be configured to receive values ​​within the interval [0,1]. Therefore, it may be necessary to normalize the brain activity characteristics to the interval [0,1], i.e., bring the values ​​of these characteristics into the interval [0,1] so that they can be processed by the algorithm in step g).Furthermore, normalizing values ​​obtained under different conditions within the same bounded interval allows for comparison of these values, the conditions being, for example, different mental states in the same user within the same measurement session, different measurement sessions with the same user, and different measurement sessions with different users.

[0031] A well-known and commonly used normalization method is the "min / max" normalization method, which involves normalizing data using a linear application that takes into account minimum and maximum values ​​within a given interval. The "min / max" normalization method can be performed by applying the following formula to the data within the interval: where x is the value of a data point to be normalized, x' is the normalized value of x, x min is the minimum value of the data over the interval, and x ma x is the maximum value of the data over the interval. In order to apply this method, it is therefore necessary to know in advance the minimum and maximum values ​​of the data in the interval in which we wish to normalize said data.

[0032] The invention aims to normalize the user's brain activity characteristics over a time interval equal to the duration of a measurement session, using a dynamic method where at least some of the processed data is unknown at the start of processing. In this context, the minimum and maximum values ​​of each brain activity characteristic may not be known for the entire measurement session. Therefore, the "min / max" normalization method is not applicable within the scope of this invention.

[0033] Furthermore, during a measurement session, extreme values ​​may appear in the brain activity characteristics, particularly due to possible discontinuities in the data measuring the user's brain electrical activity, which can negatively impact the performance of the method according to the invention. Several factors can cause such discontinuities, including intrinsic factors such as a sudden variation in the user's brain electrical activity, and extrinsic factors such as a sudden movement by the user, an impact on the measuring device used to acquire the data, or electromagnetic interference caused by a source external to the measuring device.In particular, extrinsic factors can cause data to vary by several orders of magnitude from their mean value, thus introducing significant discontinuities in the data. It should be noted that the min / max normalization method described above is also sensitive to extreme values. To illustrate this, imagine an interval within which we wish to normalize data, containing on the one hand a set of useful values ​​and on the other hand an extreme value several orders of magnitude larger (or in other words, higher) than the set of useful values. In this case, the presence of the extreme value within the interval can distort a min / max normalization operation by reducing the set of useful values ​​to a very narrow range after normalization.The sensitivity to extreme values ​​of the "min / max" normalization method described above provides an additional reason for this. which it is not advantageous to apply such a standardization method within the framework of the invention.

[0034] Within the scope of the invention, it is therefore advantageous to normalize brain activity characteristics using a method that is relatively insensitive to extreme values ​​and does not require prior knowledge of the minimum and maximum values ​​of said characteristics within the interval in which normalization is desired. The use of a sigmoid function makes this possible thanks to the saturation effect of the image values ​​of the sigmoid function near its asymptotes for extreme input values. A sigmoid function is a special case of a monotonic nonlinear function with a shallower slope for values ​​further from the origin and a steeper (or, in other words, higher or greater) slope for values ​​closer to the origin.The fact that the slope of this function is lower for values ​​further from the origin leads to a saturation of the function's image values ​​for these values ​​further from the origin. Therefore, this function allows for the normalization of brain activity characteristics while being relatively insensitive to extreme values ​​of these characteristics. Furthermore, in order to compare the normalized values ​​under the different conditions mentioned above, the sigmoid function can be parameterized by statistical parameters calculated based on the brain activity characteristics included in the time window of step e).Finally, in cases where the image values ​​of the sigmoid function would not be included in a desired bounded interval, such as the interval [0,1], it is possible to bring said values ​​into said interval with low computational complexity via an additional step of linear application of the image values ​​of the sigmoid function to said interval.

[0035] Preferably, the normalization sigmoid function used in step e) is a hyperbolic tangent parameterized by the mean and standard deviation of the brain activity features included in the time window of step e), which allows step e) of normalization of one or more brain activity features to be carried out with low computational complexity.

[0036] In particular, the normalization function of step e) can be a hyperbolic tangent parameterized by the mean p and standard deviation o of the brain activity characteristics included in the time window of step e) according to the following expression: tanh(s, [i, o') = tanh - J where s denotes the values ​​to which the normalization function applies and where A is a scaling factor (or normalization constant). This function allows us to to carry out step e) of normalisation of one or more characteristics of brain activity with reduced computational complexity.

[0037] More generally, within the scope of the invention, other normalization functions may be used in the normalization step (e). Thus, according to one embodiment, step (e) comprises applying a normalization function to the activity characteristic(s) of step (d) to produce one or more normalized characteristics, where the normalization function is a nonlinear function. Preferably, the normalization function of step (e) is a monotonic nonlinear function whose slope decreases for activity characteristics whose absolute value increases. In other words, this normalization function is a monotonic nonlinear function whose slope is shallower for activity characteristic values ​​farther from the origin and whose slope is steeper (or, in other words, higher or greater) for activity characteristic values ​​closer to the origin.In this context, the "origin" refers to a zero value for an activity characteristic from step d), to which the normalization function from step e) is applied. Thus, the slope of this function is lower for values ​​further from the origin than for values ​​closer to the origin. This leads to saturation of the function's image values ​​for these values ​​further from the origin. Therefore, this function allows for the normalization of brain activity characteristics with low sensitivity to extreme values ​​(or "outliers") of these characteristics. Furthermore, this normalization function allows step e) of normalizing one or more brain activity characteristics to be performed with reduced computational complexity by eliminating the need to consider all the data acquired during normalization step e).

[0038] In summary, step e) of normalizing the method according to the invention preferably comprises applying a normalization function to the activity characteristic(s) of step d) to produce one or more normalized characteristics. In this context, the normalization function of step e) takes as input (or as abscissas) the activity characteristic(s) of step d) and produces as output (or on the ordinates) one or more normalized characteristics.

[0039] According to one embodiment, step e) includes applying a normalization function to the activity characteristic(s) of step d) to produce one or more normalized characteristics, where the normalization function is a non-linear function.

[0040] According to one embodiment, step e) includes applying a normalization function to the activity characteristic(s) of step d) to produce a or several normalized characteristics, where the normalization function is a monotonic nonlinear function whose slope decreases for activity characteristics whose absolute value increases. In other words, the normalization function of step e) of this embodiment is a monotonic nonlinear function whose slope is smaller for activity characteristic values ​​further from the origin and whose slope is larger for activity characteristic values ​​closer to the origin. In this context, "the origin" refers to a zero value of an activity characteristic in step d), to which the normalization function of step e) is applied.

[0041] In one embodiment, step e) comprises applying a normalization function to the activity characteristic(s) of step d) to produce one or more normalized characteristics, where the normalization function is a sigmoid function, also referred to herein more simply as a sigmoid function. A sigmoid function is a special example of a monotonic nonlinear function whose slope decreases for activity characteristics whose absolute value increases.

[0042] In one embodiment, step e) comprises applying a normalization function to the activity characteristic(s) of step d) to produce one or more normalized characteristics, where the normalization function is a hyperbolic tangent. A hyperbolic tangent is a special example of a sigmoid function.

[0043] The use of a normalization function in step e) according to the embodiments described above combines advantageously, i.e., offers a synergistic effect, with the dynamic processing of the user's brain activity characteristics by the method according to the invention. Indeed, the normalization functions of step e) described above have the advantage of being relatively insensitive to extreme values ​​(or "outliers") of the acquired data, even though it is not possible to know in advance all of this acquired data, and in particular its minimum and maximum values, within the framework of the dynamic processing according to the invention, where the data are processed in limited time windows.

[0044] According to one embodiment, the method according to the invention further comprises, between steps a) and d), a step b) of converting electrical data from step a) from an analog form into a digital form, said conversion step b) comprising at least one conversion operation from the following group: - amplification; - quantification; - sampling. Converting electrical data from an analog to a digital form facilitates the processing of said electrical data in subsequent steps of the method according to the invention. Signal amplification is a process that consists of multiplying the signal values ​​by a given gain. Signal amplification may be necessary to facilitate its quantization. Signal quantization is a process that consists of approximating values ​​from a large set, i.e., comprising a large number of elements, potentially a continuous set, with values ​​from a smaller set, i.e., comprising a smaller number of elements, potentially a finite number of elements. Signal sampling is a process that consists of taking values ​​from a signal at defined, generally regular, intervals in order to produce a sequence of discrete values ​​called samples.

[0045] According to one embodiment, the method according to the invention further comprises, between steps a) and d), a step c) for processing electrical data from step a) contained within a time window, said processing step c) comprising at least one processing operation from the following group: - removal of a trend value; - bandpass filtering; - band-stop filtering; - noise reduction by wavelets. The processing operations mentioned above clean the data to improve the performance of subsequent steps in the method. Removing a trend value corrects for any potential data drift. Bandpass filtering retains data with frequencies between 1 Hz and 100 Hz. Data below 1 Hz can be considered noise and do not contribute to recognizing a user's mental state. Similarly, data above 100 Hz can be ignored because, in practice, brain electrical activity data does not exceed 100 Hz. Preferably, bandpass filtering retains data with frequencies between 5 Hz and 80 Hz.Indeed, data with frequencies between 1 Hz and 5 Hz, as well as data with frequencies between 80 Hz and 100 Hz, contribute only marginally to recognizing a user's mental state. Band-stop filtering removes the influence of the power grid on the data and consists of a very narrow band-pass filter centered on the operating frequency of the power grid, for example, 50 Hz in Europe and 60 Hz in other regions. United States. Wavelet denoising makes it possible to eliminate periodic noises and artifacts with a characteristic shape, such as noises due to facial muscle activity or artifacts due to blinking.

[0046] Preferably, the processing window for step c) has a duration of at least 1 s, and preferably 10 s. This is because the duration of the processing window used for step c) of the electrical data is at least 1 s to ensure sufficient data is available for calculating a frequency representation of the electrical data within that time window. Furthermore, the duration of this time window is preferably 10 s to limit the amount of data to be processed and thus reduce the amount of computation required for processing that data.

[0047] Steps b) and c) described above are optional and can be performed in any order between steps a) and d), preferably in alphabetical order. Where applicable, the data on which a step in steps b), c), and d) is performed is the data from the immediately preceding step. Thus, the data in steps b), c), and d) are calculated directly or indirectly from data from step a).

[0048] According to one embodiment, a brain activity characteristic produced in step d) of extracting one or more activity characteristics from electrical data is calculated as the spectral power density over a frequency band of the electrical data included in the time window of step d). Indeed, the spectral power density calculated over a frequency band of a signal represents the signal's energy over that frequency band. Thus, the spectral power density calculated for a frequency band on brain electrical activity data can provide a good representation of the intensity of brain activity for the considered frequency band.

[0049] Preferably, the electrical data used to calculate a brain activity characteristic in step d) should originate from at least one specific region of the user's brain. This is because different functions can be associated with different brain regions, such as the frontal lobe for conscious thought, the central lobe for sensorimotor information, and the occipital lobe for vision. These different functions can play specific roles in different mental states, such as the central lobe for movement intentions. Therefore, it may be advantageous to select electrical activity data from specific brain regions depending on the mental state to be recognized. rather than considering electrical activity data from all brain regions.

[0050] Preferably, the frequency band considered in the calculation of a brain activity characteristic in step d) overlaps at least partially with at least one band from the following group: - alpha band (8-13 Hz); - beta band (13-30 Hz); - gamma band (30-100 Hz); - delta band (0.5-4 Hz); - theta band (4-8 Hz). Indeed, it is well known in the field of EEG measurements that the different frequency bands mentioned above can correspond to different mental states of a user. For example, alpha band activity may be significant for a user in a relaxed, conscious state, but during intense mental activity, beta band activity may be more pronounced. Therefore, it may be advantageous to select at least a portion of one or more frequency bands based on the mental state to be identified, rather than considering all frequencies of the brain's electrical activity data.

[0051] According to one embodiment, the process for generating a multidimensional characteristic image (MCI) in step f) includes a step of creating a feature matrix comprising at least one normalized feature. One advantage of constructing a feature matrix is ​​the ability to account for the spatial origin of the signals in data processing, i.e., the brain region from which the data originate.

[0052] Preferably, any missing data in the feature matrix are calculated by interpolation based on the data present in the matrix, which can allow these missing data to be filled in with low computational complexity. Indeed, it is possible that data for a location in the matrix may be missing if no brain electrical activity data has been measured for a region corresponding to that location, for example, if there is no measurement electrode for that region.

[0053] In one embodiment, the algorithm used in step g) to produce a prediction of the user's mental state includes an artificial neural network. An artificial neural network is an algorithm that can be trained to recognize certain patterns. or structures in data, said data being images, such as ICMs.

[0054] Preferably, the artificial neural network included in the algorithm used in step g) includes a convolutional neural network, which is particularly suited to processing data in image form, such as an ICM.

[0055] According to one embodiment, the method according to the invention further comprises: - a collection of one or more predictions of a user's mental state produced in step g); - a smoothing operation of the said prediction(s) of a user's mental state carried out via an arithmetic mean of the said prediction(s) included in a smoothing time window, said smoothing time window having a duration between 0.1 s and 20s, preferably between 1 s and 10s. Indeed, there may be an advantage in smoothing over time the predictions of a user's mental state produced in step g) of the method according to the invention in order to obtain a sequence of mental states that is more homogeneous along a temporal dimension than the sequence of mental states produced in step g).

[0056] Preferably, the smoothing operation also includes an extrapolation operation to compensate for a delay introduced by the smoothing time window. Indeed, when a smoothing time window is temporally centered on the value to be smoothed, it introduces a delay essentially equal to half the duration of that window. This delay can be compensated for, for example, by an extrapolation operation that consists of estimating a future smoothed value based on one or more past smoothed values.

[0057] According to a second aspect, the invention provides a computer program for implementing a method according to an embodiment of the invention. Such a computer program is a computer program comprising instructions which, when the program is executed by a computer system comprising one or more computers, cause the computer system to implement a method according to an embodiment of the invention.

[0058] According to a third aspect, the invention proposes a device for the automatic processing of electrical activity data from a user's brain and comprising: - a receiver to receive electrical data reflecting electrical activity of the user's brain and measured by electroencephalography; - data processing means arranged to produce a prediction of a user's mental state repeatedly, with a refresh rate of at least 1 Hz, preferably at least 3 Hz, preferably 10 Hz, based on the following processing steps: o within a time window, extraction of one or more brain activity features from the received electrical data; o within a time window, normalization of said brain activity feature(s) to produce one or more normalized features; o generation of one or more multidimensional characteristic images (MCIs) based on said normalized feature(s); o processing of said MCIs by an algorithm to produce a prediction of a user's mental state. The said automatic data processing device may be a computer system, which may comprise a plurality of computers and be programmed to perform the said processing steps. Furthermore, the said receiver may be a device comprising a processor and memory, and may be configured to receive electrical data from a data source, reflecting the electrical activity of the user's brain and measured by electroencephalography. Several scenarios are possible for acquiring this data, some of which are detailed below. In the first scenario, the data is acquired prior to its transmission to the receiver. In this first scenario, the data source may be a computer file, which may be contained in a storage device, such as memory.In a second case, the data is acquired continuously and transmitted to the receiver as it is acquired. In this second case, the data source can be a device for measuring brain electrical activity data by electroencephalography (EEG), which may include one or more EEG electrodes. Furthermore, if the receiver is located remotely from the data source, the data can be made available to the receiver via a computer network, such as a wired or wireless network. This computer network can be configured, for example, as a client-server system or as an ad hoc network. Finally, the data processing means can be one or more computer devices, each comprising a processor and memory, and each configured to implement one or more of the aforementioned processing steps.

[0059] The invention also proposes a device for measuring the electrical activity of a user's brain in order to predict the user's mental state, comprising: - a brain-computer interface device for measuring the electrical activity of the user's brain and comprising: o a frontal electroencephalography electrode for measuring the electrical activity of a frontal region of the brain; o a central electroencephalography electrode for measuring the electrical activity of a central region of the brain; o an occipital electroencephalography electrode for measuring the electrical activity of an occipital region of the brain; and - an automatic data processing device for electrical activity of the user's brain according to the invention; the brain-computer interface device being coupled with the automatic data processing device in such a way that the data measured by the brain-computer interface device can be received by the automatic data processing device, the brain-computer interface device thus constituting a data source as described above.

[0060] All possible embodiments and all advantages of the method according to the invention apply mutatis mutandis to the other aspects of the invention. Brief description of the figures

[0061] Other features and advantages of the present invention will become apparent upon reading the detailed description that follows, and for understanding which reference should be made to the accompanying figures, among which: - Figure 1 illustrates the "10-20 system" for placing electroencephalography electrodes on a user's skull in relation to the user's brain regions; - Figure 2 is a functional diagram of an automatic data processing device for electrical activity of a user's brain according to one embodiment of the invention; - Figure 3 illustrates a sigmoid function that can be used during a normalization step of the method according to the invention; - Figure 4 schematically illustrates a step in generating a multidimensional characteristic image according to one embodiment of the invention; - Figure 5 is a functional diagram of a method for training an artificial neural network that can be used to produce a prediction of a user's mental state according to one embodiment of the invention.

[0062] The drawings in the figures are not to scale. Generally, similar features are denoted by similar reference numerals in the figures. Within the scope of this document, identical or analogous features may bear the same reference numerals. Furthermore, the presence of reference numerals or letters in the drawings shall not be considered limiting, even when such numerals or letters are specified in the claims. Detailed description of certain embodiments of the invention

[0063] The present invention is described with particular embodiments and references to figures, but the invention is not limited by them. The drawings or figures described are schematic only and are not limiting. Furthermore, the functions described can be performed by structures other than those described in this document.

[0064] The use of the verb "comprendre" (to understand), its variants, and its conjugations in this document does not in any way preclude the presence of elements other than those mentioned. The use of the indefinite article "un" (a / an) or the definite article "le" (the / it) to introduce an element does not preclude the presence of multiple such elements.

[0065] In the context of this document, the terms "first", "second", "third", etc. are used only to differentiate different elements and do not imply any order between these elements.

[0066] Furthermore, embodiments described as "preferred" should be interpreted as examples of implementation of the invention rather than as limiting the scope of the invention.

[0067] Figure 1 illustrates the "10-20 system" for electroencephalography (EEG) electrode placement, also known as the "international 10-20 system." This system is described in the book "Webster J., Nimunkar A., ​​Clark J., Medical Instrumentation. 4th ed. 2010." The 10-20 system is an internationally recognized method that uses anatomical landmarks to standardize EEG electrode placement. This system is based on the relationship between electrode placement and the underlying area of ​​the cerebral cortex, ensuring that all brain regions are covered.

[0068] The numbers "10" and "20" refer to the distances between adjacent electrodes, which represent either 10% or 20% of the total distance (front-to-back or right-to-left) across the skull. The total distance is based on anatomical locations on the scalp: the nasion 101 and the inion 102 (front-to-back direction) and the two preauricular points 103 and 104 (right-to-left direction), as shown in Figure 1. Using these anatomical landmarks, the electrode placement can be determined according to these directions, in the proportions specified beforehand: 10% is used between the anatomical landmarks and the first electrode in that direction, and 20% is used between the other electrodes. For example, electrode Fp1 is placed at 10% of the total distance from the nasion, and electrode Fz is placed at 20% of the total distance from electrode Fp1.

[0069] In the 10-20 system, each electrode implantation site has a letter that identifies the lobe or brain region the electrode covers: prefrontal (Fp), frontal (F), temporal (T), parietal (P), occipital (O), and central (C). In practice, there is no "central lobe." The central (C) region of the brain refers to a brain region near the boundary between the motor and somatosensory cortex. Additionally, the letter A is sometimes used to designate the earlobes. Generally, these locations are included as a reference for signal analysis. Finally, there are also "z" sites: such a site designates an electrode placed on the midline of the sagittal plane of the skull, for example, Fz, Cz, Pz. The electrode sites located on the right side of the skull have even numbers (2, 4, 6, 8), while the electrode sites located on the left side of the skull have odd numbers (1, 3, 5, 7).

[0070] Figure 2 is a functional diagram of a device 200 for automatic processing of electrical activity data from a user's brain according to one embodiment and comprising the following elements: an analog electrical data receiver 210, an analog-to-digital converter 220, a data processing device 230, a device 240 for extracting activity features from the user's brain, a device 250 for normalizing activity features, a generator 260 of multidimensional characteristic images (ICM), and an ICM processing device 270.

[0071] The automatic data processing device 200 for the user's brain electrical activity operates as follows. First, the receiver 210 receives analog electrical data reflecting the user's brain electrical activity, measured by electroencephalography. The receiver 210 then transmits this analog electrical data to the analog-to-digital converter 220, as shown by arrow 211. The analog-to-digital converter 220 converts the analog electrical data 211 into digital electrical data by means of at least one conversion operation from the following group: - amplification; - quantification; - sampling. The analog-to-digital converter 220 then transmits this digital electrical data to the data processing device 230, as represented by arrow 221. The data processing device 230 produces processed digital electrical data from the digital electrical data 221 by means of at least one processing operation from the following group: - removal of a trend value; - bandpass filtering; - band-stop filtering; - noise reduction by wavelets. The data processing device 230 then transmits this processed digital electrical data to the user brain activity feature extraction device 240, as shown by arrow 231. Device 240 extracts user brain activity features from the processed digital electrical data 231. Device 240 then transmits these user brain activity features to the activity feature normalization device 250, as shown by arrow 241. Device 250 normalizes the activity features 241 to produce one or more normalized features. Device 250 then transmits these normalized features to the ICM generator 260, as shown by arrow 251. The ICM generator 260 generates one or more ICMs from one or more normalized features 251.The ICM generator 260 then transmits the said ICM(s) to the ICM processing device 270, as represented by arrow 261. The ICM processing device 270 then produces a prediction 271 of a mental state of the user by means of an algorithm based on one or more ICMs 261. A mental state of the user is thus predicted by the device 200 repeatedly, with a refresh rate of at least 1 Hz, preferably at least 3 Hz, preferably 10 Hz, giving rise to a sequence of predicted mental states.

[0072] In the case where a user's mental state is an emotion, it could be an emotion such as anger, joy, surprise, disgust, fear, or sadness. These emotions, as well as others, can be represented in a three-dimensional space, according to the VAD model (for "Valence-Arousal- Dominance”) proposed by Mehrabian and Russell in “Russell JA, Mehrabian A., Evidence for a three-factor theory of emotions, J. Res. Personal, 1977, 11, 273-294”. This model proposes to represent emotions according to 3 independent dimensions: pleasure (valence), which ranges from unhappiness to happiness and expresses the pleasant or unpleasant feeling one experiences towards something; arousal, i.e. the level of affective activation, which ranges from sleepiness to excitement; and dominance, which reflects the level of control over the emotional state, which ranges from submission to dominance.

[0073] A user's mental state can also be a movement intention. A movement intention can include information about the direction of the intended movement, such as left or right. A movement intention can also include information about the origin of the intended movement, such as an upper limb (e.g., an arm) or a lower limb (e.g., a leg).

[0074] As explained above, the use of a sigmoid function, also called a "logistic function", during the normalization step of brain activity characteristics contained within an interval makes it possible to carry out said normalization with little sensitivity to the extreme values ​​of said characteristics that may be present in the interval, and also without having to know in advance all the values ​​of said characteristics in the interval, in particular the minimum and maximum values ​​of said characteristics.

[0075] Figure 3 illustrates a sigmoid function 300 represented in a coordinate system comprising a first axis 301 on which the domain of the function is defined, and a second axis 302 which contains the function's image values. The sigmoid function 300 satisfies the following properties: the function's image values ​​are bounded by two horizontal asymptotes 304, 305; the slope at the origin 306 of the function has a finite value; and the function has a single inflection point 307. The first axis 301 can also be called the "x-axis". The second axis 302 can also be called the "y-axis".

[0076] A special case of a sigmoid function is a "hyperbolic tangent" function, also denoted "tanh", which can be calculated using the following expression: tanh In this particular case, and denoting axes 301 and 302 respectively by "x" and "y", the asymptotes 304 and 305 are respectively of equations y = +1 and y = -1, the slope at the origin 306 is equal to 1, and the inflection point 307 is located at the origin, i.e. at coordinates (x, y) = (0, 0).

[0077] A hyperbolic tangent function can be parameterized by a mean p and a standard deviation o of values ​​s contained within an interval. In such a case, the hyperbolic tangent function can be calculated using the following expression: tanh(s, [i, o') = tanh where A is a scaling factor (or normalization constant) which can take, for example, the value 0.01. The function tanh(s,p,o) can be used to calculate normalized values ​​s' from values ​​s contained within the interval. Furthermore, it is possible to bring the normalized values, i.e., the values ​​represented by the function tanh(s,p,o), into the interval [0,1] using a linear transformation according to the following expression: 1 s' = - (tanh (s, fi, o) + 1).

[0078] Figure 4 illustrates the creation of a multidimensional characteristic image (MCI) from signals measured by electroencephalography (EEG) electrodes. Figure 4a reproduces the representation from Figure 1 illustrating the placement of EEG electrodes on a user's skull in relation to the user's brain regions covered by the EEG electrodes. In addition, Figure 4a highlights some EEG electrode sites 401 to 413. Figure 4b schematically represents a 420 matrix of features, including at least one normalized feature. Such a 420 matrix allows for consideration of the spatial origin of the normalized features, i.e., the EEG electrode sites and the associated user brain regions from which the normalized features originate. Thus, entries 421 to 433 of the 420 matrix correspond respectively to EEG electrode sites 401 to 413.Furthermore, it is possible to calculate the value of missing data in matrix 420 by interpolation based on the data present in the matrix. For example, the missing entry 434 in matrix 420 can be calculated as the average, possibly weighted, of the adjacent entries 421, 422, 430, 432, and 433.

[0079] The 420 matrix, in which missing values ​​are optionally calculated by interpolation, can serve as the basis for creating a CMI (Computer-Mixed Image) by applying the 420 matrix's inputs to certain pixels of an image. This yields an image whose pixel values ​​correspond to the amplitude of normalized features, taking into account the spatial origin of those normalized features. Furthermore, the pixel values ​​of an image lacking a corresponding entry in the 420 matrix can be calculated by interpolation from the image pixels to which the 420 matrix's inputs have been applied.

[0080] Figure 5 is a functional diagram of a method for training an artificial neural network that can be used to produce a prediction of a user's mental state according to one embodiment and comprising the following steps: - presentation 501 of a stimulus to a user 502 to elicit a mental state in the user 502, such as an emotion or an intention to move, a label 509 being associated with the stimulus and allowing to describe the mental state which the stimulus is likely to elicit in the user 502; - acquisition 503 of EEG signals in user 502 in response to stimulus presentation 501; - 504 processing of acquired EEG signals, the 504 processing being able to include one or more conversion operations as well as one or more processing operations as described above; - extraction 505 of user brain activity characteristics from processed EEG signals; - standardization 506 of activity characteristics; - generation 507 of one or more ICM 508 from the standardized characteristics; - creation 510 of a training dataset and a test dataset for training the artificial neural network, said datasets being created on the basis of ICM 508 and wording 509; - 511 training of the artificial neural network based on the created datasets.

[0081] The stimulus presented to the user in step 501 can be any type of stimulus capable of inducing electrical activity in the user's brain that can be measured by EEG to predict a mental state of the user. Examples include an audiovisual stimulus in the case of a mental state corresponding to an emotion, a movement of a part of the user's body in the case of a mental state corresponding to an intention to move, or any other sensory stimulus, such as a taste, smell, tactile stimulus, etc. The stimulus presented to the user in step 501 can also include a voluntary action by the user, such as voluntarily entering a given emotional state, like becoming angry, in the case of a mental state corresponding to an emotion, or voluntarily moving a part of their body in the case of a mental state corresponding to an intention to move.

[0082] The training dataset created in step 510 is intended to train the artificial neural network to recognize a given mental state, identified by a label 509, to Starting from one or more ICM 508s, the test dataset created in step 510 aims to measure the quality of the artificial neural network's training. Typically, several stimuli of different natures are presented to the user 502 to elicit a large number of different mental states. Furthermore, the same stimulus can be presented to the user 502 multiple times to acquire a large number of EEG signals corresponding to a mental state elicited by that stimulus. This allows for the creation of sufficiently comprehensive training and test datasets to enable effective training of the artificial neural network.

[0083] In summary, the invention relates to a method for the automatic processing of electrical activity data from a user's brain and comprising the following steps: receiving electrical data reflecting the electrical activity of the user's brain and measured by electroencephalography; within a time window, extracting one or more brain activity features from the electrical data; within a time window, normalizing said activity feature(s) to produce one or more normalized features; generating one or more multidimensional characteristic images based on said normalized feature(s); processing said multidimensional characteristic image(s) by an algorithm to produce a prediction of the user's mental state;where said steps are repeated with a refresh rate of at least 1 Hz, preferably at least 3 Hz, preferably 10 Hz.;

[0084] The present invention has been described in this document with regard to specific embodiments, which are purely illustrative and should not be considered limiting. Generally, it will be evident to a person skilled in the art that the present invention is not limited to the embodiments and examples illustrated and / or described in this document, but that the scope of the invention is more broadly defined by the claims introduced below.

Claims

Demands 1. A method for the automatic processing of electrical activity data from a user's brain, comprising the following steps: a) receiving electrical data reflecting the electrical activity of the user's brain and measured by electroencephalography; d) within a time window, extracting one or more brain activity features from the electrical data of the preceding step; e) within a time window, normalizing said activity feature(s) from step d) to produce one or more normalized features; f) generating one or more multidimensional feature images based on said normalized feature(s) from step e); g) processing said multidimensional feature image(s) from step f) by an algorithm to produce a prediction of the user's mental state;where steps a) to g) are repeated with a refresh rate of at least 1 Hz, preferably at least 3 Hz, preferably 10 Hz.

2. Method according to the preceding claim, wherein step e) comprises applying a normalization function to said activity characteristic(s) of step d) to produce one or more normalized characteristics, and wherein the normalization function is a non-linear function.

3. Method according to any one of the preceding claims, wherein step e) comprises applying a normalization function to said activity characteristic(s) of step d) to produce one or more normalized characteristics, and wherein the normalization function is a non-linear monotonic function whose slope decreases for activity characteristics whose absolute value increases.

4. A method according to any one of the preceding claims, wherein step e) comprises applying a normalization function to said activity characteristic(s) of step d) to produce one or more normalized characteristics, and wherein the normalization function is a sigmoid function.

5. Method according to any one of the preceding claims, wherein step e) comprises applying a normalization function to said activity characteristic(s) of step d) to produce one or more normalized characteristics, and wherein the normalization function is a hyperbolic tangent.

6. Method according to any one of claims 2 to 5, wherein the normalization function of step e) is parameterized by statistical parameters calculated on the basis of the brain activity characteristics included in the time window of step e).

7. A method according to the preceding claim, wherein the normalization function of step e) is a hyperbolic tangent parameterized by the mean p and standard deviation o of the brain activity characteristics included in the time window of step e) according to the following expression: tanh(s, [i, o') = tanh where s denotes the values ​​on which the normalization function is applied and where A is a scaling factor.

8. Method according to any one of claims 2 to 7, wherein step e) further comprises a linear application of the image values ​​of the normalization function to a bounded interval, preferably the interval [0,1].

9. Method according to any one of the preceding claims, wherein the duration of the time window of step d) of extraction of one or more brain activity features is between 1 s and 10s, preferably between 1 s and 5s, preferably between 1 s and 3s, preferably equal to 2s.

10. Method according to any one of the preceding claims, wherein the duration of the time window of step e) of normalization of one or more brain activity characteristics is at most equal to 10s, preferably at most equal to 5s, preferably equal to 2s.

11. A method according to any one of the preceding claims, further comprising, between steps a) and d), a step b) of converting electrical data from step a) from an analog form into a digital form, said conversion step b) comprising at least one conversion operation from the following group: - amplification; - quantification; - sampling.

12. A method according to any one of the preceding claims, further comprising, between steps a) and d), a step c) of processing electrical data from step a) contained within a time window, said processing step c) comprising at least one processing operation from the following group: - removal of a trend value; - bandpass filtering; - band-stop filtering; - noise reduction by wavelets.

13. Method according to the preceding claim, wherein the time window of the processing step c) has a duration of at least 1 s, preferably 10s.

14. Method according to any one of the preceding claims, wherein a brain activity feature produced in step d) of extracting one or more activity features from electrical data is calculated as the spectral power density over a frequency band of the electrical data included in the time window of step d).

15. Method according to the preceding claim, wherein the electrical data used for calculating a brain activity characteristic in step d) are derived from at least one specific region of the user's brain.

16. A method according to any one of the two preceding claims, wherein the frequency band considered in the calculation of a brain activity characteristic in step d) at least partially overlaps with at least one band from the following group: - alpha band (8-13 Hz); - beta band (13-30 Hz); - gamma band (30-100 Hz); - delta band (0.5-4 Hz); - theta band (4-8 Hz).

17. Method according to any one of the preceding claims, wherein the process of generating a multidimensional characteristic image in step f) includes a step of creating a feature matrix comprising at least one normalized feature.

18. Method according to the preceding claim, wherein any missing data in the feature matrix are calculated by interpolation on the basis of the data present in the matrix.

19. A method according to any one of the preceding claims, wherein the algorithm used in step g) to produce a prediction of a user's mental state comprises an artificial neural network.

20. Method according to the preceding claim, wherein the artificial neural network included in the algorithm used in step g) comprises a convolutional neural network.

21. A method according to any one of the preceding claims, and further comprising: - a collection of one or more predictions of a user's mental state produced in step g); - a smoothing operation of the said prediction(s) of a user's mental state carried out via an arithmetic mean of the said prediction(s) included in a smoothing time window, said smoothing time window having a duration between 0.1 s and 20s, preferably between 1 s and 10s.

22. Method according to the preceding claim, wherein the smoothing operation further comprises an extrapolation operation aimed at compensating for a delay introduced by the smoothing time window.

23. Computer program for implementing a method according to any one of the preceding claims.

24. Device for the automatic processing of electrical activity data from a user's brain, comprising: - a receiver to receive electrical data reflecting electrical activity of the user's brain and measured by electroencephalography; - data processing means arranged to produce a prediction of a user's mental state repeatedly, with a refresh rate of at least 1 Hz, preferably at least 3 Hz, preferably 10 Hz, based on the following processing steps: o within a time window, extraction of one or more brain activity features from the received electrical data; o within a time window, normalization of said brain activity feature(s) to produce one or more normalized features; o generation of one or more multidimensional feature images based on said normalized feature(s); o processing of said multidimensional feature image(s) by an algorithm to produce a prediction of a user's mental state.

25. Device for measuring the electrical activity of a user's brain for the purpose of predicting the user's mental state and comprising: - a brain-computer interface device for measuring electrical activity in the user's brain and comprising: o a frontal electroencephalography electrode for measuring electrical activity in a frontal region of the brain; o a central electroencephalography electrode for measuring electrical activity in a central region of the brain; o an occipital electroencephalography electrode to measure electrical activity in an occipital region of the brain; and - an automatic data processing device for electrical activity of the user's brain according to the preceding claim; the brain-computer interface device being coupled with the automatic data processing device in such a way that the data measured by the brain-computer interface device can be received by the automatic data processing device.

Images