User intention recognition method, device and equipment based on multi-modal brain-computer interface

Abstract

The invention discloses a user intention recognition method, device and equipment based on a multi-mode brain-computer interface, and relates to the technical field of artificial intelligence. The user intention recognition method based on the multi-mode brain-computer interface comprises the following steps: acquiring an electroencephalogram signal and a functional near infrared spectrum signal of a user; the intention of the user is obtained through a current recognition model according to the electroencephalogram signals and the functional near infrared spectrum signals, and an initial recognition model is obtained by training historical electroencephalogram signals and historical functional near infrared spectrum signals of the user; receiving feedback information input by the user for the intention; and performing parameter adjustment on the current identification model according to the feedback information for next intention identification. According to the method, the intention recognition accuracy can be improved, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and equipment for identifying user intention based on a multimodal brain-computer interface. Background Art

[0002] In the related technology, when the brain-computer interface (BCI) system performs intent recognition, it often relies on static models and algorithms, which makes the BCI system unable to maintain efficient user interaction and accurate intent recognition during long-term use, resulting in a poor user experience. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the purpose of the present invention is to propose a method, device and equipment for user intention recognition based on a multimodal brain-computer interface to improve the accuracy of intention recognition and enhance user experience.

[0004] In the first aspect, an embodiment of the present invention proposes a method for user intention recognition based on a multimodal brain-computer interface, including: obtaining the user's EEG signals and functional near-infrared spectral signals; using a current recognition model to obtain the user's intention based on the EEG signals and the functional near-infrared spectral signals, wherein the initial recognition model is trained on the user's historical EEG signals and historical functional near-infrared spectral signals; receiving feedback information input by the user regarding the intention; and adjusting parameters of the current recognition model based on the feedback information for the next intention recognition.

[0005] According to one embodiment of the present invention, the recognition model includes a processing module and a recognition module; the use of the current recognition model to obtain the user's intention based on the EEG signal and the functional near-infrared spectral signal includes: using the current processing module to process the EEG signal and the functional near-infrared spectral signal to obtain target input parameters; using the current recognition module to obtain the user's intention based on the target input parameters.

[0006] According to one embodiment of the present invention, the processing of the EEG signal and the functional near-infrared spectral signal to obtain target input parameters includes: performing causal analysis on the EEG signal and the functional near-infrared spectral signal to obtain causal relationship analysis results; and / or performing feature extraction on the EEG signal and the functional near-infrared spectral signal respectively to obtain EEG features and functional near-infrared spectral features.

[0007] According to one embodiment of the present invention, before processing the EEG signal and the functional near-infrared spectral signal, the method further includes: performing high-pass filtering, low-pass filtering and band-pass filtering on the EEG signal, and performing low-pass filtering on the functional near-infrared spectral signal; and / or, performing baseline correction on the EEG signal and the functional near-infrared spectral signal, respectively.

[0008] According to one embodiment of the present invention, the baseline correction of the EEG signal and the functional near-infrared spectral signal respectively includes: calculating the mean of the EEG signal and the functional near-infrared spectral signal respectively; subtracting the EEG signal from the mean of the EEG signal, and subtracting the functional near-infrared spectral signal from the mean of the functional near-infrared spectral signal.

[0009] According to one embodiment of the present invention, the causal analysis of the EEG signal and the functional near-infrared spectral signal to obtain the causal relationship analysis result includes: synchronously processing the EEG signal and the functional near-infrared spectral signal; establishing a first univariate autoregression model for the synchronously processed EEG signal, establishing a second univariate autoregression model for the synchronously processed functional near-infrared spectral signal, establishing a first bivariate autoregression model considering the influence of the synchronously processed functional near-infrared spectral signal on the synchronously processed EEG signal, and establishing a second bivariate autoregression model considering the influence of the synchronously processed EEG signal on the synchronously processed functional near-infrared spectral signal; calculating a first prediction error sum of squares according to the first univariate autoregression model, calculating a second prediction error sum of squares according to the second univariate autoregression model, calculating a third prediction error sum of squares according to the first bivariate autoregression model, and calculating a fourth prediction error sum of squares according to the second bivariate autoregression model; calculating a first F statistic according to the first prediction error sum of squares and the third prediction error sum of squares, and calculating a second F statistic according to the second prediction error sum of squares and the fourth prediction error sum of squares; obtaining the causal relationship analysis result according to the first F statistic and the second F statistic.

[0010] According to one embodiment of the present invention, the first F statistic and the second F statistic are calculated by the following formula:

[0011]

[0012] Among them, F Y→X is the first F statistic, F X→Yis the second F statistic, SSE(X) is the first prediction error sum of squares, SSE(X|Y) is the third prediction error sum of squares, SSE(Y) is the second prediction error sum of squares, SSE(Y|X) is the fourth prediction error sum of squares, n is the number of samples of the EEG signal or the functional near-infrared spectroscopy signal, p is the autoregressive model order of the EEG signal, and q is the autoregressive model order of the functional near-infrared spectroscopy signal.

[0013] According to one embodiment of the present invention, obtaining the causal relationship analysis result based on the first F statistic and the second F statistic includes: obtaining a first p-value based on the first F statistic using an F distribution table, and obtaining a second p-value based on the second F statistic using the F distribution table; obtaining the causal relationship analysis result based on the first p-value and the second p-value.

[0014] According to an embodiment of the present invention, the method further comprises: generating a control instruction according to the user's intention; wherein the feedback information is also generated for the control instruction.

[0015] In the second aspect, an embodiment of the present invention proposes a user intention recognition device based on a multimodal brain-computer interface, comprising: an acquisition unit, used to acquire the user's EEG signals and functional near-infrared spectral signals; an identification unit, used to obtain the user's intention based on the EEG signals and the functional near-infrared spectral signals using a current recognition model, wherein the initial recognition model is obtained by training the user's historical EEG signals and historical functional near-infrared spectral signals; a receiving unit, used to receive feedback information input by the user regarding the intention; and an adjustment unit, used to adjust parameters of the current recognition model based on the feedback information for the next intention recognition.

[0016] In a third aspect, an embodiment of the present invention proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the user intention recognition method based on a multimodal brain-computer interface described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present invention proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the user intention recognition method based on a multimodal brain-computer interface described in the first aspect.

[0018] The user intention recognition method, device and equipment based on the multimodal brain-computer interface of the embodiment of the present invention first obtains the user's EEG signal and functional near-infrared spectral signal; then uses the current recognition model to obtain the user's intention based on the EEG signal and the functional near-infrared spectral signal, and the initial recognition model is trained by the user's historical EEG signal and historical functional near-infrared spectral signal; then receives the user's feedback information on the intention input; finally, according to the feedback information, the parameters of the current recognition model are adjusted for the next intention recognition. In this way, the parameters of the recognition model can be dynamically adjusted according to the user's real-time EEG signal, functional near-infrared spectral signal and intention recognition feedback information, thereby optimizing the user experience and system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a method for identifying user intention based on a multimodal brain-computer interface according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a method structure for realizing a user intention recognition method based on a multimodal brain-computer interface according to an embodiment of the present invention;

[0021] Figure 3 is a structural block diagram of a user intention recognition device based on a multimodal brain-computer interface according to an embodiment of the present invention;

[0022] Figure 4 is a structural block diagram of a user intention recognition device based on a multimodal brain-computer interface according to another embodiment of the present invention;

[0023] Figure 5 It is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0025] The following describes a method, apparatus and device for identifying user intention based on a multimodal brain-computer interface according to an embodiment of the present invention with reference to the accompanying drawings.

[0026] Figure 1 It is a flowchart of a method for identifying user intention based on a multimodal brain-computer interface according to an embodiment of the present invention.

[0027] like Figure 1 As shown, the user intention recognition method based on multimodal brain-computer interface includes the following steps:

[0028] S11, obtaining the user's EEG signal and functional near-infrared spectroscopy signal.

[0029] Among them, EEG signals can be collected through EEG electrodes on head-mounted devices, which can reflect EEG activity; functional near-infrared spectroscopy signals (fNIRS) can be collected through near-infrared detectors, which can reflect changes in cerebral hemodynamics.

[0030] As an implementation method, a near-infrared emitter may be provided on the head-mounted device to emit near-infrared light. Since HbO2 (oxygenated hemoglobin) and Hb (hemoglobin) in brain tissue have different absorption spectra for near-infrared light, when near-infrared light enters the brain, it is absorbed and attenuated by HbO2 and Hb in the brain tissue before being emitted and received by the near-infrared detector, thereby acquiring a functional near-infrared spectrum signal. Optionally, the near-infrared emitter may also be provided independently of the head-mounted device.

[0031] Taking the BCI system for controlling a robotic arm as an example, the head-mounted device is equipped with EEG electrodes and near-infrared transmitters. Both the EEG electrodes and near-infrared detectors are connected to the host computer for communication, and the host computer is connected to the robotic arm for communication. When controlling the robotic arm, the user puts on the head-mounted device and generates the intention to control the robotic arm, and the near-infrared transmitter simultaneously emits near-infrared light. The EEG electrodes can transmit the collected EEG signals to the host computer, and the near-infrared detector can transmit the collected functional near-infrared spectral signals to the host computer, which then performs subsequent intention recognition based on the EEG signals and the functional near-infrared spectral signals.

[0032] S12, using the current recognition model to obtain the user's intention according to the EEG signal and the functional near-infrared spectroscopy signal, wherein the initial recognition model is trained based on the user's historical EEG signal and historical functional near-infrared spectroscopy signal.

[0033] In an embodiment of the present invention, the recognition model may include a processing module and a recognition module. The current processing module may be used to process the EEG signal and the functional near infrared spectroscopy signal to obtain the target input parameter; and then the current recognition module (which may use a neural network, such as a convolutional neural network, a recurrent neural network, an adversarial network, etc.) may be used to obtain the user's intention according to the target input parameter.

[0034] In some examples, the target input parameter includes a fusion processing parameter. Processing the EEG signal and the functional near infrared spectroscopy signal includes: fusing the EEG signal and the functional near infrared spectroscopy signal, such as splicing processing, weighting processing, etc.

[0035] In some examples, the target input parameter includes a causal relationship analysis result between the EEG signal and the functional near infrared spectroscopy signal. The causal relationship analysis result can be used to determine the dynamic influence between the EEG signal and the functional near infrared spectroscopy signal, and provide timing information for subsequent intention recognition.

[0036] In this example, the EEG signal and the functional near-infrared spectroscopy signal are processed to obtain target input parameters, including: performing causal analysis on the EEG signal and the functional near-infrared spectroscopy signal to obtain causal relationship analysis results.

[0037] Specifically, performing causal analysis on the EEG signal and the functional near infrared spectroscopy signal to obtain the causal relationship analysis result includes the following steps:

[0038] A1, synchronous processing of EEG signals and functional near-infrared spectroscopy signals.

[0039] The synchronization may include time and space synchronization. By synchronously processing the EEG signal and the functional near-infrared spectroscopy signal, the effective fusion of the EEG signal and the functional near-infrared spectroscopy signal can be ensured.

[0040] As an example, clock crystal oscillator technology can be used to synchronize EEG signals and functional near-infrared spectroscopy signals. Specifically, the EEG signal clock can be used as the master clock, and the functional near-infrared spectroscopy signal clock can be used as the slave clock. After receiving the EEG signal, the host computer uses the master clock as the reference to perform delay compensation on the slave clock to achieve synchronization of the EEG signal and the functional near-infrared spectroscopy signal.

[0041] Optionally, a timestamp calibration method, a spatial alignment algorithm, etc. may be used to synchronously process the EEG signal and the functional near-infrared spectroscopy signal.

[0042] A2, establish a first univariate autoregressive model for the synchronized EEG signal, establish a second univariate autoregressive model for the synchronized functional near-infrared spectral signal, establish a first bivariate autoregressive model considering the influence of the synchronized functional near-infrared spectral signal on the synchronized EEG signal, and establish a second bivariate autoregressive model considering the influence of the synchronized EEG signal on the synchronized functional near-infrared spectral signal.

[0043] The first univariate autoregressive model and the second univariate autoregressive model can be expressed by the following equations (1) and (2), respectively:

[0044]

[0045] Among them, X(t) and Y(t) are the predicted values ​​of EEG signal and functional near-infrared spectroscopy signal at time t, respectively. Xi ,φ Yiare the model parameters of the first univariate autoregressive model and the second univariate autoregressive model of the i-th order, p and q are the model orders of the first univariate autoregressive model and the second univariate autoregressive model, ε X (t), ε Y (t) are the prediction errors of the first univariate autoregressive model and the second univariate autoregressive model at time t respectively.

[0046] The first bivariate autoregressive model and the second bivariate autoregressive model can be expressed by the following equations (3) and (4):

[0047]

[0048] Among them, X(t) and Y(t) are the predicted values ​​of EEG signal and functional near-infrared spectroscopy signal at time t, respectively. Xi ,φ Yi ,θ Xi ,θ Yi are model parameters, p and q are model orders, ε X (t), ε Y (t) are the prediction errors at time t. It should be noted that φ in equations (3) and (4) Xi ,φ Yi , p, q, ε X (t), ε Y (t) and φ in equations (1) and (2) Xi ,φ Yi , p, q, ε X (t), ε Y (t) can take different values.

[0049] A3, calculate the first prediction error sum of squares according to the first univariate autoregression model, calculate the second prediction error sum of squares according to the second univariate autoregression model, calculate the third prediction error sum of squares according to the first bivariate autoregression model, and calculate the fourth prediction error sum of squares according to the second bivariate autoregression model.

[0050] Among them, the sum of squares of the prediction errors can be used to evaluate the goodness of fit of the model. It can be calculated based on the predicted value and observed value (i.e., actual value) of the signal. Specifically, it can be the accumulated value of the square of the difference between the predicted value and the observed value of the signal at multiple moments.

[0051] A4, calculating a first F statistic according to the first prediction error sum of squares and the third prediction error sum of squares, and calculating a second F statistic according to the second prediction error sum of squares and the fourth prediction error sum of squares.

[0052] In some examples, the first F-statistic and the second F-statistic can be calculated by the following formula:

[0053]

[0054] Among them, F Y→X is the first F statistic, F X→Y is the second F statistic, SSE(X) is the first prediction error sum of squares, SSE(X|Y) is the third prediction error sum of squares, SSE(Y) is the second prediction error sum of squares, SSE(Y|X) is the fourth prediction error sum of squares, n is the number of samples of EEG signals or functional near-infrared spectroscopy signals, p is the order of the autoregressive model of EEG signals, and q is the order of the autoregressive model of functional near-infrared spectroscopy signals.

[0055] A5, obtain the causal relationship analysis results based on the first F statistic and the second F statistic.

[0056] As an implementation mode, obtaining a causal relationship analysis result based on a first F statistic and a second F statistic includes: obtaining a first p-value based on the first F statistic using an F distribution table, and obtaining a second p-value based on the second F statistic using an F distribution table; obtaining a causal relationship analysis result based on the first p-value and the second p-value.

[0057] Through the significance test of the F statistic, it can be determined whether there is a causal relationship between the EEG signal and the functional near-infrared spectroscopy signal. In implementation, the F distribution table can be used to determine the p value for the F statistic, and then the p value can be used to determine whether there is a causal relationship between the EEG signal and the functional near-infrared spectroscopy signal.

[0058] Specifically, the F statistic reflects the ratio of the variance that the model can explain to the variance of the random error; the p-value is a key indicator of the F test result, which indicates the probability of observing the current result or a more extreme result when the null hypothesis (i.e., all independent variables have no effect on the dependent variable) is established. If the p-value is less than the significance level (such as 0.05), the null hypothesis is rejected and the model is considered statistically significant, that is, at least one independent variable has a significant effect on the dependent variable (i.e., there is a causal relationship, which can be represented by 1). If the p-value is greater than the significance level, the null hypothesis cannot be rejected and the model is considered statistically insignificant, that is, there is insufficient evidence to show that the independent variable has a significant effect on the dependent variable (i.e., there is no causal relationship, which can be represented by 0). The causal relationship analysis results obtained based on the first p-value and the second p-value can be represented by a 0-1 pair. For example, (1,1) indicates that the EEG signal and the functional near-infrared spectroscopy signal have a causal relationship with each other, and (1,0) indicates that the functional near-infrared spectroscopy signal has a causal relationship with the EEG signal, and the EEG signal has no causal relationship with the functional near-infrared spectroscopy signal. By using the results of causal relationship analysis as the input of the recognition module, the complexity of recognition such as neural network in the recognition module can be reduced and the recognition speed can be improved.

[0059] In some examples, the target input parameters include EEG features and functional near-infrared spectroscopy features.

[0060] In this example, the EEG signal and the functional near-infrared spectroscopy signal are processed to obtain target input parameters, including: extracting features from the EEG signal and the functional near-infrared spectroscopy signal to obtain EEG features and functional near-infrared spectroscopy features.

[0061] Among them, EEG features and functional near-infrared spectral features may include time, frequency and spatial dimensions, such as power spectral density, waveform features, changes in blood oxygen concentration, etc. By using EEG features and functional near-infrared spectral features as inputs to the recognition module, the diversity of intent recognition can be increased.

[0062] In some examples, the target input parameters include causal relationship analysis results between the EEG signal and the functional near-infrared spectroscopy signal, as well as EEG features and functional near-infrared spectroscopy features.

[0063] In this example, the EEG signal and the functional near infrared spectroscopy signal are processed to obtain the target input parameters, including: performing causal analysis on the EEG signal and the functional near infrared spectroscopy signal to obtain the causal relationship analysis result; and performing feature extraction on the EEG signal and the functional near infrared spectroscopy signal to obtain EEG features and functional near infrared spectroscopy features. Compared with the above example of using the causal relationship analysis result or feature alone as the input of the recognition module, this example considers both the causal relationship and the feature, which can make the recognition module more accurate.

[0064] S13, receiving feedback information from the user regarding the intended input.

[0065] Specifically, after the intent is identified, the intent can be provided to the user, such as in the form of display, broadcast, tactile signal, etc., to enhance the user experience. After the user gets the intent, he can give feedback on whether the identification is accurate, that is, the feedback information may include feedback on whether the identified intent is correct, such as accurate, inaccurate, 70% accuracy, etc. The feedback information can be in the form of text, voice, EEG signal, functional near infrared spectroscopy signal, eye movement signal, etc., and can be input by the user in the form of manual, voice, brain activity, eye movement, etc.

[0066] S14, adjusting the parameters of the current recognition model according to the feedback information for the next intention recognition.

[0067] In some embodiments, the feedback information and the EEG signals, functional near-infrared spectroscopy signals used for the current recognition, and the intention currently recognized can be used as new samples for training the recognition model. The current recognition model can be continuously trained with the new samples to adjust the parameters of the current recognition model (the adjustable parameters can be the relevant processing parameters for processing the signal, such as the parameters in the above-mentioned causal analysis process, as well as the filtering parameters, feature extraction parameters, etc. mentioned later, and can also be the relevant parameters of the neural network in the recognition module, such as weights, biases, convolution kernels, etc.). Due to changes in the user's age, mood, habits, etc., the EEG signals and functional near-infrared spectroscopy signals corresponding to the same intention will change. Therefore, by increasing and updating the training samples, the recognition accuracy of the recognition model can be improved, or the recognition model can be adaptively adjusted to follow the changes in the user's age, mood, habits, etc.

[0068] In order to improve the recognition accuracy and facilitate the recognition of the recognition model, the EEG signals and the functional near-infrared spectroscopy signals can be preprocessed before processing them, including: high-pass filtering, low-pass filtering and band-pass filtering of the EEG signals, and low-pass filtering of the functional near-infrared spectroscopy signals; and / or, baseline correction of the EEG signals and the functional near-infrared spectroscopy signals, respectively.

[0069] Specifically, for EEG signals, a high-pass filter can be applied to remove low-frequency components, a low-pass filter can be applied to remove high-frequency components, and a band-pass filter can be applied to retain signals in a specific frequency band. The order of the three is not limited, and one or two of the three can be selected. For functional near-infrared spectroscopy signals, a low-pass filter can be applied to remove high-frequency noise.

[0070] For example, the EEG signal can be high-pass filtered by the following formula:

[0071]

[0072] Among them, H hp (f) is the high-pass filter used for EEG signals, f is the frequency of the EEG signal, and f h is the first preset cutoff frequency (the value can be set as needed).

[0073] The EEG signal can be low-pass filtered by the following formula:

[0074]

[0075] Among them, H hp (f) is the low-pass filter used for EEG signals, f is the frequency of the EEG signal, and f l is the second preset cutoff frequency (the value can be set as needed), f NIRSis a third preset cutoff frequency, and its value is obtained according to the frequency of the functional near infrared spectroscopy signal, such as the maximum value of the frequency of the functional near infrared spectroscopy signal.

[0076] The EEG signal can be bandpass filtered by the following formula:

[0077]

[0078] Among them, H bp (f) is a bandpass filter for EEG signals, and f1 and f2 are the lower and upper cutoff frequencies of the bandpass filter, respectively (the values ​​can be set as needed).

[0079] The functional near-infrared spectroscopy signal can be low-pass filtered by the following formula:

[0080]

[0081] Among them, H hp ′(f) is a low-pass filter for functional near-infrared spectroscopy signal, f′ is the frequency of functional near-infrared spectroscopy signal, and f l ' is the fourth preset cut-off frequency (the value can be set as required).

[0082] Baseline correction can eliminate the offset or fluctuation of the baseline in the signal, making it more suitable for subsequent analysis and interpretation. In some examples, baseline correction is performed on the EEG signal and the functional near-infrared spectroscopy signal, respectively, including: calculating the mean of the EEG signal and the functional near-infrared spectroscopy signal, respectively; subtracting the EEG signal from the mean of the EEG signal, and subtracting the functional near-infrared spectroscopy signal from the mean of the functional near-infrared spectroscopy signal.

[0083] Specifically, the following formula can be used for baseline correction:

[0084] x corrected (t) = x(t) - μ

[0085] For EEG signals, x corrected (t) represents the EEG signal at time t after baseline correction, x(t) represents the EEG signal at time t, and μ represents the mean value of the EEG signal. Similarly, for the functional near-infrared spectroscopy signal, x corrected (t) represents the functional near-infrared spectral signal at time t after baseline correction, x(t) represents the functional near-infrared spectral signal at time t, and μ represents the mean of the functional near-infrared spectral signal.

[0086] In some embodiments of the present invention, the method further includes: generating a control instruction according to the user's intention.

[0087] In this embodiment, feedback information is also input for control instructions.

[0088] Specifically, the correspondence between the intention and the control instruction can be established in advance, for example, the intention is to control the robot to step with its left foot, and the control instruction is an instruction to control the robot to step with its left foot. In some examples, when the recognized intention and the generated control instruction are both accurate, the input feedback information is that the recognition is accurate and the control is accurate. At this time, the parameter adjustment of the recognition model can be omitted, and the target control (such as controlling the robot to step with its left foot) can be performed according to the control instruction; when the recognized intention is accurate, but the generated control instruction is inaccurate, the input feedback information is that the recognition is accurate and the control is inaccurate. At this time, the parameter adjustment of the recognition model can be omitted, but the target control is not performed, and the control instruction is regenerated until an accurate control instruction is generated (target control is performed) and the generation time does not exceed the time threshold or the number of generation times does not exceed the preset number of times. When the recognized intention is inaccurate, but the generated control instruction is accurate, the input feedback information is that the recognition is inaccurate and the control is accurate. At this time, the parameter adjustment of the recognition model is performed, and the target control is performed according to the control instruction. Thus, not only the adjustment of the accuracy rate of intention recognition can be achieved, but also the control effect can be improved, and the user's interactive experience can be improved.

[0089] For ease of understanding, Figure 2 As shown, the user intention recognition method based on multimodal brain-computer interface of an embodiment of the present invention can be divided into three parts: a data processing part, a recognition control part and a user interaction feedback part.

[0090] Specifically, see Figure 2 The data processing part includes a signal acquisition module (i.e., the above-mentioned step S11), a preprocessing module (i.e., the above-mentioned filtering and baseline correction part), a feature extraction module (i.e., the above-mentioned extraction of EEG features and functional near-infrared spectral features), a causal analysis module (i.e., the above-mentioned process of obtaining the causal relationship analysis results) and the above-mentioned step S12; the recognition control part includes an intention and control instruction generation module (generated according to the recognized intention) obtained by the data processing part; the user interaction feedback part includes a user feedback module (i.e., the above-mentioned user makes feedback on whether the recognition is accurate or not based on the recognized intention and the generated control instruction). Afterwards, the user interaction feedback part can act on the data processing part to adjust the parameters of the signal acquisition, processing module, and recognition module in the data processing part. As a result, the accuracy of intention recognition can be improved by increasing the number of signals and providing feedback on real-time recognition results, thereby improving the user experience.

[0091] Figure 3 It is a structural block diagram of a user intention recognition device based on a multimodal brain-computer interface according to an embodiment of the present invention.

[0092] like Figure 3As shown, the user intention recognition device 300 based on the multimodal brain-computer interface includes: an acquisition unit 310, a recognition unit 320, a receiving unit 330 and an adjustment unit 340.

[0093] Among them, the acquisition unit 310 is used to acquire the user's EEG signals and functional near-infrared spectral signals; the recognition unit 320 is used to use the current recognition model to obtain the user's intention based on the EEG signals and functional near-infrared spectral signals, wherein the initial recognition model is trained on the user's historical EEG signals and historical functional near-infrared spectral signals; the receiving unit 330 is used to receive feedback information input by the user regarding the intention; the adjustment unit 340 is used to adjust the parameters of the current recognition model according to the feedback information for the next intention recognition.

[0094] In some embodiments of the present invention, the recognition model includes a processing module and a recognition module; using the current recognition model to obtain the user's intention based on the EEG signal and the functional near-infrared spectral signal, including: using the current processing module to process the EEG signal and the functional near-infrared spectral signal to obtain target input parameters; using the current recognition module to obtain the user's intention based on the target input parameters.

[0095] In some embodiments of the present invention, EEG signals and functional near-infrared spectral signals are processed to obtain target input parameters, including: performing causal analysis on the EEG signals and functional near-infrared spectral signals to obtain causal relationship analysis results; and / or performing feature extraction on the EEG signals and functional near-infrared spectral signals respectively to obtain EEG features and functional near-infrared spectral features.

[0096] In some embodiments of the present invention, before the recognition module processes the EEG signal and the functional near-infrared spectroscopy signal, it also performs high-pass filtering, low-pass filtering and band-pass filtering on the EEG signal, and performs low-pass filtering on the functional near-infrared spectroscopy signal; and / or performs baseline correction on the EEG signal and the functional near-infrared spectroscopy signal, respectively.

[0097] In some embodiments of the present invention, baseline correction is performed on the EEG signal and the functional near-infrared spectroscopy signal, respectively, including: calculating the mean of the EEG signal and the functional near-infrared spectroscopy signal, respectively; subtracting the EEG signal from the mean of the EEG signal, and subtracting the functional near-infrared spectroscopy signal from the mean of the functional near-infrared spectroscopy signal.

[0098] In some embodiments of the present invention, causal analysis is performed on EEG signals and functional near-infrared spectral signals to obtain causal relationship analysis results, including: synchronously processing the EEG signals and the functional near-infrared spectral signals; establishing a first univariate autoregression model for the synchronously processed EEG signals, establishing a second univariate autoregression model for the synchronously processed functional near-infrared spectral signals, establishing a first bivariate autoregression model considering the influence of the synchronously processed functional near-infrared spectral signals on the synchronously processed EEG signals, and establishing a second bivariate autoregression model considering the influence of the synchronously processed EEG signals on the synchronously processed functional near-infrared spectral signals; calculating a first prediction error sum of squares according to the first univariate autoregression model, calculating a second prediction error sum of squares according to the second univariate autoregression model, calculating a third prediction error sum of squares according to the first bivariate autoregression model, and calculating a fourth prediction error sum of squares according to the second bivariate autoregression model; calculating a first F statistic according to the first prediction error sum of squares and the third prediction error sum of squares, and calculating a second F statistic according to the second prediction error sum of squares and the fourth prediction error sum of squares; obtaining the causal relationship analysis results according to the first F statistic and the second F statistic.

[0099] In some embodiments of the present invention, the first F statistic and the second F statistic are calculated by the following formula:

[0100]

[0101] Among them, F Y→X is the first F statistic, F X→Y is the second F statistic, SSE(X) is the first prediction error sum of squares, SSE(X|Y) is the third prediction error sum of squares, SSE(Y) is the second prediction error sum of squares, SSE(Y\X) is the fourth prediction error sum of squares, n is the number of samples of EEG signals or functional near-infrared spectroscopy signals, p is the order of the autoregressive model of EEG signals, and q is the order of the autoregressive model of functional near-infrared spectroscopy signals.

[0102] In some embodiments of the present invention, obtaining a causal relationship analysis result based on a first F statistic and a second F statistic includes: obtaining a first p-value based on the first F statistic using an F distribution table, and obtaining a second p-value based on the second F statistic using an F distribution table; obtaining a causal relationship analysis result based on the first p-value and the second p-value.

[0103] In some embodiments of the present invention, Figure 4 As shown, the user intention recognition device 300 based on the multimodal brain-computer interface also includes: a generation unit 350, and the generation unit 350 is used to generate a control instruction according to the user's intention.

[0104] In this embodiment, feedback information is also input for control instructions.

[0105] It should be noted that Figure 3 , Figure 4 The solid line in the figure represents a communication connection relationship, and the dotted line represents that the two units are connected but not necessarily connected in communication. For example, the feedback information is input by the user according to the intention output by the recognition unit 320 and received by the receiving unit 330. It can be considered that the recognition unit 320 and the receiving unit 330 are connected through the user, but there is no communication connection between the two. In addition, for other specific implementations of the user intention recognition device 100 based on the multimodal brain-computer interface of the embodiment of the present invention, please refer to the specific implementation of the user intention recognition method based on the multimodal brain-computer interface in the above embodiment.

[0106] Based on the user intention recognition method based on a multimodal brain-computer interface in the above embodiment, the present invention proposes a computer-readable storage medium.

[0107] In this embodiment, a computer program is stored on a computer-readable storage medium. When the computer program is executed by a processor, the user intention recognition method based on a multimodal brain-computer interface of the above embodiment is implemented.

[0108] Based on the intention recognition method of the above embodiment, the present invention also proposes an electronic device.

[0109] like Figure 5 As shown, the electronic device 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present invention.

[0110] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0111] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0112] The memory 503 is used to store a computer program corresponding to the user intention recognition method based on a multimodal brain-computer interface of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the content shown in the above method embodiment.

[0113] The electronic device 500 may be the above-mentioned host computer, including but not limited to: fixed terminals such as digital TV, desktop computers, etc., and mobile terminals such as smart phones, tablet computers, etc. Figure 5 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0114] It should be noted that the logical steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0115] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0117] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0118] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0119] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0120] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0121] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for identifying user intention based on a multimodal brain-computer interface, characterized in that: include: Obtaining the user's EEG signals and functional near-infrared spectroscopy signals; Obtaining the user's intention according to the EEG signal and the functional near-infrared spectroscopy signal using the current recognition model, wherein the initial recognition model is obtained by training the user's historical EEG signal and historical functional near-infrared spectroscopy signal; receiving feedback information input by the user regarding the intention; The parameters of the current recognition model are adjusted according to the feedback information for the next intention recognition.

2. The method for identifying user intention based on a multimodal brain-computer interface according to claim 1, characterized in that: The recognition model includes a processing module and a recognition module; The obtaining the user's intention according to the EEG signal and the functional near infrared spectrum signal by using the current recognition model includes: Processing the EEG signal and the functional near infrared spectroscopy signal using the current processing module to obtain target input parameters; The user's intention is obtained according to the target input parameter using the current recognition module.

3. The method for identifying user intention based on a multimodal brain-computer interface according to claim 2, characterized in that: The step of processing the EEG signal and the functional near infrared spectroscopy signal to obtain target input parameters includes: Performing causal analysis on the EEG signal and the functional near-infrared spectroscopy signal to obtain a causal relationship analysis result; and / or Feature extraction is performed on the EEG signal and the functional near-infrared spectroscopy signal respectively to obtain EEG features and functional near-infrared spectroscopy features.

4. The method for identifying user intention based on a multimodal brain-computer interface according to claim 2, characterized in that: Before processing the EEG signal and the functional near infrared spectroscopy signal, the method further includes: Performing high-pass filtering, low-pass filtering and band-pass filtering on the EEG signal, and performing low-pass filtering on the functional near-infrared spectroscopy signal; and / or Baseline correction is performed on the EEG signal and the functional near-infrared spectroscopy signal respectively.

5. The method for identifying user intention based on a multimodal brain-computer interface according to claim 4 is characterized in that: The performing baseline correction on the EEG signal and the functional near infrared spectroscopy signal respectively comprises: Calculating the mean values ​​of the EEG signal and the functional near-infrared spectroscopy signal respectively; The EEG signal is subtracted from the mean value of the EEG signal, and the functional near-infrared spectroscopy signal is subtracted from the mean value of the functional near-infrared spectroscopy signal.

6. The method for user intention recognition based on multimodal brain-computer interface according to claim 3 is characterized in that: The performing causal analysis on the EEG signal and the functional near infrared spectroscopy signal to obtain a causal relationship analysis result includes: Synchronously processing the EEG signal and the functional near-infrared spectroscopy signal; A first univariate autoregressive model is established for the synchronously processed EEG signal, a second univariate autoregressive model is established for the synchronously processed functional near-infrared spectroscopy signal, a first bivariate autoregressive model is established considering the influence of the synchronously processed functional near-infrared spectroscopy signal on the synchronously processed EEG signal, and a second bivariate autoregressive model is established considering the influence of the synchronously processed EEG signal on the synchronously processed functional near-infrared spectroscopy signal; Calculate a first prediction error sum of squares according to the first univariate autoregressive model, calculate a second prediction error sum of squares according to the second univariate autoregressive model, calculate a third prediction error sum of squares according to the first bivariate autoregressive model, and calculate a fourth prediction error sum of squares according to the second bivariate autoregressive model; Calculating a first F statistic according to the first prediction error sum of squares and the third prediction error sum of squares, and calculating a second F statistic according to the second prediction error sum of squares and the fourth prediction error sum of squares; The causal relationship analysis result is obtained according to the first F statistic and the second F statistic.

7. The method for identifying user intention based on a multimodal brain-computer interface according to claim 6, characterized in that: The first F statistic and the second F statistic are calculated by the following formula: Among them, F Y→X is the first F statistic, F X→Y is the second F statistic, SSE(X) is the first prediction error sum of squares, SSE(XY) is the third prediction error sum of squares, SSE(Y) is the second prediction error sum of squares, SSE(YX) is the fourth prediction error sum of squares, n is the number of samples of the EEG signal or the functional near-infrared spectroscopy signal, p is the autoregressive model order of the EEG signal, and q is the autoregressive model order of the functional near-infrared spectroscopy signal.

8. The method for identifying user intention based on a multimodal brain-computer interface according to claim 6, characterized in that: The obtaining the causal relationship analysis result according to the first F statistic and the second F statistic includes: Using an F distribution table to obtain a first p-value based on the first F statistic, and using the F distribution table to obtain a second p-value based on the second F statistic; The causal relationship analysis result is obtained according to the first p value and the second p value.

9. The method for user intention recognition based on multimodal brain-computer interface according to claim 1, characterized in that: The method further comprises: generating a control instruction according to the user's intention; Wherein, the feedback information is also generated for the control instruction.

10. A user intention recognition device based on a multimodal brain-computer interface, characterized in that: include: An acquisition unit, used for acquiring the user's EEG signal and functional near infrared spectroscopy signal; A recognition unit, configured to use a current recognition model to recognize the user's intention according to the EEG signal and the functional near infrared spectroscopy signal, wherein the initial recognition model is obtained by training the user's historical EEG signal and historical functional near infrared spectroscopy signal; A receiving unit, configured to receive feedback information inputted by the user in response to the intention; An adjustment unit is used to adjust parameters of the current recognition model according to the feedback information for the next intention recognition.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for identifying user intention based on a multimodal brain-computer interface described in any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for identifying user intention based on a multimodal brain-computer interface described in any one of claims 1 to 9 are implemented.

Images