Banking system interaction method and device based on hybrid brain-computer interface

Abstract

The invention discloses a bank system interaction method and device based on a hybrid brain-computer interface, and the method comprises the steps: carrying out the real-time collection of an electro-oculogram signal, and when monitoring that the electro-oculogram characteristic value of the collected electro-oculogram signal is smaller than an electro-oculogram characteristic threshold value, indicating that the electro-oculogram signal is not polluted by the electro-oculogram signal; the collected electroencephalogram signals are relatively accurate data, so that further noise reduction is performed on the electroencephalogram signals, then frequency spectrum value monitoring is performed on the denoised electroencephalogram signals to obtain power spectrum characteristic values of the denoised electroencephalogram signals, and if the power spectrum characteristic values of the denoised electroencephalogram signals are higher than an electroencephalogram characteristic threshold value, the electroencephalogram signals are determined to be the electroencephalogram signals. If the condition shows that the electroencephalogram signals are valid, further performing stimulation source classification on the electroencephalogram signals, and controlling and operating a bank system based on the classification result of the electroencephalogram signals. According to the invention, the robustness and the control precision of the bank system are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a banking system interaction method and device based on a hybrid brain-computer interface. Background Art

[0002] With the rapid development of computer, sensor and information communication technologies, as well as the cross-integration of mathematics, biology, psychology, neuroscience and cognitive science, brain-computer interface technology is currently widely used in many fields such as rehabilitation medicine, game entertainment, military applications, and home intelligence. However, different types of brain-computer interface systems (SSVEP, motor imagery, P300) still have problems such as low control accuracy and susceptibility to noise interference. Summary of the invention

[0003] The present application provides a banking system interaction method and device based on a hybrid brain-computer interface. By adopting a dual processing method based on the power spectrum eigenvalues ​​of electrooculogram signals and electroencephalogram signals, the noise interference to the banking system based on the brain-computer interface is reduced, and the robustness and control accuracy of the system are improved.

[0004] In a first aspect, a banking system interaction method based on a hybrid brain-computer interface is provided, the method comprising:

[0005] In the process of presenting a visual stimulation interface, collecting electroencephalogram signals and electrooculogram signals, wherein the visual stimulation interface includes a plurality of stimulation sources;

[0006] If the electrooculogram characteristic value of the electrooculogram signal extracted within the electrooculogram signal acquisition time window is less than the electrooculogram characteristic threshold value, denoising the electrooculogram signal collected within the electrooculogram signal acquisition time window to obtain a denoised electrooculogram signal;

[0007] Monitoring the spectrum value of the denoised EEG signal to obtain a power spectrum characteristic value of the denoised EEG signal;

[0008] If the power spectrum eigenvalue is higher than the EEG characteristic threshold, performing stimulus source classification on the denoised EEG signal to obtain a classification result of the EEG signal;

[0009] Based on the classification result of the EEG signal, the operation of the banking system is controlled.

[0010] In some embodiments, the method further comprises:

[0011] If the electrooculogram feature values ​​of multiple electrooculogram signals extracted within the electrooculogram signal acquisition time window are all greater than the electrooculogram feature threshold, further processing of the electrooculogram signal is canceled, and the user is guided to look at the visual stimulation interface again.

[0012] In some embodiments, the visual stimulation interface includes a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface includes a plurality of first-type stimulation sources, the plurality of the first-type stimulation sources are used to trigger a first-type control instruction, and the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, the second visual stimulation interface includes a plurality of second-type stimulation sources, the plurality of the second-type stimulation sources are used to trigger a second-type control instruction, the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, and the stimulation frequencies of the plurality of the first-type stimulation sources are partially or completely the same as the stimulation frequencies of the plurality of the second-type stimulation sources;

[0013] The method further comprises:

[0014] Collecting electromyographic signals, and performing feature extraction on the electromyographic signals to obtain electromyographic feature values;

[0015] If the myoelectric characteristic value is greater than the myoelectric characteristic threshold, the intelligent visual stimulator is triggered to switch between the first visual stimulation interface and the second visual stimulation interface.

[0016] In some embodiments, the controlling the operation of the banking system based on the classification result of the EEG signal comprises:

[0017] If the classification result of the EEG signal is greater than or equal to the classification threshold, based on the classification result of the EEG signal, controlling the operation of the banking system; the method further includes:

[0018] If the classification result of the EEG signal is less than the classification threshold, it is determined that the classification result of the EEG signal is invalid.

[0019] In some embodiments, the method further comprises:

[0020] Performing correlation analysis on classification results of multiple EEG signals collected from the same user to obtain a maximum correlation coefficient value of the classification results of the multiple EEG signals;

[0021] The classification threshold is adjusted based on an average value of maximum correlation coefficient values ​​of the classification results of the multiple electroencephalogram signals.

[0022] In some embodiments, the monitoring of the spectrum value of the denoised EEG signal to obtain the power spectrum characteristic value of the denoised EEG signal includes:

[0023] The denoised EEG signal is converted from the time domain to the frequency domain using a fast Fourier transform method to obtain a power spectrum characteristic value of the denoised EEG signal.

[0024] In a second aspect, a banking system interaction device based on a hybrid brain-computer interface is provided, the device comprising:

[0025] A signal acquisition module, used to collect EEG signals and EOG signals during the presentation of a visual stimulation interface, wherein the visual stimulation interface includes a plurality of stimulation sources;

[0026] A signal processing module, for, if the electrooculogram characteristic value of the electrooculogram signal extracted within the electrooculogram signal acquisition time window is less than the electrooculogram characteristic threshold, denoising the electrooculogram signal collected within the electrooculogram signal acquisition time window to obtain a denoised electrooculogram signal; monitoring the spectrum value of the denoised electrooculogram signal to obtain a power spectrum characteristic value of the denoised electrooculogram signal; and if the power spectrum characteristic value is higher than the electrooculogram characteristic threshold, classifying the denoised electrooculogram signal by stimulus source to obtain a classification result of the electrooculogram signal;

[0027] The control processing module is used to control the operation of the banking system based on the classification result of the EEG signal.

[0028] In some embodiments, the signal processing module is also used to cancel further processing of the EEG signal and guide the user to look at the visual stimulation interface again if the EEG feature values ​​of multiple EEG signals extracted within the EEG signal acquisition time window are all greater than the EEG feature threshold.

[0029] In some embodiments, the visual stimulation interface includes a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface includes a plurality of first-type stimulation sources, the plurality of the first-type stimulation sources are used to trigger a first-type control instruction, and the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, the second visual stimulation interface includes a plurality of second-type stimulation sources, the plurality of the second-type stimulation sources are used to trigger a second-type control instruction, the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, and the stimulation frequencies of the plurality of the first-type stimulation sources are partially or completely the same as the stimulation frequencies of the plurality of the second-type stimulation sources;

[0030] The signal acquisition module is further used to collect electromyographic signals and perform feature extraction on the electromyographic signals to obtain electromyographic feature values;

[0031] The signal processing module is further used to trigger the intelligent visual stimulator to switch between the first visual stimulation interface and the second visual stimulation interface if the myoelectric characteristic value is greater than the myoelectric characteristic threshold.

[0032] In some embodiments, the control processing module is used to control the operation of the banking system based on the classification result of the EEG signal if the classification result of the EEG signal is greater than or equal to the classification threshold; if the classification result of the EEG signal is less than the classification threshold, determine that the classification result of the EEG signal is invalid.

[0033] In some embodiments, the signal processing module is also used to perform correlation analysis on the classification results of multiple EEG signals collected from the same user to obtain the maximum correlation coefficient value of the classification results of the multiple EEG signals; and adjust the classification threshold based on the average value of the maximum correlation coefficient values ​​of the classification results of the multiple EEG signals.

[0034] In some embodiments, the signal processing module is used to convert the denoised EEG signal from the time domain to the frequency domain using a fast Fourier transform method to obtain a power spectrum eigenvalue of the denoised EEG signal.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] The embodiment of the present application collects the electrooculogram signal in real time. When it is monitored that the electrooculogram characteristic value of the collected electrooculogram signal is less than the electrooculogram characteristic threshold, this indicates that the EEG signal has not been contaminated by the electrooculogram signal and the collected EEG signal is relatively accurate data. Therefore, the electrooculogram signal is further denoised, and the spectrum value of the denoised EEG signal is monitored to obtain the power spectrum characteristic value of the denoised EEG signal. If the power spectrum characteristic value of the denoised EEG signal is higher than the EEG characteristic threshold, this indicates that the EEG signal is valid. The EEG signal is further classified as a stimulus source, and the operation bank system is controlled based on the classification result of the EEG signal. Since the judgment is based on both the EOG signal and the EOG feature threshold and the power spectrum eigenvalue of the EEG signal and the EEG feature threshold, it is equivalent to using a double judgment processing method to check before controlling the banking system based on the EEG signal. The more accurate EEG signal is used to control the banking system, which reduces the adverse effects on the robustness of the banking system when the banking system is controlled based on the EEG signal contaminated by the EOG signal or the EEG signal whose power spectrum eigenvalue does not meet the requirements. Therefore, the robustness and control accuracy of the banking system are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a general architecture diagram of a banking system interaction device based on a hybrid brain-computer interface provided in an embodiment of the present application;

[0038] Figure 2 is a block diagram of a signal acquisition module provided in an embodiment of the present application;

[0039] Figure 3a is a schematic diagram of a visual stimulation interface of type 1 provided in an embodiment of the present application;

[0040] Figure 3b This is a schematic diagram of a visual stimulation interface of type 2 provided in an embodiment of the present application;

[0041] Figure 4is an architecture diagram of a signal processing module provided in an embodiment of the present application;

[0042] Figure 5 is a schematic diagram of the architecture of the control processing module provided in an embodiment of the present application;

[0043] Figure 6 This is a flowchart of a banking system interaction method based on a hybrid brain-computer interface provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0045] The following is an explanation of some terminology concepts involved in the embodiments of the present application.

[0046] Brain Computer Interface (BCI): A communication method that can establish a direct connection between the brain and the external environment without relying on peripheral nerves and muscles.

[0047] Hybrid brain-computer interface (Hybrid BCI): A communication method that establishes a brain-computer interface that integrates different types of biological signals.

[0048] Electroencephalogram (EEG): The potential changes recorded from the scalp of humans or animals. It is a non-invasive brain signal acquisition technology.

[0049] Electromyography (EMG): The superposition of action potentials of motor units in many muscle fibers in time and space, usually the electrical activity on the muscle surface.

[0050] Electrooculography (EOG): Different eye movements, such as blinking and eye movement, can cause depolarization and hyperpolarization between the retina and cornea, forming a potential difference between the retina and cornea.

[0051] Steady state visual evoked potential (SSVEP): When the subjects are subjected to visual stimulation of a certain frequency, a continuous visual evoked potential corresponding to the stimulation frequency will be generated in the EEG signal in the occipital lobe area.

[0052] The following is an introduction to the application scenarios involved in the embodiments of the present application.

[0053] The current SSVEP-based brain-computer interface (SSVEP-BCI) interaction method is easily affected by the user's failure to look at the visual stimulator or the shift in gaze target, and the EEG signals of different users are affected differently by SSVEP, resulting in differences in the classification and recognition results of different users receiving the same visual frequency stimulation. In addition, the visual stimulators of most SSVEP-BCI systems are relatively simple, so they cannot meet the user's personalized adaptation and the system is not easy to use.

[0054] The current hybrid brain-computer interface-based interaction method can handle more complex commands through the fusion processing of multiple signals, thus improving the flexibility of the system. However, due to the different signal types used by various hybrid brain-computer interface systems, different EEG paradigms, different control strategies and many other reasons, the robustness of the system cannot be guaranteed.

[0055] The current brain-computer interface interaction method based on the banking industry has expanded the application field of brain-computer interface technology and realized brain-computer interface control of banking scene equipment. However, the current system has many problems such as low accuracy of EEG signals, single control strategy, or inflexible system.

[0056] In response to the above problems, the embodiment of the present application proposes a banking system interaction method and device based on a hybrid brain-computer interface, which helps disabled people with serious diseases such as stroke and spinal cord injury to achieve real-time control of the banking system by establishing a direct information exchange channel between the brain and external devices that does not rely on peripheral nerves and muscles, effectively improving the self-care ability and quality of life of disabled people, and improving the usability and portability of the system. In addition, the embodiment of the present application improves the accuracy of EEG signals, thereby improving the robustness and control accuracy of the brain-computer interaction system, reducing the user's interaction burden, and thus improving the user experience.

[0057] The device provided in the embodiment of the present application includes a signal acquisition module, a signal processing module and a control processing module. In the signal acquisition module, the collection and transmission of EEG signals, EMG signals and EOG signals are completed through a portable signal collector. Among them, after the user wears the portable signal collector, the real-time collection of EEG signals is completed by looking at a stimulation source of a certain frequency in the interface of the intelligent visual stimulator. The signal processing module includes a pre-signal processing module, a feature extraction unit and a classification and recognition unit.

[0058] The pre-signal processing module uses the wavelet analysis method to denoise the EEG signal. The feature extraction unit completes the feature extraction of the EEG signal, EMG signal and EEG signal through the EEG monitor, EMG selector and spectrum value monitor. Among them, if the EEG monitor extracts multiple EEG features within the EEG signal acquisition time window, it is determined that the EEG signal is contaminated and no further processing is performed on the EEG signal. If the EMG selector extracts the EMG feature, it can trigger the switching of the intelligent visual stimulator interface. If the characteristic value of the EEG signal extracted by the spectrum value monitor is higher than the predetermined EEG characteristic value, it is determined that the EEG signal is valid and the EEG signal continues to be classified and identified; in the classification and identification unit, the classification identifier uses the canonical correlation analysis method to classify and identify the EEG signal, and re-judges the classification result through the adaptive idle state monitor. In the control processing module, the EEG signal intention is converted into a control strategy of direct control or shared control to realize the processing of different business scenarios in the banking system, which improves the ease of use and portability of the system.

[0059] Figure 1 This is a general architecture diagram of a bank system interaction device based on a hybrid brain-computer interface provided in an embodiment of the present application. The device includes three parts: a signal acquisition module, a signal processing module, and a control processing module.

[0060] The signal acquisition module includes a portable EEG collector and an intelligent visual stimulator. The signal processing module includes a pre-signal processing module, a feature extraction unit and a classification and recognition unit. Among them, the feature extraction unit includes an electrooculogram monitor, an electromyography monitor and a spectrum value monitor, the recognition and classification unit includes a classification identifier and an adaptive idle state monitor, and the control processing module includes a control processor, an external device (bank system) and a feedback processor.

[0061] Figure 2 : is a block diagram of the signal acquisition module provided in the embodiment of the present application. The signal acquisition module is mainly used for the collection and transmission of EEG signals, EMG signals and EOG signals. Specifically, after wearing the portable signal collector, the user looks at a stimulation source of a certain frequency in the visual stimulation interface presented by the intelligent visual stimulator. While the user is looking at the stimulation source, the signal acquisition module completes the real-time collection of EEG signals, EMG signals and EOG signals in the occipital lobe area of ​​the brain, and transmits the collected EEG signals, EMG signals and EOG signals in the occipital lobe area of ​​the brain to the signal processing module via Bluetooth.

[0062] The signal acquisition module includes a signal collector and an intelligent visual stimulator. The intelligent visual stimulator is used to present two types of visual stimulation interfaces, which are respectively referred to as type 1 visual stimulation interface (also referred to as the first visual stimulation interface) and type 2 visual stimulation interface (also referred to as the second visual stimulation interface). Type 1 visual stimulation interface is as follows: Figure 3aAs shown, the visual stimulation interface of type 2 is as follows Figure 3b shown.

[0063] The visual stimulation interface of type 1 includes multiple stimulation sources of type 1, and the visual stimulation interface of type 2 includes multiple stimulation sources of type 2. The stimulation sources of type 1 are used to trigger control instructions of type 1 (or to execute business functions of type 1, such as moving up or down). The stimulation sources of type 2 are used to trigger control instructions of type 2 (or to execute business functions of type 2, such as entering a specific interface of a banking system). The stimulation frequencies of multiple stimulation sources of type 1 and the stimulation frequencies of multiple stimulation sources of type 2 partially overlap or are completely the same. For example, the stimulation sources of type 1 include stimulation sources with stimulation frequency k and stimulation frequency j, and the stimulation sources of type 2 include stimulation sources with stimulation frequency k and stimulation frequency j. In other words, the visual stimulation interface of type 1 and the visual stimulation interface of type 2 have stimulation sources with the same stimulation frequency but different functions.

[0064] Exemplarily, the visual stimulation interface of type 1 includes 8 stimulation sources, and the control instructions corresponding to the 8 stimulation sources are respectively moving up, moving down, moving left, moving right, sliding up the page, sliding down the page, confirming the click, and returning to the previous page. The 8 stimulation sources have 8 different stimulation frequencies. For example, a stimulation source with a stimulation frequency of 6Hz is used to trigger a control instruction for moving up, a stimulation source with a stimulation frequency of 7Hz is used to trigger a control instruction for moving down, a stimulation source with a stimulation frequency of 8Hz is used to trigger a control instruction for moving left, a stimulation source with a stimulation frequency of 9Hz is used to trigger a control instruction for moving right, a stimulation source with a stimulation frequency of 10Hz is used to trigger a control instruction for sliding up the page, a stimulation source with a stimulation frequency of 11Hz is used to trigger a control instruction for sliding down the page, a stimulation source with a stimulation frequency of 12Hz is used to trigger a control instruction for confirming the click, and a stimulation source with a stimulation frequency of 13Hz is used to trigger a control instruction for returning to the previous page.

[0065] The corresponding relationship between the stimulation frequency of each stimulation source and the control instructions in the visual stimulation interface of type 1 is shown in Table 1. Table 1 is only for exemplary description, and the basic functions set by the stimulation source, the flashing frequency corresponding to the basic functions and the corresponding control instructions are not limited to the above examples.

[0066] Table 1 Stimulation frequency and corresponding instructions of type 1 visual stimulation interface

[0067]

[0068]

[0069] The number of stimulus sources in the visual stimulation interface of type 2 is not fixed, and the functions in the banking system represented by the stimulus sources are also not fixed. For example, when entering the homepage of the banking system, the feedback processor in the control processing module will feed back the core functions involved in this page to the intelligent visual stimulator, and the intelligent visual stimulator will generate a corresponding visual stimulation interface to achieve real-time linkage between the intelligent visual stimulator and the banking system. For example, a stimulation frequency of 6Hz represents my account, a stimulation frequency of 7Hz represents transfer, a stimulation frequency of 7.5Hz represents popular activities, a stimulation frequency of 8Hz represents income and expenditure, a stimulation frequency of 8.57Hz represents Xiaodou Paradise, a stimulation frequency of 9Hz represents scanning, a stimulation frequency of 10Hz represents credit cards, a stimulation frequency of 11Hz represents deposits, a stimulation frequency of 12Hz represents financial products, a stimulation frequency of 13Hz represents loans, a stimulation frequency of 13.25Hz represents living expenses, and a stimulation frequency of 13.5Hz represents city districts. Table 2 shows the stimulation frequency and corresponding control instructions of the visual stimulation interface of type 2. The examples shown in Table 2 are only for illustrative purposes. The business modules included in each visual stimulation interface, the flashing frequency corresponding to the business modules, and the corresponding control instructions are not limited to the above examples.

[0070] Table 2 Stimulation frequency and corresponding control instructions of type 2 visual stimulation interface

[0071] Stimulation frequency (Hz) Control instructions 6 Click to enter my account 7 Click to enter transfer 7.5 Click to enter popular activities 8 Click to enter income and expenditure 8.57 Click to enter Xiaodou Paradise 9 Click to enter Scan 10 Click to enter credit card 11 Click to enter deposit 12 Click to enter financial products 13 Click to enter loan 13.25 Click to enter life payment 13.5 Click to enter the city area

[0072] Figure 4 This is an architecture diagram of a signal processing module provided in an embodiment of the present application. The signal processing module is mainly used for preprocessing, feature processing and classification recognition of EEG signals, as well as feature processing of EMG signals and feature processing of EO signals. The signal processing module includes a pre-signal processing module, a feature signal processing module and a classification recognition unit.

[0073] The pre-signal processing module uses the wavelet analysis method to denoise the collected EEG signals. The denoising process includes removing the electrocardiogram, electrooculogram, power frequency artifacts and environmental noise. After denoising, the EEG signal with a frequency range of 0.01-32Hz is obtained.

[0074] The feature extraction unit includes an electrooculogram monitor, an electromyography selector and a spectrum value monitor. The electrooculogram monitor is used to complete the feature extraction of electrooculogram signals, the electromyography selector is used to complete the feature extraction of electromyography signals, and the spectrum value monitor is used to complete the feature extraction of electroencephalogram signals.

[0075] The electrooculogram monitor is used to collect electrooculogram signals in real time, and pre-process the collected electrooculogram signals through wavelet transform to obtain the electrooculogram characteristic values ​​of the electrooculogram signals. If the electrooculogram characteristic values ​​of multiple electrooculogram signals (such as blinking or saccade) extracted within an electroencephalogram signal acquisition time window are greater than the set electrooculogram characteristic threshold value (for example, the user is blinking or saccade frequently), the electrooculogram monitor cancels further processing of the electroencephalogram signal. Specifically, considering that the electrooculogram signal and the electroencephalogram signal are extracted in parallel, the potential difference of the electrooculogram signal is much greater than the potential difference of the electroencephalogram signal. If the user blinks when collecting the electroencephalogram signal, it will affect the accuracy of the collected electroencephalogram signal, which is equivalent to the electroencephalogram signal being contaminated. In view of this, if the electrooculogram monitor determines that the electrooculogram characteristic values ​​of multiple electrooculogram signals extracted within an electroencephalogram signal acquisition time window are greater than the set electrooculogram characteristic threshold value, the electrooculogram monitor does not need to further process the electroencephalogram signal, thereby avoiding the unnecessary increase in the amount of calculation and processing resources caused by further processing of the already contaminated electroencephalogram signal.

[0076] The formula based on the electrooculogram monitor is as follows:

[0077]

[0078] Among them, θ is the electrooculogram characteristic value, EOG H EOG is the number of eye movements detected. V is the number of monitored eye blinks, and N is the number of EEG signal acquisition time windows.

[0079] The electromyographic selector is used to collect electromyographic signals in real time, and denoise the collected electromyographic signals through wavelet transform, and extract the features of the denoised electromyographic signals using the root mean square algorithm (RMS) to obtain electromyographic characteristic values. If the extracted electromyographic characteristic value is higher than the set electromyographic characteristic threshold, the visual stimulation interface presented in the intelligent visual stimulator is triggered to switch between the two stimulation paradigms of type 1 visual stimulation interface and type 2 visual stimulation interface.

[0080] Since the frequency characteristics of the SSVEP signal are very obvious, the spectrum value monitor converts the denoised EEG signal from the time domain to the frequency domain by using the fast Fourier transform method, thereby obtaining the power spectrum characteristic value of the denoised EEG signal. If the power spectrum characteristic value of the denoised EEG signal is higher than the set EEG characteristic threshold, the spectrum value monitor predicts that the EEG signal is valid, and continues to classify the stimulus source for the EEG characteristic threshold to obtain the classification result of the EEG signal. The classification result of the EEG signal is, for example, the stimulus frequency of the stimulus source corresponding to the EEG signal, and the classification result can characterize the EEG signal as the stimulus source of which frequency in the visual stimulation interface the user is looking at. Among them, since the EEG signal generated by the stimulus source that the user is looking at is usually the signal with the highest characteristic value in the power spectrum, the EEG signal is screened based on the EEG characteristic threshold, which can filter out the EEG signals of some stimulus sources that the user is not looking at, and reduce the amount of calculation caused by the classification of these EEG signals.

[0081] The formula based on the spectrum value monitor is as follows:

[0082]

[0083] Where x(n) is the EEG signal and the signal length is finite, W N =e -j2π / N represents the rotation factor.

[0084] After calculating the spectrum X(k) through FFT, the power spectrum estimate P(k) of x(n) is calculated according to the power spectrum density analysis theorem. The formula is as follows:

[0085]

[0086] The classification and recognition unit includes a classification and recognition device and an adaptive idle state monitor. The classification and recognition device classifies and recognizes the stimulus source of the denoised EEG signal to obtain a classification result of the EEG signal.

[0087] Afterwards, the adaptive idle state monitor compares the classification result of the EEG signal with the classification result. If the classification result of the EEG signal reaches the classification threshold, the adaptive idle state monitor determines that the classification result of the EEG signal is valid, and the adaptive idle state monitor transmits the classification result to the control processing module. If the classification result of the EEG signal does not reach the classification threshold, the adaptive idle state monitor determines that the classification result of the EEG signal is invalid, and the adaptive idle state monitor will not transmit the classification result of the EEG signal to the control processing module to prevent erroneous control operations.

[0088] The recursive relationship based on the classification identifier is as follows:

[0089] Suppose there are two sets of multidimensional random variables X and Y, and X∈R H×J,Y∈R I×J Canonical correlation analysis expects to find a pair of linear transformations w X ∈R H×1 and w Y ∈R I×1 , so that the linear combination x = w X T x and y = w Y T The correlation between Y is maximized, and the formula is as follows:

[0090]

[0091] Among them, C xx =XX T and C yy =YY T Represents the intra-class covariance matrices of two sets of multidimensional random variables X and Y, C xy =XY T Represents the inter-class covariance matrix of two sets of multidimensional random variables X and Y. x and y are projected on w X and w Y , the maximum value of ρ represents the maximum canonical correlation coefficient.

[0092] If the feedback processor in the control processing module feeds back i functions to the intelligent visual stimulator, the intelligent visual stimulator generates i stimulation sources, X is the N s EEG data of n channels collected, and the corresponding reference signal Y of the same length is i The formula is defined as follows:

[0093]

[0094] Among them, N represents the number of sampling points, S represents the sampling rate, and N h Represents the number of harmonics of the signal.

[0095] Therefore, it can be concluded that EEG signal X and EEG signal Y i The maximum correlation coefficient.

[0096] The adaptive idle state monitor is used to monitor whether the classification result of the EEG signal reaches the set classification threshold. If the classification result of the EEG signal is greater than or equal to the set classification threshold, the classification result of the EEG signal is determined to be valid. If the classification result of the EEG signal does not reach the set classification threshold, the classification result of the EEG signal is determined to be invalid. In addition, because the EEG signals of different users are affected differently by SSVEP, the maximum correlation coefficient values ​​of the classification recognition of different users under the same visual frequency stimulation are also different. Therefore, the adaptive idle state monitor sets a default initial classification threshold i. In the real-time control process of the bank system, the adaptive idle state monitor performs correlation analysis on the classification results of multiple EEG signals collected from the same user to obtain the maximum correlation coefficient value of the classification results of multiple EEG signals. The adaptive idle state monitor averages the maximum correlation coefficient values ​​of the classification results of multiple EEG signals, adjusts the initial classification threshold based on the average value, obtains the adjusted classification threshold, and judges whether the classification results of subsequent EEG signals are valid based on the adjusted classification threshold.

[0097] Since the classification threshold is adjusted based on the maximum correlation coefficient value of the classification results of multiple EEG signals collected from the same user, different classification thresholds can be used for different users, and the classification threshold can also be adjusted dynamically, thereby improving the flexibility of the system.

[0098] The formula based on the adaptive idle state monitor is as follows:

[0099]

[0100] The adaptive idle state monitor proposed in the embodiment of the present application adjusts the classification threshold based on the maximum correlation coefficient value of the classification results of multiple EEG signals of the same user, so that it can set an adaptive idle state threshold for different users, and judge whether the classification result is valid based on the classification threshold, and use the valid classification result to control the operating system, thereby improving the accuracy of the system.

[0101] Figure 5 1 is a schematic diagram of the architecture of a control processing module provided in an embodiment of the present application. The control processing module mainly includes a control processor, an external device (banking system) and a feedback processor. The control processor converts the classification result of the EEG signal output by the classification identifier in the signal processing module into a control command, which is a direct control command or a shared control command, and operates the banking system-related scenes based on the control command.

[0102] The feedback processor feeds back the identified current basic function to the intelligent visual stimulator, so that the intelligent visual stimulator can generate a corresponding visual stimulation interface.

[0103] Because based on Figure 1 The architecture shown utilizes a processing method that integrates an intelligent visual stimulator, a feedback processor, an electromyographic selector, and a control processor to achieve real-time linkage between the visual stimulation paradigm and the banking system, forming a control strategy with a feedback linkage mechanism, reducing the user's cognitive pressure and interaction burden, and improving the user experience.

[0104] Figure 6 1 is a flow chart of a banking system interaction method based on a hybrid brain-computer interface provided in an embodiment of the present application. Figure 6 As shown, the method includes the following steps.

[0105] Step S210: After the user wears the portable signal collector, the intelligent visual stimulator presents a visual stimulation interface, and the user gazes at the visual stimulation interface presented by the intelligent visual stimulator. The visual stimulation interface includes stimulation sources of multiple frequencies. The stimulation source is, for example, a light source that flashes at a certain frequency. The visual stimulation interface can be a visual stimulation interface of type 1 or a visual stimulation interface of type 2.

[0106] Step S212: The portable signal collector completes the real-time collection of EEG signals, EMG signals and EOG signals in the occipital lobe area of ​​the brain, and transmits the collected EEG signals, EMG signals and EOG signals in the occipital lobe area of ​​the brain to the signal processing module via Bluetooth.

[0107] In one possible implementation, the portable signal collector includes multiple electrode patches, which are respectively connected to the user's occipital lobe, eyes, and mouth. The electrode patch connected to the occipital lobe collects EEG signals from the occipital lobe area, the electrode patch connected to the mouth collects EMG signals, and the electrode patch connected to the eyes collects EOG signals.

[0108] Step S214, the electrooculogram monitor in the signal processing module performs feature extraction on the electrooculogram signal collected in a certain electrooculogram signal collection time window to obtain the electrooculogram characteristic value of the electrooculogram signal. The electrooculogram monitor determines whether the electrooculogram characteristic value of the electrooculogram signal exceeds the electrooculogram characteristic threshold. If the electrooculogram characteristic value of the electrooculogram signal collected in an electrooculogram signal collection time window is less than the electrooculogram characteristic threshold factor, the electrooculogram monitor executes step S216 for the electrooculogram signal collected in the electrooculogram signal collection time window. If the electrooculogram characteristic values ​​of multiple electrooculogram signals detected in an electrooculogram signal collection time window are greater than the electrooculogram characteristic threshold, the electrooculogram monitor determines that the electrooculogram signal collected in the electrooculogram signal collection time window is an invalid signal, and therefore returns to step S210 for the electrooculogram signal, and the user needs to look at the visual stimulation interface presented in the intelligent visual stimulator again.

[0109] Step S216: the pre-signal processing module performs denoising on the EEG signal collected in the EEG signal collection time window to obtain a denoised EEG signal.

[0110] In step S218, the spectrum value monitor monitors the spectrum value of the denoised EEG signal to extract the power spectrum characteristic value of the denoised EEG signal. The spectrum value monitor determines whether the power spectrum characteristic value of the EEG signal is greater than the EEG characteristic threshold. If the power spectrum characteristic value of the extracted EEG signal is less than the EEG characteristic threshold, then return to step S210, and the user needs to look at the intelligent visual stimulator interface again. If the power spectrum characteristic value of the extracted EEG signal is greater than the EEG characteristic threshold, then execute step S220, and continue to perform classification and recognition processing to obtain more accurate EEG signal data. Due to the judgment action of the electrooculogram monitor based on the electrooculogram characteristic threshold factor in step S214, and the judgment action of the spectrum value monitor based on the EEG characteristic threshold in step S218, it is equivalent to providing double protection, which improves the signal-to-noise ratio of the EEG signal and the robustness of the system.

[0111] Step S220: the classification identifier classifies and identifies the stimulus source of the EEG signal to obtain a classification result of the EEG signal.

[0112] In step S222, the adaptive idle state monitor compares the classification result of the EEG signal with the set classification threshold. If the classification result of the EEG signal is less than the set classification threshold, the process returns to step S210, and the user needs to look at the intelligent visual stimulator interface again. If the classification result of the EEG signal is greater than or equal to the set classification threshold, the process proceeds to step S224.

[0113] Taking into account that when the user is not looking at the stimulus source in the visual stimulation interface or his line of sight is shifted, EEG signals may still be collected to trigger control instructions, affecting the control accuracy of the system. In view of this, the embodiment of the present application compares the classification result of the EEG signal with the classification threshold. If the classification result of the EEG signal is less than the classification threshold, it is deemed that the user is not controlling the system by looking at the visual stimulation interface. Therefore, the control instruction will not be triggered based on the classification result of the EEG signal. If the classification result of the EEG signal is greater than the classification threshold, the control instruction will be triggered based on the classification result of the EEG signal, thereby further improving the control accuracy of the system.

[0114] Step S224: If the adaptive idle state monitor determines that the classification result of the EEG signal is greater than or equal to the set classification threshold, the adaptive idle state monitor generates a control command and sends the control command to the control processor. Since the adaptive idle state monitor processes the classification result of the EEG signal based on the classification threshold, a more accurate final classification result (a classification result greater than or equal to the classification threshold) is obtained, thereby improving the accuracy of the system.

[0115] Step S226: The control processor controls the operation of the banking system based on the control command.

[0116] Step S228: The control processor feeds back the next interface information entered after the operation to the intelligent visual stimulator, and the intelligent visual stimulator generates a corresponding visual stimulation interface based on the next interface information.

[0117] Step S230, in the above-mentioned whole processing process, the portable signal collector collects the electromyographic signal in real time, and the signal processing module extracts the features of the collected electromyographic signal to obtain the electromyographic characteristic value. If the electromyographic characteristic value extracted is greater than the electromyographic characteristic threshold value, the monitoring result is fed back to the intelligent visual stimulator and the control processor. After the intelligent visual stimulator receives the monitoring result, the type of visual stimulation interface can be switched, for example, the visual stimulation interface of the type 1 currently presented is switched to the visual stimulation interface of the type 2, or the visual stimulation interface of the type 2 currently presented is switched to the visual stimulation interface of the type 1. After the control processor receives the monitoring result, the mode of the control instruction can be switched.

[0118] Current brain-computer interface systems usually present all stimulus sources in a visual stimulation interface. For example, the first stimulus source in the visual stimulation interface triggers a control command to the left, and the second stimulus source triggers a control command to enter the home page. However, under the premise that the stimulus sources that can be presented in a visual stimulation interface are limited, one stimulus frequency can only trigger one control command, which limits the number of available control commands. For example, a visual stimulation interface can present stimulus sources with m stimulus frequencies, but can only trigger m control commands. It can be seen that the control commands that the system can execute are limited and the business functions are relatively single. In the embodiment of the present application, the electromyographic signal is equivalent to a switch for switching the visual stimulation interface. For example, when the user clenches his teeth (clenching his teeth will allow the system to detect the electromyographic signal), the visual stimulation interface of type 1 is switched to the visual stimulation interface of type 2. Since all available stimulation sources are presented in two visual stimulation interfaces, and the stimulation sources in the two visual stimulation interfaces can share the stimulation frequency, more types of control instructions can be triggered. For example, a stimulation frequency of 6 Hz in the visual stimulation interface of type 1 triggers control instruction 1, while a stimulation frequency of 6 Hz in the visual stimulation interface of type 2 triggers control instruction 2, which is equivalent to multiple control instructions being able to share one stimulation frequency. For example, the visual stimulation interface of type 1 presents stimulation sources with m1 stimulation frequencies, and the visual stimulation interface of type 2 presents stimulation sources with m2 stimulation frequencies. The stimulation sources in the visual stimulation interface of type 1 and the stimulation sources in the visual stimulation interface of type 2 have the same stimulation frequency and correspond to different control instructions, which can trigger (m1+m2) control instructions, thereby expanding the control instructions that can be executed by the system and enriching the business functions of the system. In addition, compared to moving the line of sight in a visual stimulation interface to change the stimulus source to be triggered, switching the stimulus source to be triggered through muscle movement also reduces the user's visual fatigue.

[0119] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0120] A refers to B, which means that A is the same as B or A is a simple variant of B.

[0121] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0122] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A banking system interaction method based on a hybrid brain-computer interface, characterized in that: The method comprises: In the process of presenting a visual stimulation interface, collecting electroencephalogram signals and electrooculogram signals, wherein the visual stimulation interface includes a plurality of stimulation sources; If the electrooculogram characteristic value of the electrooculogram signal extracted within the electrooculogram signal acquisition time window is less than the electrooculogram characteristic threshold value, denoising the electrooculogram signal collected within the electrooculogram signal acquisition time window to obtain a denoised electrooculogram signal; Monitoring the spectrum value of the denoised EEG signal to obtain a power spectrum characteristic value of the denoised EEG signal; If the power spectrum eigenvalue is higher than the EEG characteristic threshold, performing stimulus source classification on the denoised EEG signal to obtain a classification result of the EEG signal; Based on the classification result of the EEG signal, the operation of the banking system is controlled.

2. The method according to claim 1, characterized in that The method further comprises: If the electrooculogram feature values ​​of multiple electrooculogram signals extracted within the electrooculogram signal acquisition time window are all greater than the electrooculogram feature threshold, further processing of the electrooculogram signal is canceled, and the user is guided to look at the visual stimulation interface again.

3. The method according to claim 1, characterized in that The visual stimulation interface includes a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface includes a plurality of first-type stimulation sources, the plurality of the first-type stimulation sources are used to trigger a first-type control instruction, and the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, the second visual stimulation interface includes a plurality of second-type stimulation sources, the plurality of the second-type stimulation sources are used to trigger a second-type control instruction, the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, and the stimulation frequencies of the plurality of the first-type stimulation sources are partially the same as or completely the same as the stimulation frequencies of the plurality of the second-type stimulation sources; The method further comprises: Collecting electromyographic signals, and performing feature extraction on the electromyographic signals to obtain electromyographic feature values; If the electromyographic characteristic value is greater than the electromyographic characteristic threshold, the intelligent visual stimulator is triggered to switch between the first visual stimulation interface and the second visual stimulation interface.

4. The method according to claim 1, characterized in that: The controlling and operating banking system based on the classification result of the EEG signal comprises: If the classification result of the EEG signal is greater than or equal to the classification threshold, based on the classification result of the EEG signal, controlling the operation of the banking system; the method further includes: If the classification result of the EEG signal is less than the classification threshold, it is determined that the classification result of the EEG signal is invalid.

5. The method according to claim 4, characterized in that The method further comprises: Performing correlation analysis on classification results of multiple EEG signals collected from the same user to obtain a maximum correlation coefficient value of the classification results of the multiple EEG signals; The classification threshold is adjusted based on an average value of maximum correlation coefficient values ​​of the classification results of the multiple electroencephalogram signals.

6. The method according to claim 1, characterized in that The monitoring of the spectrum value of the denoised EEG signal to obtain the power spectrum characteristic value of the denoised EEG signal includes: The denoised EEG signal is converted from the time domain to the frequency domain using a fast Fourier transform method to obtain a power spectrum characteristic value of the denoised EEG signal.

7. A banking system interaction device based on a hybrid brain-computer interface, characterized in that: The device comprises: A signal acquisition module, used to collect EEG signals and EOG signals during the presentation of a visual stimulation interface, wherein the visual stimulation interface includes a plurality of stimulation sources; A signal processing module, for, if the electrooculogram characteristic value of the electrooculogram signal extracted within the electrooculogram signal acquisition time window is less than the electrooculogram characteristic threshold, denoising the electrooculogram signal collected within the electrooculogram signal acquisition time window to obtain a denoised electrooculogram signal; monitoring the spectrum value of the denoised electrooculogram signal to obtain a power spectrum characteristic value of the denoised electrooculogram signal; and if the power spectrum characteristic value is higher than the electrooculogram characteristic threshold, classifying the denoised electrooculogram signal by stimulus source to obtain a classification result of the electrooculogram signal; The control processing module is used to control the operation of the banking system based on the classification result of the EEG signal.

8. The device according to claim 7, characterized in that The signal processing module is also used to cancel further processing of the EEG signal and guide the user to look at the visual stimulation interface again if the EEG feature values ​​of multiple EEG signals extracted within the EEG signal acquisition time window are all greater than the EEG feature threshold.

9. The device according to claim 7, characterized in that The visual stimulation interface includes a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface includes a plurality of first-type stimulation sources, the plurality of the first-type stimulation sources are used to trigger a first-type control instruction, and the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, the second visual stimulation interface includes a plurality of second-type stimulation sources, the plurality of the second-type stimulation sources are used to trigger a second-type control instruction, the plurality of the first-type stimulation sources have a plurality of stimulation frequencies, and the stimulation frequencies of the plurality of the first-type stimulation sources are partially the same as or completely the same as the stimulation frequencies of the plurality of the second-type stimulation sources; The signal acquisition module is further used to collect electromyographic signals and perform feature extraction on the electromyographic signals to obtain electromyographic feature values; The signal processing module is further used to trigger the intelligent visual stimulator to switch between the first visual stimulation interface and the second visual stimulation interface if the myoelectric characteristic value is greater than the myoelectric characteristic threshold.

10. The device according to claim 7, characterized in that The control processing module is used to control the operation of the banking system based on the classification result of the EEG signal if the classification result of the EEG signal is greater than or equal to the classification threshold; If the classification result of the EEG signal is less than the classification threshold, it is determined that the classification result of the EEG signal is invalid.

Images