A bank system interaction method and device based on a hybrid brain-computer interface

Abstract

The application discloses a bank system interaction method and device based on a hybrid brain-computer interface. The method comprises the following steps: collecting an electro-oculogram signal in real time; when it is monitored that an electro-oculogram characteristic value of the collected electro-oculogram signal is less than an electro-oculogram characteristic threshold value, it is indicated that the electroencephalogram signal has not been contaminated by the electro-oculogram signal, and the collected electroencephalogram signal is relatively accurate data, so that the electro-oculogram signal is further denoised, the denoised electroencephalogram signal is monitored in a frequency spectrum value, and a power spectrum characteristic value of the denoised electroencephalogram signal is obtained; if the power spectrum characteristic value of the denoised electroencephalogram signal is higher than an electroencephalogram characteristic threshold value, it is indicated that the electroencephalogram signal is effective, so that the electroencephalogram signal is further classified according to a stimulation source, and a bank system is controlled based on a classification result of the electroencephalogram signal. The application improves the robustness and control accuracy of the bank system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a bank system interaction method and device based on a hybrid brain-computer interface. BACKGROUND

[0002] With the continuous high-speed development of computer, sensor and information communication technologies, and the cross-fusion among the fields of mathematics, biology, psychology, neuroscience and cognitive science, the brain-computer interface technology is widely applied to many fields such as rehabilitation medicine, game entertainment, military application and home intelligence. However, different types of brain-computer interface systems (SSVEP, motor imagination and P300) still have problems such as low control accuracy and susceptibility to noise interference. SUMMARY

[0003] The present application provides a bank system interaction method and device based on a hybrid brain-computer interface. Since a double processing mode based on the power spectrum characteristic values of electroencephalogram signals and electrooculogram signals is adopted, the noise interference on the bank system based on the brain-computer interface is reduced, and the robustness and control accuracy of the system are improved.

[0004] In a first aspect, a bank system interaction method based on a hybrid brain-computer interface is provided, and the method comprises:

[0005] In the process of presenting a visual stimulation interface, electroencephalogram signals and electrooculogram signals are collected, and the visual stimulation interface comprises a plurality of stimulation sources;

[0006] If the electrooculogram characteristic value of the electrooculogram signal extracted in the electroencephalogram signal collection time window is less than an electrooculogram characteristic threshold value, the electroencephalogram signal collected in the electroencephalogram signal collection time window is subjected to denoising processing to obtain a denoised electroencephalogram signal;

[0007] The denoised electroencephalogram signal is subjected to frequency spectrum value monitoring to obtain a power spectrum characteristic value of the denoised electroencephalogram signal;

[0008] If the power spectrum characteristic value is higher than an electroencephalogram characteristic threshold value, the denoised electroencephalogram signal is subjected to stimulation source classification to obtain a classification result of the electroencephalogram signal;

[0009] Based on the classification result of the electroencephalogram signal, a bank system is controlled.

[0010] In some embodiments, the method further comprises:

[0011] If the electrooculogram characteristic values of the multiple electrooculogram signals extracted in the electroencephalogram signal collection time window are all greater than the electrooculogram characteristic threshold value, the electroencephalogram signal is not subjected to further processing, and the user is guided to re-fixate the visual stimulation interface.

[0012] In some embodiments, the visual stimulation interface comprises a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface comprises a plurality of first type stimulation sources, the plurality of first type stimulation sources are used to trigger first type control instructions, the plurality of first type stimulation sources have a plurality of stimulation frequencies, the second visual stimulation interface comprises a plurality of second type stimulation sources, the plurality of second type stimulation sources are used to trigger second type control instructions, the plurality of first type stimulation sources have a plurality of stimulation frequencies, the stimulation frequencies of the plurality of first type stimulation sources are partially or entirely the same as the stimulation frequencies of the plurality of second type stimulation sources.

[0013] The method further comprises:

[0014] Collecting an electromyogram signal, and performing feature extraction on the electromyogram signal to obtain an electromyogram feature value;

[0015] If the electromyogram feature value is greater than an electromyogram feature threshold value, triggering the intelligent visual stimulator to switch between the first visual stimulation interface and the second visual stimulation interface.

[0016] In some embodiments, based on the classification result of the electroencephalogram signal, controlling the operation of the bank system comprises:

[0017] If the classification result of the electroencephalogram signal is greater than or equal to a classification threshold value, based on the classification result of the electroencephalogram signal, controlling the operation of the bank system; the method further comprises:

[0018] If the classification result of the electroencephalogram signal is less than the classification threshold value, determining that the classification result of the electroencephalogram signal is invalid.

[0019] In some embodiments, the method further comprises:

[0020] Performing correlation analysis on the classification results of a plurality of electroencephalogram signals collected from the same user to obtain a maximum correlation coefficient value of the classification results of the plurality of electroencephalogram signals;

[0021] Based on the average value of the maximum correlation coefficient values of the classification results of the plurality of electroencephalogram signals, adjusting the classification threshold value.

[0022] In some embodiments, the monitoring of the frequency spectrum value of the denoised electroencephalogram signal to obtain the power spectrum feature value of the denoised electroencephalogram signal comprises:

[0023] Using the fast Fourier transform method to convert the denoised electroencephalogram signal from the time domain to the frequency domain to obtain the power spectrum feature value of the denoised electroencephalogram signal.

[0024] In a second aspect, a bank system interaction device based on a hybrid brain-computer interface is provided, the device comprising:

[0025] a signal collection module, configured to collect electroencephalogram signals and electrooculogram signals in a process of presenting a visual stimulation interface, the visual stimulation interface including a plurality of stimulation sources;

[0026] a signal processing module, configured to, if an electrooculogram feature value of the electrooculogram signals extracted in an electroencephalogram signal collection time window is less than an electrooculogram feature threshold value, perform denoising processing on the electroencephalogram signals collected in the electroencephalogram signal collection time window to obtain denoised electroencephalogram signals, perform frequency spectrum value monitoring on the denoised electroencephalogram signals to obtain a power spectrum feature value of the denoised electroencephalogram signals, and if the power spectrum feature value is higher than an electroencephalogram feature threshold value, perform stimulation source classification on the denoised electroencephalogram signals to obtain a classification result of the electroencephalogram signals;

[0027] a control processing module, configured to control an operation bank system based on the classification result of the electroencephalogram signals.

[0028] In some embodiments, the signal processing module is further configured to, if the electrooculogram feature values of a plurality of electrooculogram signals extracted in the electroencephalogram signal collection time window are all greater than the electrooculogram feature threshold value, cancel further processing on the electroencephalogram signals and guide a user to re-fixate the visual stimulation interface.

[0029] In some embodiments, the visual stimulation interface includes a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface includes a plurality of first-type stimulation sources, the plurality of first-type stimulation sources are configured to trigger first-type control instructions, the plurality of first-type stimulation sources have a plurality of stimulation frequencies, the second visual stimulation interface includes a plurality of second-type stimulation sources, the plurality of second-type stimulation sources are configured to trigger second-type control instructions, the plurality of first-type stimulation sources have a plurality of stimulation frequencies, and the stimulation frequencies of the plurality of first-type stimulation sources are partially or entirely the same as the stimulation frequencies of the plurality of second-type stimulation sources.

[0030] The signal collection module is further configured to collect electromyogram signals and perform feature extraction on the electromyogram signals to obtain electromyogram feature values.

[0031] The signal processing module is further configured to, if the electromyogram feature values are greater than an electromyogram feature threshold value, trigger the intelligent visual stimulator to switch between the first visual stimulation interface and the second visual stimulation interface.

[0032] In some embodiments, the control processing module is configured to, if the classification result of the electroencephalogram signals is greater than or equal to a classification threshold value, control the operation bank system based on the classification result of the electroencephalogram signals, and if the classification result of the electroencephalogram signals is less than the classification threshold value, determine that the classification result of the electroencephalogram signals is invalid.

[0033] In some embodiments, the signal processing module is further configured to perform correlation analysis on the classification results of multiple electroencephalogram signals collected from the same user to obtain a maximum correlation coefficient value of the classification results of the multiple electroencephalogram signals; and adjust the classification threshold based on an average value of the maximum correlation coefficient values of the classification results of the multiple electroencephalogram signals.

[0034] In some embodiments, the signal processing module is configured to convert the denoised electroencephalogram signal from time domain to frequency domain by using a fast Fourier transform method to obtain a power spectrum feature value of the denoised electroencephalogram signal.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] The embodiments of the present application collect electrooculogram signals in real time. When it is monitored that the electrooculogram feature value of the collected electrooculogram signal is less than the electrooculogram feature threshold, it indicates that the electroencephalogram signal has not been contaminated by the electrooculogram signal, and the collected electroencephalogram signal is relatively accurate data. Therefore, the electrooculogram signal is further denoised, and the frequency spectrum value of the denoised electroencephalogram signal is monitored to obtain the power spectrum feature value of the denoised electroencephalogram signal. If the power spectrum feature value of the denoised electroencephalogram signal is higher than the electroencephalogram feature threshold, it indicates that the electroencephalogram signal is effective. The electroencephalogram signal is further classified according to the stimulation source, and the bank system is controlled based on the classification result of the electroencephalogram signal. Since the electrooculogram signal and the electrooculogram feature threshold are used for judgment, and the power spectrum feature value of the electroencephalogram signal and the electroencephalogram feature threshold are used for judgment, it is equivalent to using a double judgment processing mode to check before controlling the bank system based on the electroencephalogram signal. The bank system is controlled by using more accurate electroencephalogram signals, thereby reducing the adverse effects on the robustness of the bank system caused by controlling the bank system based on the electroencephalogram signal contaminated by the electrooculogram signal or the electroencephalogram signal with a power spectrum feature value that does not meet the requirements. Therefore, the robustness and control accuracy of the bank system are improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 FIG. 1 is a general architecture diagram of a bank system interaction device based on a hybrid brain-computer interface provided by the embodiments of the present application;

[0038] Figure 2 FIG. 6 is a block diagram of a signal collection module provided by the embodiments of the present application;

[0039] Figure 3a FIG. 11 is a schematic diagram of a visual stimulation interface of type 1 provided by the embodiments of the present application;

[0040] Figure 3b FIG. 12 is a schematic diagram of a visual stimulation interface of type 2 provided by the embodiments of the present application;

[0041] Figure 4is an architecture diagram of a signal processing module provided by an embodiment of the present application.

[0042] Figure 5 is an architecture diagram of a control processing module provided by an embodiment of the present application.

[0043] Figure 6 is a flowchart of a bank system interaction method based on a hybrid brain computer interface provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0045] Some terms and concepts related to the embodiments of the present application are explained below.

[0046] Brain computer interface (BCI): a communication mode that can directly connect the brain and the external environment without relying on peripheral nerves and muscles.

[0047] Hybrid brain computer interface (Hybrid BCI): a communication mode that establishes a brain computer interface that integrates different types of biological signals.

[0048] Electroencephalogram (EEG): the potential change recorded from the scalp of a human or animal, which is a non-invasive brain signal acquisition technology.

[0049] Electromyography (EMG): the superposition of motor unit action potentials in a large number of muscle fibers in time and space, which is usually the electrical activity on the surface of the muscle.

[0050] Electrooculography (EOG): different eye movements, such as blinking and eye movement, will cause depolarization and hyperpolarization between the retina and the cornea, forming a potential difference between the retina and the cornea.

[0051] Steady state visual evoked potential (SSVEP): when a subject is subjected to a certain frequency of visual stimulation, continuous visual evoked potentials corresponding to the stimulation frequency will be generated in the occipital lobe of the brain.

[0052] The application scenarios related to the embodiments of the present application are introduced below.

[0053] The current SSVEP-based brain-computer interface (SSVEP-BCI) interaction method is susceptible to the user's non-gaze at the visual stimulator or the shift of the gaze target, and the electroencephalogram signals of different users are affected differently by the SSVEP, resulting in differences in the classification and recognition results of different users under the same visual frequency stimulation. In addition, most SSVEP-BCI systems have simple visual stimulators, and therefore cannot meet the individualized adaptation of users, and the system does not have ease of use.

[0054] The current hybrid brain-computer interface interaction method can process more complex commands and improve the flexibility of the system through the fusion processing of multiple signals. However, due to the different types of signals used by different hybrid brain-computer interface systems, different electroencephalogram paradigms, different control strategies, and many other reasons, the robustness of the system cannot be guaranteed.

[0055] The current bank-based brain-computer interface interaction method expands the application field of the brain-computer interface technology and realizes the brain-computer interface control of the banking scene device, but the current system has many problems such as low accuracy of electroencephalogram signals, single control strategy, and inflexible system.

[0056] To solve the above problems, the embodiment of the present application provides a bank system interaction method and device based on a hybrid brain-computer interface. By establishing a direct information exchange channel between the brain and the external device without relying on the peripheral nerves and muscles, the embodiment helps the disabled people with stroke, spinal cord injury and other serious diseases to realize real-time control of the bank system, effectively improves the self-care ability and life quality of the disabled people, and improves the ease of use and portability of the system. In addition, the embodiment improves the accuracy of the electroencephalogram signals, thereby improving the robustness and control accuracy of the brain-computer interaction system, reducing the user's interaction burden, and thus improving the user experience.

[0057] The device provided by the embodiment of the present application includes a signal acquisition module, a signal processing module and a control processing module. In the signal acquisition module, the acquisition and transmission of electroencephalogram signals, electromyogram signals and electrooculogram signals are completed by a portable signal acquisition device. After wearing the portable signal acquisition device, the user completes the real-time acquisition of the electroencephalogram signals by gazing at a certain frequency of the stimulation source in the interface of the intelligent visual stimulator. The signal processing module includes a pre-signal processing module, a feature extraction unit and a classification and recognition unit.

[0058] The pre-signal processing module performs denoising processing on the electroencephalogram signal by wavelet analysis. The feature extraction unit completes feature extraction of the electroencephalogram signal, the electromyogram signal and the electrooculogram signal through the electrooculogram monitor, the electromyogram selector and the spectrum value monitor respectively. If the electrooculogram monitor extracts multiple electrooculogram features in the electroencephalogram signal collection time window, it is determined that the electroencephalogram signal is contaminated, and the electroencephalogram signal is no longer processed further. If the electromyogram selector extracts electromyogram features that can trigger the switching of the intelligent visual stimulator interface, the spectrum value monitor extracts a feature value of the electroencephalogram signal higher than a predetermined electroencephalogram feature value, it is determined that the electroencephalogram signal is valid, and the electroencephalogram signal is classified and recognized. In the classification and recognition unit, the classification recognizer classifies and recognizes the electroencephalogram signal by using the canonical correlation analysis method, and the adaptive idle state monitor rejudges the classification result. In the control processing module, the electroencephalogram signal intention is converted into a control strategy of direct control or shared control, different business scenarios in the bank system are processed, and the usability and portability of the system are improved.

[0059] Figure 1 is a general architecture diagram of a bank system interaction device based on a hybrid brain-computer interface provided by an embodiment of the present application. The device includes a signal acquisition module, a signal processing module and a control processing module.

[0060] The signal acquisition module includes a portable electroencephalogram collector and an intelligent visual stimulator. The signal processing module includes a pre-signal processing module, a feature extraction unit and a classification and recognition unit. The feature extraction unit includes an electrooculogram monitor, an electromyogram monitor and a spectrum value monitor, the classification and recognition unit includes a classification recognizer and an adaptive idle state monitor, and the control processing module includes a control processor, an external device (a bank system) and a feedback processor.

[0061] Figure 2 is a block diagram of the signal acquisition module provided by an embodiment of the present application. The signal acquisition module is mainly used for acquisition and transmission of electroencephalogram signals, electromyogram signals and electrooculogram signals. Specifically, after a user wears the portable signal acquisition module, the user gazes at a certain frequency of the stimulus source in the visual stimulation interface presented by the intelligent visual stimulator. The signal acquisition module completes real-time acquisition of the electroencephalogram signals in the occipital region of the brain, the electromyogram signals and the electrooculogram signals in the process that the user gazes at the stimulus source, and transmits the acquired electroencephalogram signals in the occipital region of the brain, the electromyogram signals and the electrooculogram signals to the signal processing module through Bluetooth.

[0062] The signal acquisition module includes a signal acquisition module and an intelligent visual stimulator. The intelligent visual stimulator is used to present two types of visual stimulation interfaces, which are referred to as a type 1 visual stimulation interface (also referred to as a first visual stimulation interface) and a type 2 visual stimulation interface (also referred to as a second visual stimulation interface) respectively. The type 1 visual stimulation interface is as shown in Figure 3aAs shown, the visual stimulus interface of type 2 is as follows: Figure 3b As shown.

[0063] Type 1 visual stimulus interfaces include multiple Type 1 stimulus sources, while Type 2 visual stimulus interfaces include multiple Type 2 stimulus sources. Type 1 stimulus sources are used to trigger Type 1 control commands (or execute Type 1 business functions, such as moving up or down). Type 2 stimulus sources are used to trigger Type 2 control commands (or execute Type 2 business functions, such as accessing a specific interface of a banking system). The stimulation frequencies of the multiple Type 1 stimulus sources and the multiple Type 2 stimulus sources partially overlap or are exactly the same. For example, Type 1 stimulus sources include a stimulus source with stimulation frequency k and a stimulus frequency j, while Type 2 stimulus sources also include a stimulus source with stimulation frequency k and a stimulus frequency j. In other words, Type 1 and Type 2 visual stimulus interfaces have stimulus sources with the same stimulation frequency but different functions.

[0064] For example, the Type 1 visual stimulation interface includes eight stimulation sources, and the corresponding control commands for the eight stimulation sources are: move up, move down, move left, move right, swipe up, swipe down, confirm click, and return to the previous page. The eight stimulation sources have eight different stimulation frequencies. For instance, a stimulation source with a frequency of 6Hz triggers the move up control command, a stimulation source with a frequency of 7Hz triggers the move down control command, a stimulation source with a frequency of 8Hz triggers the move left control command, a stimulation source with a frequency of 9Hz triggers the move right control command, a stimulation source with a frequency of 10Hz triggers the swipe up control command, a stimulation source with a frequency of 11Hz triggers the swipe down control command, a stimulation source with a frequency of 12Hz triggers the confirm click control command, and a stimulation source with a frequency of 13Hz triggers the return to the previous page control command.

[0065] Table 1 shows the correspondence between the stimulation frequency and control commands of each stimulus source in the Type 1 visual stimulus interface. Table 1 is for illustrative purposes only, and the basic functions of the stimulus source settings, the flashing frequencies corresponding to the basic functions, and the corresponding control commands are not limited to the examples above.

[0066] Table 1. Visual stimulus interface stimulation frequency and corresponding instructions for Type 1

[0067]

[0068]

[0069] The number of stimulation sources in the visual stimulation interface of type 2 is not fixed, and the functions represented by the stimulation sources in the bank system are also not fixed. For example, when entering the homepage of the bank system, the feedback processor in the control processing module will feed back the core functions involved in the page to the intelligent visual stimulator, and the intelligent visual stimulator generates a visual stimulation interface corresponding to the core functions, realizing the real-time linkage between the intelligent visual stimulator and the bank system. For example, a stimulation frequency of 6 Hz represents my account, a stimulation frequency of 7 Hz represents transfer, a stimulation frequency of 7.5 Hz represents popular activities, a stimulation frequency of 8 Hz represents income and expenditure, a stimulation frequency of 8.57 Hz represents small bean park, a stimulation frequency of 9 Hz represents scanning, a stimulation frequency of 10 Hz represents credit card, a stimulation frequency of 11 Hz represents deposit, a stimulation frequency of 12 Hz represents financial products, a stimulation frequency of 13 Hz represents loan, a stimulation frequency of 13.25 Hz represents life payment, and a stimulation frequency of 13.5 Hz represents city special zone. Table 2 shows the stimulation frequency and corresponding control instruction of the visual stimulation interface of type 2. The example shown in Table 2 is only illustrative, and the business modules included in each visual stimulation interface, the flicker frequency corresponding to the business modules, and the corresponding control instructions are not limited to the above examples.

[0070] Table 2 Stimulation frequency and corresponding control instruction of visual stimulation interface of type 2

[0071] Stimulation frequency (Hz) Control instruction 6 Click to enter my account 7 Click to enter transfer 7.5 Click to enter popular activities 8 Click to enter income and expenditure 8.57 Click to enter bean park 9 Click to enter scan to pay 10 Click to enter credit card 11 Click to enter deposit 12 Click to enter financial products 13 Click to enter loans 13.25 Click to enter life payment 13.5 Click to enter city special area

[0072] Figure 4 It is an architecture diagram of a signal processing module provided by the embodiment of the application. The signal processing module is mainly used for pre-processing, feature processing and classification recognition of electroencephalogram signals, and feature processing of electromyogram signals and feature processing of electrooculogram signals. The signal processing module includes a pre-signal processing module, a feature signal processing module and a classification recognition unit.

[0073] The pre-signal processing module completes the denoising processing of the collected electroencephalogram signals by wavelet analysis method. The denoising processing includes removing electrocardiogram, electrooculogram, power frequency artifact and environmental noise, and obtaining 0.01-32 Hz electroencephalogram signals after denoising.

[0074] The feature extraction unit includes an electrooculogram monitor, an electromyogram selector and a spectrum value monitor. The electrooculogram monitor is used to complete the feature extraction of the electrooculogram signal, the electromyogram selector is used to complete the feature extraction of the electromyogram signal, and the spectrum value monitor is used to complete the feature extraction of the electroencephalogram signal.

[0075] The electro-oculogram monitor is used for real-time acquisition of electro-oculogram signals, and pre-processes the acquired electro-oculogram signals through wavelet transform to obtain electro-oculogram characteristic values of the electro-oculogram signals. If the electro-oculogram characteristic values of multiple electro-oculogram signals (such as eye blinking or saccade) extracted in a brain electrical signal acquisition time window are greater than a set electro-oculogram characteristic threshold (for example, the user frequently blinks or frequently saccades), the electro-oculogram monitor cancels further processing of the brain electrical signals. Specifically, considering that the electro-oculogram signals and the brain electrical signals are extracted in parallel, the potential difference of the electro-oculogram signals is much larger than the potential difference of the brain electrical signals, and if the user blinks when the brain electrical signals are acquired, the accuracy of the acquired brain electrical signals will be affected, which is equivalent to the brain electrical signals being contaminated. Therefore, if the electro-oculogram monitor determines that the electro-oculogram characteristic values of multiple electro-oculogram signals extracted in a brain electrical signal acquisition time window are greater than a set electro-oculogram characteristic threshold, the electro-oculogram monitor does not need to further process the brain electrical signals, thereby avoiding unnecessary increase in calculation amount and processing resources caused by further processing of the brain electrical signals that have been contaminated.

[0076] The formula based on the electro-oculogram monitor is as follows:

[0077]

[0078] wherein θ is the electro-oculogram characteristic value, EOG H is the number of monitored saccadic electro-oculograms, EOG V is the number of monitored blinking electro-oculograms, and N is the number of brain electrical signal acquisition time windows.

[0079] The electromyogram selector is used for real-time acquisition of electromyogram signals, and denoises the acquired electromyogram signals through wavelet transform, and extracts features of the denoised electromyogram signals using a root mean square algorithm (RMS) to obtain electromyogram characteristic values. If the extracted electromyogram characteristic values are higher than a set electromyogram characteristic threshold, the visual stimulation interface presented in the intelligent visual stimulator is triggered to select switching between the type 1 visual stimulation interface and the type 2 visual stimulation interface.

[0080] Since the frequency characteristics of the SSVEP signal are very obvious, the spectral value monitor converts the denoised electroencephalogram signal from the time domain to the frequency domain by using the fast Fourier transform method, so as to obtain the power spectrum characteristic value of the denoised electroencephalogram signal. If the power spectrum characteristic value of the denoised electroencephalogram signal is higher than the set electroencephalogram characteristic threshold value, the spectral value monitor predicts that the electroencephalogram signal is effective, and continues to stimulate the electroencephalogram characteristic threshold value to obtain the classification result of the electroencephalogram signal. The classification result of the electroencephalogram signal is, for example, the stimulation frequency of the stimulation source corresponding to the electroencephalogram signal, and the classification result can represent which frequency of the stimulation source in the visual stimulation interface is the electroencephalogram signal generated by the user's gaze. Since the electroencephalogram signal generated by the stimulation source gazed by the user is usually the signal with the highest characteristic value in the power spectrum, the electroencephalogram signal generated by the stimulation source not gazed by the user can be filtered out based on the electroencephalogram characteristic threshold value, and the calculation amount caused by the classification of the electroencephalogram signal is reduced.

[0081] The formula based on the spectral value monitor is as follows:

[0082]

[0083] Wherein x(n) is the EEG signal, and the signal length is limited, W N -j2π / N The rotation factor is represented by e

[0084] After the spectrum X(k) is calculated by the FFT, the power spectrum estimate P(k) of x(n) is calculated according to the power spectrum density analysis theorem, and the formula is as follows:

[0085]

[0086] The classification and recognition unit includes a classification and recognition unit and an adaptive idle state monitor. The classification and recognition unit classifies and recognizes the stimulation source of the denoised electroencephalogram signal to obtain the classification result of the electroencephalogram signal.

[0087] Then, the adaptive idle state monitor compares the classification result of the electroencephalogram signal with the classification threshold value. If the classification result of the electroencephalogram signal reaches the classification threshold value, the adaptive idle state monitor determines that the classification result of the electroencephalogram signal is effective, and the adaptive idle state monitor transmits the classification result to the control processing module. If the classification result of the electroencephalogram signal does not reach the classification threshold value, the adaptive idle state monitor determines that the classification result of the electroencephalogram signal is invalid, and the adaptive idle state monitor will not transmit the classification result of the electroencephalogram signal to the control processing module, so as to prevent the occurrence of incorrect control operation.

[0088] The recursive relationship based on the classification and recognition unit is as follows:

[0089] Suppose there are two groups of multidimensional random variables X and Y, and X∈R H×J ​Y∈R I×J Typical correlation analysis expects to seek a pair of linear transformations w X ∈R H×1 and w Y ∈R I×1 , such that the linear combination x = w X T X and y = w Y T Y between the correlation reaches the maximum, the formula is as follows:

[0090]

[0091] Where, C xx = XX T and C yy = YY T respectively represent the two groups of multi-dimensional random variables X and Y within-class covariance matrix, C xy = XY T represent the two groups of multi-dimensional random variables X and Y between-class covariance matrix. X and y are projected on w X and w Y , the maximum value of p represents the maximum canonical correlation coefficient.

[0092] If the feedback processor in the control processing module feeds back i functions to the intelligent visual stimulator, the intelligent visual stimulator generates i stimulus sources, X is the collected n-channel N s electroencephalogram data, and the corresponding reference signal Y i The formula is defined as follows:

[0093]

[0094] Where, N represents the number of sampling points, S represents the sampling rate, N h represents the harmonic number of the signal.

[0095] Therefore, the maximum correlation coefficient of the electroencephalogram signal X and the electroencephalogram signal Y i can be obtained.

[0096] The adaptive idle state monitor is used to monitor whether the classification result of the electroencephalogram signal reaches a set classification threshold. If the classification result of the electroencephalogram signal is greater than or equal to the set classification threshold, it is determined that the classification result of the electroencephalogram signal is valid, and if the classification result of the electroencephalogram signal does not reach the set classification threshold, it is determined that the classification result of the electroencephalogram signal is invalid. In addition, because the electroencephalogram signals of different users are affected differently by the SSVEP, the maximum correlation coefficient values of the classification recognition of different users under the same visual frequency stimulation also differ, so the adaptive idle state monitor sets a default initial classification threshold i. In the process of real-time operation of the banking system, the adaptive idle state monitor performs correlation analysis on the classification results of multiple electroencephalogram signals collected from the same user to obtain the maximum correlation coefficient values of the classification results of the multiple electroencephalogram signals. The adaptive idle state monitor takes the average of the maximum correlation coefficient values of the classification results of the multiple electroencephalogram signals, adjusts the initial classification threshold based on the average to obtain an adjusted classification threshold, and judges whether the classification result of a subsequent electroencephalogram signal is valid based on the adjusted classification threshold.

[0097] Because the classification threshold is adjusted based on the maximum correlation coefficient values of the classification results of the multiple electroencephalogram signals collected from the same user, different classification thresholds can be used for different users, and the classification threshold can also be dynamically adjusted, thereby improving the flexibility of the system.

[0098] The formula based on the adaptive idle state monitor is as follows:

[0099]

[0100] The adaptive idle state monitor proposed in the embodiments of the present application adjusts the classification threshold based on the maximum correlation coefficient values of the classification results of the multiple electroencephalogram signals of the same user, so that adaptive idle state thresholds can be set for different users, and the classification result is judged to be valid or invalid based on the classification threshold, and the operation system is controlled by using the valid classification result, thereby improving the accuracy of the system.

[0101] Figure 5 is an architectural schematic diagram of a control processing module provided by the embodiments of the present application. The control processing module mainly includes a control processor, an external device (a banking system), and a feedback processor. The control processor converts the classification result of the electroencephalogram signal output by the classification recognizer in the signal processing module into a control command, which is a direct control command or a shared control command, and operates the relevant scenes of the banking system based on the control command.

[0102] The feedback processor feeds back the recognized current basic function to the intelligent visual stimulator, so as to facilitate the intelligent visual stimulator to generate a visual stimulation interface corresponding thereto.

[0103] Because the classification threshold is adjusted based on the maximum correlation coefficient values of the classification results of the multiple electroencephalogram signals collected from the same user, different classification thresholds can be used for different users, and the classification threshold can also be dynamically adjusted, thereby improving the flexibility of the system.Figure 1 The architecture shown utilizes the processing mode of the fusion of the intelligent visual stimulator, the feedback processor, the electromyographic selector and the control processor, realizes the real-time linkage of the visual stimulation paradigm and the bank system, forms a control strategy with a feedback linkage mechanism, reduces the cognitive pressure and the interaction burden of the user, and improves the user experience.

[0104] Figure 6 is a flowchart of a bank system interaction method based on a hybrid brain-computer interface provided by the embodiment of the present application. As shown in the figure, the method comprises the following steps. Figure 6

[0105] Step S210, after the user wears the portable signal collector, the intelligent visual stimulator presents a visual stimulation interface, and the user gazes at the visual stimulation interface presented by the intelligent visual stimulator. The visual stimulation interface comprises a plurality of frequency stimulation sources. The stimulation source is, for example, a light source that flashes at a certain frequency. The visual stimulation interface can be a type 1 visual stimulation interface or a type 2 visual stimulation interface.

[0106] Step S212, the portable signal collector completes the real-time collection of the brain electrical signals in the occipital region, the electromyographic signals and the electrooculographic signals, and transmits the collected brain electrical signals in the occipital region, the electromyographic signals and the electrooculographic signals to the signal processing module through Bluetooth.

[0107] In a possible implementation manner, the portable signal collector comprises a plurality of electrode pieces, the plurality of electrode pieces are connected to the occipital lobe, the eyes and the mouth of the user respectively, the electrode pieces connected to the occipital lobe collect the brain electrical signals in the occipital region, the electrode pieces connected to the mouth collect the electromyographic signals, and the electrode pieces connected to the eyes collect the electrooculographic signals.

[0108] Step S214, the electrooculographic monitor in the signal processing module extracts features from the electrooculographic signals collected in a certain brain electrical signal collection time window to obtain electrooculographic feature values of the electrooculographic signals. The electrooculographic monitor judges whether the electrooculographic feature values of the electrooculographic signals exceed an electrooculographic feature threshold value. If the electrooculographic feature values of the electrooculographic signals collected in a brain electrical signal collection time window are less than an electrooculographic feature threshold value factor, the electrooculographic monitor executes step S216 on the brain electrical signals collected in the brain electrical signal collection time window. If the electrooculographic feature values of the electrooculographic signals detected in a brain electrical signal collection time window are greater than the electrooculographic feature threshold value, the electrooculographic monitor determines that the brain electrical signals collected in the brain electrical signal collection time window are invalid signals, and thus returns to step S210 for the brain electrical signals, and the user needs to gaze at the visual stimulation interface presented in the intelligent visual stimulator again.

[0109] Step S216, the pre-signal processing module performs denoising processing on the brain electrical signals collected in the brain electrical signal collection time window to obtain denoised brain electrical signals.​

[0110] Step S218, the spectrum value monitor monitors the denoised electroencephalogram to extract the power spectrum characteristic value of the denoised electroencephalogram. The spectrum value monitor determines whether the power spectrum characteristic value of the electroencephalogram is greater than the electroencephalogram characteristic threshold. If the extracted power spectrum characteristic value of the electroencephalogram is less than the electroencephalogram characteristic threshold, return to step S210, and the user needs to re-fix the smart visual stimulator interface. If the extracted power spectrum characteristic value of the electroencephalogram is greater than the electroencephalogram characteristic threshold, execute step S220, and more accurate electroencephalogram data is obtained by continuing the classification and recognition process. Due to the judgment action of the electrooculogram monitor based on the electrooculogram characteristic threshold factor in step S214 and the judgment action of the spectrum value monitor based on the electroencephalogram characteristic threshold in step S218, double protection is provided, and the signal-to-noise ratio of the electroencephalogram and the robustness of the system are improved.

[0111] Step S220, the classification recognizer classifies and identifies the electroencephalogram to obtain the classification result of the electroencephalogram.

[0112] Step S222, the adaptive idle state monitor compares the classification result of the electroencephalogram with the set classification threshold. If the classification result of the electroencephalogram is less than the set classification threshold, return to step S210, and the user needs to re-fix the smart visual stimulator interface. If the classification result of the electroencephalogram is greater than or equal to the set classification threshold, execute step S224.

[0113] Considering that in the case where the user does not fix the stimulus source in the visual stimulation interface or the line of sight is diverted, the electroencephalogram may still be collected to trigger the control instruction, affecting the control accuracy of the system. Therefore, based on the comparison between the classification threshold and the classification result of the electroencephalogram, if the classification result of the electroencephalogram is less than the classification threshold, it is considered that the user does not control the system by fixing the visual stimulation interface, and therefore the control instruction is not triggered based on the classification result of the electroencephalogram. If the classification result of the electroencephalogram is greater than the classification threshold, the control instruction is triggered based on the classification result of the electroencephalogram, thereby further improving the control accuracy of the system.

[0114] Step S224, if the adaptive idle state monitor determines that the classification result of the electroencephalogram is greater than or equal to the set classification threshold, the adaptive idle state monitor generates a control command and sends the control command to the control processor. Since the adaptive idle state monitor processes the classification result of the electroencephalogram based on the classification threshold, a more accurate final classification result (greater than or equal to the classification threshold) is obtained, thereby improving the accuracy of the system.

[0115] Step S226, the control processor controls the operation of the bank system based on the control command.

[0116] Step S228, the control processor feeds back the next interface information entered after the operation to the intelligent visual stimulator, and the intelligent visual stimulator generates a corresponding visual stimulation interface based on the next interface information.

[0117] Step S230, during the entire processing process, the portable signal collector collects the electromyographic signals in real time, and the signal processing module extracts the electromyographic features from the collected electromyographic signals to obtain electromyographic feature values. If the extracted electromyographic feature values are greater than the electromyographic feature threshold, the monitoring results are fed back to the intelligent visual stimulator and the control processor. After receiving the monitoring results, the intelligent visual stimulator switches the type of the visual stimulation interface, for example, switches the currently presented type 1 visual stimulation interface to a type 2 visual stimulation interface, or switches the currently presented type 2 visual stimulation interface to a type 1 visual stimulation interface. After receiving the monitoring results, the control processor switches the mode of the control instructions.

[0118] The current brain-computer interface system usually presents all stimulation sources in one visual stimulation interface, for example, the first stimulation source in the visual stimulation interface triggers a left control instruction, and the second stimulation source triggers a control instruction to enter the home page. However, under the premise that the stimulation sources that can be presented in one visual stimulation interface are limited, one stimulation frequency can only trigger one control instruction, which limits the number of available control instructions. For example, if a visual stimulation interface can present m stimulation sources of stimulation frequency, it can only trigger m control instructions. It can be seen that the system can execute limited control instructions and the business function is relatively single. In the embodiments of the present application, the electromyographic signal is equivalent to a switch for switching the visual stimulation interface. For example, when the user clenches his teeth (clenching teeth will cause the system to detect electromyographic signals), the visual stimulation interface is switched from type 1 to type 2. Since all available stimulation sources are presented in two visual stimulation interfaces, and the stimulation sources in the two visual stimulation interfaces can share the stimulation frequency, more types of control instructions can be triggered. For example, the 6Hz stimulation frequency in the type 1 visual stimulation interface triggers control instruction 1, and the 6Hz stimulation frequency in the type 2 visual stimulation interface triggers control instruction 2, which is equivalent to multiple control instructions sharing one stimulation frequency. For example, the type 1 visual stimulation interface presents m1 stimulation sources of stimulation frequency, and the type 2 visual stimulation interface presents m2 stimulation sources of stimulation frequency. The stimulation sources in the type 1 visual stimulation interface and the stimulation sources in the type 2 visual stimulation interface have the same stimulation frequency and correspond to different control instructions, and can trigger (m1+m2) control instructions, thereby expanding the control instructions that the system can execute and making the business function of the system more abundant. In addition, compared with moving the line of sight in one visual stimulation interface to change the stimulation source to be triggered, switching the stimulation source to be triggered through muscle movement also reduces the visual fatigue of the user.

[0119] The various embodiments described in the specification are progressive in nature, and identical or similar parts among the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments.

[0120] A refers to B means that A is identical to B or A is a simple transformation of B.

[0121] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)), etc.

[0122] The above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A bank system interaction method based on a hybrid brain-computer interface, characterized by, The method comprises: In the process of presenting a visual stimulation interface, collecting electroencephalogram signals and electrooculogram signals, the visual stimulation interface comprising a plurality of stimulation sources; If the electrooculogram feature value of the electrooculogram signal extracted in the electroencephalogram signal collection time window is less than the electrooculogram feature threshold value, performing denoising processing on the electroencephalogram signal collected in the electroencephalogram signal collection time window to obtain a denoised electroencephalogram signal; Performing spectrum value monitoring on the denoised electroencephalogram signal to obtain a power spectrum feature value of the denoised electroencephalogram signal; If the power spectrum feature value is higher than the electroencephalogram feature threshold value, performing stimulation source classification on the denoised electroencephalogram signal to obtain a classification result of the electroencephalogram signal; If the classification result of the electroencephalogram signal is greater than or equal to a classification threshold value, controlling an operation bank system based on the classification result of the electroencephalogram signal; If the classification result of the electroencephalogram signal is less than the classification threshold value, determining that the classification result of the electroencephalogram signal is invalid.

2. The method of claim 1, wherein, The method further comprises: If the electrooculogram feature values of a plurality of electrooculogram signals extracted in the electroencephalogram signal collection time window are all greater than the electrooculogram feature threshold value, canceling further processing of the electroencephalogram signal and guiding the user to re-fixate on the visual stimulation interface.

3. The method of claim 1, wherein, The visual stimulation interface comprises a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface comprising a plurality of first-type stimulation sources, the plurality of first-type stimulation sources being used to trigger first-type control instructions, the plurality of first-type stimulation sources having a plurality of stimulation frequencies, the second visual stimulation interface comprising a plurality of second-type stimulation sources, the plurality of second-type stimulation sources being used to trigger second-type control instructions, the plurality of first-type stimulation sources having a plurality of stimulation frequencies, the stimulation frequencies of the plurality of first-type stimulation sources being partially the same as or all the same as the stimulation frequencies of the plurality of second-type stimulation sources; The method further comprises: Collecting electromyogram signals and performing feature extraction on the electromyogram signals to obtain electromyogram feature values; If the electromyogram feature values are greater than electromyogram feature threshold values, triggering the intelligent visual stimulator to switch between the first visual stimulation interface and the second visual stimulation interface.

4. The method of claim 1, wherein, The method further comprises: Performing correlation analysis on the classification results of a plurality of electroencephalogram signals collected from the same user to obtain a maximum correlation coefficient value of the classification results of the plurality of electroencephalogram signals; Adjusting the classification threshold value based on an average value of the maximum correlation coefficient values of the classification results of the plurality of electroencephalogram signals.

5. The method of claim 1, wherein, The method further comprises: Using the fast Fourier transform method to convert the denoised electroencephalogram signal from the time domain to the frequency domain to obtain the power spectrum feature value of the denoised electroencephalogram signal.

6. A hybrid brain-machine interface based bank system interaction device, characterized in that, The device comprises: A signal collection module for collecting electroencephalogram signals and electrooculogram signals in the process of presenting a visual stimulation interface, the visual stimulation interface comprising a plurality of stimulation sources; The signal processing module is configured to, if an electrooculogram feature value of the electrooculogram signal extracted in the electroencephalogram signal collection time window is less than an electrooculogram feature threshold value, perform denoising processing on the electroencephalogram signal collected in the electroencephalogram signal collection time window to obtain a denoised electroencephalogram signal; perform frequency spectrum value monitoring on the denoised electroencephalogram signal to obtain a power spectrum feature value of the denoised electroencephalogram signal; and if the power spectrum feature value is higher than an electroencephalogram feature threshold value, perform stimulation source classification on the denoised electroencephalogram signal to obtain a classification result of the electroencephalogram signal. The control processing module is configured to, if the classification result of the electroencephalogram signal is greater than or equal to a classification threshold value, control an operation bank system based on the classification result of the electroencephalogram signal; and if the classification result of the electroencephalogram signal is less than the classification threshold value, determine that the classification result of the electroencephalogram signal is invalid.

7. The apparatus of claim 6, wherein, The signal processing module is further configured to, if electrooculogram feature values of multiple electrooculogram signals extracted in the electroencephalogram signal collection time window are all greater than the electrooculogram feature threshold value, cancel further processing on the electroencephalogram signal, and guide a user to re-fixate on the visual stimulation interface.

8. The apparatus of claim 6, wherein, The visual stimulation interface includes a first visual stimulation interface or a second visual stimulation interface, the first visual stimulation interface includes a plurality of first-type stimulation sources, the plurality of first-type stimulation sources are configured to trigger a first-type control instruction, the plurality of first-type stimulation sources have a plurality of stimulation frequencies, the second visual stimulation interface includes a plurality of second-type stimulation sources, the plurality of second-type stimulation sources are configured to trigger a second-type control instruction, the plurality of first-type stimulation sources have a plurality of stimulation frequencies, and the stimulation frequencies of the plurality of first-type stimulation sources and the stimulation frequencies of the plurality of second-type stimulation sources are partially the same or all the same. The signal collection module is further configured to collect an electromyogram signal, and perform feature extraction on the electromyogram signal to obtain an electromyogram feature value. The signal processing module is further configured to, if the electromyogram feature value is greater than an electromyogram feature threshold value, trigger the intelligent visual stimulator to switch between the first visual stimulation interface and the second visual stimulation interface.

Images