A visual attention adaptive method and system based on FBCCA

Abstract

The present invention relates to a method and system for self-adapting visual attention based on FBCCA, and belongs to the field of brain-computer interface technology. The method includes: constructing a visual stimulation paradigm, and setting a visual image interface according to the visual stimulation paradigm; collecting the electroencephalogram (EEG) signal of the person being tested who observes the visual image interface through an external electrode electroencephalogram (EEG) cap; obtaining a cyclic movement recognition template through the time displacement relationship between the multi-channel template and the stimulation target; filtering the original data of the EEG signal through a spatiotemporal filter to obtain test signal data; calculating the correlation coefficient between the test signal data and the cyclic movement recognition template, and determining the visual attention stimulation target in the stimulation paradigm according to the correlation coefficient; generating a control instruction according to the visual attention stimulation target, and sending the control instruction to the main control system of the device through a wireless communicator. The robustness of the brain-computer interface system is improved, and the recognition accuracy and information transmission rate of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer interface, and in particular relates to a visual attention adaptation method and system based on FBCCA. Background Art

[0002] Brain-computer interface (BCI) technology is a revolutionary human-computer interaction that bypasses peripheral nerves and muscles to directly establish a new communication and control channel between the brain and external devices. It has the functions of monitoring, replacing, improving / restoring, enhancing, and supplementing the output and input of damaged or impaired natural central nervous systems, and has important potential applications in the field of medical health. When the human body is stimulated by a fixed-frequency flickering or changing pattern of visual stimulation, the potential activity of the cerebral cortex will be modulated, resulting in a continuous response related to the stimulation frequency (at the fundamental frequency or multiple of the stimulation frequency). This response has a periodic rhythm similar to that of visual stimulation, namely Steady-State Visual Evoked Potentials (SSVEP). The SSVEP signal is manifested in the EEG signal as a spectral peak at the stimulation frequency or harmonics in the power spectrum. According to the specific frequency corresponding to the position of the spectral peak, the stimulus source of the subject's visual gaze can be detected, so that the subject's intention can be identified.

[0003] Canonical Correlation Analysis (CCA) and its improved algorithm are the mainstream algorithms for SSVEP frequency recognition. The core idea of ​​CCA is to calculate the correlation coefficient between multi-channel SSVEP EEG signals and different reference signals, and consider the frequency corresponding to the reference signal with the largest correlation as the target stimulation frequency. This algorithm is not only simple to implement but also has a high recognition accuracy, so it is widely used in SSVEP frequency recognition.

[0004] At present, the BCI system based on FBCCA (Filter Bank Canonical Correlation Analysis, FBCCA) still has the following areas for improvement:

[0005] To obtain EEG signals with high signal-to-noise ratio in high-noise and high-interference scenarios such as industrial operations, it is necessary to accumulate EEG parameter databases in various operating environments and continuously optimize engineering algorithm filters. To improve the overall signal quality, more efficient and stable system support is required. The conditions in industrial scenarios limit the use of only a small number of information channels, which places higher demands on algorithms for real-time brain state analysis under low-information conditions. In the large-scale commercial use of brain-computer interface systems in industrial scenarios, the suppression effect of noise and motion artifacts needs to be improved, and the differences between different groups and individuals pose a severe challenge to robustness. Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a visual attention adaptation method and system based on FBCCA.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] S1: constructing a visual stimulation paradigm and setting a visual image interface according to the visual stimulation paradigm;

[0009] S2: collecting and observing the electroencephalogram (EEG) signal of the person under test who is observing the visual image interface through an external electroencephalogram (EEG) cap;

[0010] S3: Generate multi-channel training signal data based on the stimulation cycle through the visual stimulation paradigm and the electroencephalogram signal, extract the target recognition template of the multi-channel training signal data through the multi-channel template of the spatiotemporal filter, and obtain the cyclic movement recognition template according to the time displacement relationship of the stimulation target and the target recognition template; filter the original data of the electroencephalogram signal through the spatiotemporal filter to obtain test signal data; calculate the correlation coefficient between the test signal data and the cyclic movement recognition template, and determine the visual attention stimulation target in the stimulation paradigm according to the correlation coefficient;

[0011] S4: Generate a control instruction according to the visual attention stimulation target, and send the control instruction to the device main control system through the wireless communicator.

[0012] Specifically, the visual stimulation paradigm uses a liquid crystal display as a stimulation source, and sets different frequencies and phases for each stimulation block. At each sampling moment, the brightness of the visual stimulation block is updated in a sinusoidal wave paradigm to trigger a steady-state visual evoked potential signal. The screen brightness sequence of the visual stimulation paradigm is:

[0013] ,

[0014] in, is the positive correlation coefficient of screen brightness, f , ϕare the stimulation frequency and phase, f s is the screen refresh rate, k It is the stimulation block index number under single sampling.

[0015] Specifically, the stimulation cycle is the number of shift registers in a linear shift register sequence corresponding to a pseudo-random code sequence generated by modulating the stimulation target.

[0016] As a preferred technical solution of the present invention, the space-time filter expands the signal source in the spatial position to obtain the filtered signal, and the calculation formula is:

[0017] ,

[0018] in, y ( n ) is the filtered signal, c is the channel count, C is the total number of channels, is the delay time, Ne is the delay parameter based on the order of the space-time filter, The delay time for channel c is The weight of The delay time for channel c is signal;

[0019] Said The optimal weight solution of is obtained by performing inverse operation on the multi-channel training signal data, and the calculation formula is:

[0020] ,

[0021] in, for The optimal weight of m Counts a single stimulation cycle, M is the total number of stimulation cycles in the signal processing stage, n is the shift register count, N is the total amount of the shift register, y m (n) is the multi-channel training signal data, s ( n ) is a one-dimensional target recognition template, and λ is the regularization parameter of cross-validation.

[0022] Specifically, the target recognition template is generated by averaging the training data under multiple stimulation cycles, and the calculation formula is:

[0023] ,

[0024] in, T r (n ) is the target recognition template, It is a multi-channel template;

[0025] The time displacement relationship is determined by the sampling interval of the visual evoked potential signal of the stimulus target and the reference target and the number of bits of the shift of the continuous pseudo-random code sequence generated by the stimulus target. The calculation formula for calculating the cyclic movement recognition template through the time displacement relationship is:

[0026] ,

[0027] in, k is the serial number of the stimulus target, r is the serial number of the reference target, T k ( n ) is a cyclic movement recognition template, T r For target recognition template, b s is the discrete sampling interval of the visual evoked potential signal of the stimulus target and the reference target, N is the total number of sampling points, L is the number of bits in the pseudo-random code sequence.

[0028] Specifically, the calculation formula of the correlation coefficient is:

[0029] ,

[0030] in, is the correlation coefficient, y t ( n ) is the test signal data, T k ( n ) is a cyclic movement recognition template, E is the inner product calculation function;

[0031] The template number corresponding to the maximum value of the correlation coefficient As a target of visual attention stimulation.

[0032] Specifically, the wireless communicator obtains the control instruction through serial port communication, and sends the control instruction to the edge main control system through the Bluetooth communication module.

[0033] A visual attention adaptive system based on FBCCA, comprising: a visual stimulation module, a signal acquisition module, a signal processing module, and a device control module;

[0034] The visual stimulation module is used to construct a visual stimulation paradigm and set a visual image interface according to the visual stimulation paradigm;

[0035] The signal acquisition module is used to acquire the electroencephalogram (EEG) signal of the person being tested who observes the visual image interface through an external electrode EEG cap;

[0036] The signal processing module is used to generate multi-channel training signal data based on the stimulation cycle through the visual stimulation paradigm and the electroencephalogram signal, extract the target recognition template of the multi-channel training signal data through the multi-channel template of the spatiotemporal filter, and obtain the cyclic movement recognition template according to the time displacement relationship of the stimulation target and the target recognition template; filter the original data of the electroencephalogram signal through the spatiotemporal filter to obtain test signal data; calculate the correlation coefficient between the test signal data and the cyclic movement recognition template, and determine the visual attention stimulation target in the stimulation paradigm according to the correlation coefficient;

[0037] The device control module is used to generate a control instruction according to the visual attention stimulation target, and send the control instruction to the device main control system through a wireless communicator.

[0038] The beneficial effects of the present invention are:

[0039] By designing a stimulation paradigm based on sinusoidal coding, the visual stimulation block is no longer limited to black and white states. The stimulation is concentrated in the central area of ​​the receptive field, which improves the overall response and obtains high-quality response signals. It can also reduce the visual fatigue of the subjects to a certain extent and improve comfort. By generating multi-channel training signal data based on the stimulation cycle and performing feature extraction and classification more effectively according to the time displacement relationship of the stimulation target, the algorithm is continuously optimized according to the actual scene data, the data model is iterated, the robustness of the brain-computer interface system is improved, and the recognition accuracy and information transmission rate of the system are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0041] Figure 1 A schematic diagram of a flow chart of a visual attention adaptation method based on FBCCA of the present invention;

[0042] Figure 2 It is a structural schematic diagram of a visual attention adaptive system based on FBCCA of the present invention. DETAILED DESCRIPTION

[0043] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0044] See also Figure 1 , a visual attention adaptation method based on FBCCA:

[0045] S1: constructing a visual stimulation paradigm and setting a visual image interface according to the visual stimulation paradigm;

[0046] S2: collecting and observing the electroencephalogram (EEG) signal of the person under test who is observing the visual image interface through an external electroencephalogram (EEG) cap;

[0047] S3: Generate multi-channel training signal data based on the stimulation cycle through the visual stimulation paradigm and the electroencephalogram signal, extract the target recognition template of the multi-channel training signal data through the multi-channel template of the spatiotemporal filter, and obtain the cyclic movement recognition template according to the time displacement relationship of the stimulation target and the target recognition template; filter the original data of the electroencephalogram signal through the spatiotemporal filter to obtain test signal data; calculate the correlation coefficient between the test signal data and the cyclic movement recognition template, and determine the visual attention stimulation target in the stimulation paradigm according to the correlation coefficient;

[0048] S4: Generate a control instruction according to the visual attention stimulation target, and send the control instruction to the device main control system through the wireless communicator.

[0049] In this embodiment, Figure 2 As shown, the EEG acquisition system uses the Synamp2 device of Neuroscan Company, including an electrode cap, equipped with 64 lead electrodes, and the software Scan4.5 is matched with the amplifier. The stimulation signal is generated by using the LCD liquid crystal display and the computer with a parallel interface by using Visual Studio programming on a computer with a parallel interface. The computer is connected to the EEG amplifier through the parallel interface, and a synchronization signal is sent at the same time to realize the synchronous recording of the EEG signal and the synchronous recording of the trigger signal, and at the same time modulate the pseudo-random code to stimulate the target. When the subject looks at the target of the stimulation and generates EEG, the EEG signal is recorded and amplified through the electrode cap worn on the head through the NeuroScan Synamp2 system. The EEG signal is converted to A / D with a sampling rate of 1000Hz, and then the digital signal is transmitted to the acquisition module on the PC through the USB interface. Finally, the EEG signal is analyzed using the SCAN4.5 software installed on the computer.

[0050] Specifically, the visual stimulation paradigm uses a liquid crystal display as a stimulation source, and sets different frequencies and phases for each stimulation block. At each sampling moment, the brightness of the visual stimulation block is updated in a sinusoidal wave paradigm to trigger a steady-state visual evoked potential signal. The screen brightness sequence of the visual stimulation paradigm is:

[0051] ,

[0052] in, is the positive correlation coefficient of screen brightness, f , ϕ are the stimulation frequency and phase, f s is the screen refresh rate, k It is the stimulation block index number under single sampling.

[0053] Specifically, the stimulation cycle is the number of shift registers in a linear shift register sequence corresponding to a pseudo-random code sequence generated by modulating the stimulation target.

[0054] As a preferred technical solution of the present invention, the space-time filter expands the signal source in the spatial position to obtain the filtered signal, and the calculation formula is:

[0055] ,

[0056] in, y ( n ) is the filtered signal, c is the channel count, C is the total number of channels, is the delay time, Ne is the delay parameter based on the order of the space-time filter, The delay time for channel c is The weight of The delay time for channel c is signal;

[0057] Said The optimal weight solution of is obtained by performing inverse operation on the multi-channel training signal data, and the calculation formula is:

[0058] ,

[0059] in, for The optimal weight of m Counts a single stimulation cycle, M is the total number of stimulation cycles in the signal processing stage, n is the shift register count, N is the total amount of the shift register, y m (n) is the multi-channel training signal data, s ( n ) is a one-dimensional target recognition template, and λ is the regularization parameter of cross-validation.

[0060] Specifically, the target recognition template is generated by averaging the training data under multiple stimulation cycles, and the calculation formula is:

[0061] ,

[0062] in, T r ( n ) is the target recognition template, It is a multi-channel template;

[0063] The time displacement relationship is determined by the sampling interval of the visual evoked potential signal of the stimulus target and the reference target and the number of bits of the shift of the continuous pseudo-random code sequence generated by the stimulus target. The calculation formula for calculating the cyclic movement recognition template through the time displacement relationship is:

[0064] ,

[0065] in, k is the serial number of the stimulus target, r is the serial number of the reference target, T k ( n ) is a cyclic movement recognition template, T r For target recognition template, b s is the discrete sampling interval of the visual evoked potential signal of the stimulus target and the reference target, N is the total number of sampling points, L is the number of bits in the pseudo-random code sequence.

[0066] Specifically, the calculation formula of the correlation coefficient is:

[0067] ,

[0068] in, is the correlation coefficient, y t ( n ) is the test signal data, T k ( n ) is a cyclic movement recognition template, E is the inner product calculation function;

[0069] The template number corresponding to the maximum value of the correlation coefficient As a target of visual attention stimulation.

[0070] Specifically, the wireless communicator obtains the control instruction through serial port communication, and sends the control instruction to the edge main control system through the Bluetooth communication module.

[0071] In this embodiment, the wireless communicator first detects whether there is a pre-stored device name after the Flash is powered on. If so, it detects whether there is a device with the same device name nearby, and then connects to the device that meets the requirements. The method to achieve active pairing is: the wireless communicator sends an active pairing command through the serial port, scans the UUIDs of surrounding devices, and reads the device name UUID that meets the settings; if the first 5 characters of the device name meet the settings, the signal strength of the device is read, and the device name UUID and device name that meet the requirements and have the highest signal strength are stored in the Flash and restart the system. The restarted system can use the name pre-stored in the Flash for pairing. The meaning of active pairing is that each edge device of the main control system has a different name, and until the next active pairing command is issued, the same Bluetooth receiver of active pairing can be connected to edge devices with different names. After receiving the control command through the serial port, it can be sent out through Bluetooth after the Bluetooth connection.

[0072] A visual attention adaptive system based on FBCCA, comprising: a visual stimulation module, a signal acquisition module, a signal processing module, and a device control module;

[0073] The visual stimulation module is used to construct a visual stimulation paradigm and set a visual image interface according to the visual stimulation paradigm;

[0074] The signal acquisition module is used to acquire the electroencephalogram (EEG) signal of the person being tested who observes the visual image interface through an external electrode EEG cap;

[0075] The signal processing module is used to generate multi-channel training signal data based on the stimulation cycle through the visual stimulation paradigm and the electroencephalogram signal, extract the target recognition template of the multi-channel training signal data through the multi-channel template of the spatiotemporal filter, and obtain the cyclic movement recognition template according to the time displacement relationship of the stimulation target and the target recognition template; filter the original data of the electroencephalogram signal through the spatiotemporal filter to obtain test signal data; calculate the correlation coefficient between the test signal data and the cyclic movement recognition template, and determine the visual attention stimulation target in the stimulation paradigm according to the correlation coefficient;

[0076] The device control module is used to generate a control instruction according to the visual attention stimulation target, and send the control instruction to the device main control system through a wireless communicator.

[0077] The working principle and use process of the present invention:

[0078] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0079] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0080] The program code included on the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0081] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A visual attention adaptation method based on FBCCA, characterized in that: include: S1: constructing a visual stimulation paradigm and setting a visual image interface according to the visual stimulation paradigm; The visual stimulation paradigm uses a liquid crystal display as a stimulation source, and sets different frequencies and phases for each stimulation block. At each sampling moment, the brightness of the visual stimulation block is updated in a sinusoidal wave paradigm to trigger a steady-state visual evoked potential signal. The screen brightness sequence of the visual stimulation paradigm is: ， in, is the positive correlation coefficient of screen brightness, f , ϕ are the stimulation frequency and phase, f s is the screen refresh rate, k is the stimulus block index number under single sampling; S2: collecting and observing the electroencephalogram (EEG) signal of the person under test who is observing the visual image interface through an external electroencephalogram (EEG) cap; S3: Generate multi-channel training signal data based on the stimulation cycle through the visual stimulation paradigm and the electroencephalogram signal, extract the target recognition template of the multi-channel training signal data through the multi-channel template of the spatiotemporal filter, and obtain the cyclic movement recognition template according to the time displacement relationship of the stimulation target and the target recognition template; filter the original data of the electroencephalogram signal through the spatiotemporal filter to obtain test signal data; calculate the correlation coefficient between the test signal data and the cyclic movement recognition template, and determine the visual attention stimulation target in the stimulation paradigm according to the correlation coefficient; The space-time filter expands the signal source in space to obtain a filtered signal, and the calculation formula is: ， in, y ( n ) is the filtered signal, c is the channel count, C is the total number of channels, is the delay time, Ne is the delay parameter based on the order of the space-time filter, The delay time for channel c is The weight of The delay time for channel c is signal; The target recognition template is generated by averaging the training data under multiple stimulation cycles, and the calculation formula is: ， in, T r ( n ) is the target recognition template, It is a multi-channel template; The time displacement relationship is determined by the sampling interval of the visual evoked potential signal of the stimulus target and the reference target and the number of bits of the shift of the continuous pseudo-random code sequence generated by the stimulus target. The calculation formula for calculating the cyclic movement recognition template through the time displacement relationship is: ， in, k is the serial number of the stimulus target, r is the serial number of the reference target, T k ( n ) is a cyclic movement recognition template, T r For target recognition template, b s is the discrete sampling interval of the visual evoked potential signal of the stimulus target and the reference target, N is the total number of sampling points, L is the number of bits of the pseudo-random code sequence; The calculation formula of the correlation coefficient is: ， in, is the correlation coefficient, y t ( n ) is the test signal data, T k ( n ) is a cyclic movement recognition template, E is the inner product calculation function; The template number corresponding to the maximum value of the correlation coefficient As a target of visual attention stimulation; S4: Generate a control instruction according to the visual attention stimulation target, and send the control instruction to the device main control system through the wireless communicator.

2. The method according to claim 1, characterized in that The stimulation cycle is the number of shift registers in a linear shift register sequence corresponding to a pseudo-random code sequence generated by modulating the stimulation target.

3. The method according to claim 1, characterized in that The wireless communicator obtains the control instruction through serial port communication, and sends the control instruction to the edge main control system through the Bluetooth communication module.

4. A visual attention adaptive system based on FBCCA, used to execute the method according to any one of claims 1 to 3, characterized in that: include: Visual stimulation module, signal acquisition module, signal processing module, equipment control module; The visual stimulation module is used to construct a visual stimulation paradigm and set a visual image interface according to the visual stimulation paradigm; The signal acquisition module is used to acquire the electroencephalogram (EEG) signal of the person being tested who observes the visual image interface through an external electrode EEG cap; The signal processing module is used to generate multi-channel training signal data based on the stimulation cycle through the visual stimulation paradigm and the electroencephalogram signal, extract the target recognition template of the multi-channel training signal data through the multi-channel template of the spatiotemporal filter, and obtain the cyclic movement recognition template according to the time displacement relationship of the stimulation target and the target recognition template; and filter the original data of the electroencephalogram signal through the spatiotemporal filter to obtain the test signal data; Calculating the correlation coefficient between the test signal data and the cyclic movement recognition template, and determining the visual attention stimulation target in the stimulation paradigm according to the correlation coefficient; The device control module is used to generate a control instruction according to the visual attention stimulation target, and send the control instruction to the device main control system through a wireless communicator.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the FBCCA-based visual attention adaptation method as described in any one of claims 1-3 is implemented.

6. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the FBCCA-based visual attention adaptation method as described in any one of claims 1-3 when executed by a computer processor.

Images