Stimulation modulation and decoding method of SSVEP brain-computer interface under dynamic background

Abstract

The present invention discloses a method for modulating and decoding SSVEP brain-computer interface stimulation in a dynamic background. The modulation method includes suspending SSVEP stimulation blocks of preset shapes at multiple preset positions in the dynamic background; using a preset foreground modulation method and / or background modulation method to change the contrast between each SSVEP stimulation block and the corresponding background area; using sampled sine coding to smoothly change the transparency of the stimulation block so that the stimulation block continuously flashes based on its own frequency to stimulate the subject to generate SSVEP EEG signals. This can improve the contrast between the stimulation block and the background area, avoid it being submerged in the background, and facilitate viewing by the subject. The decoding method preprocesses the collected SSVEP EEG signals of the subject and decodes them using a preset DBDN network to obtain attribute information of the SSVEP stimulation block corresponding to the SSVEP EEG signal. This can improve decoding accuracy under the SSVEP paradigm for dynamic backgrounds.

Description

Technical Field

[0001] The present invention belongs to the field of signal processing, and specifically relates to a SSVEP brain-computer interface stimulation modulation and decoding method under a dynamic background. Background Art

[0002] Brain-computer interface (BCI) technology is an important bridge connecting the human brain and external devices. A product of the fusion of computer science and biological science, BCI technology is a crucial means for exploring the brain and studying human-computer interaction. It has applications in military operations, skills training, and disease rehabilitation.

[0003] Steady-State Visual Evoked Potential (SSVEP), currently the fastest and most stable brain-computer interface (BCI) paradigm, has been applied in fields such as control and rehabilitation. Compared to BCIs that use other input signals, SSVEP BCI systems offer significant advantages due to their high information transmission rates, stable performance, and short training time.

[0004] The SSVEP BCI primarily collects and analyzes SSVEP EEG (electroencephalogram) signals. This is because EEG signals are acquired directly from the scalp using non-invasive methods, such as wearing an EEG cap. Compared to other EEG signal acquisition methods, it offers advantages such as ease of acquisition, flexibility, and high security. It also meets the high real-time requirements of brain-computer interface systems.

[0005] Among the visual stimulus sources commonly used by SSVEP BCI to induce SSVEP signals, graphic stimulus sources are a common type. They mainly use flickering stimulation on a computer display screen, that is, making a single simple square, complex multiple squares, pictures of different colors and other graphics as stimulus blocks, and flickering the stimulus at a certain frequency. In SSVEP BCI, the experimental paradigm design is mainly based on the encoding of stimulus blocks. The classic paradigm is that on a black background, multiple white squares containing different semantic information (such as letters, numbers, actions, etc.) flash simultaneously. Each stimulus block is encoded by a stimulus frequency, and the frequencies used by different stimulus blocks are different. Therefore, the frequency encoding of the stimulus block uniquely "marks" each stimulus block.

[0006] Currently, with the popularization of communication and VR (Virtual Reality) technologies, more precise and real-time requirements are being placed on SSVEP control peripherals. A typical control scenario involves the SSVEP stimulation block floating above the dynamic image transmission signal of the peripheral, allowing the operator to synchronously observe the image transmission as the background on the computer display screen of the brain-computer interface system and generate SSVEP EEG signals under the stimulation of the SSVEP stimulation block, thereby controlling the device. However, since the image transmission signal changes in real time, the contrast between the traditional monochrome SSVEP stimulation block and the background will change dynamically. In extreme cases, the monochrome SSVEP stimulation block may be submerged in the background, making it difficult for the operator to observe these stimulation blocks. When the operator looks at these stimulation blocks, the decoding accuracy of the existing SSVEP decoding algorithm is limited and the robustness is poor. It is difficult to accurately identify the stimulation block the operator is looking at, thereby reducing the efficiency of the SSVEP BCI and affecting the accuracy of subsequent control links.

[0007] Therefore, how to propose a stimulation block modulation method and decoding method for the dynamic background in SSVEP BCI, such as the dynamic image transmission signal displayed on the display screen or other browsing backgrounds with changing color and brightness, is a technical problem that needs to be solved urgently. Summary of the Invention

[0008] The purpose of the embodiments of the present invention is to provide a method for modulating and decoding stimulation blocks in a dynamic SSVEP BCI environment, thereby improving the efficiency of the SSVEP BCI. The specific technical solution is as follows:

[0009] In a first aspect, an embodiment of the present invention provides an SSVEP brain-computer interface stimulation modulation method under a dynamic background, which is applied to a display modulation module corresponding to a display screen in a brain-computer interface system. The method includes:

[0010] SSVEP stimulation blocks of preset shapes are suspended at a plurality of preset positions of the dynamic background displayed on the display screen;

[0011] The contrast between each SSVEP stimulus block and the corresponding background area is changed using a preset foreground modulation method and / or background modulation method, and the transparency of each SSVEP stimulus block obtained by smoothing the change using sampled sine coding is used to make multiple SSVEP stimulus blocks flash continuously based on their respective frequencies to stimulate the subject to produce SSVEP EEG signals.

[0012] In one embodiment of the present invention, the preset foreground modulation method includes:

[0013] For each SSVEP stimulus block, the three-channel data of each pixel in the background image within the range of the SSVEP stimulus block at the current moment are complemented by the pixel maximum value 255 to obtain the complemented three-channel data of each pixel;

[0014] The three-channel data after complementation of the same pixel are merged, and all the merged pixels are used to obtain the SSVEP stimulus block after foreground modulation of the SSVEP stimulus block at the current moment.

[0015] In one embodiment of the present invention, the preset foreground modulation method includes:

[0016] For each SSVEP stimulus block, the three-channel data of each pixel in the background image within the range of the SSVEP stimulus block at the current moment are complemented by the pixel maximum value 255 to obtain the complemented three-channel data of each pixel;

[0017] Calculate the mean of the complemented data of all pixels in the same channel to obtain the updated data of the corresponding channel;

[0018] The updated data of the three channels are merged to obtain the updated pixel of each pixel, and all the updated pixels are used to obtain the SSVEP stimulation block after the foreground modulation of the SSVEP stimulation block at the current moment.

[0019] In one embodiment of the present invention, the preset background modulation method includes:

[0020] For each SSVEP stimulus block, convert the pixel format of the image within the range of the background block surrounding the SSVEP stimulus block at the current moment into the HSV format, and obtain the three-channel data of each pixel in the background block in the HSV format;

[0021] Compressing the V channel data of each pixel in the background block in the HSV format according to a preset brightness compression ratio;

[0022] The three-channel data of the same pixel after brightness compression are merged, and all the merged pixels are used to obtain the modulated background block corresponding to the SSVEP stimulus block at the current moment.

[0023] In one embodiment of the present invention, the preset background modulation method includes:

[0024] For each SSVEP stimulation block, a defocus blur operation is performed on each pixel point in the image within the range of the background block surrounding the SSVEP stimulation block at the current moment, and the modulated background block corresponding to the SSVEP stimulation block at the current moment is obtained by using each pixel point processed by the defocus blur operation.

[0025] In one embodiment of the present invention, the method of smoothly changing the transparency of each SSVEP stimulus block obtained by sampling sinusoidal coding so that the multiple SSVEP stimulus blocks continuously flicker based on their respective frequencies includes:

[0026] At the current moment, for each SSVEP stimulation block after foreground modulation using the preset foreground modulation method, the transparency change sequence of the stimulation block is determined using the sampled sine coding method, and the transparency of the stimulation block is changed according to the corresponding transparency change sequence; wherein the transparency change sequence of each SSVEP stimulation block is determined at least based on its own frequency.

[0027] In one embodiment of the present invention, the method of smoothly changing the transparency of each SSVEP stimulus block obtained by sampling sinusoidal coding so that the multiple SSVEP stimulus blocks continuously flicker based on their respective frequencies includes:

[0028] At the current moment, for each modulated SSVEP stimulus block, the stimulus block is superimposed on the modulated background block corresponding to itself at the current moment, and the transparency change sequence of the stimulus block is determined using the sampled sine coding method, and the transparency of the stimulus block is changed according to the corresponding transparency change sequence; wherein, each modulated SSVEP stimulus block includes each SSVEP stimulus block obtained after foreground modulation using the preset foreground modulation method or a traditional monochrome SSVEP stimulus block; the transparency change sequence of each SSVEP stimulus block is determined based on at least its own frequency.

[0029] In a second aspect, an embodiment of the present invention provides a method for decoding SSVEP brain-computer interface stimulation in a dynamic background, which is applied to a signal processing module in a brain-computer interface system. The method includes:

[0030] Preprocessing the collected SSVEP EEG signals of the subject; wherein the SSVEP EEG signals are generated by suspending SSVEP stimulation blocks of preset shapes at multiple preset positions of a dynamic background displayed on the display screen by a display modulation module corresponding to the display screen of the brain-computer interface system; using a preset modulation method to change the contrast between each SSVEP stimulation block and the corresponding background area, and using sampled sine coding to smoothly change the transparency of each SSVEP stimulation block, so that the multiple SSVEP stimulation blocks continuously flash at their respective frequencies to stimulate the subject; the preset modulation method includes a preset foreground modulation method and / or a preset background modulation method;

[0031] The pre-processed SSVEP EEG signal is decoded using a preset DBDN network to obtain attribute information of the SSVEP stimulation block corresponding to the SSVEP EEG signal; wherein the preset DBDN network includes a feature extraction layer, a feature compression layer and a classification layer.

[0032] In one embodiment of the present invention, the feature extraction layer includes a serial multi-scale temporal feature extraction unit, a spatial feature extraction unit, and a first activation function operation module; wherein the multi-scale temporal feature extraction unit includes a preset number of parallel temporal convolution layers with different temporal convolution kernel lengths, a batch normalization processing module corresponding to each temporal convolution layer, and a feature fusion module for fusing processing results of each batch normalization processing module; the spatial feature extraction unit includes a serial spatial convolution layer and a batch normalization processing module;

[0033] The feature compression layer includes an average pooling layer, a random drop processing module, a depth-separable convolution module, a batch normalization processing module and a second activation function operation module in series;

[0034] The classification layer includes a serial global average pooling layer and a third activation function operation module.

[0035] In one embodiment of the present invention, the preset number is 4;

[0036] In the multi-scale time feature extraction unit, the time convolution kernel lengths of the preset number of time convolution layers are respectively the SSVEP EEG signal sampling rate Among them, n=1, 2, 3, 4.

[0037] Beneficial effects of the present invention:

[0038] The SSVEP brain-computer interface stimulation modulation method under dynamic background provided by the embodiment of the present invention can improve the contrast between each SSVEP stimulation block and the corresponding background area by utilizing a preset foreground modulation method and / or background modulation method for the dynamic background, avoid the stimulation block being submerged in the background with a similar color, and enable the subject to more easily observe the flickering of the SSVEP stimulation block. Therefore, it can improve the efficiency of the SSVEP brain-computer interface under dynamic background.

[0039] The SSVEP brain-computer interface stimulation decoding method under a dynamic background provided by an embodiment of the present invention utilizes a pre-built and trained preset DBDN network to decode pre-processed SSVEP EEG signals to obtain attribute information of the SSVEP stimulation block corresponding to the SSVEP EEG signal. Since the SSVEP EEG signal to be decoded is the result of a display modulation module corresponding to the display screen of the brain-computer interface system, SSVEP stimulation blocks of preset shapes are suspended at multiple preset positions on the dynamic background displayed on the display screen. The contrast between each SSVEP stimulation block and the corresponding background area is changed using a preset modulation method, and the transparency of each resulting SSVEP stimulation block is smoothly changed using sampled sine coding, resulting in multiple SSVEP stimulation blocks continuously flashing at their respective frequencies to stimulate the subject. The preset modulation method includes a preset foreground modulation method and / or a preset background modulation method. Therefore, the feature extraction layer, feature compression layer, and classification layer included in the preset DBDN network can be used to improve the decoding accuracy of SSVEP EEG signals for the dynamic background SSVEP paradigm. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic flow chart of a method for SSVEP brain-computer interface stimulation modulation under a dynamic background provided by an embodiment of the present invention;

[0041] Figure 2 This is a diagram showing the main components of the brain-computer interface system in an embodiment of the present invention;

[0042] Figure 3 This is a flow chart of the SSVEP brain-computer interface stimulation modulation method under a dynamic background proposed in an embodiment of the present invention;

[0043] Figure 4 This is a flow chart of the implementation principle of the single-point inverted color modulation method proposed in an embodiment of the present invention;

[0044] Figure 5 This is a flow chart of the implementation principle of the global inverted color modulation method proposed in an embodiment of the present invention;

[0045] Figure 6 A schematic diagram illustrating the principle of the brightness compression modulation method proposed in an embodiment of the present invention;

[0046] Figure 7 A schematic diagram illustrating the principle of the background blur modulation method proposed in an embodiment of the present invention;

[0047] Figure 8 (a)~ Figure 8 (c) The effect diagram of stimulus blocks with different transparency determined by the sampled sine coding method;

[0048] Figure 9 (a)~ Figure 9 (b) is a diagram showing the effects of single-point inversion modulation and sampled sine encoding of two background images selected in an embodiment of the present invention at a certain moment in the SSVEP paradigm;

[0049] Figure 10 (a)~ Figure 10 (b) is a diagram showing the effects of global inversion modulation and sampled sine encoding of two background images selected in an embodiment of the present invention at a certain moment in the SSVEP paradigm;

[0050] Figure 11(a) to Figure 11(b) This is a diagram showing the effects of brightness compression modulation and sampled sine coding of two background images selected in an embodiment of the present invention at a certain moment in the SSVEP paradigm;

[0051] Figure 12(a) to Figure 12(b) This is a diagram showing the effects of background blur modulation and sampled sine coding of two background images selected in an embodiment of the present invention at a certain moment in the SSVEP paradigm;

[0052] Figure 13 A schematic flow chart of a method for decoding SSVEP brain-computer interface stimulation in a dynamic context provided by an embodiment of the present invention;

[0053] Figure 14 A schematic diagram of the structure of a DBDN network provided in an embodiment of the present invention;

[0054] Figure 15 A schematic diagram of an SSVEP EEG signal input by a preset DBDN network according to an embodiment of the present invention;

[0055] Figure 16 The function image of the ELU activation function used in the embodiment of the present invention;

[0056] Figure 17 Schematic diagram of the principle of deep convolution;

[0057] Figure 18 Schematic diagram of the point-by-point convolution principle;

[0058] Figure 19 Comparison chart of fully connected layer and global average pooling;

[0059] Figure 20 This is a comparison chart of the average accuracy of various decoding algorithms under different modulation methods in the experiments of the embodiment of the present invention;

[0060] Figure 21 This is a comparison chart of the average scores of four subjective questions under various modulation methods in the experiment of the embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0062] To facilitate understanding of the solutions of the embodiments of the present invention, the main technical problems faced are briefly introduced below.

[0063] For the structural composition of SSVEP BCI, please refer to Figure 1 The brain-computer interface system is understood. Among them, the display screen will show a dynamic background and stimulus blocks. The dynamic background is derived from the dynamic image transmission signal as an example. Since the image transmission signal changes in real time, the contrast between the traditional monochrome SSVEP stimulus block and the background will change dynamically, and in extreme cases, it may be submerged in the background. Figure 2 , Figure 2 These are four examples of a stimulus block submerged in the background. As the relative positions of the stimulus block and background image change, the stimulus block suspended above the background gradually becomes submerged in a similarly colored background, making it difficult for the operator to observe the stimulus block. When the operator focuses on the stimulus block, existing EEG decoding methods are insufficiently tailored to the SSVEP paradigm, making it difficult to accurately identify the stimulus block the operator is focusing on. Therefore, research is needed on modulation and decoding methods for SSVEP stimulus color relative to dynamic backgrounds to further improve the efficiency of SSVEP brain-computer interfaces in dynamic backgrounds.

[0064] In order to achieve the purpose of SSVEP BCI dynamic background, a stimulation block modulation method and a decoding method are proposed to improve the efficiency of SSVEP BCI. The embodiments of the present invention provide an SSVEP brain-computer interface stimulation modulation method under dynamic background and an SSVEP brain-computer interface stimulation decoding method under dynamic background.

[0065] In the first aspect, an embodiment of the present invention provides a method for modulating SSVEP brain-computer interface stimulation in a dynamic background. The method is applied to a display modulation module corresponding to a display screen in a brain-computer interface system, such as Figure 3 As shown, the method includes:

[0066] S1, suspending SSVEP stimulation blocks of preset shapes at multiple preset positions of a dynamic background displayed on a display screen.

[0067] It should be noted that the execution subject of the SSVEP brain-computer interface stimulation modulation method under dynamic background proposed in the embodiment of the present invention can be a device, corresponding to the display modulation module corresponding to the display screen in the brain-computer interface system. Figure 1 Combined with the existing knowledge about brain-computer interface systems, it can be understood that the display screen in the brain-computer interface system can generally be a computer display screen. Of course, it is not ruled out that it can be the display screen of other devices such as mobile phones. This is all reasonable. Taking a computer display screen as an example, under the control of the display modulation software in the computer, a dynamic background can be displayed on the display screen, and multiple stimulation blocks can be generated by floating on the dynamic background. Therefore, the SSVEP brain-computer interface stimulation modulation method under the dynamic background of the embodiment of the present invention can be understood as being implemented by the display modulation software that controls the display screen, such as computer display modulation software.

[0068] For information on the display process of a dynamic background on a display screen, please refer to the relevant prior art. The embodiments of the present invention may be directed to a scenario where a dynamic background is already displayed on the display screen. It is understood that due to the real-time variability of the dynamic background, the dynamic background at each moment is actually a background image, the size of which is determined by the display screen.

[0069] In the embodiment of the present invention, for the dynamic background of a predetermined size, a plurality of preset positions on the background image may be predetermined, and an SSVEP stimulation block of a preset shape is suspended at each preset position.

[0070] Specifically, for all moments, the multiple preset positions can be fixed; or, the multiple preset positions can also change according to different moments, and can be reasonably set according to the needs of the scene. For example, for a directional control scene, four fixed preset positions can be set for the dynamic background, and a SSVEP stimulus block of a preset shape can be suspended on the background of each preset position so that the SSVEP stimulus block covers the corresponding background area. The SSVEP stimulus blocks at the four preset positions of up, down, left and right represent the directional control of forward, backward, left turn and right turn, etc.

[0071] The preset shapes of the SSVEP stimulation blocks can include rectangles, circles, triangles, and the like, without limitation. The SSVEP stimulation blocks at each preset location can have the same preset shape, or they can have different preset shapes. Regardless of the shapes, the range of each SSVEP stimulation block is fixed at each moment.

[0072] For S1, the color and pattern of each stimulation block can be set arbitrarily. To facilitate subsequent processing, in an optional embodiment, in S1, the pixel values ​​in each stimulation block are the pixel values ​​of the background image within the range of the corresponding stimulation block.

[0073] S2, using a preset foreground modulation method and / or background modulation method to change the contrast between each SSVEP stimulus block and the corresponding background area, and using sampled sine coding to smoothly change the transparency of each SSVEP stimulus block obtained, so that multiple SSVEP stimulus blocks flash continuously based on their respective frequencies to stimulate the subject to produce SSVEP EEG signals.

[0074] Because the image transmission signal changes in real time, the contrast between traditional monochrome SSVEP stimulation blocks and the background will change dynamically. This can lead to the possibility that traditional monochrome SSVEP stimulation blocks will be submerged in the background in extreme cases. Embodiments of the present invention utilize preset foreground modulation methods and / or background modulation methods to change the contrast between each SSVEP stimulation block and the corresponding background area. The purpose is to improve the contrast between each SSVEP stimulation block and the corresponding background area, making it less likely that each SSVEP stimulation block will appear similar in color to the corresponding background, thereby eliminating the situation where the stimulation block is submerged in the background.

[0075] Foreground modulation refers to the modulation of the stimulus blocks involved in the SSVEP paradigm. This modulation aims to maintain the background of the SSVEP stimulus blocks unchanged and improve the contrast between the SSVEP stimulus blocks and the corresponding background areas. Background modulation refers to the modulation of the background image at the location of the stimulus blocks and a certain adjacent area. This modulation aims to maintain the display of the SSVEP stimulus blocks unchanged and improve the contrast between the SSVEP stimulus blocks and the corresponding background areas. These two methods are explained below.

[0076] (1) In an optional embodiment, the preset foreground modulation method includes:

[0077] A1, for each SSVEP stimulus block, the three-channel data of each pixel in the background image within the range of the SSVEP stimulus block at the current moment are complemented by the pixel maximum value 255 to obtain the complemented three-channel data of each pixel;

[0078] A2: Merge the complementary three-channel data of the same pixel, and use all the merged pixels to obtain the SSVEP stimulus block after foreground modulation of the SSVEP stimulus block at the current moment.

[0079] The image format involved in the embodiment of the present invention is RGB format. For simplicity, this foreground modulation method can be referred to as single-point inverted color modulation. Figure 4The figure below is a flowchart of the implementation principle of the single-point inversion modulation method. Here, the original image is split into three channels to obtain the original RGB three-channel data. After each channel data is complemented with 255, the complemented RGB three-channel data of the image is obtained. Then, the channels are merged to obtain the single-point inverted image.

[0080] Specifically for steps A1-A2, for each SSVEP stimulation block, the background image within the range of the SSVEP stimulation block at the current moment is obtained as an image block, and for each pixel of the image block, the RGB three-channel data is split, and the data of each split channel is complemented by 255 to obtain the complemented three-channel data of the pixel, and then the complemented three-channel data of the pixel are merged to obtain a merged pixel. A new image block can be obtained by arranging all the merged pixels according to the original pixel positions, and the SSVEP stimulation block after single-point inversion modulation of the SSVEP stimulation block at the current moment is obtained.

[0081] The complemented three-channel data of each pixel is calculated according to the following formula:

[0082]

[0083] Among them, R background , G background 、B background They represent the R channel data, G channel data, and B channel data of a single pixel point in the background image within the range of the SSVEP stimulus block before modulation; R flicker , G flicker 、B flicker They respectively represent the complementary R channel data, complementary G channel data, and complementary B channel data obtained after single-point inversion modulation.

[0084] It can be understood that after single-point inversion modulation, at each moment, an SSVEP stimulus block is actually an image block with a pattern, and the pixel values ​​are not the same.

[0085] (2) In another optional embodiment, the preset foreground modulation method includes:

[0086] B1, for each SSVEP stimulus block, the three-channel data of each pixel in the background image within the range of the SSVEP stimulus block at the current moment are complemented by the pixel maximum value 255 to obtain the complemented three-channel data of each pixel;

[0087] B2, calculate the mean of the complemented data of all pixels in the same channel to obtain the updated data of the corresponding channel;

[0088] B3, merging the updated data of the three channels to obtain the updated pixel of each pixel, and using all the updated pixels to obtain the SSVEP stimulation block after foreground modulation of the SSVEP stimulation block at the current moment.

[0089] For simplicity, this foreground modulation method can be referred to as global inversion modulation. Figure 5 The figure below is a flowchart of the implementation principle of the global inversion modulation method. The original image is split into three channels to obtain the original RGB three-channel data. Each channel data is complemented by 255 to obtain the complemented RGB three-channel data of the image. All the complemented data of the same channel are averaged to obtain the averaged RGB three-channel data of the image. Finally, the channels are merged to obtain the globally inverted image.

[0090] The present invention takes into account that traditional SSVEP stimulus blocks are pure white. Therefore, when designing the global inversion modulation method, the goal is to create a pure color stimulus block that changes with the background. Global inversion is similar to single-point inversion, differing in that after obtaining the complemented three-channel data for each pixel, the complemented data of all pixels in the same channel are averaged to obtain the updated data for that channel. The updated data of the three channels are then combined to obtain the updated pixel for each pixel. Therefore, it can be understood that after global inversion, all pixels are identical, resulting in a pure color stimulus block determined by the background.

[0091] The update data of the corresponding channel is calculated according to the following formula:

[0092]

[0093] Among them, R background , G background 、B background Respectively represent the R channel data, G channel data and B channel data of a single pixel point in the background image within the range of the SSVEP stimulus block before modulation; Rchannel flicker , Gchannel flicker , Bchannel flicker They represent the updated data of the R channel, the G channel, and the B channel after global inversion modulation respectively; ∑() represents the summation of the complemented data of the same channel of all pixels; mean represents the mean.

[0094] It can be understood that after global inversion modulation, at each moment, an SSVEP stimulus block is actually a pure color image block with the same pixel values.

[0095] Of course, the preset foreground modulation method provided by the embodiment of the present invention is not limited to the single-point inversion modulation and global inversion modulation described above. Any method that maintains the background of the SSVEP stimulation block unchanged and modulates the display effect of the SSVEP stimulation block to improve the contrast between the SSVEP stimulation block and the corresponding background area can be included in the protection scope of the preset foreground modulation method of the embodiment of the present invention.

[0096] (3) In an optional embodiment, the preset background modulation method includes:

[0097] C1, for each SSVEP stimulus block, convert the pixel format of the image within the range of the background block surrounding the SSVEP stimulus block at the current moment into HSV format, and obtain the three-channel data of each pixel in the background block in HSV format;

[0098] C2, compress the V channel data of each pixel in the background block in HSV format according to the preset brightness compression ratio;

[0099] C3, merge the three-channel data of the same pixel after brightness compression, and use all the merged pixels to obtain the modulated background block corresponding to the SSVEP stimulus block at the current moment.

[0100] Images captured in natural environments are easily affected by natural lighting, occlusion, and shadows, making them sensitive to brightness. The human visual system is sensitive to brightness information in addition to chrominance. Based on these characteristics, embodiments of the present invention propose a brightness compression method as a background modulation method. For simplicity, this background modulation method can be referred to as brightness compression modulation.

[0101] The most common image storage format is RGB, in which the red (R), green (G), and blue (B) color components are highly correlated. Continuous adjustments to an image's color or brightness require corrections to all three components. It's difficult to accurately deduce the values ​​of the three components for a single color. Therefore, while the RGB color space is suitable for display systems, it's not ideal for naturally representing images.

[0102] In addition to RGB, HSV is also a common image storage format. HSV represents a color image using three components: hue, saturation, and value. Compared to RGB, HSV is closer to human perception of color and allows for easier color comparison.

[0103] Therefore, for C1, the embodiment of the present invention first converts the pixel format of the image within the range of the background block surrounding each SSVEP stimulation block at the current moment into the HSV format, thereby obtaining three-channel data for each pixel in the background block in the HSV format. The background block surrounding the SSVEP stimulation block means that the range of the background block is larger than the range of the SSVEP stimulation block. For example, if the SSVEP stimulation block is a rectangle, its background block can be a larger rectangle that can completely enclose the rectangle corresponding to the SSVEP stimulation block.

[0104] The specific conversion formula from RGB to HSV is shown below:

[0105]

[0106] If H<0, then H=H+360. Finally, we get 0≤V≤1, 0≤S≤1, and 0≤H≤360. For the specific process of converting RGB to HSV, please refer to the relevant existing technology for understanding.

[0107] For C2, the V channel data of each pixel in the background block in HSV format is compressed according to the preset brightness compression ratio, which can be performed according to the following formula:

[0108]

[0109] Among them, H old 、S old 、V old Respectively represent the H channel, S channel and V channel data of the background block before brightness compression; H new 、S new 、V new They represent the H channel, S channel and V channel data after the background block brightness is compressed respectively; γ represents the preset brightness compression ratio, which has a value between [0, 1] and can be reasonably selected as needed, such as 0.5, 0.75, 0.25, etc.

[0110] Regarding the modulated background block in C3, please combine Figure 6 understand, Figure 6 This is a schematic diagram illustrating the principle of the brightness compression modulation method proposed in an embodiment of the present invention. An original image undergoes brightness compression to produce a brightness-compressed image, reducing the brightness. Therefore, it can be understood that by maintaining the display effect of the SSVEP stimulus block while proportionally compressing the brightness of the corresponding background block, the brightness of the background block is reduced, thereby increasing the contrast between the SSVEP stimulus block and the background.

[0111] (4) In another optional embodiment, the preset background modulation method includes:

[0112] For each SSVEP stimulus block, a defocus blur operation is performed on each pixel point in the image within the range of the background block surrounding the SSVEP stimulus block at the current moment, and the modulated background block corresponding to the SSVEP stimulus block at the current moment is obtained by using each pixel point processed by the defocus blur operation.

[0113] For simplicity, this background modulation method can be referred to as background blur modulation. This method applies a defocus blur to every pixel in the background patch and uses the defocused background patch as the new background patch for the SSVEP stimulus patch. Background blur modulation aims to reduce background detail to highlight the stimulus patch.

[0114] The defocus blur method is described by the point spread function (PSF). The PSF simulates how an imaging system captures a single point in the world. The most commonly used blur model is a linear model, where the blurred image b is represented as a convolution of a kernel k and noise, which can be expressed as:

[0115]

[0116] Where, in formula (5), i represents a pixel; represents convolution; n~Ν(0,σ 2 ), represents and additive Gaussian noise model.

[0117] The embodiment of the present invention can use any existing defocus blur function to realize background blur modulation. In an optional implementation, the defocus blur function used can be 2×radius+1 centered on the pixel to be processed, where radius is the radius of the circular area, and the RGB three channels of all pixels in the circular area are averaged respectively, and the value of the pixel after averaging is used to replace the value of the original pixel. Therefore, this method is also called circular mean filtering. It can be understood that if the circular mean filter is used and the shape of the background block is selected as a rectangle, then a smaller rectangular area can be selected to be retained as the background block within the larger circular mean filter range, while the remaining areas within the circle still use the original pixels, and so on.

[0118] For the principle of background blur modulation, please refer to Figure 7 It can be understood that after the original image is subjected to background blurring processing, a background blurred image is obtained. It can be seen that the background blurring modulation significantly weakens the details of the background.

[0119] Of course, the preset background modulation method provided in the embodiment of the present invention is not limited to the brightness compression modulation and background blur modulation described above. Any method that maintains the display effect of the SSVEP stimulation block unchanged and modulates the background area of ​​the SSVEP stimulation block to improve the contrast between the SSVEP stimulation block and the corresponding background area can be included in the protection scope of the preset background modulation method in the embodiment of the present invention.

[0120] It should be noted that background modulation is to modulate the background area of ​​the SSVEP stimulation block, but does not change the display effect of the SSVEP stimulation block. Therefore, the display effect of the SSVEP stimulation block can be achieved in any way.

[0121] That is, in the embodiments of the present invention, any one of the preset foreground modulation methods can be used alone, any one of the preset background modulation methods can be used alone, or a combination of one of the preset foreground modulation methods and one of the preset background modulation methods can be used, all of which are reasonable. It is understood that after foreground modulation and / or background modulation, the patterns or colors of the multiple SSVEP stimulation blocks are determined for the current background image, and according to the requirements of the dynamic background evoked paradigm, each SSVEP stimulation block also needs to achieve SSVEP stimulation at a different frequency.

[0122] According to the principle of dynamic background evoked paradigm, different SSVEP signals can be induced by different frequency stimulus sources. By controlling the speed of change of the brightness of the computer monitor, SSVEP stimuli of different frequencies can be generated. The classic method of generating SSVEP stimulation patterns of different frequencies is to make the monitor switch between black and white every fixed number of frames. Usually the monitor has a fixed screen refresh rate, which is set to R. When using the above classic method to generate different SSVEP frequency stimuli, the achievable stimulation frequency is Where n is a positive integer representing the number of frames in which the screen displays a color. Obviously, implementing frequency stimulation in this way will result in a very small number of frequencies available within the SSVEP generation frequency band.

[0123] The sampled sine coding method can effectively solve the shortcoming of limited available frequency. Sampled sine coding based on transparency change is often used in paradigm implementation. Its specific implementation principle is to change the transparency of each stimulus block on the display so that the stimulus block changes in the form of a sine wave on the display screen. The frequency of the sine wave is the final flicker stimulus frequency. Specifically, the transparency change sequence of the stimulus block determined by the sampled sine coding method can be determined according to the following formula:

[0124]

[0125] The sin function is used to generate a sine wave; n is the frame number; f is the frequency; φ is the phase; alpha is the transparency of the stimulus block, and its value range is 0≤alpha≤1, where 0 indicates that the stimulus block is completely transparent and 1 indicates that the stimulus block is completely opaque. Figure 8 (a)~ Figure 8 (c) is the effect diagram of different transparency stimulus blocks determined by the sampling sine coding method, where: Figure 8 (a)~ Figure 8 The alpha values ​​of (c) are 1, 0.5 and 0 respectively.

[0126] In the embodiment of the present invention, specifically, for an SSVEP stimulation block that uses a preset foreground modulation method for foreground modulation, the transparency of each SSVEP stimulation block obtained by smoothly changing the sampled sine coding is used to make multiple SSVEP stimulation blocks continuously flicker based on their respective frequencies, including:

[0127] At the current moment, for each SSVEP stimulus block after foreground modulation using a preset foreground modulation method, the transparency change sequence of the stimulus block is determined using a sampled sine coding method, and the transparency of the stimulus block is changed according to the corresponding transparency change sequence.

[0128] It can be understood that after the pixel value of each SSVEP stimulation block is multiplied by alpha(n, f, φ) after foreground modulation using the preset foreground modulation method, its own transparency can be changed according to the corresponding transparency change sequence and superimposed on the corresponding background area. The contrast with the corresponding background area changes in real time, which can avoid the situation where the color is similar to the corresponding background while generating SSVEP stimulation, and can eliminate the situation where the stimulation block is submerged in the background.

[0129] Among them, the transparency change sequence of each SSVEP stimulation block is determined at least based on its own frequency, which can be seen in formula (6). The specific frequency can be set as needed, for example, the stimulation frequency flickering within the 9-12 Hz frequency band can be achieved. When the number of SSVEP stimulation blocks is small, for example, less than the set number threshold, such as less than 16, the phase φ of all SSVEP stimulation blocks can use the same initial phase; and when the number of SSVEP stimulation blocks is greater than 16, such as when the number of SSVEP stimulation blocks is 64, 128, etc., each SSVEP stimulation block can use a different phase φ, so that each SSVEP stimulation block has not only differences in frequency but also differences in phase, so as to increase the discrimination of SSVEP stimulation.

[0130] For details about the effect of transparency changes of each SSVEP stimulus block after foreground modulation using the preset foreground modulation method and the sampled sine coding method, see Figure 9 and Figure 10 understand.

[0131] Figure 9 (a)~ Figure 9 (b) is a diagram showing the effects of single-point inversion modulation and sampled sine encoding of two background images selected in an embodiment of the present invention at a certain moment in the SSVEP paradigm; Figure 10 (a)~ Figure 10 (b) is a diagram showing the effects of global inverted modulation and sampled sine encoding of the two background images selected for the embodiment of the present invention at a certain moment in the SSVEP paradigm; these two background images are images of two typical application scenarios, and are used as background images of the dynamic background SSVEP induction paradigm. For the sake of comparison, these two background images are used in all foreground modulation and background modulation of the embodiment of the present invention.

[0132] Specifically, Figure 9 Each figure shows the effect of the SSVEP stimulus block when its transparency is 1 (i.e., opaque). Figure 9 (a) shows the effect of single-point inverted color modulation and sampled sine encoding in the drone's perspective scene, using a picture taken during the drone's flight; Figure 9 (b) shows the effects of single-point inversion modulation and sampled sinusoidal encoding for a scene from the perspective of a ground vehicle. These two background images are used to simulate real-time control scenarios for devices with cameras. These two images demonstrate the effectiveness of single-point inversion modulation and sampled sinusoidal encoding, demonstrating that the single-point inversion modulation method effectively preserves background details. This allows the operator, or subject, to better notice real-time changes in the background while focusing on the SSVEP stimulus.

[0133] Figure 10 (a) A diagram showing the effect of global inversion modulation and sampled sine encoding in the drone’s perspective scene; Figure 10 (b) is a diagram showing the effect of global inversion modulation and sampling sine coding in the scene from the perspective of ground vehicles. Figure 10 As can be seen from these two figures, the global inversion modulation method generates pure color stimulus blocks that change with the background image.

[0134] In the embodiment of the present invention, specifically, for an SSVEP stimulation block that uses a preset background modulation method for background modulation, in the embodiment of the present invention, the transparency of each SSVEP stimulation block obtained by smoothly changing the sampled sine coding is used to make multiple SSVEP stimulation blocks continuously flash based on their respective frequencies, including:

[0135] At the current moment, for each modulated SSVEP stimulus block, the stimulus block is superimposed on the modulated background block corresponding to itself at the current moment, and the transparency change sequence of the stimulus block is determined using the sampled sine coding method, and the transparency of the stimulus block is changed according to the corresponding transparency change sequence.

[0136] Wherein, each modulated SSVEP stimulation block includes each SSVEP stimulation block obtained after foreground modulation using a preset foreground modulation method or a traditional monochrome SSVEP stimulation block; the transparency change sequence of each SSVEP stimulation block is determined based on at least its own frequency, which can be seen in formula (6).

[0137] It should be noted that each modulated SSVEP stimulus block in this embodiment may include each SSVEP stimulus block obtained by foreground modulation using a preset foreground modulation method or a traditional single-color SSVEP stimulus block, and may also include SSVEP stimulus blocks obtained by other existing technologies. In this embodiment, the processing method for each modulated SSVEP stimulus block is similar to that of the previous embodiment and is not repeated here.

[0138] For the effect of each SSVEP stimulus block after background modulation using a preset background modulation method and transparency change using a sampled sine coding method, please refer to Figures 11 and 12 for understanding.

[0139] Figure 11(a) to Figure 11(b) This is a diagram showing the effects of brightness compression modulation and sampled sine encoding of two background images selected in an embodiment of the present invention at a certain moment in the SSVEP paradigm. Figure 11(a) to Figure 11(b) The images on the left correspond to the scene from the perspective of the drone and the scene from the perspective of the ground vehicle, respectively. The image on the left in the same row is only the background image generated after the brightness compression background modulation corresponding to the background image, and does not contain the stimulus block. The image on the right is an example of the SSVEP paradigm used in the actual experiment, which contains both the modulated background image and the pure white stimulus block. Figure 11(a) to Figure 11(b) It can be seen that after the brightness compression modulation method compresses the background brightness, the background blocks near the stimulus block actually achieve a display effect similar to the traditional SSVEP paradigm, that is, the white stimulus block flashes against a black background.

[0140] Figure 12(a) to Figure 12(b) This is a diagram showing the effects of background blur modulation and sampled sine coding of two background images selected in an embodiment of the present invention at a certain moment in the SSVEP paradigm. Figure 12(a) to Figure 12(b)The left image in the same row of images is only the background image generated by the background blur modulation corresponding to the background image, and does not contain the stimulus block. The right image is an example of the SSVEP paradigm used in the actual experiment, which contains both the modulated background image and the pure white stimulus block. Figure 12(a) to Figure 12(b) It can be seen that the background blur modulation method significantly weakens the details of the background, allowing the operator (subject) to better focus on the flickering of the stimulus block.

[0141] The SSVEP brain-computer interface stimulation modulation method under dynamic background provided by the embodiment of the present invention can improve the contrast between each SSVEP stimulation block and the corresponding background area by utilizing a preset foreground modulation method and / or background modulation method for the dynamic background, avoid the stimulation block being submerged in the background with a similar color, and enable the subject to more easily observe the flickering of the SSVEP stimulation block. Therefore, it can improve the efficiency of the SSVEP brain-computer interface under dynamic background.

[0142] In the second aspect, the embodiment of the present invention provides a method for decoding SSVEP brain-computer interface stimulation under a dynamic background, which is applied to the signal processing module in the brain-computer interface system. Figure 13 As shown, the method may include the following steps:

[0143] S001, pre-processing the collected SSVEP EEG signal of the subject.

[0144] It can be understood that after the signal acquisition module in the brain-computer interface system collects the subject's SSVEP EEG signal, it transmits it to the signal processing module for processing.

[0145] The execution subject of the SSVEP brain-computer interface stimulation decoding method under dynamic background proposed in the embodiment of the present invention can be a device, which corresponds to the signal processing module in the brain-computer interface system. Figure 1 Combined with the existing knowledge about brain-computer interface systems, it can be understood that the signal processing module can be a computer or other device.

[0146] It is understandable that during the process of collecting EEG signals, EEG signals are often contaminated by noise from various sources. These artifacts may come from blinking, electrocardiogram (ECG), electromyogram (EMG), and any external sources related to the equipment involved in the system. These artifacts may have amplitudes similar to those of the EEG signal and are therefore likely to interfere with the subsequent tasks. The purpose of signal preprocessing is to reduce the noise in the original signal and eliminate the artifacts in the original signal, output a clean signal, and simplify subsequent processing operations without losing relevant information, so that reliable features can be extracted in the subsequent links.

[0147] Specifically, the preprocessing process can filter out most artifacts by setting hardware filters. Usually, a high-pass filter with a cutoff frequency below 0.5Hz can be set to remove interfering extremely low-frequency components, such as breathing components. In addition, high-frequency noise can be reduced by using a low-pass filter with a cutoff frequency of approximately 50-70Hz. In addition to hardware filtering, software methods can also be used for preprocessing. Principal Component Analysis (PCA), Independent Component Analysis (ICA) and Adaptive Filters (AF) are all suitable for preprocessing in this step.

[0148] In an optional implementation manner, the preprocessing process of this step may include filtering, downsampling or normalization processing, etc.

[0149] Among them, filtering is used to eliminate noise; downsampling is used to reduce data dimension; normalization processing is used to unify data format. It can be selected according to needs and any corresponding method or device can be selected for implementation. No limitation or detailed description is given here.

[0150] In an embodiment of the present invention, the EEG signal of SSVEP is generated by a display modulation module corresponding to the display screen in the brain-computer interface system, which suspends SSVEP stimulation blocks of preset shapes at multiple preset positions of the dynamic background displayed on the display screen; uses a preset modulation method to change the contrast between each SSVEP stimulation block and the corresponding background area, and uses sampled sine coding to smoothly change the transparency of each SSVEP stimulation block obtained, so that multiple SSVEP stimulation blocks continuously flash at their respective frequencies to stimulate the subject; the preset modulation method includes a preset foreground modulation method and / or a background modulation method.

[0151] Among them, SSVEP stimulation blocks of preset shapes are respectively suspended at multiple preset positions of the dynamic background displayed on the display screen; the contrast between each SSVEP stimulation block and the corresponding background area is changed by using a preset foreground modulation method and / or background modulation method, and the transparency of each SSVEP stimulation block obtained by smoothing the change using sampled sine coding, so that multiple SSVEP stimulation blocks continuously flash based on their own frequencies to stimulate the subject to generate SSVEP EEG signals. For the process, please refer to the relevant content of the SSVEP brain-computer interface stimulation modulation method under the dynamic background provided in the first aspect, which will not be repeated here.

[0152] The preset modulation method includes a preset foreground modulation method and / or background modulation method, which means that the SSVEP brain-computer interface stimulation decoding method under a dynamic background of an embodiment of the present invention is not limited to being implemented on the basis of the SSVEP brain-computer interface stimulation modulation method under a dynamic background provided in the first aspect, but can also decode the EEG signal of SSVEP generated by the SSVEP stimulation block under the existing technology, such as decoding the EEG signal of SSVEP generated by the traditional monochrome SSVEP stimulation block flashing to stimulate the subject, or decoding the EEG signal of SSVEP generated by the SSVEP stimulation block flashing to stimulate the subject obtained by other modulation methods.

[0153] S002 , using a preset DBDN network to decode the pre-processed SSVEP EEG signal to obtain attribute information of the SSVEP stimulation block corresponding to the SSVEP EEG signal.

[0154] The purpose of decoding the EEG signal of SSVEP is to obtain the boundaries between the description classes and to mark them according to the characteristics of the signals to be identified. Specifically, in the brain-computer interface system, this task is to identify the user's intention based on the feature vector of the brain activity characteristics provided by the feature extraction step. Decoding the EEG signal of SSVEP is also generally regarded as classifying it. The purpose of decoding the EEG signal of SSVEP is to obtain the pattern corresponding to the EEG signal of SSVEP, that is, to determine which SSVEP stimulation block on the display screen is the EEG signal that stimulates the subject to produce SSVEP, so as to perform corresponding control on the device, etc. This is a very important link in the brain-computer interface system.

[0155] Therefore, the attribute information of the decoded SSVEP stimulation block can include characteristic information that can distinguish the SSVEP stimulation block, such as the position and number of the SSVEP stimulation block. Specifically, the attribute information can further include the instruction information carried by the identified SSVEP stimulation block. For example, in the direction control scenario described in the previous example, if the attribute information of the SSVEP stimulation block corresponding to the decoded SSVEP EEG signal is corresponding to the preset position "up", then its attribute information can further include the current control instruction "forward". The remaining content and scenarios of the attribute information are not given here one by one.

[0156] The existing feature extraction process for decoding SSVEP EEG signals involves extracting features from SVEP EEG signals for use in brain-computer interfaces, such as amplitude, band power, power spectral density (PSD), autoregressive, and time-frequency features. Different features correspond to different feature extraction techniques, and different signal decoding methods are also used for different features.

[0157] However, the existing SSVEP decoding algorithm has limited decoding accuracy and poor robustness under the SSVEP paradigm with dynamic background. Therefore, it is necessary to develop a decoding algorithm that conforms to the characteristics of the dynamic background stimulation paradigm to improve the decoding accuracy.

[0158] In this embodiment of the present invention, a novel network for decoding SSVEP EEG signals under dynamic backgrounds, called a Dynamic Background Decoding Network (DBDN), is pre-built and trained using sample data. Therefore, the pre-set DBDN network in this embodiment of the present invention refers to a trained DBDN network. The DBDN network can be trained using existing neural network training processes, such as gradient descent, and the specific process is not described here.

[0159] Among them, the preset DBDN network includes a feature extraction layer, a feature compression layer and a classification layer.

[0160] In an optional embodiment, the feature extraction layer includes a serial multi-scale temporal feature extraction unit, a spatial feature extraction unit, and a first activation function operation module; wherein the multi-scale temporal feature extraction unit includes a preset number of parallel temporal convolution layers with different temporal convolution kernel lengths, a batch normalization processing module corresponding to each temporal convolution layer, and a feature fusion module for fusing processing results of each batch normalization processing module; the spatial feature extraction unit includes a serial spatial convolution layer and a batch normalization processing module;

[0161] The feature compression layer includes a serial average pooling layer, a random dropout processing module, a depth-separable convolution module, a batch normalization processing module, and a second activation function operation module;

[0162] The classification layer includes a serial global average pooling layer and a third activation function operation module.

[0163] The structure of the preset DBDN network is shown in Figure 14 The preprocessed SSVEP EEG signal is input into the preset DBDN network as input data, and the feature extraction layer extracts the basic features of the SSVEP EEG signal. Figure 15 This is a schematic diagram of the SSVEP EEG signal input by the preset DBDN network of an embodiment of the present invention; the SSVEP EEG signal input by the preset DBDN network is a multi-channel time series signal with the dimension of channel*time.

[0164] The characteristics of the EEG signal of SSVEP include temporal features and spatial features. Therefore, the feature extraction layer of DBDN sequentially performs two different types of convolutions: multi-scale temporal convolution and spatial convolution.

[0165] Specifically, the multi-scale temporal feature extraction unit and the spatial feature extraction unit are shown in the dotted line boxes respectively. The multi-scale temporal feature extraction unit includes a preset number of parallel temporal convolution layers. The role of temporal convolution is similar to a bandpass frequency filter, which uses a convolution kernel of size [1, c] (where c is the convolution kernel length) to extract temporal features. Usually, the temporal convolution kernels used in the network are of uniform specifications, such as EEGNet. In the embodiment of the present invention, the temporal convolution kernel lengths used in each temporal convolution layer are different from each other, so as to extract more diverse features in the time dimension. The length of the temporal convolution kernel used can be expressed as Indicates, 2 is the attenuation factor, and n is usually a multiple of 2. In actual use, the output of each temporal convolution kernel has the same dimension as the EEG signal input of SSVEP. The output of each temporal convolution kernel is the feature of the time dimension, which can be expressed as a temporal feature map.

[0166] Figure 14 In the example, the preset number is 4, TimeConv1~TimeConv4 are four temporal convolution layers, and the lengths of the temporal convolution kernels used are c1, c2, c3, and c4 respectively. When the preset number is 4, in the multi-scale temporal feature extraction unit, the lengths of the temporal convolution kernels of the preset number of temporal convolution layers are respectively 1 / 4 of the EEG signal sampling rate of SSVEP. Where n = 1, 2, 3, 4. That is, c1, c2, c3, and c4 are 1 / 2, 1 / 4, 1 / 8, and 1 / 16 respectively.

[0167] It can be seen that the embodiment of the present invention extracts features from the original data used as network input, that is, the pre-processed SSVEP EEG signal, in the time dimension from multiple different scales or granularities.

[0168] According to the research of the embodiments of the present invention, neuroscience research models the brain as a hierarchical functional architecture with different granularity levels, wherein the lower-level granularity information contains the short-term temporal correlation of the EEG signal, and the higher-level granularity information retains a large amount of global information. Therefore, the embodiments of the present invention utilize multiple time convolution kernels of different lengths to extract temporal features from multiple scales in parallel, using a lower granularity level (i.e., a lower scale level) to extract the short-term temporal correlation of the EEG signal of SSVEP, and a higher granularity level (i.e., a higher scale level) to extract the global information of the EEG signal of SSVEP. Therefore, the data output by multiple time convolution layers contains features of different time lengths, data with a low granularity level contains the short-term local features of the data, and data with a high granularity level contains the long-term global features of the data. By analyzing data at different granularity levels, feature information at different granularity levels can be extracted, thereby enabling more sufficient and reasonable feature extraction of the data, which can greatly improve the classification accuracy when classifying the EEG signal of SSVEP in real time.

[0169] The temporal features extracted by each temporal convolution layer are batch normalized and then concatenated in the feature map dimension. The concatenated temporal features serve as the input to the spatial convolution layer.

[0170] The spatial convolution layer uses a spatial convolution kernel of size [k, 1] to extract spatial features, where k is the number of channels in the input signal. The properties of the spatial convolution kernel can effectively reduce the dimensionality of the feature map. In addition, adding spatial convolution provides a direct method for learning each temporal filter, thereby generating a spatial filter that effectively extracts specific frequencies.

[0171] The feature map extracted by spatial convolution is subjected to batch normalization operation to speed up the network convergence. In addition, after batch normalization, the first activation function operation module is used to process the nonlinear activation function, which can increase the nonlinear expression ability of the network. The activation function used by the first activation function operation module can be the ELU activation function. The expression of the ELU activation function is shown in formula (7), and its function image is shown in Figure 16 shown.

[0172]

[0173] The ELU activation function combines the sigmoid and ReLU activation functions. Its left side exhibits the saturation characteristics of a sigmoid, while its right side exhibits the linearity of a ReLU. The linearity on the right side of the ELU activation function effectively mitigates the vanishing gradient problem, while the soft saturation characteristics on the left side make it more robust to input variations or noise. Furthermore, the mean output of the ELU activation function approaches 0, so using it as an activation function can achieve faster network convergence.

[0174] The spatiotemporal features of the SSVEP EEG signal extracted by the feature extraction layer are processed by the feature compression layer. The feature compression layer searches for the most valuable features within the feature maps extracted by the feature extraction layer—those most beneficial for signal classification. Furthermore, the feature compression layer reduces computational effort and mitigates the risk of overfitting.

[0175] The feature compression layer first uses the average pooling layer to perform an average pooling operation on the spatiotemporal feature maps to aggregate the features. This average pooling operation also reduces the size of the feature maps. To prevent overfitting and improve generalization, the pooled feature maps are processed by a random dropout module using dropout technology before being fed into the depthwise separable convolution module. In an optional implementation, the dropout ratio of the random dropout module can be set to 0.5.

[0176] The feature map after the Dropout operation is subjected to the depthwise separable convolution module for depthwise separable convolution operation. The depthwise separable convolution is the core of the feature compression layer. The depthwise separable convolution consists of two parts: depthwise convolution and pointwise convolution. The principle of depthwise convolution can be found in Figure 17 As shown in Figure 2. The convolution kernels of depthwise convolution correspond one-to-one to the feature map channels, meaning one depthwise convolution kernel is responsible for one channel. The number of feature maps after depthwise convolution is the same as or an integer multiple of the number of input channels, which is determined by the depth parameter D. In this paper, the model uses D = 1.

[0177] The point-by-point convolution operation is similar to the conventional convolution, except that the convolution kernel size is fixed to [1,1]. Point-by-point convolution performs weighted summation of the feature maps generated by depthwise convolution in the depth direction to generate a new feature map. The principle of point-by-point convolution can be found in Figure 18 shown.

[0178] There are two advantages of using depthwise separable convolution to extract features: (1) it can effectively reduce the number of parameters to be fitted; (2) it first learns multiple convolution kernels that summarize the features of a single feature map, and then uses point-by-point convolution to optimally combine the features in the depth direction.

[0179] The feature map generated by the depth-wise separable convolution operation is batch normalized using the batch normalization processing module, and then processed using the second activation function operation module. The activated feature map will be used as the input of the classification layer, where the second activation function operation module can still use the ELU activation function.

[0180] Traditional convolutional neural networks (CNNs) convolve the input in the first few layers of the network to generate feature maps related to the input. For classification tasks, a common practice is to vectorize the feature maps of the last convolutional layer and input them into a fully connected layer, where a softmax activation function is used to obtain the resulting prediction value. However, fully connected layers often have large parameters. On the one hand, this increases the computational complexity of network training and testing, reducing the speed; on the other hand, this large number of parameters can easily lead to overfitting, which hinders the network's generalization ability. Furthermore, due to the black-box nature of fully connected layers, it is difficult to explain how the category information from the target layer is passed back to the previous convolutional layer.

[0181] In view of the characteristics of the fully connected layer, an embodiment of the present invention applies the global average pooling (GAP) method in the classification layer to replace the fully connected layer. The implementation of this operation requires that the number of feature maps output by the feature compression layer is consistent with the classification category. Compared with the fully connected layer, global average pooling can reduce the risk of overfitting of the fully connected layer classification, shift the network learning pressure forward, and has better interpretability when realizing the correspondence between feature maps and categories. The feature map can be easily interpreted as a category confidence map. Moreover, there are no parameters to be optimized in global average pooling, and it can itself be regarded as a structural regularizer, so this layer will not overfit. In addition, global average pooling summarizes spatial information and has better robustness to spatial transformation of the input. After the global average pooling layer, the third activation function operation module is used for processing to obtain the classification result. Among them, the third activation function operation module can adopt the softmax activation function.

[0182] Figure 19 The figure shows a comparison between the fully connected layer and the global average pooling. The fully connected layer first expands and splices the feature map of the convolutional layer, obtains the spliced ​​feature vector, and then fully connects the feature vector to the output node. Global average pooling performs average pooling of the feature map size on each feature map, and each feature map is mapped to an output node. At the end of the network, after performing the global average pooling operation on the feature map, a softmax is used to obtain the predicted value of each category. Therefore, the classification layer of the embodiment of the present invention maps the feature map directly to the corresponding output node, reducing the complexity of the network while increasing the interpretability of the network.

[0183] The classification result is a vector. The number of elements in the vector is the number of categories, that is, the number of SSVEP stimulation blocks. Each element corresponds to an SSVEP stimulation block. The element value represents the probability of being classified into that category, that is, the probability that the actual stimulation block that stimulates the EEG signal of the subject to produce SSVEP is the SSVEP stimulation block corresponding to the element.

[0184] The SSVEP brain-computer interface stimulation decoding method under a dynamic background provided by an embodiment of the present invention utilizes a pre-built and trained preset DBDN network to decode pre-processed SSVEP EEG signals to obtain attribute information of the SSVEP stimulation block corresponding to the SSVEP EEG signal. Since the SSVEP EEG signal to be decoded is generated by the display modulation module corresponding to the display screen of the brain-computer interface system, which suspends SSVEP stimulation blocks of preset shapes at multiple preset positions on the dynamic background displayed on the display screen, using a preset modulation method to change the contrast between each SSVEP stimulation block and the corresponding background area, and using sampled sine coding to smoothly change the transparency of each resulting SSVEP stimulation block, so that the multiple SSVEP stimulation blocks continuously flicker at their respective frequencies to stimulate the subject. The preset modulation method includes a preset foreground modulation method and / or a preset background modulation method. Therefore, the feature extraction layer, feature compression layer, and classification layer included in the preset DBDN network can be used to improve the decoding accuracy of the SSVEP EEG signal for the dynamic background SSVEP paradigm.

[0185] To verify the effectiveness of the SSVEP brain-computer interface stimulation decoding method under dynamic backgrounds in the embodiments of the present invention, the following experimental data is used to illustrate. The experimental results of the embodiments of the present invention will be evaluated and analyzed from both objective and subjective perspectives. The objective evaluation is mainly based on various modulation methods, and the effects of the modulation and decoding methods proposed in the embodiments of the present invention are analyzed in detail based on the accuracy of different decoding methods. The subjective evaluation mainly analyzes different modulation methods based on the subjective perception of the participants in the experiment.

[0186] (1) Objective analysis of experimental results

[0187] The main experimental parameter settings include: the EEG acquisition system is BioSemiActiveTwo; the channels used are: P3, PO7, O1, Oz, POz, Pz, P4, PO8, O2; the sampling frequency is 1024 Hz; the filter is 2-32 Hz; and the downsampling is 256 Hz.

[0188] Table 1 shows the decoding accuracy of the CCA-SVM (Canonical Correlation Analysis-Support Vector Machine) for various modulation methods. As shown in Table 1, luminance compression modulation achieved the highest decoding accuracy, followed by single-point inversion modulation, global inversion modulation, unmodulated modulation, and background blur modulation. Using the unmodulated decoding accuracy as a baseline, luminance compression modulation achieved the greatest improvement, reaching 19.59%. Single-point inversion modulation improved by 14.22% compared to unmodulated modulation; global inversion modulation improved by 12.19%. However, background blur modulation decreased by 22.34%.

[0189] Table 1 CCA-SVM results

[0190]

[0191] Table 2 shows the decoding accuracy of SSVEP-EEGNet for various modulation methods. The decoding accuracy ranking for each modulation method is the same as that for CCA-SVM: luminance compression modulation achieves the highest decoding accuracy, followed by single-point inversion modulation, global inversion modulation, unmodulated data, and background blur modulation. Using the unmodulated decoding accuracy as a baseline, luminance compression modulation improves by 27.61%, single-point inversion modulation by 13.42%, global inversion modulation by 6.57%, and background blur modulation by 22.58%.

[0192] Table 2 SSVEP-EEGNet results

[0193]

[0194]

[0195] Table 3 shows the decoding accuracy of the DBDN of the present invention for various modulation methods. The decoding accuracy rankings for each modulation method are the same as for the previous two methods: luminance compression modulation has the highest decoding accuracy, followed by single-point inversion modulation, followed by global inversion modulation, unmodulated modulation, and background blur modulation. Using the unmodulated decoding accuracy as a baseline, luminance compression modulation improved by 24.68%; single-point inversion modulation improved by 20.72%; global inversion modulation improved by 18.56%; and background blur modulation decreased by 20.63%.

[0196] Table 3 DBDN results display

[0197]

[0198] Figure 20This figure compares the average accuracy of various decoding algorithms under different modulation methods in the experiments of the present invention. Three decoding methods are included for single-point inversion, global inversion, luminance compression, background blur, and unmodulated. The left, middle, and right columns represent CCA-SVM, SSVEP-EEGNet, and DBDN, respectively. It can be seen that the dynamic background SSVEP brain-computer interface stimulation decoding method proposed in the present invention, referred to as DBDN, outperforms the other two comparison algorithms in decoding performance under all modulation methods. In single-point inversion modulation, DBDN's decoding accuracy is 11.52% higher than CCA-SVM and 13.55% higher than SSVEP-EEGNet; in global inversion modulation, DBDN's decoding accuracy is 11.39% higher than CCA-SVM and 18.24% higher than SSVEP-EEGNet; in brightness compression modulation, DBDN's decoding accuracy is 10.11% higher than CCA-SVM and 3.32% higher than SSVEP-EEGNet; in background blur modulation, DBDN's decoding accuracy is 6.73% higher than CCA-SVM and 8.20% higher than SSVEP-EEGNet; even in the unmodulated method, DBDN's decoding accuracy is 5.02% higher than CCA-SVM and 6.25% higher than SSVEP-EEGNet.

[0199] Combined with Tables 1 to 3, and Figure 20 , a comparative analysis of the decoding accuracy of each subject under each modulation method shows that there are obvious differences in the degree of adaptation of the subjects to the SSVEP elicitation paradigm involved in the embodiment of the present invention. Subject 4 performed better than other subjects in all five modulation methods, while subject 6 performed significantly worse than other subjects in the four modulation methods except the brightness compression modulation method. From the perspective of decoding accuracy, the accuracy of brightness compression modulation ranks first, followed by single-point inversion modulation, followed by global inversion, unmodulated, and background blur. The decoding accuracy of the DBDN of the embodiment of the present invention is the best under each modulation method.

[0200] It should be noted that the different modulation methods in the experimental part: single-point inversion, global inversion, brightness compression, and background blur refer to the use of single-point inversion modulation method, global inversion modulation method, brightness compression modulation method, and background blur modulation method to modulate the SSVEP stimulus block or background and complete the sampling sinusoidal encoding process of the stimulus block.

[0201] (2) Subjective analysis of experimental results

[0202] This part of the experiment in the embodiment of the present invention can use a subjective evaluation form to let the subjects evaluate and score. The subjective evaluation form can be seen in Table 4.

[0203] Table 4 Subjective evaluation table

[0204]

[0205] Figure 21 The average scores of the four subjective questions Q1 to Q4 under each modulation method are compared. Figure 21 It can be seen that the display effects of the four modulation methods proposed in the embodiments of the present invention are all better than those of unmodulated ones. Single-point inversion modulation has the least impact on actual operation, possibly because the stimulation block generated by single-point operation retains all the details of the background; background blur modulation has the greatest impact on actual operation, possibly because the defocus blur used for background blur weakens the details of the background, making it difficult to grasp the real-time situation of the background; global inversion and brightness compression modulation have a lower impact on operation than background blur modulation, and slightly higher than the unmodulated method. Background blur modulation has the highest viewing comfort, possibly because this modulation method creates a transition effect between the background and the stimulation block; brightness compression modulation has the second highest viewing comfort, global inversion modulation ranks third, and single-point inversion modulation has the lowest viewing comfort, consistent with unmodulated ones. After the experiment, the subjects' fatigue level was inversely proportional to their comfort level. The subjects under single-point inversion modulation had the highest fatigue level, possibly because the stimulation block retained too much background detail, which caused the greatest mental exhaustion to the subjects. That is, (1) brightness compression and background blurring have the best display effect at the same time; (2) single-point color inversion has the least impact on operation; (3) background blurring has the highest subject comfort; (4) background blurring has the lowest subject fatigue.

[0206] In summary, the embodiments of the present invention propose four different SSVEP modulation methods for dynamic backgrounds. From the perspective of decoding accuracy, the three modulation methods, single-point inversion, global inversion, and brightness compression, all outperform unmodulated methods, significantly improving decoding accuracy. Among them, brightness compression modulation has the highest decoding accuracy, followed by single-point inversion modulation, and global inversion modulation ranks third, while background blur modulation has a lower decoding accuracy than unmodulated methods. From a comprehensive subjective and objective perspective, single-point inversion modulation has the least impact on actual operation but can easily cause subject fatigue; global inversion modulation is slightly lower than single-point inversion modulation in decoding accuracy, but it also alleviates subject fatigue to a certain extent; brightness compression modulation has the highest recognition accuracy under all decoding algorithms; background blur modulation, at the expense of decoding accuracy, offers the highest subject comfort and is more user-friendly. Compared to unmodulated methods, the four modulation methods proposed in the embodiments of the present invention all show improvements in certain aspects, demonstrating the feasibility of the four modulation methods.

[0207] Furthermore, the proposed decoding algorithm, DBDN, was validated on a dataset collected during a dynamic SSVEP experiment. Its decoding accuracy across all modulation schemes outperformed the baseline algorithms, CCA-SVM and SSVEP-EEGNet, used for comparison. Across modulation schemes, DBDN achieved classification accuracy improvements of up to 11.52% compared to CCA-SVM and up to 18.24% compared to SSVEP-EEGNet, demonstrating the effectiveness of the DBDN algorithm for SSVEP decoding in dynamic environments.

[0208] The above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A SSVEP brain-computer interface stimulation modulation method under dynamic background, characterized in that: The display modulation module corresponding to the display screen in the brain-computer interface system is applied. The SSVEP brain-computer interface stimulation modulation method under dynamic background includes: SSVEP stimulation blocks of preset shapes are suspended at multiple preset positions of the dynamic background displayed on the display screen; Using a preset foreground modulation method and / or background modulation method to change the contrast between each SSVEP stimulus block and the corresponding background area, and using sampled sine coding to smoothly change the transparency of each SSVEP stimulus block, so that multiple SSVEP stimulus blocks continuously flash based on their respective frequencies to stimulate the subject to generate SSVEP EEG signals; The preset foreground modulation method includes: For each SSVEP stimulus block, the three-channel data of each pixel in the background image within the range of the SSVEP stimulus block at the current moment are complemented by the pixel maximum value 255 to obtain the complemented three-channel data of each pixel; The three-channel data of the same pixel after complementation are merged, and the SSVEP stimulus block after foreground modulation of the SSVEP stimulus block at the current moment is obtained using all the merged pixels; The preset background modulation method includes: For each SSVEP stimulus block, convert the pixel format of the image within the range of the background block surrounding the SSVEP stimulus block at the current moment into the HSV format, and obtain the three-channel data of each pixel in the background block in the HSV format; Compressing the V channel data of each pixel in the background block in the HSV format according to a preset brightness compression ratio; The three-channel data of the same pixel after brightness compression are merged, and all the merged pixels are used to obtain the modulated background block corresponding to the SSVEP stimulus block at the current moment; The method of using sampled sine coding to smoothly change the transparency of each SSVEP stimulation block, so that the multiple SSVEP stimulation blocks continuously flicker based on their respective frequencies, includes: At the current moment, for each modulated SSVEP stimulus block, the stimulus block is superimposed on the modulated background block corresponding to itself at the current moment, and the transparency change sequence of the stimulus block is determined using the sampled sine coding method, and the transparency of the stimulus block is changed according to the corresponding transparency change sequence; wherein, each modulated SSVEP stimulus block includes each SSVEP stimulus block obtained after foreground modulation using the preset foreground modulation method or a traditional monochrome SSVEP stimulus block; the transparency change sequence of each SSVEP stimulus block is determined based on at least its own frequency.

2. The SSVEP brain-computer interface stimulation modulation method under dynamic background according to claim 1, characterized in that The preset foreground modulation method further includes: For each SSVEP stimulus block, the three-channel data of each pixel in the background image within the range of the SSVEP stimulus block at the current moment are complemented by the pixel maximum value 255 to obtain the complemented three-channel data of each pixel; Calculate the mean of the complemented data of all pixels in the same channel to obtain the updated data of the corresponding channel; The updated data of the three channels are merged to obtain the updated pixel of each pixel, and all the updated pixels are used to obtain the SSVEP stimulation block after the foreground modulation of the SSVEP stimulation block at the current moment.

3. The SSVEP brain-computer interface stimulation modulation method under dynamic background according to claim 1, characterized in that: The preset background modulation method further includes: For each SSVEP stimulation block, a defocus blur operation is performed on each pixel point in the image within the range of the background block surrounding the SSVEP stimulation block at the current moment, and the modulated background block corresponding to the SSVEP stimulation block at the current moment is obtained by using each pixel point processed by the defocus blur operation.

4. The SSVEP brain-computer interface stimulation modulation method under dynamic background according to any one of claims 1 to 3, characterized in that: The method further comprises: using the sampled sine code to smoothly change the transparency of each SSVEP stimulation block, so that the multiple SSVEP stimulation blocks continuously flicker based on their respective frequencies; At the current moment, for each SSVEP stimulation block after foreground modulation using the preset foreground modulation method, the transparency change sequence of the stimulation block is determined using the sampled sine coding method, and the transparency of the stimulation block is changed according to the corresponding transparency change sequence; wherein the transparency change sequence of each SSVEP stimulation block is determined at least based on its own frequency.

5. A method for decoding SSVEP brain-computer interface stimulation under dynamic background, characterized in that: The signal processing module applied to the brain-computer interface system, the SSVEP brain-computer interface stimulation decoding method under dynamic background includes: Preprocessing the collected SSVEP EEG signals of the subject; wherein the SSVEP EEG signals are generated by suspending SSVEP stimulation blocks of preset shapes at multiple preset positions of the dynamic background displayed on the display screen by a display modulation module corresponding to the display screen in the brain-computer interface system; changing the contrast between each SSVEP stimulation block and the corresponding background area by a preset modulation method, and smoothly changing the transparency of each SSVEP stimulation block obtained by sampling sinusoidal coding, so that multiple SSVEP stimulation blocks continuously flash based on their respective frequencies to stimulate the subject; the preset modulation method includes a preset foreground modulation method and / or a background modulation method; wherein the processing of the display modulation module corresponding to the display screen in the brain-computer interface system is implemented based on the SSVEP brain-computer interface stimulation modulation method under the dynamic background according to any one of claims 1-4; The pre-processed SSVEP EEG signal is decoded using a preset DBDN network to obtain attribute information of the SSVEP stimulation block corresponding to the SSVEP EEG signal; wherein the preset DBDN network includes a feature extraction layer, a feature compression layer and a classification layer.

6. The SSVEP brain-computer interface stimulation decoding method under dynamic background according to claim 5, characterized in that The feature extraction layer includes a serial multi-scale temporal feature extraction unit, a spatial feature extraction unit, and a first activation function operation module; wherein the multi-scale temporal feature extraction unit includes a preset number of parallel temporal convolution layers with different temporal convolution kernel lengths, a batch normalization processing module corresponding to each temporal convolution layer, and a feature fusion module for fusing processing results of each batch normalization processing module; the spatial feature extraction unit includes a serial spatial convolution layer and a batch normalization processing module; The feature compression layer includes an average pooling layer, a random drop processing module, a depth-separable convolution module, a batch normalization processing module and a second activation function operation module in series; The classification layer includes a serial global average pooling layer and a third activation function operation module.

7. The SSVEP brain-computer interface stimulation decoding method under dynamic background according to claim 6, characterized in that: The preset number is 4; In the multi-scale time feature extraction unit, the lengths of the temporal convolution kernels of the preset number of temporal convolution layers are respectively the SSVEP EEG signal sampling rate ,in, .

Images