Control method and device of brain-computer interface device and brain-computer interface device

Abstract

The embodiment of the present application provides a control method, device and brain-computer interface device for a brain-computer interface device. In the control method of the above-mentioned brain-computer interface device, after the brain-computer interface device obtains the user's EEG data, it extracts multimodal features from the above-mentioned EEG data and the image data displayed by the brain-computer interface device. Then, the brain-computer interface device obtains a fuzzy set corresponding to the above-mentioned multimodal features based on the above-mentioned multimodal features, and then obtains a control instruction corresponding to the above-mentioned multimodal features based on the fuzzy set corresponding to the above-mentioned multimodal features. Finally, the brain-computer interface device executes the above-mentioned control instruction, thereby realizing the control of the brain-computer interface device through the user's EEG data and the image data displayed by the brain-computer interface device, improving the interactivity between the user and the brain-computer interface device and the user's immersion, and bringing a more intelligent control experience to the user.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent terminal technology, and in particular to a control method and apparatus for a brain-computer interface device, and a brain-computer interface device. Background Art

[0002] Virtual reality (VR), as the name suggests, is the fusion of virtuality and reality. Theoretically, VR technology is a computer simulation system that can create and experience a virtual world. VR technology uses computers to generate a simulated environment, immersing the user in it.

[0003] VR technology is gaining increasing recognition among users. VR devices offer a truly immersive experience, with simulated environments so realistic they're indistinguishable from the real world, giving users a sense of being there. Furthermore, VR devices offer auditory, visual, tactile, gustatory, and / or olfactory sensations. Finally, VR devices boast powerful simulation systems, enabling true human-computer interaction, allowing users to freely manipulate the environment and receive realistic feedback. VR devices are gaining popularity with a growing number of users precisely because of their presence, multi-sensory nature, and interactivity.

[0004] However, the solutions for controlling VR devices provided by existing related technologies have the problems of poor interactivity between users and VR devices and weak user immersion. Summary of the Invention

[0005] The embodiments of the present application provide a control method, device and brain-computer interface device for a brain-computer interface device. The embodiments of the present application also provide a computer-readable storage medium to control the brain-computer interface device through the user's EEG data and image data displayed by the brain-computer interface device, thereby improving the interactivity between the user and the brain-computer interface device and the user's immersion, and bringing a more intelligent control experience to the user.

[0006] In a first aspect, an embodiment of the present application provides a method for controlling a brain-computer interface device, comprising: the brain-computer interface device obtains the user's EEG data; extracts multimodal features from the EEG data and image data displayed by the brain-computer interface device; obtains a fuzzy set corresponding to the multimodal feature based on the multimodal feature; obtains a control instruction corresponding to the multimodal feature based on the fuzzy set corresponding to the multimodal feature; and executes the control instruction.

[0007] In the control method of the above-mentioned brain-computer interface device, after the brain-computer interface device obtains the user's EEG data, it extracts multimodal features from the EEG data and the image data displayed by the brain-computer interface device. Then, based on the multimodal features, the brain-computer interface device obtains a fuzzy set corresponding to the multimodal features, and then, based on the fuzzy set corresponding to the multimodal features, obtains a control instruction corresponding to the multimodal features. Finally, the brain-computer interface device executes the control instruction, thereby enabling the brain-computer interface device to be controlled through the user's EEG data and the image data displayed by the brain-computer interface device, improving the interactivity between the user and the brain-computer interface device and the user's sense of immersion, and providing the user with a more intelligent control experience.

[0008] In one possible implementation, obtaining the fuzzy set corresponding to the multimodal feature based on the multimodal feature includes: performing feature mapping on the multimodal feature; and fuzzifying the result of the feature mapping to obtain the fuzzy set corresponding to the multimodal feature.

[0009] In one possible implementation, obtaining the control instruction corresponding to the multimodal feature according to the fuzzy set corresponding to the multimodal feature includes: obtaining a relationship matrix according to the fuzzy set corresponding to the multimodal feature and a preset control accuracy set; selecting a fuzzy output result from the relationship matrix according to the fuzzy membership; and obtaining the control instruction corresponding to the multimodal feature according to the fuzzy output result.

[0010] In one possible implementation, the extracting multimodal features from the EEG data and the image data displayed by the brain-computer interface device includes: encoding the EEG data to obtain an EEG vector; and encoding the image data to obtain an image vector; performing feature conversion on the EEG vector to obtain an EEG feature vector, and performing feature conversion on the image vector to obtain an image feature vector; and obtaining the multimodal features based on the EEG feature vector and the image feature vector.

[0011] In one possible implementation, encoding the EEG data to obtain an EEG vector includes: performing discrete wavelet transform on the EEG data of each channel to obtain an EEG two-dimensional feature map of each channel; wherein the EEG two-dimensional feature map of each channel includes time domain and frequency domain features of each channel; merging the obtained EEG two-dimensional feature maps to obtain three-dimensional EEG time-frequency domain features; performing feature extraction on the three-dimensional EEG time-frequency domain features on each channel to obtain a first depth feature of each channel in the time dimension; and performing cross-channel feature extraction on the three-dimensional EEG time-frequency domain features to obtain a second depth feature between channels in the spatial dimension; and obtaining the EEG vector based on the first depth feature and the second depth feature.

[0012] In one possible implementation, the image data includes local image data of the user's viewpoint at the current moment in the image displayed by the brain-computer interface device; encoding the image data to obtain an image vector includes: performing image preprocessing on the local image data to obtain a viewpoint focus image; using a basic convolutional layer and a depth residual layer to extract a depth feature image from the viewpoint focus image; and obtaining the image vector based on the depth feature image.

[0013] In one possible implementation, the fuzzifying the result of the feature mapping to obtain the fuzzy set corresponding to the multimodal feature includes: obtaining the Euclidean distance between the result of the feature mapping and the cluster center corresponding to the result of the feature mapping; discretizing the Euclidean distance to obtain a discrete domain value corresponding to the Euclidean distance; determining the control adjustment amount corresponding to the multimodal feature based on the discrete domain value corresponding to the Euclidean distance; and obtaining the fuzzy set corresponding to the multimodal feature based on the control adjustment amount corresponding to the multimodal feature.

[0014] In a second aspect, an embodiment of the present application provides a control device for a brain-computer interface device, which is included in the brain-computer interface device and has the function of implementing the behavior of the brain-computer interface device in the first aspect and the possible implementation of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, an acquisition module, an extraction module, and an execution module.

[0015] In a third aspect, an embodiment of the present application provides a brain-computer interface device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the brain-computer interface device, enable the brain-computer interface device to execute the method provided in the first aspect.

[0016] It should be understood that the second and third aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0019] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the structure of a brain-computer interface device provided in one embodiment of the present application;

[0021] Figure 2 A flowchart of a method for controlling a brain-computer interface device according to one embodiment of the present application;

[0022] Figure 3 A flowchart of a method for controlling a brain-computer interface device provided in another embodiment of the present application;

[0023] Figure 4 A flowchart of a method for controlling a brain-computer interface device provided in yet another embodiment of the present application;

[0024] Figure 5 A schematic structural diagram of a brain-computer interface device provided in another embodiment of the present application;

[0025] Figure 6 A schematic structural diagram of a brain-computer interface device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0026] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0027] Based on the problems of poor interactivity between users and brain-computer interface devices and weak user immersion in existing related technologies, an embodiment of the present application provides a control method for a brain-computer interface device, which can control the brain-computer interface device using fuzzy control based on the user's EEG data and the image data displayed by the brain-computer interface device, thereby improving the interactivity between the user and the brain-computer interface device and the user's immersion, and bringing a more intelligent control experience to the user.

[0028] The following first introduces the fuzzy control and EEG data mentioned in the embodiments of this application.

[0029] 1. Fuzzy control

[0030] Fuzzy control is a control method based on fuzzy logic, which is used to deal with complex, uncertain or fuzzy systems. Fuzzy control is a rule-based control method that uses fuzzy reasoning and fuzzy rules to make control decisions.

[0031] In fuzzy control, input and output variables are defined as fuzzy variables, which can have fuzzy, imprecise values. Unlike traditional precise control methods, fuzzy control handles fuzzy variables by using fuzzy sets and fuzzy rules.

[0032] The following are the general steps of fuzzy control:

[0033] Fuzzification: Converting input variables from real values ​​to fuzzy values. This involves mapping the real values ​​to a set of fuzzy sets that represent the fuzzy properties of the variables. Common mapping methods include triangular functions, trapezoidal functions, and others.

[0034] Rule base: Defines a set of fuzzy rules that describe the relationship between input variables and output variables. Fuzzy rules are usually in the form of "if-then" statements and are based on empirical knowledge or expert experience.

[0035] Inference engine: An inference engine uses fuzzy rules to derive fuzzy output variables from fuzzy input variables. Common inference methods include fuzzy association, fuzzy reasoning, and / or fuzzy conjunction.

[0036] Defuzzification: Converting fuzzy output variables to real values. This involves mapping the fuzzy variables back to real values ​​to obtain the control output of the system. Common defuzzification methods include maximum and / or weighted average methods.

[0037] Fuzzy control has a wide range of applications in many fields, including automation, robotics, traffic control, and power system control. Fuzzy control can handle nonlinear, time-varying, fuzzy, and uncertain systems, and it has low requirements for system modeling and strong adaptability. However, fuzzy control also has some limitations, such as the need for domain expertise in rule design and fuzzy set selection, and the potential for a large number of rules when dealing with complex systems.

[0038] 2. EEG data

[0039] Electroencephalogram (EEG) data is a recording obtained by measuring electrical activity in the brain's neurons. EEG data reflects the electrical activity of neurons in the brain and is used to study and diagnose various conditions and diseases related to brain function. EEG data is measured by placing electrodes on the scalp. These electrodes record changes in electrical potential in different brain regions, forming a multi-channel voltage time series.

[0040] 1) EEG Data Characteristics: EEG data typically has a wide frequency spectrum ranging from low to high frequencies. EEG waves in different frequency bands include δ (0.5-4 Hz), θ (4-8 Hz), α (8-13 Hz), β (13-30 Hz), and γ (30 Hz and above).

[0041] 2) Application areas: EEG data is widely used in fields such as neuroscience, clinical medicine, and cognitive psychology. It can be used to study brain functions such as sleep, consciousness, attention, emotion, and / or memory, as well as to diagnose and monitor diseases such as epilepsy, sleep disorders, and / or brain damage.

[0042] 3) Data Analysis Methods: Common methods for analyzing EEG data include time domain analysis, frequency domain analysis, and time-frequency domain analysis. Time domain analysis focuses on the amplitude and waveform characteristics of EEG signals, frequency domain analysis is used to study EEG activity in different frequency bands, and time-frequency domain analysis combines both time and frequency information.

[0043] 4) Preprocessing: EEG data preprocessing typically includes filtering, denoising, and artifact removal. Filtering is used to remove frequency components of no interest. Denoising methods can include mean filtering, wavelet denoising, and / or independent component analysis.

[0044] 5) Feature extraction: Extracting useful features from EEG data is a key step for further analysis and research. Common features include spectral power, time-domain statistical characteristics, coherence, and / or phase synchronization.

[0045] 6) Machine Learning Applications: EEG data is also widely used in machine learning, for example, in the classification and control of brain-computer interfaces (BCIs), as well as in tasks such as emotion recognition and cognitive load assessment based on EEG.

[0046] The control method of the brain-computer interface device provided in the embodiment of the present application can be applied to the brain-computer interface device, wherein the above-mentioned brain-computer interface device can be a smart phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / VR device, a laptop computer or an ultra-mobile personal computer (UMPC) and other devices; the embodiment of the present application does not impose any restrictions on the specific type of the brain-computer interface device. In the following embodiments of the present application, the brain-computer interface device 100 is taken as a VR device as an example for illustration.

[0047] For example, Figure 1 A schematic diagram of the structure of a brain-computer interface device provided in one embodiment of the present application is shown in FIG. Figure 1As shown, the brain-computer interface device 100 includes a processor 110 and a communication interface 120. Optionally, the brain-computer interface device 100 may also include a memory 130. The processor 110, the communication interface 120, and the memory 130 may communicate with each other through internal connection paths to transmit control and / or data signals. The memory 130 is used to store computer programs, and the processor 110 is used to call and execute the computer programs from the memory 130.

[0048] The processor 110 and the memory 130 may be combined into a processing device, or more commonly, they are independent components. The processor 110 is used to execute program codes stored in the memory 130. In a specific implementation, the memory 130 may also be integrated into the processor 110, or independent of the processor 110.

[0049] In addition, in order to make the functions of the brain-computer interface device 100 more complete, the brain-computer interface device 100 may also include one or more of an input unit 160 and a display unit 170 .

[0050] Optionally, the brain-computer interface device 100 may further include a power supply 150 for providing power to various devices or circuits in the brain-computer interface device 100 .

[0051] It should be understood that Figure 1 The processor 110 in the brain-computer interface device 100 shown may be a system on a chip (SOC), which may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU).

[0052] For ease of understanding, the following examples of this application will be described with Figure 1 Taking the brain-computer interface device of the structure shown as an example, the control method of the brain-computer interface device provided by the embodiment of the present application is specifically explained in combination with the accompanying drawings and application scenarios.

[0053] Figure 2 A flowchart of a method for controlling a brain-computer interface device provided in one embodiment of the present application is shown in FIG. Figure 2 As shown, the control method of the above-mentioned brain-computer interface device may include:

[0054] Step 201: The brain-computer interface device 100 obtains the user's EEG data.

[0055] In a specific implementation, electrodes can be placed on the user's scalp to measure the user's EEG data. The brain-computer interface device 100 is connected to the electrodes to obtain the EEG data measured by the electrodes. In this step, the connection between the brain-computer interface device 100 and the electrodes can be mechanical or communication, which is not limited in this embodiment.

[0056] In step 202 , the brain-computer interface device 100 extracts multimodal features from the above-mentioned EEG data and the image data displayed by the brain-computer interface device 100 .

[0057] In this embodiment, the user can interact with the brain-computer interface device 100 through EEG data to control the brain-computer interface device 100 to execute instructions. After the brain-computer interface device 100 executes the instructions, it will display an image corresponding to the instructions.

[0058] After the brain-computer interface device 100 obtains the user's EEG data, multimodal features can be extracted from the above-mentioned EEG data and the image data displayed by the brain-computer interface device 100, and then the multimodal features can be used to control the brain-computer interface device 100. Since the above-mentioned multimodal features are extracted from the EEG data and the image data displayed by the brain-computer interface device 100, the interactivity between the user and the brain-computer interface device 100 and the user's immersion can be enhanced.

[0059] In step 203 , the brain-computer interface device 100 obtains a fuzzy set corresponding to the multimodal features according to the multimodal features.

[0060] In step 204 , the brain-computer interface device 100 obtains a control instruction corresponding to the multimodal feature according to the fuzzy set corresponding to the multimodal feature.

[0061] Specifically, the brain-computer interface device 100 obtains the control instructions corresponding to the above-mentioned multimodal features based on the fuzzy sets corresponding to the above-mentioned multimodal features by: obtaining a relationship matrix based on the fuzzy sets corresponding to the multimodal features and a pre-set control accuracy set; then, selecting a fuzzy output result from the above-mentioned relationship matrix based on the fuzzy membership; finally, obtaining the control instructions corresponding to the above-mentioned multimodal features based on the above-mentioned fuzzy output results.

[0062] Among them, according to the fuzzy set corresponding to the multimodal feature and the preset control precision set, the relationship matrix can be obtained by calculating the Cartesian product of the fuzzy set corresponding to the multimodal feature and the preset control precision set, and using the calculated Cartesian product as the relationship matrix.

[0063] Selecting the fuzzy output result from the above relationship matrix according to the fuzzy membership can be: selecting the component with the largest fuzzy membership from the above relationship matrix as the fuzzy output result.

[0064] Obtaining the control instruction corresponding to the multimodal feature according to the fuzzy output result may be: using a defuzzification method, such as a maximum method and / or a weighted average method, to map the fuzzy output result to the control instruction.

[0065] Step 205: The brain-computer interface device 100 executes the above control instructions.

[0066] In the control method of the above-mentioned brain-computer interface device, after the brain-computer interface device 100 obtains the user's EEG data, it extracts multimodal features from the EEG data and the image data displayed by the brain-computer interface device 100. Then, based on the multimodal features, the brain-computer interface device 100 obtains a fuzzy set corresponding to the multimodal features, and then obtains a control instruction corresponding to the multimodal features based on the fuzzy set corresponding to the multimodal features. Finally, the brain-computer interface device 100 executes the control instruction, thereby realizing control of the brain-computer interface device 100 through the user's EEG data and the image data displayed by the brain-computer interface device, improving the interactivity between the user and the brain-computer interface device and the user's immersion, and providing the user with a more intelligent control experience.

[0067] Figure 3 A flowchart of a method for controlling a brain-computer interface device provided in another embodiment of the present application is shown in FIG. Figure 3 As shown, this application Figure 2 In the illustrated embodiment, step 202 may include:

[0068] In step 301 , the brain-computer interface device 100 encodes the EEG data to obtain an EEG vector; and encodes the image data to obtain an image vector.

[0069] In some examples, the brain-computer interface device 100 encodes the EEG data to obtain an EEG vector. This can be done by: the brain-computer interface device 100 performs a discrete wavelet transform on the EEG data of each channel to obtain a two-dimensional EEG feature map for each channel; wherein the two-dimensional EEG feature map of each channel includes the time domain and frequency domain features of each channel. Then, the brain-computer interface device 100 merges the obtained two-dimensional EEG feature maps to obtain three-dimensional EEG time-frequency domain features; on each channel, performs feature extraction on the above-mentioned three-dimensional EEG time-frequency domain features to obtain a first depth feature of each channel in the time dimension; and performs cross-channel feature extraction on the above-mentioned three-dimensional EEG time-frequency domain features to obtain a second depth feature between channels in the spatial dimension. Finally, the brain-computer interface device 100 obtains the above-mentioned EEG vector based on the first depth feature and the second depth feature.

[0070] In some examples, the above-mentioned image data may include local image data of the user's viewpoint at the current moment in the image displayed by the brain-computer interface device 100; the brain-computer interface device 100 encodes the above-mentioned image data, and obtaining the image vector can be: performing image preprocessing on the above-mentioned local image data to obtain a viewpoint focus image; using the basic convolutional layer and the depth residual layer to extract the depth feature image from the above-mentioned viewpoint focus image; and obtaining the above-mentioned image vector based on the above-mentioned depth feature image.

[0071] In step 302 , the brain-computer interface device 100 performs feature conversion on the EEG vector to obtain an EEG feature vector, and performs feature conversion on the image vector to obtain an image feature vector.

[0072] In step 303 , the brain-computer interface device 100 obtains the multimodal feature according to the EEG feature vector and the image feature vector.

[0073] In specific implementation, the brain-computer interface device 100 can use an EEG feature encoder to encode the EEG data to obtain a one-dimensional EEG vector; and use a visual encoder to encode the above image data to obtain a one-dimensional image vector.

[0074] Then, the brain-computer interface device 100 uses a set of 1×1 convolutions to perform a convolution operation on the one-dimensional EEG vector to obtain an EEG feature vector; and uses another set of 1×1 convolutions to perform a convolution operation on the one-dimensional image vector to obtain an image feature vector; the above image feature vector has the same dimension as the EEG feature vector.

[0075] Finally, the brain-computer interface device 100 can perform positional addition on the EEG feature vector and the image feature vector, and then activate them using a sigmoid function to obtain multimodal features.

[0076] The following describes the process of the brain-computer interface device 100 encoding EEG data using the EEG feature encoder to obtain a one-dimensional EEG vector. The process of encoding EEG data may include:

[0077] Step 1: Use a bandpass filter to process the EEG data to filter out high-frequency signals and low-frequency signals outside the band;

[0078] Step 2: Process each channel of the filtered EEG data using discrete wavelet transform to obtain a two-dimensional EEG feature map of each channel; wherein the two-dimensional EEG feature map of each channel includes time domain and frequency domain features of each channel;

[0079] Step 3: Merge the two-dimensional EEG feature maps of all channels to obtain three-dimensional EEG time-frequency domain features;

[0080] Step 4: Use 1×1 convolution on each channel to extract the above three-dimensional EEG time-frequency domain features to obtain the first depth feature of each channel in the time dimension;

[0081] Step 5: Use 1×1 convolution to perform cross-channel feature extraction on the above 3D EEG time-frequency domain features to obtain the second depth feature between channels in the spatial dimension;

[0082] Step 6: Use spatial average pooling to compress the first depth feature and the second depth feature into a one-dimensional vector to obtain a one-dimensional EEG vector.

[0083] The following describes a process in which the brain-computer interface device 100 uses a visual encoder to encode the image data to obtain a one-dimensional image vector. The process of encoding the image data may include:

[0084] Step 1: Process the image data through the basic convolution layer. In this step, you can specify the number of channels of the input feature map to be 3, the number of channels of the output feature map to be 32, the convolution kernel size to be 3, the jump step size to be 2, and the number of padded pixel columns to be 1. After this step, the width and height of the feature map are halved, and the number of channels becomes 32. The result is recorded as c1.

[0085] Step 2: Process the result of step 1 through the basic convolution layer. In this step, you can specify the number of channels of the input feature map to be 32, the number of channels of the output feature map to be 64, the convolution kernel size to be 3, the jump step size to be 2, and the number of padded pixel columns to be 1. After this step, the width and height of the feature map are halved, the number of channels becomes 64, and the result is recorded as c2.

[0086] Step 3: Process the result of step 2 through the depth residual layer. In this step, the parameters C1 and C2 can be specified as 64 and 64, respectively. After this step, the width and height of the feature map are halved, and the number of channels becomes 128. The result is recorded as r1.

[0087] Step 4: Process the result of step 3 through the depth residual layer. In this step, you can specify parameters C1 and C2 as 128. After this step, the width and height of the feature map are halved, the number of channels becomes 256, and the result is recorded as r2.

[0088] Step 5: Process the result of step 4 through the depth residual layer. In this step, you can specify parameters C1 and C2 as 256. After this step, the width and height of the feature map are halved, the number of channels becomes 256, and the result is recorded as r3.

[0089] Step 6: Process the result of step 5 through the depth residual layer. In this step, you can specify the number of channels of the input feature map to be 512, the number of channels of the output feature map to be 512, the convolution kernel size to be 3, the jump step size to be 1, and the number of padded pixel columns to be 1. After this step, the width and height of the feature map remain unchanged, the number of channels becomes 512, and the result is recorded as c3.

[0090] Step 7: Use spatial average pooling to perform convolution on c3 to obtain a feature vector c4 of size 1×1×512.

[0091] Step 8: Use a set of 1×1 convolutions to perform a convolution operation on the feature vector c4 to obtain a one-dimensional image vector; wherein the one-dimensional image vector has the same dimension as the one-dimensional EEG vector.

[0092] The following describes the calculation process of the depth residual layer used in the encoding process of image data, which may include:

[0093] Step 1: Assume that the size of the input feature map is W×H×C1, where W represents the width of the input feature map, H represents the height of the input feature map, and C1 represents the number of channels of the input feature map. The embodiment of the present application uses a square image, so W is equal to H, and the input feature map is represented by U.

[0094] Step 2: Process the input feature map U through the basic convolution layer. In this step, the number of channels of the input feature map can be specified as C1, the number of channels of the output feature map can be specified as C2, the convolution kernel size can be specified as 3, the jump step size can be specified as 1, and the number of padded pixel columns can be specified as 1. Specifically, the output feature map can be obtained according to formula (1).

[0095]

[0096] In formula (1), W represents the width of the input feature map, F represents the size of the convolution kernel, P represents the number of columns of padded pixels, S represents the jump step size, and N represents the width of the output feature map. Since a square image is used, the height of the output feature map is equal to N. After this step, the width and height of the feature map remain unchanged, and the number of channels becomes C2. The result of this step is recorded as R.

[0097] Step 3: Channel splitting. Split the output feature map R obtained in step 2 into the left half and the right half according to the channel direction, and retain the right half, which is recorded as R2.

[0098] In step 4, the feature map R2 is processed by the basic convolution layer. In this step, the number of channels of the input feature map can be specified as C2 / 2, the number of channels of the output feature map can be specified as C2 / 2, the convolution kernel size can be specified as 3, the jump step size can be specified as 1, and the number of padded pixel columns can be specified as 1. The calculation method of step 4 is consistent with that of step 2. After step 4, the width and height of the feature map remain unchanged, the number of channels becomes C2 / 2, and the result is recorded as K1.

[0099] In step 5, the feature map K1 is processed by the basic convolution layer. In this step, the number of channels of the input feature map can be specified as C2 / 2, the number of channels of the output feature map can be specified as C2 / 2, the convolution kernel size can be specified as 3, the jump step size can be specified as 1, and the number of padded pixel columns can be specified as 1. After step 5, the width and height of the feature map remain unchanged, the number of channels becomes C2 / 2, and the result is recorded as K2.

[0100] Step 6: Merge the channels of feature map K1 and feature map K2 to obtain feature map K3. The number of channels of feature map K3 is C2.

[0101] In step 7, the feature map K3 is processed by the basic convolution layer. In this step, the number of channels of the input feature map can be specified as C2, the number of channels of the output feature map can be specified as C2, the convolution kernel size can be specified as 3, the jump step size can be specified as 1, and the number of padded pixel columns can be specified as 1. After step 7, the width and height of the feature map remain unchanged, the number of channels becomes C2, and the result is recorded as K4.

[0102] Step 8: Merge the channels of the feature map R and the feature map K4 to obtain the feature map K5. The number of channels of the feature map K5 is 2×C2.

[0103] In step 9, the feature map K5 is subjected to the maximum pooling operation to obtain the feature map X. After step 9, the width and height of the feature map are reduced to half of the original, and the number of channels remains unchanged. Therefore, the size of the feature map X is: (W / 2)×(H / 2)×(2×C2).

[0104] Figure 4 A flowchart of a control method for a brain-computer interface device provided in yet another embodiment of the present application is shown in FIG. Figure 4 As shown, this application Figure 2 In the illustrated embodiment, step 203 may include:

[0105] In step 401 , the brain-computer interface device 100 performs feature mapping on the multimodal features.

[0106] In this embodiment, the process of the brain-computer interface device 100 performing feature mapping on the multimodal features may include:

[0107] Step 1: Build a competitive learning network. The competitive learning network includes an input layer and a competition layer. The number of neuron nodes in the input layer is the same as the dimension of the multimodal feature. The competition layer is composed of N neurons, where N is the same as the number of types of actions performed by the user-controlled brain-computer interface device 100. Each neuron in the competition layer is weightedly connected to all neurons in the input layer.

[0108] Step 2: Use the winner-takes-all learning rule to train the competitive learning network, which may include:

[0109] Step 21: Normalize the input data: Normalize the multimodal features corresponding to the EEG data and image data at a certain moment and use them as the input vector of the input layer, which is recorded as the input mode vector X p ;

[0110] Step 22, normalize the inner star weight vector: normalize the inner star weight vector corresponding to each neuron in the competitive layer to obtain the normalized inner star weight vector W j ;

[0111] Step 23, find the winning neuron: input pattern vector X p and the normalized inner star weight vector W j The similarity is compared by inner product, and the normalized inner star weight vector W with the largest inner product result in the competition layer is j The corresponding neuron is the winning neuron, and the inner star weight vector of the winning neuron is recorded as W j *;

[0112] Step 24, network output and weight adjustment: the winning neuron output is 1, and the other outputs are zero. Only the winning neuron has the right to adjust the inner star weight vector W j *, the adjustment method can be shown as formula (2);

[0113] W j *(t+1)=W j *(t)+ΔW j = W j *(t)+α (X p -W j *(t)) (2)

[0114] In formula (2), α is the learning rate, which is between 0 and 1 and gradually decreases as the learning progresses. j *(t) is the inner star weight vector at the current moment, W j *(t+1) is the adjusted inner star weight vector;

[0115] In step 25, training ends when the learning rate α decreases to 0. The competitive learning network parameters are solidified for subsequent classification tasks. The output of the competitive learning network is an N-dimensional vector with only one component set to 1. The inner star weight vectors corresponding to the N neurons are the cluster centers of their corresponding patterns. A component with a value of 1 indicates that the corresponding neuron is activated and will execute the action corresponding to the control instruction.

[0116] In step 402 , the brain-computer interface device 100 fuzzifies the result of the feature mapping to obtain a fuzzy set corresponding to the multimodal features.

[0117] Specifically, the brain-computer interface device 100 fuzzifies the result of the feature mapping, and obtains the fuzzy set corresponding to the above-mentioned multimodal feature as follows: the brain-computer interface device 100 obtains the Euclidean distance between the result of the feature mapping and the cluster center corresponding to the result of the feature mapping; then discretizes the above-mentioned Euclidean distance to obtain the discrete domain value corresponding to the above-mentioned Euclidean distance; according to the discrete domain value corresponding to the above-mentioned Euclidean distance, determines the control adjustment amount corresponding to the above-mentioned multimodal feature; finally, according to the control adjustment amount corresponding to the above-mentioned multimodal feature, obtains the fuzzy set corresponding to the above-mentioned multimodal feature.

[0118] The above process is described in detail below.

[0119] Step 1: After the brain-computer interface device 100 obtains the Euclidean distance between the result of the feature mapping and the cluster center corresponding to the result of the feature mapping, the Euclidean distance is discretized to obtain a discrete domain value corresponding to the Euclidean distance.

[0120] Specifically, the Euclidean distance can be discretized using the formula shown in formula (3).

[0121]

[0122] In formula (3), k is the quantization factor, a is the Euclidean distance between the feature mapping result of the multimodal feature and the corresponding cluster center, d is the maximum Euclidean distance of the statistic, round[] represents the rounding operation, b is the discrete domain value corresponding to the Euclidean distance, and n is the quantization accuracy control parameter.

[0123] Step 2: Determine the control adjustment amount corresponding to the multimodal feature according to the discrete domain value corresponding to the Euclidean distance.

[0124] Specifically, after obtaining the discrete universe value, the category to which the Euclidean distance belongs can be determined according to the position of the discrete universe value in the discrete universe.

[0125] In one example, assuming that the quantization precision control parameter n is equal to 4, the discrete domain can be:

[0126] V={-4,-3,-2,-1,0,1,2,3,4}

[0127] The components in V are represented from left to right. The Euclidean distance between the result of the feature mapping and the cluster center corresponding to the result of the above feature mapping is negative large, negative large, negative medium, negative small, zero, positive small, positive medium, positive large and positive large.

[0128] In this way, when the discrete domain values ​​b obtained in step 1 are -4, -3, -2, -1, 0, 1, 2, 3, and 4 respectively, the brain-computer interface device 100 can determine that the Euclidean distances between the result of the feature mapping and the cluster centers corresponding to the result of the above feature mapping are negative large, negative large, negative medium, negative small, zero, positive small, positive medium, positive large, and positive large respectively.

[0129] If the Euclidean distance is positive, the control adjustment corresponding to the multimodal feature is negative.

[0130] If the above Euclidean distance is positive and large, then the control adjustment amount corresponding to the multimodal feature is negative and large;

[0131] If the above Euclidean distance is positive, then the control adjustment corresponding to the multimodal feature is negative;

[0132] If the above Euclidean distance is small and positive, then the control adjustment amount corresponding to the multimodal feature is small and negative;

[0133] If the above Euclidean distance is zero, then the control adjustment amount corresponding to the multimodal feature is zero;

[0134] If the above Euclidean distance is negative and small, then the control adjustment amount corresponding to the multimodal feature is positive and large;

[0135] If the above Euclidean distance is negative, then the control adjustment corresponding to the multimodal feature is positive;

[0136] If the Euclidean distance is negative and large, then the control adjustment amount corresponding to the multimodal feature is positive and large;

[0137] If the above Euclidean distance is negative, the control adjustment amount corresponding to the multimodal feature is positive.

[0138] Step 3: Obtain the fuzzy set corresponding to the multimodal feature according to the control adjustment amount corresponding to the multimodal feature.

[0139] In this embodiment, the control adjustment amount and the corresponding fuzzy set can be expressed as formula (4):

[0140] PBB=(0,0,0,0,0,0,0,0.5,1)

[0141] PB=(0,0,0,0,0,0,0.5,1,0)

[0142] PM=(0,0,0,0,0,0.5,1,0.5,0)

[0143] PS=(0,0,0,0,0.5,1,0.5,0,0)

[0144] ZE=(0,0,0,0.5,1,0.5,0,0,0) (4)

[0145] NS=(0,0,0.5,1,0.5,0,0,0,0)

[0146] NM=(0,0.5,1,0.5,0,0,0,0,0)

[0147] NB=(0,1,0.5,0,0,0,0,0,0)

[0148] NBB=(1,0.5,0,0,0,0,0,0,0)

[0149] In formula (4), PBB, PB, PM, PS, ZE, NS, NM, NB, and NBB respectively represent the control adjustment amount as positive large, positive large, positive medium, positive small, zero, negative small, negative medium, negative large, and negative large.

[0150] For example, assuming that the discrete domain value b obtained in step 1 is 2, the brain-computer interface device 100 determines that the Euclidean distance between the result of the feature mapping and the cluster center corresponding to the result of the feature mapping is positive based on the discrete domain value, and thus can determine that the control adjustment amount corresponding to the multimodal feature is negative. Then, according to formula (4), the brain-computer interface device 100 can obtain the fuzzy set corresponding to the multimodal feature as NM = (0, 0.5, 1, 0.5, 0, 0, 0, 0).

[0151] Next, the brain-computer interface device 100 can calculate the Cartesian product of the fuzzy set NM and the preset control precision set P to obtain a relationship matrix.

[0152] In this example, the control accuracy set P can be expressed as formula (5).

[0153]

[0154] In formula (5), m represents the maximum value of the control amount, and s is the control accuracy parameter; wherein the above-mentioned control amount can be the action or operation to be performed by the brain-computer interface device 100.

[0155] After obtaining the relationship matrix, the brain-computer interface device 100 can select the component with the largest fuzzy membership from the above relationship matrix as the fuzzy output result, and then use a defuzzification method, such as the maximum method and / or weighted average method, to map the above fuzzy output result to the above control instruction.

[0156] Finally, the BCI device 100 executes the control instructions. This allows the BCI device 100 to perform actions or operations based on the user's EEG data and the image data displayed by the BCI device 100. This improves the interactivity between the user and the BCI device 100 and the user's sense of immersion, providing the user with a more intelligent and personalized control experience. In actual applications, customized design and development can also be performed based on specific control requirements and user experience.

[0157] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in a different order than those presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0158] It is understandable that, in order to realize the above functions, the brain-computer interface device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0159] This embodiment can divide the brain-computer interface device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0160] Figure 5 This is a structural diagram of a brain-computer interface device provided by another embodiment of the present application, in which each functional module is divided into corresponding functional modules. Figure 5 A possible schematic diagram of the composition of the brain-computer interface device 500 involved in the above embodiment is shown. Figure 5 As shown, the brain-computer interface device 500 may include: an acquisition module 501, an extraction module 502 and an execution module 503;

[0161] The acquisition module 501 is used to acquire the user's EEG data;

[0162] An extraction module 502 is used to extract multimodal features from the EEG data and the image data displayed by the brain-computer interface device;

[0163] The acquisition module 501 is further configured to obtain, based on the multimodal features, a fuzzy set corresponding to the multimodal features; and obtain, based on the fuzzy set corresponding to the multimodal features, a control instruction corresponding to the multimodal features;

[0164] The execution module 503 is used to execute the above control instructions.

[0165] It should be noted that this application Figure 2 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0166] The brain-computer interface device 500 provided in this embodiment is used to execute the present application Figure 2 The control method of the brain-computer interface device provided by the illustrated embodiment can therefore achieve the same effect as the above-mentioned method.

[0167] Figure 6 A schematic diagram of the structure of a brain-computer interface device provided in another embodiment of the present application is provided. Figure 5 Compared with the brain-computer interface devices shown, Figure 6 In the brain-computer interface device 500 shown, the acquisition module 501 may include: a feature mapping submodule 5011, a fuzzification submodule 5012, a matrix acquisition submodule 5013, a selection submodule 5014 and an instruction acquisition submodule 5015;

[0168] A feature mapping submodule 5011 is used to perform feature mapping on multimodal features;

[0169] The fuzzification submodule 5012 is used to fuzzify the result of the feature mapping to obtain the fuzzy set corresponding to the multimodal feature.

[0170] In this embodiment, the matrix acquisition submodule 5013 is used to obtain a relationship matrix based on the fuzzy set corresponding to the multimodal features and a preset control precision set;

[0171] A selection submodule 5014 is used to select a fuzzy output result from the above relationship matrix according to the fuzzy membership degree;

[0172] The instruction acquisition submodule 5015 is used to obtain the control instruction corresponding to the above multimodal feature according to the above fuzzy output result.

[0173] In this embodiment, the extraction module 502 may include: an encoding submodule 5021, a feature conversion submodule 5022, and a feature acquisition submodule 5023;

[0174] The encoding submodule 5021 is configured to encode the EEG data to obtain an EEG vector; and to encode the image data to obtain an image vector.

[0175] A feature conversion submodule 5022 is configured to perform feature conversion on the EEG vector to obtain an EEG feature vector, and to perform feature conversion on the image vector to obtain an image feature vector;

[0176] The feature acquisition submodule 5023 is used to obtain the multimodal features according to the EEG feature vector and the image feature vector.

[0177] In some examples, the encoding submodule 5021 is specifically used to perform discrete wavelet transform on the EEG data of each channel to obtain a two-dimensional EEG feature map of each channel; wherein the two-dimensional EEG feature map of each channel includes the time domain and frequency domain features of each channel; the obtained two-dimensional EEG feature maps are merged to obtain three-dimensional EEG time-frequency domain features; on each channel, feature extraction is performed on the three-dimensional EEG time-frequency domain features to obtain the first depth features of each channel in the time dimension; and cross-channel feature extraction is performed on the above-mentioned three-dimensional EEG time-frequency domain features to obtain the second depth features between channels in the spatial dimension; based on the first depth features and the second depth features, the above-mentioned EEG vector is obtained.

[0178] In some examples, the above-mentioned image data may include local image data of the user's viewpoint at the current moment in the image displayed by the brain-computer interface device 100; the encoding sub-module 5021 is specifically used to perform image preprocessing on the above-mentioned local image data to obtain a viewpoint focus image; use the basic convolutional layer and the depth residual layer to extract the depth feature image from the above-mentioned viewpoint focus image; and obtain the above-mentioned image vector based on the above-mentioned depth feature image.

[0179] In some examples, the fuzzification submodule 5012 is specifically used to obtain the Euclidean distance between the result of the feature mapping and the cluster center corresponding to the result of the above feature mapping, discretize the above Euclidean distance, and obtain the discrete domain value corresponding to the above Euclidean distance; and determine the control adjustment amount corresponding to the above multimodal feature based on the discrete domain value corresponding to the above Euclidean distance; and obtain the fuzzy set corresponding to the above multimodal feature based on the control adjustment amount corresponding to the above multimodal feature.

[0180] It should be noted that this application Figures 2 to 4 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0181] The brain-computer interface device 500 provided in this embodiment is used to execute the present application Figures 2 to 4 The control method of the brain-computer interface device provided by the illustrated embodiment can therefore achieve the same effect as the above-mentioned method.

[0182] It should be understood that the brain-computer interface device 500 may correspond to Figure 1 The brain-computer interface device 100 shown in FIG. The functions of the acquisition module 501, the extraction module 502 and the execution module 503 can be Figure 1 The processor 110 in the brain-computer interface device 100 is implemented.

[0183] In the case of an integrated unit, the brain-computer interface device 500 may include a processing module, a storage module, and a communication module.

[0184] The processing module can be used to control and manage the actions of the brain-computer interface device 500. For example, it can be used to support the brain-computer interface device 500 in executing the steps performed by the above modules. The storage module can be used to support the brain-computer interface device 500 in storing program code and data. The communication module can be used to support communication between the brain-computer interface device 500 and other devices.

[0185] Among them, the processing module can be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other brain-computer interface devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0186] In one embodiment, when the processing module is a processor and the storage module is a memory, the brain-computer interface device 500 involved in this embodiment can be a Figure 1 Device with the structure shown.

[0187] The present invention also provides a computer-readable storage medium in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present invention. Figures 2 to 4 The method provided by the illustrated embodiment.

[0188] The present invention also provides a computer program product, which includes a computer program that, when executed on a computer, enables the computer to execute the present invention. Figures 2 to 4 The method provided by the illustrated embodiment.

[0189] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0190] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0191] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0192] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A method for controlling a brain-computer interface device, characterized in that: include: The brain-computer interface device obtains the user's EEG data; Extracting multimodal features from the EEG data and image data displayed by the brain-computer interface device; wherein the image data displayed by the brain-computer interface device includes image data corresponding to instructions executed by the brain-computer interface device; According to the multimodal feature, obtaining a fuzzy set corresponding to the multimodal feature; Obtaining a control instruction corresponding to the multimodal feature according to the fuzzy set corresponding to the multimodal feature; executing the control instruction; Wherein, obtaining the fuzzy set corresponding to the multimodal feature according to the multimodal feature includes: Performing feature mapping on the multimodal features; Fuzzifying the result of feature mapping to obtain a fuzzy set corresponding to the multimodal feature; The performing feature mapping on the multimodal features includes: Building a competitive learning network, the competitive learning network comprising an input layer and a competitive layer, wherein the number of neuron nodes in the input layer is the same as the dimension of the multimodal feature, the competitive layer is composed of N neurons, where N is the same as the number of types of actions performed by the user controlling the brain-computer interface device, and each neuron in the competitive layer is weightedly connected to all neurons in the input layer; Training the competitive learning network using a winner-takes-all learning rule; The fuzzifying the result of the feature mapping to obtain the fuzzy set corresponding to the multimodal feature includes: Obtaining the Euclidean distance between a result of the feature mapping and a cluster center corresponding to the result of the feature mapping; Discretizing the Euclidean distance to obtain a discrete domain value corresponding to the Euclidean distance; Determining a control adjustment amount corresponding to the multimodal feature according to a discrete domain value corresponding to the Euclidean distance; Obtaining a fuzzy set corresponding to the multimodal feature according to the control adjustment amount corresponding to the multimodal feature; The step of extracting multimodal features from the EEG data and the image data displayed by the brain-computer interface device includes: Encoding the EEG data to obtain an EEG vector; and encoding the image data to obtain an image vector; Performing feature conversion on the EEG vector to obtain an EEG feature vector, and performing feature conversion on the image vector to obtain an image feature vector; Obtaining the multimodal feature according to the EEG feature vector and the image feature vector; The encoding of the EEG data to obtain an EEG vector comprises: Performing discrete wavelet transform on the EEG data of each channel to obtain a two-dimensional EEG feature map of each channel; wherein the two-dimensional EEG feature map of each channel includes time domain and frequency domain features of each channel; The obtained two-dimensional EEG feature maps are merged to obtain three-dimensional EEG time-frequency domain features; On each channel, feature extraction is performed on the three-dimensional EEG time-frequency domain features to obtain a first depth feature of each channel in the time dimension; and cross-channel feature extraction is performed on the three-dimensional EEG time-frequency domain features to obtain a second depth feature between channels in the spatial dimension; The EEG vector is obtained according to the first depth feature and the second depth feature.

2. The method according to claim 1, characterized in that The obtaining, according to the fuzzy set corresponding to the multimodal feature, the control instruction corresponding to the multimodal feature comprises: Obtaining a relationship matrix according to the fuzzy set corresponding to the multimodal features and a preset control precision set; Selecting a fuzzy output result from the relationship matrix according to the fuzzy membership degree; A control instruction corresponding to the multimodal feature is obtained according to the fuzzy output result.

3. The method according to claim 1, characterized in that The image data includes local image data of the user's viewpoint at the current moment in the image displayed by the brain-computer interface device; The encoding of the image data to obtain an image vector comprises: Performing image preprocessing on the local image data to obtain a viewpoint focus image; Extracting a depth feature image from the viewpoint attention image using a basic convolutional layer and a depth residual layer; The image vector is obtained according to the depth feature image.

4. A control device for a brain-computer interface device, characterized in that: Set in a brain-computer interface device, the control device includes: Acquisition module, used to obtain the user's EEG data; an extraction module, configured to extract multimodal features from the EEG data and image data displayed by the brain-computer interface device; wherein the image data displayed by the brain-computer interface device includes image data corresponding to instructions executed by the brain-computer interface device; The acquisition module is further configured to obtain, based on the multimodal feature, a fuzzy set corresponding to the multimodal feature; and obtain, based on the fuzzy set corresponding to the multimodal feature, a control instruction corresponding to the multimodal feature; An execution module, configured to execute the control instruction; Wherein, the acquisition module includes: a feature mapping submodule and a fuzzification submodule; The feature mapping submodule is used to perform feature mapping on the multimodal features; The fuzzification submodule is used to fuzzify the result of feature mapping to obtain the fuzzy set corresponding to the multimodal feature; The feature mapping submodule is specifically used to build a competitive learning network, which includes an input layer and a competition layer. The number of neuron nodes in the input layer is the same as the dimension of the multimodal feature. The competition layer is composed of N neurons, where N is the same as the number of types of actions performed by the user controlling the brain-computer interface device. Each neuron in the competition layer is weightedly connected to all neurons in the input layer. The competitive learning network is trained using a winner-takes-all learning rule. The fuzzification submodule is specifically configured to obtain a Euclidean distance between a result of a feature mapping and a cluster center corresponding to the result of the feature mapping, discretize the Euclidean distance, and obtain a discrete domain value corresponding to the Euclidean distance; and determine a control adjustment amount corresponding to the multimodal feature based on the discrete domain value corresponding to the Euclidean distance; and obtain a fuzzy set corresponding to the multimodal feature based on the control adjustment amount corresponding to the multimodal feature. Wherein, the extraction module includes: an encoding submodule, a feature conversion submodule and a feature acquisition submodule; The encoding submodule is used to encode the EEG data to obtain EEG vectors; and to encode the image data to obtain image vectors; The feature conversion submodule is used to perform feature conversion on the EEG vector to obtain an EEG feature vector, and to perform feature conversion on the image vector to obtain an image feature vector; The feature acquisition submodule is used to obtain the multimodal feature according to the EEG feature vector and the image feature vector; Among them, the encoding submodule is specifically used to perform discrete wavelet transform on the EEG data of each channel to obtain a two-dimensional EEG feature map of each channel; wherein the two-dimensional EEG feature map of each channel includes the time domain and frequency domain features of each channel; the obtained two-dimensional EEG feature maps are merged to obtain three-dimensional EEG time-frequency domain features; on each channel, feature extraction is performed on the three-dimensional EEG time-frequency domain features to obtain the first depth features of each channel in the time dimension; and cross-channel feature extraction is performed on the three-dimensional EEG time-frequency domain features to obtain the second depth features between channels in the spatial dimension; according to the first depth features and the second depth features, the EEG vector is obtained.

5. A brain-computer interface device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the brain-computer interface device, enable the brain-computer interface device to perform the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 3.

Images