Brain-computer interface stimulation normal form generation method and system and electronic equipment

Abstract

The invention provides a brain-computer interface stimulation paradigm generation method and system and electronic equipment, and is applied to a brain-computer interface stimulation paradigm generation system.The method comprises the steps that a simulated driving environment comprising a visual evoked stimulation source and a virtual vehicle is constructed, original electroencephalogram signals generated when the tested driver controls the virtual vehicle to run under the action of the visual evoked stimulation source are obtained; a potential classification model and an electroencephalogram signal processing model are used for conducting classification and recognition on the original electroencephalogram signals under different dimensions, and a vehicle control instruction is determined based on classification and recognition results under different dimensions; and training the stimulation paradigm mapping model by using the vehicle control instruction so as to determine a brain-computer interface stimulation paradigm corresponding to the simulated driving environment by using the trained stimulation paradigm mapping model. By adopting the brain-computer interface stimulation normal form generation method and system and the electronic equipment, the problem of low accuracy of signal control and signal identification of the brain-computer interface of an intelligent automobile is solved.

Description

Technical Field

[0001] The present application relates to the field of brain-computer interface technology, and more specifically, to a method, system and electronic device for generating a brain-computer interface stimulation paradigm. Background Art

[0002] Brain-computer interface technology can establish a connection between the brain and the external environment, and use the EEG signals generated by the brain's conscious activities to control external devices. Steady State Visual Evoked Potential (SSVEP) refers to the change in the potential of EEG signals generated after a fixed-frequency visual stimulation of the human brain. Due to its advantages such as short training time, fast communication rate, and simple equipment configuration, it is widely used in medical, aerospace, and military fields. At present, with the substantial increase in the number of cars, smart cars have been widely used and popularized, and human beings have increased their requirements for the intelligence of smart cars, and the demand for intelligent brain-computer interaction functions has also increased rapidly.

[0003] However, at present, the brain-computer interface technology for smart cars is still in its infancy, and the relevant technologies for brain-computer interface function testing are particularly weak. In addition, EEG signals are easily affected by external stimuli, prone to produce erroneous control signals, and have a high misrecognition rate. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a brain-computer interface stimulation paradigm generation method, system and electronic device to solve the problem of low accuracy of signal control and signal recognition of the brain-computer interface of smart cars.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a brain-computer interface stimulation paradigm, which is applied to a brain-computer interface stimulation paradigm generation system, and the method includes:

[0006] Constructing a simulated driving environment including a visual induced stimulus source and a virtual vehicle, and obtaining the original EEG signals generated by the test driver when controlling the driving of the virtual vehicle under the action of the visual induced stimulus source;

[0007] Using the potential classification model and the EEG signal processing model, the original EEG signals are classified and identified in different dimensions, and the vehicle control instructions are determined based on the classification and identification results in different dimensions;

[0008] The stimulation paradigm mapping model is trained using vehicle control instructions and original EEG signals, so that the trained stimulation paradigm mapping model can be used to determine the brain-computer interface stimulation paradigm of the test driver in a simulated driving environment.

[0009] Optionally, the classification and recognition results under different dimensions include a first signal recognition result and a second signal recognition result. The potential classification model and the EEG signal processing model are used to classify and recognize the original EEG signal under different dimensions, including: using the potential classification model to classify and recognize the steady-state visual potential component in the original EEG signal to obtain the first signal recognition result; using the EEG signal processing model to classify and recognize other components in the original EEG signal to obtain the second signal recognition result, where the other components are components in the original EEG signal other than the steady-state visual potential component.

[0010] Optionally, the potential classification model includes a spatial filter and a first classifier, and the potential classification model is used to classify and identify the steady-state visual potential components in the original EEG signal, including: filtering the original EEG signal using the spatial filter to obtain the filtered original EEG signal; and classifying the filtered original EEG signal using the first classifier to obtain a first signal recognition result.

[0011] Optionally, the EEG signal processing model includes a preprocessing model, a separation model and a second classifier. The EEG signal processing model is used to classify and identify other components in the original EEG signal, including: preprocessing the original EEG signal using the preprocessing model to obtain the preprocessed original EEG signal; performing component separation processing on the preprocessed original EEG signal using the separation model to obtain a pure EEG signal; and inputting the pure EEG signal into the second classifier to obtain a second signal recognition result.

[0012] Optionally, a vehicle control instruction is determined based on the classification recognition results in different dimensions, including: comparing the first signal recognition result with the second signal recognition result, and determining whether the recognition result meets the preset recognition requirements based on the comparison result, and the preset recognition requirement refers to the requirement that the two recognition results match; if the preset recognition requirements are met, the first signal recognition result and / or the second signal recognition result is converted into a recognizable vehicle control instruction to control the virtual vehicle to execute the vehicle control instruction.

[0013] Optionally, the stimulation paradigm mapping model is trained using vehicle control instructions and original EEG signals, including: obtaining the instruction truth value corresponding to the original EEG signal, and using the instruction truth value as the standard value, and using the original EEG signal and the vehicle control instruction as the input data and output data of the stimulation paradigm mapping model respectively; using the standard value, input data and output data to train the stimulation paradigm mapping model.

[0014] Optionally, constructing a simulated driving environment including a visual induced stimulus source and a virtual vehicle includes: constructing an initial simulated driving environment, the initial simulated driving environment including a virtual vehicle; and providing at least one visual induced stimulus source flashing at a fixed frequency in the simulated driving environment using a shader.

[0015] Optionally, the classification results of the first classifier and / or the second classifier each include multiple classification results, and the multiple classification results are determined based on a binary classification method.

[0016] In a second aspect, the embodiment of the present application further provides a brain-computer interface stimulation paradigm generation system, the brain-computer interface stimulation paradigm generation system comprising a driving environment construction module, a signal processing module, and a vehicle control module:

[0017] A driving environment construction module, used to construct a simulated driving environment including a visual induced stimulus source and a virtual vehicle;

[0018] Collecting the original EEG signals generated by the test driver when controlling the virtual vehicle under the action of the visual induced stimulus source, and sending the original EEG signals to the EEG signal processing module;

[0019] The signal processing module is used to classify and identify the original EEG signals based on different dimensions using the potential classification model and the EEG signal processing model;

[0020] Determine the vehicle control command based on the classification recognition result, and use the vehicle control command and the original EEG signal to train the stimulation paradigm mapping model, so as to use the trained stimulation paradigm mapping model to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment;

[0021] The vehicle control module is used to control the virtual vehicle to operate in a simulated driving environment according to the vehicle control instructions.

[0022] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the brain-computer interface stimulation paradigm generation method as described above are performed.

[0023] The embodiments of the present application bring the following beneficial effects:

[0024] The embodiments of the present application provide a brain-computer interface stimulation paradigm generation method, system and electronic device, which can induce the test driver to generate original EEG signals in the current simulated driving environment through a visually induced stimulus source, and use a potential classification model and an EEG signal processing model to classify and identify the original EEG signals in different dimensions, thereby avoiding the problems of erroneous control signals and misidentification caused by abnormal fluctuations in EEG signals due to differences in drivers and driving environments. By training the stimulation paradigm mapping model using the classification and identification results in different dimensions, the accuracy of model training is also improved. Compared with the brain-computer interface stimulation paradigm generation method in the prior art, the problem of low accuracy of signal control and signal recognition of the brain-computer interface of smart cars is solved.

[0025] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A flow chart of a method for generating a brain-computer interface stimulation paradigm provided in an embodiment of the present application is shown;

[0028] Figure 2 A schematic diagram showing the electrode placement positions of the electroencephalometer provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of the structure of a brain-computer interface stimulation paradigm generation system provided in an embodiment of the present application is shown;

[0030] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0032] It is worth noting that before this application was filed, brain-computer interface technology was able to establish a connection between the brain and the external environment, and use the EEG signals generated by the brain's conscious activities to control external devices. Steady-state visual evoked potential (SSVEP) refers to the change in the potential of EEG signals generated after a fixed-frequency visual stimulation is applied to the human brain. It is widely used in medical, aerospace, and military fields due to its advantages such as short training time, fast communication rate, and simple equipment configuration. At present, with the substantial increase in the number of cars, smart cars have been widely used and popularized, and human beings have increased their requirements for the intelligence of smart cars, and the demand for intelligent brain-computer interaction functions has also increased rapidly. However, at this stage, the brain-computer interface technology of smart cars is still in its infancy, and the relevant technologies for brain-computer interface function testing are particularly weak, and the EEG signals of smart cars are easily affected by external stimuli, which are prone to produce erroneous control signals, and the misrecognition rate is high.

[0033] Based on this, an embodiment of the present application provides a method for generating a brain-computer interface stimulation paradigm to improve the accuracy of signal control and signal recognition of the brain-computer interface of an intelligent vehicle.

[0034] See also Figure 1 , Figure 1 A flowchart of a method for generating a brain-computer interface stimulation paradigm provided in an embodiment of the present application. Figure 1 As shown, the brain-computer interface stimulation paradigm generation method provided in the embodiment of the present application is applied to the brain-computer interface stimulation paradigm generation system, including:

[0035] Step S101, constructing a simulated driving environment including a visual induced stimulus source and a virtual vehicle, and obtaining the original EEG signal generated by the test driver when controlling the virtual vehicle to drive under the action of the visual induced stimulus source;

[0036] Step S102, using the potential classification model and the EEG signal processing model, classify and identify the original EEG signal in different dimensions, and determine the vehicle control command based on the classification and identification results in different dimensions;

[0037] Step S103, using the vehicle control instructions and the original EEG signals to train the stimulation paradigm mapping model, so as to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment using the trained stimulation paradigm mapping model.

[0038] Through the above-mentioned brain-computer interface stimulation paradigm generation method, the present application induces the test driver to generate original EEG signals in the current simulated driving environment through a visually induced stimulus source, and uses a potential classification model and an EEG signal processing model to classify and identify the original EEG signals in different dimensions, thereby avoiding the problems of erroneous control signals and misidentification caused by abnormal fluctuations in EEG signals due to differences in drivers and driving environments. By using the classification and recognition results in different dimensions to train the stimulation paradigm mapping model, the accuracy of the model training is also improved, and the problem of low accuracy of signal control and signal recognition of the brain-computer interface of smart cars is solved.

[0039] Below Figure 1 Each step is described in detail.

[0040] In step S101, a simulated driving environment including a visual induced stimulus source and a virtual vehicle is constructed, and original EEG signals generated when the test driver controls the driving of the virtual vehicle under the action of the visual induced stimulus source are obtained.

[0041] The visual evoked stimulus source may refer to a stereoscopic visual evoked stimulus source flickering at a fixed frequency, and the visual evoked stimulus source is used to stimulate the brain of the test driver to produce a SSVEP response.

[0042] The virtual vehicle may refer to a virtual three-dimensional vehicle model in a simulated driving environment, which is used to simulate a real vehicle so as to display control feedback for the virtual vehicle in real time in the simulated driving environment.

[0043] The simulated driving environment may refer to an autonomous driving simulation test environment, and the simulated driving environment includes an autonomous driving simulation test scenario. Exemplarily, the autonomous driving simulation test scenario is a virtual test scenario.

[0044] In an embodiment of the present application, a simulated driving environment can be constructed by a driving environment construction module in a brain-computer interface stimulation paradigm generation system. In order to construct a highly realistic virtual driving environment, including various road conditions, weather conditions and traffic conditions, and to record data in the driving process in real time, such as the state parameters of a virtual vehicle, here, the SCANeRstudio autonomous driving simulation software can be used to build an autonomous driving simulation test scene. Specifically, the road surface corresponding to the autonomous driving simulation test scene can be created in the TERRAIN interface of the autonomous driving simulation software according to the test requirements, and the road type can be selected, such as straight, crossroads, roundabouts, etc., and then the road is parameterized to define the specific values ​​of the length and width of the road. The virtual vehicle is modeled in the VEHICLE interface, and the vehicle model provided in the autonomous driving simulation software can also be directly called. Finally, in the SCENARIO mode, based on the parameters of the constructed scene library, traffic participants, road facilities, etc. are set, and in the script editing box, some special trigger conditions including intelligent voice control, weather, etc. are controlled. In this way, an initial simulated driving environment is constructed, and the initial simulated driving environment includes a virtual vehicle.

[0045] At the same time, visual induced stimuli can be added to the initial simulated driving environment through the autonomous driving simulation software, or an additional display screen can be set in front of the test driver, and the visual induced stimuli can be set in the display screen. Regardless of which of the above methods is used to add visual induced stimuli, a shader can be used to provide at least one stereoscopic visual induced stimulus source that flashes at a fixed frequency in the simulated driving environment, thereby stimulating the brain of the test driver to produce an SSVEP response. Specifically, a vertex shader is set in the stimulus source control software of the additional display screen or the autonomous driving simulation software, and the vertex data of the stereoscopic visual induced stimulus source model is converted to the clipping coordinate system through matrix transformation in the vertex shader, and a matrix C is defined in the vertex shader to store the R, G, and B channel data of different colors of the stereoscopic visual induced stimulus source. For example: assuming that the stereoscopic visual induced stimulus source needs to switch to display two colors, the matrix C can be expressed as:

[0046]

[0047] In the above formula, R i Represents the value of the R channel of the i-th color, G i Indicates the value of the G channel of the i-th color, B i Represents the value of the B channel of the i-th color.

[0048] Then, according to the preset display position, adjust the row index L at the current moment i (starting from time t=0), where the row index calculation formula is:

[0049] L i = Fl[(2·t·f)$%2];

[0050] In the above formula, Fl[] represents rounding down; t represents the time count after the program is run; f represents the flickering frequency of the stereoscopic vision induced stimulus source; %2 represents the remainder of 2.

[0051] After the row index is calculated, the color value corresponding to the row index can be passed into the fragment shader, and the color value can be output in the fragment shader to display the alternatingly flickering stereoscopic vision inducing stimulus source on the screen.

[0052] In addition, the simulated driving environment also includes an electroencephalogram (EEG), which is placed on the head of the test driver to collect the EEG signals of the test driver. The EEG signals include delta waves, theta waves, alpha waves and beta waves. The EEG can adopt the international 10-20 system electrode placement method, which has a total of 14+2 channels (including 2 reference points).

[0053] Refer to the following Figure 2 Let's introduce the position of each channel in the EEG device.

[0054] Figure 2 A schematic diagram showing the electrode placement positions of the electroencephalometer provided in the embodiment of the present application is shown, Figure 2 As shown, the EEG device includes multiple channels, channels AF3, F3, F7, AF4, F4 and F8 are set in the frontal lobe area, channels P3, P7, P4 and P8 are set in the parietal lobe area, channels FC5, T7, FC6 and T8 are set in the temporal lobe area, channels O1 and O2 are set in the occipital lobe area, wherein AF represents frontal, F represents frontal, P represents parietal, FC represents frontocentral, T represents temporal and O represents occipital.

[0055] Finally, the human-machine environment synchronization cloud platform system ErgoLAB, the autonomous driving simulation software SCANeR studio and the SimEASY driving simulator were connected to build a brain-computer interface simulation test platform. The brain-computer interface simulation test platform records the EEG signals of the test driver in real time through the ErgoLAB system and synchronously records the data of the virtual vehicle through the autonomous driving simulation software, thus completing the construction of the simulated driving environment.

[0056] After completing the construction of the simulated driving environment, the test driver executes the preset vehicle control instructions to make the virtual vehicle run in the autonomous driving simulation test scene. At the same time, the original EEG signals generated by the test driver when controlling the virtual vehicle to drive or run under the action of visual induced stimulation sources are obtained. Vehicle control instructions include but are not limited to: vehicle left turn, vehicle right turn, vehicle acceleration, braking, window lifting, in-car music switching, air conditioning temperature adjustment, etc.

[0057] In step S102, the potential classification model and the EEG signal processing model are used to classify and identify the original EEG signals in different dimensions, and the vehicle control instructions are determined based on the classification and identification results in different dimensions.

[0058] The potential classification model may refer to a neural network model, and the potential classification model is a model used to extract SSVEP signals from EEG signals and obtain classification and recognition results based on the extracted SSVEP signals.

[0059] The EEG signal processing model may refer to a model used to extract normal EEG signals and obtain classification and recognition results based on the normal EEG signals.

[0060] Different dimensions include SSVEP signal dimensions and other signal dimensions, and other signal dimensions may refer to dimensions of other EEG signals except SSVEP signals.

[0061] The classification and recognition results under different dimensions include the first signal recognition result and the second signal recognition result. The first signal recognition result is obtained by the potential classification model based on the SSVEP signal in the original EEG signal, and the second signal recognition result is obtained by the EEG signal processing model based on the other signals in the original EEG signal except the SSVEP signal.

[0062] Vehicle control instructions may refer to instructions for controlling a virtual vehicle.

[0063] In the embodiment of the present application, since SSVEP is a special EEG signal, it needs to be processed separately. To this end, the potential classification model can be used to extract the steady-state visual potential component in the original EEG signal, that is, the SSVEP signal is extracted, and the extracted steady-state visual potential component is classified and identified to obtain the first signal recognition result. At the same time, the EEG signal processing model is used to classify and identify other components in the original EEG signal to obtain the second signal recognition result, wherein the other components are other normal EEG signals in the original EEG signal except the steady-state visual potential component.

[0064] Before using the potential classification model for classification and identification, historical EEG signal data including SSVEP signals can be obtained, and the potential classification model can be trained offline using the historical EEG signal data. The potential classification model includes a spatial filter and a first classifier.

[0065] Here, when the potential classification model is trained offline, in order to effectively extract the frequency response characteristics of the signal, it is first necessary to filter the signal in a specific frequency band. For the fundamental wave (f) and the second harmonic (2f) of the EEG signal under each target channel, corresponding bandpass filters are set. The corresponding frequency band ranges of the bandpass filters are [f-0.25, f+0.25] and [2f-0.25, 2f+0.25] respectively.

[0066] In one example, when using the potential classification model for classification and recognition, the original EEG signal can be filtered using a spatial filter to obtain the filtered original EEG signal. Specifically, after the EEG signal preprocessing is completed, the offline historical EEG signal data can be trained through the Common Spatial Patterns (CSP) to obtain a set of spatial filter matrices, thereby extracting the features of the EEG signal.

[0067] Here, when constructing the spatial filter, if the classification recognition result is two categories, then in the case of two categories, the construction process of the spatial filter is as follows: let the i-th EEG signal sample in the two categories and The normalized covariance matrix of is:

[0068]

[0069] In the above formula, tr() represents the sum of the elements on the diagonal of the matrix.

[0070] The construction formula of the spatial filter is W=PB, where P represents the whitening matrix of the covariance matrix R, and B represents the combination of the eigenvectors corresponding to the maximum eigenvalue and the minimum eigenvalue of the matrix after whitening. The historical EEG signal data is used as the signal data sample, and the characteristic matrix Z of the signal data sample after spatial filtering is expressed as Z=WX, where W represents the spatial filter matrix and X represents the signal data sample. Then, the logarithm of the filtered eigenvector is taken as the EEG signal feature, where the EEG signal feature of the jth row is expressed as:

[0071]

[0072] In the above formula, var(Z j ) represents the variance of the j-th row in the matrix Z, j = 1, 2, …, 2m, 2m represents the number of rows in the matrix Z.

[0073] In one example, the classification results of the first classifier and / or the second classifier include multiple classification results, and the multiple classification results are determined based on the binary classification method. If the classification recognition result is multiple classifications, the multiple classifications are converted into multiple binary classifications for processing by the "one-to-many" method. Here, one of the n classifications can be selected to set the corresponding covariance matrix Then the covariance matrix of the remaining other categories is combined with the class frequency for weighted averaging as use and The spatial filter under the two classifications is solved, and n spatial filters can be constructed in sequence by analogy. Among them, the number of classifications of the classification recognition results is set, and those skilled in the art can determine the specific number of classifications of the classification recognition results according to actual conditions, and this application does not limit it here.

[0074] Then, the filtered original EEG signal is classified and processed using the first classifier to obtain the first signal recognition result. At this time, it is necessary to construct the first classifier first. When constructing the first classifier, the linear discriminant analysis method can be used to construct the first classifier. This method distinguishes different types of signals by linearly combining the sample signal features. Among them, the sample signal features are the features extracted by the spatial filter. If the classification and recognition result is binary, the sample signal features are projected onto a straight line so that the sample signal features of the same type are as close as possible and the sample signal features of different types are as far away as possible, so as to obtain the goal of maximizing the linear discriminant analysis (LDA).

[0075] The specific implementation process is as follows: Assume that the projection line is ω, take any sample signal feature x i , define the intra-class scatter matrix as S ω , S ω =∑(x-μ0)(x-μ0) T +∑(x-μ1)(x-μ1) T , where μ0 and μ1 are the center points of the two types of samples. Define the inter-class scatter matrix as S b , S b =(μ0-μ1)(μ0-μ1) T . Among them, the objective function J(w) maximized by LDA is:

[0076]

[0077] Similarly, if the classification recognition result is n categories, it can also be expanded into multiple "two-classification" for solution through "one-to-many". That is, take one of the n categories, and then combine the remaining n-1 categories into another category. Through training, the first classifier in this classification situation can be obtained. And so on, a total of n first classifiers can be obtained.

[0078] In this way, after constructing the spatial filter and the first classifier, the EEG signals of the test driver collected online can be preprocessed, and the features of the preprocessed EEG signals can be extracted through the spatial filter. After the extracted features are classified through the first classifier, the first signal recognition result can be obtained.

[0079] In one example, the EEG signal processing model includes a preprocessing model, a separation model and a second classifier. When the EEG signal processing model is used for classification and recognition, the preprocessing model can be used to preprocess the original EEG signal to obtain the preprocessed original EEG signal. Then, the separation model is used to separate the components of the preprocessed original EEG signal to obtain a pure EEG signal; the pure EEG signal is input into the second classifier to obtain a second signal recognition result. Here, the construction method of the second classifier is the same as the construction method of the first classifier, which will not be repeated here.

[0080] Specifically, the EEG signal processing model can be obtained by constructing and training a neural network model, or it can be implemented by setting software. Exemplarily, the setting software can be MATLAB software. When the original EEG signal is processed using MATLAB software, since MATLAB software includes the EEGLAB toolbox, the original EEG signal of the test driver collected can be imported into the EEGLAB toolbox, so as to use the pre-processing plug-in to perform electroencephalogram electrode positioning, low-pass / high-pass filtering, effective data segmentation, and baseline correction processing. At the same time, the ICLabel plug-in in the toolbox is used as a separation model, and the ICLabel plug-in is used to perform independent component analysis (ICA) to separate the mixed EEG signal into the pure EEG signal of the independent component. Finally, the pure EEG signal after pre-processing and independent component analysis is input into the second classifier to obtain the second signal recognition result.

[0081] When determining the vehicle control command, the first signal recognition result and the second signal recognition result can be compared, and whether the recognition result meets the preset recognition requirement can be determined based on the comparison result. The preset recognition requirement refers to the requirement that the two recognition results match. If the first signal recognition result matches the second signal recognition result, it is determined that the preset recognition requirement is met. Otherwise, it is determined that the preset recognition requirement is not met. For example, if the first signal recognition result is a left turn, and the second signal recognition result is also a left turn, it is determined that the first signal recognition result matches the second signal recognition result and meets the preset recognition requirement.

[0082] Regardless of whether the preset recognition requirements are met, the first signal recognition result and the second signal recognition result are converted into a vehicle control instruction, and the vehicle control instruction is used as the output of the stimulation paradigm model to train the stimulation paradigm model. Further, if the preset recognition requirements are met, the first signal recognition result and / or the second signal recognition result are converted into a recognizable vehicle control instruction to control the virtual vehicle to execute the vehicle control instruction, and the execution result of the vehicle control instruction is displayed in the simulated driving scene.

[0083] In step S103, the stimulation paradigm mapping model is trained using the vehicle control instructions and the original EEG signals, so as to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment using the trained stimulation paradigm mapping model.

[0084] The stimulation paradigm mapping model may refer to a neural network model trained using raw EEG signals and vehicle control commands. The stimulation paradigm mapping model may characterize the brain-computer interface stimulation paradigm of the test driver in a simulated driving environment.

[0085] In an embodiment of the present application, when training the stimulation paradigm mapping model, the instruction truth value corresponding to the original EEG signal can be obtained, and the instruction truth value is used as the standard value, and the original EEG signal and the vehicle control instruction are used as the input data and output data of the stimulation paradigm mapping model respectively. Then, the stimulation paradigm mapping model is trained using the standard value, input data and output data. Among them, the instruction truth value may refer to the vehicle control instruction actually issued by the test driver. For example, it is set in advance that the test driver needs to issue a "turn left" vehicle control instruction, and the vehicle control instruction is the instruction truth value. When the output result of the brain-computer interface stimulation paradigm generation system deviates from the instruction truth value, the stimulation paradigm mapping model is trained with the instruction truth value as the standard value. Through the trained stimulation paradigm mapping model, the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment can be determined.

[0086] The stimulus paradigm mapping model is a neural network model, in which It represents the linear coefficient from the jth neuron in the l-1th layer to the kth neuron in the lth layer; the lth layer of the neural network is denoted by L l ; The activation function of each layer of network nodes is σ(z); The linear relationship is is the mth bias of the lth layer network, the calculation method of m conforms to the 2n+1 rule, and n is the number of input nodes.

[0087] Taking the input and output calculation rules of the lth layer of neurons as an example, the output of each neuron is recorded as l is the number of layers, k is the number of neurons, and the calculation method is as follows:

[0088]

[0089] Furthermore, the above calculation method is organized, and the mathematical expression of the output of each layer of neuron nodes in the model is as follows:

[0090]

[0091] In the above formula, x is the total number of signals generated by the stimulation paradigm, is the kth neuron output of the lth layer in the neural network; R q is the output vector of the qth neural network model; R p As the stimulus paradigm mapping sequence, that is, the output sequence used to train the qth neural network model; I p , R p The corresponding stimulus paradigm mapping model is denoted by N p , that is, the deep neural network mapping model corresponding to the pth stimulation paradigm. In this way, the establishment of the brain-computer interface stimulation paradigm mapping model is completed.

[0092] On the basis of completing the above-mentioned brain-computer interface stimulation paradigm mapping model, the EEG signal data of the test driver in different simulated driving environments is input as input data into each feature model to obtain the stimulation paradigm mapping output sequence Rp. Then, the goodness of fit is used to test the difference between the stimulation paradigm mapping output sequence and each original EEG signal. The calculation method is as follows:

[0093]

[0094] In the above formula, R 3 represents the measure of goodness of fit, t and r represent the output sequence and the original EEG signal respectively, R 3The closer the value is to 1, the closer it is to the linear law of the stimulation paradigm mapping model. Therefore, it can be seen that the feature model with the highest goodness of fit can more accurately reflect the driving task of the test driver (i.e., the true value of the instruction), thereby achieving the effect of brain-computer interface stimulation paradigm mapping.

[0095] In addition, the number of test drivers can be one or more. When there are multiple test drivers, the EEG signals of multiple test drivers can be collected, and the commonalities of the brain-computer interface stimulation paradigm can be extracted using the stimulation paradigm mapping model, thereby establishing a brain-computer interface stimulation paradigm suitable for a wider user group.

[0096] Based on the same inventive concept, a brain-computer interface stimulation paradigm generation system corresponding to the brain-computer interface stimulation paradigm generation method is also provided in the embodiments of the present application. Since the principle of solving the problem by the system in the embodiments of the present application is similar to the above-mentioned brain-computer interface stimulation paradigm generation method in the embodiments of the present application, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0097] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a brain-computer interface stimulation paradigm generation system provided in an embodiment of the present application. Figure 3 As shown in , the brain-computer interface stimulation paradigm generation system 200 includes a driving environment construction module 210, a signal processing module 220, a command conversion module 230 and a vehicle control module 240;

[0098] A driving environment construction module 210 is used to construct a simulated driving environment including a visual induced stimulus source and a virtual vehicle;

[0099] Collecting the original EEG signals generated by the test driver when controlling the virtual vehicle under the action of the visual induced stimulus source, and sending the original EEG signals to the signal processing module 220;

[0100] The signal processing module 220 is used to classify and identify the original EEG signals based on different dimensions using the potential classification model and the EEG signal processing model, and send the classification and identification results to the instruction conversion module 230;

[0101] Determine the vehicle control command based on the classification recognition result, and use the vehicle control command and the original EEG signal to train the stimulation paradigm mapping model, so as to use the trained stimulation paradigm mapping model to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment;

[0102] The instruction conversion module 230 is used to convert the classification recognition result to obtain a vehicle control instruction, and send the vehicle control instruction to the vehicle control module 240;

[0103] The vehicle control module 240 is used to receive the vehicle control instructions sent by the instruction conversion module 230, and control the virtual vehicle to run in the simulated driving environment according to the vehicle control instructions.

[0104] It should be pointed out that relying solely on the autonomous driving simulation software cannot realize the execution of all vehicle control instructions. Therefore, it is necessary to construct a vehicle control module 240 to complete the vehicle control in the required scenario.

[0105] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 300 includes a processor 310 , a memory 320 and a bus 330 .

[0106] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 via the bus 330. When the machine-readable instructions are executed by the processor 310, the above-mentioned Figure 1 The specific implementation of the steps of the brain-computer interface stimulation paradigm generation method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0107] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps of the brain-computer interface stimulation paradigm generation method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0112] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0113] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a brain-computer interface stimulation paradigm, characterized in that: Applied to a brain-computer interface stimulation paradigm generation system, the method includes: Constructing a simulated driving environment including a visual induced stimulus source and a virtual vehicle, and obtaining original EEG signals generated by the test driver when controlling the virtual vehicle to drive under the action of the visual induced stimulus source; Using a potential classification model and an EEG signal processing model, the original EEG signal is classified and identified in different dimensions, and a vehicle control instruction is determined based on the classification and identification results in different dimensions; The stimulation paradigm mapping model is trained using the vehicle control instructions and the original EEG signal, so as to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment using the trained stimulation paradigm mapping model.

2. The method according to claim 1, characterized in that: The classification and recognition results under different dimensions include a first signal recognition result and a second signal recognition result. The potential classification model and the EEG signal processing model are used to classify and recognize the original EEG signal under different dimensions, respectively, including: Using the potential classification model, classifying and identifying the steady-state visual potential component in the original EEG signal to obtain a first signal recognition result; The electroencephalogram signal processing model is used to classify and identify other components in the original electroencephalogram signal to obtain a second signal recognition result, wherein the other components are other components in the original electroencephalogram signal except the steady-state visual potential component.

3. The method according to claim 2, characterized in that The potential classification model includes a spatial filter and a first classifier. The use of the potential classification model to classify and identify the steady-state visual potential component in the original EEG signal includes: Using the spatial filter to filter the original EEG signal to obtain a filtered original EEG signal; The first classifier is used to classify the filtered original EEG signal to obtain a first signal recognition result.

4. The method according to claim 2, characterized in that The EEG signal processing model includes a preprocessing model, a separation model and a second classifier. The EEG signal processing model is used to classify and identify other components in the original EEG signal, including: Preprocessing the original EEG signal using the preprocessing model to obtain a preprocessed original EEG signal; Using the separation model to perform component separation on the preprocessed original EEG signal to obtain a pure EEG signal; The clean EEG signal is input into the second classifier to obtain a second signal recognition result.

5. The method according to claim 2, characterized in that: The determining of the vehicle control instruction based on the classification and recognition results in different dimensions includes: Comparing the first signal recognition result with the second signal recognition result, and determining whether the recognition result meets a preset recognition requirement based on the comparison result, wherein the preset recognition requirement refers to a requirement that the two recognition results match; If the preset recognition requirements are met, the first signal recognition result and / or the second signal recognition result are converted into a recognizable vehicle control instruction to control the virtual vehicle to execute the vehicle control instruction.

6. The method according to claim 1, characterized in that The method of training the stimulation paradigm mapping model by using the vehicle control command and the original EEG signal includes: Acquire the instruction truth value corresponding to the original EEG signal, and use the instruction truth value as the standard value, and use the original EEG signal and the vehicle control instruction as the input data and output data of the stimulation paradigm mapping model respectively; The stimulation paradigm mapping model is trained using the standard value, the input data and the output data.

7. The method according to claim 1, characterized in that The construction includes a visual induced stimulus source and a simulated driving environment of a virtual vehicle, including: Constructing an initial simulated driving environment, wherein the initial simulated driving environment includes a virtual vehicle; At least one visually induced stimulus source flickering at a fixed frequency is provided in the simulated driving environment using a shader.

8. The method according to claim 3 or 4, characterized in that: The classification results of the first classifier and / or the second classifier each include multiple classification results, and the multiple classification results are determined based on a binary classification method.

9. A brain-computer interface stimulation paradigm generation system, characterized in that: The brain-computer interface stimulation paradigm generation system includes a driving environment construction module, a signal processing module, and a vehicle control module: The driving environment construction module is used to construct a simulated driving environment including a visual induced stimulus source and a virtual vehicle; collecting original EEG signals generated by the test driver when controlling the virtual vehicle to travel under the action of the visual induced stimulus source, and sending the original EEG signals to a signal processing module; The signal processing module is used to classify and identify the original EEG signals based on different dimensions using a potential classification model and an EEG signal processing model; Determine a vehicle control instruction based on the classification and recognition result, and train a stimulation paradigm mapping model using the vehicle control instruction and the original EEG signal, so as to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment using the trained stimulation paradigm mapping model; The vehicle control module is used to control the virtual vehicle to run in the simulated driving environment according to the vehicle control instruction.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the brain-computer interface stimulation paradigm generation method as described in any one of claims 1 to 8.

Images