Brain-computer interface stimulation paradigm generation method, system and electronic device

Abstract

The application provides a brain-computer interface stimulation paradigm generation method, system and electronic equipment, which is applied to a brain-computer interface stimulation paradigm generation system. The method comprises the following steps: constructing a simulation driving environment comprising a visual evoked stimulus source and a virtual vehicle, and obtaining original electroencephalogram signals generated by a test driver when the test driver controls the virtual vehicle to run under the action of the visual evoked stimulus source; using a potential classification model and an electroencephalogram signal processing model to respectively perform classification and recognition of the original electroencephalogram signals in different dimensions, and determining a vehicle control instruction based on the classification and recognition results in different dimensions; and training a stimulation paradigm mapping model by using the vehicle control instruction, so as to determine a brain-computer interface stimulation paradigm corresponding to the simulation driving environment by using the trained stimulation paradigm mapping model. By using the brain-computer interface stimulation paradigm generation method, system and electronic equipment, the problem of low accuracy of signal control and signal recognition of the brain-computer interface of an intelligent automobile is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brain-computer interface, in particular to a brain-computer interface stimulation paradigm generation method and system and an electronic device. BACKGROUND

[0002] The brain-computer interface technology can establish a connection between the brain and the external environment, and control the external device by using the brain electrical signals generated by the brain consciousness activity. The steady state visual evoked potential (SSVEP) is the change of the brain electrical signal potential generated after the brain is subjected to a fixed frequency visual stimulation. Due to its short training time, fast communication rate, simple device configuration and other advantages, it is widely used in medical, aerospace and military fields. At present, with the substantial increase of the number of automobiles, intelligent automobiles are widely used and popularized, and the intelligent degree of intelligent automobiles is required to be improved, and the demand for intelligent brain-computer interaction function is also rapidly increasing.

[0003] However, the brain-computer interface technology of intelligent automobile is still in its infancy at present, and the related technology of brain-computer interface function test is particularly weak, and the brain electrical signal is easily affected by external stimulation, and error control signals are easily generated, and the misrecognition rate is high. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a brain-computer interface stimulation paradigm generation method, system and electronic device to solve the problem of low accuracy of signal control and signal recognition of intelligent automobile brain-computer interface.

[0005] In a first aspect, the embodiments of the present application provide a brain-computer interface stimulation paradigm generation method applied to a brain-computer interface stimulation paradigm generation system, and the method comprises the following steps:

[0006] A simulation driving environment including a visual evoked stimulation source and a virtual vehicle is constructed, and original brain electrical signals generated by a subject driver when controlling the virtual vehicle to drive under the action of the visual evoked stimulation source are obtained;

[0007] The original brain electrical signals are classified and recognized in different dimensions by using a potential classification model and a brain electrical signal processing model, and a vehicle control instruction is determined based on the classification and recognition results in different dimensions.

[0008] The stimulation paradigm mapping model is trained by using the vehicle control instruction and the original brain electrical signals, so as to determine the brain-computer interface stimulation paradigm of the subject driver in the simulation driving environment by using the trained stimulation paradigm mapping model.

[0009] Optionally, the classification and recognition results in different dimensions include a first signal recognition result and a second signal recognition result, and the original EEG signals are classified and recognized in different dimensions by using the potential classification model and the EEG signal processing model, including: the steady-state visual potential component in the original EEG signals is classified and recognized by using the potential classification model to obtain the first signal recognition result; and other components in the original EEG signals are classified and recognized by using the EEG signal processing model to obtain the second signal recognition result, the other components being other components in the original EEG signals except the steady-state visual potential component.

[0010] Optionally, the potential classification model includes a spatial filter and a first classifier, and the steady-state visual potential component in the original EEG signals is classified and recognized by using the potential classification model, including: the original EEG signals are filtered by using the spatial filter to obtain filtered original EEG signals; and the filtered original EEG signals are classified by using the first classifier to obtain the first signal recognition result.

[0011] Optionally, the EEG signal processing model includes a preprocessing model, a separation model and a second classifier, and the other components in the original EEG signals are classified and recognized by using the EEG signal processing model, including: the original EEG signals are preprocessed by using the preprocessing model to obtain preprocessed original EEG signals; the preprocessed original EEG signals are subjected to component separation processing by using the separation model to obtain pure EEG signals; and the pure EEG signals are input into the second classifier to obtain the second signal recognition result.

[0012] Optionally, the vehicle control instruction is determined based on the classification and recognition results in different dimensions, including: the first signal recognition result and the second signal recognition result are compared, and whether the recognition results meet a preset recognition requirement is determined according to a comparison result, the preset recognition requirement being a requirement that the two recognition results match; if the preset recognition requirement is met, the first signal recognition result and / or the second signal recognition result is converted into an identifiable vehicle control instruction to control the virtual vehicle to execute the vehicle control instruction.

[0013] Optionally, the stimulus paradigm mapping model is trained by using the vehicle control instruction and the original EEG signals, including: an instruction true value corresponding to the original EEG signals is obtained, the instruction true value is taken as a standard value, and the original EEG signals and the vehicle control instruction are taken as input data and output data of the stimulus paradigm mapping model respectively; and the stimulus paradigm mapping model is trained by using the standard value, the input data and the output data.

[0014] Optionally, the simulation driving environment including the visual evoked stimulus source and the virtual vehicle is constructed, including: constructing an initial simulation driving environment, the initial simulation driving environment including the virtual vehicle; providing at least one visual evoked stimulus source flickering at a fixed frequency in the simulation driving environment by using a shader.

[0015] Optionally, the classification results of the first classifier and / or the second classifier each include a plurality, and the plurality of classification results are determined based on a binary classification method.

[0016] In a second aspect, the embodiments of the present application further provide a brain-computer interface stimulation paradigm generation system, the brain-computer interface stimulation paradigm generation system including a driving environment construction module, a signal processing module, and a vehicle control module:

[0017] The driving environment construction module is configured to construct a simulation driving environment including a visual evoked stimulus source and a virtual vehicle.

[0018] The original brain electrical signals generated by the subject driver when controlling the virtual vehicle to run under the action of the visual evoked stimulus source are collected, and the original brain electrical signals are sent to the brain electrical signal processing module.

[0019] The signal processing module is configured to use a potential classification model and a brain electrical signal processing model to respectively perform classification and recognition of the original brain electrical signals based on different dimensions.

[0020] The vehicle control instructions are determined based on the classification and recognition results, and the stimulation paradigm mapping model is trained using the vehicle control instructions and the original brain electrical signals, so as to determine the brain-computer interface stimulation paradigm of the subject driver in the simulation driving environment by using the trained stimulation paradigm mapping model.

[0021] The vehicle control module is configured to control the virtual vehicle to run in the simulation driving environment according to the vehicle control instructions.

[0022] In a third aspect, the embodiments of the present application further provide an electronic device, including a processor, a memory, and a bus, the memory storing machine-readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the brain-computer interface stimulation paradigm generation method as described above.

[0023] The embodiments of the present application bring the following beneficial effects:

[0024] The brain-computer interface stimulation paradigm generation method, system and electronic device provided by the embodiment of the application can induce the original brain electrical signal of the subject driver in the current simulation driving environment through the visual evoked stimulation source, and utilize the potential classification model and the brain electrical signal processing model to respectively classify and identify the original brain electrical signal in different dimensions, thereby avoiding the problems of error control signals and misidentification caused by abnormal fluctuations of the brain electrical signal due to the differences of the driver and the driving environment, and improving the accuracy of model training by utilizing the classification and identification results in different dimensions to train the stimulation paradigm mapping model, compared with the brain-computer interface stimulation paradigm generation method in the prior art, the problems of low accuracy of signal control and signal identification of the intelligent automobile brain-computer interface are solved.

[0025] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0027] Figure 1 A flowchart of the brain-computer interface stimulation paradigm generation method provided by the embodiment of the present application is shown;

[0028] Figure 2 A schematic diagram of the electrode placement position of the electroencephalograph provided by the embodiment of the present application is shown;

[0029] Figure 3 A structural schematic diagram of the brain-computer interface stimulation paradigm generation system provided by the embodiment of the present application is shown;

[0030] Figure 4 A structural schematic diagram of the electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0032] It is worth noting that prior to this application, brain-computer interface (BCI) technology could establish a connection between the brain and the external environment, using brain electrical signals generated by conscious brain activity to control external devices. Steady-state visual evoked potentials (SSVEPs) refer to the changes in brain electrical signal potentials generated after applying visual stimulation at a fixed frequency to the human brain. Due to their advantages such as short training time, fast communication speed, and simple equipment configuration, they are widely used in medical, aerospace, and military fields. Currently, with the significant increase in car ownership and the widespread application and popularization of intelligent vehicles, human demands for the level of intelligence in intelligent vehicles are increasing, and the demand for intelligent brain-computer interface functions is also rapidly growing. However, at present, intelligent vehicle BCI technology is still in its early stages, and related technologies for testing BCI functions are particularly weak. Furthermore, intelligent vehicle brain electrical signals are easily affected by external stimuli, prone to generating erroneous control signals, and have a high misidentification rate.

[0033] Based on this, this application provides a method for generating brain-computer interface stimulation paradigms to improve the accuracy of signal control and signal recognition in intelligent vehicle brain-computer interfaces.

[0034] Please see Figure 1 , Figure 1 This is a flowchart illustrating a brain-computer interface stimulation paradigm generation method provided in an embodiment of this application. Figure 1 As shown, the brain-computer interface stimulation paradigm generation method provided in this application embodiment is applied to a brain-computer interface stimulation paradigm generation system, including:

[0035] Step S101: Construct a simulated driving environment including visual evoked stimuli and a virtual vehicle, and acquire the original electroencephalogram (EEG) signals generated by the test driver when controlling the virtual vehicle under the action of visual evoked stimuli.

[0036] In step S102, the original brain electrical signals are classified and recognized in different dimensions by using the potential classification model and the brain electrical signal processing model, and a vehicle control instruction is determined based on the classification and recognition results in different dimensions.

[0037] In step S103, the stimulus paradigm mapping model is trained by using the vehicle control instruction and the original brain electrical signals, so as to determine the brain-computer interface stimulus paradigm of the subject driver in the simulation driving environment by using the trained stimulus paradigm mapping model.

[0038] By the brain-computer interface stimulus paradigm generation method, the subject driver generates original brain electrical signals in the current simulation driving environment by using the visual evoked stimulus source, and the original brain electrical signals are classified and recognized in different dimensions by using the potential classification model and the brain electrical signal processing model. The problem of false control signals and misrecognition caused by abnormal fluctuations of brain electrical signals due to differences in drivers and driving environments is avoided. The accuracy of model training is improved by training the stimulus paradigm mapping model by using the classification and recognition results in different dimensions, and the problem of low accuracy of signal control and signal recognition of the intelligent automobile brain-computer interface is solved.

[0039] Each step in the method will be described in detail below. Figure 1

[0040] In step S101, a simulation driving environment including a visual evoked stimulus source and a virtual vehicle is constructed, and original brain electrical signals generated by a subject driver when controlling the virtual vehicle to drive under the action of the visual evoked stimulus source are obtained.

[0041] The visual evoked stimulus source can refer to a stereoscopic visual evoked stimulus source that flashes at a fixed frequency, and the visual evoked stimulus source is used to stimulate the brain of the subject driver to generate an SSVEP response.

[0042] The virtual vehicle can refer to a virtual three-dimensional vehicle model in the simulation driving environment, which is used to simulate a real vehicle to display real-time control feedback for the virtual vehicle in the simulation driving environment.

[0043] The simulation driving environment can refer to an automatic driving simulation test environment, and the simulation driving environment includes an automatic driving simulation test scene. For example, the automatic driving simulation test scene is a virtual test scene.

[0044] ​In the embodiment of the present application, the driving environment construction module in the brain-computer interface stimulation paradigm generation system can be used to construct a simulation driving environment. In order to construct a highly realistic virtual driving environment including various road conditions, weather conditions and traffic situations, and record data in real time during driving, such as the state parameters of the virtual vehicle, here, the SCANeR studio automatic driving simulation software can be used to build an automatic driving simulation test scene. Specifically, according to the test requirements, the road surface corresponding to the automatic driving simulation test scene can be created under the TERRAIN interface of the automatic driving simulation software, and the road type can be selected, for example, straight, crossroads, roundabout, etc., and then the road can be parameterized to define the length, width and other specific values of the road. The virtual vehicle can be modeled under the VEHICLE interface, or the vehicle model provided by the automatic driving simulation software can be directly called. Finally, under the SCENARIO mode, based on the parameters of the constructed scene library, traffic participants, road facilities, etc. are set, and in the script editing box, some special trigger conditions including intelligent voice control, weather, etc. are controlled. In this way, an initial simulation driving environment is constructed, and the initial simulation driving environment includes a virtual vehicle.

[0045] Meanwhile, the automatic driving simulation software can be used to add a visual evoked stimulus source to the initial simulation driving environment, or a display screen can be additionally provided in front of the test driver, and a visual evoked stimulus source can be set in the display screen. Regardless of which of the above ways is used to add a visual evoked stimulus source, a shader can be used to provide at least one stereoscopic visual evoked stimulus source that flashes at a fixed frequency in the simulation driving environment, so as to stimulate the brain of the test driver to generate an SSVEP response. Specifically, a vertex shader is set in the stimulus source control software of the additionally provided display screen or the automatic driving simulation software, the vertex data of the stereoscopic visual evoked stimulus source model is converted to the clipping coordinate system through matrix transformation in the vertex shader, and a matrix C is defined in the vertex shader to store the R, G and B channel data of different colors of the stereoscopic visual evoked stimulus source. For example, assuming that the stereoscopic visual evoked stimulus source needs to switch between displaying two colors, the matrix C can be represented as:

[0046]

[0047] In the above formula, R i represents the value of the R channel of the i-th color, G i represents the value of the G channel of the i-th color, and B i represents the value of the B channel of the i-th color.

[0048] Then, according to the preset display position, the row index L i at the current time (starting from t=0) is adjusted, where the row index calculation formula is:

[0049] L i = Fl[(2·t·f)$%2];

[0050] In the above formula, Fl[] represents the floor function; t represents the time count after the program runs; f represents the flicker frequency of the stereoscopic vision evoked stimulus source; and %2 is the remainder when divided by 2.

[0051] After the row index is calculated, the color value corresponding to the row index can be transmitted into the fragment shader, and the color value is output in the fragment shader to display the stereoscopic vision evoked stimulus source that alternately flickers on the screen.

[0052] In addition, the simulated driving environment also includes an electroencephalograph arranged on the head of the test driver to collect the electroencephalogram of the test driver. The electroencephalogram includes delta waves, theta waves, alpha waves and beta waves. The electroencephalograph can use the international 10-20 system electrode placement method, which has a total of 14+2 channels (including 2 reference points).

[0053] The following will be introduced Figure 2 to introduce the positions of the channels in the electroencephalograph.

[0054] Figure 2 The schematic diagram of the electrode placement position of the electroencephalograph provided by the embodiment of the application is shown, as shown in Figure 2 The electroencephalograph includes a plurality of channels, the channels AF3, F3, F7, AF4, F4 and F8 are arranged in the frontal lobe region, the channels P3, P7, P4 and P8 are arranged in the parietal lobe region, the channels FC5, T7, FC6 and T8 are arranged in the temporal lobe region, and the channels O1 and O2 are arranged in the occipital lobe region. Among them, AF represents the forehead, F represents the frontal lobe, P represents the parietal lobe, FC represents the frontal central, T represents the temporal lobe, and O represents the occipital lobe.

[0055] Finally, the human-machine environment synchronization cloud platform system ErgoLAB, the automatic driving simulation software SCANeR studio and the SimEASY driving simulator are connected to build a brain-computer interface simulation test platform. The brain-computer interface simulation test platform records the electroencephalogram of the test driver in real time through the ErgoLAB system, and records the data of the virtual vehicle synchronously through the automatic driving simulation software, so as to complete the construction of the simulated driving environment.

[0056] After the construction of the simulated driving environment is completed, the test driver executes the preset vehicle control instruction, so that the virtual vehicle runs in the automatic driving simulation test scene, and at the same time, the original electroencephalogram generated by the test driver when controlling the virtual vehicle to drive or run under the action of the visual evoked stimulus source is obtained. The vehicle control instruction includes but is not limited to: vehicle left turn, vehicle right turn, vehicle acceleration, brake, vehicle window lifting, vehicle music switching, air conditioning temperature adjustment, etc.

[0057] In step S102, the original brain electrical signal is classified and recognized in different dimensions by using the potential classification model and the brain electrical signal processing model, and a vehicle control instruction is determined based on the classification and recognition results in different dimensions.

[0058] The potential classification model can be a neural network model, and the potential classification model is used to extract the SSVEP signal in the brain electrical signal and obtain a classification and recognition result based on the extracted SSVEP signal.

[0059] The brain electrical signal processing model can be a model used to extract normal brain electrical signals and obtain a classification and recognition result based on the normal brain electrical signals.

[0060] The different dimensions include SSVEP signal dimensions and other signal dimensions, and the other signal dimensions can refer to the dimensions of other brain electrical signals except the SSVEP signal.

[0061] The classification and recognition results in different dimensions include a first signal recognition result and a second signal recognition result. The first signal recognition result is obtained by classifying and recognizing the SSVEP signal in the original brain electrical signal by the potential classification model, and the second signal recognition result is obtained by classifying and recognizing other signals in the original brain electrical signal except the SSVEP signal by the brain electrical signal processing model.

[0062] The vehicle control instruction can be an instruction for controlling a virtual vehicle.

[0063] In the embodiments of the present application, since the SSVEP is a special brain electrical signal, it needs to be processed separately. Therefore, the steady-state visual potential component in the original brain electrical signal, i.e., the SSVEP signal, can be extracted by using the potential classification model, and the steady-state visual potential component is classified and recognized to obtain a first signal recognition result. At the same time, other components in the original brain electrical signal are classified and recognized by using the brain electrical signal processing model to obtain a second signal recognition result, wherein the other components are other normal brain electrical signals except the steady-state visual potential component in the original brain electrical signal.

[0064] Before classifying and recognizing by using the potential classification model, historical brain electrical signal data including the SSVEP signal can be obtained, and the potential classification model is trained offline by using the historical brain electrical signal data. The potential classification model includes a spatial filter and a first classifier.

[0065] Here, when the potential classification model is trained offline, in order to effectively extract the frequency response characteristics of the signal, the signal needs to be filtered in a specific frequency band first. A band-pass filter corresponding to the fundamental wave (f) and the second harmonic (2f) of the electroencephalogram signal under each target channel is set, and the frequency band range of the band-pass filter is [f-0.25, f+0.25] and [2f-0.25, 2f+0.25] respectively.

[0066] In an example, when classification and recognition are performed by using the potential classification model, a spatial filter can be used to filter the original electroencephalogram signal to obtain a filtered original electroencephalogram signal. Specifically, after the electroencephalogram signal is preprocessed, a set of spatial filter matrices can be obtained by training the offline historical electroencephalogram signal data through common spatial patterns (CSP), so as to extract the features of the electroencephalogram signal.

[0067] Here, when the spatial filter is constructed, if the classification and recognition result is two classifications, in the case of two classifications, the construction process of the spatial filter is as follows: let the i-th electroencephalogram signal sample in the two categories be and the normalized covariance matrix of is:

[0068]

[0069] In the above formula, tr() represents the sum of the elements on the diagonal of the matrix.

[0070] The construction formula of the spatial filter is W=PB, where P represents the whitening matrix of the covariance matrix R, and B represents the combination of the eigenvectors corresponding to the maximum eigenvalue and the minimum eigenvalue of the matrix after whitening. The historical electroencephalogram signal data is used as a signal data sample, and the feature matrix Z of the signal data sample after spatial filtering is represented as Z=WX, where W represents the spatial filtering matrix, and X represents the signal data sample. Then, the logarithm of the filtered eigenvector is taken as the electroencephalogram signal feature, and the electroencephalogram signal feature of the j-th row is represented as:

[0071]

[0072] In the above formula, var(Z j ) represents the variance of the j-th row of the matrix Z, j=1, 2, …, 2m, and 2m represents the number of rows of the matrix Z.

[0073] In an example, the classification results of the first classifier and / or the second classifier each include multiple, and the multiple classification results are determined based on binary classification. If the classification recognition result is multiple classification, the multiple classification is converted into multiple binary classification for processing by a "one-to-many" method. Here, one of the n classifications can be selected to set the corresponding covariance matrix Then, the covariance matrix of the remaining other classifications is combined with the class frequency to perform weighted average as Solve the spatial filter under this binary classification by using and , and in this way, n spatial filters can be sequentially constructed. The number of classifications of the classification recognition result is set, and the specific number of classifications of the classification recognition result can be determined by a person skilled in the art according to the actual situation, which is not limited in the present application.

[0074] Then, the filtered original electroencephalogram signal is classified and processed by using the first classifier to obtain a first signal recognition result. At this time, the first classifier needs to be constructed, and when constructing the first classifier, a linear discriminant analysis method can be used to construct the first classifier. This method distinguishes different types of signals by linearly combining sample signal features. The sample signal feature is the feature extracted by the spatial filter. If the classification recognition result is binary classification, the sample signal feature is projected onto a straight line, so that the same sample signal features are as close as possible, and the different sample signal features are as far away as possible, to obtain the maximum target of linear discriminant analysis (LDA).

[0075] The specific implementation process is as follows: assuming that the projection straight line is ω, and taking an arbitrary sample signal feature x i , the within-class scatter matrix S ω is defined as S ω =∑(x-μ0)(x-μ0) T +∑(x-μ1)(x-μ1) T , where μ0 and μ1 are the center points of the two classes of samples. The between-class scatter matrix S b is defined as S b =(μ0-μ1)(μ0-μ1) T . The objective function J(w) of LDA maximization is:

[0076]

[0077] Likewise, if the classification recognition result is n classifications, it can also be expanded to multiple "binary classifications" through "one-to-many" to solve. That is, one of the n classifications is taken out, and the remaining n-1 classifications are combined into another classification, and the first classifier under this classification condition can be obtained through training. By analogy, a total of n first classifiers can be obtained.

[0078] In this way, after the spatial filter and the first classifier are constructed, the online collected brain electrical signals of the test driver can be preprocessed, and the preprocessed brain electrical signals are extracted through the spatial filter, and after the extracted features are classified through the first classifier, the first signal recognition result is obtained.

[0079] In an example, the brain electrical signal processing model includes a preprocessing model, a separation model, and a second classifier. When using the brain electrical signal processing model for classification recognition, the original brain electrical signal can be preprocessed using the preprocessing model to obtain the preprocessed original brain electrical signal. Then, the preprocessed original brain electrical signal is processed by the separation model to obtain a pure brain electrical signal; the pure brain electrical signal is input into the second classifier to obtain a second signal recognition result. Here, the construction method of the second classifier is the same as that of the first classifier, which will not be described here.

[0080] Specifically, the brain electrical signal processing model can be obtained by constructing and training a neural network model, or can be realized by setting software. Exemplarily, the setting software can be MATLAB software. When using MATLAB software to process the original brain electrical signal, since MATLAB software includes an EEGLAB toolbox, the collected original brain electrical signal of the test driver can be imported into the EEGLAB toolbox to use the preprocessing plug-in for electroencephalograph electrode positioning, low-pass / high-pass filtering, effective data segmentation, and baseline correction processing. At the same time, the ICLabel plug-in in the toolbox is used as a separation model, and the ICLabel plug-in is used for independent component analysis (ICA) to separate the mixed brain electrical signal into a pure brain electrical signal of independent components. Finally, the pure brain electrical signal processed by the preprocessing and independent component analysis is input into the second classifier to obtain a second signal recognition result.

[0081] In determining the vehicle control instruction, the first signal recognition result and the second signal recognition result can be compared, and it is determined whether the recognition result meets the preset recognition requirement according to the comparison result. The preset recognition requirement refers to the requirement that the two recognition results match. If the first signal recognition result matches the second signal recognition result, it is determined that the preset recognition requirement is met, otherwise, it is determined that the preset recognition requirement is not met. For example, the first signal recognition result is to turn left, and the second signal recognition result is also to turn left, it is determined that the first signal recognition result matches the second signal recognition result, and the preset recognition requirement is met.

[0082] Whether the preset recognition requirement is met or not, the first signal recognition result and the second signal recognition result are converted into a vehicle control instruction, and the vehicle control instruction is taken as the output of the stimulus paradigm model for training of the stimulus paradigm model. Further, if the preset recognition requirement is met, the first signal recognition result and / or the second signal recognition result is converted into an identifiable vehicle control instruction to control the virtual vehicle to execute the vehicle control instruction, and to display the execution result of executing the vehicle control instruction in the simulation driving scene.

[0083] In step S103, the stimulus paradigm mapping model is trained by using the vehicle control instruction and the original brain electrical signal, so as to determine the brain-computer interface stimulus paradigm of the test driver in the simulation driving environment by using the trained stimulus paradigm mapping model.

[0084] The stimulus paradigm mapping model can refer to a neural network model trained by using the original brain electrical signal and the vehicle control instruction. The stimulus paradigm mapping model can represent the brain-computer interface stimulus paradigm of the test driver in the simulation driving environment.

[0085] In the embodiment of the present application, when the stimulus paradigm mapping model is trained, the instruction true value corresponding to the original brain electrical signal can be obtained, and the instruction true value is taken as the standard value, and the original brain electrical signal and the vehicle control instruction are taken as the input data and the output data of the stimulus paradigm mapping model respectively. Then, the standard value, the input data and the output data are used to train the stimulus paradigm mapping model. The instruction true value can refer to the vehicle control instruction actually issued by the test driver. For example, it is previously set that the test driver needs to issue the vehicle control instruction of "turning left", and the vehicle control instruction is the instruction true value. When the output result of the brain-computer interface stimulus paradigm generation system deviates from the instruction true value, the instruction true value is taken as the standard value to train the stimulus paradigm mapping model. Through the trained stimulus paradigm mapping model, the brain-computer interface stimulus paradigm of the test driver in the simulation driving environment can be determined.

[0086] The stimulus paradigm mapping model is a neural network model, and in the neural network model, represents the linear coefficient from the jth neuron of the l-1th layer to the kth neuron of the lth layer; the lth layer neural network is denoted as L l ; the activation function of each layer network node is σ(z); the linear relationship is is the mth bias of the lth layer network, and the calculation method of m conforms to the 2n+1 rule, and n is the number of input nodes.

[0087] Taking the input and output calculation rule of the lth layer neuron as an example, the output of each neuron is denoted as l is the number of layers, and k is the serial number of the neuron, and the calculation method is as follows:

[0088]

[0089] Further, the above calculation method is arranged, and the mathematical expression of the output of each layer neuron node in the model is as follows:

[0090]

[0091] In the above formula, x is the total number of signals generated by the stimulation paradigm, is the output of the kth neuron of the lth layer in the neural network; R q is the output vector of the qth neural network model; R p is the output sequence of the qth neural network model, which is used for training; I p , R p The corresponding stimulation paradigm mapping model is denoted as N p , that is, the deep neural network mapping model corresponding to the pth stimulation paradigm. In this way, the establishment of the brain-computer interface stimulation paradigm mapping model is completed.

[0092] On the basis of the above brain-computer interface stimulation paradigm mapping model, the electroencephalogram data of the subject driver in different simulation driving environments is input into each feature model as input data, so that the stimulation paradigm mapping output sequence Rp can be obtained. Then, the difference between the stimulation paradigm mapping output sequence and each original electroencephalogram signal is tested by goodness of fit, and the calculation method is as follows:

[0093]

[0094] In the above formula, R 3 represents a measurement index of goodness of fit, t and r represent the output sequence and the original electroencephalogram signal respectively, R 3The closer the value is to 1, the closer the linear rule of the stimulation paradigm mapping model is, and thus it can be known that the characteristic model with the highest goodness of fit can more accurately reflect the driving task (i.e., the instruction true value) of the subject driver, so as to achieve the effect of brain-computer interface stimulation paradigm mapping.

[0095] In addition, the subject driver can be one or multiple. When the subject driver is multiple, the commonality of the brain-computer interface stimulation paradigm can be extracted by collecting the brain electrical signals of the multiple subject drivers and using the stimulation paradigm mapping model, so as to establish a brain-computer interface stimulation paradigm suitable for a wider user group.

[0096] Based on the same inventive concept, the present application also provides a brain-computer interface stimulation paradigm generation system corresponding to the brain-computer interface stimulation paradigm generation method. Since the system in the present application solves the problem by the similar principle as the above-mentioned brain-computer interface stimulation paradigm generation method in the present application, the implementation of the system can be referred to the implementation of the method, and the repeated parts will not be described here.

[0097] Please refer to Figure 3 , Figure 3 The structure of a brain-computer interface stimulation paradigm generation system provided by the present application is shown in FIG. 2. As shown in FIG. 2, the brain-computer interface stimulation paradigm generation system 200 includes a driving environment construction module 210, a signal processing module 220, an instruction conversion module 230, and a vehicle control module 240. Figure 3

[0098] The driving environment construction module 210 is configured to construct a simulated driving environment including a visual evoked stimulus source and a virtual vehicle.

[0099] The original brain electrical signals generated by the subject driver when controlling the virtual vehicle to drive under the action of the visual evoked stimulus source are collected, and the original brain electrical signals are sent to the signal processing module 220.

[0100] The signal processing module 220 is configured to use a potential classification model and a brain electrical signal processing model to respectively perform classification and recognition of the original brain electrical signals based on different dimensions, and send the classification and recognition results to the instruction conversion module 230.

[0101] Based on the classification and recognition results, the vehicle control instructions are determined, and the stimulation paradigm mapping model is trained using the vehicle control instructions and the original brain electrical signals, so as to determine the brain-computer interface stimulation paradigm of the subject driver in the simulated driving environment by using the trained stimulation paradigm mapping model.

[0102] The instruction conversion module 230 is configured to convert the classification and recognition results to obtain the vehicle control instructions, and send the vehicle control instructions to the vehicle control module 240.

[0103] ​The vehicle control module 240 is configured to receive the vehicle control instruction sent by the instruction conversion module 230, and control the virtual vehicle to run in the simulation driving environment according to the vehicle control instruction.

[0104] It should be noted that the execution of all vehicle control instructions cannot be realized by relying on the automatic driving simulation software alone, and therefore, the vehicle control module 240 needs to be constructed to complete the vehicle control in the required scene.

[0105] Referring to Figure 4 , Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 4 , the electronic device 300 includes a processor 310, a memory 320 and a bus 330.

[0106] The memory 320 stores machine readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate through the bus 330. When the machine readable instructions are executed by the processor 310, the steps of the brain-computer interface stimulation paradigm generation method in the method embodiment shown in the above Figure 1 may be implemented. For specific implementation manners, refer to the method embodiment, which will not be described here.

[0107] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the brain-computer interface stimulation paradigm generation method in the method embodiment shown in the above Figure 1 may be implemented. For specific implementation manners, refer to the method embodiment, which will not be described here.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0109] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0111] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0112] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and various program code storage media.

[0113] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, used to illustrate the technical solutions of the present application, and not to limit them, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: any skilled person in the art within the technical scope disclosed by the present application, they can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and all should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating brain-computer interface stimulation paradigms, characterized in that, Applied to brain-computer interface stimulation paradigm generation systems, the methods include: A simulated driving environment including visual evoked stimuli and a virtual vehicle was constructed, and the original electroencephalogram (EEG) signals generated by the test driver when controlling the virtual vehicle under the action of the visual evoked stimuli were obtained. Using a potential classification model and an EEG signal processing model, the raw EEG signals are classified and identified in different dimensions, and vehicle control commands are determined based on the classification and identification results in different dimensions. The stimulus paradigm mapping model is trained using the vehicle control commands and the original EEG signals, so as to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment using the trained stimulus paradigm mapping model. The stimulus paradigm mapping model is used to characterize the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment. The classification and recognition results under different dimensions include first signal recognition results and second signal recognition results. The process of using a potential classification model and an EEG signal processing model to classify and recognize the original EEG signal under different dimensions includes: Using the aforementioned potential classification model, the steady-state visual potential components in the original EEG signal are classified and identified to obtain a first signal identification result; Using the EEG signal processing model, other components in the original EEG signal are classified and identified to obtain a second signal identification result. The other components are components in the original EEG signal other than the steady-state visual potential component. The training of the stimulus paradigm mapping model using the vehicle control commands and the raw EEG signals includes: Obtain the command truth value corresponding to the original EEG signal, and use the command truth value as the standard value. Use the original EEG signal and the vehicle control command as the input data and output data of the stimulus paradigm mapping model, respectively. The stimulus paradigm mapping model is trained using the standard values, the input data, and the output data.

2. The method according to claim 1, characterized in that, The potential classification model includes a spatial filter and a first classifier. The process of classifying and identifying the steady-state visual potential components in the original EEG signal using the potential classification model includes: The original EEG signal is filtered using the spatial filter to obtain the filtered original EEG signal. The filtered raw EEG signal is classified using the first classifier to obtain a first signal recognition result.

3. The method according to claim 2, characterized in that, The EEG signal processing model includes a preprocessing model, a separation model, and a second classifier. The step of using the EEG signal processing model to classify and identify other components in the original EEG signal includes: The raw EEG signal is preprocessed using the preprocessing model to obtain the preprocessed raw EEG signal. The separation model is used to perform component separation processing on the preprocessed raw EEG signal to obtain a pure EEG signal; The pure EEG signal is input into the second classifier to obtain the second signal recognition result.

4. The method according to claim 1, characterized in that, The determination of vehicle control commands based on classification and recognition results under different dimensions includes: The first signal recognition result is compared with the second signal recognition result, and it is determined whether the recognition result meets the preset recognition requirements based on the comparison result. The preset recognition requirements refer to the requirement that the two recognition results match. If the preset recognition requirements are met, the first signal recognition result and / or the second signal recognition result are converted into a recognizable vehicle control command to control the virtual vehicle to execute the vehicle control command.

5. The method according to claim 1, characterized in that, The construction of the simulated driving environment includes visual evoked stimuli and virtual vehicles, including: Construct an initial simulated driving environment, which includes a virtual vehicle; The shader is used to provide at least one visually evoked stimulus that flashes at a fixed frequency in the simulated driving environment.

6. The method according to claim 3, characterized in that, The classification results of the first classifier and / or the second classifier each include multiple classification results, which are determined based on a binary classification method.

7. A brain-computer interface stimulation paradigm generation system, characterized in that, The brain-computer interface stimulation paradigm generation system includes a driving environment construction module, a signal processing module, and a vehicle control module. The driving environment construction module is used to construct a simulated driving environment that includes visual stimuli and virtual vehicles. The raw electroencephalogram (EEG) signals generated by the test driver when controlling the virtual vehicle under the action of the visual evoked stimulus were collected, and the raw EEG signals were sent to the signal processing module. The signal processing module is used to classify and identify the original EEG signal based on different dimensions using a potential classification model and an EEG signal processing model. Based on the classification and recognition results, vehicle control commands are determined, and the vehicle control commands and the original EEG signals are used to train the stimulus paradigm mapping model. The trained stimulus paradigm mapping model is used to determine the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment. The stimulus paradigm mapping model is used to characterize the brain-computer interface stimulation paradigm of the test driver in the simulated driving environment. The vehicle control module is used to control the virtual vehicle to operate in the simulated driving environment according to the vehicle control instructions; The classification and recognition results under different dimensions include a first signal recognition result and a second signal recognition result. The signal processing module is specifically used for: Using the aforementioned potential classification model, the steady-state visual potential components in the original EEG signal are classified and identified to obtain a first signal identification result; Using the EEG signal processing model, other components in the original EEG signal are classified and identified to obtain a second signal identification result. The other components are components in the original EEG signal other than the steady-state visual potential component. Obtain the command truth value corresponding to the original EEG signal, and use the command truth value as the standard value. Use the original EEG signal and the vehicle control command as the input data and output data of the stimulus paradigm mapping model, respectively. The stimulus paradigm mapping model is trained using the standard values, the input data, and the output data.

8. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the brain-computer interface stimulation paradigm generation method as described in any one of claims 1 to 6.

Images