Electroencephalogram and eye movement detection method and device based on AR / VR scene simulation and storage medium

Abstract

The invention relates to the technical field of cognitive neuroscience, and particularly discloses an electroencephalogram and eye movement detection method and device based on AR / VR scene simulation and a storage medium. The method comprises the steps that an electroencephalogram signal and an eye movement signal are obtained; performing data preprocessing on the electroencephalogram signals and the eye movement signals; according to a short-time Fourier transform algorithm and the characteristics of the iris tremor effect, electroencephalogram signal and eye movement signal characteristics are extracted; smoothing the features of the electroencephalogram signals and the eye movement signals through linear power system smoothing; performing eye movement signal feature saliency detection on the eye movement signal by using single-factor variance analysis; a dual-mode depth automatic encoder is adopted, and a deep neural network is combined with an electroencephalogram signal and an eye movement signal to carry out emotion recognition. The problems that in the prior art, data collected by electroencephalogram equipment and eye movement equipment are complex, and analysis needs to be processed and interpreted by professionals are solved.

Description

Technical Field

[0001] The present application relates to the field of cognitive neuroscience technology, and specifically to a method, device and storage medium for detecting electroencephalogram and eye movement based on AR / VR scene simulation. Background Art

[0002] The combination of EEG and eye tracking technology provides a multi-dimensional approach to studying brain activity, which can deeply understand people's cognitive processes and psychological states from both physiological and behavioral levels. With the development of emerging technologies such as computer vision, graphics, and artificial intelligence, the combination of EEG and eye tracking technology, especially when used synchronously in a fully immersive virtual reality (VR) environment, has demonstrated its potential in neuroscience research. For example, by implementing a brain-computer interface speller based on hybrid steady-state visual evoked potentials in a VR environment, potential use cases for synchronized data streams were studied. This integrated approach not only improves the synchronization accuracy of data, but also opens up new ways to study brain activity in an ecologically valid VR environment. In the field of human-computer interaction, combining EEG data and eye movement data can more accurately identify the user's interaction intention. However, existing EEG and eye movement devices are bulky and inconvenient to carry, which limits their flexible application in daily life. They cannot collect people's eye movement data and EEG data at the same time. The collected data is complex, and analysis requires professional personnel to process and interpret. Traditional equipment is inconvenient to wear and may cause discomfort during use, affecting the accuracy and reliability of data collection. Summary of the invention

[0003] The purpose of this application is to provide an EEG and eye movement detection method, device and storage medium based on AR / VR scene simulation, so as to solve the problem that the data collected by EEG equipment and eye movement equipment in the prior art are complex and the analysis requires professional personnel to process and interpret.

[0004] To achieve the above-mentioned purpose, the embodiment of the present application provides a method for detecting EEG and eye movement based on AR / VR scene simulation, comprising the following steps: obtaining EEG signals and eye movement signals;

[0005] Performing data preprocessing on the EEG signal and the eye movement signal;

[0006] Extract EEG signal and eye movement signal features based on the short-time Fourier transform algorithm and the characteristics of iris tremor effect;

[0007] By performing linear dynamic system smoothing, the characteristics of EEG signals and eye movement signals are smoothed;

[0008] One-way ANOVA was used to detect the significance of eye movement signal features.

[0009] A bimodal deep autoencoder is used, and a deep neural network is used to combine EEG signals and eye movement signals for emotion recognition.

[0010] Optionally, the data preprocessing includes:

[0011] The EEG signal is preprocessed by downsampling the EEG signal at a sampling rate of 200 Hz by using a 1-75 Hz bandpass filtering method; and the collected eye movement signal is preprocessed by using principal component analysis, specifically including:

[0012] The 1-75Hz bandpass filtering method is used to filter out the low-frequency electrooculogram signals and high-frequency electromyography signals in the original EEG signals, and the EEG signals are downsampled to 200Hz to preprocess the EEG signals.

[0013] Principal component analysis was used to estimate the common illumination reflection pattern of pupil diameter data of all subjects in the eye tracking signal. For one The matrix contains the pupil diameter information, represents the number of samples, represents the number of subjects in the sample, represents the number of variables in each sample, There are three signals:

[0014] ,

[0015] in, It is the strongest signal and is the reaction of pupil diameter to light; It is the parasympathetic nervous system’s response signal for emotion and attention; It's other noise.

[0016] Using principal component analysis Decomposition, the first principal component obtained can be considered as an approximate estimate of light reflection, remove the first principal component obtained, and get the remaining .

[0017] Optionally, the principal component analysis is used to Decomposition, remove the first principal component, and get the remaining , including:

[0018] ,

[0019] ,

[0020] ,

[0021] ,

[0022] ,

[0023] in, represents the matrix containing pupil diameter information, express The transpose of Represents the original signal Transpose The product of The column vector in the decomposition of principal component analysis is The matrix of eigenvectors of The diagonal line is The diagonal matrix of the eigenvalues ​​of Represents a column vector as The matrix of eigenvectors of express The transpose of Indicated by and The matrix composed of express The transpose of represents the illumination reflectance component matrix reconstructed using the first principal component, express The first principal component matrix of express The transpose of the matrix of the eigenvector corresponding to the first principal component of ; Represents the original signal Subtract the first principal component After the remaining part.

[0024] Optionally, the extracting features of EEG signals and eye movement signals according to the short-time Fourier transform algorithm and the features of iris tremor effect specifically includes:

[0025] The short-time Fourier transform algorithm is used to convert the EEG time series signal into frequency domain features, and then the energy is calculated on the obtained frequency domain features. The eye movement signal is short-time Fourier transform, and the characteristics of the iris tremor effect are used to align the eye movement signal features with the EEG signal features on the time axis.

[0026] Optionally, the linear dynamic system smoothing process is performed to smooth the features of the EEG signal and the eye movement signal, specifically including:

[0027] The linear dynamic system method is used to process EEG signals and eye movement signals, and the probability model is used to estimate the weights to retain the information related to emotions;

[0028] Optionally, the detecting of eye movement signal feature significance by using one-way analysis of variance on the eye movement signal specifically includes:

[0029] A one-way ANOVA was performed on the collected eye movement signals and all eye movement signal features to obtain the significance of pupil diameter, fixation duration, fixation deviation, saccade duration, saccade amplitude, and blink duration, to verify whether each eye movement signal indicator has significant differences between different emotion categories.

[0030] Optionally, the dual-modal deep autoencoder is used to perform emotion recognition using a deep neural network combined with EEG signals and eye movement signals, specifically including:

[0031] The EEG signal and the eye movement signal are fused by using a model fusion strategy or a fuzzy integral-based model fusion or a deep neural network-based model;

[0032] The model fusion strategy is used to fuse the EEG signal and the eye movement signal, including:

[0033] The model fusion strategies include maximum rule, sum rule and boda counting method. The maximum rule, sum rule and boda counting method are used to select the signal with the largest reflection in EEG signal and eye movement signal, and use it as the signal of emotion fusion.

[0034] The method of using a fuzzy integral-based model fusion to fuse the EEG signal and the eye movement signal includes:

[0035] Obtaining a fuzzy metric, using the fuzzy metric to represent the importance of each feature of the EEG signal and the eye movement signal, and using the fuzzy metric to perform weighted summation on the classifiers corresponding to different features, obtaining the overall confidence, and obtaining a fused model at the decision level;

[0036] The method of using a model based on a deep neural network to fuse the EEG signal and the eye movement signal includes:

[0037] Emotion recognition is performed using a deep neural network combined with EEG signals and eye tracking signals, which includes two parts: encoding and decoding. In the encoding part, two restricted Boltzmann Machine (RBM) models are trained using EEG signal features and eye movement signal features respectively. In the decoding part, two layers of RBM are expanded to reconstruct the input features to form a deep autoencoder.

[0038] To achieve the above purpose, the present application also provides an EEG and eye movement detection device based on AR / VR scene simulation, including: an EEG cap, a glasses body, a memory, and a processor, wherein:

[0039] The EEG cap, the glasses body and the processor are respectively connected to the first memory;

[0040] The EEG cap and the glasses body are used to synchronously collect EEG signals and eye movement signals, and transmit the EEG signals and eye movement signals to the first memory;

[0041] The processor is used to obtain the EEG signals and eye movement signals stored in the first memory, convert the electrical signals into digital signals, and perform data preprocessing on the EEG signals and eye movement signals, extract the features of the EEG signals and eye movement signals according to the short-time Fourier transform algorithm and the characteristics of the iris tremor effect, smooth the features of the EEG signals and eye movement signals by performing linear dynamic system smoothing processing, perform eye movement signal feature significance detection on the eye movement signals using one-way variance analysis, adopt a dual-modal deep autoencoder, and use a deep neural network in combination with the EEG signals and eye movement signals to perform emotion recognition.

[0042] To achieve the above objectives, the present application also provides a computer storage medium on which a computer program is stored, wherein the computer program implements the steps of the above method when executed by a machine.

[0043] The embodiments of the present application have the following advantages:

[0044] Through the above method, the user's interaction intention can be more accurately identified by combining EEG signals and eye movement signals. The recognition accuracy can be improved through feature extraction and machine learning pattern recognition network, which is of great significance for designing adaptive human-computer interaction systems and can better understand people's real thoughts and psychological processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the implementation methods of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings described below are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0046] Figure 1 A flowchart of a method for detecting EEG and eye movements based on AR / VR scene simulation provided by at least one embodiment of the present application;

[0047] Figure 2 A schematic diagram of the overall structure of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided by at least one embodiment of the present application;

[0048] Figure 3A schematic diagram of the shell structure of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided by at least one embodiment of the present application;

[0049] Figure 4 A schematic diagram of the internal structure of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided by at least one embodiment of the present application;

[0050] Figure 5 A schematic diagram of a glasses body of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided by at least one embodiment of the present application;

[0051] Figure 6 A schematic diagram of a memory of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided by at least one embodiment of the present application;

[0052] Figure 7 Another schematic diagram of a memory of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided for at least one embodiment of the present application.

[0053] Description of reference numerals:

[0054] Shell 1, EEG cap 2, glasses body 3, first memory 4, brain-computer connector 5, connecting cable 6, EEG interface 7, glasses conduction interface 8, brain-muscle wire 9, sensor 15, camera device 14, second memory 17, conduction connector 16, USB data connector 10, second data interface 19, first bandage 11, second bandage 12, first Velcro 13, first data interface 18, buckle 20, second Velcro 21, restraint belt 24, reel 25, battery compartment 26, cover 22, cover 23. DETAILED DESCRIPTION

[0055] The following is a description of the implementation of the present application by specific embodiments. People familiar with the technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0056] In the description of the present application, terms such as "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", and "outside" to indicate directions or positional relationships are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present application.

[0057] Unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0058] It should be noted that, in the claims and description of the present application, the steps may be executed substantially in parallel or in the reverse order under appropriate circumstances, depending on the functions involved, and the terms "first", "second" and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance.

[0059] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0060] An embodiment of the present application provides a method for detecting EEG and eye movement based on AR / VR scene simulation. Figure 1 , Figure 1 A flowchart of a method for detecting EEG and eye movements based on AR / VR scene simulation provided in at least one embodiment of the present application. It should be understood that the method may also include additional boxes not shown and / or the boxes shown may be omitted, and the scope of the present application is not limited in this respect. The following embodiments take the detection of EEG and eye movements during children's games as an example. It should be understood that the method provided in the present application may also be applied to other scenarios. The method includes:

[0061] Acquire physiological signals, including EEG signals and eye movement signals, through a brain-computer interface provided on the EEG cap, and store them in a first memory;

[0062] Performing data preprocessing on the EEG signal and the eye movement signal;

[0063] Extract EEG signals and eye movement signals based on the Short-term Fourier Transform (STFT) algorithm and the characteristics of iris tremor effects;

[0064] According to the linear dynamic system, the characteristics of the EEG signal and the eye movement signal are smoothed by performing linear dynamic system smoothing processing;

[0065] One-way ANOVA was used to detect the significance of eye movement signal features.

[0066] A bimodal deep autoencoder is used, and a deep neural network is used to combine EEG signals and eye movement signals for emotion recognition.

[0067] In some embodiments, the data preprocessing includes:

[0068] The EEG signals and eye movement signals in the first memory are transmitted to the computer via the first USB data interface, and the EEG signals are downsampled to a sampling rate of 200 Hz by adopting a bandpass filtering method of 1 to 75 Hz, and the EEG signals and eye movement signals collected during the children's game are preprocessed using principal component analysis.

[0069] In some embodiments, the specific steps of data preprocessing are as follows:

[0070] (1.1) The original EEG signal is preprocessed by downsampling the EEG signal at a sampling rate of 200 Hz using a 1-75 Hz bandpass filtering method.

[0071] (1.2) Preprocess the eye movement signal by principal component analysis, including: For one The matrix contains the pupil diameter information, represents the number of samples, represents the number of subjects in the sample, represents the number of variables in each sample, There are three signals:

[0072] (1-1)

[0073] in, is the strongest signal, the reaction of pupil diameter to light, It is the parasympathetic nervous system's emotional and attentional response signal. is other noise; the sources of these three components are independent of each other, so the decorrelation of principal component analysis can separate them.

[0074] Using principal component analysis Decomposition, the first principal component obtained can be considered as an approximate estimate of the light reflection. Then remove the first principal component obtained to obtain the remaining .

[0075] In some embodiments, the principal component analysis is used to Decomposition, remove the first principal component, and get the remaining , including:

[0076] ,

[0077] ,

[0078] ,

[0079] ,

[0080] (1-2)

[0081] in, represents the matrix containing pupil diameter information, express The transpose of Represents the original signal Transpose The product of The column vector in the decomposition of principal component analysis is The matrix of eigenvectors of The diagonal line is The diagonal matrix of the eigenvalues ​​of Represents a column vector as The matrix of eigenvectors of express The transpose of ; where and is an orthogonal matrix, is a diagonal matrix containing eigenvalues; Indicated by and The matrix composed of Also said The transpose of represents the illumination reflectance component matrix reconstructed using the first principal component, express The first principal component matrix of express The transpose of the matrix of the eigenvector corresponding to the first principal component of ; Represents the original signal Subtract the first principal component The remaining part contains other components and noise.

[0082] Specifically, in principal component analysis, the matrix is decomposed into the product of three matrices: and By selecting the eigenvector corresponding to the largest eigenvalue, the most important component can be reconstructed , which usually corresponds to the most varying signal in the data.

[0083] In some embodiments, the extracting features of EEG signals and eye movement signals based on the short-time Fourier transform algorithm and the features of the iris tremor effect specifically includes:

[0084] The short-time Fourier transform algorithm is used to convert the EEG time series signal into frequency domain features, and then the energy is calculated on the obtained frequency domain features.

[0085] Assume that the EEG timing signal in a certain time window is ,in is the number of samples in this window. g[n] , its short-time Fourier transform is:

[0086] (1-3)

[0087] in, Indicates the moving position of the window function in the signal; is the angular frequency, where is the frequency index, , is the total number of samples in the EEG signal; Indicates at location and angular frequency The short-time Fourier transform result under ; is the window function; is a complex exponential function used to calculate the spectrum value, where Is an imaginary unit.

[0088] Specifically, iris tremor is a tiny oscillation of the pupil diameter in the frequency band of 0.05-0.3 Hz, with an amplitude of 1 mm. The characteristics of the iris tremor effect are contained in the first two frequency bands (i.e. [0,0.2] Hz, [0.2,0.4] Hz) after the pupil diameter undergoes short-time Fourier transform. After feature extraction, the pupil diameter has 6 (4+1+1) eigenvalues. Since it has two dimensions, horizontal and vertical coordinates, there are a total of 12 features.

[0089] In some embodiments, the linear dynamic system smoothing process is used to smooth the features of the EEG signal and the eye movement signal, specifically including:

[0090] The linear dynamic system method is used to process EEG signals and eye movement signals, and the probability model is used to effectively estimate the weights to retain emotion-related information.

[0091] Specifically, the linear dynamic system method is used to process the EEG signal, which includes:

[0092] Assume that the observed EEG signal sequence is , which is called the observation sequence, and To represent the EEG signal features related to emotions, we call it the hidden sequence, and estimate the hidden sequence from the observed sequence by constructing the state space equation;

[0093] Assume that the initial emotion-related EEG features obey Gaussian distribution (Normal distribution function, ND), and estimate the hidden sequence from the observed sequence by constructing the state space equation. Assume that the estimated value is ,Right now:

[0094] (1-4)

[0095] The relationship between the hidden state sequence and the observation sequence is expressed by a series of linear relationships:

[0096] ,

[0097] ,

[0098] (1-5)

[0099] in, is the state transfer matrix, is the observation matrix, , and All are noise, Indicated in data samples is the noise in the hidden state, Indicated in The noise in the observation state is a data sample, and the noise term also follows a Gaussian distribution:

[0100] ,

[0101] ,

[0102] (1-6)

[0103] is the noise in the hidden state, with a mean of 0 and a covariance matrix of ; is the noise in the observation state, with a mean of 0 and a covariance matrix of ; is the noise associated with the initial state, with a mean of 0 and a covariance matrix of ;

[0104] The parameter set of this linear dynamic system can be expressed as .

[0105] Through sequencing , the hidden state The marginal posterior distribution can be To estimate the mean that is The maximum a posteriori estimate of and The estimation is done through an iterative algorithm.

[0106] The specific formula is:

[0107] ,

[0108] ,

[0109] (1-7)

[0110] in,

[0111] (1-8)

[0112] Refers to the sample The covariance of the hidden state estimate when ; yes The inversion of a matrix; is the observation matrix, Refers to the sample The Kalman gain at time , Refers to the sample The covariance matrix of the state estimate at ; refers to the covariance matrix of the hidden state noise, is the covariance matrix of the observation noise, Refers to the identity matrix.

[0113] The initial conditions of formula (1-7) are:

[0114] ,

[0115] (1-9)

[0116] in,

[0117] (1-10)

[0118] refers to the mean estimate of the hidden state at time 1, Refers to the mean of the initial hidden states.

[0119] When the observation sequence is known and parameters When , the above formula is used to iteratively estimate and The value of , thus obtaining the observation sequence The maximum a posteriori estimate of .

[0120] If you want to calculate the offline linear dynamic system, you can use the following formula:

[0121] ,

[0122] (1-11)

[0123] in,

[0124] (1-12)

[0125] The initial condition is,

[0126] ,

[0127] (1-13)

[0128] Therefore, in the observation sequence and parameters If known, estimate and , thus obtaining Maximum a posteriori estimate of ; refers to the estimated state mean, refers to the estimated state covariance; Refers to the gain matrix, which is used to update the estimated value.

[0129] For parameters , estimated using the Expectation Maximization (EM) algorithm. The specific steps of the algorithm are as follows:

[0130] (2.1) Initialize the parameters to ;

[0131] (2.2) Expectations : Calculate the posterior distribution ;

[0132] (2.3) Maximization : Use the expected estimate result in (2.2) to estimate the parameter Make an estimate:

[0133] (1-14)

[0134] in,

[0135] Q(θ, θ old )= E x 1 ,…, x na | θ old [ ln q ( x 1 ,…, x na , y 1 ,…, y na |θ)] (1-15)

[0136] If the model has converged, exit the algorithm; otherwise, , jump to step 2 to continue executing the algorithm.

[0137] Find the expectation in step (2.2) , you can use formula (1-11). The update of model parameters in step 3 is:

[0138] μ 0 new = E [ x 1 ] ,

[0139] A new = ∑ na = 2 NA E [ x na x na - 1 T ] ∑ na = 2 NA E [ x na x na - 1 T ] -1 ,

[0140] ,

[0141] Γ new = 1 NA - 1 ∑ na = 2 NA { E [ x na x na T ] - A new E [ x na - 1 x na T ]- E [ x na x na - 1 T ] (A new ) T + A new E [ x -1 x na - 1 T ] (A new ) T } ,

[0142] Σ new = 1 NA ∑ na = 1 NA { y na y na T - C new E [ x na ] y na T - y na E [ x na T ] (C new ) T + C new E [ x na x na T ] (C new ) T } (1-16)

[0143] in,

[0144] ,

[0145] E [ x na x na - 1 T ]= J na - 1 V na ̃ + μ na ̃ μ na - 1 ̃ T ,

[0146] E [ x na x na T ]= V na ̃ + μ na ̃ μ na ̃ T (1-17)

[0147] In the parameter set middle, is the state transition matrix, which represents the degree of change of the numerical value of the EEG signal feature; is the observation matrix, which represents the changing trend of EEG signal characteristics. Both parameters can be set to 1. It represents the mean of the initial hidden state. The average value of the EEG signal characteristics of children in a calm state before playing the game can be used as the value of this parameter. Set to a smaller value, parameter and The specific value can be manually selected through cross-validation method; refers to the estimated state mean, refers to the estimated state covariance; Refers to the new state transfer matrix, involving and The expected product of Refers to the new observation matrix, involving and The expected product of . and Refers to a new matrix, where the formula is more complicated and involves multiple expectations and matrix operations. Represents the total number of data samples.

[0148] In actual operation, the above method is used to obtain the values ​​of all parameters, and then the values ​​are used all the time without further training.

[0149] The eye movement signal characteristics are also smoothed using a linear dynamic system.

[0150] In some embodiments, the use of one-way analysis of variance to perform eye movement signal feature significance detection on the eye movement signal specifically includes:

[0151] A one-way ANOVA was performed on the collected eye movement signals and all eye movement signal features to obtain the significance of pupil diameter, fixation duration, fixation deviation, saccade duration, saccade amplitude, and blink duration, to verify whether each eye movement signal indicator has significant differences between different emotion categories.

[0152] In some embodiments, the dual-modal deep autoencoder is used to perform emotion recognition using a deep neural network combined with EEG signals and eye movement signals, specifically including:

[0153] Using different model fusion strategies (i.e. Figure 1 The emotion model in the game includes a simple model fusion strategy or a model fusion based on fuzzy integral or a model based on deep neural network), which fuses the EEG signal and the eye movement signal, combines the useful information related to emotions in the two modalities, and uses their complementarity in emotion representation ability to improve the accuracy of emotion recognition during children's games. The model fusion strategy is mainly divided into two categories: feature layer fusion and decision layer fusion.

[0154] In some embodiments, the method of fusing the EEG signal and the eye movement signal using a simple model fusion strategy includes:

[0155] Model fusion strategies include maximum rule, sum rule and boda counting method. By using the maximum rule, sum rule and boda counting method, the signal with the largest reflection in EEG signals and eye movement signals is selected and used as the signal of emotion fusion.

[0156] (3.1) Maximum value rule

[0157] Assume that there is a Different classifiers, categories, each classifier has a probability distribution for the data , The maximum value rule is to select the category with the highest probability among all classifiers for each category as the category to which the test data belongs. Its mathematical form is:

[0158] For samples ,choose satisfy

[0159] (2-1)

[0160] (3.2) Sum rule

[0161] For samples ,choose satisfy

[0162] (2-2)

[0163] (3.3) Borda counting method

[0164] For samples ,choose satisfy

[0165] (2-3)

[0166] in, represents the number of classifiers, Indicates the number of categories; Indicates Classifiers for data Belongs to category The probability distribution of represents the test sample, Indicates categories; Indicates The classifier transforms the data Classify into categories The number of votes corresponding to the ranking.

[0167] In some embodiments, the method of fusing the EEG signal and the eye movement signal by using a fuzzy integral-based model fusion (learning of fuzzy metrics) includes:

[0168] The formula is used to calculate the fuzzy metric, which is used to represent the importance of each feature of the EEG signal and the eye movement signal. The fuzzy metric is used to perform weighted summation on the classifiers corresponding to different features, and the overall confidence is calculated to obtain the fused model at the decision level. Specifically, it includes:

[0169] make , that is, two types of features (two classifiers). Now assume that there are There are two types of training data, labeled .

[0170] The rules for fuzzy metric learning are as follows:

[0171] Minimize the squared error between the classifier's expected output and its actual output:

[0172] (2-4)

[0173] The minimization problem is transformed into a problem with variables and The quadratic optimization problem with constraints can be written as follows:

[0174] (2-5)

[0175] in, is a dimensional vector, containing the fuzzy metric coefficients, namely u=[ u 1 T u 2 T ] T and

[0176] ,

[0177] μ j d 1 , d 2 … μ j d n-1 , d n … μ j d 2 , d 3, … d n ] T (2-6)

[0178] In the above formula, represents the number of features or classifiers, here Indicates that there are two classifiers (EEG signals and eye movement signals); represents the total amount of training data, , Respectively represent the amount of training data for the two types of features, Represents the sum of the training data for the two types of features; Represents the labeled training data; represents the fuzzy metric function; , Represents the membership function of two different categories; , Indicates two categories; , Indicates training samples for two types of features; represents the minimized square error function; Indicates a dimensional vector, containing the fuzzy metric coefficients; represents the diagonal matrix associated with the fuzzy metric coefficients; represents the vector associated with the fuzzy metric coefficient; Indicates A vector of fuzzy metric coefficients, Represents a single feature.

[0179] After obtaining the fuzzy metric, it is used to indicate the importance of each feature. The fuzzy metric is used to perform weighted summation on the classifiers corresponding to different features, and then the overall confidence is calculated to finally obtain a fused model at the decision level.

[0180] In some embodiments, the method of using a deep neural network-based model (restricted Boltzmann machine of a dual-modal deep autoencoder to fuse the EEG signal and the eye movement signal comprises:

[0181] The deep neural network is used to combine EEG signals and eye tracking signals for emotion recognition, further improving the accuracy of emotion recognition. It includes two parts: encoding and decoding. In the encoding part, two RBM models are trained using EEG signal features and eye movement signal features respectively. In the decoding part, two layers of RBM are expanded to reconstruct the input features to form a deep autoencoder. Specifically, it includes:

[0182] (2-7)

[0183] The parameters , In the visible layer node and hidden layer nodes The symmetric weights between and They are visible layer nodes and hidden layer nodes The bias.

[0184] The joint probability distribution of visible layer nodes and hidden layer nodes is:

[0185] (2-8)

[0186] in, is the normalization factor, which is specifically defined as:

[0187] (2-9)

[0188] In the above formula, Represents the visible layer node in RBM and hidden layer nodes The energy function of is the set of parameters of the model; Represents the visible layer node and hidden layer nodes The symmetric weights between Indicates the visible layer The state of the node, represents the hidden layer The state of the node, Represents the visible layer node Bias of Represents the hidden layer nodes Bias of Indicated in the parameter Lower visible layer node and hidden layer nodes The joint probability distribution of represents the normalization factor, which is used to ensure that the sum of the probability distribution is 1; Represents the base of natural logarithms, used to calculate exponential functions; Indicates the number of nodes in the visible layer, Indicates the number of nodes in the hidden layer.

[0189] Through the above method, the user's interaction intention can be more accurately identified by combining EEG signals and eye movement signals. The recognition accuracy can be improved through feature extraction and machine learning pattern recognition network, which is of great significance for designing adaptive human-computer interaction systems and can better understand people's real thoughts and psychological processes.

[0190] refer to Figure 2 - Figure 7 , Figure 2 The overall structure diagram of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided by at least one embodiment of the present application. The device includes: an electroencephalogram cap 2, a glasses body 3, a first memory 4, and a processor, wherein:

[0191] The EEG cap 2, the glasses body 3 and the processor are respectively connected to the first memory 4;

[0192] The EEG cap 2 and the glasses body 3 are used to synchronously collect EEG signals and eye movement signals of children during the game, and transmit the EEG signals and eye movement signals to the first memory 4 through the brain-muscle wire 9 and the USB data connector 10;

[0193] The processor is used to obtain the EEG signals and eye movement signals stored in the first memory 4 through the Type-C second data interface 19, convert the electrical signals into digital signals, and perform data preprocessing on the EEG signals and eye movement signals, extract the features of the EEG signals and eye movement signals according to the short-time Fourier transform algorithm and the characteristics of the iris tremor effect, smooth the features of the EEG signals and eye movement signals by performing linear dynamic system smoothing processing, perform eye movement signal feature significance detection on the eye movement signals using one-way variance analysis, adopt a dual-modal deep autoencoder, and use a deep neural network in combination with the EEG signals and eye movement signals to perform emotion recognition.

[0194] Figure 4 A schematic diagram of the internal structure of an EEG and eye movement detection device based on AR / VR scene simulation provided for at least one embodiment of the present application. In some embodiments, the EEG cap 2 is provided with a brain-computer connector 5, a connecting cable 6, an EEG interface 7, a glasses conduction interface 8, and a brain-muscle wire 9;

[0195] The EEG signals measured by each brain-computer connector 5 are collected and connected to the brain-muscle wire 9 through the connecting cable 6, and the brain-muscle wire 9 is connected to the first memory 4 through the USB data connector 10;

[0196] One end of the glasses conduction interface 8 is connected to the brain-muscle wire 9, and the other end of the glasses conduction interface 8 is connected to the glasses body 3;

[0197] The EEG interface 7 is a position corresponding to the brain-computer connector 5 on the inner side of the EEG cap 2 , and is used to enable the brain-computer connector 5 to pass through the hair and form a good contact with the scalp.

[0198] In some embodiments, the EEG cap 2 is provided with 35 sites for brain-computer connectors 5, and the 35 brain-computer connectors 5 are evenly distributed on the scalp. The 35 brain-computer connectors 5 are all non-invasive, contact electrodes, and from the inside to the outside are respectively: conductive layer: made of highly conductive material silver sodium chloride, responsible for contacting the scalp and transmitting bioelectric signals; supporting matrix: the supporting structure of the conductive layer, made of soft, flexible and elastic polymer. The supporting matrix enables the electrodes to adapt to the curves and movements of the scalp, while providing appropriate structural support. The electrodes at different sites are interconnected by the supporting matrix to form a complete EEG cap structure suitable for different head shapes; transparent protective layer: the transparent protective layer coated on the supporting matrix, which is a transparent polymer or similar material, used to protect the conductive layer from contamination, oxidation or other damage.

[0199] Specifically, all 35 brain-computer connectors 5 are connected with connecting cables 6: the outside of the connecting cables 6 is made of soft and durable material, and the inside is a wire for transmitting the collected bioelectric signals.

[0200] In some embodiments, a first bandage 11 and a second bandage 12 are respectively provided on both sides of the EEG cap 2, and a first Velcro 13 is provided at the end of the first bandage 11. The EEG cap 2 is fixed to the child's head by the first bandage 11, the second bandage 12 and the first Velcro 13.

[0201] In some embodiments, the 35 said brain-computer connection 5 includes 7 midline electrodes: after the EEG cap 2 is fixed on the head, it is located in the center of the top of the head;

[0202] 14 frontal electrodes: After the EEG cap 2 is fixed on the head, they are located at the center of the forehead, the left forehead, and the right forehead;

[0203] 7 temporal electrodes: After the EEG cap 2 is fixed on the head, they are located at the left and right temporal regions;

[0204] 7 occipital electrodes: After the EEG cap 2 is fixed on the head, they are located on the left occipital region and the right occipital region respectively.

[0205] Figure 5 A schematic diagram of a glasses body of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided for at least one embodiment of the present application. In some embodiments, the structure of the glasses body 3 is a semi-frame type, including a frame and a lens, the frame is located above the lens, and a sensor 15, a camera device 14, a second memory 17, and a conductive connector 16 are arranged in the frame, the conductive connector 16 is connected to the glasses conductive interface 8, and the sensor 15 and the camera device 14 are both connected to the second memory 17.

[0206] Specifically, when the user wears the glasses, the sensor 15 is used to track the user's pupil position and mark the gaze point; the camera 14 at the center of the glasses is used to capture the view seen by the user; the second memory 17 receives the eye movement signal transmitted by the camera 14 and the sensor 15. Working principle: The sensor 15 infers the user's gaze position, calculates the precise contour of the visual reflection signal of the camera 14 in different situations, and learns the deep learning model formulated for the virtual reality scene in the configuration device, so as to achieve acceptable tracking accuracy and realize the eye tracking function.

[0207] Specifically, when the glasses and the EEG cap are worn at the same time, one end of the glasses conduction interface 8 is connected to the brain-muscle wire 9, and the other end is connected to the conduction connector 16 provided on the glasses body 3; when the glasses are worn alone, the glasses conduction interface 16 is connected to the second memory 17.

[0208] Specifically, the lenses are transparent under normal conditions and do not affect normal vision. The AR / VR display screen is attached to the lenses using micro-display technology. There are two types of programs that control the display screen of AR / VR glasses:

[0209] The first type of detection program is manually controlled and displays specific screens in sequence after startup;

[0210] The second category is adjustment applications, which capture the usage environment in real time and present the user with the least distracting and most conducive scene images through dynamic feedback adjustment. When the algorithm determines that the user is in a focused state and is disturbed by external factors, it automatically activates the AR / VR anti-interference mode to block interference objects, so that the user always sees the interference-free initial focus environment.

[0211] Figure 6 and Figure 7 A memory schematic diagram of an EEG and eye movement detection device based on AR / VR scene simulation provided for at least one embodiment of the present application. In some embodiments, the first memory 4 includes a USB first data interface 18, a Type-C second data interface 19, a buckle 20, a second Velcro 21, a strap 24, and a reel 25, wherein the USB first data interface 18 is connected to the USB data connector 10; the EEG and eye movement signals transmitted by the brain-muscle wire 9 are stored in the first memory 4, and the processor is connected through the Type-C second data interface 19, the electrical signal in the first memory 4 is converted into a digital signal, and the signal is pre-processed; the strap 24 includes at least one pair, one end of the strap 24 is arranged in the shell of the first memory 4, and is wound on the reel 25, and the other ends of at least one pair of straps 24 extend out of the shell of the first memory 4 from both sides, and the buckle 20 and the second Velcro 21 are respectively arranged at the ends of the straps 24 extending out of the first memory 4 on both sides.

[0212] In some embodiments, it further includes: a cover plate 22, a cover opening 23, and a battery compartment 26, wherein:

[0213] The cover plate 22 is disposed on the housing of the first storage device 4, and one side of the cover plate 22 is rotatably connected to the housing of the first storage device 4. The cover plate 22 covers the reel 25 and the battery compartment 26 disposed in the housing of the first storage device 4;

[0214] The cover opening 23 is a groove or a hole provided on the cover plate 22 .

[0215] Specifically, the first memory 4 is fixed to the child's back through a retractable strap 24 with a buckle 20 and a second Velcro 21. The strap 24 can be freely retracted and retracted through an internal reel 25, that is, the reel 25 is connected to a spring and can automatically rotate under the action of elastic force, thereby retracting the strap 24 wound on the reel 25 into the shell of the first memory 4. The cover 22 can be opened through the cover 23 to place the battery in the battery compartment 26 to prevent damage to the instrument caused by long-term power outages.

[0216] In some embodiments, it also includes: a shell 1, and the EEG cap 2 is arranged in the shell 1. Figure 3 A schematic diagram of the shell structure of an electroencephalogram and eye movement detection device based on AR / VR scene simulation provided in at least one embodiment of the present application.

[0217] The specific implementation method is referred to the aforementioned method embodiment and will not be described in detail here.

[0218] The present application may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present application.

[0219] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, a static random access memory, a portable compact disk read-only memory, a digital versatile disk, a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0220] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0221] The computer program instructions for performing the operation of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network or a wide area network, or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array, or a programmable logic array, may be personalized by using the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present application.

[0222] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0223] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0224] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0225] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0226] Note that, unless otherwise directly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) may be replaced by alternative features for achieving the same, equivalent or similar purposes. Therefore, unless otherwise explicitly stated, each feature disclosed is only an example of a group of equivalent or similar features. Where used, further, preferably, further and more preferably are simple beginnings for elaborating another embodiment based on the aforementioned embodiment, and the content of further, preferably, further or more preferably followed by the aforementioned embodiment is combined with the complete composition of another embodiment. Several further, preferably, further or more preferably settings following the same embodiment can be arbitrarily combined to form another embodiment.

[0227] Although the present application has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements may be made to the present application. Therefore, these modifications or improvements made without departing from the spirit of the present application all fall within the scope of protection claimed in the present application.

Claims

1. A method for detecting EEG and eye movement based on AR / VR scene simulation, characterized in that: Includes steps: Obtain EEG signals and eye movement signals; Performing data preprocessing on the EEG signal and the eye movement signal; Extract EEG signal and eye movement signal features based on the short-time Fourier transform algorithm and the characteristics of iris tremor effect; By performing linear dynamic system smoothing, the characteristics of EEG signals and eye movement signals are smoothed; One-way ANOVA was used to detect the significance of eye movement signal features. A bimodal deep autoencoder is used, and a deep neural network is used to combine EEG signals and eye movement signals for emotion recognition.

2. The method for detecting electroencephalogram and eye movement based on AR / VR scene simulation according to claim 1, characterized in that: The data preprocessing includes: The original EEG signal is preprocessed by adopting a 1-75 Hz bandpass filtering method and down-sampling the EEG signal at a sampling rate of 200 Hz, and the collected eye movement signal is preprocessed by principal component analysis.

3. The method for detecting electroencephalogram and eye movement based on AR / VR scene simulation according to claim 2 is characterized in that: The principal component analysis performs data preprocessing on the collected eye movement signals, specifically including: make For one The matrix contains the pupil diameter information, represents the number of samples, represents the number of subjects in the sample, represents the number of variables in each sample, There are three signals: , in, It is the strongest signal and is the reaction of pupil diameter to light; It is the parasympathetic nervous system’s response signal for emotion and attention; It’s other noise; Using principal component analysis Decomposition, the first principal component obtained can be considered as an approximate estimate of light reflection, remove the first principal component obtained, and get the remaining .

4. The method for detecting electroencephalogram and eye movement based on AR / VR scene simulation according to claim 3 is characterized in that: The principal component analysis Decomposition, remove the first principal component, and get the remaining , specifically including: , , , , , in, represents the matrix containing pupil diameter information, express The transpose of Represents the original signal Transpose The product of The column vector in the decomposition of principal component analysis is The matrix of eigenvectors of The diagonal line is The diagonal matrix of the eigenvalues ​​of Represents a column vector as The matrix of eigenvectors of express The transpose of Indicated by and The matrix composed of express The transpose of represents the illumination reflectance component matrix reconstructed using the first principal component, express The first principal component matrix of express The transpose of the matrix of the eigenvector corresponding to the first principal component of ; Represents the original signal Subtract the first principal component After the remaining part.

5. The method for detecting electroencephalogram and eye movement based on AR / VR scene simulation according to claim 1, characterized in that: The extracting of features of EEG signals and eye movement signals according to the short-time Fourier transform algorithm and the features of the iris tremor effect specifically includes: The short-time Fourier transform algorithm is used to convert the EEG time series signal into frequency domain features, and then the energy is calculated on the obtained frequency domain features.

6. The method for detecting electroencephalogram and eye movement based on AR / VR scene simulation according to claim 1, characterized in that: The linear dynamic system smoothing process is used to smooth the characteristics of the EEG signal and the eye movement signal, specifically including: The linear dynamic system method is used to process EEG signals and eye movement signals, and the probability model is used to estimate the weights to retain emotion-related information.

7. The method for detecting electroencephalogram and eye movement based on AR / VR scene simulation according to claim 1, characterized in that: The use of one-way analysis of variance to perform eye movement signal feature significance detection on the eye movement signal specifically includes: A one-way ANOVA was performed on the collected eye movement signals and all eye movement signal features to obtain the significance of pupil diameter, fixation duration, fixation deviation, saccade duration, saccade amplitude, and blink duration, to verify whether each eye movement signal indicator has significant differences between different emotion categories.

8. The method for detecting electroencephalogram and eye movement based on AR / VR scene simulation according to claim 1, characterized in that: The dual-modal deep autoencoder is used to perform emotion recognition using a deep neural network combined with EEG signals and eye movement signals, specifically including: The EEG signal and the eye movement signal are fused by using a model fusion strategy or a fuzzy integral-based model fusion or a deep neural network-based model; The model fusion strategy is used to fuse the EEG signal and the eye movement signal, including: The model fusion strategies include maximum rule, sum rule and boda counting method. The maximum rule, sum rule and boda counting method are used to select the signal with the largest reflection in EEG signal and eye movement signal, and use it as the signal of emotion fusion. The method of using a fuzzy integral-based model fusion to fuse the EEG signal and the eye movement signal includes: Obtaining a fuzzy metric, using the fuzzy metric to represent the importance of each feature of the EEG signal and the eye movement signal, and using the fuzzy metric to perform weighted summation on the classifiers corresponding to different features, obtaining the overall confidence, and obtaining a fused model at the decision level; The method of using a model based on a deep neural network to fuse the EEG signal and the eye movement signal includes: Emotion recognition is performed using a deep neural network combined with EEG signals and eye tracking signals, which includes two parts: encoding and decoding. In the encoding part, two restricted Boltzmann machine models are trained using EEG signal features and eye movement signal features respectively. In the decoding part, two layers of RBM are expanded to reconstruct the input features to form a deep autoencoder.

9. An electroencephalogram and eye movement detection device based on AR / VR scene simulation, characterized in that: include: EEG cap, glasses body, memory, processor, among which, The EEG cap, the glasses body and the processor are respectively connected to the first memory; The EEG cap and the glasses body are used to synchronously collect EEG signals and eye movement signals, and transmit the EEG signals and eye movement signals to the first memory; The processor is used to obtain the EEG signals and eye movement signals stored in the first memory, convert the electrical signals into digital signals, and perform data preprocessing on the EEG signals and eye movement signals, extract the features of the EEG signals and eye movement signals according to the short-time Fourier transform algorithm and the characteristics of the iris tremor effect, smooth the features of the EEG signals and eye movement signals by performing linear dynamic system smoothing processing, perform eye movement signal feature significance detection on the eye movement signals using one-way variance analysis, adopt a dual-modal deep autoencoder, and use a deep neural network in combination with the EEG signals and eye movement signals to perform emotion recognition.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a machine, the steps of the method according to any one of claims 1 to 8 are implemented.

Images