A recognition method and system based on electroencephalogram and eye movement

Abstract

The embodiment of the application relates to a recognition method and a recognition system based on electroencephalogram and eye movement, the method comprising: creating a collection paradigm of electroencephalogram signals and eye movement signals, and collecting the electroencephalogram signals and the eye movement signals according to the collection paradigm; creating a feature fusion network model based on an attention mechanism; inputting the collected electroencephalogram signals and the eye movement signals into the feature fusion network model for feature extraction and classification processing, and obtaining a recognition result of a corresponding psychological state of the electroencephalogram signals and the eye movement signals. The electroencephalogram signals and the eye movement signals are collected through the set signal collection paradigm, the collected signals are input into the created feature fusion network model for feature extraction to obtain feature signals, and the feature signals are further classified to obtain the corresponding recognition result, so that the two kinds of recognition signals are collected for recognition processing, and thus the corresponding emotional state recognition processing can be realized, and the technical effect of improving the emotional recognition rate is achieved.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of intelligent recognition of brain-computer interface, and particularly relates to a recognition method and system based on electroencephalogram and eye movement. BACKGROUND

[0002] Emotion is a basic factor in human daily life, affecting decision-making, perception, interpersonal interaction and human intelligence. Emotion recognition helps better understand language processing and non-verbal communication in human-computer interaction environment. Recent research shows that emotional state can be predicted from biomedical signals.

[0003] Current emotion recognition technology is mainly divided into three categories: (1) facial expression and voice; (2) external physiological signals; (3) brain signals generated by central nervous system. Among these measurements, the audio-visual-based detector that interprets facial expressions and voices realizes non-contact recognition of emotions, but they cannot always return reliable results because people can easily fake their emotions without being noticed. In contrast, physiological signals show relatively high recognition accuracy because users cannot control it. The features extracted from peripheral physiological signals such as electrocardiogram (ECG), skin conductance (SC) and pulse can provide detailed and complex information for recognizing emotional state. Compared with external physiological signals, electroencephalogram (EEG) signals captured from central nervous system can directly reflect brain activity and have an inherent connection with human emotional state. SUMMARY

[0004] In view of this, to solve the technical problem of low emotion recognition rate, the embodiment of the present application provides a recognition method and system based on electroencephalogram and eye movement.

[0005] In the first aspect, the embodiment of the present application provides a recognition method based on electroencephalogram and eye movement, comprising:

[0006] creating a collection paradigm of electroencephalogram signals and eye movement signals, and collecting the electroencephalogram signals and eye movement signals according to the collection paradigm;

[0007] creating a feature fusion network model based on attention mechanism;

[0008] inputting the collected electroencephalogram signals and eye movement signals into the feature fusion network model for feature extraction and classification processing, to obtain a recognition result of the corresponding psychological state of the electroencephalogram signals and eye movement signals.

[0009] In a possible implementation, the creating of the acquisition paradigm of the electroencephalogram signal and the eye movement signal, and the acquisition of the electroencephalogram signal and the eye movement signal according to the acquisition paradigm, comprise:

[0010] creating an acquisition paradigm of electroencephalogram signal and eye movement signal according to a set experiment paradigm, the acquisition paradigm containing target stimulus and interference stimulus;

[0011] acquiring target electroencephalogram signal and target eye movement signal based on the target stimulus;

[0012] acquiring interference electroencephalogram signal and interference eye movement signal based on the interference stimulus.

[0013] In a possible implementation, the creating of the feature fusion network model based on attention mechanism comprises:

[0014] creating a deep time sequence convolution layer network, a multi-spectrum convolution layer network and a feature fusion classification layer network;

[0015] performing attention mechanism processing based on the deep time sequence convolution layer network, the multi-spectrum convolution layer network and the feature fusion classification layer network, and creating a corresponding feature fusion network model.

[0016] In a possible implementation, the inputting of the acquired electroencephalogram signal and eye movement signal into the feature fusion network model for feature extraction and classification processing to obtain the recognition result of the corresponding psychological state of the electroencephalogram signal and the eye movement signal comprises:

[0017] inputting the acquired electroencephalogram signal into the corresponding deep time sequence convolution layer network of the feature fusion network model for time sequence convolution processing to obtain high-dimensional time sequence representation corresponding to the electroencephalogram signal;

[0018] inputting the acquired eye movement signal into the corresponding multi-spectrum convolution layer network of the feature fusion network model for wavelet convolution processing to obtain multi-spectrum features corresponding to the eye movement signal;

[0019] inputting the high-dimensional time sequence representation and the multi-spectrum features into the feature fusion classification layer network for fusion processing, and performing classification processing on the fused features to obtain the recognition result of the corresponding psychological state of the electroencephalogram signal and the eye movement signal.

[0020] In a possible implementation, the inputting of the high-dimensional time sequence representation and the multi-spectrum features into the feature fusion classification layer network for fusion processing, and the performing of classification processing on the fused features to obtain the recognition result of the corresponding psychological state of the electroencephalogram signal and the eye movement signal comprise:

[0021] input the high-dimensional time sequence representation and the multi-spectrum feature into a feature fusion classification layer network for global average pooling processing to obtain a dimension statistical feature;

[0022] perform first full connection processing, non-linear processing and second full connection processing on the dimension statistical feature to obtain a fusion feature vector;

[0023] perform classification processing based on a softmax function on the fusion feature vector to obtain a recognition result of a pre-set psychological state.

[0024] In one possible implementation, the psychological state includes a positive emotional state and a negative emotional state.

[0025] In a first aspect, an embodiment of the present application provides an identification system applying the identification method of electroencephalogram and eye movement according to the first aspect, comprising:

[0026] a synchronous signal acquisition module, a data management module, an online state detection module and a display module;

[0027] The synchronous signal acquisition module is configured to acquire brain electrical signals and eye movement signals in real time and synchronously.

[0028] The data management module is configured to store multi-modal data corresponding to the brain electrical signals and the eye movement signals.

[0029] The online state detection module is configured to perform online classification detection and analysis processing on the acquired brain electrical signals and eye movement signals, and determine a psychological state and an analysis result corresponding to the brain electrical signals and the eye movement signals.

[0030] The display module is configured to display the psychological state and the analysis result obtained through online detection.

[0031] In one possible implementation, the data management module is further configured to perform quality analysis processing on the multi-modal data corresponding to the brain electrical signals and the eye movement signals, and filter the multi-modal data, so that an original database and a feature database in the data management module are updated.

[0032] In one possible implementation, the online state detection module is further configured to perform preprocessing on online multi-modal data to obtain online multi-modal information, and perform fusion classification processing based on the online multi-modal information to determine an online psychological state and an online analysis result of the online multi-modal data.

[0033] In one possible implementation, the identification system further comprises an offline detection module.

[0034] The offline detection module is configured to perform preprocessing on original multi-modal data to obtain offline multi-modal information.

[0035] Based on the offline multi-modal information, a fusion classification processing is performed to determine an offline psychological state and an offline analysis result of the offline multi-modal data.

[0036] The recognition scheme based on electroencephalogram and eye movement provided by the embodiment of the present application creates a signal collection paradigm of electroencephalogram and eye movement signal, collects the electroencephalogram and eye movement signal according to the signal collection paradigm, creates a feature fusion network model based on attention mechanism, inputs the collected electroencephalogram and eye movement signal into the feature fusion network model for feature extraction and classification processing, and obtains a recognition result of the psychological state corresponding to the electroencephalogram and eye movement signal. The electroencephalogram and eye movement signal are collected through the set signal collection paradigm, the collected signal is input into the created feature fusion network model for feature extraction to obtain a feature signal, and the feature signal is further classified to obtain a corresponding recognition result, so that the two kinds of recognition signals are collected for recognition processing. Through the present scheme, the corresponding emotional state can be recognized and processed, and the technical effect of improving the emotional recognition rate is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0038] Figure 1 A flowchart of a recognition method based on electroencephalogram and eye movement provided by the embodiment of the present application is shown in the figure;

[0039] Figure 2 A flowchart of another recognition method based on electroencephalogram and eye movement provided by the embodiment of the present application is shown in the figure;

[0040] Figure 3 A flowchart of an example scene provided by the embodiment of the present application is shown in the figure;

[0041] Figure 4 A flowchart of another example scene provided by the embodiment of the present application is shown in the figure;

[0042] Figure 5 A structural diagram of a recognition system provided by the embodiment of the present application is shown in the figure;

[0043] Figure 6 A structural diagram of a recognition system in an example scene provided by the embodiment of the present application is shown in the figure;

[0044] Figure 7 An effect diagram of a recognition system in another example scene provided by the embodiment of the present application is shown in the figure;

[0045] Figure 8A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] The terms 'comprise' and 'have' in the embodiments of the present application are used to represent an open-ended inclusion, and refer to the presence of additional elements / components / etc. in addition to the listed elements / components / etc.; the terms 'first' and'second' are used only as labels, and are not intended to limit the number of objects. In addition, different elements and regions in the drawings are only schematically shown, and thus the present application is not limited to the sizes or distances shown in the drawings.

[0048] To facilitate the understanding of the embodiments of the present application, further explanation and description will be made below with reference to the drawings in specific embodiments, and the embodiments do not constitute a limitation on the embodiments of the present application.

[0049] The attention mechanism originates from the research on human vision. In cognitive science, due to the bottleneck of information processing, humans selectively focus on part of all information while ignoring other visible information. The above mechanism is usually referred to as the attention mechanism.

[0050] Figure 1 A flowchart of an identification method based on electroencephalogram and eye movement provided by an embodiment of the present application. The execution subject of the present application is an emotion identification system. According to the Figure 1 The identification method based on electroencephalogram and eye movement specifically includes:

[0051] S101, creating an acquisition paradigm of electroencephalogram signals and eye movement signals, and acquiring the electroencephalogram signals and the eye movement signals according to the acquisition paradigm.

[0052] The present application is applied to emotion identification. By acquiring the electroencephalogram signals and the eye movement signals of a tester, inputting the acquired signals into a created feature fusion network model to extract feature signals, and further classifying the feature signals to obtain corresponding identification results, the identification processing through the acquisition of two kinds of identification signals is achieved.

[0053] The brain electrical signal can be understood as the brain wave signal collected from the brain of the detection object. The eye movement signal can be understood as the information of the retinal focus and pupil dilation of the eye part of the detection object. The paradigm can be understood as the pre-set signal collection mode.

[0054] Further, before collecting the brain electrical signal and the eye movement signal of the detection object, the signal collection mode is designed, and the brain electrical signal and the eye movement signal of the detection object are collected according to the designed collection mode, so as to provide processing data for recognizing the emotion category of the detection object.

[0055] S102, a feature fusion network model based on attention mechanism is created.

[0056] The attention mechanism can be understood as a processing process of selectively focusing on part of all information. The feature fusion network model can be understood as a recognition model used for classification processing after aggregating features.

[0057] Further, according to the attention mechanism principle, a recognition model is constructed, and a network model aggregating feature characteristics is created to provide a processing model for emotion recognition.

[0058] S103, the collected brain electrical signal and eye movement signal are input into the feature fusion network model for feature extraction and classification processing, and the recognition result of the corresponding psychological state of the brain electrical signal and eye movement signal is obtained.

[0059] The feature extraction can be understood as the process of extracting the feature vector in the signal. The classification processing can be understood as the process of identifying the collected signal according to the emotion category. The psychological state can be understood as the emotion category, including negative emotion and positive emotion. The recognition result can be understood as the specific category determined according to the emotion classification.

[0060] Further, after the brain electrical signal and the eye movement signal are preprocessed and input into the trained feature fusion network model for feature extraction, the extracted feature data is classified and recognized according to the emotion, and the emotion category represented by the collected brain electrical signal and eye movement signal is obtained as the recognition result of the psychological state, so as to realize the recognition processing of the corresponding emotion state and improve the technical effect of the emotion recognition rate.

[0061] The recognition method based on electroencephalogram and eye movement provided by the embodiment of the present application creates a signal collection paradigm of electroencephalogram and eye movement signal, and collects the electroencephalogram and eye movement signal according to the collection paradigm; creates a feature fusion network model based on an attention mechanism; inputs the collected electroencephalogram and eye movement signal into the feature fusion network model for feature extraction and classification processing, and obtains the recognition result of the corresponding psychological state of the electroencephalogram and eye movement signal. The electroencephalogram and eye movement signal are collected through the set signal collection paradigm, the collected signal is input into the created feature fusion network model for feature extraction to obtain a feature signal, and the feature signal is further classified to obtain the corresponding recognition result, so that the two kinds of recognition signals are collected for recognition processing. According to the present application, the corresponding emotional state can be recognized and processed, and the technical effect of improving the emotional recognition rate is achieved.

[0062] Figure 2 The flowchart of another recognition method based on electroencephalogram and eye movement provided by the embodiment of the present application is provided. The execution subject of the present application is an emotional recognition system. Figure 2 The above embodiment is introduced on the basis of the above embodiment. Referring to Figure 2 The recognition method based on electroencephalogram and eye movement provided by the embodiment of the present application specifically further includes:

[0063] S201, a signal collection paradigm of electroencephalogram and eye movement signal is created according to a set experiment paradigm, and the collection paradigm contains target stimulation and interference stimulation.

[0064] The present application is applied to emotional recognition. The electroencephalogram and eye movement signal of a tester are collected, the collected signal is input into a created feature fusion network model for feature extraction to obtain a feature signal, and the feature signal is further classified to obtain the corresponding recognition result, so that the two kinds of recognition signals are collected for recognition processing.

[0065] The electroencephalogram signal mentioned here can be understood as the brain wave signal collected by the brain of the detector. The detector mentioned here can be but is not limited to a human or an animal. The eye movement signal mentioned here can be understood as the information such as the retinal focusing condition and the pupil dilation condition of the eye part of the detection object. The paradigm mentioned here can be understood as a pre-set signal collection mode. For example, a static picture is set, then a specified video content is watched, and finally the video is closed, and the paradigm cycle process of eye rest. The target stimulation mentioned here can be understood as video stimulation with conflicting visual perception. For example, the acceleration process of a racing car in a rugged terrain, or a video clip of natural phenomena such as a tsunami and a mudslide. The interference stimulation mentioned here can be understood as video stimulation with a peaceful visual perception. For example, a picture of a clear sky with a blue sky, or a picture of a mountain road with flowing water.

[0066] Further, before collecting the brain electrical signals and eye movement signals of the detection object, a signal collection paradigm is designed, and a complete experimental paradigm is created according to different categories of target stimuli and interference stimuli, so as to prepare for the next step of collecting the brain electrical signals and eye movement signals.

[0067] S202, collecting target brain electrical signals and target eye movement signals based on target stimuli.

[0068] S203, collecting interference brain electrical signals and interference eye movement signals based on interference stimuli.

[0069] The target brain electrical signals mentioned here can be understood as brain electrical signals generated by the brain after watching target stimulus videos. The target eye movement signals mentioned here can be understood as eye movement signals generated by the pupil and focusing of the eye after watching target stimuli. The interference brain electrical signals mentioned here can be understood as brain electrical signals generated by the brain after watching interference stimulus videos. The interference eye movement signals mentioned here can be understood as eye movement signals generated by the pupil and focusing of the eye after watching interference stimuli.

[0070] Further, when the detection object watches different visual stimuli, the corresponding brain electrical signals of the target stimuli are collected as target brain electrical signals, and the corresponding eye movement signals are collected as target eye movement signals. Similarly, for the interference stimuli in the visual stimuli, the corresponding brain electrical signals are collected as interference brain electrical signals, and the corresponding eye movement signals are collected as interference eye movement signals, so as to obtain the brain electrical signals and eye movement signals of the detection object collected according to the designed collection paradigm, and provide data basis for identifying the emotion category of the detection object.

[0071] In a possible example scenario, based on the go-nogo experimental paradigm, an E-Prime software is used to design the experimental paradigm, and 2 types of stimuli (i.e. target stimuli and interference stimuli) are designed. The detection object wears an electrode cap and looks at the video screen playing the stimuli in the front direction. At the beginning, a cross is displayed in the center of the screen for 1s, then the visual stimuli are randomly presented for 2s, and finally there is a 1s rest. After each round of testing, the detection object needs to perform self-evaluation in time and record the real emotional experience.

[0072] S204, creating a deep time sequence convolution layer network, a multi-spectrum convolution layer network, and a feature fusion classification layer network.

[0073] S205, performing attention mechanism processing based on the deep time sequence convolution layer network, the multi-spectrum convolution layer network, and the feature fusion classification layer network, and creating a corresponding feature fusion network model.

[0074] The deep time sequence convolutional layer network mentioned here can be understood as a network model for processing time sequence data features to obtain time correlation features. The multi-spectrum convolutional layer network mentioned here can be understood as a model for performing spectrum analysis on spectrum features. The attention mechanism mentioned here can be understood as a process of selectively focusing on part of all information. The feature fusion classification layer network mentioned here can be understood as an identification model for performing classification processing after fusion processing.

[0075] Further, a deep time sequence convolutional layer network, a multi-spectrum convolutional layer network, and a feature fusion classification layer network are respectively created, and the attention mechanism is processed on the three network models, the correlation degrees between the three network models are established, and a feature fusion network model for performing feature fusion on multiple feature data and achieving the purpose of classifying and identifying features after fusion processing is created, to prepare for the next step of emotion classification and identification.

[0076] S206, input the collected electroencephalogram signal into the corresponding deep time sequence convolutional layer network of the feature fusion network model for time sequence convolution processing to obtain high-dimensional time sequence representation corresponding to the electroencephalogram signal.

[0077] The convolution processing mentioned here can be understood as a multi-layer convolution operation. The high-dimensional time sequence representation mentioned here can be understood as multi-dimensional time sequence feature data.

[0078] Further, the created feature fusion network model is used to perform time sequence convolution operation on the collected electroencephalogram signal, to improve the feature dimension of the electroencephalogram signal, and to obtain high-dimensional time sequence representation corresponding to the electroencephalogram signal after multi-dimensional transformation, to prepare for identifying emotion categories.

[0079] In one possible example scenario, the deep time sequence convolutional layer network in the feature fusion network model extracts high-dimensional time features of the electroencephalogram signal, and designs multiple subunits as time sequence convolution units. Each time sequence convolution unit starts from a max-pooling layer, and the kernel size is set to (1x3) to generate corresponding electroencephalogram representation from the collected electroencephalogram signal. A time convolution with a convolution kernel size of (1x11) is used to generate deeper information, thereby achieving the purpose of improving the dimension. At the same time, a nonlinear activation function is set to increase the nonlinearity of the electroencephalogram signal, and to obtain high-dimensional time sequence representation corresponding to the electroencephalogram signal.

[0080] S207, input the collected eye movement signal into the corresponding multi-spectrum convolutional layer network of the feature fusion network model for wavelet convolution processing to obtain multi-spectrum features corresponding to the eye movement signal.

[0081] The wavelet convolution mentioned here can be understood as a spectrum processing operation method. The multi-spectrum features mentioned here can be understood as features data that highlight the main frequency components and are easier to identify the components of the signal.

[0082] Further, the wavelet convolution operation is performed on the collected eye movement signals by the created feature fusion network model to extract the frequency spectrum features, the feature dimension of the eye movement signals is improved, and the multi-frequency spectrum features corresponding to the eye movement signals are obtained to prepare for recognizing the emotion categories.

[0083] In a possible example scenario, a convolution operator, i.e., a wavelet convolution layer, is used to integrate the function of frequency domain feature extraction into the feature fusion network model. The Db4 (Daubechies order-4) wavelet is used to decompose the eye movement signals into wavelet coefficients of multiple frequency bands. After a series of wavelet convolutions, in order to perform multi-frequency spectrum analysis, these frequency spectrum representations are further spliced into compact feature maps. Let x be the eye movement signal of T sampling points, and let the wavelet transform at the sampling point t be defined as follows:

[0084]

[0085]

[0086] where u and v represent a pair of wavelet filters, respectively named as approximation filter and detail filter. x A and x D are the approximation coefficients and the detail coefficients, respectively. K and s are the kernel size and the step size of the wavelet convolution. The number of wavelet layers V is determined by the sampling rate of the eye movement signal:

[0087]

[0088] where f s is the original sampling rate of the signal. The step size of the wavelet convolution kernel is set to 2 each time, and the kernel size is set to 8, which is consistent with the order of the Db4 wavelet filter. Since the wavelet is implemented in the form of convolution, the number of output channels is twice the number of input channels, which ensures that the number of two channels obtained after separation is the same. Assuming that the number of input channels is N, the 2N output channels are separated into approximation coefficients and detail coefficients by the defined selection method:

[0089] x A = {x w (c) | c = 1, 3, …, 2R-1} Equation 4

[0090] x D = {x w (c) | c = 2, 4, …, 2R} Equation 5

[0091] where x W is the output of each wavelet convolution layer, and c is the selected channel index. For each input channel, using a pair of wavelet filters (u, v) will produce two output channels, i.e., x A and xD is alternating. One padding method is chosen to output x A The results are periodically smoothed to alleviate the distortion problem at the head and tail of the signal after wavelet convolution:

[0092]

[0093]

[0094] where K is the length of the signal and h is the kernel size of the wavelet convolution. In order to integrate all the frequency domain features into a compact model architecture, the obtained spectral features are concatenated to obtain a set of multi-spectral features.

[0095] S208, input the high-dimensional time sequence representation and the multi-spectral feature into the feature fusion classification layer network for fusion processing, and classify the fused features to obtain the recognition result of the psychological state corresponding to the electroencephalogram signal and the eye movement signal.

[0096] wherein the psychological state includes a positive emotional state and a negative emotional state.

[0097] Further, by inputting the obtained high-dimensional time sequence representation representing the electroencephalogram signal and the multi-spectral feature representing the eye movement signal into the feature fusion classification layer network for feature fusion, and classifying the obtained fusion feature, the recognition result is obtained as the psychological state of the electroencephalogram signal and the eye movement signal, and then the emotional category is obtained, completing the emotion recognition based on the electroencephalogram signal and the eye movement signal.

[0098] Further, the classification and recognition process in step S208 includes the following steps:

[0099] Step one: input the high-dimensional time sequence representation and the multi-spectral feature into the feature fusion classification layer network for global average pooling processing to obtain a dimension statistical feature.

[0100] Step two: perform first full connection processing, non-linear processing, and second full connection processing on the dimension statistical feature to obtain a fusion feature vector.

[0101] Step three: perform classification processing based on the softmax function on the fusion feature vector to obtain the recognition result of the pre-set psychological state.

[0102] In a possible example scenario, for the input feature data corresponding to the electroencephalogram signal and the eye movement signal, the attention module performs a "compression" (i.e., fusion processing) operation to recalibrate the features, and aggregates the feature data along the time dimension. The "excitation" operation takes the output of the previous "compression" block as input, explores the channel correlation, calculates the weight of all channels, and weights the feature data, to realize the channel attention mechanism. In order to utilize the channel correlation, the compression operation performs global average pooling on the input feature data to compress the global information, for generating channel dimension statistical features. Formally, the statistical information calculated by the global average pooling is defined as shown in equation 8:

[0103]

[0104] where x s is the input feature data, T is the time series length, M, c, t represent the three dimensions of the feature data, the number, the width and the length. After extracting the channel information from the compression operation, the subsequent "excitation" operation fully utilizes the channel correlation. This operation is realized by two consecutive fully connected layers, a nonlinear layer and a softmax function, and finally the output of the attention module is defined as shown in equation 9:

[0105] z se = σ (W2ε (W1z sq )) x s Equation 9

[0106] where W1 and W2 are the first and second fully connected layers respectively, ε is a nonlinear function, and σ is a softmax function. All features are mapped into a one-dimensional feature vector and input into the fully connected layer. The softmax function is used to obtain the probability of different classifications, and the classification with the maximum probability is used as the final recognition result of the electroencephalogram signal and the eye movement signal emotion recognition, to achieve the purpose of detecting the emotional changes based on the electroencephalogram signal and the eye movement signal.

[0107] In a possible example scenario, Figure 3 A flowchart of an example scenario provided by an embodiment of the present application is shown. Referring to Figure 3The provided diagram first acquires the EEG and eye movement signals of the detection object by detecting the brain electricity of the object wearing the electroencephalogram and eye movement signal acquisition device watching the visual stimulation video or image, and then acquires the EEG signal and eye movement signal. The next step is to preprocess the acquired signals respectively to obtain the multi-layer dynamic graph convolution data corresponding to the EEG signal and the time sequence and spectrum convolution data corresponding to the eye movement signal. Then the obtained convolution data is input into the feature fusion network model based on the attention mechanism for feature fusion and classification to obtain the recognition result of the psychological state. The recognition result mentioned here can be classified into two categories of positive emotion and negative emotion, wherein the positive emotion is further refined into calm emotion, relaxed emotion or happy emotion. The negative emotion can be further refined into fatigue emotion, tension emotion or fear emotion. After obtaining the recognition result, the model performance evaluation is optionally added, and the feature extraction method is modified in the unqualified case, and the current feature fusion network model is detected in real time in the qualified state, the signal data is obtained by using the multi-modal data management module, the classification and recognition data are adjusted through online detection, and the detection result is displayed through the display module to achieve the purpose of real-time detection.

[0108] Optionally, in a possible example scenario, Figure 4 The flowchart of another example scenario provided by the embodiment of the present application is given. Referring to Figure 4 The provided diagram acquires the EEG signal and eye movement signal, performs multi-layer convolution processing, pooling processing and nonlinear function processing on the EEG signal by using a deep time sequence convolution layer to obtain high-dimensional time sequence features corresponding to the EEG signal. The eye movement signal is subjected to multi-layer wavelet convolution processing in a multi-spectrum convolution layer to obtain multi-spectrum features. Then the obtained high-dimensional time sequence features and multi-spectrum features are input into a feature fusion network model based on the attention mechanism for fusion processing of the high-dimensional time sequence features and multi-spectrum features. The fused features are classified, including global pooling processing, twice full connection processing, nonlinear processing and classification processing, compared according to the pre-set emotion categories, and finally output the recognition result to realize the recognition processing of the corresponding emotional state and improve the emotional recognition rate.

[0109] Another EEG and eye movement based recognition method provided by the embodiment of the present application, by designing to realize the paradigm to acquire the EEG signal and eye movement signal, using the deep time sequence convolution layer network to extract the features of the EEG signal, obtaining the high-dimensional time sequence features, and then using the multi-spectrum convolution layer network to extract the features of the eye movement signal, obtaining the multi-spectrum features, inputting the obtained high-dimensional time sequence features and multi-spectrum features into the feature fusion classification layer network based on the attention mechanism for feature fusion, and classifying the fused feature data to obtain the recognition result of the emotion category, realizing the recognition processing of the corresponding emotional state and improving the technical effect of the emotional recognition rate.

[0110] Figure 5 A structural schematic diagram of an identification system is provided for an embodiment of the present application. The executive body of the present application is an emotion identification system. Referring to Figure 5 The identification system specifically comprises:

[0111] The synchronous signal acquisition module 51, the data management module 52, the state online detection module 53 and the display module 54.

[0112] Further, the synchronous signal acquisition module is used for real-time synchronous acquisition of the electroencephalogram signal and the eye movement signal; the data management module is used for storage of the multi-modal data corresponding to the electroencephalogram signal and the eye movement signal; the state online detection module is used for online classification detection and analysis processing of the acquired electroencephalogram signal and eye movement signal, determination of the psychological state and analysis result corresponding to the electroencephalogram signal and the eye movement signal; and the display module is used for display of the psychological state and analysis result obtained through online detection.

[0113] In a possible example scenario, the data management module is further used for quality analysis processing of the multi-modal data corresponding to the electroencephalogram signal and the eye movement signal, and screening of the multi-modal data, so as to enable feedback updating of the original database and the feature database in the data management module.

[0114] In a possible example scenario, the state online detection module is further used for preprocessing of the online multi-modal data to obtain online multi-modal information; and fusion classification processing based on the online multi-modal information, determination of the online psychological state and the online analysis result of the online multi-modal data.

[0115] In a possible example scenario, the identification system further comprises an offline detection module; the offline detection module is used for preprocessing of the original multi-modal data to obtain offline multi-modal information; and fusion classification processing based on the offline multi-modal information, determination of the offline psychological state and the offline analysis result of the offline multi-modal data.

[0116] Optionally, in a possible example scenario, Figure 6The structural schematic diagram of the recognition system in an example scenario provided for the embodiment of the present application is shown in the figure. First, the test object wears the electroencephalogram signal and eye movement signal synchronous acquisition module, which integrates the high-sampling-rate Neuroscan electroencephalogram acquisition device and the portable Tobii Pro Glasses 3 eye movement tracker, to realize the synchronous acquisition of the electroencephalogram signal and the eye movement physiological information with the advantages of high time resolution and high stability. The data management module corresponding to the electroencephalogram signal and the eye movement signal stores the recorded electroencephalogram signal, eye movement tracking and other physiological information to the electroencephalogram eye movement database on the server side for management, and quantitatively analyzes and determines the data quality according to the multi-modal feature processing result, optimizes the data, makes the database continuously feedback and update, and realizes the dynamic storage and management of the test physiological information big data. Then, the online psychological state detection module integrates the multi-modal data preprocessing and multi-modal data information fusion processing of the electroencephalogram signal and the eye movement signal with the powerful computing capability of the server side, first constructs the multi-modal time-frequency feature extraction and fusion decoding network model based on the historical storage data of the data management module corresponding to the electroencephalogram signal and the eye movement signal offline, analyzes the online detection of the real-time synchronous acquisition of the multi-modal physiological data corresponding to the electroencephalogram signal and the eye movement signal, and corrects and updates the decoding network parameters according to the detection result, and then realizes the rapid and accurate detection of the psychological state. Finally, the dynamic psychological state online detection and analysis result synchronized with the model is presented through the client platform display module set, and the emotional state recognition processing is realized, and the technical effect of improving the emotional recognition rate is realized.

[0117] Optionally, in a possible example scenario, Figure 7 The effect diagram of the recognition system in another example scenario provided for the embodiment of the present application is shown in the figure. Figure 7 The user psychological state real-time detection system corresponding to the detection object can directly observe the current data of the detection object by inputting the user basic information and displaying the sampling device state, and can display the current emotional category negative emotion through the current psychological state display module, and can be refined into a fatigue state, and can accurately obtain the position of the detection object by locking the coordinate latitude of the detection object. At the same time, the historical detection report display interface is set to display the historical detection result of the detection object, as a judgment of the emotional change of the detection object, realize the recognition processing of the corresponding emotional state, and improve the technical effect of improving the emotional recognition rate.

[0118] The recognition system provided by the embodiment of the application realizes real-time online psychological state detection, the emotion recognition system based on electroencephalogram and eye movement signals, in combination with the characteristics of the neural mechanism of visual cognition and the cognitive process, carries out psychological state detection such as fatigue, tension and excitement, combines visual attention mechanism, human experience knowledge and machine intelligence, forms a man-machine fusion system platform, realizes the recognition and processing of emotional state, and improves the technical effect of emotion recognition rate.

[0119] Figure 8 A structural schematic diagram of an electronic device provided by the embodiment of the application is provided, Figure 8 The electronic device 800 shown includes at least one processor 801, a memory 802, at least one network interface 804 and other user interfaces 803. The various components in the electronic device 800 are coupled together through a bus system 805. It can be understood that the bus system 805 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 805 also includes a power supply bus, a control bus and a state signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 805 in the Figure 8

[0120] The user interface 803 can include a display, a keyboard or a clicking device (for example, a mouse, a trackball, a touchpad or a touch screen, etc.).

[0121] ​It is to be understood that the memory 802 in embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of RAM are available, for example, static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). Memory 802 described herein is intended to include, without being limited to, these and any other suitable types of memory.

[0122] In some embodiments, the memory 802 stores the following elements, executable units or data structures, or a subset of them, or an extended set of them: an operating system 8021 and application programs 8022.

[0123] The operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application programs 8022 include various application programs, such as a media player, a browser, etc., for implementing various application services. Programs implementing the methods of embodiments of the present application can be included in the application programs 8022.

[0124] In embodiments of the present application, by invoking programs or instructions stored in the memory 802, specifically, programs or instructions stored in the application programs 8022, the processor 801 is configured to execute the method steps provided by various method embodiments, for example, including:

[0125] The collection paradigm of the electroencephalogram signal and the eye movement signal is created, and the electroencephalogram signal and the eye movement signal are collected according to the collection paradigm; a feature fusion network model based on an attention mechanism is created; the collected electroencephalogram signal and eye movement signal are input into the feature fusion network model for feature extraction and classification processing, and the recognition result of the corresponding psychological state of the electroencephalogram signal and the eye movement signal is obtained.

[0126] The method disclosed in the embodiments of the present application can be applied to the processor 801 or implemented by the processor 801. The processor 801 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the above method can be completed by hardware integrated logic circuits or software form instructions in the processor 801. The processor 801 can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software units in the decoding processor for execution. The software unit can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 802, and the processor 801 reads the information in the memory 802, and combines the hardware to complete the steps of the above method.

[0127] It can be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP Devices, DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.

[0128] For software implementation, the techniques described herein can be implemented with a processing unit that executes program components or modules. The software code can be stored in memory and executed by a processor. Memory can be implemented within the processor or external to the processor.

[0129] The electronic device provided by the embodiments can be an electronic device as shown in Figure 8 , can perform all steps of the recognition method based on electroencephalogram and eye movement as described in Figures 1-2 and Figure 4 , and further realize the technical effects of the recognition method based on electroencephalogram and eye movement as shown in Figures 1-2 and Figure 4 . For details, please refer to the relevant description in Figures 1-2 and Figure 4 . For brevity, no further description is given here.

[0130] The embodiments of the present application also provide a storage medium (computer readable storage medium). The storage medium herein stores one or more programs. Wherein, the storage medium can include volatile memory, such as random access memory; the storage medium can also include non-volatile memory, such as read only memory, flash memory, hard disk or solid state disk; the storage medium can also include a combination of the above kinds of memory.

[0131] When the one or more programs in the storage medium can be executed by one or more processors to implement the above-mentioned recognition method based on electroencephalogram and eye movement executed on the side of the recognition device based on electroencephalogram and eye movement.

[0132] The processor is configured to execute the recognition program based on electroencephalogram and eye movement stored in the memory to implement the following steps of the recognition method based on electroencephalogram and eye movement executed on the side of the recognition device based on electroencephalogram and eye movement:

[0133] The collection paradigm of the electroencephalogram signal and the eye movement signal is created, and the electroencephalogram signal and the eye movement signal are collected according to the collection paradigm; a feature fusion network model based on an attention mechanism is created; the collected electroencephalogram signal and eye movement signal are input into the feature fusion network model for feature extraction and classification processing, to obtain the recognition result of the corresponding psychological state of the electroencephalogram signal and eye movement signal.

[0134] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, each example has been described in general terms above as including functional components and steps. Whether such functions are implemented in hardware or software depends on the particular application and design constraints imposed on the overall architecture. Those skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present application.

[0135] The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can be located in random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0136] The above detailed description of the specific implementation of the present application further describes the purpose, technical solutions, and beneficial effects of the present application. It should be understood that the above description is only a specific implementation of the present application and is not intended to limit the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included within the scope of protection of the present application.

Claims

1. A recognition method based on electroencephalogram and eye movement, characterized in that, The method comprises the following steps: creating a collection paradigm of electroencephalogram signals and eye movement signals, and collecting the electroencephalogram signals and eye movement signals according to the collection paradigm; creating a feature fusion network model based on an attention mechanism; inputting the collected electroencephalogram signals and eye movement signals into the feature fusion network model for feature extraction and classification processing to obtain the recognition result of the psychological state corresponding to the electroencephalogram signals and eye movement signals, comprising: inputting the collected electroencephalogram signals into the corresponding deep time sequence convolution layer network of the feature fusion network model for time sequence convolution processing to obtain the high-dimensional time sequence representation corresponding to the electroencephalogram signals; inputting the collected eye movement signals into the corresponding multi-spectrum convolution layer network of the feature fusion network model for wavelet convolution processing to obtain the multi-spectrum feature corresponding to the eye movement signals; inputting the high-dimensional time sequence representation and the multi-spectrum feature into the feature fusion classification layer network for fusion processing, and classifying the fused features to obtain the recognition result of the psychological state corresponding to the electroencephalogram signals and eye movement signals.

2. The method of claim 1, wherein, The method of creating a collection paradigm of electroencephalogram signals and eye movement signals, and collecting the electroencephalogram signals and eye movement signals according to the collection paradigm, comprises: creating a collection paradigm of electroencephalogram signals and eye movement signals according to a set experiment paradigm, wherein the collection paradigm comprises target stimuli and interference stimuli; collecting target electroencephalogram signals and target eye movement signals based on the target stimuli; collecting interference electroencephalogram signals and interference eye movement signals based on the interference stimuli.

3. The method of claim 2, wherein, The method of creating a feature fusion network model based on an attention mechanism comprises: creating a deep time sequence convolution layer network, a multi-spectrum convolution layer network and a feature fusion classification layer network; performing attention mechanism processing based on the deep time sequence convolution layer network, the multi-spectrum convolution layer network and the feature fusion classification layer network to create a corresponding feature fusion network model.

4. The method of claim 1, wherein, The method of inputting the high-dimensional time sequence representation and the multi-spectrum feature into the feature fusion classification layer network for fusion processing, and classifying the fused features to obtain the recognition result of the psychological state corresponding to the electroencephalogram signals and eye movement signals, comprises: inputting the high-dimensional time sequence representation and the multi-spectrum feature into the feature fusion classification layer network for global average pooling processing to obtain a dimension statistical feature; performing first full connection processing, non-linear processing and second full connection processing on the dimension statistical feature to obtain a fusion feature vector; performing classification processing based on a softmax function on the fusion feature vector to obtain the recognition result of the pre-set psychological state.

5. The method of claim 1, wherein, The psychological state comprises a positive emotional state and a negative emotional state.

6. A recognition system applying the electroencephalogram and eye movement recognition method of claim 1, characterized by, The method comprises the following steps: synchronous signal collection module, data management module, state online detection module and display module; The synchronous signal collection module is used for real-time synchronous collection of electroencephalogram signals and eye movement signals; The data management module is used for storing the multi-modal data corresponding to the electroencephalogram signals and eye movement signals; The state online detection module is used for online classification detection and analysis processing of the collected electroencephalogram signals and eye movement signals to determine the psychological state and analysis result corresponding to the electroencephalogram signals and eye movement signals; The display module is used for displaying the online detected psychological state and analysis result.

7. The identification system of claim 6, wherein, The data management module is also used for quality analysis and processing of the multi-modal data corresponding to the electroencephalogram signal and eye movement signal, and screening of the multi-modal data, so that the original database and the feature database in the data management module are updated in feedback.

8. The identification system of claim 6, wherein, The state online detection module is also used for preprocessing of the online multi-modal data to obtain online multi-modal information, fusion classification processing based on the online multi-modal information, determination of the online psychological state and online analysis result of the online multi-modal data.

9. The identification system of claim 6, wherein, The recognition system further comprises an offline detection module. The offline detection module is used for preprocessing of the original multi-modal data to obtain offline multi-modal information. Fusion classification processing based on the offline multi-modal information is performed to determine the offline psychological state and offline analysis result of the offline multi-modal data.

Images