Fine-grained emotional EEG detection method and system in virtual reality

Abstract

The present invention provides a fine-grained emotional EEG detection method and system in virtual reality. The method uses immersion and can effectively isolate environmental interference to play emotional videos to stimulate user emotions, improve the quality of collected EEG signals, and enhance the usability of the system. A fine-grained emotional EEG decoding network is then used to extract spatial and temporal features from emotional EEG signals. Compared with the traditional method that cannot effectively express the strength of user emotions, the present invention can further obtain the user's emotional intensity on the premise of decoding the user's emotional state. Moreover, after completing the training phase, the present invention can be directly used, whether for offline analysis of user status or online decoding to obtain user emotions. It is suitable for application in a variety of tasks and can broaden the application scenarios of emotional EEG.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotional state recognition, and in particular relates to a fine-grained emotional EEG detection method and system in virtual reality. Background Art

[0002] Emotion detection, a crucial component of brain-computer interaction, is a research hotspot at the intersection of psychology and neuroengineering. Currently, emotion labeling in EEG computations is primarily approached from two perspectives: dimensional models and discrete models. Dimensional models consist of two dimensions: valence and arousal. These quantify emotions into nine numerical values, which are detailed but difficult to understand, lacking the ability to intuitively define emotions. Discrete models generally consist of six basic emotions: happiness, sadness, surprise, fear, anger, and disgust. Discrete models are widely used in practice, but their drawback is their inability to describe continuous emotional states in a fine-grained manner. These two emotion definitions are often used in EEG emotion experiments, but their primary challenge is identifying continuous emotions in real-world scenarios. In real-world scenarios, emotions do not change in type over a short period of time, but rather in intensity.

[0003] Emotion detection has broad application prospects. For example, in production activities, people working in aerospace companies, national security departments, mines, construction sites, or high-risk work environments are often in a state of high tension. Regular emotion detection can promptly identify individuals with abnormal emotions, allowing for early intervention to prevent their mental state from further deteriorating, thereby reducing production risks and potential safety hazards. In the field of education, emotion detection can provide valuable information, such as assessing the psychological conditions of students and teachers and promptly intervening in individuals in unhealthy states. It can also detect students' emotional states during the learning process and compare them with teachers' teaching activities to optimize the teaching environment and improve teaching quality. Accurately obtaining emotional states can improve the efficiency of emotional regulation and achieve more targeted intervention. Therefore, fine-grained emotion classification technology is very important and has great application potential.

[0004] Current emotion detection methods are primarily categorized as those based on non-physiological signals and those based on physiological signals. Non-physiological signal detection includes facial expressions, speech, or gestures, while physiological signal detection includes electrocardiogram (ECG), electroencephalogram (EEG), electromyography (EMG), and respiratory signals. Because EEG signals are directly related to consciousness and are difficult to conceal, suppress, or mask, EEG-based emotion detection has garnered significant attention.

[0005] The concept of intensity in fine-grained emotions has been studied in the psychology of emotion. One of the foundations of psychologist Robert Plutchik's emotion wheel theory is that emotions exhibit varying intensities. He argues that emotions of varying intensities influence each other, generating new and distinct emotions and thus establishing new emotional states.

[0006] Traditional EEG-based emotion computing recognizes broad emotion categories, such as happiness, calmness, and sadness. However, the intensity of emotions within these broad categories still varies at fine-grained levels, such as happiness, moderate happiness, and extreme happiness. Traditional EEG-based emotion recognition fails to identify fine-grained emotional states. This is primarily due to the homology of EEG patterns within the same broad emotion category. Consequently, intra-class variability within these broad categories is minimal, and feature distributions are similar, making fine-grained emotion classification difficult. Summary of the Invention

[0007] In order to solve the above problems existing in the prior art, the present invention provides a fine-grained emotional EEG detection method and system in virtual reality. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0008] In a first aspect, the present invention provides a fine-grained emotional EEG detection method in virtual reality, comprising:

[0009] Collect users' real-time emotional EEG data;

[0010] The user's real-time emotional EEG data is input into a trained fine-grained emotional EEG decoding network, so that the fine-grained emotional EEG decoding network sequentially extracts the time features of each EEG channel in the user's real-time emotional EEG data, performs time domain feature enhancement on the time features, extracts the spatial features hidden in each EEG channel in the time features after the time domain feature enhancement, and returns to perform time domain feature enhancement again, and calculates the confidence of different target classifications based on the features after the enhancement, thereby outputting the fine-grained emotion category to which the user's real-time emotional EEG data belongs.

[0011] Optionally, the fine-grained emotion EEG decoding network includes a time feature extraction module, a channel attention enhancement module, a spatial feature extraction module and a classifier module; the time feature extraction module is used to extract the time features in each EEG channel, and this module is implemented using grouped convolution; the channel attention module is used to enhance the time domain features of the EEG signal after the time features are extracted, and this module is implemented using a channel-based attention mechanism; the spatial feature extraction module is used to extract spatial features hidden in multiple EEG channels, and this module is implemented by using cross-channel convolution kernels; the classifier module contains a fully connected layer, which is used to obtain the final classification result of the emotion EEG data.

[0012] Optionally, the trained fine-grained emotion EEG decoding network is trained through the following steps:

[0013] Stimulate users' emotions by playing emotional response videos, thereby collecting users' offline fine-grained emotional EEG data;

[0014] The offline fine-grained emotional EEG data carries classification labels based on user emotion scores;

[0015] Preprocess the collected offline fine-grained emotional EEG data;

[0016] Divide the pre-processed emotional EEG data into training sets and test sets according to the preset ratio;

[0017] The training set is input into the constructed fine-grained emotion EEG decoding network for iterative training until the loss function converges to obtain the trained fine-grained emotion EEG decoding network;

[0018] The trained fine-grained emotion EEG decoding network is tested using the test set to determine whether the trained fine-grained emotion EEG decoding network is well trained.

[0019] Optionally, stimulating the user's emotions by playing an emotional response video, thereby collecting the user's offline fine-grained emotional EEG data, includes:

[0020] Play a video in the form of a parent segment and its three corresponding child segments, and collect EEG signals while the user is watching the parent segment;

[0021] Obtaining the score classification given to the EEG signal by the user after watching the parent segment according to the first scoring rule, and collecting the emotional intensity of the user trying to recall the matching position of the parent segment while watching the child segment, and classifying the score given to the EEG signal according to the second scoring rule;

[0022] The first scoring rule is a classification evaluation rule that evaluates the user's viewing emotion as positive or negative. The second scoring rule is a fine-grained emotion label evaluation rule that evaluates the user's viewing emotion when watching the sub-segment based on the classification evaluation of the parent segment as low positive emotion, high positive emotion, low negative emotion, or high negative emotion.

[0023] The scoring classification of the second scoring rule is used as the classification label of the EEG signal, and the collected EEG signal is determined as offline fine-grained emotional EEG data.

[0024] Optionally, preprocessing the collected offline fine-grained emotional EEG data includes:

[0025] Perform baseline removal processing on the collected EEG signals;

[0026] The EEG signal after baseline removal was subjected to 1-75 Hz bandpass filtering to remove power frequency interference;

[0027] According to the position of the sub-segment on the parent segment, the position of the target signal segment containing fine-grained emotional information is determined from the EEG signal;

[0028] According to the location of the target signal segment, the EEG signal with power frequency interference removed is cut using a 1s time window to obtain the target signal segment;

[0029] The target signal segment is taken as the EEG signal after preprocessing.

[0030] Optionally, the temporal feature extraction module includes two temporal convolutional layers connected in sequence, where the first temporal convolutional block includes one layer of 1*3 convolution, one layer of batch normalization, and one layer of ELU activation function; the second temporal convolutional block includes one layer of 1*7 convolution, one layer of batch normalization, and one layer of ELU activation function;

[0031] The channel attention module consists of one global average pooling layer to obtain the weight of each EEG channel, one 1*1 convolutional layer to reduce the feature channel dimension and improve the operation speed, one ELU activation function layer, and one fully connected layer to restore the channel dimension and generate attention weight information.

[0032] The spatial feature extraction module consists of two sequentially connected spatial convolution blocks. The first spatial convolution block contains a layer of 3*1 convolution, a layer of batch normalization, and a layer of ELU activation function. The second temporal convolution block contains a layer of 7*1 convolution, a layer of batch normalization, and a layer of ELU activation function.

[0033] The classifier module includes a fully connected layer, which is used to output the classification confidence of four types of targets and select the category with the largest confidence as the output category.

[0034] Optionally, after outputting the fine-grained emotion category to which the user's real-time emotional EEG data belongs, the method for detecting fine-grained emotional EEG in virtual reality further includes:

[0035] Display the fine-grained emotion category to which the user's real-time emotional EEG data belongs.

[0036] In a second aspect, the present invention provides a fine-grained emotional EEG detection system in virtual reality, comprising:

[0037] EEG signal acquisition module, used to collect real-time emotional EEG data of users;

[0038] The fine-grained emotion EEG decoding network module is used to input the user's real-time emotion EEG data into the trained fine-grained emotion EEG decoding network, so that the fine-grained emotion EEG decoding network sequentially extracts the time features of each EEG channel in the user's real-time emotion EEG data, performs time domain feature enhancement on the time features, extracts the spatial features hidden in each EEG channel in the time features after the time domain feature enhancement and returns to perform time domain feature enhancement again, and calculates the confidence of different target classifications based on the features after the enhancement, thereby outputting the fine-grained emotion category to which the user's real-time emotion EEG data belongs.

[0039] Optionally, the fine-grained emotion EEG decoding network includes a time feature extraction module, a channel attention enhancement module, a spatial feature extraction module and a classifier module; the time feature extraction module is used to extract the time features in each EEG channel, and this module is implemented using grouped convolution; the channel attention module is used to enhance the time domain features of the EEG signal after the time features are extracted, and this module is implemented using a channel-based attention mechanism; the spatial feature extraction module is used to extract spatial features hidden in multiple EEG channels, and this module is implemented by using cross-channel convolution kernels; the classifier module contains a fully connected layer, which is used to obtain the final classification result of the emotion EEG data.

[0040] Optionally, the fine-grained emotion EEG detection system includes:

[0041] The induction module is used to stimulate the user's emotions by playing an emotional response video, so that the EEG signal acquisition module can collect the user's offline fine-grained emotional EEG data;

[0042] The offline fine-grained emotional EEG data carries classification labels based on user emotion scores;

[0043] The EEG signal preprocessing module is used to preprocess the collected offline fine-grained emotional EEG data;

[0044] Divide the pre-processed emotional EEG data into training sets and test sets according to the preset ratio;

[0045] The network training module is used to input the training set into the constructed fine-grained emotion EEG decoding network for iterative training until the loss function converges to obtain the trained fine-grained emotion EEG decoding network;

[0046] The network forward reasoning module is used to test the trained fine-grained emotion EEG decoding network using the test set to determine whether the trained fine-grained emotion EEG decoding network is well trained.

[0047] Beneficial effects of the present invention:

[0048] The present invention provides a fine-grained emotional EEG detection method and system in virtual reality. The method uses immersion and can effectively isolate environmental interference to play emotional videos to stimulate user emotions, improve the quality of collected EEG signals, and enhance the usability of the system. A fine-grained emotional EEG decoding network is then used to extract spatial and temporal features from emotional EEG signals. Compared with the traditional method that cannot effectively express the strength of user emotions, the present invention can further obtain the user's emotional intensity on the premise of decoding the user's emotional state. Moreover, after completing the training phase, the present invention can be directly used, whether for offline analysis of user status or online decoding to obtain user emotions. It is suitable for application in a variety of tasks and can broaden the application scenarios of emotional EEG.

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of a fine-grained emotional EEG detection method in virtual reality provided by the present invention;

[0051] Figure 2 A diagram showing the fine-grained emotional EEG decoding network structure provided by the present invention;

[0052] Figure 3 Schematic diagram of the fine-grained emotional brain-computer interaction system provided by the present invention;

[0053] Figure 4 Schematic diagram of the fine-grained emotional brain-computer interaction system paradigm provided by the present invention;

[0054] Figure 5 The task-state workflow of the fine-grained emotional brain-computer interaction system provided by the present invention

[0055] Figure 6 This is a system block diagram of the fine-grained emotional EEG classification method under virtual reality provided by the present invention; DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0057] like Figure 1 As shown, the present invention provides a fine-grained emotional EEG detection method in virtual reality, which includes:

[0058] S1, collects users’ real-time emotional EEG data;

[0059] S2, input the user's real-time emotional EEG data into the trained fine-grained emotional EEG decoding network, so that the fine-grained emotional EEG decoding network sequentially extracts the time features of each EEG channel in the user's real-time emotional EEG data, performs time domain feature enhancement on the time features, extracts the spatial features hidden in each EEG channel in the time features after the time domain feature enhancement and returns to perform time domain feature enhancement again, and calculates different target classification confidences based on the features after the enhancement, thereby outputting the fine-grained emotion category to which the user's real-time emotional EEG data belongs.

[0060] The network structure refers to 2. The fine-grained emotion EEG decoding network of the present invention includes a time feature extraction module, a channel attention enhancement module, a spatial feature extraction module and a classifier module; the time feature extraction module is used to extract the time features in each EEG channel, and this module is implemented by group convolution; the channel attention module is used to enhance the time domain features of the EEG signal after the time features are extracted, and this module is implemented by a channel-based attention mechanism; the spatial feature extraction module is used to extract the spatial features hidden in multiple EEG channels, and this module is implemented by using a cross-channel convolution kernel; the classifier module contains a fully connected layer, which is used to obtain the final classification result of the emotion EEG data.

[0061] The temporal feature extraction module of the present invention includes two temporal convolutional layers connected in sequence, the first temporal convolutional block includes one layer of 1*3 convolution, one layer of batch normalization, and one layer of ELU activation function; the second temporal convolutional block includes one layer of 1*7 convolution, one layer of batch normalization, and one layer of ELU activation function; the channel attention module includes one layer of global average pooling layer for obtaining the weight of each EEG channel, one layer of 1*1 convolutional layer for reducing the feature channel dimension and improving the calculation speed, one layer of ELU activation function, and one layer of fully connected layer for restoring the channel dimension and generating attention weight information; the spatial feature extraction module includes: two sequentially connected spatial convolutional blocks, the first spatial convolutional block includes one layer of 3*1 convolution, one layer of batch normalization, and one layer of ELU activation function; the second temporal convolution block includes one layer of 7*1 convolution, one layer of batch normalization, and one layer of ELU activation function; the classifier module includes a fully connected layer, which is used to output the classification confidence of four categories of targets and select the category with the largest confidence as the output category.

[0062] The present invention can display the fine-grained emotion category to which the user's real-time emotional EEG data belongs.

[0063] The present invention classifies online process and offline state as training state and task state, see Figure 3The training phase refers to the algorithm training process. This phase involves designing a fine-grained emotion paradigm, collecting offline fine-grained emotion EEG data, forming a personalized EEG signal dataset, and constructing a personalized model using a fine-grained emotion EEG network for subsequent fine-grained emotion monitoring in the task phase. The core technology of this phase lies in the construction of a fine-grained emotion EEG network; the task phase refers to the practical application of laboratory technology. This phase primarily deploys the personalized model and deep network constructed during the training phase on training tasks, enabling real-time online monitoring of the test subject's fine-grained emotional state and assisting in training.

[0064] In this embodiment, users will use the algorithm model trained offline to perform flight missions. During the training phase, the user's emotional EEG information is collected and fed into a neural network to determine network parameters. After the training phase, during the mission phase, the user's EEG information is collected online and fed into the neural network. The network's forward inference module outputs a fine-grained emotion classification result, which is then transmitted back to the virtual reality display module for feedback and display, prompting the user to regulate their emotions.

[0065] As an optional embodiment of the present invention, the trained fine-grained emotion EEG decoding network is obtained by training through the following steps:

[0066] Step a1: Stimulate the user's emotions by playing an emotional response video, thereby collecting the user's offline fine-grained emotional EEG data;

[0067] The offline fine-grained emotional EEG data carries classification labels based on user emotion scores;

[0068] It is worth noting that the fine-grained emotional brain-computer interaction experimental paradigm refers to the classic paradigm of emotional brain-computer interface and improves it by adding fine-grained emotional fragment evaluation. Figure 4 As shown in Figure 2, the playback order of each experimental video consists of one parent segment and three corresponding child segments. The EEG signals of users were recorded while they watched the parent segment, and users immediately rated the segment after watching it. The scoring criteria were either positive or negative.

[0069] Step b1: preprocessing the collected offline fine-grained emotional EEG data;

[0070] This step can remove the baseline of the collected EEG signal; perform 1-75 Hz bandpass filtering on the EEG signal after removing the baseline to remove power frequency interference; determine the location of the target signal segment containing fine-grained emotional information from the EEG signal based on the position of the sub-segment on the parent segment; cut the EEG signal after removing power frequency interference using a 1s time window according to the location of the target signal segment to obtain the target signal segment; and use the target signal segment as the EEG signal after preprocessing.

[0071] Step c1: Divide the pre-processed emotional EEG data into a training set and a test set according to a preset ratio;

[0072] Step d1: Input the training set into the constructed fine-grained emotion EEG decoding network for iterative training until the loss function converges to obtain the trained fine-grained emotion EEG decoding network;

[0073] Step e1: Use the test set to test the trained fine-grained emotion EEG decoding network to determine whether the trained fine-grained emotion EEG decoding network is well trained.

[0074] In the present invention, the user's emotions are stimulated by playing an emotional response video, thereby collecting the user's offline fine-grained emotional EEG data, including:

[0075] Step a2: Play a video in the form of a parent segment and its corresponding three child segments, and collect EEG signals of the user while watching the parent segment;

[0076] The EEG signals collected by the present invention are 64-channel signals with a sampling rate of 1024 Hz.

[0077] Step b2: Obtain the score classification given to the EEG signal by the user after watching the parent segment according to the first scoring rule, and collect the emotional intensity when the user attempts to recall the matching position of the parent segment while watching the child segment, and classify the score given to the EEG signal according to the second scoring rule;

[0078] The first scoring rule is a classification evaluation rule that evaluates the user's viewing emotion as positive or negative. The second scoring rule is a fine-grained emotion label evaluation rule that evaluates the user's viewing emotion when watching the sub-segment based on the classification evaluation of the parent segment as low positive emotion, high positive emotion, low negative emotion, or high negative emotion.

[0079] Step c2: Using the scoring classification of the second scoring rule as the classification label of the EEG signal, and determining the collected EEG signal as offline fine-grained emotional EEG data.

[0080] It is worth noting that the flight mission is completed in a virtual reality training cabin, and the mission workflow is as follows: Figure 5The success of a flight mission is judged by the number of operational errors during the flight, with 20 errors being the standard for this experiment. The tester pilots the aircraft within a training cabin. At the start of the mission, the cabin sends a synchronization signal to the data acquisition terminal to ensure data synchronization. Simultaneously, the data is transmitted to the data processing terminal in real time, outputting the tester's fine-grained emotional state every 1 second. If fine-grained emotions are detected that are detrimental to mission completion, a signal is sent to the training cabin, providing tactile feedback to the tester, allowing them to adjust their emotional state promptly.

[0081] The fine-grained emotional EEG online visualization system displays the intensity of the test subject's current emotional state in real-time as an image. Before the experiment begins, click "Start Monitoring" and attempt to connect to the fine-grained emotion output terminal. Once connected, a message will be displayed in the text box in the lower left corner indicating whether communication is working properly. Once monitoring begins, four different fine-grained emotions will be displayed on the right side of the interface based on the monitoring results. If a harmful emotion is detected, feedback will be triggered to the test subject to assist in adjusting their emotional state.

[0082] like Figure 6 As shown, the present invention provides a fine-grained emotional EEG detection system in virtual reality, comprising:

[0083] EEG signal acquisition module, used to collect real-time emotional EEG data of users;

[0084] The fine-grained emotion EEG decoding network module is used to input the user's real-time emotion EEG data into the trained fine-grained emotion EEG decoding network, so that the fine-grained emotion EEG decoding network sequentially extracts the time features of each EEG channel in the user's real-time emotion EEG data, performs time domain feature enhancement on the time features, extracts the spatial features hidden in each EEG channel in the time features after the time domain feature enhancement and returns to perform time domain feature enhancement again, and calculates the confidence of different target classifications based on the features after the enhancement, thereby outputting the fine-grained emotion category to which the user's real-time emotion EEG data belongs.

[0085] The fine-grained emotion EEG decoding network includes a time feature extraction module, a channel attention enhancement module, a spatial feature extraction module and a classifier module; the time feature extraction module is used to extract the time features in each EEG channel, and this module is implemented using grouped convolution; the channel attention module is used to enhance the time domain features of the EEG signal after the time features are extracted, and this module is implemented using a channel-based attention mechanism; the spatial feature extraction module is used to extract the spatial features hidden in multiple EEG channels, and this module is implemented by using cross-channel convolution kernels; the classifier module contains a fully connected layer, which is used to obtain the final classification result of the emotion EEG data.

[0086] The fine-grained emotional EEG detection system includes:

[0087] The induction module is used to stimulate the user's emotions by playing an emotional response video, so that the EEG signal acquisition module can collect the user's offline fine-grained emotional EEG data;

[0088] The offline fine-grained emotional EEG data carries classification labels based on user emotion scores;

[0089] The EEG signal preprocessing module is used to preprocess the collected offline fine-grained emotional EEG data;

[0090] Divide the pre-processed emotional EEG data into training sets and test sets according to the preset ratio;

[0091] The network training module is used to input the training set into the constructed fine-grained emotion EEG decoding network for iterative training until the loss function converges to obtain the trained fine-grained emotion EEG decoding network;

[0092] The network forward reasoning module is used to test the trained fine-grained emotion EEG decoding network using the test set to determine whether the trained fine-grained emotion EEG decoding network is well trained.

[0093] In order to effectively detect the intensity of emotions, improve the accuracy of emotional EEG classification, and combine the characteristics of virtual reality to enhance the quality of user-computer interaction, the system includes an induction module, a virtual reality display module, an EEG signal acquisition and processing module, a network training module, an emotional EEG decoding module, and a task state module. Each module is described as follows:

[0094] The induction module uses virtual reality to play film clips and elicit emotional EEG signals. To ensure accurate emotional induction, in addition to sufficient time between trials, the present invention also designs the playback sequence based on the emotional attributes of the videos to avoid emotional priming effects. Furthermore, to investigate the cross-trial capabilities of the designed neural network, users are required to conduct fine-grained emotional brain-computer interaction experiments again after a period of at least one week.

[0095] The virtual reality display module plays emotional EEG clips, generating time stamps. This time stamp generates a time synchronization signal that is sent to the EEG acquisition module. Based on the EEG decoding results, feedback is displayed in the form of easily understandable facial expressions.

[0096] The EEG signal acquisition module collects EEG data from the user in an induced emotional state at a fixed sampling rate, and this data will be used as raw EEG data.

[0097] The EEG signal preprocessing module filters the raw EEG data and segments the filtered data using a 1-second time window for subsequent decoding.

[0098] The network training module extracts features from the preprocessed EEG data according to the user's emotional intensity and generates network parameters in the fine-grained emotional EEG decoding network module.

[0099] Fine-grained emotion EEG decoding network module, which stores the structural information and parameter information of the fine-grained emotion EEG decoding network.

[0100] The network forward reasoning module uses the network structure and network parameter information to decode the emotional EEG output emotion classification results, and sends the results to the virtual reality display module for visual information feedback.

[0101] The present invention provides a fine-grained emotional EEG detection method and system in virtual reality, which uses immersion and the ability to effectively isolate environmental interference to play emotional videos to stimulate user emotions, improve the quality of collected EEG signals, and enhance the usability of the system. A fine-grained emotional EEG decoding network is then used to extract spatial and temporal features from the emotional EEG signals. On the premise of decoding the user's emotional state, the user's emotional intensity can be further obtained. After completing the training phase, the present invention can be directly used, whether for offline analysis of user status or online decoding to obtain user emotions. It is suitable for application in a variety of tasks and can broaden the application scenarios of emotional EEG.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0103] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A fine-grained emotional EEG detection method in virtual reality, characterized in that: include: Collect users' real-time emotional EEG data; The user's real-time emotional EEG data is input into a trained fine-grained emotional EEG decoding network, so that the fine-grained emotional EEG decoding network sequentially extracts the time features of each EEG channel in the user's real-time emotional EEG data, performs time domain feature enhancement on the time features, extracts the spatial features hidden in each EEG channel in the time features after the time domain feature enhancement, and returns to perform time domain feature enhancement again, and calculates the confidence of different target classifications based on the features after the enhancement, thereby outputting the fine-grained emotion category to which the user's real-time emotional EEG data belongs; wherein the trained fine-grained emotional EEG decoding network stimulates the user's emotions by playing an emotional response video, thereby collecting the user's offline fine-grained emotional EEG data, and uses the offline fine-grained emotional EEG data to train the constructed fine-grained emotional EEG decoding network; the method of stimulating the user's emotions by playing an emotional response video, thereby collecting the user's offline fine-grained emotional EEG data, includes: Play a video in the form of a parent segment and its three corresponding child segments, and collect EEG signals while the user is watching the parent segment; Obtaining the score classification given to the EEG signal by the user after watching the parent segment according to the first scoring rule, and collecting the emotional intensity of the user trying to recall the matching position of the parent segment while watching the child segment, and classifying the score given to the EEG signal according to the second scoring rule; The first scoring rule is a classification evaluation rule that evaluates the user's viewing emotion as positive or negative. The second scoring rule is a fine-grained emotion label evaluation rule that evaluates the user's viewing emotion when watching the sub-segment based on the classification evaluation of the parent segment as low positive emotion, high positive emotion, low negative emotion, or high negative emotion. The scoring classification of the second scoring rule is used as the classification label of the EEG signal, and the collected EEG signal is determined as offline fine-grained emotional EEG data.

2. The fine-grained emotional EEG detection method under virtual reality according to claim 1 is characterized in that: The fine-grained emotion EEG decoding network includes a temporal feature extraction module, a channel attention enhancement module, a spatial feature extraction module, and a classifier module; the temporal feature extraction module is used to extract temporal features from each EEG channel, and this module is implemented using grouped convolution; the channel attention module is used to enhance the temporal features of the EEG signal after temporal feature extraction, and this module is implemented using a channel-based attention mechanism; The spatial feature extraction module is used to extract spatial features hidden in multiple EEG channels. This module is implemented by utilizing cross-channel convolution kernels. The classifier module contains a fully connected layer to obtain the final classification results of emotional EEG data.

3. The fine-grained emotional EEG detection method under virtual reality according to claim 2, characterized in that: The trained fine-grained emotion EEG decoding network is obtained by training through the following steps: Stimulate users' emotions by playing emotional response videos, thereby collecting users' offline fine-grained emotional EEG data; The offline fine-grained emotional EEG data carries classification labels based on user emotion scores; Preprocess the collected offline fine-grained emotional EEG data; Divide the pre-processed emotional EEG data into training sets and test sets according to the preset ratio; The training set is input into the constructed fine-grained emotion EEG decoding network for iterative training until the loss function converges to obtain the trained fine-grained emotion EEG decoding network; The trained fine-grained emotion EEG decoding network is tested using the test set to determine whether the trained fine-grained emotion EEG decoding network is well trained.

4. The fine-grained emotional EEG detection method under virtual reality according to claim 3 is characterized in that: The preprocessing of the collected offline fine-grained emotional EEG data includes: Perform baseline removal processing on the collected EEG signals; The EEG signal after baseline removal was subjected to 1-75 Hz bandpass filtering to remove power frequency interference; According to the position of the sub-segment on the parent segment, the position of the target signal segment containing fine-grained emotional information is determined from the EEG signal; According to the location of the target signal segment, the EEG signal with power frequency interference removed is cut using a 1s time window to obtain the target signal segment; The target signal segment is taken as the EEG signal after preprocessing.

5. The fine-grained emotional EEG detection method under virtual reality according to claim 2, characterized in that: The temporal feature extraction module consists of two sequentially connected temporal convolutional layers. The first temporal convolutional block contains one layer of 1*3 convolution, one layer of batch normalization, and one layer of ELU activation function; the second temporal convolutional block contains one layer of 1*7 convolution, one layer of batch normalization, and one layer of ELU activation function. The channel attention module consists of one global average pooling layer to obtain the weight of each EEG channel, one 1*1 convolutional layer to reduce the feature channel dimension and improve the operation speed, one ELU activation function layer, and one fully connected layer to restore the channel dimension and generate attention weight information. The spatial feature extraction module consists of two sequentially connected spatial convolution blocks. The first spatial convolution block contains a layer of 3*1 convolution, a layer of batch normalization, and a layer of ELU activation function. The second temporal convolution block contains a layer of 7*1 convolution, a layer of batch normalization, and a layer of ELU activation function. The classifier module includes a fully connected layer, which is used to output the classification confidence of four types of targets and select the category with the largest confidence as the output category.

6. The fine-grained emotional EEG detection method in virtual reality according to claim 1, characterized in that: After outputting the fine-grained emotion category to which the user's real-time emotional EEG data belongs, the fine-grained emotional EEG detection method in virtual reality further includes: Display the fine-grained emotion category to which the user's real-time emotional EEG data belongs.

7. A fine-grained emotional EEG detection system in virtual reality, characterized by: include: EEG signal acquisition module, used to collect real-time emotional EEG data of users; A fine-grained emotion EEG decoding network module is used to input the user's real-time emotion EEG data into a trained fine-grained emotion EEG decoding network, so that the fine-grained emotion EEG decoding network sequentially extracts the time features of each EEG channel in the user's real-time emotion EEG data, performs time domain feature enhancement on the time features, extracts the spatial features hidden in each EEG channel in the enhanced time features, returns to perform time domain feature enhancement again, and calculates different target classification confidences based on the enhanced features, thereby outputting the fine-grained emotion category to which the user's real-time emotion EEG data belongs; The trained fine-grained emotion EEG decoding network stimulates the user's emotions by playing an emotion-response video, thereby collecting the user's offline fine-grained emotion EEG data, and uses the offline fine-grained emotion EEG data to train the constructed fine-grained emotion EEG decoding network; the method of stimulating the user's emotions by playing an emotion-response video, thereby collecting the user's offline fine-grained emotion EEG data, includes: Play a video in the form of a parent segment and its three corresponding child segments, and collect EEG signals while the user is watching the parent segment; Obtaining the score classification given to the EEG signal by the user after watching the parent segment according to the first scoring rule, and collecting the emotional intensity of the user trying to recall the matching position of the parent segment while watching the child segment, and classifying the score given to the EEG signal according to the second scoring rule; The first scoring rule is a classification evaluation rule that evaluates the user's viewing emotion as positive or negative. The second scoring rule is a fine-grained emotion label evaluation rule that evaluates the user's viewing emotion when watching the sub-segment based on the classification evaluation of the parent segment as low positive emotion, high positive emotion, low negative emotion, or high negative emotion. The scoring classification of the second scoring rule is used as the classification label of the EEG signal, and the collected EEG signal is determined as offline fine-grained emotional EEG data.

8. The fine-grained emotional EEG detection system under virtual reality according to claim 7, characterized in that: The fine-grained emotion EEG decoding network includes a temporal feature extraction module, a channel attention enhancement module, a spatial feature extraction module, and a classifier module; the temporal feature extraction module is used to extract temporal features from each EEG channel, and this module is implemented using grouped convolution; the channel attention module is used to enhance the temporal features of the EEG signal after temporal feature extraction, and this module is implemented using a channel-based attention mechanism; The spatial feature extraction module is used to extract spatial features hidden in multiple EEG channels. This module is implemented by utilizing cross-channel convolution kernels. The classifier module contains a fully connected layer to obtain the final classification results of emotional EEG data.

9. The fine-grained emotional EEG detection system under virtual reality according to claim 7, characterized in that: The fine-grained emotional EEG detection system includes: The induction module is used to stimulate the user's emotions by playing an emotional response video, so that the EEG signal acquisition module can collect the user's offline fine-grained emotional EEG data; The offline fine-grained emotional EEG data carries classification labels based on user emotion scores; The EEG signal preprocessing module is used to preprocess the collected offline fine-grained emotional EEG data; Divide the pre-processed emotional EEG data into training sets and test sets according to the preset ratio; The network training module is used to input the training set into the constructed fine-grained emotion EEG decoding network for iterative training until the loss function converges to obtain the trained fine-grained emotion EEG decoding network; The network forward reasoning module is used to test the trained fine-grained emotion EEG decoding network using the test set to determine whether the trained fine-grained emotion EEG decoding network is well trained.

Images