Emotion mode fusion recognition method and device for electroencephalogram signals

Abstract

The invention discloses an emotion mode fusion recognition method and device for electroencephalogram signals. The method comprises the steps that an electroencephalogram signal training data set is obtained; the electroencephalogram signal training data set comprises an electroencephalogram signal sequence and corresponding emotion label information; performing training processing on an emotion mode fusion recognition model by utilizing the electroencephalogram signal training data set to obtain a trained emotion mode fusion recognition model; and processing the acquired electroencephalogram signals by using the trained emotion mode fusion recognition model to obtain an emotion mode recognition result. According to the method, emotion fusion recognition is carried out on the fused electroencephalogram features by utilizing the fact that EEG is highly related to the emotional state in time, space and frequency domains and has complementary characteristics, and the emotion recognition ability and the recognition model accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the fields of electroencephalogram signal processing and artificial intelligence, and in particular to a method and device for fusion recognition of emotion patterns of electroencephalogram signals. Background Art

[0002] EEG signals are a multi-channel time series that contain a large amount of central nervous system information that is directly related to higher-level cognitive processes, including emotions. Electroencephalogram signals (EEG) are highly correlated with emotional states in time, space, and frequency domains, and have complementary characteristics.

[0003] In existing research on emotion recognition of EEG signals, generally only one or two types of features are used for encoding, and most of them use a combination of spatial and frequency domain features. This does not take into account that EEG is highly correlated with emotional state in time, space, and frequency domains and has complementary characteristics. There is a lack of emotion fusion recognition that integrates EEG features in the three aspects of "time, space, and frequency", which leads to a decrease in emotion recognition ability and a low accuracy rate of the constructed recognition model. Summary of the invention

[0004] The present invention mainly solves the problem that the existing emotion recognition methods do not take into account that EEG is highly correlated with emotional state in time, space and frequency domains and has complementary characteristics, lack emotion fusion recognition that integrates EEG characteristics in the three aspects of "time, space and frequency", resulting in reduced emotion recognition ability and low accuracy of the constructed recognition model. The present invention discloses a method and device for emotion pattern fusion recognition of EEG signals.

[0005] In a first aspect of the embodiments of the present application, a method for fusion recognition of emotion patterns of EEG signals is disclosed, comprising:

[0006] S1, obtaining an EEG signal training data set; the EEG signal training data set includes an EEG signal sequence and corresponding emotion label information;

[0007] S2, using the EEG signal training data set to train the emotion pattern fusion recognition model to obtain a trained emotion pattern fusion recognition model;

[0008] S3, using the trained emotion pattern fusion recognition model to process the collected EEG signal to obtain an emotion pattern recognition result.

[0009] The emotion pattern fusion recognition model includes: a preprocessing module, a frequency domain coding module, a space coding module, a feature fusion module, a time coding module and a classifier module;

[0010] The preprocessing module is used to preprocess the received EEG signal sequence to obtain a spatial signal set and a frequency domain signal set;

[0011] The frequency domain coding module is connected to the preprocessing module and the feature fusion module respectively, and is used to perform frequency domain coding processing on the received frequency domain signal set to obtain a frequency domain coded signal;

[0012] The spatial coding module is connected to the preprocessing module and the feature fusion module respectively, and is used to perform spatial coding processing on the received spatial signal set to obtain a spatially coded signal;

[0013] The feature fusion module is connected to the time coding module and is used to splice the received frequency domain coded signal and the space coded signal in time sequence to obtain a fused coded signal;

[0014] The time coding module is used to perform time coding processing on the received fusion coded signal to obtain a time coded signal;

[0015] The classifier module is connected to the time coding module and is used to classify the received time coding signal to obtain an emotion pattern recognition result.

[0016] The preprocessing process of the preprocessing module includes:

[0017] Performing data cleaning processing on the received EEG signal sequence to obtain a first signal sequence;

[0018] performing pattern detection processing on the first signal sequence to obtain a second signal sequence;

[0019] Perform spectrum transformation on the second signal sequence to obtain a frequency domain signal set; the frequency domain signal set X t1 The expression is

[0020] X t1 =[a1, a2, …, a F ],

[0021] Among them, a i is the frequency domain signal of the i-th frequency band, and F is the total number of frequency bands;

[0022] Perform spectrum transformation processing on the signal collected by each electrode in the second signal sequence to obtain a spatial signal set; the spatial signal set X t2 The expression is

[0023]

[0024] Among them, b i is the frequency domain signal of the signal collected by the i-th electrode, T represents the transposition operation, and C is the total number of electrodes.

[0025] The mode detection process comprises:

[0026] For each data of the first signal sequence, data acquisition information of the data is used as a known independent variable, and data values ​​of the data are used as a known dependent variable. A curve to be approximated is constructed by using the known independent variable and the known dependent variable, and a function approximation method is used to perform curve fitting on the curve to be approximated to obtain an optimal consistent approximation polynomial f(Ix) of the class data attribute;

[0027] Using the best consistent approximation polynomial f(Ix), the known independent variable is calculated to obtain an approximate dependent variable;

[0028] Determine whether the absolute value of the difference between the approximate dependent variable and the corresponding known dependent variable is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the first signal sequence; if it is less than or equal to the first regression discrimination threshold, do not process the data.

[0029] The frequency domain coding module includes an input module, a mask processing module, a position coding module, a frequency domain encoder, and a frequency domain decoder;

[0030] The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence; the masked sequence has a corresponding signal sequence;

[0031] The position encoding module is connected to the frequency domain encoder and is used to perform position encoding processing on the masked sequence to obtain a first feature sequence;

[0032] The frequency domain encoder is connected to the frequency domain decoder and is used to encode the first feature sequence to obtain a frequency domain coded signal;

[0033] The frequency domain decoder is used to decode the frequency domain coded signal to obtain a frequency domain decoded signal;

[0034] The spatial encoding module includes an input module, a mask processing module, a position encoding module, a spatial encoder, and a spatial decoder;

[0035] The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence;

[0036] The position encoding module is connected to the spatial encoder and is used to perform position encoding processing on the masked sequence to obtain a second feature sequence;

[0037] The spatial encoder is connected to the spatial decoder and is used to encode the second feature sequence to obtain a spatially coded signal;

[0038] The spatial decoder is used to decode the spatially encoded signal to obtain a spatially decoded signal;

[0039] The time encoding module includes an input module, a mask processing module, a position encoding module, a time encoder, and a time decoder;

[0040] The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence;

[0041] The position encoding module is connected to the time encoder and is used to perform position encoding processing on the masked sequence to obtain a third feature sequence;

[0042] The time encoder is connected to the time decoder and is used to encode the third characteristic sequence to obtain a time code signal;

[0043] The time decoder is used to decode the time coded signal to obtain a time decoded signal.

[0044] The training process of the emotion pattern fusion recognition model includes:

[0045] Pre-training the frequency domain coding module, the space coding module, and the time coding module to obtain a pre-trained emotion pattern fusion recognition model;

[0046] Initialize the number of training iterations;

[0047] Using the EEG signal sequence in the EEG signal training data set as input data, inputting it into the pre-trained emotion pattern fusion recognition model to obtain an emotion prediction value;

[0048] Performing difference calculation processing on the emotion label information and the emotion prediction value of the input data to obtain a difference value;

[0049] Determine whether the difference value satisfies a convergence condition, and obtain a first determination result;

[0050] When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result;

[0051] When the second judgment result is no, determining that the model training state does not meet the training termination condition;

[0052] When the second judgment result is yes, determining that the model training state satisfies the training termination condition;

[0053] When the first judgment result is yes, determining that the model training state satisfies a training termination condition;

[0054] When the model training state does not meet the training termination condition, the classifier module is updated with the first parameter update model, the training iteration number is increased by 1, and the execution is triggered to use the EEG signal sequence in the EEG signal training data set as input data;

[0055] When the model training state satisfies the training termination condition, the training process of the pre-trained emotion pattern fusion recognition model is completed to obtain a trained emotion pattern fusion recognition model.

[0056] The pre-training process comprises:

[0057] Initialize the number of pre-training iterations;

[0058] Using the masked sequence as input data, and inputting it into the encoding module to be pre-trained;

[0059] Processing the input data using the encoding module to be pre-trained, and using the output of the decoder contained therein as a prediction signal sequence;

[0060] Performing difference calculation processing on the signal sequence corresponding to the masked sequence and the predicted signal sequence to obtain a difference value;

[0061] Determine whether the difference value satisfies a convergence condition, and obtain a third determination result;

[0062] When the third judgment result is no, judging whether the pre-training iteration count value is equal to the pre-training iteration count threshold, and obtaining a fourth judgment result;

[0063] When the fourth judgment result is no, determining that the model training state does not meet the training termination condition;

[0064] When the fourth judgment result is yes, determining that the model training state satisfies the training termination condition;

[0065] When the third judgment result is yes, determining that the model training state satisfies the training termination condition;

[0066] When the model training state does not meet the training termination condition, the second parameter update model is used to update the parameters of the encoding module to be pre-trained, the pre-training iteration number value is increased by 1, and the masked sequence is used as input data and input into the encoding module to be pre-trained.

[0067] When the model training state satisfies the training termination condition, the pre-training process is completed to obtain a pre-trained encoding module;

[0068] After the pre-training process of the frequency domain coding module, the space coding module and the time coding module is completed, a pre-trained emotion pattern fusion recognition model is obtained.

[0069] In a second aspect of an embodiment of the present invention, a device for fusion and recognition of emotion patterns of EEG signals is disclosed, the device comprising:

[0070] A memory storing executable program code;

[0071] a processor coupled to the memory;

[0072] The processor calls the executable program code stored in the memory to execute the method for emotional pattern fusion and recognition of EEG signals.

[0073] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the method for emotional pattern fusion and recognition of EEG signals.

[0074] According to a fourth aspect of the embodiments of the present invention, an information data processing terminal is disclosed, wherein the information data processing terminal is used to implement the method for emotional pattern fusion recognition of EEG signals.

[0075] The beneficial effects of the present invention are:

[0076] Before the formal training, the present invention first trains each coding module by a pre-training method. Pre-training has the following important advantages: the pre-training model has learned a common feature representation on a large amount of data, which can greatly reduce the amount of supervised data required; for classification tasks with scarce data, pre-training can help the model learn and generalize quickly; the pre-training model has learned rich semantic features and contextual information, which can be better summarized and migrated to new classification tasks; compared with training from scratch, the pre-training model has stronger generalization and is not easy to overfit; the pre-training model can converge quickly, and the number of training rounds and time required are greatly reduced; the classification model can be initialized using pre-trained parameters to accelerate the convergence process; the pre-training model can be flexibly migrated to different classification tasks, and only a small number of layers need to be fine-tuned; this transfer learning method greatly reduces the cost and difficulty of model development. In summary, pre-training technology can significantly improve the data efficiency, generalization and training efficiency of classification tasks, and has important value in classification tasks.

[0077] In order to enhance the model's ability to learn features, the present invention adopts a mask-based pre-training method. Pre-training can greatly reduce the amount of supervised data required and improve the data efficiency of classification tasks; the pre-trained model has stronger generalization and is not prone to overfitting; the classification model can be initialized using pre-trained parameters to accelerate the convergence process and improve training efficiency. The mask-based pre-training method can also reconstruct missing parts in the data to achieve a complementary effect between data. The use of pre-training technology can further improve the model's emotion recognition performance.

[0078] The present invention converts EEG into a three-dimensional representation of time, space and frequency for encoding to obtain features, and utilizes the complementary characteristics of EEG information in time, space and frequency to improve the accuracy of model emotion recognition.

[0079] The present invention uses two-layer coding. The first layer is parallel coding in the space and frequency domains. The obtained feature sequences are spliced ​​and then passed through the second layer of time coding. The global perception ability, multi-scale feature learning characteristics, model generalization ability and scalability are used to improve the performance of model emotion recognition.

[0080] The present invention uses a mask-based pre-training method. Pre-training can significantly reduce the amount of supervised data required and improve the data efficiency of classification tasks. In addition, the pre-trained model has stronger generalization and is not prone to overfitting. The pre-trained parameters can also be used to initialize the classification model to accelerate the convergence process and improve training efficiency. The mask-based pre-training method can also reconstruct the missing parts of the data to achieve a complementary effect between the data. The use of pre-training technology can improve the model's emotion recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a schematic diagram of the frequency domain and space encoding modules of the emotion pattern fusion recognition model of the present invention;

[0082] Figure 2 Schematic diagram of the composition of the spatial coding module of the present invention;

[0083] Figure 3 It is a schematic diagram of the composition of the frequency domain coding module of the present invention;

[0084] Figure 4 It is a schematic diagram of the composition of the time encoding module and the emotion classifier of the present invention. DETAILED DESCRIPTION

[0085] In order to better understand the content of the present invention, an embodiment is given here.

[0086] Figure 1 It is a schematic diagram of the frequency domain and space encoding modules of the emotion pattern fusion recognition model of the present invention; Figure 2 Schematic diagram of the composition of the spatial coding module of the present invention; Figure 3It is a schematic diagram of the composition of the frequency domain coding module of the present invention; Figure 4 It is a schematic diagram of the composition of the time encoding module and the emotion classifier of the present invention.

[0087] Figure 1 In the figure, 101 is a preprocessing module, 102 is a combination of a frequency domain coding module, a spatial coding module, and a feature fusion module, 103 is a time coding module, and 104 is a classifier module.

[0088] In a first aspect of the embodiments of the present application, a method for fusion recognition of emotion patterns of EEG signals is disclosed, comprising:

[0089] S1, obtaining an EEG signal training data set; the EEG signal training data set includes an EEG signal sequence and corresponding emotion label information;

[0090] S2, using the EEG signal training data set to train the emotion pattern fusion recognition model to obtain a trained emotion pattern fusion recognition model;

[0091] S3, using the trained emotion pattern fusion recognition model to process the collected EEG signals to obtain emotion pattern recognition results;

[0092] The emotion pattern fusion recognition model includes: a preprocessing module, a frequency domain coding module, a space coding module, a feature fusion module, a time coding module and a classifier module;

[0093] The preprocessing module is used to preprocess the received EEG signal sequence to obtain a spatial signal set and a frequency domain signal set;

[0094] The frequency domain coding module is connected to the preprocessing module and the feature fusion module respectively, and is used to perform frequency domain coding processing on the received frequency domain signal set to obtain a frequency domain coded signal;

[0095] The spatial coding module is connected to the preprocessing module and the feature fusion module respectively, and is used to perform spatial coding processing on the received spatial signal set to obtain a spatially coded signal;

[0096] The feature fusion module is connected to the time coding module and is used to splice the received frequency domain coded signal and the space coded signal in time sequence to obtain a fused coded signal;

[0097] The time coding module is used to perform time coding processing on the received fusion coded signal to obtain a time coded signal;

[0098] The classifier module is connected to the time coding module and is used to classify the received time coding signal to obtain an emotion pattern recognition result.

[0099] The classification processing of the received time-coded signal is to obtain a classification result value after processing the received time-coded signal, and each classification result value has a corresponding emotion pattern recognition result. For example, when the classification result value is 1, the emotion pattern recognition result is happiness; when the classification result value is 2, the emotion pattern recognition result is sadness; when the classification result value is 3, the emotion pattern recognition result is disappointment; when the classification result value is 4, the emotion pattern recognition result is calm; when the classification result value is 5, the emotion pattern recognition result is excitement.

[0100] The preprocessing process of the preprocessing module includes:

[0101] Performing data cleaning processing on the received EEG signal sequence to obtain a first signal sequence;

[0102] performing pattern detection processing on the first signal sequence to obtain a second signal sequence;

[0103] Perform spectrum transformation on the second signal sequence to obtain a frequency domain signal set; the frequency domain signal set X t1 The expression is

[0104] X t1 =[a1, a2, …, a F ],

[0105] Among them, a i is the frequency domain signal of the i-th frequency band, and F is the total number of frequency bands;

[0106] Perform spectrum transformation processing on the signal collected by each electrode in the second signal sequence to obtain a spatial signal set; the spatial signal set X t2 The expression is

[0107]

[0108] Among them, b i is the frequency domain signal of the signal collected by the i-th electrode, T represents the transposition operation, and C is the total number of electrodes;

[0109] The mode detection process comprises:

[0110] For each data of the first signal sequence, the data acquisition information of the data is used as a known independent variable, the data value of the data is used as a known dependent variable, the known independent variable and the known dependent variable are used to construct a curve to be approximated, and the function approximation method is used to perform curve fitting on the curve to be approximated to obtain the best consistent approximation polynomial f(Ix) of the class data attribute; the best consistent approximation polynomial f(Ix) is used to calculate and process the known independent variable to obtain an approximate dependent variable;

[0111] Determine whether the absolute value of the difference between the approximate dependent variable and the corresponding known dependent variable is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the first signal sequence; if it is less than or equal to the first regression discrimination threshold, do not process the data.

[0112] The first regression discrimination threshold value may be 22;

[0113] The frequency domain coding module includes an input module, a mask processing module, a position coding module, a frequency domain encoder, and a frequency domain decoder;

[0114] The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence; the masked sequence has a corresponding signal sequence;

[0115] The position encoding module is connected to the frequency domain encoder and is used to perform position encoding processing on the masked sequence to obtain a first feature sequence;

[0116] The frequency domain encoder is connected to the frequency domain decoder and is used to encode the first feature sequence to obtain a frequency domain coded signal;

[0117] The frequency domain decoder is used to decode the frequency domain coded signal to obtain a frequency domain decoded signal;

[0118] The spatial encoding module includes an input module, a mask processing module, a position encoding module, a spatial encoder, and a spatial decoder;

[0119] The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence;

[0120] The position encoding module is connected to the spatial encoder and is used to perform position encoding processing on the masked sequence to obtain a second feature sequence;

[0121] The spatial encoder is connected to the spatial decoder and is used to encode the second feature sequence to obtain a spatially coded signal;

[0122] The spatial decoder is used to decode the spatially encoded signal to obtain a spatially decoded signal;

[0123] The time encoding module includes an input module, a mask processing module, a position encoding module, a time encoder, and a time decoder;

[0124] The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence;

[0125] The position encoding module is connected to the time encoder and is used to perform position encoding processing on the masked sequence to obtain a third feature sequence;

[0126] The time encoder is connected to the time decoder and is used to encode the third characteristic sequence to obtain a time code signal;

[0127] The time decoder is used to decode the time coded signal to obtain a time decoded signal;

[0128] The training process of the emotion pattern fusion recognition model includes:

[0129] Pre-training the frequency domain coding module, the space coding module, and the time coding module to obtain a pre-trained emotion pattern fusion recognition model;

[0130] Initialize the number of training iterations;

[0131] Using the EEG signal sequence in the EEG signal training data set as input data, inputting it into the pre-trained emotion pattern fusion recognition model to obtain an emotion prediction value;

[0132] Performing difference calculation processing on the emotion label information and the emotion prediction value of the input data to obtain a difference value;

[0133] Determine whether the difference value satisfies a convergence condition, and obtain a first determination result;

[0134] When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result;

[0135] When the second judgment result is no, determining that the model training state does not meet the training termination condition;

[0136] When the second judgment result is yes, determining that the model training state satisfies the training termination condition;

[0137] When the first judgment result is yes, determining that the model training state satisfies a training termination condition;

[0138] When the model training state does not meet the training termination condition, the classifier module is updated with the first parameter update model, the training iteration number is increased by 1, and the execution is triggered to use the EEG signal sequence in the EEG signal training data set as input data;

[0139] When the model training state satisfies the training termination condition, the training process of the pre-trained emotion pattern fusion recognition model is completed to obtain a trained emotion pattern fusion recognition model.

[0140] The pre-training process comprises:

[0141] Initialize the number of pre-training iterations;

[0142] Using the masked sequence as input data, and inputting it into the encoding module to be pre-trained;

[0143] Processing the input data using the encoding module to be pre-trained, and using the output of the decoder contained therein as a prediction signal sequence;

[0144] Performing difference calculation processing on the signal sequence corresponding to the masked sequence and the predicted signal sequence to obtain a difference value;

[0145] Determine whether the difference value satisfies a convergence condition, and obtain a third determination result;

[0146] When the third judgment result is no, judging whether the pre-training iteration count value is equal to the pre-training iteration count threshold, and obtaining a fourth judgment result;

[0147] When the fourth judgment result is no, determining that the model training state does not meet the training termination condition;

[0148] When the fourth judgment result is yes, determining that the model training state satisfies the training termination condition;

[0149] When the third judgment result is yes, determining that the model training state satisfies the training termination condition;

[0150] When the model training state does not meet the training termination condition, the second parameter update model is used to update the parameters of the encoding module to be pre-trained, the pre-training iteration number value is increased by 1, and the masked sequence is used as input data and input into the encoding module to be pre-trained.

[0151] When the model training state satisfies the training termination condition, the pre-training process is completed to obtain a pre-trained encoding module;

[0152] After the pre-training process of the frequency domain coding module, the space coding module and the time coding module is completed, a pre-trained emotion pattern fusion recognition model is obtained.

[0153] The convergence condition refers to that the difference value is less than a preset judgment threshold, and the value of the judgment threshold can be 20.

[0154] The pre-training times threshold is preset and may be 100.

[0155] The difference calculation process between the signal sequence corresponding to the masked sequence and the predicted signal sequence is implemented by using a loss function, and its expression is shown in formula (8).

[0156] The coding module to be pre-trained may be a frequency domain coding module, a spatial coding module, or a temporal coding module.

[0157] The first parameter updating model is:

[0158]

[0159] θ←θ+v,

[0160] In the formula, x (i) is the ith EEG signal sequence in the EEG signal training data set, is the loss function, v is the parameter update value, θ is the parameter of the classifier module, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤1, It means to find the partial derivative of variable θ, f(x (i) ; θ) represents the emotion prediction value calculated by the pre-trained emotion pattern fusion recognition model for the i-th EEG signal sequence of the EEG signal training data set, f(·) is the calculation function corresponding to the pre-trained emotion pattern fusion recognition model; exp represents the power operation of the constant e; η and α are preset values.

[0161] The second parameter update model is:

[0162]

[0163] θ←θ+v,

[0164] Where v is the parameter update value, θ is the parameter of a certain encoding module, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤1, It means to find partial derivatives with respect to variable θ. represents the difference value calculated between the i-th masked sequence and the predicted signal sequence, exp represents the power operation of the constant e; η and α are preset values.

[0165] The masking process is to randomly set some variables in the signal sequence to 0 according to a preset ratio.

[0166] The frequency domain encoder, time encoder and space encoder can be implemented by using the encoder module in Transformer.

[0167] The frequency domain decoder, time decoder and space decoder can be implemented by using the decoder module in Transformer.

[0168] The splicing process can be implemented by using the contact function.

[0169] The classifier module can be implemented using a support vector machine (SVM).

[0170] The values ​​of emotional patterns include happiness, sadness, disappointment, calmness, excitement, etc.

[0171] The curve fitting of the curve to be approximated by the function approximation method may be performed by using the best consistent linear approximation method. The best consistent approximation polynomial f(Ix) is expressed as:

[0172] f(Ix)=α P1 (Ix) P1 +α P1-1 (Ix) P1-1 +…+α2(Ix) 2 +α1(Ix)+α0,

[0173] Where P1 is the order of the best consistent approximation polynomial f(Ix), α0, α1, α2, …, α P1 are the coefficients of the optimal consistent approximation polynomial f(Ix);

[0174] The data collection information of the data includes data collection time information and collection electrode sequence number information;

[0175] The symbols of the formulas in the embodiment of the present invention are shown in Table 1.

[0176] Table 1 Formula symbols

[0177]

[0178] Specifically, the encoding module can be implemented using a Transformer structure. After the Transformer structure is used for implementation, the implementation steps of the method of the present invention include:

[0179] Step 1: First, in module 101, the EEG is divided into several non-overlapping time windows, and these time windows are preprocessed. The EEG signal of each time window is a space-spectrum image, represented by X∈R C×F , where C is the number of EEG electrodes and F is the number of frequency bands. These images are serialized in the spatial and frequency domains and then put into the Transformer encoder for encoding, as shown in Figure 1 .

[0180] Step 1.1: First, collect EEG signals with emotion labels to form a data set. Perform appropriate preprocessing on all EEG signals, including noise removal, artifact correction, and filtering, to ensure data quality and signal accuracy.

[0181] Step 1.2: The EEG of each trial is divided into several non-overlapping 5-second windows. These 5-second EEG windows are all spatial-spectral images, denoted as X∈R C×F , where C is the number of EEG electrodes and F is the frequency domain resolution, see module 101.

[0182] Step 1.3: Spatial serialization. The spatial block method divides the SFR (Spatial Frequency Response) into multiple spatial blocks, each block contains all electrodes. This design utilizes the spatial information in the EEG signal. The spatial serialization method divides the segmented fine spectrum representation of the EEG signal into multiple spatial sequences, each sequence contains all electrodes. These sequences are then used as the sequence input of an EEG trial. This design utilizes the spatial information in the EEG signal. The formula of the spatial serialization method is:

[0183] X t =[a1, a2, …, a F ] (1)

[0184] where a i ∈R C , C is the number of EEG signal electrodes, and t is the window index. The spatial representation is calculated by the spatial Transformer block as follows, see Figure 2 :

[0185]

[0186] Step 1.4: Spectral serialization. In addition to spatial information, spectral information is another important dimension to represent EEG signals. In order to enable Transformer to effectively capture the global emotional characteristics in the frequency domain, the electrode-by-electrode serialization method is adopted. Each block contains complete frequency domain information. The formula of the spectral serialization method is:

[0187]

[0188] where b i ∈R F , T is the transpose operation, and F is the number of frequency bands. The spectral representation is calculated by the spectral Transformer block as follows, see Figure 3 :

[0189]

[0190] Step 2: Put the processed sequence corresponding to each time window into the spatial and frequency domain Transformer encoders for encoding to obtain features, and concatenate the two types of features obtained for the input of the second layer of time Transformer, such as Figure 1 .

[0191] Step 2.1: In module 102, the grouped data is passed through the frequency domain and spatial Transformer blocks to obtain the feature sequences in the spatial and frequency domains for feature concatenation. The formula is as follows:

[0192]

[0193] Step 2.2: The concatenated features are then arranged into sequences according to the time steps and input into the trial-level transformer model as shown in the following formula:

[0194]

[0195] Step 3: Put the concatenated features obtained from the entire trial into the temporal Transformer encoder in chronological order for encoding, and finally use the obtained features for emotion classification, such as Figure 4 Module 103. The temporal representation is computed by the temporal Transformer block as follows:

[0196]

[0197] Step 4: In modules 102 and 103, two encoders are pre-trained using a mask-based self-supervised learning method to improve classification accuracy. Take the spectrum branch as an example (see Figure 2 ), the segmented fine spectral representation of the EEG signal is divided into multiple blocks in the frequency domain to form a sequence. On this sequence, a certain proportion of the sequence is masked using a random selection process. The visible sequence is then used to predict the masked sequence. The loss of the difference between the predicted sequence and the masked sequence is calculated. The loss function of self-supervised learning consists of two parts, as described below:

[0198]

[0199] Step 5: The sequence obtained after two layers of encoding is used for model training in module 104 and finally used for emotion classification, such as Figure 4 The emotion calculation classification formula is as follows:

[0200] Y emo =f(S final ; Θ cls ) (9)

[0201] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for fusion recognition of emotional patterns of EEG signals, characterized in that: include: S1, obtain the EEG signal training data set; The EEG signal training data set includes an EEG signal sequence and corresponding emotion label information; S2, using the EEG signal training data set to train the emotion pattern fusion recognition model to obtain a trained emotion pattern fusion recognition model; S3, using the trained emotion pattern fusion recognition model to process the collected EEG signal to obtain an emotion pattern recognition result.

2. The method for emotional pattern fusion recognition of EEG signals as claimed in claim 1, characterized in that: The emotion pattern fusion recognition model includes: a preprocessing module, a frequency domain coding module, a space coding module, a feature fusion module, a time coding module and a classifier module; The preprocessing module is used to preprocess the received EEG signal sequence to obtain a spatial signal set and a frequency domain signal set; The frequency domain coding module is connected to the preprocessing module and the feature fusion module respectively, and is used to perform frequency domain coding processing on the received frequency domain signal set to obtain a frequency domain coded signal; The spatial coding module is connected to the preprocessing module and the feature fusion module respectively, and is used to perform spatial coding processing on the received spatial signal set to obtain a spatially coded signal; The feature fusion module is connected to the time coding module and is used to splice the received frequency domain coded signal and the space coded signal in time sequence to obtain a fused coded signal; The time coding module is used to perform time coding processing on the received fusion coded signal to obtain a time coded signal; The classifier module is connected to the time coding module and is used to classify the received time coding signal to obtain an emotion pattern recognition result.

3. The method for emotional pattern fusion recognition of EEG signals as claimed in claim 1, characterized in that: The preprocessing process of the preprocessing module includes: Performing data cleaning processing on the received EEG signal sequence to obtain a first signal sequence; performing pattern detection processing on the first signal sequence to obtain a second signal sequence; Perform spectrum transformation on the second signal sequence to obtain a frequency domain signal set; the frequency domain signal set X t1 The expression is X t1 =[a1,a2,…,a F ], Among them, a i is the frequency domain signal of the i-th frequency band, and F is the total number of frequency bands; Perform spectrum transformation processing on the signal collected by each electrode in the second signal sequence to obtain a spatial signal set; the spatial signal set X t2 The expression is Among them, b i is the frequency domain signal of the signal collected by the i-th electrode, T represents the transposition operation, and C is the total number of electrodes.

4. The method for emotional pattern fusion recognition of EEG signals as claimed in claim 1, characterized in that: The mode detection process comprises: For each data of the first signal sequence, data acquisition information of the data is used as a known independent variable, and data values ​​of the data are used as a known dependent variable. A curve to be approximated is constructed by using the known independent variable and the known dependent variable, and a function approximation method is used to perform curve fitting on the curve to be approximated to obtain an optimal consistent approximation polynomial f(Ix) of the class data attribute; Using the best consistent approximation polynomial f(Ix), the known independent variable is calculated to obtain an approximate dependent variable; Determine whether the absolute value of the difference between the approximate dependent variable and the corresponding known dependent variable is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the first signal sequence; if it is less than or equal to the first regression discrimination threshold, do not process the data.

5. The method for emotional pattern fusion recognition of EEG signals as claimed in claim 4, characterized in that: The frequency domain coding module includes an input module, a mask processing module, a position coding module, a frequency domain encoder, and a frequency domain decoder; The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence; The masked sequence has a corresponding signal sequence; The position encoding module is connected to the frequency domain encoder and is used to perform position encoding processing on the masked sequence to obtain a first feature sequence; The frequency domain encoder is connected to the frequency domain decoder and is used to encode the first feature sequence to obtain a frequency domain coded signal; The frequency domain decoder is used to decode the frequency domain coded signal to obtain a frequency domain decoded signal; The spatial encoding module includes an input module, a mask processing module, a position encoding module, a spatial encoder, and a spatial decoder; The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence; The position encoding module is connected to the spatial encoder and is used to perform position encoding processing on the masked sequence to obtain a second feature sequence; The spatial encoder is connected to the spatial decoder and is used to encode the second feature sequence to obtain a spatially coded signal; The spatial decoder is used to decode the spatially encoded signal to obtain a spatially decoded signal; The time encoding module includes an input module, a mask processing module, a position encoding module, a time encoder, and a time decoder; The mask processing module is connected to the input module and the position encoding module, and is used to perform mask processing on the signal sequence received by the input module according to a preset ratio value to obtain a masked sequence; The position encoding module is connected to the time encoder and is used to perform position encoding processing on the masked sequence to obtain a third feature sequence; The time encoder is connected to the time decoder and is used to encode the third characteristic sequence to obtain a time code signal; The time decoder is used to decode the time coded signal to obtain a time decoded signal.

6. The method for emotional pattern fusion recognition of EEG signals as claimed in claim 5, characterized in that: The training process of the emotion pattern fusion recognition model includes: Pre-training the frequency domain coding module, the space coding module, and the time coding module to obtain a pre-trained emotion pattern fusion recognition model; Initialize the number of training iterations; Using the EEG signal sequence in the EEG signal training data set as input data, inputting it into the pre-trained emotion pattern fusion recognition model to obtain an emotion prediction value; Performing difference calculation processing on the emotion label information and the emotion prediction value of the input data to obtain a difference value; Determine whether the difference value satisfies a convergence condition, and obtain a first determination result; When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result; When the second judgment result is no, determining that the model training state does not meet the training termination condition; When the second judgment result is yes, determining that the model training state satisfies the training termination condition; When the first judgment result is yes, determining that the model training state satisfies a training termination condition; When the model training state does not meet the training termination condition, the classifier module is updated with the first parameter update model, the training iteration number is increased by 1, and the execution is triggered to use the EEG signal sequence in the EEG signal training data set as input data; When the model training state satisfies the training termination condition, the training process of the pre-trained emotion pattern fusion recognition model is completed to obtain a trained emotion pattern fusion recognition model.

7. The method for emotional pattern fusion recognition of EEG signals as claimed in claim 6, characterized in that: The pre-training process comprises: Initialize the number of pre-training iterations; Using the masked sequence as input data, and inputting it into the encoding module to be pre-trained; Processing the input data using the encoding module to be pre-trained, and using the output of the decoder contained therein as a prediction signal sequence; Performing difference calculation processing on the signal sequence corresponding to the masked sequence and the predicted signal sequence to obtain a difference value; Determine whether the difference value satisfies a convergence condition, and obtain a third determination result; When the third judgment result is no, judging whether the pre-training iteration count value is equal to the pre-training iteration count threshold, and obtaining a fourth judgment result; When the fourth judgment result is no, determining that the model training state does not meet the training termination condition; When the fourth judgment result is yes, determining that the model training state satisfies the training termination condition; When the third judgment result is yes, determining that the model training state satisfies the training termination condition; When the model training state does not meet the training termination condition, the second parameter update model is used to update the parameters of the encoding module to be pre-trained, the pre-training iteration number value is increased by 1, and the masked sequence is used as input data and input into the encoding module to be pre-trained. When the model training state satisfies the training termination condition, the pre-training process is completed to obtain a pre-trained encoding module; After the pre-training process of the frequency domain coding module, the space coding module and the time coding module is completed, a pre-trained emotion pattern fusion recognition model is obtained.

8. An EEG signal emotion pattern fusion recognition device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the method for emotional pattern fusion and recognition of EEG signals according to any one of claims 1 to 7.

9. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the method for emotional pattern fusion and recognition of EEG signals according to any one of claims 1 to 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the emotion pattern fusion recognition method of the electroencephalogram signal as described in any one of claims 1 to 7.

Images