Self-supervised emotion recognition method based on electroencephalogram signals in virtual reality scene

Abstract

The embodiment of the invention discloses a self-supervised emotion recognition method based on electroencephalogram signals in a virtual reality scene. One specific embodiment of the method comprises the following steps: collecting original full-channel electroencephalogram data, and generating preprocessed full-channel electroencephalogram data; obtaining each differential entropy feature; generating each embedded feature; obtaining a time visible frame feature and a time masking frame feature; obtaining each piece of time code embedded information; obtaining each real two-dimensional coordinate, and generating each spatial position code; obtaining each space visible frame feature and each space masking frame feature; each piece of space code embedding information is obtained; obtaining each space decoding feature; obtaining reconstructed differential entropy features; determining a loss value, and determining a pre-training model; obtaining an emotion predictor; and obtaining an emotion category. According to the embodiment, the emotion recognition capability of the few-channel electroencephalogram data can be enhanced, and the problem that the emotion recognition accuracy is low due to individual differences can be solved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of emotion computing and cognitive rehabilitation based on electroencephalogram (EEG) signals, and in particular to a self-supervised emotion recognition method based on electroencephalogram (EEG) signals in a virtual reality scenario. Background Art

[0002] Affective Brain-Computer Interface (aBCI) allows machines to recognize and regulate human emotions. aBCI technology not only has great potential in treating mental illness, but can also be used as an assessment tool for ordinary people in daily life. aBCI uses multiple modes, including functional magnetic resonance imaging (fMRI), stereo-electroencephalography (SEEG), electroencephalography (EEG), etc., among which EEG signals are non-invasive and relatively easy to collect, while revealing subtle changes in emotions with high temporal resolution. Among all the basic technologies, emotion recognition is the basis of aBCIs and has become a key research topic. Accurately assessing people's emotional state is also helpful for mental health assessment. At the same time, virtual reality scenes provide users with a large number of immersive visual, auditory and other immersive experiences. In virtual scenes, users' emotions are fully mobilized and effectively induced. Therefore, emotion recognition based on EEG stimulated by virtual reality (VR) scenes has developed rapidly in recent years and has attracted extensive attention in the fields of cognitive rehabilitation and VR emotion induction research.

[0003] With the rapid development of deep learning technology, significant progress has been made in decoding EEG data into detailed and lightweight representations using supervised models such as convolutional neural networks, recurrent neural networks, graph convolutional neural networks, and attention mechanisms to achieve accurate emotion recognition. However, these deep learning networks usually require a large amount of manually annotated multi-channel EEG training data, which is expensive and time-consuming. Moreover, multi-channel EEG devices are mainly used in laboratory scenarios. In real scenarios, the inherent fragility and sensitivity to noise of EEG data often lead to data corruption or loss in certain channels. Using only complete few-channel labeled data will seriously damage the classification accuracy achieved by the network. Therefore, these excellent supervised models are difficult to generalize to general daily applications. Therefore, in order to overcome these obstacles in practical applications, the present invention proposes an efficient decoding paradigm for self-supervised learning based on EEG for learning EEG spatiotemporal frequency feature representation. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a self-supervised emotion recognition method based on EEG signals in a virtual reality scene to solve one or more of the technical problems mentioned in the above background technology section.

[0006] The technical problem solved by the present invention is to propose a self-supervised emotion recognition method based on EEG signals in a virtual reality scene, which is used for learning universal representation of EEG features. The framework can be extended to new subjects, and good classification results can be obtained by fine-tuning some channel data, which enhances the emotion recognition performance of EEG data with few channels, effectively solves the problem of individual differences, and provides new ideas for expanding the emotion recognition model to practical applications.

[0007] Some embodiments of the present disclosure provide a self-supervised emotion recognition method based on EEG signals in a virtual reality scene, the method comprising: step (1): collecting original full-channel EEG data of all users, and preprocessing each collected original full-channel EEG data to generate each preprocessed full-channel EEG data; step (2): performing frequency domain feature processing on each of the above preprocessed full-channel EEG data to obtain each differential entropy feature; step (3): generating each embedded feature based on each of the above differential entropy features, and inputting each of the above embedded features into an initial neural network, wherein each of the above embedded features corresponds to each time frame; step (4): for each of the above embedded features, For each embedded feature in the feature, add a position code to each time frame corresponding to the above embedded feature, and based on a preset frame masking ratio, randomly mask the time frame of the preset position code corresponding to the above embedded feature to obtain the time visible frame feature and the time masked frame feature corresponding to the above embedded feature; step (5): input the obtained each time visible frame feature into the time encoder in the initial neural network to obtain each time code embedding information; step (6): project each pre-acquired real three-dimensional coordinate onto a two-dimensional plane to obtain each real two-dimensional coordinate, and generate each spatial position code based on the above each real two-dimensional coordinate; step (7): add each time code in the above each time code embedding information The code embedding information and the above-mentioned spatial position codes are combined into high-dimensional features, and the combined high-dimensional features are spatially masked to obtain each spatial visible frame feature and each spatial masked frame feature; step (8): inputting each of the above-mentioned spatial visible frame features into the spatial encoder in the initial neural network to obtain each spatial code embedding information; step (9): inputting each of the above-mentioned spatial code embedding information and each of the above-mentioned spatial masked frame features into the spatial decoder in the initial neural network to obtain each spatial decoding feature; step (10): inputting each of the above-mentioned spatial decoding features and the corresponding temporal masked frame feature into the temporal decoder in the initial neural network to obtain the reconstructed differential entropy. feature; step (11): determining the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each reconstructed differential entropy feature obtained, and based on a preset number of iterations, looping steps (3) to (11) to obtain the loss value, and determining the initial neural network after the iterative loop as the pre-trained model; step (12): for the target user, based on the calibration data corresponding to the above target user, fine-tuning the pre-trained model to obtain an emotion predictor, and using a linear layer to predict the emotion category, and measuring the classification loss by cross entropy; step (13): using the test data corresponding to the above target user to verify the effectiveness of the above emotion predictor, and obtaining the emotion category corresponding to the above target user.

[0008] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through a self-supervised emotion recognition method based on EEG signals in a virtual reality scene in some embodiments of the present disclosure, good classification results can be obtained by only fine-tuning part of the channel data, the emotion recognition performance of few-channel EEG data can be enhanced, the problem of low emotion recognition accuracy caused by individual differences can be solved, and new ideas are provided for expanding emotion recognition models to practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0010] Figure 1 is a flowchart of some embodiments of a self-supervised emotion recognition method based on EEG signals in a virtual reality scene according to the present disclosure;

[0011] Figure 2 Schematic diagram of preprocessing and feature extraction of raw full-channel EEG data;

[0012] Figure 3 A schematic diagram of brain partitioning based on different neurological prior knowledge;

[0013] Figure 4 Schematic diagram of the generalized pre-training stage based on the spatiotemporal masking strategy;

[0014] Figure 5 Schematic diagram of the personalized fine-tuning and testing phase. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0017] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0018] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0019] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0020] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0021] Figure 1 A process 100 of some embodiments of a self-supervised emotion recognition method based on EEG signals in a virtual reality scene according to the present disclosure is shown. The self-supervised emotion recognition method based on EEG signals in a virtual reality scene comprises the following steps:

[0022] Step 101 , collecting original full-channel EEG data of all users, and preprocessing each collected original full-channel EEG data to generate each preprocessed full-channel EEG data.

[0023] In some embodiments, the original full-channel EEG data of all users can be collected, and each collected original full-channel EEG data can be preprocessed to generate each preprocessed full-channel EEG data. Among them, the above-mentioned original full-channel EEG data can be the brain waves of the user collected through a first preset number of different electrodes. The above-mentioned preprocessed full-channel EEG data can be the original full-channel EEG data after preprocessing. The above-mentioned first preset number can be a specific numerical value. Here, the specific setting of the above-mentioned first preset number is not limited. In practice, the collected original full-channel EEG data can be downsampled, filtered, re-referenced, and independently component analyzed to generate each preprocessed full-channel EEG data.

[0024] Step 102, performing frequency domain feature extraction processing on each pre-processed full-channel EEG data to obtain each differential entropy feature.

[0025] In some embodiments, frequency domain feature extraction processing may be performed on each of the pre-processed full-channel EEG data to obtain each differential entropy feature, wherein the differential entropy feature may be the differential entropy feature corresponding to each of the pre-processed full-channel EEG data in five bands.

[0026] In some optional implementations of some embodiments, frequency domain feature extraction processing may be performed on each of the above-mentioned pre-processed full-channel EEG data to obtain each differential entropy feature through the following steps:

[0027] The five bands in the above-mentioned pre-processed full-channel EEG data are subjected to frequency domain feature extraction processing to obtain various differential entropy features. Among them, the above-mentioned five bands include δ (0.1-3Hz), θ (4-7Hz), α (8-12Hz), β (13-30Hz), and γ (31-47Hz). The δ in δ (0.1-3Hz) is the name of the frequency band of 0.1-3Hz corresponding to the above-mentioned pre-processed full-channel EEG data, and (0.1-3Hz) indicates that the frequency corresponding to the above-mentioned pre-processed full-channel EEG data is 0.1-3Hz. The θ in θ (4-7Hz) is the name of the frequency band of 4-7Hz corresponding to the above-mentioned pre-processed full-channel EEG data, and (4-7Hz) indicates that the frequency corresponding to the above-mentioned pre-processed full-channel EEG data is 4-7Hz. The α in α(8-12Hz) is the name of the frequency band of 8-12Hz corresponding to the above-mentioned preprocessed full-channel EEG data, and (8-12Hz) indicates that the frequency of the above-mentioned preprocessed full-channel EEG data is 8-12Hz. The β in β(13-30Hz) is the name of the frequency band of 13-30Hz corresponding to the above-mentioned preprocessed full-channel EEG data, and (13-30Hz) indicates that the frequency of the above-mentioned preprocessed full-channel EEG data is 13-30Hz. The γ in γ(31-47Hz) is the name of the frequency band of 31-47Hz corresponding to the above-mentioned preprocessed full-channel EEG data, and (31-47Hz) indicates that the frequency of the above-mentioned preprocessed full-channel EEG data is 31-47Hz. The differential entropy feature is the Shannon information entropy-∑ x p(x)log(p(x))dx, generalized to continuous variables, is calculated as follows:

[0028]

[0029] DE represents differential entropy characteristics, p(x) represents the probability density function of the continuous variable x, ∫ represents the integral symbol, a represents the lower limit of integration, b represents the upper limit of integration, dx represents the differential of x, log represents the logarithm with base 2, x represents the independent variable, and the value of x represents the potential value of the brain wave at a specific time point. For a specific length, it approximately follows the Gaussian distribution N(μ, σ 2 )’s preprocessed full-channel EEG data, its differential entropy feature is:

[0030]

[0031] Wherein, DE represents differential entropy feature, ∫ is the integral symbol, -∞ represents the lower limit of integration, ∞ represents the upper limit of integration, dx represents the differential with respect to x, log represents the logarithm with base 2, π is the circumference of the circle, e is a natural constant, x is the independent variable, the value of x is the potential value of the above-mentioned preprocessed full-channel EEG data at the preset time point, μ is the mean, and σ is the standard deviation. The above-mentioned preset time point can be any time point within the time range from the start of collecting the original full-channel EEG data corresponding to the above-mentioned preprocessed full-channel EEG data to the end of the collection.

[0032] Step 103, generating each embedding feature based on each differential entropy feature, and inputting each embedding feature into the initial neural network.

[0033] In some embodiments, each embedded feature can be generated based on the above-mentioned differential entropy features, and each embedded feature can be input into the initial neural network. Among them, each embedded feature in the above-mentioned embedded features corresponds to each time frame. The above-mentioned embedded features can be feature vectors obtained after converting the above-mentioned differential entropy features and embedding the corresponding original full-channel EEG data. The above-mentioned initial neural network can be a neural network that takes each embedded feature as input and takes the reconstructed differential entropy features corresponding to each embedded feature as output. For example, the above-mentioned initial neural network can be a long short-term memory network. The above-mentioned initial neural network can include a time encoder, a spatial encoder, a spatial decoder, a time decoder and a linear layer.

[0034] In some optional implementations of some embodiments, various embedding features may be generated based on the various differential entropy features described above by the following steps:

[0035] The first step is to transform the above differential entropy features to obtain the transformation features. In practice, the above differential entropy features can be transformed by overlapping window technology to obtain the transformation features. The above differential entropy features are represented by X = (x 1 , x 2 , ..., x N )∈R (N×C×F) . R (N×C×F) The R in the formula represents the real number domain, R (N×C×F) The (N×C×F) in the figure represents the dimensions N, C, F, and R. (N×C×F) represents a three-dimensional matrix composed of real numbers with corresponding dimensions N, C, and F. X represents the set of differential entropy features. 1 , x 2 , ..., x N In the equation, x represents the differential entropy feature. 1, 2, and N represent the serial number of the differential entropy feature. 1 , x 2 , ..., x NIt represents the transformation from the first differential entropy feature, the second differential entropy feature to the Nth differential entropy feature. The above transformation features are expressed as Represents a collection of individual conversion characteristics. In Indicates the transformation feature. 1, 2, N indicates the sequence number of the transformation feature. Represents the first transformation feature, the second transformation feature, and the Nth transformation feature. (N×C×T×F) The R in the formula represents the real number domain, R (N×C×T×F) The (N×C×T×F) in the formula indicates that the dimensions of R are N, C, F, and T respectively. (N×C×T×F) Represents a four-dimensional matrix composed of real numbers and the corresponding dimensions are N, C, F and T respectively. C represents the set of electrodes corresponding to the acquisition of the above-mentioned original full-channel EEG data. N is the set of the above-mentioned original full-channel EEG data obtained at various time points. F represents the set of bands when performing frequency domain feature processing. T represents that the overlapping window set when the above-mentioned differential entropy features are transformed by overlapping window technology is T seconds, which means that each transformed feature corresponds to T time frames.

[0036] In the second step, for each of the above-mentioned transformation features, project it into the D-dimensional space through a linear layer to embed the frequency domain information of each transformation feature to obtain each embedded feature. Among them, the above-mentioned frequency domain information can be the original full-channel EEG data corresponding to each transformation feature. The feature dimension of each of the above-mentioned embedded features is C×T×D, and the embedding formula is as follows:

[0037]

[0038] in, In represents the above embedded features, i represents the serial number corresponding to the embedded feature, represents the i-th embedded feature among the above embedded features. (s) is the weight vector of the above linear layer. (s) ∈R (F×D) . R (F×D) The R in the formula represents the real number domain, R (F×D) The (F×D) in the figure represents the dimensions F and D, R (F×D) represents a two-dimensional matrix consisting of real numbers with corresponding dimensions F and D. (s) is the deviation of the above linear layer. b (s) ∈R (D) . R (D) The R in the formula represents the real number domain, (D) represents the dimension D, and R (D)represents a D-dimensional vector consisting of real numbers. s represents the state variable, which is only used to distinguish between before and after embedding. The above-mentioned linear layer can be a network layer that takes the above-mentioned conversion features as input and takes the embedding features corresponding to the above-mentioned conversion features as output. In practice, for each of the above-mentioned conversion features, the dimension F corresponding to the above-mentioned conversion feature can be mapped to the D-dimensional space through the linear layer. When mapping, each band in the above-mentioned F can be mapped to the D-dimensional space. After mapping, each dimension in the D-dimensional space is the new band information obtained after the mapping of each band in the above-mentioned F. D can represent the set of new band information obtained after mapping each band in the above-mentioned F.

[0039] Step 104, for each embedded feature in each embedded feature, add a position code to each time frame corresponding to the embedded feature, and based on a preset frame masking ratio, randomly mask the time frame of the preset position code corresponding to the above embedded feature to obtain the temporally visible frame feature and the temporally masked frame feature corresponding to the above embedded feature.

[0040] In some embodiments, for each of the above-mentioned embedded features, position coding can be added to each time frame corresponding to the above-mentioned embedded feature, and based on a preset frame masking ratio, the time frame of the preset position coding corresponding to the above-mentioned embedded feature can be randomly masked to obtain the time visible frame feature and the time masked frame feature corresponding to the above-mentioned embedded feature. The above-mentioned position coding can be a label for distinguishing each time frame. The above-mentioned frame masking ratio can be the proportion of the time frame to be masked in all time frames. In practice, position coding can be added to each time frame corresponding to the above-mentioned embedded feature through a learnable position coding technology.

[0041] In some optional implementations of some embodiments, the following steps may be performed to add position codes to each time frame corresponding to the above-mentioned embedded feature, and based on a preset frame masking ratio, randomly mask the time frame of the preset position code corresponding to the above-mentioned embedded feature to obtain the temporally visible frame feature and the temporally masked frame feature corresponding to the above-mentioned embedded feature:

[0042] The first step is to add position codes to each time frame corresponding to the above embedded features.

[0043] In the second step, based on a preset frame masking ratio, the time frame encoded at the preset position corresponding to the above embedded feature is randomly masked to obtain the temporally visible frame feature and the temporally masked frame feature corresponding to the above embedded feature. The temporally visible frame feature can be the feature vector corresponding to each unmasked time frame in each time frame corresponding to the above embedded feature. The temporally masked frame feature can be the feature vector corresponding to each masked time frame in each time frame corresponding to the above embedded feature. The above embedded feature is represented as R (C×T×D) The R in the formula represents the real number field, R (C×T×D) The (C×T×D) in the figure indicates that the dimensions are C, T, and D, respectively. (C×T×D) represents a three-dimensional matrix composed of real numbers with corresponding dimensions C, T, and D. The above time-visible frame feature is expressed as represents the above-mentioned time-visible frame feature, i represents the sequence number corresponding to the time-visible frame feature, Represents the features of the i-th visible frame. The R in it represents the field of real numbers. (C×T v ×D) indicates that the dimensions are C, T v , D, Represents a real number with corresponding dimensions C, T v , D is a three-dimensional matrix. The above temporal masked frame feature is expressed as In represents the time masked frame feature, i represents the sequence number corresponding to the time masked frame feature, represents the i-th temporal masked frame feature. The R in it represents the field of real numbers. (C×T m ×D) indicates that the dimensions are C, T m , D, Represents a real number with corresponding dimensions C, T m , D is a three-dimensional matrix. T is the above embedded features The set of corresponding time frames. v is the set of unmasked time frames in T above. m is the set of masked time frames in T above. v ∪T m=T, ∪ is the union symbol. In practice, first, the total number of time frames contained in the set T of time frames corresponding to each embedded feature can be determined. Then, the product of the above-mentioned frame masking ratio and the total number of time frames corresponding to each embedded feature can be determined as the masking number. Then, the time frames corresponding to the above-mentioned masked number can be randomly selected from the set T of the above-mentioned time frames as the respective masked time frames. Then, the respective potential values ​​corresponding to the above-mentioned respective masked time frames in the above-mentioned embedded features can be set to the preset masking values ​​to obtain the temporally visible frame features. Then, the time frames other than the above-mentioned respective masked time frames in the set T of the above-mentioned time frames can be determined as visible time frames. Then, the respective potential values ​​corresponding to the above-mentioned visible time frames in the above-mentioned embedded features can be set to the preset masking values ​​to obtain the temporally visible frame features. Among them, the above-mentioned preset masking value can be a preset value. Here, the specific setting of the above-mentioned preset masking value is not limited.

[0044] Step 105, inputting the obtained features of each temporally visible frame into the temporal encoder in the initial neural network to obtain each temporal encoding embedding information.

[0045] In some embodiments, the obtained time-visible frame features can be input into the time encoder in the initial neural network to obtain each time code embedding information. The above-mentioned time code embedding information can be a feature vector obtained by encoding the above-mentioned time-visible frame features through the time encoder. The above-mentioned time encoder can be an encoder composed of a multi-head self-attention block and a multi-layer perceptron module, taking the above-mentioned time-visible frame features as input and taking the various time code embedding information as output.

[0046] In some optional implementations of some embodiments, the obtained time-visible frame features may be input into the time encoder in the initial neural network through the following steps to obtain each time encoding embedding information:

[0047] The obtained features of each temporally visible frame are input into the temporal encoder in the initial neural network. The formula for obtaining the temporal encoding embedding information is:

[0048]

[0049] Attention() is the attention mechanism, Q represents the query vector corresponding to the above time encoder, K represents the key vector corresponding to the above time encoder. V represents the value vector corresponding to the above time encoder. Softmax() represents the normalized exponential function. K T T in it is the transposition symbol, K represents the key vector corresponding to the above time encoder, K Tis the transpose of the key vector K. TD is the product of character T and character D. The character T represents the dimension of the query vector Q, that is, the number of columns corresponding to the query vector Q. The character D represents the dimension of the key vector K, that is, the number of columns of the key vector K. Each time code embedded information finally output by the time encoder is represented as In represents the time code embedded information, i represents the sequence number corresponding to the time code embedded information, Represents the i-th time code embedding information. The R in it represents the field of real numbers. (C×T v ×D) indicates that the dimensions are C, T v , D, Represents a real number with corresponding dimensions C, T v , a three-dimensional matrix of D.

[0050] Step 106 , projecting the previously acquired real three-dimensional coordinates onto a two-dimensional plane to obtain real two-dimensional coordinates, and generating spatial position codes based on the real two-dimensional coordinates.

[0051] In some embodiments, each pre-acquired real three-dimensional coordinate can be projected onto a two-dimensional plane to obtain each real two-dimensional coordinate, and each spatial position code can be generated based on the above-mentioned real two-dimensional coordinates. Among them, the above-mentioned real three-dimensional coordinates can be the three-dimensional coordinates of each electrode used when collecting the user's original full-channel EEG data. In practice, the original full-channel EEG data can be obtained from a data storage database. The above-mentioned data storage server can be a database for managing and storing original full-channel EEG data. The above-mentioned real two-dimensional coordinates can be the two-dimensional coordinates obtained by projecting the real three-dimensional coordinates onto a two-dimensional plane.

[0052] In some optional implementations of some embodiments, the following steps may be performed to project the pre-acquired real three-dimensional coordinates onto a two-dimensional plane to obtain the real two-dimensional coordinates, and to generate the spatial position codes based on the real two-dimensional coordinates:

[0053] The first step is to project the real three-dimensional coordinates acquired in advance onto a two-dimensional plane to obtain the real two-dimensional coordinates. Among them, each real two-dimensional coordinate in the above-mentioned real two-dimensional coordinates includes a horizontal coordinate and a vertical coordinate. Each real two-dimensional coordinate is represented as R(x, y). The x in R(x, y) represents the horizontal coordinate, and the y represents the vertical coordinate. The R in R(x, y) can be used to refer to the above-mentioned real two-dimensional coordinates. R(x, y) represents a real two-dimensional coordinate with a horizontal coordinate of x and a vertical coordinate of y. In practice, the real three-dimensional coordinates acquired in advance can be projected onto a two-dimensional plane by equidistant azimuthal projection to obtain the real two-dimensional coordinates.

[0054] The second step is to sort the horizontal coordinates included in the above-mentioned real two-dimensional coordinates to obtain a horizontal coordinate sequence. In practice, the horizontal coordinates included in the above-mentioned real two-dimensional coordinates can be sorted in order from large to small to obtain a horizontal coordinate sequence.

[0055] The third step is to sort the ordinates included in the above-mentioned real two-dimensional coordinates to obtain a ordinate sequence. In practice, the ordinates included in the above-mentioned real two-dimensional coordinates can be sorted in order from large to small to obtain a ordinate sequence.

[0056] Step 4: For each horizontal coordinate in the above horizontal coordinate sequence, perform the following steps:

[0057] In the first sub-step, the ordinate corresponding to the abscissa in the ordinate sequence is determined as the ordinate to be encoded. The abscissa can be expressed as x p .x p The x in the equation represents the horizontal coordinate, and p represents the sequence number of the horizontal coordinate x in the above horizontal coordinate sequence. p The above-mentioned ordinate to be encoded can be expressed as y p .y p The y in y represents the ordinate, and p represents the sequence number of the ordinate y in the above ordinate sequence. p In practice, the ordinate in the ordinate sequence with the same sequence number as the horizontal coordinate can be determined as the ordinate to be encoded.

[0058] The second sub-step is to generate the sinusoidal position codes of each abscissa corresponding to the abscissa based on the abscissa, wherein the generating formula is as follows:

[0059]

[0060] Among them, x prepresents the pth horizontal coordinate of the above horizontal coordinate sequence. i represents the index of the feature dimension during encoding, sin represents the sine function, d represents the number of electrodes used when collecting the above original full-channel EEG data, Ω represents a large constant, usually set to 10000, Encode represents encoding, Encode(x p , 2i) represents the horizontal coordinate x p Encode on the feature dimension whose corresponding index i is an even number.

[0061] The third sub-step is to generate the cosine position codes of each horizontal coordinate corresponding to the horizontal coordinate based on the horizontal coordinate, wherein the generation formula is as follows:

[0062]

[0063] Among them, cos represents the cosine function, Encode(x p , 2i+1) represents the horizontal coordinate x p Encode on the feature dimension where the corresponding index i is an odd number.

[0064] The fourth sub-step is to determine the sum of the above-mentioned abscissa sine position codes and the above-mentioned abscissa cosine position codes as the abscissa position code.

[0065] The fifth sub-step is to generate the sinusoidal position codes of each ordinate corresponding to the ordinate to be encoded based on the ordinate to be encoded, wherein the generating formula is as follows:

[0066]

[0067] Among them, y p Indicates the above-mentioned ordinate to be encoded, Encode(y p , 2i) represents the above-mentioned ordinate y to be encoded p Encode on the feature dimension whose corresponding index i is an even number.

[0068] The sixth sub-step is to generate the cosine position codes of each ordinate corresponding to the ordinate to be encoded based on the ordinate to be encoded, wherein the generation formula is as follows:

[0069]

[0070] Among them, Encode(y p , 2i+1) means that the ordinate to be encoded is encoded on the feature dimension where the corresponding index i is an odd number.

[0071] The seventh sub-step is to determine the sum of the sine position codes of the ordinates and the cosine position codes of the ordinates as the ordinate position code corresponding to the ordinate to be encoded.

[0072] In the eighth sub-step, the horizontal coordinate position code and the vertical coordinate position code are connected to obtain a spatial position code, wherein the connection formula is as follows:

[0073] PE x，y，i =CONCAT|Encode(x p ,i),Encode(y p ,i)|,

[0074] Among them, i represents the index of the feature dimension in the encoding, PE x，y，i PE in x, y, i represents the spatial position encoding, and x in x, y, i represents the horizontal coordinate x above. p . The y in x, y, i represents the above-mentioned ordinate y to be encoded p The i in x, y, i represents the index of the feature dimension during encoding. x，y，i Indicates the spatial position encoding of the horizontal coordinate x and the vertical coordinate y when the feature dimension is i. CONCAT represents the concat function. Encode(x p , i) represents the horizontal coordinate x p Encoding on all feature dimensions. Encode(y p , i) represents the ordinate y to be encoded p Encoding across all feature dimensions.

[0075] Step 107, combining each time code embedding information and each spatial position code in the above-mentioned time code embedding information into a high-dimensional feature, and performing spatial masking processing on each combined high-dimensional feature to obtain each spatial visible frame feature and each spatial masked frame feature.

[0076] In some embodiments, each of the time code embedded information and the spatial position codes in the above-mentioned time code embedded information can be combined into a high-dimensional feature, and the combined high-dimensional features can be subjected to spatial masking processing to obtain each spatial visible frame feature and each spatial masked frame feature. The above-mentioned high-dimensional feature can be a feature vector obtained by adding the time code embedded information and the corresponding spatial position code.

[0077] In some optional implementations of some embodiments, each of the time code embedding information in the above-mentioned time code embedding information and the above-mentioned each spatial position code may be combined into a high-dimensional feature through the following steps, and spatial masking processing is performed on each of the combined high-dimensional features to obtain each spatial visible frame feature and each spatial masked frame feature:

[0078] In the first step, for each time code embedded information in the above-mentioned time code embedded information, the following steps are performed:

[0079] In the first sub-step, the time coding embedding information and the spatial position coding are combined into a high-dimensional feature.

[0080] The second sub-step is to perform spatial masking on the high-dimensional features to obtain spatial visible frame features and spatial masked frame features. The spatial visible frame features are expressed as In represents the spatially visible frame feature, i represents the sequence number corresponding to the spatially visible frame feature, represents the i-th spatially visible frame feature. The R in it represents the field of real numbers. (C v ×T v ×D) indicates that the dimensions are C v 、T v , D, represents a real number with corresponding dimensions C v 、T v , D is a three-dimensional matrix. The above spatial masked frame feature is expressed as In represents the spatial masking frame feature, and i represents the sequence number corresponding to the spatial masking frame feature. represents the i-th spatial masked frame feature. The R in it represents the field of real numbers. (C m ×T v ×D) indicates that the dimensions are C m 、T v 、D. represents a real number with dimensions C m 、T v , D is a three-dimensional matrix. C v ∪C m = C, where C represents the set of electrodes used to collect the original full-channel EEG data of all users. v represents the set of electrodes that are not spatially masked in the set of electrodes C. mRepresents the set of electrodes that have been spatially masked in the set C of electrodes mentioned above. ∪ is the union symbol. In practice, all electrodes C can be divided into a preset number of regions based on neurological prior knowledge. Then, the high-dimensional features can be spatially masked by using an intra-regional masking strategy, that is, randomly masking a fixed number of electrodes in each region, to obtain spatially visible frame features and spatially masked frame features. In practice, an inter-regional masking strategy can also be used, that is, randomly masking several fixed regions, in which case the electrodes in the same region are completely masked, to perform spatial masking on the high-dimensional features, to obtain spatially visible frame features and spatially masked frame features. The above intra-regional masking strategy and the above inter-regional masking strategy are randomly used during the pre-training process.

[0081] Step 108, input each spatial visible frame feature into the spatial encoder in the initial neural network to obtain each spatial encoding embedding information.

[0082] In some embodiments, the above-mentioned various spatially visible frame features can be input into the spatial encoder in the initial neural network to obtain various spatially coded embedding information. The above-mentioned spatially coded embedding information can be a feature vector obtained after inputting the above-mentioned various spatially visible frame features into the spatial encoder. The above-mentioned spatial encoder can be an encoder formed by stacking various multi-head spatial attention modules, taking the above-mentioned various spatially visible frame features as input and taking the above-mentioned various spatially coded embedding information as output.

[0083] In some optional implementations of some embodiments, the above-mentioned various spatially visible frame features may be input into the spatial encoder in the initial neural network through the following steps to obtain various spatial encoding embedding information:

[0084] For each of the above-mentioned spatially visible frame features, the above-mentioned spatially visible frame features are input into the spatial encoder in the initial neural network to obtain the spatial encoding embedding information corresponding to the above-mentioned spatially visible frame features, wherein the formula for obtaining the spatial encoding embedding information is:

[0085]

[0086] Attention() is the attention mechanism, Q represents the query vector corresponding to the above spatial encoder, K represents the key vector corresponding to the above spatial encoder, V represents the value vector corresponding to the above spatial encoder, Softmax() represents the normalized exponential function, K T T in it is the transposition symbol, K represents the key vector corresponding to the above spatial encoder, K Tis the transpose of the key vector K. TD is the product of character T and character D. The character T represents the dimension of the query vector Q, that is, the number of columns corresponding to the query vector Q. The character D represents the dimension of the key vector K, that is, the number of columns of the key vector K. Each spatial encoding embedding information finally output by the spatial encoder can be expressed as In represents the spatial coding embedded information, i represents the serial number corresponding to the spatial coding embedded information, represents the i-th spatial encoding embedding information. The R in it represents the field of real numbers. (C v ×T v ×D) indicates that the dimensions are C v 、T v , D, represents a real number with corresponding dimensions C v 、T v , a three-dimensional matrix of D.

[0087] Step 109, input each spatial coding embedding information and each spatial masking frame feature into the spatial decoder in the initial neural network to obtain each spatial decoding feature.

[0088] In some embodiments, the above-mentioned spatial coding embedding information and the above-mentioned spatial masked frame features can be input into the spatial decoder in the initial neural network to obtain various spatial decoding features. The above-mentioned spatial decoding features can be feature vectors obtained after the above-mentioned spatial coding embedding information and the above-mentioned spatial masked frame features are input into the spatial decoder. The above-mentioned spatial decoder can be a decoder composed of various identical multi-head attention layers, with various spatial coding embedding information and the above-mentioned spatial masked frame features as input and various spatial decoding features as output.

[0089] Step 110, input each spatial decoding feature and the corresponding temporal masked frame feature in each spatial decoding feature into the temporal decoder in the initial neural network to obtain a reconstructed differential entropy feature.

[0090] In some embodiments, each of the above-mentioned spatial decoding features and the corresponding temporal masked frame features can be input into the temporal decoder in the initial neural network to obtain a reconstructed differential entropy feature. The reconstructed differential entropy feature can be a differential entropy feature obtained after being processed by the initial neural network. The temporal decoder can be a decoder composed of the same multi-head attention layers, with spatial decoding features and temporal masked frame features as inputs and corresponding temporal decoding features as outputs. The temporal decoding feature can be a feature vector obtained by inputting the above-mentioned spatial decoding feature and the corresponding temporal masked frame feature into the temporal decoder.

[0091] In some optional implementations of some embodiments, each of the spatial decoding features and the corresponding temporal masked frame features in the above-mentioned spatial decoding features may be input into a temporal decoder in the initial neural network through the following steps to obtain a reconstructed differential entropy feature:

[0092] In the first step, for each of the above spatial decoding features, perform the following steps:

[0093] In the first sub-step, the spatial decoding feature and the temporal mask frame feature corresponding to the spatial decoding feature are input into the temporal decoder in the initial neural network to obtain the temporal decoding feature. The temporal decoding feature is represented as In Represents the above-mentioned temporal decoding features. The i in represents the sequence number corresponding to the temporal decoding feature. represents the i-th time decoding feature. The R in represents the field of real numbers. (C×T v ×D) indicates that the dimensions are C, T v 、D. Represents a real number with corresponding dimensions C, T v , a three-dimensional matrix of D.

[0094] In the second sub-step, the above-mentioned time decoding features are input into the linear layer to obtain the reconstructed differential entropy features. The above-mentioned reconstructed differential entropy features are expressed as In represents the above reconstructed differential entropy feature. The i in represents the sequence number corresponding to the reconstructed differential entropy feature. Represents the i-th reconstructed differential entropy feature. R (C×T×F) The R in R represents the real number field. (C×T×F) The (C×T×F) in R indicates that the dimensions are C, T, and F respectively. (C×T×F)represents a three-dimensional matrix composed of real numbers and the corresponding dimensions are C, T, and F. The above linear layer can be a network layer that takes the above time decoding features as input and outputs the reconstructed differential entropy features corresponding to the above time decoding features.

[0095] Step 111, determine the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each reconstructed differential entropy feature obtained, and based on a preset number of iterations, loop through steps 3 to 111 to obtain the loss value, and determine the initial neural network after the iterative loop as a pre-trained model.

[0096] In some embodiments, the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in the obtained reconstructed differential entropy features can be determined, and based on a preset number of iterations, steps 3 to 111 are looped to obtain the loss value, and the initial neural network after the iterative loop is determined as a pre-trained model.

[0097] In some optional implementations of some embodiments, the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each of the obtained reconstructed differential entropy features may be determined by the following steps:

[0098] Based on the above-mentioned reconstructed differential entropy features, the loss values ​​between the above-mentioned reconstructed differential entropy features and the above-mentioned differential entropy features are generated. The formula for generating the loss value is as follows:

[0099]

[0100] Among them, Loss represents the loss value, S i The S in represents the differential entropy feature. i represents the serial number of the differential entropy feature. i represents the i-th differential entropy feature. S * Represents the reconstructed differential entropy feature. i represents the sequence number of the reconstructed differential entropy feature. represents the i-th reconstructed differential entropy feature. N represents the number of the above-mentioned reconstructed differential entropy features, and is also the number of the above-mentioned differential entropy features. ∑ is the accumulation symbol, It means that i increases from 1 to N.

[0101] Step 112, for the target user, based on the calibration data corresponding to the target user, fine-tune the pre-trained model to obtain an emotion predictor, and use a linear layer to predict the emotion category, and measure the classification loss by cross entropy.

[0102] In some embodiments, for the target user, based on the calibration data corresponding to the target user, the pre-trained model can be fine-tuned to obtain an emotion predictor, and a linear layer can be used to predict the emotion category, and the classification loss can be measured by cross entropy. Among them, the target user can be any user who has collected the original full-channel EEG data. The calibration data can be the few-channel EEG data corresponding to the target user. The few-channel EEG data can be brain waves collected using a second preset number of electrodes. Among them, the second preset number can be a value less than the first preset number. Here, the specific setting of the second preset number is not limited. In practice, the Adam optimizer (Adaptive Moment Estimation) can be used to fine-tune the pre-trained model. Among them, the linear layer can be a fully connected layer with the calibration data corresponding to the target user as input and the probability distribution of the calibration data and the preset emotion categories as output. The probability distribution can be used to characterize the probability that the calibration data belongs to the above-mentioned various emotion categories. The emotion category can be a label used to characterize the user's emotions. For example, the emotion category can be "angry", "happy", and "sad".

[0103] Step 113: Use the test data corresponding to the target user to verify the effectiveness of the emotion predictor and obtain the emotion category corresponding to the target user.

[0104] In some embodiments, the test data corresponding to the target user can be used to verify the validity of the emotion predictor and obtain the emotion category corresponding to the target user. The test data can be brain waves different from the original full-channel EEG data collected when collecting the original full-channel EEG data corresponding to the target user.

[0105] Figure 2 The flowchart of obtaining each differential entropy feature is shown in Figure 1. Figure 2 As shown, the original EEG signal is any one of the above-mentioned original full-channel EEG data. EEG signal filtering is the preprocessed full-channel EEG data obtained after downsampling, filtering, re-referencing, and independent component analysis of the above-mentioned original EEG signal. The α, β, and γ bands are used to represent the above-mentioned five bands δ (0.1-3Hz), θ (4-7Hz), α (8-12Hz), β (13-30Hz), and γ (31-47Hz). Frequency domain analysis is the step of performing frequency domain feature extraction processing on the above-mentioned preprocessed full-channel EEG data in the above-mentioned five bands. δ (0.1-3Hz) in the above-mentioned five bands corresponds to Figure 2 The θ (4-7Hz) in the above five bands corresponds to Figure 2Theta waves in the above five bands correspond to α (8-12Hz) Figure 2 The β (13-30Hz) in the above five bands corresponds to Figure 2 The γ (31-47Hz) in the above five bands corresponds to Figure 2 “Gamma waves” in the film.

[0106] Figure 3 A diagram of brain division based on different neurological prior knowledge. According to the international 10-20 system, when using electrodes to collect brain waves, the cerebral cortex can be divided into four main areas: frontal lobe area, occipital lobe area, parietal lobe area and temporal lobe area.

[0107] like Figure 3 As shown, in Figure (a), the above four main areas are further subdivided into 11 sub-areas. Each of the above 11 sub-areas includes each electrode placement mark. The above electrode placement mark can be the name of the electrode. When using electrodes to collect the above original full-channel EEG data, the above electrode placement mark can be used to characterize the placement position of the electrode on the user's scalp. In Figure (b), the above four main areas are further subdivided into 14 sub-areas. Among them, the frontal lobe area is subdivided into 6 sub-areas. The above 6 sub-areas are symmetrical in the frontal lobe area to obtain more discriminative frontal lobe information. Each of the above 14 sub-areas includes each electrode placement mark. In Figure (c), the above four main areas are further subdivided into 17 sub-areas that are symmetrical in the left and right hemispheres. Each of the above 17 sub-areas includes each electrode placement mark. In Figure (d), the above four main areas are further subdivided into 10 sub-areas. Each of the above 10 sub-areas includes each electrode placement mark. The above each electrode placement mark corresponds to Figure 3 The individual characters in .

[0108] Figure 4 FIG. 1 is a flowchart for encoding and decoding the above-mentioned time-visible frame features. Figure 4 As shown, the masked EEG signal can represent the time-visible frame features corresponding to the above-mentioned original full-channel EEG data. The encoder can be the above-mentioned time encoder and the above-mentioned space encoder. The decoder can be the above-mentioned space decoder and the above-mentioned time decoder. The reconstructed EEG signal can be the above-mentioned reconstructed differential entropy features. Figure 4 The δ, θ, α, β, and Y in the figure correspond to the above five bands δ (0.1-3 Hz), θ (4-7 Hz), α (8-12 Hz), β (13-30 Hz), and γ (31-47 Hz).

[0109] Figure 5Flowchart for fine-tuning the pre-trained model and validating the fine-tuned emotion predictor using test data. Figure 5 As shown, in stage 2: personalized calibration, the calibration data is the calibration data corresponding to the target user. Feature extraction and linear layer is the process of fine-tuning the pre-trained model using the calibration data to obtain the emotion predictor. In stage 3: personalized test, the test data is the test data corresponding to the target user. Emotion recognition is the process of using the emotion predictor to perform emotion recognition on the test data. Figure 5 As shown, For the above emotion categories, The corresponding emotion category may be "happy". For the above emotion categories, The corresponding emotion category may be "calm". For the above emotion categories, The corresponding emotion category may be "sad".

[0110] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A self-supervised emotion recognition method based on EEG signals in a virtual reality scene, comprising: Step (1): collecting original full-channel EEG data of all users, and preprocessing each collected original full-channel EEG data to generate each preprocessed full-channel EEG data; Step (2): performing frequency domain feature extraction processing on each of the pre-processed full-channel EEG data to obtain each differential entropy feature; Step (3): generating each embedding feature based on each differential entropy feature, and inputting each embedding feature into an initial neural network, wherein each of the embedding features corresponds to each time frame; Step (4): for each of the embedded features, adding a position code to each time frame corresponding to the embedded feature, and based on a preset frame masking ratio, randomly masking the time frame of the preset position code corresponding to the embedded feature, to obtain a temporally visible frame feature and a temporally masked frame feature corresponding to the embedded feature; Step (5): Input the obtained features of each temporally visible frame into the temporal encoder in the initial neural network to obtain the embedded information of each temporal code; Step (6): projecting each real three-dimensional coordinate obtained in advance onto a two-dimensional plane to obtain each real two-dimensional coordinate, and generating each spatial position code based on the each real two-dimensional coordinate; Step (7): combining each time code embedding information in each time code embedding information and each spatial position code into a high-dimensional feature, and performing spatial masking processing on each combined high-dimensional feature to obtain each spatial visible frame feature and each spatial masked frame feature; Step (8): inputting each of the spatial visible frame features into the spatial encoder in the initial neural network to obtain each of the spatial encoding embedding information; Step (9): inputting each of the spatial coding embedding information and each of the spatial masking frame features into a spatial decoder in the initial neural network to obtain each of the spatial decoding features; Step (10): inputting each of the spatial decoding features and the corresponding temporal masked frame features into the temporal decoder in the initial neural network to obtain a reconstructed differential entropy feature; Step (11): determining a loss value between each reconstructed differential entropy feature and a corresponding differential entropy feature in each of the obtained reconstructed differential entropy features, and looping through steps (3) to (11) based on a preset number of iterations to obtain a loss value, and determining the initial neural network after the iterative loop as a pre-trained model; Step (12): for the target user, fine-tune the pre-trained model based on the calibration data corresponding to the target user to obtain an emotion predictor, and use a linear layer to predict the emotion category, and measure the classification loss by cross entropy; Step (13): Use the test data corresponding to the target user to verify the effectiveness of the emotion predictor and obtain the emotion category corresponding to the target user.

2. The method according to claim 1, wherein: The frequency domain feature extraction process is performed on each of the pre-processed full-channel EEG data to obtain each differential entropy feature, including: Frequency domain feature extraction processing is performed on the five bands in each of the preprocessed full-channel EEG data to obtain various differential entropy features, wherein the five bands include δ (0.1-3 Hz), θ (4-1 Hz), α (8-12 Hz), β (13-30 Hz), and γ (31-47 Hz), δ in δ (0.1-3 Hz) is the name of the frequency band of 0.1-3 Hz corresponding to each of the preprocessed full-channel EEG data, (0.1-3 Hz) indicates that the frequency of each of the preprocessed full-channel EEG data is 0.1-3 Hz, θ in θ (4-7 Hz) is the name of the frequency band of 4-7 Hz corresponding to each of the preprocessed full-channel EEG data, and (4-7 Hz) indicates that the frequency of each of the preprocessed full-channel EEG data is 4-7H z, α in α(8-12Hz) is the name of the frequency band of 8-12Hz corresponding to each preprocessed full-channel EEG data, (8-12Hz) indicates that the frequency of each preprocessed full-channel EEG data is 8-12Hz, β in β(13-30Hz) is the name of the frequency band of 13-30Hz corresponding to each preprocessed full-channel EEG data, (13-30Hz) indicates that the frequency of each preprocessed full-channel EEG data is 13-30Hz, γ in γ(31-47Hz) is the name of the frequency band of 31-47Hz corresponding to each preprocessed full-channel EEG data, (31-47Hz) indicates that the frequency of each preprocessed full-channel EEG data is 31-47Hz, and the differential entropy feature is Shannon information entropy-∑ x p(x)log(p(x))dx, generalized to continuous variables, is calculated as follows: Where DE represents the differential entropy feature, p(x) represents the probability density function of the continuous variable x, x is the independent variable, the value of x is the potential value of the brain wave at a specific time point, ∫ is the integral symbol, a represents the lower limit of the integral, b represents the upper limit of the integral, dx represents the differential of x, log represents the logarithm with base 2, and for a specific length, it is approximately subject to the Gaussian distribution N(μ, σ 2 )’s preprocessed full-channel EEG data, its differential entropy feature is: Wherein, DE represents differential entropy feature, ∫ is the integral symbol, -∞ represents the lower limit of integration, ∞ represents the upper limit of integration, dx represents the differential with respect to x, log represents the logarithm with base 2, π is pi, e is a natural constant, x is an independent variable, the value of x is the potential value of the preprocessed full-channel EEG data at a preset time point, μ is the mean, and σ is the standard deviation.

3. The method according to claim 1, wherein: The generating each embedding feature based on each differential entropy feature includes: Each differential entropy feature is transformed to obtain each transformed feature, wherein each differential entropy feature is represented by X=(x1, x2, ..., x N )∈R (N×C×F) , R (N×C×F) The R in the formula represents the real number domain, R (N×C×F) represents a three-dimensional matrix composed of real numbers, whose dimensions are N, C, F, x1, x2, ..., x N represents each differential entropy feature, and each transformation feature is expressed as Represents each transformation feature, R (N×C×T×F) The R in the formula represents the real number domain, R (N×C×T×F) represents a four-dimensional matrix composed of real numbers, whose dimensions are N, C, F and T. C represents the set of electrodes corresponding to the acquisition of each original full-channel EEG data, N is the set of each original full-channel EEG data obtained at each time point; F represents the set of bands when performing frequency domain feature processing, T represents the overlapping window set to T seconds when the differential entropy features are transformed by overlapping window technology, that is, each transformation feature corresponds to T time frames; For each of the conversion features, a linear layer is used to project it into a D-dimensional space to embed frequency domain information of each conversion feature to obtain each embedded feature, wherein the feature dimension of each of the embedded features is C×T×D, and the embedding formula is as follows: in, represents any one of the embedding features, w (s) is the weight vector, w (s) ∈R (F ×D) , F (F×D) The R in the formula represents the real number domain, R (F×D) represents a two-dimensional matrix composed of real numbers, whose dimensions are F and D, b (s) is the deviation, b (s) ∈R (D) , R (D) The R in the formula represents the real number field, whose dimension is D, R (D) represents a D-dimensional vector consisting of real numbers, and s represents the state variable, which is only used to distinguish before and after embedding.

4. The method according to claim 1, wherein: The adding of position codes to the respective time frames corresponding to the embedded features, and randomly masking the time frames of the preset position codes corresponding to the embedded features based on a preset frame masking ratio, to obtain the temporally visible frame features and the temporally masked frame features corresponding to the embedded features, include: Adding position codes to each time frame corresponding to the embedded features; Based on a preset frame masking ratio, the time frame encoded at the preset position corresponding to the embedded feature is randomly masked to obtain the temporal visible frame feature and the temporal masked frame feature corresponding to the embedded feature, wherein the embedded feature is expressed as R (C×T×D) The R in the formula represents the real number field, R (C×T×D) represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T and D respectively, and the temporal visible frame feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T v and D, the temporal masked frame feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T m and D, T are the embedding features The corresponding set of time frames, T v is the set of unmasked time frames, T m is the set of masked time frames, T v ∪T m =T, ∪ is the union symbol.

5. The method according to claim 1, wherein: The step of inputting the obtained features of each temporally visible frame into the temporal encoder in the initial neural network to obtain each temporal encoding embedding information includes: The obtained features of each temporally visible frame are input into the temporal encoder in the initial neural network, where the formula for obtaining the temporal encoding embedding information is: Attention() is an attention mechanism, Q represents the query vector corresponding to the time encoder, K represents the key vector corresponding to the time encoder, V represents the value vector corresponding to the time encoder, Softmax() represents the normalized exponential function, K represents the T T in it is the transposition symbol, K represents the key vector corresponding to the time encoder, K T is the transpose of the key vector K, TD is the product of character T and character D, the character T represents the dimension of the query vector Q, that is, the number of columns corresponding to the query vector Q, the character D represents the dimension of the key vector K, that is, the number of columns of the key vector K, and each time code embedding information finally output by the time encoder is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T v and D.

6. The method according to claim 3, wherein: The projecting of the pre-acquired real three-dimensional coordinates onto a two-dimensional plane to obtain real two-dimensional coordinates, and generating spatial position codes based on the real two-dimensional coordinates, includes: Projecting each pre-acquired real three-dimensional coordinate onto a two-dimensional plane to obtain each real two-dimensional coordinate, wherein each of the real two-dimensional coordinates includes a horizontal coordinate and a vertical coordinate, and each real two-dimensional coordinate is represented by R (x, y); Sorting the abscissas included in the real two-dimensional coordinates to obtain a sequence of abscissas; Sorting the ordinates included in the real two-dimensional coordinates to obtain a ordinate sequence; For each abscissa in the abscissa sequence, perform the following steps: The ordinate corresponding to the abscissa in the ordinate sequence is determined as the ordinate to be encoded, wherein the abscissa can be expressed as x p , the ordinate to be encoded is represented by y p , p represents the sequence number of the horizontal coordinate in the horizontal coordinate sequence, and also represents the sequence number of the vertical coordinate to be encoded in the vertical coordinate sequence; Based on the abscissa, each abscissa sinusoidal position code corresponding to the abscissa is generated, wherein the generating formula is as follows: Among them, x p represents the horizontal axis, i represents the index of the feature dimension in the encoding, sin represents the sine function, d represents the number of electrodes used when collecting the original full-channel EEG data, Ω represents a large constant, usually set to 10000, Encode represents encoding, Encode(x p , 2i) represents the horizontal coordinate x p Encode on the feature dimension where the corresponding index i is an even number; Based on the abscissa, each abscissa cosine position code corresponding to the abscissa is generated, wherein the generation formula is as follows: Among them, cos represents the cosine function, Encode(x p , 2i+1) represents the horizontal coordinate x p Encode on the feature dimension where the corresponding index i is an odd number; Determine the sum of each of the abscissa sine position codes and each of the abscissa cosine position codes as the abscissa position code; Based on the ordinate to be encoded, the sinusoidal position codes of each ordinate corresponding to the ordinate to be encoded are generated, wherein the generation formula is as follows: Among them, y p Indicates the ordinate to be encoded, Encode(y p , 2i) represents the ordinate y to be encoded p Encode on the feature dimension where the corresponding index i is an even number; Based on the ordinate to be encoded, the cosine position codes of each ordinate corresponding to the ordinate to be encoded are generated, wherein the generation formula is as follows: Among them, Encode(y p , 2i+1) means that the ordinate to be encoded is encoded on the feature dimension where the corresponding index i is an odd number; Determine the sum of the sine position codes of each ordinate and the cosine position codes of each ordinate as the ordinate position code corresponding to the ordinate to be encoded; The horizontal coordinate position code and the vertical coordinate position code are connected to obtain a spatial position code, wherein the connection formula is as follows: PE x，y，i =CONCAT[Encode(x p ,i),Encode(y p ,i)], Among them, i represents the index of the feature dimension in the encoding, PE x，y，i represents the spatial position encoding of the abscissa x and the ordinate y corresponding to the abscissa x when the feature dimension is i, CONCAT represents the concat function, Encode(y p , i) represents the horizontal coordinate x p Encoding on all feature dimensions, Encode(y p , i) represents the ordinate y to be encoded p Encoding across all feature dimensions.

7. The method according to claim 1, wherein: The step of combining each time code embedding information in each time code embedding information and each spatial position code into a high-dimensional feature, and performing spatial masking processing on each combined high-dimensional feature to obtain each spatial visible frame feature and each spatial masked frame feature includes: For each time code embedded information in the respective time code embedded information, the following steps are performed: Combining the time code embedding information and the respective spatial position codes into a high-dimensional feature; The high-dimensional features are subjected to spatial masking processing to obtain spatially visible frame features and spatially masked frame features, wherein the spatially visible frame features are expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C v , T v and D, the spatial masked frame feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C m , T v and D, C v ∪C m = C, C represents the set of electrodes used to collect the original full-channel EEG data of all users, C v represents the set of electrodes without spatial masking, C m represents the set of electrodes that have been spatially masked, and ∪ is the union symbol.

8. The method according to claim 1, wherein: The step of inputting each spatially visible frame feature into a spatial encoder in an initial neural network to obtain each spatially encoded embedding information includes: For each of the spatially visible frame features, the spatially visible frame feature is input into a spatial encoder in the initial neural network to obtain spatial encoding embedding information corresponding to the spatially visible frame feature, wherein the formula for obtaining the spatial encoding embedding information is: Among them, Attention() is the attention mechanism, Q represents the query vector corresponding to the spatial encoder, K represents the key vector corresponding to the spatial encoder, V represents the value vector corresponding to the spatial encoder, Softmax() represents the normalized exponential function, K T Where T is the transposition symbol, TD is the product of character T and character D, the character T represents the dimension of the query vector Q, that is, the number of columns corresponding to the query vector Q, the character D represents the dimension of the key vector K, that is, the number of columns of the key vector K, and each spatial encoding embedding information finally output by the spatial encoder can be expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C v , T v and D.

9. The method according to claim 1, wherein: The step of inputting each of the spatial decoding features and the corresponding temporal mask frame feature into the temporal decoder in the initial neural network to obtain a reconstructed differential entropy feature comprises: For each of the spatial decoding features, the following steps are performed: The spatial decoding feature and the temporal masked frame feature corresponding to the spatial decoding feature are input into the temporal decoder in the initial neural network to obtain the temporal decoding feature, wherein the temporal decoding feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T v and D; The time decoding feature is input into the linear layer to obtain the reconstructed differential entropy feature, where the reconstructed differential entropy feature is expressed as R (C×T×F) The R in the formula represents the real number domain, R (C×T×F) represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T, and F respectively.

10. The method according to claim 1, wherein: The determining of the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each of the obtained reconstructed differential entropy features includes: Based on the respective reconstructed differential entropy features, a loss value between the respective reconstructed differential entropy features and the respective differential entropy features is generated, and the formula for generating the loss value is as follows: Among them, Loss represents the loss value, S i is the differential entropy characteristic, is the reconstructed differential entropy feature, i is the serial number of the differential entropy feature and the reconstructed differential entropy feature, N represents the number of the reconstructed differential entropy features, ∑ is the accumulation symbol, It means that i increases from 1 to N.

Images