A self-supervised emotion recognition method based on electroencephalogram signals in a virtual reality scene

Abstract

Embodiments of the present disclosure disclose a self-supervised emotion recognition method based on electroencephalogram signals in a virtual reality scene. A specific embodiment of the method comprises: collecting original full-channel electroencephalogram data to generate preprocessed full-channel electroencephalogram data; obtaining each differential entropy feature; generating each embedding feature; obtaining time visible frame features and time masking frame features; obtaining each time encoding embedding information; obtaining each real two-dimensional coordinate to generate each spatial position encoding; obtaining each spatial visible frame feature and each spatial masking frame feature; obtaining each spatial encoding embedding information; obtaining each spatial decoding feature; obtaining reconstructed differential entropy features; determining a loss value, determining a pre-training model; obtaining an emotion predictor; and obtaining an emotion category. The embodiment can enhance the emotion recognition capability of the few-channel electroencephalogram data and can solve the problem of low emotion recognition accuracy caused by individual differences.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the fields of affective computing and cognitive rehabilitation based on EEG signals, and more particularly to a self-supervised emotion recognition method based on EEG signals in a virtual reality scenario. Background Art

[0002] Affective Brain-Computer Interfaces (aBCIs) allow machines to recognize and regulate human emotions. aBCI technology not only has great potential in treating mental illnesses but can also serve as an assessment tool for ordinary people in their daily lives. aBCI employs a variety of modalities, including functional magnetic resonance imaging (fMRI), stereo-electroencephalography (SEEG), and electroencephalography (EEG). EEG signals are non-invasive and relatively easy to collect, while revealing subtle changes in emotions with high temporal resolution. Among all the underlying technologies, emotion recognition is the foundation of aBCIs and has become a key research topic. Accurately assessing people's emotional states can also aid in mental health assessments. At the same time, virtual reality scenarios provide users with a wealth of immersive visual, auditory, and other experiences, allowing users' emotions to be fully mobilized and effectively induced in virtual scenes. Therefore, emotion recognition based on EEG stimuli in virtual reality (VR) scenes has developed rapidly in recent years and has attracted widespread attention in the fields of cognitive rehabilitation and VR emotion induction research.

[0003] With the rapid development of deep learning technology, significant progress has been made in decoding EEG data into detailed and lightweight representations using supervised models such as convolutional neural networks, recurrent neural networks, graph convolutional neural networks, and attention mechanisms to achieve accurate emotion recognition. However, these deep learning networks usually require a large amount of manually annotated multi-channel EEG training data, which is expensive and time-consuming. Moreover, multi-channel EEG equipment is mainly used in laboratory scenarios. In real-world scenarios, the inherent fragility and sensitivity of EEG data to noise often lead to data corruption or loss in certain channels. Using only complete, few-channel labeled data will seriously damage the classification accuracy achieved by the network. Therefore, these excellent supervised models are difficult to generalize to general daily applications. Therefore, in order to overcome these obstacles in practical applications, the present invention proposes an EEG-based self-supervised learning efficient decoding paradigm for EEG spatiotemporal-frequency feature representation learning. Summary of the Invention

[0004] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Some embodiments of the present disclosure propose a self-supervised emotion recognition method based on EEG signals in a virtual reality scene to solve one or more of the technical problems mentioned in the above background technology section.

[0006] This invention addresses the technical problem of proposing a self-supervised emotion recognition method based on EEG signals in virtual reality scenarios, which is used to learn universal representations of EEG features. This framework can be generalized to new subjects, achieving good classification results with only minor adjustments to partial channel data. This enhances the emotion recognition performance of low-channel EEG data, effectively addresses individual differences, and provides new insights for expanding emotion recognition models into practical applications.

[0007] Some embodiments of the present disclosure provide a self-supervised emotion recognition method based on EEG signals in a virtual reality scene, the method comprising: step (1): collecting original full-channel EEG data of all users, and pre-processing each collected original full-channel EEG data to generate each pre-processed full-channel EEG data; step (2): performing frequency domain feature processing on each of the above pre-processed full-channel EEG data to obtain each differential entropy feature; step (3): generating each embedded feature based on each of the above differential entropy features, and inputting each of the above embedded features into an initial neural network, wherein each of the above embedded features corresponds to each time frame; step (4): for each of the above embedded features, For each embedded feature in the feature, position coding is added to each time frame corresponding to the above embedded feature, and based on a preset frame masking ratio, the time frame of the preset position coding corresponding to the above embedded feature is randomly masked to obtain the time visible frame feature and the time masked frame feature corresponding to the above embedded feature; step (5): inputting the obtained each time visible frame feature into the time encoder in the initial neural network to obtain each time coding embedding information; step (6): projecting each real three-dimensional coordinate obtained in advance onto a two-dimensional plane to obtain each real two-dimensional coordinate, and generating each spatial position coding based on the above each real two-dimensional coordinate; step (7): adding each time code in the above each time coding embedding information to the time encoder; The code embedding information and the above-mentioned spatial position codes are combined into high-dimensional features, and the combined high-dimensional features are spatially masked to obtain each spatial visible frame feature and each spatial masked frame feature; step (8): inputting each of the above-mentioned spatial visible frame features into the spatial encoder in the initial neural network to obtain each spatial code embedding information; step (9): inputting each of the above-mentioned spatial code embedding information and each of the above-mentioned spatial masked frame features into the spatial decoder in the initial neural network to obtain each spatial decoding feature; step (10): inputting each of the above-mentioned spatial decoding features and the corresponding temporal masked frame feature into the temporal decoder in the initial neural network to obtain the reconstructed differential entropy. feature; step (11): determining the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each reconstructed differential entropy feature obtained, and based on a preset number of iterations, looping steps (3) to (11) to obtain the loss value, and determining the initial neural network after the iterative loop as the pre-trained model; step (12): for the target user, based on the calibration data corresponding to the above target user, fine-tuning the pre-trained model to obtain an emotion predictor, and using a linear layer to predict the emotion category, and measuring the classification loss by cross entropy; step (13): using the test data corresponding to the above target user to verify the effectiveness of the above emotion predictor, and obtaining the emotion category corresponding to the above target user.

[0008] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through a self-supervised emotion recognition method based on EEG signals in a virtual reality scene in some embodiments of the present disclosure, good classification results can be obtained by only fine-tuning part of the channel data, the emotion recognition performance of small-channel EEG data can be enhanced, and the problem of low emotion recognition accuracy caused by individual differences can be solved, providing new ideas for expanding the emotion recognition model to practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0010] Figure 1 is a flowchart of some embodiments of a method for self-supervised emotion recognition based on EEG signals in a virtual reality scene according to the present disclosure;

[0011] Figure 2 Schematic diagram of preprocessing and feature extraction of raw full-channel EEG data;

[0012] Figure 3 Schematic diagram of brain partitioning based on different neurological prior knowledge;

[0013] Figure 4 Schematic diagram of the generalized pre-training stage based on the spatiotemporal masking strategy;

[0014] Figure 5 Schematic diagram of the personalized fine-tuning and testing phase. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0016] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0017] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0018] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0019] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0020] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0021] Figure 1 A process 100 of some embodiments of a self-supervised emotion recognition method based on EEG signals in a virtual reality scene according to the present disclosure is shown. The self-supervised emotion recognition method based on EEG signals in a virtual reality scene includes the following steps:

[0022] Step 101 : collect original full-channel EEG data of all users, and pre-process each collected original full-channel EEG data to generate each pre-processed full-channel EEG data.

[0023] In some embodiments, the original full-channel EEG data of all users can be collected, and the collected original full-channel EEG data can be preprocessed to generate various preprocessed full-channel EEG data. The above-mentioned original full-channel EEG data can be the brain waves of the user collected through a first preset number of different electrodes. The above-mentioned preprocessed full-channel EEG data can be the original full-channel EEG data after preprocessing. The above-mentioned first preset number can be a specific numerical value. Here, there is no limitation on the specific setting of the above-mentioned first preset number. In practice, the collected original full-channel EEG data can be downsampled, filtered, re-referenced, and independently component analyzed to generate various preprocessed full-channel EEG data.

[0024] Step 102 : performing frequency domain feature extraction processing on each pre-processed full-channel EEG data to obtain each differential entropy feature.

[0025] In some embodiments, frequency domain feature extraction processing can be performed on each of the pre-processed full-channel EEG data to obtain each differential entropy feature. The differential entropy feature can be the differential entropy feature corresponding to each of the pre-processed full-channel EEG data in five bands.

[0026] In some optional implementations of some embodiments, frequency domain feature extraction processing may be performed on each of the above-mentioned pre-processed full-channel EEG data to obtain each differential entropy feature through the following steps:

[0027] Frequency domain feature extraction is performed on the five bands in the above-mentioned preprocessed full-channel EEG data to obtain various differential entropy features. Among them, the above-mentioned five bands include δ (0.1-3Hz), θ (4-7Hz), α (8-12Hz), β (13-30Hz), and γ (31-47Hz). The δ in δ (0.1-3Hz) is the name of the frequency band of 0.1-3Hz corresponding to the above-mentioned preprocessed full-channel EEG data, and (0.1-3Hz) indicates that the frequency corresponding to the above-mentioned preprocessed full-channel EEG data is 0.1-3Hz. The θ in θ (4-7Hz) is the name of the frequency band of 4-7Hz corresponding to the above-mentioned preprocessed full-channel EEG data, and (4-7Hz) indicates that the frequency corresponding to the above-mentioned preprocessed full-channel EEG data is 4-7Hz. The α in α(8-12Hz) is the name of the frequency band of 8-12Hz corresponding to each of the above-mentioned preprocessed full-channel EEG data, and (8-12Hz) indicates that the frequency of each of the above-mentioned preprocessed full-channel EEG data is 8-12Hz. The β in β(13-30Hz) is the name of the frequency band of 13-30Hz corresponding to each of the above-mentioned preprocessed full-channel EEG data, and (13-30Hz) indicates that the frequency of each of the above-mentioned preprocessed full-channel EEG data is 13-30Hz. The γ in Y(31-47Hz) is the name of the frequency band of 31-47Hz corresponding to each of the above-mentioned preprocessed full-channel EEG data, and (31-47Hz) indicates that the frequency of each of the above-mentioned preprocessed full-channel EEG data is 31-47Hz. The differential entropy feature is the Shannon information entropy-∑ x p(x)log(p(x))dx, a generalized form for continuous variables, is calculated as follows:

[0028]

[0029] DE represents the differential entropy feature, p(x) represents the probability density function of the continuous variable x, ∫ represents the integral symbol, a represents the lower limit of integration, b represents the upper limit of integration, dx represents the differential of x, log represents the logarithm with base 2, x is the independent variable, and the value of x is the potential value of the brain wave at a specific time point. For a specific length, it is approximately subject to the Gaussian distribution N(μ, σ 2 )’s preprocessed full-channel EEG data, its differential entropy feature is:

[0030]

[0031] Where DE represents the differential entropy feature, ∫ is the integral symbol, -∞ represents the lower limit of integration, ∞ represents the upper limit of integration, dx represents the differential with respect to x, log represents the logarithm with base 2, π is the circumference of the circle, e is a natural constant, x is the independent variable, the value of x is the potential value of the above-mentioned preprocessed full-channel EEG data at the preset time point, μ is the mean, and σ is the standard deviation. The above-mentioned preset time point can be any time point within the time range from the start of acquisition of the original full-channel EEG data corresponding to the above-mentioned preprocessed full-channel EEG data to the end of acquisition.

[0032] Step 103: Generate each embedding feature based on each differential entropy feature, and input each embedding feature into the initial neural network.

[0033] In some embodiments, each embedded feature can be generated based on the above-mentioned differential entropy features, and each embedded feature can be input into the initial neural network. Each of the above-mentioned embedded features corresponds to a time frame. The above-mentioned embedded features can be feature vectors obtained by converting the above-mentioned differential entropy features and embedding them into the corresponding original full-channel EEG data. The above-mentioned initial neural network can be a neural network that takes each embedded feature as input and takes the reconstructed differential entropy features corresponding to each embedded feature as output. For example, the above-mentioned initial neural network can be a long short-term memory network. The above-mentioned initial neural network can include a time encoder, a spatial encoder, a spatial decoder, a time decoder and a linear layer.

[0034] In some optional implementations of some embodiments, each embedding feature may be generated based on the above-mentioned differential entropy features through the following steps:

[0035] The first step is to transform the above differential entropy features to obtain the transformed features. In practice, the above differential entropy features can be transformed by overlapping window technology to obtain the transformed features. The above differential entropy features are represented by X = (x1, x2, ..., x N )∈R (N×C×F) . R (N×C×F) The R in the formula represents the real number field, R (N×C×F) (N×C×F) represents the dimensions N, C, F, R (N×C×F) represents a three-dimensional matrix composed of real numbers with corresponding dimensions N, C, and F. X represents the set of differential entropy features. x1, x2, ..·, x N In the equation, x represents the differential entropy feature. 1, 2, and N represent the sequence number of the differential entropy feature. x1, x2, ..·, x N Represents the transformation from the first differential entropy feature, the second differential entropy feature to the Nth differential entropy feature. The above transformation features are expressed as Represents a collection of individual conversion characteristics. in Indicates the transformation feature. 1, 2, and N are the sequence numbers of the transformation features. Represents the transformation from the first transformation feature, the second transformation feature to the Nth transformation feature. (N×C×T×F) The R in the formula represents the real number field, R (N×C×T×F) (N×C×T×F) in the formula indicates that the dimensions of R are N, C, F, and T respectively. (N×C×T×F) Represents a four-dimensional matrix composed of real numbers with corresponding dimensions N, C, F and T respectively. C represents the set of electrodes corresponding to the acquisition of each of the above-mentioned original full-channel EEG data. N is the set of each of the above-mentioned original full-channel EEG data obtained at each time point. F represents the set of bands when performing frequency domain feature processing. T represents the overlapping window set to T seconds when transforming each of the above-mentioned differential entropy features using the overlapping window technology, which means that each transformed feature corresponds to T time frames.

[0036] In the second step, for each of the above-mentioned transformation features, it is projected into a D-dimensional space through a linear layer to embed the frequency domain information of each transformation feature to obtain each embedded feature. The above-mentioned frequency domain information can be the original full-channel EEG data corresponding to each transformation feature. The feature dimension of each of the above-mentioned embedded features is C×T×D, and the embedding formula is as follows:

[0037]

[0038] in, in represents the above embedded features, i represents the serial number corresponding to the embedded feature, represents the i-th embedded feature among the above embedded features. (s) is the weight vector of the above linear layer. (s) ∈R (F×D) . R (F×D) The R in the formula represents the real number field, R (F×D) (F×D) represents the dimensions F and D, R (F×D) represents a two-dimensional matrix composed of real numbers with corresponding dimensions F and D. (s) is the deviation of the above linear layer. b (s) ∈R (D) . R (D) R in the formula represents the real number field, (D) represents the dimension D, and R (D) Represents a D-dimensional vector consisting of real numbers. s represents the state variable, which is only used to distinguish before and after embedding. The linear layer can be a network layer that takes the aforementioned transformed features as input and outputs the embedded features corresponding to the transformed features. In practice, for each of the aforementioned transformed features, the dimension F corresponding to the transformed feature can be mapped to a D-dimensional space via a linear layer. During this mapping, each band in F can be mapped to the D-dimensional space. After mapping, each dimension in the D-dimensional space represents the new band information obtained by mapping each band in F. D can represent the set of new band information obtained by mapping each band in F.

[0039] Step 104: For each embedded feature in each embedded feature, add a position code to each time frame corresponding to the embedded feature, and based on a preset frame masking ratio, randomly mask the time frame of the preset position code corresponding to the above embedded feature to obtain the temporally visible frame feature and the temporally masked frame feature corresponding to the above embedded feature.

[0040] In some embodiments, for each of the above-mentioned embedded features, position coding can be added to each time frame corresponding to the above-mentioned embedded feature, and based on a preset frame masking ratio, the time frame of the preset position coding corresponding to the above-mentioned embedded feature can be randomly masked to obtain the time visible frame feature and the time masked frame feature corresponding to the above-mentioned embedded feature. The above-mentioned position coding can be a label used to distinguish each time frame. The above-mentioned frame masking ratio can be the proportion of the time frame to be masked in all time frames. In practice, position coding can be added to each time frame corresponding to the above-mentioned embedded feature through a learnable position coding technology.

[0041] In some optional implementations of some embodiments, the following steps may be performed to add position codes to each time frame corresponding to the embedded feature, and randomly mask the time frames with preset position codes corresponding to the embedded feature based on a preset frame masking ratio, to obtain temporally visible frame features and temporally masked frame features corresponding to the embedded feature:

[0042] The first step is to add position encoding to each time frame corresponding to the above embedded features.

[0043] In the second step, based on the preset frame masking ratio, the time frame of the preset position encoding corresponding to the above embedded feature is randomly masked to obtain the temporal visible frame feature and temporal masked frame feature corresponding to the above embedded feature. The temporal visible frame feature can be the feature vector corresponding to each time frame that is not masked in each time frame corresponding to the above embedded feature. The temporal masked frame feature can be the feature vector corresponding to each time frame that is masked in each time frame corresponding to the above embedded feature. The above embedded feature is represented as R (C×T×D) The R in the formula represents the real number field, R(C ×T×D) (C×T×D) in the figure indicates that the dimensions are C, T, and D, respectively. (C×T×D) Represents a three-dimensional matrix composed of real numbers and corresponding dimensions are C, T, and D. The above time visible frame feature is expressed as represents the above-mentioned time-visible frame feature, i represents the sequence number corresponding to the time-visible frame feature, Represents the features of the i-th visible frame. The R in represents the field of real numbers. (C×T v ×D) indicates that the dimensions are C, T v , D, Represents a real number with corresponding dimensions C, T v , D is a three-dimensional matrix. The above temporal masked frame feature is expressed as in represents the time mask frame feature, i represents the sequence number corresponding to the time mask frame feature, represents the i-th temporal masked frame feature. The R in represents the field of real numbers. (C×T m ×D) indicates that the dimensions are C, T m , D, Represents a real number with corresponding dimensions C, T m , D is a three-dimensional matrix. T is the above embedded features The set of corresponding time frames. v is the set of unmasked time frames in T above. m is the set of masked time frames in T above. v ∪T m =T, ∪ is the union symbol. In practice, first, the total number of time frames contained in the set T of time frames corresponding to each embedded feature can be determined. Then, the product of the above-mentioned frame masking ratio and the total number of time frames corresponding to each embedded feature can be determined as the masking number. Then, the time frames corresponding to the above-mentioned masked number can be randomly selected from the set T of time frames as the respective masked time frames. Then, the respective potential values ​​corresponding to the above-mentioned masked time frames in the above-mentioned embedded features can be set as preset masking values ​​to obtain the temporally visible frame feature. Then, the time frames other than the above-mentioned masked time frames in the set T of time frames can be determined as visible time frames. Then, the respective potential values ​​corresponding to the above-mentioned visible time frames in the above-mentioned embedded features can be set as preset masking values ​​to obtain the temporally visible frame feature. Among them, the above-mentioned preset masking value can be a pre-set value. Here, there is no limitation on the specific setting of the above-mentioned preset masking value.

[0044] Step 105: Input the obtained features of each temporally visible frame into the temporal encoder in the initial neural network to obtain each temporal encoding embedding information.

[0045] In some embodiments, the obtained temporally visible frame features can be input into a temporal encoder in an initial neural network to obtain temporal code embedding information. The temporal code embedding information can be a feature vector obtained by encoding the temporally visible frame features via the temporal encoder. The temporal encoder can be composed of a multi-head self-attention block and a multi-layer perceptron module, taking the temporally visible frame features as input and outputting the temporal code embedding information.

[0046] In some optional implementations of some embodiments, the obtained temporally visible frame features may be input into a temporal encoder in an initial neural network through the following steps to obtain temporal encoding embedding information:

[0047] The obtained features of each temporally visible frame are input into the temporal encoder in the initial neural network. The formula for obtaining the temporal encoding embedding information is:

[0048]

[0049] Attention() is the attention mechanism, Q represents the query vector corresponding to the above time encoder, K represents the key vector corresponding to the above time encoder. V represents the value vector corresponding to the above time encoder. Softmax() represents the normalized exponential function. K T T is the transpose symbol, K represents the key vector corresponding to the above time encoder, K T is the transpose of the key vector K. TD is the product of character T and character D, where character T represents the dimension of the query vector Q, i.e., the number of columns corresponding to the query vector Q. The character D represents the dimension of the key vector K, i.e., the number of columns of the key vector K. Each time code embedded information finally output by the time encoder is represented as in Indicates the time code embedded information, i indicates the serial number corresponding to the time code embedded information, represents the i-th time code embedding information. The R in represents the field of real numbers. (C×T v ×D) indicates that the dimensions are C, T v , D, Represents a real number with corresponding dimensions C, T v , a three-dimensional matrix of D.

[0050] Step 106 : Projecting the previously acquired real three-dimensional coordinates onto a two-dimensional plane to obtain real two-dimensional coordinates, and generating spatial position codes based on the real two-dimensional coordinates.

[0051] In some embodiments, each pre-acquired real three-dimensional coordinate can be projected onto a two-dimensional plane to obtain each real two-dimensional coordinate, and based on the above-mentioned each real two-dimensional coordinate, each spatial position code is generated. The above-mentioned real three-dimensional coordinates can be the three-dimensional coordinates of each electrode used to collect the user's original full-channel EEG data. In practice, the original full-channel EEG data can be obtained from a data storage database. The above-mentioned data storage server can be a database for managing and storing original full-channel EEG data. The above-mentioned real two-dimensional coordinates can be the two-dimensional coordinates obtained by projecting the real three-dimensional coordinates onto a two-dimensional plane.

[0052] In some optional implementations of some embodiments, the following steps may be performed to project the pre-acquired real three-dimensional coordinates onto a two-dimensional plane to obtain the real two-dimensional coordinates, and to generate the spatial position codes based on the real two-dimensional coordinates:

[0053] The first step is to project the real three-dimensional coordinates obtained in advance onto a two-dimensional plane to obtain the real two-dimensional coordinates. Among them, each real two-dimensional coordinate in the above-mentioned real two-dimensional coordinates includes a horizontal coordinate and a vertical coordinate. Each real two-dimensional coordinate is expressed as R(x, y). The x in R(x, y) represents the horizontal coordinate, and the y represents the vertical coordinate. The R in R(x, y) can be used to refer to the above-mentioned real two-dimensional coordinates. R(x, y) represents a real two-dimensional coordinate with a horizontal coordinate of x and a vertical coordinate of y. In practice, the real three-dimensional coordinates obtained in advance can be projected onto a two-dimensional plane by equidistant azimuthal projection to obtain the real two-dimensional coordinates.

[0054] The second step is to sort the horizontal coordinates included in the above-mentioned real two-dimensional coordinates to obtain a horizontal coordinate sequence. In practice, the horizontal coordinates included in the above-mentioned real two-dimensional coordinates can be sorted in descending order to obtain the horizontal coordinate sequence.

[0055] The third step is to sort the ordinates included in the above-mentioned real two-dimensional coordinates to obtain a ordinate sequence. In practice, the ordinates included in the above-mentioned real two-dimensional coordinates can be sorted in descending order to obtain a ordinate sequence.

[0056] Step 4: For each horizontal coordinate in the above horizontal coordinate sequence, perform the following steps:

[0057] The first sub-step is to determine the ordinate in the ordinate sequence corresponding to the abscissa as the ordinate to be encoded. The abscissa can be expressed as x p .x p The x in the equation represents the horizontal coordinate, and p represents the number of the horizontal coordinate x in the above horizontal coordinate sequence. p The above-mentioned ordinate sequence to be encoded can be expressed as y p .y p The y in y represents the ordinate, and p represents the sequence number of the ordinate y in the above ordinate sequence. p In practice, the vertical coordinate in the vertical coordinate sequence with the same sequence number p as the horizontal coordinate can be determined as the vertical coordinate to be encoded.

[0058] The second sub-step is to generate the sinusoidal position codes of the abscissas corresponding to the abscissas based on the abscissas, wherein the generation formula is as follows:

[0059]

[0060] Among them, x p represents the pth horizontal coordinate of the above horizontal coordinate sequence. i represents the index of the feature dimension during encoding, sin represents the sine function, d represents the number of electrodes used when collecting the above raw full-channel EEG data, Ω represents a large constant, usually set to 10000, Encode represents encoding, Encode(x p , 2i) represents the horizontal coordinate x p Encoding is performed on the feature dimension where the corresponding index i is an even number.

[0061] The third sub-step is to generate the cosine position codes of the abscissas corresponding to the abscissas based on the abscissas, wherein the generation formula is as follows:

[0062]

[0063] Among them, cos represents the cosine function, Encode(x p , 2i+1) represents the horizontal coordinate x p Encode on the feature dimension where the corresponding index i is an odd number.

[0064] The fourth sub-step is to determine the sum of the above-mentioned abscissa sine position codes and the above-mentioned abscissa cosine position codes as the abscissa position code.

[0065] The fifth sub-step is to generate the sinusoidal position codes of the respective ordinates corresponding to the ordinates to be encoded based on the ordinates to be encoded, wherein the generating formula is as follows:

[0066]

[0067] Among them, y p Indicates the above-mentioned ordinate to be encoded, Encode(y p , 2i) represents the above-mentioned ordinate y to be encoded p Encoding is performed on the feature dimension where the corresponding index i is an even number.

[0068] The sixth sub-step is to generate the cosine position codes of the respective ordinates corresponding to the ordinates to be encoded based on the ordinates to be encoded, wherein the generation formula is as follows:

[0069]

[0070] Among them, Encode(y p , 2i+1) means that the ordinate to be encoded is encoded on the feature dimension where the corresponding index i is an odd number.

[0071] In a seventh sub-step, the sum of the sine position codes of the respective ordinates and the cosine position codes of the respective ordinates is determined as the ordinate position code corresponding to the ordinate to be encoded.

[0072] In the eighth sub-step, the horizontal coordinate position code and the vertical coordinate position code are concatenated to obtain a spatial position code, wherein the concatenation formula is as follows:

[0073] PE x，y，i =CONCAT[Encode(x p ,i),Encode(y p , i)],

[0074] Among them, i represents the index of the feature dimension in the encoding, PE x，y，i PE in x, y, i represents the spatial position code, and x in x, y, i represents the horizontal coordinate x above. p The y in x,y,i represents the vertical coordinate y to be encoded above. p The i in x, y, i represents the index of the feature dimension during encoding. x，y，i Indicates the spatial position encoding of the horizontal coordinate x and the vertical coordinate y when the feature dimension is i. CONCAT represents the concat function. Encode(x p , i) represents the horizontal coordinate x p Encoding on all feature dimensions. Encode(y p , i) represents the ordinate y to be encoded p Encoding across all feature dimensions.

[0075] Step 107: Combine each time code embedding information and each spatial position code in the above-mentioned time code embedding information into a high-dimensional feature, and perform spatial masking processing on each combined high-dimensional feature to obtain each spatial visible frame feature and each spatial masked frame feature.

[0076] In some embodiments, each of the time code embedded information and the spatial position codes may be combined into a high-dimensional feature, and spatial masking may be performed on the combined high-dimensional features to obtain spatially visible frame features and spatially masked frame features. The high-dimensional feature may be a feature vector obtained by adding the time code embedded information and the corresponding spatial position code.

[0077] In some optional implementations of some embodiments, each time code embedding information in the above-mentioned time code embedding information and the above-mentioned spatial position codes can be combined into high-dimensional features through the following steps, and spatial masking processing is performed on the combined high-dimensional features to obtain each spatial visible frame feature and each spatial masked frame feature:

[0078] In the first step, for each time code embedded information in the above time code embedded information, the following steps are performed:

[0079] In the first sub-step, the time code embedding information and the spatial position codes are combined into high-dimensional features.

[0080] The second sub-step is to perform spatial masking on the above high-dimensional features to obtain spatial visible frame features and spatial masked frame features. The above spatial visible frame features are represented as in Represents the spatially visible frame feature, i represents the sequence number corresponding to the spatially visible frame feature, represents the i-th spatially visible frame feature. The R in represents the field of real numbers. (C v ×T v ×D) indicates that the dimensions are C v 、T v , D, Represents a real number with corresponding dimensions C v 、T v , D is a three-dimensional matrix. The above spatial masked frame feature is expressed as in represents the spatial masking frame feature, and i represents the sequence number corresponding to the spatial masking frame feature. represents the i-th spatial masked frame feature.

[0081] The R in represents the field of real numbers. (C m ×T v ×D) indicates that the dimensions are C m 、T v 、D. Represents a real number with dimensions C m 、T v , D is a three-dimensional matrix. C v ∪ C m = C, where C represents the set of electrodes used to collect the original full-channel EEG data of all users. v Indicates the set of electrodes that are not spatially masked in the set of electrodes C. m Represents the set of electrodes that have been spatially masked in the set C of electrodes mentioned above. ∪ is the union symbol. In practice, all electrodes C can be divided into a preset number of regions based on neurological prior knowledge. Then, the above high-dimensional features can be spatially masked by using an intra-regional masking strategy, that is, randomly masking a fixed number of electrodes in each region, to obtain spatially visible frame features and spatially masked frame features. In practice, an inter-regional masking strategy can also be used, that is, randomly masking a few fixed regions. At this time, the electrodes in the same region are completely masked to perform spatial masking on the above high-dimensional features, to obtain spatially visible frame features and spatially masked frame features. The above intra-regional masking strategy and the above inter-regional masking strategy are randomly used during the pre-training process.

[0082] In step 108, each spatial visible frame feature is input into the spatial encoder in the initial neural network to obtain each spatial coding embedding information.

[0083] In some embodiments, the aforementioned spatially visible frame features can be input into a spatial encoder in an initial neural network to obtain spatial code embedding information. The spatial code embedding information can be a feature vector obtained by inputting the aforementioned spatially visible frame features into the spatial encoder. The spatial encoder can be an encoder composed of a stack of multi-head spatial attention modules, taking the aforementioned spatially visible frame features as input and outputting the aforementioned spatial code embedding information.

[0084] In some optional implementations of some embodiments, the above-mentioned spatially visible frame features may be input into the spatial encoder in the initial neural network through the following steps to obtain spatial encoding embedding information:

[0085] For each of the above spatially visible frame features, the above spatially visible frame feature is input into the spatial encoder in the initial neural network to obtain spatial encoding embedding information corresponding to the above spatially visible frame feature. The formula for obtaining the spatial encoding embedding information is:

[0086]

[0087] Attention() is the attention mechanism, Q represents the query vector corresponding to the above spatial encoder, K represents the key vector corresponding to the above spatial encoder, V represents the value vector corresponding to the above spatial encoder, Softmax() represents the normalized exponential function, K T T is the transpose symbol, K represents the key vector corresponding to the above spatial encoder, K T is the transpose of the key vector K. TD is the product of character T and character D. The character T represents the dimension of the query vector Q, that is, the number of columns corresponding to the query vector Q. The character D represents the dimension of the key vector K, that is, the number of columns of the key vector K. Each spatial encoding embedding information finally output by the spatial encoder can be expressed as in Represents the spatial coding embedded information, i represents the serial number corresponding to the spatial coding embedded information, represents the i-th spatial encoding embedding information. The R in represents the field of real numbers. (C v ×T v ×D) indicates that the dimensions are C v 、T v , D, Represents a real number with corresponding dimensions C v 、T v , a three-dimensional matrix of D.

[0088] Step 109: Input each spatial coding embedding information and each spatial masking frame feature into the spatial decoder in the initial neural network to obtain each spatial decoding feature.

[0089] In some embodiments, the aforementioned spatial encoding embedding information and the aforementioned spatial masked frame features can be input into a spatial decoder in an initial neural network to obtain respective spatial decoding features. The aforementioned spatial decoding features can be feature vectors obtained by inputting the aforementioned spatial encoding embedding information and the aforementioned spatial masked frame features into a spatial decoder. The aforementioned spatial decoder can be a decoder composed of identical multi-head attention layers, taking the respective spatial encoding embedding information and the aforementioned spatial masked frame features as input and outputting the respective spatial decoding features.

[0090] In step 110 , each spatial decoding feature and the corresponding temporal mask frame feature are input into a temporal decoder in the initial neural network to obtain a reconstructed differential entropy feature.

[0091] In some embodiments, each of the above-mentioned spatial decoding features and the corresponding temporal masked frame features can be input into a temporal decoder in an initial neural network to obtain a reconstructed differential entropy feature. The reconstructed differential entropy feature can be a differential entropy feature obtained after processing by the initial neural network. The temporal decoder can be a decoder composed of identical multi-head attention layers, taking spatial decoding features and temporal masked frame features as input and corresponding temporal decoding features as output. The temporal decoding feature can be a feature vector obtained by inputting the above-mentioned spatial decoding feature and the corresponding temporal masked frame feature into the temporal decoder.

[0092] In some optional implementations of some embodiments, each of the above spatial decoding features and the corresponding temporal mask frame feature may be input into a temporal decoder in the initial neural network through the following steps to obtain a reconstructed differential entropy feature:

[0093] In the first step, for each of the above spatial decoding features, perform the following steps:

[0094] In the first sub-step, the spatial decoding features and the temporal mask frame features corresponding to the spatial decoding features are input into the temporal decoder in the initial neural network to obtain the temporal decoding features. The temporal decoding features are represented as in Represents the above-mentioned temporal decoding features. The i in represents the sequence number corresponding to the temporal decoding feature. represents the i-th temporal decoding feature. The R in represents the field of real numbers. (C×T v ×D) indicates that the dimensions are C, T v 、D. Represents a real number with corresponding dimensions C, T v , a three-dimensional matrix of D.

[0095] In the second sub-step, the above-mentioned time decoding features are input into the linear layer to obtain the reconstructed differential entropy features. The above-mentioned reconstructed differential entropy features are expressed as in represents the above reconstructed differential entropy feature. The i in represents the sequence number corresponding to the reconstructed differential entropy feature. Represents the i-th reconstructed differential entropy feature. R (C×T×F) The R in R represents the real number field. (C×T×F) The (C×T×F) in R represents the dimensions C, T, and F respectively. (C×T×F) represents a three-dimensional matrix composed of real numbers with corresponding dimensions C, T, and F. The above-mentioned linear layer can be a network layer that takes the above-mentioned temporal decoding features as input and outputs the reconstructed differential entropy features corresponding to the above-mentioned temporal decoding features.

[0096] Step 111, determining the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each reconstructed differential entropy feature obtained, and looping steps 3 to 111 based on a preset number of iterations to obtain the loss value, and determining the initial neural network after the iterative loop as a pre-trained model.

[0097] In some embodiments, the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in the obtained reconstructed differential entropy features can be determined, and based on a preset number of iterations, steps 3 to 111 are looped to obtain the loss value, and the initial neural network after the iterative loop is determined as a pre-trained model.

[0098] In some optional implementations of some embodiments, the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each obtained reconstructed differential entropy feature may be determined by the following steps:

[0099] Based on the above reconstructed differential entropy features, the loss value between the above reconstructed differential entropy features and the above differential entropy features is generated. The formula for generating the loss value is as follows:

[0100]

[0101] Among them, Loss represents the loss value, S i The S in represents the differential entropy feature. i represents the serial number of the differential entropy feature. i represents the i-th differential entropy feature. S in * Represents the reconstructed differential entropy feature. i represents the sequence number of the reconstructed differential entropy feature. Represents the i-th reconstructed differential entropy feature. N represents the number of the above-mentioned reconstructed differential entropy features, which is also the number of the above-mentioned differential entropy features. ∑ is the accumulation symbol, Indicates that i is accumulated from 1 to N.

[0102] In step 112 , for the target user, the pre-trained model is fine-tuned based on the calibration data corresponding to the target user to obtain an emotion predictor, and a linear layer is used to predict the emotion category, and the classification loss is measured by cross entropy.

[0103] In some embodiments, a pre-trained model can be fine-tuned for a target user based on the calibration data corresponding to the target user to obtain an emotion predictor. A linear layer can be used to predict emotion categories, and the classification loss can be measured using cross-entropy. The target user can be any user for whom raw full-channel EEG data was collected. The calibration data can be low-channel EEG data corresponding to the target user. The low-channel EEG data can be brain waves collected using a second preset number of electrodes. The second preset number can be a value less than the first preset number. The specific setting of the second preset number is not limited. In practice, the pre-trained model can be fine-tuned using the Adam optimizer (Adaptive Moment Estimation). The linear layer can be a fully connected layer that takes the calibration data corresponding to the target user as input and outputs a probability distribution of the calibration data and each preset emotion category. The probability distribution can be used to represent the probability that the calibration data belongs to each emotion category. The emotion category can be a label used to represent the user's emotion. For example, the emotion category can be "angry," "happy," or "sad."

[0104] Step 113: Use the test data corresponding to the target user to verify the effectiveness of the emotion predictor and obtain the emotion category corresponding to the target user.

[0105] In some embodiments, the validity of the emotion predictor can be verified using test data corresponding to the target user to obtain the emotion category corresponding to the target user. The test data can be brain waves collected when collecting the original full-channel EEG data corresponding to the target user, which are different from the original full-channel EEG data.

[0106] Figure 2 The flowchart of obtaining each differential entropy feature is shown in Figure 1. Figure 2 As shown, the original EEG signal is any one of the above-mentioned original full-channel EEG data. EEG signal filtering is the pre-processed full-channel EEG data obtained after downsampling, filtering, re-referencing, and independent component analysis of the above-mentioned original EEG signal. The α, β, and γ bands are used to represent the above-mentioned five bands δ (0.1-3Hz), θ (4-7Hz), α (8-12Hz), β (13-30Hz), and γ (31-47Hz). Frequency domain analysis is the step of performing frequency domain feature extraction processing on the above-mentioned pre-processed full-channel EEG data in the above-mentioned five bands. δ (0.1-3Hz) in the above-mentioned five bands corresponds to Figure 2 The θ (4-7Hz) in the above five bands corresponds to Figure 2 The "Theta waves" in the above five bands correspond to α (8-12Hz) Figure 2 The β (13-30Hz) in the above five bands corresponds to Figure 2 The γ (31-47Hz) in the above five bands corresponds to Figure 2 "Gamma waves" in.

[0107] Figure 3 This is a diagram of brain regions based on different neurological priors. According to the international 10-20 system, when using electrodes to collect brain waves, the cerebral cortex can be divided into four main regions: the frontal lobe, the occipital lobe, the parietal lobe, and the temporal lobe.

[0108] like Figure 3 As shown, Figure 3 Part a further subdivides the four main regions into 11 subregions. Each of the 11 subregions includes an electrode placement identifier. The electrode placement identifier can be the name of the electrode. When using electrodes to collect the raw full-channel EEG data, the electrode placement identifier can be used to indicate the placement of the electrode on the user's scalp. Figure 3 Part b of the figure further subdivides the four main regions into 14 subregions. The frontal lobe region is subdivided into six subregions. These six subregions are symmetrically arranged across the frontal lobe to provide more discriminative frontal lobe information. Each of the 14 subregions includes electrode placement markers. Figure 3 In part c of FIG, the four main regions are further divided into 17 sub-regions that are symmetrical in the left and right hemispheres. Each of the 17 sub-regions includes electrode placement markers. Figure 3 In part d of FIG, the four main regions are further divided into 10 sub-regions. Each of the 10 sub-regions includes an electrode placement marker. Each electrode placement marker corresponds to Figure 3 The individual characters in .

[0109] Figure 4 FIG. 1 is a flowchart for encoding and decoding the above-mentioned time-visible frame features. Figure 4 As shown, the masked EEG signal can represent the time-visible frame features corresponding to the above-mentioned original full-channel EEG data. The encoder can be the above-mentioned temporal encoder and the above-mentioned spatial encoder. The decoder can be the above-mentioned spatial decoder and the above-mentioned temporal decoder. The reconstructed EEG signal can be the above-mentioned reconstructed differential entropy features. Figure 4The δ, θ, α, β, and γ in the figure correspond to the five bands δ (0.1-3 Hz), θ (4-7 Hz), α (8-12 Hz), β (13-30 Hz), and γ (31-47 Hz).

[0110] Figure 5 Flowchart for fine-tuning the pre-trained model and validating the fine-tuned emotion predictor using test data. Figure 5 As shown in Figure 2, in stage 2: personalized calibration, the calibration data is the calibration data corresponding to the target user. Feature extraction and linear layer are the process of fine-tuning the pre-trained model using the calibration data to obtain the emotion predictor. In stage 3: personalized testing, the test data is the test data corresponding to the target user. Emotion recognition is the process of using the emotion predictor to perform emotion recognition on the test data. Figure 5 As shown, For the above emotion categories, The corresponding emotion category may be "happy". For the above emotion categories, The corresponding emotion category may be "calm". For the above emotion categories, The corresponding emotion category may be "sad".

[0111] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A self-supervised emotion recognition method based on EEG signals in a virtual reality scenario, comprising: Step (1): collecting original full-channel EEG data of all users, and preprocessing each collected original full-channel EEG data to generate each preprocessed full-channel EEG data; Step (2): performing frequency domain feature extraction processing on each of the pre-processed full-channel EEG data to obtain each differential entropy feature; Step (3): generating respective embedding features based on the respective differential entropy features, and inputting the respective embedding features into an initial neural network, wherein each of the respective embedding features corresponds to a respective time frame; Step (4): for each of the embedded features, adding a position code to each time frame corresponding to the embedded feature, and randomly masking the time frame of the preset position code corresponding to the embedded feature based on a preset frame masking ratio, to obtain a temporally visible frame feature and a temporally masked frame feature corresponding to the embedded feature; Step (5): Input the obtained features of each temporally visible frame into the temporal encoder in the initial neural network to obtain the embedded information of each temporal code; Step (6): Projecting the previously acquired real three-dimensional coordinates onto a two-dimensional plane to obtain real two-dimensional coordinates, and generating spatial position codes based on the real two-dimensional coordinates; Step (7): combining each time code embedding information in each time code embedding information and each spatial position code into a high-dimensional feature, and performing spatial masking processing on each combined high-dimensional feature to obtain each spatial visible frame feature and each spatial masked frame feature; Step (8): inputting each spatial visible frame feature into the spatial encoder in the initial neural network to obtain each spatial coding embedding information; Step (9): inputting the respective spatial coding embedding information and the respective spatial masking frame features into the spatial decoder in the initial neural network to obtain respective spatial decoding features; Step (10): inputting each of the spatial decoding features and the corresponding temporal mask frame features into the temporal decoder in the initial neural network to obtain a reconstructed differential entropy feature; Step (11): determining a loss value between each of the obtained reconstructed differential entropy features and the corresponding differential entropy feature, and looping through steps (3) to (11) based on a preset number of iterations to obtain the loss value, and determining the initial neural network after the iterative loop as a pre-trained model; Step (12): For the target user, fine-tune the pre-trained model based on the calibration data corresponding to the target user to obtain an emotion predictor, and use a linear layer to predict the emotion category, and measure the classification loss by cross entropy; Step (13): using the test data corresponding to the target user to verify the validity of the emotion predictor and obtain the emotion category corresponding to the target user.

2. The method according to claim 1, wherein The frequency domain feature extraction process is performed on each of the pre-processed full-channel EEG data to obtain each differential entropy feature, including: Frequency domain feature extraction processing is performed on the five bands in each preprocessed full-channel EEG data to obtain various differential entropy features, wherein the five bands include δ (0.1-3 Hz), θ (4-7 Hz), α (8-12 Hz), β (13-30 Hz), and γ (31-47 Hz), δ in δ (0.1-3 Hz) is the name of the frequency band of 0.1-3 Hz corresponding to each preprocessed full-channel EEG data, (0.1-3 Hz) indicates that the frequency of each preprocessed full-channel EEG data is 0.1-3 Hz, θ in θ (4-7 Hz) is the name of the frequency band of 4-7 Hz corresponding to each preprocessed full-channel EEG data, (4-7 Hz) indicates that the frequency of each preprocessed full-channel EEG data is 4-7H z, α in α(8-12Hz) is the name of the frequency band of 8-12Hz corresponding to each preprocessed full-channel EEG data, (8-12Hz) indicates that the frequency of each preprocessed full-channel EEG data is 8-12Hz, β in β(13-30Hz) is the name of the frequency band of 13-30Hz corresponding to each preprocessed full-channel EEG data, (13-30Hz) indicates that the frequency of each preprocessed full-channel EEG data is 13-30Hz, γ in γ(31-47Hz) is the name of the frequency band of 31-47Hz corresponding to each preprocessed full-channel EEG data, (31-47Hz) indicates that the frequency of each preprocessed full-channel EEG data is 31-47Hz, and the differential entropy feature is the Shannon information entropy-∑ x p(x)log(p(x))dx, a generalized form for continuous variables, is calculated as follows: Where DE represents the differential entropy feature, p(x) represents the probability density function of the continuous variable x, x is the independent variable, the value of x is the potential value of the brain wave at a specific time point, ∫ is the integral symbol, a represents the lower limit of integration, b represents the upper limit of integration, dx represents the differential of x, log represents the logarithm with base 2, and for a specific length, it is approximately subject to the Gaussian distribution N(μ, σ 2 )’s preprocessed full-channel EEG data, its differential entropy feature is: Wherein, DE represents the differential entropy feature, ∫ is the integration symbol, -∞ represents the lower limit of integration, ∞ represents the upper limit of integration, dx represents the differential with respect to x, log represents the logarithm with base 2, π is pi, e is a natural constant, x is the independent variable, the value of x is the potential value of the preprocessed full-channel EEG data at a preset time point, μ is the mean, and σ is the standard deviation.

3. The method according to claim 1, wherein Generating each embedding feature based on each differential entropy feature includes: Each differential entropy feature is transformed to obtain each transformed feature, wherein each differential entropy feature is represented by X=(x1, x2, ..., x N )∈R (N×C×F) , R (N×C×F) The R in the formula represents the real number field, R (N×C×F) Represents a three-dimensional matrix composed of real numbers, whose dimensions are N, C, F, x1, x2, ..., x N Represents each differential entropy feature, and each transformation feature is expressed as Represents each transformation feature, R (N×C×T×F) The R in the formula represents the real number field, R (N×C×T×F) represents a four-dimensional matrix composed of real numbers, whose dimensions are N, C, F and V respectively; C represents the set of electrodes corresponding to the acquisition of each original full-channel EEG data, N is the set of each original full-channel EEG data obtained at each time point; F represents the set of bands when performing frequency domain feature processing, T represents the overlapping window set to T seconds when transforming each differential entropy feature through the overlapping window technology, that is, each transformed feature corresponds to T time frames; For each of the conversion features, it is projected into the D-dimensional space through a linear layer to embed the frequency domain information of each conversion feature to obtain each embedded feature, wherein the feature dimension of each embedded feature in the embedding features is C×T×D, and the embedding formula is as follows: in, represents any one of the embedding features, w (s) is the weight vector, w (s) ∈R (F×D) , R (F×D) The R in the formula represents the real number field, R (F×D) Represents a two-dimensional matrix composed of real numbers, whose dimensions are F and D, b (s) is the deviation, b (s) ∈R (D) , R (D) The R in the formula represents the real number field, whose dimension is D, R (D) Represents a D-dimensional vector consisting of real numbers, S represents the state variable, which is only used to distinguish before and after embedding 4. The method according to claim 1, wherein The adding of position codes to the respective time frames corresponding to the embedded features, and randomly masking the time frames of the preset position codes corresponding to the embedded features based on a preset frame masking ratio, to obtain the temporally visible frame features and the temporally masked frame features corresponding to the embedded features, include: Adding position codes to each time frame corresponding to the embedded features; Based on a preset frame masking ratio, the time frame of the preset position encoding corresponding to the embedded feature is randomly masked to obtain the temporal visible frame feature and the temporal masked frame feature corresponding to the embedded feature, wherein the embedded feature is expressed as R (C×T×D) The R in the formula represents the real number field, R (C×T×D) Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T and D respectively. The temporal visible frame feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T v and D, the temporal masked frame feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T m and D, T are the embedding features The set of corresponding time frames, T v is the set of unmasked time frames, T m is the set of masked time frames, T v ∪T m =T, ∪ is the union symbol.

5. The method according to claim 1, wherein The step of inputting the obtained features of each temporally visible frame into the temporal encoder in the initial neural network to obtain each temporal encoding embedding information includes: The obtained features of each temporally visible frame are input into the temporal encoder in the initial neural network, where the formula for obtaining the temporal encoding embedding information is: Attention() is an attention mechanism, Q represents the query vector corresponding to the time encoder, K represents the key vector corresponding to the time encoder, V represents the value vector corresponding to the time encoder, Softmax() represents the normalized exponential function, K T T is the transpose symbol, K represents the key vector corresponding to the time encoder, K T is the transpose of the key vector K, TD is the product of character T and character D, the character T represents the dimension of the query vector Q, that is, the number of columns corresponding to the query vector Q, the character D represents the dimension of the key vector K, that is, the number of columns of the key vector K, and each time code embedding information finally output by the time encoder is represented as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T v and D.

6. The method according to claim 3, wherein: The projecting of the pre-acquired real three-dimensional coordinates onto a two-dimensional plane to obtain real two-dimensional coordinates, and generating spatial position codes based on the real two-dimensional coordinates, includes: Projecting the previously acquired real three-dimensional coordinates onto a two-dimensional plane to obtain real two-dimensional coordinates, wherein each of the real two-dimensional coordinates includes a horizontal coordinate and a vertical coordinate, and each real two-dimensional coordinate is represented as R(x, y); Sorting the abscissas included in the real two-dimensional coordinates to obtain a sequence of abscissas; Sorting the ordinates included in the real two-dimensional coordinates to obtain a ordinate sequence; For each abscissa in the abscissa sequence, perform the following steps: The ordinate corresponding to the abscissa in the ordinate sequence is determined as the ordinate to be encoded, wherein the abscissa can be expressed as x p , the ordinate to be encoded is represented by y p , p represents the sequence number of the abscissa in the abscissa sequence, and also represents the sequence number of the ordinate to be encoded in the ordinate sequence; Based on the abscissa, each abscissa sinusoidal position code corresponding to the abscissa is generated, wherein the generation formula is as follows: Among them, x p represents the horizontal axis, i represents the index of the feature dimension in the encoding, sin represents the sine function, d represents the number of electrodes used when collecting the original full-channel EEG data, Ω represents a constant, Encode represents the encoding, Dncode(x p , 2i) represents the horizontal coordinate x p Encode on the feature dimension where the corresponding index i is an even number; Based on the abscissa, each abscissa cosine position code corresponding to the abscissa is generated, wherein the generation formula is as follows: Among them, cos represents the cosine function, Encode(x p , 2i+1) represents the horizontal coordinate x p Encode on the feature dimension where the corresponding index i is an odd number; Determine the sum of each of the abscissa position codes and each of the abscissa cosine position codes as the abscissa position code; Based on the ordinate to be encoded, generate the sinusoidal position codes of the ordinates corresponding to the ordinate to be encoded, wherein the generation formula is as follows: Among them, y p Indicates the ordinate to be encoded, Encode(y p , 2i) represents the ordinate y to be encoded p Encode on the feature dimension where the corresponding index i is an even number; Based on the ordinate to be encoded, the cosine position codes of the ordinates corresponding to the ordinate to be encoded are generated, wherein the generation formula is as follows: Among them, Encode(y p , 2i+1) means that the ordinate to be encoded is encoded on the feature dimension where the corresponding index i is an odd number; Determine the sum of the sine position codes of each ordinate and the cosine position codes of each ordinate as the ordinate position code corresponding to the ordinate to be encoded; The horizontal coordinate position code and the vertical coordinate position code are concatenated to obtain a spatial position code, wherein the concatenation formula is as follows: PE x，y，i =CONCAT[Encode(x p ,i),Encode(y p ,i)], Among them, i represents the index of the feature dimension in the encoding, PE x，y，i It represents the spatial position encoding of the horizontal coordinate x and the vertical coordinate y corresponding to the horizontal coordinate x when the feature dimension is l, CONCAT represents the concat function, Encode(x p , i) represents the horizontal coordinate x p Encoding on all feature dimensions, Encode(y p , i) represents the ordinate y to be encoded p Encoding across all feature dimensions.

7. The method according to claim 1, wherein The combining of each time code embedding information in each time code embedding information and each spatial position code into a high-dimensional feature, and performing spatial masking processing on each combined high-dimensional feature to obtain each spatial visible frame feature and each spatial masked frame feature, includes: For each time code embedded information in the respective time code embedded information, the following steps are performed: Combining the time code embedding information and the respective spatial position codes into a high-dimensional feature; The high-dimensional features are subjected to spatial masking processing to obtain spatially visible frame features and spatially masked frame features, wherein the spatially visible frame features are expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C v 、T v and D, the spatial masked frame feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C m 、T v and D, C v ∪C m = C, C represents the set of electrodes used to collect the original full-channel EEG data of all users, C v represents the set of electrodes without spatial masking, C m represents the set of electrodes that have been spatially masked, and ∪ is the union symbol.

8. The method according to claim 1, wherein Inputting each spatially visible frame feature into a spatial encoder in an initial neural network to obtain each spatial encoding embedding information includes: For each of the spatially visible frame features, the spatially visible frame feature is input into the spatial encoder in the initial neural network to obtain spatial encoding embedding information corresponding to the spatially visible frame feature, wherein the formula for obtaining the spatial encoding embedding information is: Among them, Attention() is the attention mechanism, Q represents the query vector corresponding to the spatial encoder, K represents the key vector corresponding to the spatial encoder, V represents the value vector corresponding to the spatial encoder, Softmax() represents the normalized exponential function, K T Where T is the transpose symbol, TD is the product of character T and character D, the character T represents the dimension of the query vector Q, that is, the number of columns corresponding to the query vector Q, the character D represents the dimension of the key vector K, that is, the number of columns of the key vector K, and each spatial encoding embedding information finally output by the spatial encoder can be expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C v 、T v and D.

9. The method according to claim 1, wherein The step of inputting each of the spatial decoding features and the corresponding temporal mask frame feature into a temporal decoder in the initial neural network to obtain a reconstructed differential entropy feature comprises: For each of the spatial decoding features, performing the following steps: The spatial decoding feature and the temporal mask frame feature corresponding to the spatial decoding feature are input into the temporal decoder in the initial neural network to obtain the temporal decoding feature, wherein the temporal decoding feature is expressed as The R in represents the field of real numbers. Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, T v and D; The time decoding feature is input into the linear layer to obtain the reconstructed differential entropy feature, wherein the reconstructed differential entropy feature is expressed as R (C×T×F) The R in the formula represents the real number field, R (C×T×F) Represents a three-dimensional matrix composed of real numbers, whose dimensions are C, , T, and F respectively.

10. The method according to claim 1, wherein The determining of the loss value between each reconstructed differential entropy feature and the corresponding differential entropy feature in each of the obtained reconstructed differential entropy features includes: Based on the respective reconstructed differential entropy features, a loss value between the respective reconstructed differential entropy features and the respective differential entropy features is generated. The formula for generating the loss value is as follows: Among them, Loss represents the loss value, S i is the differential entropy characteristic, is the reconstructed differential entropy feature, i is the serial number of the differential entropy feature and the reconstructed differential entropy feature, N represents the number of the reconstructed differential entropy features, ∑ is the accumulation symbol, Indicates that i is accumulated from 1 to N.

Images