Emotion recognition method based on brain activation region multi-view comparative learning

Abstract

The invention provides an emotion recognition method based on brain activation region multi-view comparative learning. The method comprises the following steps: acquiring a to-be-recognized electroencephalogram emotion signal; performing space-time transformation on the to-be-recognized electroencephalogram emotion signal to obtain a to-be-recognized space-time signal; inputting the space-time signal to be recognized into a pre-trained double-path decoupling multi-view model to obtain an emotion recognition result; two encoders used for extracting time features and space features are arranged in a final double-path decoupling multi-view encoder, so that multi-dimensional information in a space-time signal to be identified can be fully excavated and utilized, and decoupling of corresponding space-time features is realized; a space-time signal sample is divided according to the activation degree of a brain region to obtain a high activation region and a low activation region, a targeted data enhancement strategy can be executed according to different activation degrees of the brain region, and on the basis, a pre-trained double-path decoupling multi-view model is obtained through training by using the sample subjected to data enhancement; and the emotion recognition accuracy and reliability of the pre-training double-path decoupling multi-view model are improved.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of biological signal processing and artificial intelligence, and in particular to an emotion recognition method based on multi-view contrast learning of brain activation areas. Background Art

[0002] As an important part of the complex physiological and psychological state of human beings, emotions occupy a core position in cognition, decision-making, behavior and interpersonal interaction. In recent years, with the advancement of technology, emotion recognition technology has shown broad application potential in many fields such as human-computer interaction, mental health assessment and affective computing. Compared with external emotional signals such as facial expressions and voice intonation, electroencephalogram (EEG), as an intrinsic physiological indicator with high temporal and spatial resolution, can more directly and objectively reflect the true emotional state of an individual. Therefore, it has received great attention in emotion recognition research. EEG signals capture the electrical activity of the brain through multiple electrodes and have rich temporal and spatial characteristics, providing unique and complementary information for a deep understanding of brain function.

[0003] However, most current EEG emotion recognition methods rely mainly on the supervised learning paradigm. Although this method has excellent performance, its success is highly dependent on a large amount of labeled training data. Unlike image and text data, EEG signals lack intuitive human-recognizable patterns, which makes the annotation process of EEG data complex and time-consuming, greatly increasing the cost of data preparation. To solve this problem, some researchers have begun to explore self-supervised learning methods such as contrastive learning, hoping to reduce the dependence on labeled data by designing effective data enhancement and contrast tasks. For example, some scholars applied contrastive learning to EEG emotion recognition, using data enhancement techniques such as temporal masking, linear scaling, and Gaussian noise to learn the representation of EEG signals by maximizing the consistency of the same sample after different enhancements. In addition, some scholars have proposed a graph-based contrastive learning method that uses frequency and spatial jigsaw puzzles to generate enhanced data, aiming to better capture the spatial and frequency characteristics of EEG signals.

[0004] Although contrastive learning methods have alleviated the dependence on labeled data to a certain extent, there are still several shortcomings. First, existing data augmentation strategies often treat all brain regions equally, resulting in the failure to effectively protect key emotional information. Second, strong data augmentation may destroy key semantic information in EEG signals, while weak augmentation is difficult to provide sufficient sample diversity, affecting the generalization ability of the model. Finally, existing methods lack effective decoupling methods when processing the spatiotemporal features of EEG signals, and are unable to fully mine and utilize the multidimensional information in the signals, limiting the accuracy and reliability of emotion recognition. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an emotion recognition method based on multi-view contrast learning of brain activation areas.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] The present invention provides an emotion recognition method based on multi-view contrast learning of brain activation areas, comprising:

[0008] Obtaining the EEG emotion signal to be identified;

[0009] Performing space-time transformation on the EEG emotion signal to be identified to obtain the space-time signal to be identified;

[0010] The spatiotemporal signal to be identified is input into a pre-trained dual-path decoupled multi-view model to obtain an emotion recognition result; the pre-trained dual-path decoupled multi-view model includes: a final dual-path decoupled multi-view encoder and a pre-trained classifier; the final dual-path decoupled multi-view encoder is provided with two encoders, which are respectively used to identify the temporal features and spatial features of the spatiotemporal signal to be identified; the final dual-path decoupled multi-view encoder is jointly constructed based on the contrastive learning of high-confidence positive samples under the cross-view knowledge exchange strategy of temporal features and spatial features; wherein, the high-confidence positive samples of the contrastive learning are obtained by executing the corresponding data enhancement strategy on the spatiotemporal signal samples according to the activation degree of the preset brain area.

[0011] Optionally, performing a spatiotemporal transformation on the EEG emotion signal to be identified to obtain the spatiotemporal signal to be identified includes:

[0012] The EEG emotion signal to be identified is segmented in the time dimension using a sliding window method to obtain multiple segment signals to be identified;

[0013] Each to-be-identified segment signal is segmented using a sliding window to obtain a plurality of to-be-identified sub-segments containing time segments;

[0014] Cosine similarity calculation is performed on multiple sub-segments to be identified according to electrode channels, and a spatiotemporal signal to be identified is generated according to the result of the cosine similarity calculation.

[0015] Optionally, performing cosine similarity calculation on a plurality of sub-segments to be identified according to electrode channels, and generating a spatiotemporal signal to be identified according to the result of the cosine similarity calculation, includes:

[0016] Performing cosine similarity calculation on multiple to-be-identified sub-segments according to electrode channels to obtain a first cosine similarity result;

[0017] An edge is established between the sub-segments to be identified corresponding to the first cosine similarity results greater than the similarity threshold, so as to obtain the spatiotemporal signal to be identified.

[0018] Optionally, the final two-way decoupled multi-view encoder is obtained by fine-tuning the pre-trained two-way decoupled multi-view encoder using spatiotemporal label samples; the training process of the pre-trained two-way decoupled multi-view encoder includes:

[0019] Obtain EEG emotion signal samples;

[0020] Performing space-time transformation on the EEG emotion signal samples to obtain space-time signal samples;

[0021] Calculate the class activation map corresponding to the spatiotemporal signal sample in the preset brain area, and divide the spatiotemporal signal to be identified into high activation area and low activation area according to the class activation map;

[0022] Performing data enhancement and signal fusion processing on the high activation area and the low activation area in turn to obtain a fused signal sample; the data enhancement includes: at least one of space jigsaw puzzle, time jigsaw puzzle, frequency band jigsaw puzzle, random masking, noise injection or scaling transformation processing;

[0023] The fused signal samples are input into the initial two-way decoupled multi-view encoder and trained by contrastive learning;

[0024] The initial two-way decoupled multi-view encoder that meets the preset iteration stop condition is used as the pre-trained two-way decoupled multi-view encoder;

[0025] Among them, the preset brain areas include: prefrontal lobe, frontal lobe, left frontal lobe, right frontal lobe, left temporal lobe, right temporal lobe, central area, left parietal lobe, right parietal lobe and occipital lobe.

[0026] Optionally, calculating the class activation map corresponding to the spatiotemporal signal sample in the preset brain area, and dividing the spatiotemporal signal to be identified into a high activation area and a low activation area according to the class activation map, including:

[0027] Global average pooling and linear projection are performed on the spatiotemporal signal samples and static benchmarks in turn, and the sample global embedding and static global embedding are obtained accordingly; the static benchmark is the mean of all spatiotemporal signal samples;

[0028] Calculate the cosine similarity between each sample global embedding and the static global embedding to obtain the second cosine similarity result;

[0029] The gradient information of the second cosine similarity result is calculated by back propagation to obtain the cosine similarity gradient, and the cosine similarity gradient is globally averaged pooled to obtain the gradient pooling result;

[0030] The gradient pooling results are averaged according to the preset brain regions to obtain the class activation map;

[0031] The class activation map is compared with the preset activation threshold, and the spatiotemporal signal to be identified is divided into high activation area and low activation area according to the result of the threshold comparison.

[0032] Optionally, the class activation map is threshold-compared with a preset activation threshold, and the spatiotemporal signal to be identified is divided into a high activation area and a low activation area according to the result of the threshold comparison, including:

[0033] Compare the class activation map with a preset activation threshold;

[0034] The brain area corresponding to the class activation map greater than the preset activation threshold is set as a high activation area;

[0035] The brain area corresponding to the class activation map that is less than the preset activation threshold is set as a low activation area.

[0036] Optionally, data enhancement and signal fusion processing are performed on the high activation area and the low activation area in sequence to obtain a fused signal sample, including:

[0037] Perform weak data enhancement processing on the high activation area to obtain weakly enhanced data;

[0038] Perform strong data enhancement processing on low activation areas to obtain strongly enhanced data;

[0039] Data fusion processing is performed on the weakly enhanced data and the strongly enhanced data to obtain fused signal samples.

[0040] Optionally, the initial two-way decoupled multi-view encoder has the same structure as the pre-trained two-way decoupled multi-view encoder, and the initial two-way decoupled multi-view encoder includes: a spatial encoder and a temporal encoder;

[0041] The spatial encoder and the temporal encoder are connected in parallel; the spatial encoder is provided with: a first unit, a spatial conversion module, a first word embedding module, a first Transformer encoding module, and a first pooling module connected in series in sequence;

[0042] The temporal encoder is provided with: a second unit, a temporal conversion module, a second word embedding module, a second Transformer encoding module, and a second pooling module connected in series in sequence;

[0043] The first unit is provided with a first branch unit and a second branch unit connected in parallel; the first branch unit is provided with a first linear projection module; the second branch unit is provided with a graph encoding module and a second linear projection module connected in series;

[0044] The first unit and the second unit have the same structure.

[0045] Optionally, the fused signal samples are input into an initial two-way decoupled multi-view encoder and trained by contrastive learning, including:

[0046] Input the fused signal samples into the initial dual-path decoupled multi-view encoder to obtain time embedding, global embedding and spatial embedding; the time embedding is the output result of the time encoder, the spatial embedding is the output result of the spatial encoder; the global embedding is the concatenation result of the time embedding and the spatial embedding;

[0047] Calculate the similarity between the temporal embedding, global embedding, and spatial embedding and all embeddings in the corresponding queue, and obtain the temporal similarity result, global similarity result, and spatial similarity result respectively;

[0048] Sort the time similarity results in a first descending order, and use the first K time embeddings in the results of the first descending order as the first guidance information;

[0049] Sorting the spatial similarity results in a second descending order, and using the first K spatial embeddings in the results of the second descending order as second guidance information;

[0050] Sort the global similarity results in a third descending order, and use the first K global embeddings in the results of the third descending order as the third guidance information;

[0051] The first guidance information, the second guidance information, and the third guidance information are used as high-confidence positive samples;

[0052] The initial two-way decoupled multi-view encoder is trained based on high-confidence positive samples and contrastive learning to obtain a pre-trained two-way decoupled multi-view encoder.

[0053] Optionally, the generation process of the pre-trained two-way decoupled multi-view model includes:

[0054] Get the pre-trained dual-channel decoupled multi-view encoder and labeled spatiotemporal samples;

[0055] Input the labeled spatiotemporal samples into the pre-trained dual-path decoupled multi-view encoder and the initial classifier for iterative fine-tuning training;

[0056] The corresponding pre-trained two-way decoupled multi-view encoder when the stopping condition is reached is used as the final two-way decoupled multi-view encoder;

[0057] The initial classifier corresponding to the stopping condition is used as the pre-trained classifier;

[0058] The final two-way decoupled multi-view encoder and the pre-trained classifier are combined to form a pre-trained two-way decoupled multi-view model.

[0059] The present invention provides an emotion recognition method based on multi-view contrastive learning of brain activation areas, comprising: obtaining an electroencephalogram emotion signal to be recognized; performing spatiotemporal transformation on the electroencephalogram emotion signal to be recognized to obtain a spatiotemporal signal to be recognized; inputting the spatiotemporal signal to be recognized into a pre-trained dual-path decoupled multi-view model to obtain an emotion recognition result; the pre-trained dual-path decoupled multi-view model comprises: a final dual-path decoupled multi-view encoder and a pre-trained classifier; the final dual-path decoupled multi-view encoder is provided with two encoders, which are respectively used to recognize the temporal features and spatial features of the spatiotemporal signal to be recognized; the final dual-path decoupled multi-view encoder is jointly constructed based on contrastive learning of high-confidence positive samples under a cross-view knowledge exchange strategy of temporal features and spatial features; wherein the high-confidence positive samples of contrastive learning are obtained by executing a corresponding data enhancement strategy on the spatiotemporal signal samples according to the activation degree of a preset brain area. In the present invention, by setting two encoders for extracting temporal features and spatial features in the final dual-path decoupled multi-view encoder, the multi-dimensional information in the spatiotemporal signal to be identified can be fully mined and utilized, thereby realizing the decoupling of the corresponding spatiotemporal features; in addition, by dividing the spatiotemporal signal samples according to the activation degree of the brain region to obtain high-activation areas and low-activation areas, targeted data enhancement strategies can be executed according to the different activation degrees of the brain regions, thereby protecting the key information therein and enhancing the diversity of the spatiotemporal signal samples. On this basis, the pre-trained dual-path decoupled multi-view model is obtained by training with the samples after data enhancement, thereby improving the generalization ability of the pre-trained dual-path decoupled multi-view model as well as the accuracy and reliability of emotion recognition.

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flowchart of an emotion recognition method based on multi-view contrast learning of brain activation areas provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of the structure of an initial dual-path decoupled multi-view encoder provided by an embodiment of the present invention;

[0063] Figure 3 The overall architecture of contrastive learning based on the initial two-way decoupled multi-view encoder provided in the embodiment of the present invention;

[0064] Figure 4 A schematic diagram of the training process of a pre-trained dual-path decoupled multi-view model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0066] In order to improve the accuracy and reliability of emotion recognition, an embodiment of the present invention provides an emotion recognition method based on multi-view contrast learning of brain activation areas. Figure 1 The flowchart of an emotion recognition method based on multi-view contrast learning of brain activation areas provided by an embodiment of the present invention is as follows. Figure 1 As shown, including:

[0067] S101, obtaining an electroencephalogram emotion signal to be identified.

[0068] The EEG emotion signal to be identified is an emotion signal collected by brain electrodes. In the embodiment of the present invention, the emotion signal includes but is not limited to: anger, disgust, fear, sadness, pleasure, joy, enlightenment, tenderness, arousal, valence, familiarity and love.

[0069] S102, performing space-time transformation on the EEG emotion signal to be identified to obtain the space-time signal to be identified.

[0070] Optionally, S102 may specifically include:

[0071] The EEG emotion signal to be identified is segmented in the time dimension using a sliding window method to obtain multiple segment signals to be identified;

[0072] Each to-be-identified segment signal is segmented using a sliding window to obtain a plurality of to-be-identified sub-segments containing time segments;

[0073] Cosine similarity calculation is performed on multiple sub-segments to be identified according to electrode channels, and a spatiotemporal signal to be identified is generated according to the result of the cosine similarity calculation.

[0074] Optionally, performing cosine similarity calculation on a plurality of sub-segments to be identified according to electrode channels, and generating a spatiotemporal signal to be identified according to the result of the cosine similarity calculation, includes:

[0075] Performing cosine similarity calculation on multiple to-be-identified sub-segments according to electrode channels to obtain a first cosine similarity result;

[0076] An edge is established between the sub-segments to be identified corresponding to the first cosine similarity results greater than the similarity threshold, so as to obtain the spatiotemporal signal to be identified.

[0077] S103, inputting the spatiotemporal signal to be recognized into the pre-trained dual-path decoupled multi-view model to obtain an emotion recognition result.

[0078] The pre-trained dual-path decoupled multi-view model includes: a final dual-path decoupled multi-view encoder and a pre-trained classifier; the final dual-path decoupled multi-view encoder is provided with two encoders, which are respectively used to identify the temporal features and spatial features of the spatiotemporal signal to be identified; the final dual-path decoupled multi-view encoder is based on the contrastive learning of high-confidence positive samples, and is jointly constructed under the cross-view knowledge exchange strategy of temporal features and spatial features; wherein the high-confidence positive samples of contrastive learning are obtained by executing the corresponding data enhancement strategy on the spatiotemporal signal samples according to the activation degree of the preset brain area.

[0079] The embodiment of the present invention provides an emotion recognition method based on multi-view contrast learning of brain activation areas. By setting two encoders for extracting temporal features and spatial features in the final dual-path decoupled multi-view encoder, the multi-dimensional information in the spatiotemporal signal to be identified can be fully mined and utilized, thereby realizing the decoupling of the corresponding spatiotemporal features. In addition, by dividing the spatiotemporal signal samples into high-activation areas and low-activation areas according to the activation levels of the brain regions, targeted data enhancement strategies can be performed according to the different activation levels of the brain regions, thereby protecting the key information therein and enhancing the diversity of the spatiotemporal signal samples. On this basis, a pre-trained dual-path decoupled multi-view model is obtained by using the sample training after data enhancement, thereby improving the generalization ability of the pre-trained dual-path decoupled multi-view model as well as the accuracy and reliability of emotion recognition.

[0080] Optionally, the final two-way decoupled multi-view encoder is obtained by fine-tuning the pre-trained two-way decoupled multi-view encoder using spatiotemporal label samples; the training process of the pre-trained two-way decoupled multi-view encoder includes:

[0081] Obtain EEG emotion signal samples;

[0082] Performing space-time transformation on the EEG emotion signal samples to obtain space-time signal samples;

[0083] Calculate the class activation map corresponding to the spatiotemporal signal sample in the preset brain area, and divide the spatiotemporal signal to be identified into high activation area and low activation area according to the class activation map;

[0084] Performing data enhancement and signal fusion processing on the high activation area and the low activation area in turn to obtain a fused signal sample; the data enhancement includes: at least one of space jigsaw puzzle, time jigsaw puzzle, frequency band jigsaw puzzle, random masking, noise injection or scaling transformation processing;

[0085] The fused signal samples are input into the initial two-way decoupled multi-view encoder and trained by contrastive learning;

[0086] The initial two-way decoupled multi-view encoder that meets the preset iteration stop condition is used as the pre-trained two-way decoupled multi-view encoder;

[0087] Among them, the preset brain areas include: prefrontal lobe, frontal lobe, left frontal lobe, right frontal lobe, left temporal lobe, right temporal lobe, central area, left parietal lobe, right parietal lobe and occipital lobe.

[0088] In an embodiment of the present invention, EEG emotion signal samples are obtained through public EEG emotion recognition data sets, such as SEED, SEED-IV and FACED. These EEG emotion recognition data sets are composed of EEG signals recorded when subjects watch emotion-inducing videos, covering different emotional states and subject groups. The SEED data set contains EEG data of 15 subjects, which are collected by 62 electrodes. The SEED-IV data set is also provided by 15 subjects, and the data of each subject is also collected through 62 channels (electrodes). The FACED data set contains data of 123 subjects, and the EEG data of each subject is collected by 30 channels. In addition, the EEG emotion signals of the above three data sets are subjected to necessary preprocessing steps, such as filtering, denoising, standardization, etc., to improve data quality. In this embodiment, electrode positioning is performed according to the international 10-20 EEG system positioning method. Table 1 is a relationship table between brain areas and corresponding electrodes provided in an embodiment of the present invention.

[0089] Table 1 Brain regions and corresponding electrodes

[0090]

[0091] Optionally, calculating the class activation map corresponding to the spatiotemporal signal sample in the preset brain area, and dividing the spatiotemporal signal to be identified into a high activation area and a low activation area according to the class activation map, including:

[0092] Global average pooling and linear projection are performed on the spatiotemporal signal samples and static benchmarks in turn, and the sample global embedding and static global embedding are obtained accordingly; the static benchmark is the mean of all spatiotemporal signal samples;

[0093] Calculate the cosine similarity between each sample global embedding and the static global embedding to obtain the second cosine similarity result;

[0094] The gradient information of the second cosine similarity result is calculated by back propagation to obtain the cosine similarity gradient, and the cosine similarity gradient is globally averaged pooled to obtain the gradient pooling result;

[0095] The gradient pooling results are averaged according to the preset brain regions to obtain the class activation map;

[0096] The class activation map is compared with the preset activation threshold, and the spatiotemporal signal to be identified is divided into high activation area and low activation area according to the result of the threshold comparison.

[0097] Specifically, in the embodiment of the present invention, the gradient information of the second cosine similarity result is calculated by back propagation and negation.

[0098] Optionally, the class activation map is threshold-compared with a preset activation threshold, and the spatiotemporal signal to be identified is divided into a high activation area and a low activation area according to the result of the threshold comparison, including:

[0099] Compare the class activation map with a preset activation threshold;

[0100] The brain area corresponding to the class activation map greater than the preset activation threshold is set as a high activation area;

[0101] The brain area corresponding to the class activation map that is less than the preset activation threshold is set as a low activation area.

[0102] Optionally, data enhancement and signal fusion processing are performed on the high activation area and the low activation area in sequence to obtain a fused signal sample, including:

[0103] Perform weak data enhancement processing on the high activation area to obtain weakly enhanced data;

[0104] Perform strong data enhancement processing on low activation areas to obtain strongly enhanced data;

[0105] Data fusion processing is performed on the weakly enhanced data and the strongly enhanced data to obtain fused signal samples.

[0106] In the present invention, data enhancement includes at least one of space jigsaw puzzle, time jigsaw puzzle, frequency band jigsaw puzzle, random masking, noise injection or scaling transformation processing.

[0107] Specifically, the processing flow of using random masking to perform data enhancement processing in the embodiment of the present invention is as follows:

[0108] Randomly mask EEG samples (high activation area or low activation area) according to the feature dimension. Each EEG sample is independently sampled according to the binomial distribution with a masking ratio r to generate the feature mask matrix Mask C,v ,. The channel is retained as a whole, that is, all elements in each row are 1 to retain all the information of the channel, thereby generating the channel retention matrix Mask V,v . A probability matrix U∈[0,1] is randomly generated, and the elements of the probability matrix are sampled from a uniform distribution. A probability threshold σ is set to control the weights of the two masking strategies. For each channel v, according to the corresponding channel probability value U in the probability matrix U v Select a masking strategy:

[0109]

[0110] Among them, x i,vRepresents the vth channel (region) of the i-th spatiotemporal signal sample. For x i,v The corresponding masked samples (weakly enhanced data or strongly enhanced data), ⊙ represents the element-by-element multiplication operation.

[0111] Furthermore, in this embodiment, weak data enhancement processing can be applied to high activation areas, that is, a higher mask ratio r=0.7 and a probability threshold σ=0.7 are set to reduce masking of high activation areas, retain more original features, and obtain weakly enhanced data. Strong data enhancement processing is applied to low activation areas, and a lower mask ratio r=0.3 and a lower probability threshold σ=0.3 are set to obtain strongly enhanced data to increase data diversity.

[0112] Furthermore, the data enhancement processing strategy shown above is only exemplary, and any method that can adaptively perform matching enhancement processing on low activation areas and high activation areas can be used as an alternative to the data enhancement processing of this solution.

[0113] Optionally, the initial two-way decoupled multi-view encoder has the same structure as the pre-trained two-way decoupled multi-view encoder, and the initial two-way decoupled multi-view encoder includes: a spatial encoder and a temporal encoder;

[0114] Figure 2 The schematic diagram of the structure of the initial dual-channel decoupled multi-view encoder provided by the embodiment of the present invention. Figure 2 As shown, the spatial encoder and the temporal encoder are connected in parallel; the spatial encoder is provided with: a first unit, a spatial conversion module, a first word embedding module, a first Transformer encoding module and a first pooling module connected in series in sequence;

[0115] The temporal encoder is provided with: a second unit, a temporal conversion module, a second word embedding module, a second Transformer encoding module, and a second pooling module connected in series in sequence;

[0116] The first unit is provided with a first branch unit and a second branch unit connected in parallel; the first branch unit is provided with a first linear projection module; the second branch unit is provided with a graph encoding module and a second linear projection module connected in series;

[0117] The first unit and the second unit have the same structure.

[0118] In the embodiment of the present invention, the space conversion module is used to convert the input signal into the space dimension. The time conversion module is used to convert the input signal into the time dimension.

[0119] Optionally, the fused signal samples are input into an initial two-way decoupled multi-view encoder and trained by contrastive learning, including:

[0120] Input the fused signal samples into the initial dual-path decoupled multi-view encoder to obtain time embedding, global embedding and spatial embedding; the time embedding is the output result of the time encoder, the spatial embedding is the output result of the spatial encoder; the global embedding is the concatenation result of the time embedding and the spatial embedding;

[0121] Calculate the similarity between the temporal embedding, global embedding, and spatial embedding and all embeddings in the corresponding queue, and obtain the temporal similarity result, global similarity result, and spatial similarity result respectively;

[0122] Sort the time similarity results in a first descending order, and use the first K time embeddings in the results of the first descending order as the first guidance information;

[0123] Sorting the spatial similarity results in a second descending order, and using the first K spatial embeddings in the results of the second descending order as second guidance information;

[0124] Sort the global similarity results in a third descending order, and use the first K global embeddings in the results of the third descending order as the third guidance information;

[0125] The first guidance information, the second guidance information, and the third guidance information are used as high-confidence positive samples;

[0126] The initial two-way decoupled multi-view encoder is trained based on high-confidence positive samples and contrastive learning to obtain a pre-trained two-way decoupled multi-view encoder.

[0127] Figure 3 The overall architecture of contrastive learning based on the initial two-way decoupled multi-view encoder provided by the embodiment of the present invention. Figure 3As shown, in the contrastive learning framework, the spatiotemporal signal samples are trained synchronously in the training channel and the contrast channel. First, the spatiotemporal signal samples are divided into brain regions of high activation and low activation according to the preset brain regions, and the corresponding data enhancement processing is performed according to the degree of activation. The fused signal samples obtained after the enhancement processing are input into the initial dual-channel decoupled multi-view encoder for training to obtain time embedding, spatial embedding and global embedding, and the time embedding, spatial embedding and global embedding obtained by the contrast channel are combined to form positive samples. Finally, the positive samples obtained through the training channel and the negative samples in the corresponding queue learn from each other based on the contrast loss to obtain a pre-trained dual-channel decoupled multi-view encoder. Among them, the time, global and spatial queues are used to store the time embedding, spatial embedding and global embedding output by the initial dual-channel decoupled multi-view encoder before the current batch in turn, and correspond to the negative samples as the current batch samples. In addition, after obtaining the above positive samples, the high-confidence positive samples can also be determined according to the similarity results obtained by cosine similarity calculation, and the initial dual-channel decoupled multi-view encoder is trained based on the high-confidence positive samples, negative samples and contrastive learning to obtain a pre-trained dual-channel decoupled multi-view encoder. The process of determining the high-confidence positive sample has been described in the above embodiment, and will not be repeated in this embodiment.

[0128] In addition, the contrastive learning in the embodiment of the present invention uses contrastive loss to determine the end of iteration. That is, when the number of iterations is greater than a preset iteration threshold or the contrastive loss value is continuously less than a preset loss threshold, it indicates that the contrastive learning training process is terminated.

[0129] Optionally, the generation process of the pre-trained two-way decoupled multi-view model includes:

[0130] Get the pre-trained dual-channel decoupled multi-view encoder and labeled spatiotemporal samples;

[0131] Input the labeled spatiotemporal samples into the pre-trained dual-path decoupled multi-view encoder and the initial classifier for iterative fine-tuning training;

[0132] The corresponding pre-trained two-way decoupled multi-view encoder when the stopping condition is reached is used as the final two-way decoupled multi-view encoder;

[0133] The initial classifier corresponding to the stopping condition is used as the pre-trained classifier;

[0134] The final two-way decoupled multi-view encoder and the pre-trained classifier are combined to form a pre-trained two-way decoupled multi-view model.

[0135] After obtaining the pre-trained two-way decoupled multi-view encoder, a classification head is added on this basis for the specific emotion recognition task, and then the entire model (pre-trained two-way decoupled multi-view encoder + initial classifier) ​​is fine-tuned using a small number of labeled spatiotemporal samples. During the fine-tuning process, the parameters of the pre-trained two-way decoupled multi-view encoder are slightly adjusted through end-to-end training to better adapt to the specific needs of emotion recognition, while retaining the rich and common feature representations obtained from the contrastive learning stage. This enables the pre-trained two-way decoupled multi-view model to show good generalization ability even with limited labeled data. In the training process of the pre-trained two-way decoupled multi-view model, the cross entropy loss function is used, and the deep learning framework uses PyTorch.

[0136] Figure 4 Schematic diagram of the training process of the pre-trained dual-path decoupled multi-view model provided in an embodiment of the present invention. Figure 4 As shown in the figure, in the first stage, a pre-trained dual-path decoupled multi-view encoder is first obtained through unsupervised training. Specifically, the pre-trained dual-path decoupled multi-view encoder is obtained through contrastive learning training using spatiotemporal signal samples under the corresponding data enhancement strategy. In the second stage, a small number of labeled samples (labeled spatiotemporal samples) are used to fine-tune the pre-trained dual-path decoupled multi-view encoder and the initial classifier at the same time, and finally a pre-trained dual-path decoupled multi-view model is obtained.

[0137] In summary, the emotion recognition method based on multi-view contrast learning of brain activation areas provided by the embodiment of the present invention optimizes the data enhancement strategy, adaptively designs strong or weak enhancement strategies for low activation areas and high activation areas, effectively protects the key information of high activation areas, and introduces sufficient diversity for low activation areas, avoiding the loss of semantic information and the problem of too high similarity of data samples. In addition, the adaptive data enhancement method based on activation level makes data processing more flexible and intelligent, and can dynamically adjust the enhancement strategy according to the activation of different brain areas, thereby improving the flexibility and adaptability of the overall model. Through adaptive data transformation and multi-view consistency learning, the model shows stronger generalization ability and wider adaptability in different data distributions and practical application scenarios. Specifically, the use of dual-path multi-view encoders can make full use of the temporal and spatial features in EEG signals, ensuring the independent extraction and fusion of temporal and spatial features, and improving the accuracy and comprehensiveness of feature expression. Finally, by exchanging high-confidence positive samples (time, space and global), the consistency of the contrast context under different perspectives is maintained, which promotes stable and meaningful knowledge mining and utilization.

[0138] The method provided in the embodiment of the present invention can be applied to an electronic device. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc., which is not limited in the embodiment of the present invention.

[0139] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0140] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0141] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes of the above disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and the meaning of "multiple" is two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0142] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. An emotion recognition method based on multi-view contrast learning of brain activation areas, characterized in that: include: Obtaining the EEG emotion signal to be identified; Performing space-time transformation on the EEG emotion signal to be identified to obtain a space-time signal to be identified; Inputting the to-be-recognized spatiotemporal signal into a pre-trained dual-path decoupled multi-view model to obtain an emotion recognition result; The pre-trained dual-path decoupled multi-view model includes: a final dual-path decoupled multi-view encoder and a pre-trained classifier; the final dual-path decoupled multi-view encoder is provided with two encoders, which are respectively used to identify the temporal features and spatial features of the spatiotemporal signal to be identified; the final dual-path decoupled multi-view encoder is based on the contrastive learning of high-confidence positive samples, and is jointly constructed under the cross-view knowledge exchange strategy of temporal features and spatial features; wherein the high-confidence positive samples of the contrastive learning are obtained by executing the corresponding data enhancement strategy on the spatiotemporal signal samples according to the activation degree of the preset brain area.

2. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 1, characterized in that: The step of performing a spatiotemporal transformation on the EEG emotion signal to be identified to obtain the spatiotemporal signal to be identified includes: Using a sliding window method, the EEG emotion signal to be identified is segmented in the time dimension to obtain a plurality of segment signals to be identified; Using a sliding window to segment each of the to-be-identified segment signals to obtain a plurality of to-be-identified sub-segments containing time segments; The cosine similarity calculation is performed on the multiple sub-segments to be identified according to the electrode channels, and the spatiotemporal signal to be identified is generated according to the result of the cosine similarity calculation.

3. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 2 is characterized in that: The step of performing cosine similarity calculation on the plurality of sub-segments to be identified according to electrode channels, and generating the spatiotemporal signal to be identified according to the result of the cosine similarity calculation, comprises: Performing cosine similarity calculation on the multiple sub-segments to be identified according to electrode channels to obtain a first cosine similarity result; An edge is established between the sub-segments to be identified corresponding to the first cosine similarity results that are greater than a similarity threshold, so as to obtain the spatiotemporal signal to be identified.

4. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 1, characterized in that: The final two-way decoupled multi-view encoder is obtained by fine-tuning the pre-trained two-way decoupled multi-view encoder using spatiotemporal label samples; The training process of the pre-trained dual-path decoupled multi-view encoder includes: Obtain EEG emotion signal samples; Performing space-time transformation on the EEG emotion signal sample to obtain a space-time signal sample; Calculating a class activation map corresponding to the spatiotemporal signal sample in a preset brain region, and dividing the spatiotemporal signal to be identified into a high activation region and a low activation region according to the class activation map; Performing data enhancement and signal fusion processing on the high activation area and the low activation area in sequence to obtain a fused signal sample; the data enhancement includes: at least one of space jigsaw puzzle, time jigsaw puzzle, frequency band jigsaw puzzle, random masking, noise injection or scaling transformation processing; Inputting the fused signal samples into an initial dual-path decoupled multi-view encoder and training it through contrastive learning; Using the initial two-way decoupled multi-view encoder that meets the preset iteration stop condition as the pre-trained two-way decoupled multi-view encoder; Among them, the preset brain areas include: prefrontal lobe, frontal lobe, left frontal lobe, right frontal lobe, left temporal lobe, right temporal lobe, central area, left parietal lobe, right parietal lobe and occipital lobe.

5. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 4, characterized in that: The calculating the class activation map corresponding to the spatiotemporal signal sample in the preset brain region, and dividing the spatiotemporal signal to be identified into a high activation area and a low activation area according to the class activation map, comprises: The spatiotemporal signal samples and static benchmarks are subjected to global average pooling and linear projection in sequence, and a sample global embedding and a static global embedding are obtained correspondingly; the static benchmark is the mean of all spatiotemporal signal samples; Calculating the cosine similarity between each of the sample global embeddings and the static global embedding to obtain a second cosine similarity result; Calculating the gradient information of the second cosine similarity result by back propagation to obtain a cosine similarity gradient, and performing global average pooling processing on the cosine similarity gradient to obtain a gradient pooling result; Performing average processing on the gradient pooling results according to the preset brain regions to obtain the class activation map; The class activation map is compared with a preset activation threshold, and the to-be-identified spatiotemporal signal is divided into the high activation region and the low activation region according to a result of the threshold comparison.

6. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 5, characterized in that: The step of comparing the class activation map with a preset activation threshold, and dividing the to-be-identified spatiotemporal signal into the high activation region and the low activation region according to the result of the threshold comparison, comprises: comparing the class activation map with the preset activation threshold; Setting the brain area corresponding to the class activation map greater than the preset activation threshold as the high activation area; The brain area corresponding to the class activation map that is smaller than the preset activation threshold is set as the low activation area.

7. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 4, characterized in that: The step of sequentially performing data enhancement and signal fusion processing on the high activation region and the low activation region to obtain a fused signal sample includes: Performing weak data enhancement processing on the high activation area to obtain weakly enhanced data; Performing strong data enhancement processing on the low activation area to obtain strongly enhanced data; Data fusion processing is performed on the weakly enhanced data and the strongly enhanced data to obtain the fused signal sample.

8. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 4, characterized in that: The initial two-way decoupled multi-view encoder has the same structure as the pre-trained two-way decoupled multi-view encoder, and the initial two-way decoupled multi-view encoder includes: a spatial encoder and a temporal encoder; The spatial encoder and the temporal encoder are connected in parallel; the spatial encoder is provided with: a first unit, a spatial conversion module, a first word embedding module, a first Transformer encoding module and a first pooling module connected in series in sequence; The temporal encoder is provided with: a second unit, a temporal conversion module, a second word embedding module, a second Transformer encoding module and a second pooling module connected in series in sequence; The first unit is provided with a first branch unit and a second branch unit connected in parallel; the first branch unit is provided with a first linear projection module; the second branch unit is provided with a graph encoding module and a second linear projection module connected in series; The first unit and the second unit have the same structure.

9. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 4, characterized in that: The step of inputting the fused signal samples into an initial dual-path decoupled multi-view encoder and training the encoder by contrastive learning comprises: Input the fused signal sample into the initial dual-path decoupled multi-view encoder to obtain time embedding, global embedding and spatial embedding; the time embedding is the output result of the time encoder, the spatial embedding is the output result of the spatial encoder; the global embedding is the concatenation result of the time embedding and the spatial embedding; Calculate the similarity between the time embedding, the global embedding and the spatial embedding and all embeddings in the corresponding queue, and obtain a time similarity result, a global similarity result and a spatial similarity result respectively; Sorting the time similarity results in a first descending order, and embedding the first K time in the results of the first descending order as first guidance information; Sorting the spatial similarity results in a second descending order, and embedding the first K spaces in the results of the second descending order as second guidance information; Sorting the global similarity results in a third descending order, and using the first K global embeddings in the results of the third descending order as third guidance information; Taking the first guidance information, the second guidance information, and the third guidance information as the high-confidence positive samples; The initial two-way decoupled multi-view encoder is trained based on the high-confidence positive samples and the contrastive learning to obtain the pre-trained two-way decoupled multi-view encoder.

10. The emotion recognition method based on multi-view contrast learning of brain activation areas according to claim 9, characterized in that: The generation process of the pre-trained dual-path decoupled multi-view model includes: Obtain the pre-trained dual-path decoupled multi-view encoder and labeled spatiotemporal samples; Inputting the labeled spatiotemporal samples into the pre-trained dual-path decoupled multi-view encoder and the initial classifier for iterative fine-tuning training; The pre-trained two-way decoupled multi-view encoder corresponding to the stop condition is used as the final two-way decoupled multi-view encoder; The initial classifier corresponding to the stopping condition is used as the pre-trained classifier; The final two-way decoupled multi-view encoder and the pre-trained classifier together constitute the pre-trained two-way decoupled multi-view model.

Images