Emotion closed-loop regulation system and method based on electroencephalogram feedback

Abstract

The application is specifically an emotion closed-loop regulation system and method based on electroencephalogram feedback, relates to the technical field of brain-computer interface and emotion computing, and comprises: inputting the multi-dimensional emotion hidden variable vector as a condition into a generator of a conditional generative adversarial network to generate audio-visual content consistent with the semantic of the current emotion state.In the application, the multi-dimensional emotion hidden variable vector is input as a condition to drive the conditional generative adversarial network to generate audio-visual content, the technical problem that abstract electroencephalogram emotion features cannot be directly converted into user-perceptible sensory stimulation is solved, and the technical effect that the electroencephalogram emotion features are mapped into sensory content with specific audio-visual attributes is achieved.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interface and emotion computing technology, and in particular to an emotion closed-loop regulation system and method based on electroencephalographic feedback. Background Technology

[0002] With the development of brain-computer interface technology, electroencephalogram (EEG) signals are widely used to reflect users' real-time emotional states. However, the emotional characteristics contained in EEG signals are highly abstract, and users cannot directly perceive and understand the emotional states represented by their own brainwaves, resulting in an information gap between EEG signals and user-perceptible content.

[0003] In the existing technology, content generation methods based on generative adversarial networks have been applied to emotion-related scenarios. These methods generate content with a certain degree of visual or auditory fidelity through adversarial training.

[0004] However, existing methods mainly focus on the visual or auditory fidelity of the generated content, lacking effective constraints on the causal mapping relationship between EEG emotional features and the generated audiovisual content, resulting in a discrepancy between the emotional semantics of the generated content and the emotional state decoded by EEG.

[0005] Furthermore, due to significant differences in emotion perception and aesthetic preferences among individuals, generative models with fixed parameters are difficult to adapt to the emotion regulation needs of different users. Moreover, during continuous closed-loop regulation, problems such as mapping drift or drastic content changes are prone to occur, affecting the stability and effectiveness of emotion feedback regulation and making it impossible to form a stable personalized closed-loop emotion regulation circuit. Summary of the Invention

[0006] The purpose of this invention is to propose an emotion closed-loop regulation system and method based on electroencephalogram (EEG) feedback in order to solve the above-mentioned problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] EEG-feedback-based closed-loop emotion regulation methods include:

[0009] The user's raw EEG signal is collected, and after noise reduction and artifact removal, frequency domain features and time-frequency domain features are extracted to obtain a multidimensional emotion latent variable vector. The Z-score standardization is then applied to the values ​​of each dimension of the multidimensional emotion latent variable vector.

[0010] The multidimensional emotion latent variable vector is used as a conditional input to the generator of the conditional generative adversarial network to generate audiovisual content that is semantically consistent with the current emotion state; wherein, the discriminator of the conditional generative adversarial network includes an authenticity discrimination branch and an emotion consistency discrimination branch, and the generator and the discriminator are optimized for parameters through adversarial training;

[0011] The audiovisual content is input into a pre-trained emotion recognition teacher network to obtain a virtual emotion feature vector. The cycle consistency loss value between the multidimensional emotion latent variable vector and the virtual emotion feature vector is calculated. The cycle consistency loss value is backpropagated to the generator to update the generator's parameters.

[0012] The audiovisual content is presented to the user, and the user's post-intervention EEG signal is collected within a preset feedback time window. The post-intervention EEG signal is processed in the same way as the original EEG signal to obtain the post-intervention emotion latent variable vector. The reward signal is calculated based on the difference between the post-intervention emotion latent variable vector and the multidimensional emotion latent variable vector and the preset target emotion regulation direction.

[0013] Based on the reward signal, the weights of the fully connected layers related to the emotion mapping in the generator are updated using the policy gradient algorithm, and the updated generator is applied to the next closed-loop iteration.

[0014] Preferably, each dimension of the multidimensional emotion latent variable vector corresponds to a different emotion representation index, including valence index, arousal index, tension index, relaxation depth index, attention margin index, and emotion stability entropy value; wherein, the valence index is calculated based on the asymmetry of the power of alpha waves in the left and right frontal lobes, the arousal index is calculated based on the average power spectral density of beta waves in the whole scalp, the tension index is calculated based on the ratio of the power of theta waves to alpha waves in the frontal lobe, the relaxation depth index is calculated based on the power spectral density of alpha waves in the parieto-occipital lobe, the attention margin index is calculated based on the ratio of the power of beta waves to theta waves in the frontal lobe, and the emotion stability entropy value is calculated based on the nonlinear dynamic index of EEG signals.

[0015] Preferably, after obtaining the multidimensional emotion latent variable vector, the method further includes the following steps:

[0016] The preprocessed EEG signal is segmented into a time window of K seconds to obtain an EEG time sequence. The EEG time sequence is then input into a Long Short-Term Memory (LSTM) network, which outputs a time-series encoded feature vector containing EEG context information. Here, K is the preset time window length, ranging from 2 to 10 seconds.

[0017] The temporal encoded feature vector is concatenated with the multidimensional emotion latent variable vector to obtain an emotion feature vector that integrates temporal context, which serves as the conditional input to the generator of the conditional generative adversarial network.

[0018] Preferably, the introduction of cross-attention processing during the generation process of the generator specifically includes:

[0019] The conditional input is used as the key vector and value vector in the cross-attention processing. The feature map of the intermediate layer of the generator is flattened and used as the query vector. The attention weight matrix between the conditional input and the intermediate layer feature map is calculated through the scaling dot product attention mechanism. The intermediate layer feature map is weighted and modulated based on the attention weight matrix to obtain a modulated feature map that integrates EEG emotion feature guidance information. The modulated feature map is input to the subsequent layers of the generator for processing and outputs the audiovisual content.

[0020] Preferably, the addition of a physiological feedback prior constraint loss term to the generator's loss function is performed as follows: extracting normalized attribute value vectors of physiologically related sensory attributes from the audiovisual content, querying a preset physiological and psychological rule base based on the current EEG feature state to obtain a target attribute value vector, calculating the square of the Euclidean norm between the normalized attribute value vector and the target attribute value vector, multiplying it by the weight coefficient of the prior constraint, and obtaining the physiological feedback prior constraint loss term.

[0021] Wherein, when the audiovisual content is an image, the sensory attributes include the average saturation value of the image in the HSV color space and the proportion of low-frequency components in the power spectrum of the image after two-dimensional Fourier transform; when the audiovisual content is audio, the sensory attributes include the power proportion of the audio signal in the frequency range of 0.1 to 4 Hz after short-time Fourier transform.

[0022] Preferably, an orthogonal regularization loss term is added to the loss function of the generator. The orthogonal regularization loss term is calculated as follows: the modulation vectors corresponding to each dimension of the multidimensional emotion latent variable vector are arranged in columns to form a modulation vector matrix; the product of the transpose of the modulation vector matrix and the modulation vector matrix is ​​calculated, and then the identity matrix of the same order is subtracted; the square of the Frobenius norm of the resulting difference matrix is ​​taken to obtain the orthogonal regularization loss term; the orthogonal regularization loss term is added to the total loss function of the generator for joint optimization, so that the modulation vectors of different EEG feature dimensions are orthogonal to each other in the intermediate latent space.

[0023] Preferably, the reward signal is calculated as follows: when the target emotion regulation direction is to reduce the user's tension, the increase in the prefrontal alpha wave power and the improvement in the left and right frontal asymmetry index of the EEG signal after intervention are obtained compared to before intervention. The sum of the increase multiplied by a first weighting coefficient and the improvement multiplied by a second weighting coefficient is used as the reward signal; wherein, the increase and the improvement are both calculated based on the EEG feature values ​​after Z-score standardization, and the first weighting coefficient and the second weighting coefficient are both preset positive real numbers.

[0024] Preferably, after calculating the reward signal, the method further includes the following step:

[0025] Within the preset feedback time window, the user's heart rate variability data is collected by the heart rate sensor, and the user's skin conductance response data is collected by the skin conductance sensor.

[0026] The heart rate variability data and skin conductance response data are compared with the emotional state offset vector calculated from the electroencephalogram (EEG) signal. When the trend of emotional change indicated by the auxiliary physiological signal is consistent with the trend indicated by the emotional state offset vector, the reward signal is confirmed to be valid. When the two are inconsistent, the confidence weight of the reward signal is reduced.

[0027] Preferably, after each generator parameter update, the step further includes the following step:

[0028] Calculate the perceptual distance between the audiovisual content generated in the current round and the audiovisual content generated in the previous round. The perceptual distance is the sum of the weighted Euclidean distances between the feature representations of each layer in the pre-trained perceptual feature extraction network for the two generated contents. The weights of each feature representation are preset weights.

[0029] When the perceived distance value exceeds the preset smoothing threshold, a KL divergence penalty term is applied to the latent vector adjustment of the generator to compress the adjustment range of the current round, so that the audiovisual content generated in two consecutive rounds maintains a smooth transition at the perception level.

[0030] EEG-feedback-based closed-loop emotion regulation systems include:

[0031] The EEG signal acquisition and feature extraction module is used to acquire the user's raw EEG signal, and after denoising and artifact removal processing of the raw EEG signal, extract frequency domain features and time-frequency domain features to obtain a multidimensional emotion latent variable vector processed by Z-score normalization.

[0032] The audiovisual content generation module is used to input the multidimensional emotion latent variable vector as a condition into the generator of the conditional generative adversarial network to generate audiovisual content that is semantically consistent with the current emotion state.

[0033] The cross-modal consistency constraint module is used to input the audiovisual content into a pre-trained emotion recognition teacher network to obtain a virtual emotion feature vector, calculate the cycle consistency loss value between the multidimensional emotion latent variable vector and the virtual emotion feature vector, and backpropagate the cycle consistency loss value to the generator to update the parameters.

[0034] The emotion regulation effect quantification module is used to collect post-intervention EEG signals and obtain post-intervention emotion latent variable vectors within a preset feedback time window after presenting the audiovisual content to the user, and calculate the reward signal based on the difference between the post-intervention emotion latent variable vector and the multidimensional emotion latent variable vector and the preset target emotion regulation direction.

[0035] The generator online update module is used to update the weights of the fully connected layers related to the emotion mapping in the generator based on the reward signal using the policy gradient algorithm, and apply the updated generator to the next round of closed-loop iteration.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] 1. This invention solves the technical problem that abstract EEG emotional features cannot be directly converted into sensory stimuli that users can perceive by using multidimensional emotion latent variable vectors as conditional inputs to drive conditional generative adversarial networks to generate audiovisual content. It achieves the technical effect of mapping EEG emotional features into sensory content with specific audiovisual attributes.

[0038] 2. This invention solves the technical problem of deviation and mapping drift between the generated content and the original EEG emotional feature vector by constraining the consistency between the generated content emotional semantics and the EEG decoding results through bidirectional cyclic consistency verification. It achieves the technical effect of maintaining a stable causal mapping relationship between the generated content emotional semantics and the EEG decoding results. Through physiological feedback prior constraints and latent space orthogonal regularization, it solves the technical problem of the lack of physiological and psychological basis for the sensory attributes of the generated content and the mutual coupling of the control of various emotional dimensions. Attached Figure Description

[0039] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0040] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0041] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0042] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0043] Example 1

[0044] Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0045] In this embodiment, it includes:

[0046] Step 1: Acquire raw EEG signals and extract multidimensional emotion latent variable vectors.

[0047] The user's raw brainwave signals are collected in real time using a multi-channel EEG acquisition device. ,right Noise reduction and artifact removal are performed to obtain preprocessed EEG signals. Frequency domain and time-frequency domain features are extracted from the preprocessed EEG signals to obtain a multidimensional emotion latent variable vector. ,in Let be the number of dimensions of the latent variables related to emotion. express A multidimensional real vector space. After extracting the feature values ​​of each dimension, the multidimensional emotion latent variable vector... The values ​​of each dimension are Z-score standardized to give each dimension a uniform scale, thereby eliminating the impact of differences in scale between different emotion representation indicators on subsequent vector operations and network input.

[0048] It should be noted that the above multidimensional emotion latent variable vector Each dimension corresponds to a different emotional representation index, including but not limited to the valence index, arousal index, tension index, relaxation depth index, attention margin index, and emotional stability entropy. The values ​​of each dimension are calculated by analyzing the power spectral density and nonlinear dynamic characteristics of the preprocessed EEG signals in the corresponding frequency bands. Specifically, the valence index is calculated based on the asymmetry of power in the left and right frontal alpha waves (8 to 13 Hz); the arousal index is calculated based on the average power spectral density of the whole scalp beta waves (13 to 30 Hz); the tension index is calculated based on the ratio of the power of the frontal theta waves (4 to 8 Hz) to the power of the alpha waves; the relaxation depth index is calculated based on the power spectral density of the parietal-occipital alpha waves; the attention margin index is calculated based on the ratio of the power of the frontal beta waves to the theta waves; and the emotional stability entropy is calculated based on nonlinear dynamic indicators such as the approximate entropy or sample entropy of the EEG signals.

[0049] Furthermore, to ensure the continuity of subsequently generated content in the time series, the following steps are also included: processing the preprocessed EEG signals according to the preceding... The time window is divided into seconds to obtain a sequence of EEG time segments. The sequence of EEG time segments is then input into a Long Short-Term Memory (LSTM) network, which outputs a temporal encoded feature vector containing EEG context information. ,in The preset time window length. Long Short-Term Memory (LSTM) networks use feature vectors from each time step in the EEG temporal sequence as input, incorporating previous data... The time-series encoded feature vectors representing second-order dependencies are used as outputs. The algorithm is trained using backpropagation combined with a mean squared error loss function, and optimized using the Adam optimization algorithm. With multidimensional emotion latent variable vector The data is concatenated to obtain a sentiment feature vector that incorporates the temporal context. This serves as the conditional input for subsequent generation steps.

[0050] Furthermore, the length of the aforementioned time window The value ranges from 2 to 10 seconds, and the specific value is determined based on the response latency requirements of the target emotion regulation scenario: when a low system response latency to emotion changes is required, a smaller value is chosen. A larger value is used to reduce the latency introduced by time-series coding; when time-series coding is required to capture long-term emotional dynamics, a larger value is used. Values ​​are used to cover a longer historical EEG context.

[0051] Step 2: Generate audiovisual content with consistent emotional semantics based on conditional generative adversarial networks.

[0052] Multidimensional emotion latent variable vector (or sentiment feature vectors that integrate temporal context) When the temporal encoding in step 1 is performed, it is used as a conditional input to the generator of the conditional generative adversarial network to generate audiovisual content that is semantically consistent with the current emotional state. .

[0053] The aforementioned conditional generative adversarial network consists of two components: a generator and a discriminator.

[0054] The generator takes conditional input and a random noise vector as input, and processes them through multiple upsampling and feature transformations to generate audiovisual content. As output, the discriminator uses the generated audiovisual content as its output. The input can be real natural audiovisual content samples, with the outputs being the results of judging the authenticity of the input content and the results of judging its emotional consistency. The generator and discriminator optimize their parameters through adversarial training. The specific process of adversarial training is as follows: In each iteration, the generator parameters are first fixed, and the discriminator parameters are updated with real sample labels of 1 and generated sample labels of 0, enabling the discriminator to accurately distinguish between real and generated content; then, the discriminator parameters are fixed again, and the generator parameters are updated with generated sample labels of 1, enabling the generator to generate outputs that the discriminator might misjudge as real content. This process is repeated iteratively until convergence. The optimization strategy is the Adam optimization algorithm. The output layer of the discriminator's authenticity judgment branch is a fully connected layer that outputs a single scalar authenticity score, using the least squares loss function; the output layer of the emotional consistency judgment branch is a fully connected layer that outputs the probability distribution of emotional categories, using the cross-entropy loss function.

[0055] It should be noted that when the generated audiovisual content is an image, the generator described above employs a style modulation-based generative architecture. This architecture comprises two sub-units: a mapping network and a synthesis network. The mapping network receives the conditional input and maps it into an intermediate latent space vector through a multi-layer fully connected transformation. The synthetic network is based on the intermediate latent space vector Affine transformations are performed on the feature maps of each layer to implement style modulation, controlling visual attributes such as the color temperature, line smoothness, and element density of the generated image. When the generated audiovisual content is audio, the generator adopts a Mel-spectrum-based audio generation architecture. This architecture receives conditional inputs as conditional signals embedded in the batch normalization layer, controlling auditory attributes such as the frequency distribution, rhythm, and harmonic richness of the generated waveform.

[0056] It should be noted that the discriminator described above includes a authenticity discrimination branch and an emotion consistency discrimination branch. The authenticity discrimination branch is used to determine the degree of difference between the image or audio distribution of the input content and the distribution of real natural samples. The emotion consistency discrimination branch is an auxiliary classifier that receives the input content and its corresponding target emotion label (the target emotion label is obtained based on EEG signal decoding), and determines whether the emotional semantics of the input content is consistent with the target emotion label.

[0057] Furthermore, to establish explicit associations between specific visual or auditory elements in the generated content and the numerical values ​​of various dimensions of EEG features, cross-attention processing is introduced during the generator's generation process. Specifically, the conditional input is used as the key and value vectors in the cross-attention processing, and the flattened feature map of the generator's intermediate layer is used as the query vector. An attention weight matrix between the conditional input and the intermediate layer feature map is calculated using a scaling dot product attention mechanism. Based on the attention weight matrix, the intermediate layer feature map is weighted and modulated to obtain a modulated feature map that incorporates EEG emotional feature guidance information. This modulated feature map is then input into subsequent layers of the generator for processing, outputting the generated audiovisual content. Through the aforementioned cross-attention processing, an explicit mapping relationship is formed between the visual attributes of the image elements in the generated content (such as the size, speed of movement, and degree of deformation of a specific graphic) and the corresponding dimensions in the conditional input.

[0058] Step 3: Perform cross-modal consistency constraint verification on the generated audiovisual content.

[0059] The generated audiovisual content The input is fed into a pre-trained emotion recognition teacher network to obtain the generated audiovisual content. The corresponding virtual emotion feature vector The virtual emotion feature vector output by the pre-trained emotion recognition teacher network mentioned above. Compared with the multidimensional latent emotion variable vector after Z-score standardization in step 1 Since they reside in the same standardized feature space to ensure consistency in their dimensions, vector similarity calculations can be performed directly. This involves calculating the multidimensional emotion latent variable vector. With virtual emotion feature vector Cycle consistency loss value The calculation formula is:

[0060] ;

[0061] in, Let be the dot product of two vectors. and Let be the Euclidean norms of the two vectors. Then, consider the cycle consistency loss value. The constraint loss signal is backpropagated to the generator in step 2 to update the generator's parameters. During the backpropagation process described above, the cycle consistency loss value... The gradient is sequentially passed through a pre-trained emotion recognition teacher network (with fixed parameters, only the gradient is passed) and the generated audiovisual content. Ultimately, the trainable parameters passed to the generator drive the generator to adjust the generated content to express its emotional semantics into a multi-dimensional emotional latent variable vector. near.

[0062] The above process constitutes a bidirectional cyclic consistency verification. The forward pathway is: raw EEG signals. The multidimensional emotion latent variable vector is obtained after processing in step 1. The generated audiovisual content is obtained after processing in step 2. The reverse prediction path is: to generate audiovisual content... The input is fed into a pre-trained emotion recognition teacher network to obtain virtual emotion feature vectors. The pre-trained emotion recognition teacher network mentioned above is an emotion classification or regression network pre-trained based on a large-scale emotion-annotated audiovisual dataset. The input of the pre-trained emotion recognition teacher network is the audiovisual content, and the output is the emotion feature vector that the audiovisual content should evoke. The parameters of the pre-trained emotion recognition teacher network are fixed and do not participate in the update during this closed-loop feedback process.

[0063] Furthermore, in order to incorporate prior knowledge from the fields of color psychology and audio psychoacoustics to constrain the generated content, the following steps are also included: adding a physiological feedback prior constraint loss term to the generator's loss function in step 2. The calculation formula is:

[0064] ;

[0065] in, This is a function for extracting physiologically relevant sensory attributes from generated content. This is a vector of target attribute values ​​obtained from a pre-defined physiological and psychological rule base based on EEG feature state queries. These are the weighting coefficients of the prior constraints. It is the Euclidean norm. and All are dimensionless normalized attribute value vectors; specifically, for The extracted raw values ​​of each sensory attribute and the corresponding raw values ​​of the target attribute in the rule base are all mapped to the mean normalization method based on the range. Substitute the values ​​into the above formula after dividing the interval to ensure that the dimensions of the two are consistent, and then you can directly perform the difference calculation. The specific calculation method is as follows: when the generated audiovisual content When it is an image, Extract the mean saturation value of the image in the HSV color space and the proportion of low-frequency components in the power spectrum after the image's two-dimensional Fourier transform, and output an attribute vector composed of the above two normalized scalars; when the generated audiovisual content When it is audio, A short-time Fourier transform is performed on the audio signal to extract the power proportion within the frequency range of 0.1 to 4 Hz as the binaural beat component intensity, and the normalized scalar value is output. Specifically, when the multidimensional emotion latent variable vector... The frequency band characteristics indicate high Beta wave power and low Alpha wave power (corresponding to an anxiety state). The generated image is constrained to have a hue saturation lower than a preset saturation threshold, and its spatial frequency is concentrated in the low-frequency region; when the multidimensional emotion latent variable vector The frequency band characteristics indicate when the Theta wave power increases (corresponding to the relaxed state). The constrained generated audio contains binaural beat components with frequencies ranging from 0.1 to 4 Hz.

[0066] Furthermore, the aforementioned weighting coefficients The value ranges from 0.01 to 1.0, and the specific value is determined based on the relative proportion between the prior constraints and the adversarial training loss: when the physiological feedback prior constraint loss term... When the magnitude of the loss is significantly greater than that of the adversarial training loss, reduce the weight coefficient. To prevent prior constraints from dominating the generator's optimization direction; when the guiding effect of prior constraints is insufficient, increase the weight coefficients. This is to strengthen the constraints of physiological and psychological rules on the sensory attributes of generated content.

[0067] Furthermore, to ensure that each dimension of the multidimensional emotion latent variable vector can independently control different attributes of the generated content and avoid coupling interference between dimensions, the following steps are also included in step 2: In the generator's mapping network, the multidimensional emotion latent variable vector... Orthogonal regularization constraints are applied to the modulation vectors corresponding to each dimension. Specifically, the matrix composed of the modulation vectors of each dimension is calculated. Orthogonal regularization loss:

[0068] ;

[0069] in, It is a matrix composed of modulation vectors of each dimension arranged column by column. This represents the matrix transpose operation. To and Identity matrices of the same order The Frobenius norm is used. The orthogonal regularization loss is then applied. By incorporating the overall loss function of the generator into joint optimization, the modulation vectors of different EEG feature dimensions are made orthogonal to each other in the intermediate latent space, so that changes in relaxation depth will not unexpectedly alter other attributes such as image brightness.

[0070] Step 4: Collect EEG signals after intervention and quantify the effect of emotion regulation.

[0071] The generated audiovisual content After being presented to the user via display and audio playback devices, the user's post-intervention EEG signals are collected using a multi-channel EEG acquisition device within a preset feedback time window. The post-intervention EEG signals undergo the same preprocessing, feature extraction, and Z-score standardization operations as in step 1 to obtain the post-intervention emotion latent variable vector. Calculate the vector of latent emotion variables after intervention. Compared with the multidimensional latent emotion vector before intervention The difference between them yields the emotional state offset vector. Based on emotional state offset vector Calculate the reward signal based on the preset target emotion regulation direction. .

[0072] It should be noted that the aforementioned preset feedback time window is a time interval of 2 to 5 seconds after the audiovisual content is presented. The aforementioned reward signal... The calculation method is related to the preset target emotion regulation direction. When the target is to reduce the user's tension, the reward signal... The calculations were based on the increase in prefrontal alpha wave power and the improvement in the left-right frontal asymmetry index in the EEG signals after intervention. The increase in prefrontal alpha wave power after intervention compared to before intervention. The improvement in the left and right frontal asymmetry index after intervention compared to before intervention is represented by two values. Both are calculated based on EEG feature values ​​standardized by Z-score, possessing a unified dimensional scale and allowing for direct weighted summation. (Reward signal) The calculation formula is:

[0073] ;

[0074] in, and These are the corresponding weighting coefficients, both of which are preset positive real numbers used to balance the contribution ratio of the two indicators to the reward signal. When The larger or The larger the value, the stronger the reward signal. The larger the value, the better.

[0075] Furthermore, the aforementioned reward signals Time parameters in This indicates the iteration number of the current closed-loop iteration, i.e., the [number]th iteration. The reward signal value corresponding to the next closed-loop iteration cycle is used to distinguish the rewards of different iteration rounds in the policy gradient algorithm, so as to support the cross-round calculation of the cumulative reward in step 5.

[0076] Furthermore, to reduce the impact of artifacts in EEG signals on the accuracy of reward signals, the following steps are also included: collecting the user's heart rate variability data via a heart rate sensor within a preset feedback time window. The skin conductance data of the user is collected through a skin conductance sensor. Heart rate variability data and skin conductance response data Emotional state offset vector calculated from EEG signals Perform a consistency comparison. When the trend of emotion changes indicated by auxiliary physiological signals matches the emotion state offset vector... Confirm reward signals when the indicated trend is consistent. Effective; when the two are inconsistent, reduce the reward signal. The confidence weight or the reward signal of the current feedback cycle is marked as pending review.

[0077] Step 5: Update the generator parameters online based on the reward signal.

[0078] Based on reward signals The policy gradient algorithm is used to update the weights of the fully connected layers directly related to the sentiment mapping in the generator from step 2, and the updated generator is output. The input to the policy gradient algorithm is the current multidimensional sentiment latent variable vector. and reward signals The adjustment of the weights of the fully connected layers in the generator that are directly related to the emotion mapping is taken as the action in the continuous action space, and the reward signal is used. As the reward for this action, the gradient update direction and magnitude of the fully connected layer weights are output, thus obtaining the generator with updated parameters. Specifically, the policy gradient algorithm aims to maximize the cumulative reward, which is the reward for the current round. Reward signals for subsequent rounds The cumulative discount value, discount factor The weighting used to control the decay of future rewards relative to current rewards is calculated by the reward signal. The gradient of the fully connected layer weights is used to update the weights along the gradient ascent direction, making the generator more inclined to generate audiovisual content that yields higher rewards in subsequent iterations. The updated generator is then applied to step 2 of the next closed-loop iteration to generate adjusted audiovisual content from the newly acquired EEG signals.

[0079] Furthermore, to reduce the number of closed-loop iterations required in the online adaptation phase, the following steps are included in addition to step 5: In the offline phase, based on historical EEG data and emotional feedback data from multiple different users, the generator's initialization weights are pre-trained using a model-independent meta-learning algorithm to obtain generator initialization weights sensitive to individual differences. The input to the model-independent meta-learning algorithm consists of a small number of EEG emotional feature samples and corresponding emotional feedback data from multiple users, and the output is a set of generator initialization weights. These generator initialization weights have the characteristic of adapting to the individual aesthetic preferences and emotional mapping differences of new users through a small number of closed-loop iterations. In the online phase, the generator initialization weights are used as the starting parameters of the generator, and the parameter update operation in step 5 is executed.

[0080] Furthermore, to prevent drastic changes in generated content during continuous closed-loop iterations from causing unexpected fluctuations in user emotions, the following steps are added to step 5: After each generator parameter update, the perceptual distance between the audiovisual content generated in the current round and the audiovisual content generated in the previous round is calculated. This perceptual distance is obtained by calculating the weighted Euclidean distance between the feature representations of the two generated contents in each layer of the pre-trained perceptual feature extraction network. Specifically, let the perceptual distance between the generated content in the current round and the perceptual feature extraction network be... The features of a layer are represented as The feature representation of the content generated in the previous round at the same layer is as follows: Then the perceived distance value The calculation formula is:

[0081] ;

[0082] in, For the first Preset weights for layer feature representation, Layer indexes for the perceptual feature extraction network. Given the Euclidean norm, summation is performed across all selected layers of the perceptual feature extraction network. When the perceptual distance value... When the preset smoothing threshold is exceeded, a KL divergence penalty term is applied to the latent vector adjustment of the generator to compress the adjustment range of the current round, so that the audiovisual content generated in two consecutive rounds maintains a smooth transition at the perceptual level.

[0083] Furthermore, the above-mentioned preset smoothing threshold is determined as follows: In the offline stage, based on the subjective evaluation experiment of multiple test users, the test users' acceptableness of the perceived change between two consecutive frames of generated content is statistically analyzed. The minimum value of the perceived distance value that the test users rated as "the change is too abrupt" is taken as the upper limit of the smoothing threshold, and the maximum value of the perceived distance value that the test users rated as "the change is smooth and natural" is taken as the reference value of the smoothing threshold. A specific threshold is selected from this range and then fixed for use in the online stage.

[0084] It should be noted that steps 1 to 5 above constitute a complete closed-loop iterative cycle. In actual operation, steps 1 to 5 are executed cyclically, and the generator used in step 2 of each iteration is the generator updated by step 5 of the previous iteration, thus forming a continuous adaptive emotion feedback adjustment process.

[0085] Example 2

[0086] The EEG-feedback-based closed-loop emotion regulation system includes the following components:

[0087] The EEG signal acquisition and feature extraction module is used to acquire the user's raw EEG signal, and after denoising and artifact removal of the raw EEG signal, extract frequency domain features and time-frequency domain features to obtain a multidimensional emotion latent variable vector after Z-score normalization.

[0088] The audiovisual content generation module is used to input the multidimensional emotion latent variable vector as a condition into the generator of the conditional generative adversarial network to generate audiovisual content that is semantically consistent with the current emotion state.

[0089] The cross-modal consistency constraint module is used to input audiovisual content into a pre-trained emotion recognition teacher network to obtain virtual emotion feature vectors, calculate the cycle consistency loss value between the multidimensional emotion latent variable vector and the virtual emotion feature vector, and backpropagate the cycle consistency loss value to the generator to update the parameters.

[0090] The emotion regulation effect quantification module is used to collect post-intervention EEG signals and obtain post-intervention emotion latent variable vectors within a preset feedback time window after presenting audiovisual content to the user. The reward signal is calculated based on the difference between the post-intervention emotion latent variable vector and the multidimensional emotion latent variable vector and the preset target emotion regulation direction.

[0091] The generator online update module is used to update the weights of the fully connected layers related to the sentiment mapping in the generator based on the reward signal and using the policy gradient algorithm, and then apply the updated generator to the next closed-loop iteration.

[0092] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0093] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0094] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0095] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A closed-loop emotion regulation method based on EEG feedback, characterized in that, include: The user's raw EEG signal is collected, and after noise reduction and artifact removal, frequency domain features and time-frequency domain features are extracted to obtain a multidimensional emotion latent variable vector. The Z-score standardization is then applied to the values ​​of each dimension of the multidimensional emotion latent variable vector. The multidimensional emotion latent variable vector is used as a conditional input to the generator of the conditional generative adversarial network to generate audiovisual content that is semantically consistent with the current emotion state; wherein, the discriminator of the conditional generative adversarial network includes an authenticity discrimination branch and an emotion consistency discrimination branch, and the generator and the discriminator are optimized for parameters through adversarial training; The audiovisual content is input into a pre-trained emotion recognition teacher network to obtain a virtual emotion feature vector. The cycle consistency loss value between the multidimensional emotion latent variable vector and the virtual emotion feature vector is calculated. The cycle consistency loss value is backpropagated to the generator to update the generator's parameters. The audiovisual content is presented to the user, and the user's post-intervention EEG signal is collected within a preset feedback time window. The post-intervention EEG signal is processed in the same way as the original EEG signal to obtain the post-intervention emotion latent variable vector. The reward signal is calculated based on the difference between the post-intervention emotion latent variable vector and the multidimensional emotion latent variable vector and the preset target emotion regulation direction. Based on the reward signal, the weights of the fully connected layers related to the emotion mapping in the generator are updated using the policy gradient algorithm, and the updated generator is applied to the next closed-loop iteration.

2. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, Each dimension of the multidimensional emotion latent variable vector corresponds to a different emotion representation index, including valence index, arousal index, tension index, relaxation depth index, attention margin index, and emotion stability entropy. Specifically, the valence index is calculated based on the asymmetry of alpha wave power in the left and right frontal lobes; the arousal index is calculated based on the average power spectral density of beta waves across the entire scalp; the tension index is calculated based on the ratio of theta wave power to alpha wave power in the frontal lobe; the relaxation depth index is calculated based on the alpha wave power spectral density in the parieto-occipital lobe; the attention margin index is calculated based on the ratio of beta wave power to theta wave power in the frontal lobe; and the emotion stability entropy is calculated based on the nonlinear dynamics of EEG signals.

3. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, After obtaining the multidimensional emotion latent variable vector, the following steps are also included: The preprocessed EEG signal is segmented into a time window of K seconds to obtain an EEG time sequence. The EEG time sequence is then input into a Long Short-Term Memory (LSTM) network, which outputs a time-series encoded feature vector containing EEG context information. Here, K is the preset time window length, ranging from 2 to 10 seconds. The temporal encoded feature vector is concatenated with the multidimensional emotion latent variable vector to obtain an emotion feature vector that integrates temporal context, which serves as the conditional input to the generator of the conditional generative adversarial network.

4. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, Cross-attention processing is introduced during the generation process of the generator, specifically including: The conditional input is used as the key vector and value vector in the cross-attention processing. The feature map of the intermediate layer of the generator is flattened and used as the query vector. The attention weight matrix between the conditional input and the intermediate layer feature map is calculated through the scaling dot product attention mechanism. The intermediate layer feature map is weighted and modulated based on the attention weight matrix to obtain a modulated feature map that integrates EEG emotion feature guidance information. The modulated feature map is input to the subsequent layers of the generator for processing and outputs the audiovisual content.

5. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, A physiological feedback prior constraint loss term is added to the loss function of the generator. The physiological feedback prior constraint loss term is calculated as follows: extract the normalized attribute value vector of physiologically related sensory attributes from the audiovisual content, query the preset physiological and psychological rule base based on the current EEG feature state to obtain the target attribute value vector, calculate the square of the Euclidean norm between the normalized attribute value vector and the target attribute value vector, multiply it by the weight coefficient of the prior constraint, and obtain the physiological feedback prior constraint loss term. Wherein, when the audiovisual content is an image, the sensory attributes include the average saturation value of the image in the HSV color space and the proportion of low-frequency components in the power spectrum of the image after two-dimensional Fourier transform; when the audiovisual content is audio, the sensory attributes include the power proportion of the audio signal in the frequency range of 0.1 to 4 Hz after short-time Fourier transform.

6. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, An orthogonal regularization loss term is added to the loss function of the generator. The orthogonal regularization loss term is calculated as follows: the modulation vectors corresponding to each dimension of the multidimensional emotion latent variable vector are arranged in columns to form a modulation vector matrix. The product of the transpose of the modulation vector matrix and the modulation vector matrix is ​​calculated, and then the identity matrix of the same order is subtracted. The square of the Frobenius norm of the resulting difference matrix is ​​taken to obtain the orthogonal regularization loss term. The orthogonal regularization loss term is added to the total loss function of the generator for joint optimization, so that the modulation vectors of different EEG feature dimensions are orthogonal to each other in the intermediate latent space.

7. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, The reward signal is calculated as follows: when the target emotion regulation direction is to reduce the user's tension, the increase in the prefrontal alpha wave power and the improvement in the left and right frontal asymmetry index of the EEG signal after intervention are obtained compared to before intervention. The sum of the increase multiplied by a first weighting coefficient and the improvement multiplied by a second weighting coefficient is used as the reward signal. The increase and the improvement are calculated based on the EEG feature values ​​after Z-score standardization, and the first and second weighting coefficients are preset positive real numbers.

8. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, After calculating the reward signal, the following steps are also included: Within the preset feedback time window, the user's heart rate variability data is collected by the heart rate sensor, and the user's skin conductance response data is collected by the skin conductance sensor. The heart rate variability data and skin conductance response data are compared with the emotional state offset vector calculated from the electroencephalogram (EEG) signal. When the trend of emotional change indicated by the auxiliary physiological signal is consistent with the trend indicated by the emotional state offset vector, the reward signal is confirmed to be valid. When the two are inconsistent, the confidence weight of the reward signal is reduced.

9. The emotion closed-loop regulation method based on EEG feedback according to claim 1, characterized in that, After each generator parameter update, the following steps are also included: Calculate the perceptual distance between the audiovisual content generated in the current round and the audiovisual content generated in the previous round. The perceptual distance is the sum of the weighted Euclidean distances between the feature representations of each layer in the pre-trained perceptual feature extraction network for the two generated contents. The weights of each feature representation are preset weights. When the perceived distance value exceeds the preset smoothing threshold, a KL divergence penalty term is applied to the latent vector adjustment of the generator to compress the adjustment range of the current round, so that the audiovisual content generated in two consecutive rounds maintains a smooth transition at the perception level.

10. An emotion closed-loop regulation system based on EEG feedback, wherein the emotion closed-loop regulation method based on EEG feedback according to any one of claims 1-9 is characterized in that, include: The EEG signal acquisition and feature extraction module is used to acquire the user's raw EEG signal, and after denoising and artifact removal processing of the raw EEG signal, extract frequency domain features and time-frequency domain features to obtain a multidimensional emotion latent variable vector processed by Z-score normalization. The audiovisual content generation module is used to input the multidimensional emotion latent variable vector as a condition into the generator of the conditional generative adversarial network to generate audiovisual content that is semantically consistent with the current emotion state. The cross-modal consistency constraint module is used to input the audiovisual content into a pre-trained emotion recognition teacher network to obtain a virtual emotion feature vector, calculate the cycle consistency loss value between the multidimensional emotion latent variable vector and the virtual emotion feature vector, and backpropagate the cycle consistency loss value to the generator to update the parameters. The emotion regulation effect quantification module is used to collect post-intervention EEG signals and obtain post-intervention emotion latent variable vectors within a preset feedback time window after presenting the audiovisual content to the user, and calculate the reward signal based on the difference between the post-intervention emotion latent variable vector and the multidimensional emotion latent variable vector and the preset target emotion regulation direction. The generator online update module is used to update the weights of the fully connected layers related to the emotion mapping in the generator based on the reward signal using the policy gradient algorithm, and apply the updated generator to the next round of closed-loop iteration.

Images