Cross-subject five-channel emotion modeling and emotion regulation method, device and storage medium

Abstract

The application discloses a cross-subject five-channel emotion modeling and emotion regulation method, equipment and a storage medium, and the method comprises the following steps: when playing multimedia data for a user, calling a brain-computer interface to collect original electroencephalogram signals of the user under O1, O2, C3, C4 and FP2 channels; determining actual emotions induced by the multimedia data to the user; preprocessing the original electroencephalogram signals to obtain target electroencephalogram signals; dividing the target electroencephalogram signals into a plurality of electroencephalogram segment signals; under the attention of O1, O2, C3, C4 and FP2 channels, training an emotion classifier by taking the electroencephalogram segment signals as samples and the actual emotions as labels. According to the emotion classification evaluation result, visual and auditory feedback is provided, and the user adjusts / trains emotions according to the feedback. The embodiment greatly reduces the number of channels, thereby greatly reducing the data amount of the electroencephalogram signals, effectively reducing the resources consumed during operation, reducing the time consumption of operation, and thereby ensuring real-time performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brain-computer interface, and in particular to a cross-subject five-channel emotion modeling and emotion regulation method, device and storage medium. BACKGROUND

[0002] In the business scenarios such as watching movies, playing motion games, mindfulness meditation, etc., the emotions (such as happy, neutral, sad, etc.) of the user can be monitored to assist the operation in the business scenario.

[0003] Further, based on the brain-computer interface, 32-channel or 64-channel electroencephalogram signals of the user are collected, and features in the time domain or frequency domain are constructed from the electroencephalogram signals, and the features are input into the SVM (Support Vector Machine) to infer the emotions of the user.

[0004] In this process, the number of channels of the electroencephalogram signals is large, so that the data volume is large, and the feature engineering is relatively cumbersome, such as Fourier transform (FFT), power spectral density (PSD), wavelet transform, etc., thereby causing high time consumption of operation. SUMMARY

[0005] Therefore, the present application provides a cross-subject five-channel emotion modeling and emotion regulation method, device and storage medium, which can reduce the time consumption of detecting the emotions of the user.

[0006] The first aspect of the present application provides a cross-subject five-channel emotion modeling method, comprising:

[0007] When playing multimedia data for the user, calling a brain-computer interface to collect original electroencephalogram signals of the user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel;

[0008] Determining an ideal emotion induced by the multimedia data to the user;

[0009] Pretreating the original electroencephalogram signals to obtain target electroencephalogram signals;

[0010] Segmenting the target electroencephalogram signals into a plurality of electroencephalogram segment signals;

[0011] Under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, training an emotion classifier with the electroencephalogram segment signals as samples and the ideal emotion as labels.

[0012] The second aspect of the present application provides an emotion regulation method, comprising:

[0013] In a process of playing multimedia data for a user, a brain-computer interface is invoked to collect original electroencephalogram signals of the user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel;

[0014] The original electroencephalogram signals are preprocessed to obtain target electroencephalogram signals;

[0015] The target electroencephalogram signals are divided into a plurality of electroencephalogram segment signals;

[0016] An emotion classifier trained according to the method of any one of claims 1-5 is loaded;

[0017] The electroencephalogram segment signals are input into the emotion classifier to identify a desired emotion generated by the user under attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0018] A business operation is performed according to the desired emotion to guide the user to adjust the emotion.

[0019] A third aspect of the present application provides a five-channel emotion modeling device across subjects, comprising:

[0020] An electroencephalogram signal collection module is configured to invoke a brain-computer interface to collect original electroencephalogram signals of a user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel in a process of playing multimedia data for the user;

[0021] An ideal emotion determination module is configured to determine an ideal emotion induced by the multimedia data to the user;

[0022] A preprocessing module is configured to preprocess the original electroencephalogram signals to obtain target electroencephalogram signals;

[0023] An electroencephalogram signal division module is configured to divide the target electroencephalogram signals into a plurality of electroencephalogram segment signals;

[0024] An emotion classifier training module is configured to train an emotion classifier under attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, with the electroencephalogram segment signals as samples and the ideal emotion as a label.

[0025] A fourth aspect of the present application provides an emotion adjustment device, comprising:

[0026] An electroencephalogram signal collection module is configured to invoke a brain-computer interface to collect original electroencephalogram signals of a user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel in a process of playing multimedia data for the user;

[0027] A preprocessing module is configured to preprocess the original electroencephalogram signal to obtain a target electroencephalogram signal.

[0028] An electroencephalogram signal segmentation module is configured to segment the target electroencephalogram signal into a plurality of electroencephalogram segment signals.

[0029] An emotion classifier loading module is configured to load an emotion classifier trained according to the method of Embodiment One.

[0030] An expected emotion generation module is configured to input the electroencephalogram segment signal into the emotion classifier to identify an expected emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0031] A business operation execution module is configured to perform a business operation according to the expected emotion to guide the user to adjust the emotion.

[0032] A fifth aspect of the present application provides an electronic device, which comprises:

[0033] at least one processor; and

[0034] a memory connected to the at least one processor in communication; wherein

[0035] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the cross-subject five-channel emotion modeling method according to the first aspect or the emotion regulation method according to the second aspect.

[0036] A sixth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the cross-subject five-channel emotion modeling method according to the first aspect or the emotion regulation method according to the second aspect.

[0037] A seventh aspect of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the cross-subject five-channel emotion modeling method according to the first aspect or the emotion regulation method according to the second aspect.

[0038] In the embodiment, when playing multimedia data for a user, the brain-computer interface is called to collect raw electroencephalogram signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel; an ideal emotion induced by the multimedia data to the user is determined; the raw electroencephalogram signals are preprocessed to obtain target electroencephalogram signals; the target electroencephalogram signals are segmented into a plurality of electroencephalogram segment signals; and the electroencephalogram segment signals are taken as samples and the ideal emotion is taken as a label to train an emotion classifier under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel. The embodiment focuses on the electroencephalogram signals of the five channels of O1, O2, C3, C4 and FP2 for emotion recognition, not only retains the information related to emotion recognition in the electroencephalogram signals, guarantees the accuracy of emotion recognition, but also greatly reduces the number of channels, thereby greatly reducing the data amount of the electroencephalogram signals. On this basis, deep learning can be introduced to simplify feature engineering, which can effectively reduce the resources consumed during operation and reduce the time consumption of operation, thereby guaranteeing real-time performance.

[0039] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creating laborious work.

[0041] Figure 1 is a flowchart of a cross-subject five-channel emotion modeling method provided by the embodiment one of the present application.

[0042] Figure 2 is an example diagram of an electronic device provided by the embodiment one of the present application.

[0043] Figure 3 is an example diagram of a head-mounted device provided by the embodiment one of the present application.

[0044] Figure 4 is a schematic diagram of an emotion classifier provided by the embodiment one of the present application.

[0045] Figure 5 is a schematic diagram of a channel attention module provided by the embodiment one of the present application.

[0046] Figure 6 is a flowchart of an emotion regulation method provided by the embodiment two of the present application.

[0047] Figure 7 is a flow chart of an emotion regulation method provided by Embodiment Three of the present application.

[0048] Figure 8 is a structural schematic diagram of a cross-subject five-channel emotion modeling device provided by Embodiment Four of the present application.

[0049] Figure 9 is a structural schematic diagram of an emotion regulation device provided by Embodiment Five of the present application.

[0050] Figure 10 is a structural schematic diagram of an electronic device provided by Embodiment Six of the present application. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the scope of protection of the present application.

[0052] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can encompass orders of implementation other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product, or device.

[0053] Embodiment One

[0054] Referring to Figure 1 , a flow chart of a cross-subject five-channel emotion modeling method provided by Embodiment One of the present application is shown, which can be executed by a cross-subject five-channel emotion modeling device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device.

[0055] In one case, the electronic device is an integrated device, especially a head-mounted device such as a head ring, a helmet, glasses, etc., wherein the electronic device is provided with a computing unit and a brain-computer interface, and the computing unit is loaded with an emotion classifier.

[0056] In another case, the electronic device can also be a physically separable or detachable device, in which case, as shown in Figure 2 , the electronic device includes a head-mounted device 201 and a computing device 202 independent of each other, the computing device 202 can be deployed locally or in the cloud, and is loaded with an emotion classifier, and the head-mounted device 201 and the computing device 202 can be connected through a local area network or the Internet in a wired or wireless manner, wherein the head-mounted device 201 is configured with a brain-computer interface.

[0057] As shown in Figure 1 , the method comprises:

[0058] Step 101, when playing multimedia data for a user, calling a brain-computer interface to collect raw electroencephalogram signals of the user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0059] In actual application, 19 recording electrodes and 2 reference electrodes are included in the 10-20 system.

[0060] Two lines are determined on the scalp surface, one is the front and back connecting line of nasion to external occipital protuberance, which is 100%, and the other is the left and right connecting line between the preauricular notch, which is 100%. The intersection of the two at the top of the head is the position of the Cz electrode. 10% from the nasion is FPz (midline of frontal pole), and every 20% from FPz is the position of an electrode, in order of Fz (midline of frontal), Cz (midline of central), Pz (midline of parietal) and Oz (midline of occipital).

[0061] The distance between Oz and external occipital protuberance is 10%. The distance between the preauricular notch and T3 (left middle temporal) electrode position is 10%. Every 20% to the right is an electrode, in order of C3 (left central), Cz, C4 (right central) and T4 (right middle temporal). The distance between T4 and the right preauricular notch is 10%. The connecting line from FPz to Oz is the left temporal connecting line, and 10% to the left of FPz is FP1 (left frontal pole). Every 20% from FP1 is an electrode, in order of F7 (left anterior temporal), T3 (left middle temporal), T5 (left posterior temporal) and O1 (left occipital), wherein T3 is the intersection of this line and the preauricular notch connecting line, and O1 is 10% away from Oz. The right temporal connecting line corresponds to this, and from front to back, in order of FP2 (right temporal pole), F8 (right anterior temporal), T4 (right middle temporal), T6 (right posterior temporal) and O2 (right occipital).

[0062] Connect a line from FP1 to O1 and from FP2 to O2, which are left and right sagittal side lines, and each 20% from FP1 and FP2 is an electrode site, and the left side is F3 (left forehead), C3 (left central), P3 (left parietal), and O1 (left occipital), and the right side is F4 (right forehead), C4 (right central), P4 (right parietal), and O2 (right occipital) in turn. In the 10-20 system, FPz and Oz are not included in the 19 recording points.

[0063] The induction process of the user's emotion is relatively complex, involving the interaction of multiple specific brain regions, and the five channels O1, O2, C3, C4, and FP2 are screened out by experiments and the like in the embodiment. The five channels involve multiple brain regions of the forehead region, the central region, and the parietal region, and can effectively collect the electroencephalogram signals of the emotion induction brain region. Compared with 32 channels or 64 channels, the number of channels is greatly reduced, thereby greatly reducing the data amount of the electroencephalogram signals, and maintaining the information amount in the electroencephalogram signals which has a higher correlation with the emotion, thereby maintaining the accuracy of emotion recognition.

[0064] It should be noted that the five channels O1, O2, C3, C4, and FP2 are only examples and do not constitute a limitation on the protection scope of the present application. In addition to the five channels O1, O2, C3, C4, and FP2, various modifications, combinations, sub-combinations, and substitutions can be made to the channels according to design requirements and other factors.

[0065] In the training mode, when the user wears the head-mounted device, the electrodes in the brain-computer interface can be pasted to the positions corresponding to O1, O2, C3, C4, and FP2 (such as position 4, position 5, position 2, position 3, and position 1 shown in the figure). Figure 3 When the user experiences the multimedia data, the corresponding emotion is induced.

[0066] The multimedia data can include text data, audio data, video data, and the like, and has various forms, such as novels, cross-talks, songs, short videos, and the like.

[0067] To ensure that the content of the multimedia data focuses and the user induces a specific emotion, the duration of the multimedia data is usually short, such as 2 minutes, 4 minutes, and the like.

[0068] To improve the accuracy of the test, the user can be prompted to maintain the emotional state before the multimedia data is played, so that the user adjusts his own emotional state for a short time (such as 3-10 seconds).

[0069] Considering the positions of O1, O2, C3, C4, and FP2, dry electrodes can be used in the brain-computer interface, which is more convenient for the user to wear than wet electrodes.

[0070] At this time, the brain-computer interface can collect electroencephalogram signals under five channels (O1, O2, C3, C4, and FP2), align the time stamp of the electroencephalogram signals with the time stamp of the multimedia data, and when the alignment is completed, the electroencephalogram signals during the playing of the multimedia data are cropped, which are denoted as original electroencephalogram signals.

[0071] In step 102, the ideal emotion of the user induced by the multimedia data is determined.

[0072] In a specific implementation, the labels (i.e., emotions) of the multimedia data can be labeled in advance according to the content of each multimedia data, and then the label labeled on the multimedia data can be queried according to the ID and other data as the ideal emotion induced by the user.

[0073] Further, the label of the multimedia data can be labeled manually, or the label of the multimedia data can be labeled automatically using the attribute information (such as type) of the multimedia, NLP (Natural Language Processing), computer vision, and the like, and the like, which is not limited in the embodiment.

[0074] For example, when the type of the multimedia data is cross-talk, the label of cross-talk can be automatically labeled as happy.

[0075] For another example, when the multimedia data is a short video, the comment information input by the group users on the short video can be collected, the emotion expressed by the group users in the comment information is analyzed using the NLP technology, and the emotion is labeled as the label of the short video.

[0076] In addition, when the desired emotion of the user is identified subsequently, the user can be fed back visually and aurally, the user can judge whether the emotional state of the user is consistent with the desired emotion, the desired emotion is corrected when the emotional state is consistent with the desired emotion, and the correct ideal emotion is provided when the emotional state is inconsistent with the desired emotion.

[0077] In step 103, the original electroencephalogram signals are preprocessed to obtain target electroencephalogram signals.

[0078] In the embodiment, various preprocessing can be performed on the original electroencephalogram signals, such as amplification, filtering, normalization, and the like, to obtain the target electroencephalogram signals, so as to improve the quality of the target electroencephalogram signals.

[0079] When filtering, the original electroencephalogram signals are input into a second-order Butterworth high-pass filter to filter out low-frequency signals less than a first frequency threshold (such as 0.1 Hz or the like) to obtain first candidate electroencephalogram signals.

[0080] The first candidate electroencephalogram signals are input into a second-order band-stop Butterworth filter to filter out high-frequency signals between a second frequency threshold (such as 48 Hz or the like) and a third frequency threshold (such as 52 Hz or the like) to obtain second candidate electroencephalogram signals.

[0081] The second candidate electroencephalogram signal is input into a second-order low-pass Butterworth high filter to filter out high-frequency signals greater than a fourth frequency threshold (such as 70 Hz) to obtain a target electroencephalogram signal.

[0082] The first frequency threshold is less than the second frequency threshold, the second frequency threshold is less than the third frequency threshold, and the third frequency threshold is less than the fourth frequency threshold.

[0083] Step 104, the target electroencephalogram signal is divided into a plurality of electroencephalogram segment signals.

[0084] In this embodiment, under the condition of greatly reducing the data amount of the electroencephalogram signal, an end-to-end emotion classifier can be constructed based on deep learning.

[0085] At this time, the target electroencephalogram signal can be divided in an equal division manner to obtain a plurality of electroencephalogram segment signals, and the electroencephalogram segment signal is taken as the input of the emotion classifier, thereby omitting feature engineering and simplifying the processing procedure, and the operation time consumption can be reduced.

[0086] In a specific implementation, a slidable window can be added to the target electroencephalogram signal from the end point of the target electroencephalogram signal.

[0087] Each time the window is slid by a preset step, the signal in the window is extracted as an electroencephalogram segment signal.

[0088] The width of the window is greater than the sliding step, for example, the width of the window is 5 seconds, and the sliding step is 3 seconds.

[0089] Step 105, training the emotion classifier with the electroencephalogram segment signal as the sample and the ideal emotion as the label under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0090] In actual application, the emotion classifier embeds a channel-based attention mechanism, and through the attention mechanism, the feature information of the electroencephalogram segment signal in the five channels (O1, O2, C3, C4 and FP2) can be explored, the attention between the five channels (O1, O2, C3, C4 and FP2) is added to the electroencephalogram segment signal, the weight of the signal related to the emotion is enhanced, so as to increase the stability of the emotion classifier and the accuracy of recognizing the user emotion, in this case, a small amount of test user (such as 15 users) electroencephalogram segment signals can be taken as the sample and the ideal emotion as the label to supervise the training of the emotion classifier, so that the emotion classifier can realize the intra-subject, cross-subject and cross-time user emotion recognition.

[0091] Since the data volume of the electroencephalogram signals of the five channels (O1, O2, C3, C4 and FP2) is small, if the emotion classifier selects random parameters to start training, the accuracy of identifying the emotion of the user is low in the early stage of training, therefore, the embodiment adopts the pre-training mode to obtain a general emotion classifier, so as to facilitate the emotion classifier to learn the commonality of the electroencephalogram signals of different users under the head-mounted device, so as to improve the accuracy of identifying the emotion.

[0092] In an embodiment of the present application, step 105 can include the following steps:

[0093] Step 1051, loading an emotion classifier.

[0094] In the embodiment, the constructed emotion classifier can be loaded into the memory for running, as shown in the following table: Figure 4 The emotion classifier includes an attention module Attention Module and an emotion classification head Head.

[0095] The attention module Attention Module is used to add weights to the electroencephalogram signals according to the attention among the five channels (O1, O2, C3, C4 and FP2).

[0096] The emotion classification head Head is used to identify the emotion of the user according to the characteristics of the electroencephalogram signals.

[0097] Further, the structure of the emotion classification head Head is not limited to the artificial designed neural network, for example, the emotion classification head Head can reuse a pre-trained model, such as an electroencephalogram network EEGNet, at this time, the pre-trained model is modified to have an output layer, such as a FC (Fully Connected Layer), according to the classification task of identifying the emotion of the user, so as to save the cost of training and maintain a high accuracy, or a neural network optimized by a model quantization method, a neural network searched by a NAS (Neural Architecture Search) method according to the characteristics of the electroencephalogram signals, etc., the embodiment does not limit this.

[0098] Step 1052, inputting the electroencephalogram segment signal into the attention module Attention Module to add the attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, to obtain electroencephalogram emotion characteristics.

[0099] In the embodiment, as shown in the following table: Figure 4 The electroencephalogram segment signal of the five channels (O1, O2, C3, C4 and FP2) can be input into the attention module Attention Module to add the attention weights under the five channels (O1, O2, C3, C4 and FP2), to obtain the electroencephalogram emotion characteristics.

[0100] Exemplarily, as shown in Figure 5 The channel attention module includes two fully connected layers FC.

[0101] In this example, an average pooling operation Avg Pooling can be performed on the electroencephalogram segment signal to obtain a first candidate electroencephalogram feature.

[0102] The first candidate electroencephalogram feature is input into the first fully connected layer FC to be mapped into a second channel feature.

[0103] The second channel feature is activated using a hyperbolic tangent function Tanh to obtain a third channel feature.

[0104] The third channel feature is input into the second fully connected layer FC to be mapped into a fourth channel feature.

[0105] The fourth channel feature is activated using a flexible maximum transfer function Softmax to obtain attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0106] The electroencephalogram segment signal and the attention weights are fused into an electroencephalogram emotion feature using functions such as Concat, Add, etc.

[0107] Step 1053, the electroencephalogram emotion feature is input into the emotion classification head to identify the expected emotion generated by the user.

[0108] In this embodiment, as shown in Figure 4 The electroencephalogram emotion feature is input into the emotion classification head Head, and the emotion classification head Head classifies the electroencephalogram emotion feature according to its structure to identify the emotion (such as happy, neutral) generated by the user, which is recorded as the expected emotion.

[0109] Generally, the emotion classification head outputs the probability score of each emotion, and selects the emotion with the highest probability score as the expected emotion generated by the user.

[0110] Step 1054, update the emotion classifier according to the loss value between the ideal emotion and the expected emotion.

[0111] For the same electroencephalogram segment signal, the ideal emotion and the expected emotion can be brought into a preset loss function (such as cross entropy) to calculate a loss value, and the loss value can be substituted into an optimization algorithm such as SGD (stochastic gradient descent) or Adam (Adaptive momentum) to calculate the update amplitude of the parameters in the emotion classifier (channel attention module Attention Module and emotion classification head Head), and the parameters in the emotion classifier (channel attention module Attention Module and emotion classification head Head) are updated according to the update amplitude.

[0112] Step 1055, if the update is completed, it is judged whether the preset training condition is met; if yes, step 1056 is executed; if no, step 1051 is returned to execute.

[0113] Step 1056, the emotion classifier is determined to complete the training.

[0114] In this embodiment, the training condition can be set in advance as the condition for stopping training the emotion classifier (channel attention module Attention Module and emotion classification head Head), for example, the number of iterations reaches a certain threshold, the loss value is less than a certain threshold, the change amplitude of the loss value in multiple iterations is less than a certain threshold, and the like.

[0115] In each round of iterative training, if the parameters of the emotion classifier (channel attention module Attention Module and emotion classification head Head) are updated, it can be judged whether the training condition is met in the current round of iterative training.

[0116] If the training condition is met, it can be considered that the emotion classifier (channel attention module Attention Module and emotion classification head Head) completes the training.

[0117] If the training condition is not met, the next round of iterative training can be entered, and steps 1051-1055 can be executed again, and the iterative training is cycled until the emotion classifier (channel attention module Attention Module and emotion classification head Head) completes the training.

[0118] In the embodiment, when playing multimedia data for a user, a brain-computer interface is called to collect raw electroencephalogram signals of the user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel; an ideal emotion induced by the multimedia data to the user is determined; the raw electroencephalogram signals are preprocessed to obtain target electroencephalogram signals; the target electroencephalogram signals are segmented into a plurality of electroencephalogram segment signals; and the electroencephalogram segment signals are taken as samples and the ideal emotion is taken as a label to train an emotion classifier under attention of O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel. The embodiment focuses on the electroencephalogram signals of the five channels of O1, O2, C3, C4 and FP2 for emotion recognition, not only retains the information related to emotion recognition in the electroencephalogram signals, guarantees the accuracy of emotion recognition, but also greatly reduces the number of channels, thereby greatly reducing the data amount of the electroencephalogram signals. On this basis, deep learning can be introduced to simplify feature engineering, which can effectively reduce the resources consumed during operation and reduce the time consumption of operation, thereby guaranteeing real-time performance.

[0119] Embodiment Two

[0120] Referring to Figure 6 , a flowchart of an emotion regulation method provided by Embodiment Two of the present application is shown, which can be executed by an emotion regulation device. The cross-subject five-channel emotion modeling and emotion regulation device can be realized in the form of hardware and / or software, and the emotion regulation device can be configured in an electronic device.

[0121] In one case, the electronic device is an integrated device, especially a head-mounted device such as a head ring, a helmet, glasses, etc. The brain-computer interface is arranged in the electronic device.

[0122] In another case, the electronic device can also be a physically separable or detachable device. In this case, the electronic device includes a head-mounted device and a computing device which are independent of each other. The computing device can be deployed locally or in the cloud. The head-mounted device and the computing device can be connected through a local area network or the Internet. The brain-computer interface is arranged in the head-mounted device.

[0123] As Figure 6 shown, the method includes the following steps.

[0124] Step 601, when playing multimedia data for a user, a brain-computer interface is called to collect raw electroencephalogram signals of the user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0125] In the free mode, the user uses other devices to play multimedia data facing the user in the business scenarios such as watching movies, playing motion games, mindfulness meditation, etc. When the user experiences the multimedia data, the corresponding emotions are induced. During this period, when wearing the head-mounted device, the electrodes in the brain-computer interface can be pasted to the positions corresponding to O1, O2, C3, C4 and FP2.

[0126] Generally, the multimedia data is customized content in the business scenario, and the time length of the multimedia data is not the same, such as 5 minutes, 1 hour, etc., which can induce one or more emotions of the user.

[0127] At this time, the electroencephalogram signals under the five channels (O1, O2, C3, C4 and FP2) collected by the brain-computer interface can be read, the time stamp of the electroencephalogram signals is aligned with the time stamp of the multimedia data, and when the alignment is completed, the electroencephalogram signals during playing the multimedia data are cropped, which are recorded as original electroencephalogram signals.

[0128] Step 602, pre-processing the original electroencephalogram signals to obtain target electroencephalogram signals.

[0129] In this embodiment, various pre-processing can be performed on the original electroencephalogram signals, such as amplification, filtering, normalization, etc., to obtain the target electroencephalogram signals, so as to improve the quality of the target electroencephalogram signals.

[0130] When filtering, the original electroencephalogram signals can be input into a second-order Butterworth high-pass filter to filter out low-frequency signals less than a first frequency threshold (such as 0.1 Hz, etc.), to obtain first candidate electroencephalogram signals.

[0131] The first candidate electroencephalogram signals are input into a second-order band-stop Butterworth filter to filter out high-frequency signals between a second frequency threshold (such as 48 Hz, etc.) and a third frequency threshold (such as 52 Hz, etc.), to obtain second candidate electroencephalogram signals.

[0132] The second candidate electroencephalogram signals are input into a second-order low-pass Butterworth high filter to filter out high-frequency signals greater than a fourth frequency threshold (such as 70 Hz), to obtain target electroencephalogram signals.

[0133] Among them, the first frequency threshold is less than the second frequency threshold, the second frequency threshold is less than the third frequency threshold, and the third frequency threshold is less than the fourth frequency threshold.

[0134] Step 603, cutting the target electroencephalogram signals into a plurality of electroencephalogram segment signals.

[0135] In this embodiment, under the condition of greatly reducing the data amount of the electroencephalogram signals, an end-to-end emotion classifier can be constructed based on deep learning.

[0136] At this time, the target electroencephalogram signal can be divided into multiple electroencephalogram segment signals in an equal division manner, the electroencephalogram segment signal is taken as an input of the emotion classifier, feature engineering is omitted, a processing procedure is simplified, and operation time consumption can be reduced.

[0137] In a specific implementation, a slidable window can be added to the target electroencephalogram signal from an endpoint of the target electroencephalogram signal.

[0138] The signal in the window is extracted as an electroencephalogram segment signal each time the window is slid by a preset step.

[0139] The width of the window is greater than the sliding step, for example, the width of the window is 5 seconds, and the sliding step is 3 seconds.

[0140] Step 604, loading the emotion classifier.

[0141] In the embodiment, the emotion classifier constructed and trained in advance based on the method of the first embodiment can be loaded into the memory for running

[0142] Step 605, inputting the electroencephalogram segment signal into the emotion classifier to identify the expected emotion of the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0143] In actual application, the emotion classifier embeds a channel-based attention mechanism, the electroencephalogram segment signal can be input into the emotion classifier, the channel-based attention mechanism can explore the feature information of the electroencephalogram segment signal in the five channels (O1, O2, C3, C4 and FP2), the attention between the five channels (O1, O2, C3, C4 and FP2) is added to the electroencephalogram segment signal, the weight of the signal related to the emotion is enhanced, so as to increase the stability of the emotion classifier and the accuracy of identifying the emotion of the user.

[0144] Further, the embodiment provides a cross-user and cross-time emotion classifier, which does not rely on the pre-acquisition of the electroencephalogram signal of the current user for training, and the current user can directly use the emotion classifier to identify the current emotional state.

[0145] In an embodiment of the present application, the emotion classifier includes an attention module Attention Module and an emotion classification head Head.

[0146] The attention module Attention Module is used to add weight to the electroencephalogram signal according to the attention between the five channels (O1, O2, C3, C4 and FP2).

[0147] The emotion classification head Head is used to identify the emotion of the user according to the features of the electroencephalogram signal.

[0148] Further, the structure of the emotion classification head Head is not limited to an artificially designed neural network. For example, the emotion classification head Head can be reused in a pre-trained model, such as an electroencephalogram network EEGNet, and the like. In this case, the pre-trained model is modified to have an output layer, such as an FC, according to the classification task of identifying the emotion of the user, so as to save the cost of training and maintain a high accuracy. The neural network can also be optimized by a model quantization method, a neural network searched by a NAS method according to the characteristics of the electroencephalogram signal, and the like. The present embodiment is not limited thereto.

[0149] In the present embodiment, step 605 can include the following steps:

[0150] Step 6051, adding the attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel, and the FP2 channel to the input of the channel attention module of the electroencephalogram segment signal to obtain the electroencephalogram emotion feature.

[0151] In the present embodiment, the attention weights under the five channels (O1, O2, C3, C4, and FP2) of the electroencephalogram segment signal are added to the input of the channel attention module to obtain the electroencephalogram emotion feature.

[0152] For example, the channel attention module includes two fully connected layers FC.

[0153] In the present example, an average pooling operation Avg Pooling is performed on the electroencephalogram segment signal to obtain a first candidate electroencephalogram feature.

[0154] The first candidate electroencephalogram feature is input into the first fully connected layer FC to be mapped into a second channel feature.

[0155] The second channel feature is activated by using a hyperbolic tangent function Tanh to obtain a third channel feature.

[0156] The third channel feature is input into the second fully connected layer FC to be mapped into a fourth channel feature.

[0157] The fourth channel feature is activated by using a flexible maximum transmission function Softmax to obtain the attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel, and the FP2 channel.

[0158] The electroencephalogram segment signal and the attention weights are fused into the electroencephalogram emotion feature by using functions such as Concat, Add, and the like.

[0159] Step 6052, the electroencephalogram emotion feature is input into the emotion classification head to identify the expected emotion generated by the user.

[0160] In this embodiment, the EEG emotion features are input into the emotion classification head Head, which classifies the EEG emotion features according to its structure, identifies the emotion generated by the user, and is recorded as the expected emotion.

[0161] Generally, the emotion classification head outputs the probability score of each emotion, and selects the emotion with the highest probability score as the expected emotion generated by the user.

[0162] The expected emotion can provide more accurate visual feedback to the user, allowing the user to better understand their current emotional state, so that the embodiment can effectively identify emotions for a specific user.

[0163] In this embodiment, the brain-computer interface is called to collect the raw EEG signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel; the raw EEG signals are preprocessed to obtain target EEG signals; the target EEG signals are divided into multiple EEG segment signals; the emotion classifier is loaded; and the EEG segment signals are input into the emotion classifier to identify the expected emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0164] Step 606, performing business operations according to the expected emotion to guide the user to adjust the emotion.

[0165] In this embodiment, some business scenarios have emotional needs, and it can be determined whether the expected emotion of the user meets the emotional needs of the business scenario. When the expected emotion deviates from the emotional needs of the business scenario, the expected emotion is used to perform corresponding business operations in the business scenario to guide the user to adjust the emotion until the emotional needs of the business scenario are met.

[0166] In a specific implementation, the ideal emotion induced by the multimedia data to the user is determined, then for the same multimedia data, the ideal emotion and the expected emotion are simultaneously possessed, the ideal emotion is the emotion induced by the user according to the content of the multimedia data, and the expected emotion is the emotion predicted by the emotion classifier according to the EEG signal of the user. When the performance of the emotion classifier meets the requirements, the expected emotion can be considered as the actual emotion generated by the user.

[0167] At this time, a variable can be generated, denoted as the co-occurrence number, which is the number of times when the expected emotion and the ideal emotion are the same, indicating that the user correctly induces the emotion according to the guidance of the multimedia data.

[0168] Generally, the co-occurrence number is valid in one business scenario, and the co-occurrence number is released when the business scenario ends.

[0169] In the one-time service scenario, if the ideal emotion and the expected emotion are the same for the same multimedia data, the same occurrence quantity can be accumulated by 1, and if the ideal emotion and the expected emotion are different, the operation can be ignored.

[0170] The ratio between the same occurrence quantity and the total quantity of the ideal emotion is calculated to obtain the accuracy of the user-induced emotion, and thus the service operation provided by the current service scenario is performed according to the accuracy to guide the user to adjust the emotion.

[0171] For example, the accuracy of the user-induced emotion is displayed in the form of a column chart or the like on a UI (user interface), the accuracy of the user-induced emotion is intuitively displayed to the user, guidance information for adjusting the emotion is generated for the user according to the information of the user-induced emotion (such as the accuracy of the user-induced emotion, the type distribution of the incorrectly induced emotion, etc.) by using an LLM (Large Language Model) or the like, and the guidance information for adjusting the emotion is played to the user by using a virtual digital person to guide the user to adjust the emotion.

[0172] The embodiment focuses on the electroencephalogram signals of the five channels of O1, O2, C3, C4 and FP2 for emotion recognition, not only retains the information related to emotion recognition in the electroencephalogram signals, guarantees the accuracy of emotion recognition, thereby improving the efficiency of emotion adjustment, but also greatly reduces the number of channels, thereby greatly reducing the data amount of the electroencephalogram signals, on the basis of which deep learning can be introduced, feature engineering is simplified, resources consumed during operation can be effectively reduced, time consumption of operation can be reduced, thereby guaranteeing real-time performance.

[0173] Embodiment Three

[0174] Referring to Figure 7 , a flowchart of a cross-subject five-channel emotion modeling and emotion adjustment method provided by Embodiment Three of the present application is shown, and the embodiment increases the operation of updating the emotion classifier on the basis of the foregoing embodiments. As Figure 7 shown, the method comprises:

[0175] Step 701, when playing multimedia data for a user, calling a brain-computer interface to collect original electroencephalogram signals of the user under O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0176] Step 702, preprocessing the original electroencephalogram signals to obtain target electroencephalogram signals.

[0177] Step 703, dividing the target electroencephalogram signals into a plurality of electroencephalogram segment signals.

[0178] Step 704, loading an emotion classifier.

[0179] Step 705, inputting the EEG segment signal into the emotion classifier, and identifying the expected emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0180] Step 706, performing a business operation according to the expected emotion to guide the user to adjust the emotion.

[0181] Step 707, determining the actual emotion induced by the user in the current time period.

[0182] In the process of identifying the expected emotion of the user, a plurality of shorter time periods can be divided, such as every 4 minutes as a time period.

[0183] In some cases, the actual emotion induced by the user can be determined by information in the business scenario.

[0184] Taking multimedia data as an example, the attribute information (such as type, etc.) of the multimedia data, NLP, computer vision and other technologies can be used to label the corresponding label (i.e. emotion) according to the content of each multimedia data, as the actual emotion induced by the user.

[0185] For example, when the type of multimedia data is cross-talk, happy can be automatically labeled as the label of cross-talk.

[0186] For another example, when the multimedia data is a short video, the comment information input by the group user to the short video can be collected, and the emotion expressed by the group user in the comment information can be analyzed using NLP technology, and the emotion is labeled as the label of the short video.

[0187] Step 708, in the current time period, updating the emotion classifier with the EEG segment signal as the sample and the actual emotion as the label under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0188] In actual application, attention between O1, O2, C3, C4 and FP2 can be added to the EEG segment signal to enhance the weight of the signal related to the emotion, so as to increase the accuracy of identifying the user's emotion. In this case, the EEG segment signal of the current user in the current time period can be used as the sample, and the actual emotion can be used as the label to supervise the update of the emotion classifier using Fine-tuing (fine-tuning) technology, so that the emotion classifier can realize personalized user emotion recognition and improve the accuracy of the emotion classifier in cross-user emotion recognition.

[0189] Generally, a lower learning rate can be used to supervise the update of the emotion classifier, that is, the learning rate of updating the emotion classifier is lower than the learning rate of training the emotion classifier.

[0190] Under the condition of greatly reducing the data amount of the brain electrical signals, the resource consumption for updating the emotion classifier can be reduced, so as to realize online short-time and few-sample updating of the emotion classifier.

[0191] Step 709, in the next time period, the updated emotion classifier replaces the current emotion classifier.

[0192] If the training of the emotion classifier is completed in the current time period, the updated emotion classifier can replace the current emotion classifier in the next time period to provide the service of emotion recognition for the user.

[0193] Further, in the next time period, the updated emotion classifier and the current emotion classifier run at the same time.

[0194] On the one hand, for the updated emotion classifier, the expected emotion output by the updated emotion classifier is compared with the actual emotion induced by the user, and the first performance index of the updated emotion classifier in recognizing the emotion of the user is calculated.

[0195] On the other hand, for the current emotion classifier, the expected emotion output by the current emotion classifier is compared with the actual emotion induced by the user, and the second performance index of the current emotion classifier in recognizing the emotion of the user is calculated.

[0196] The type of the first performance index is the same as the type of the second performance index, for example, accuracy, precision, etc.

[0197] The first performance index is compared with the second performance index.

[0198] If the first performance index is better than the second performance index, which means that the performance of the updated emotion classifier is better than the performance of the current emotion classifier, the updated emotion classifier is allowed to replace the current emotion classifier.

[0199] If the first performance index is worse than the second performance index, which means that the performance of the updated emotion classifier is worse than the performance of the current emotion classifier, the updated emotion classifier is prohibited to replace the current emotion classifier, that is, the updated emotion classifier is ignored, and the current emotion classifier is maintained unchanged.

[0200] Embodiment four

[0201] Referring to Figure 8 , a structure schematic diagram of a cross-subject five-channel emotion modeling device provided by an embodiment four of the present application is shown. As Figure 8 shown, the device comprises:

[0202] The electroencephalogram signal collection module 801 is configured to collect original electroencephalogram signals of a user under O1, O2, C3, C4 and FP2 channels when playing multimedia data for the user.

[0203] The ideal emotion determination module 802 is configured to determine an ideal emotion induced by the multimedia data for the user.

[0204] The preprocessing module 803 is configured to preprocess the original electroencephalogram signals to obtain target electroencephalogram signals.

[0205] The electroencephalogram signal segmentation module 804 is configured to segment the target electroencephalogram signals into a plurality of electroencephalogram segment signals.

[0206] The emotion classifier training module 805 is configured to train an emotion classifier under attention of the O1, O2, C3, C4 and FP2 channels, with the electroencephalogram segment signals as samples and the ideal emotion as a label.

[0207] In an embodiment of the present application, the preprocessing module 803 comprises:

[0208] The first filtering module is configured to input the original electroencephalogram signals into a second-order Butterworth high-pass filter to filter out signals less than a first frequency threshold to obtain first candidate electroencephalogram signals.

[0209] The second filtering module is configured to input the first candidate electroencephalogram signals into a second-order band-stop Butterworth filter to filter out signals between a second frequency threshold and a third frequency threshold to obtain second candidate electroencephalogram signals.

[0210] The third filtering module is configured to input the second candidate electroencephalogram signals into a second-order low-pass Butterworth filter to filter out signals greater than a fourth frequency threshold to obtain target electroencephalogram signals.

[0211] The first frequency threshold is less than the second frequency threshold, the second frequency threshold is less than the third frequency threshold, and the third frequency threshold is less than the fourth frequency threshold.

[0212] In an embodiment of the present application, the electroencephalogram signal segmentation module 804 comprises:

[0213] The window adding module is configured to add a window to the target electroencephalogram signals.

[0214] The window sliding module is configured to extract signals in the window as electroencephalogram segment signals when sliding the window by a preset step each time.

[0215] The width of the window is greater than the step.

[0216] In an embodiment of the present application, the emotion classifier training module 805 comprises:

[0217] an emotion classifier loading module for loading an emotion classifier; the emotion classifier comprises a channel attention module and an emotion classification head;

[0218] an electroencephalogram emotion feature generation module for inputting the electroencephalogram segment signal into the channel attention module to add attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, and obtain electroencephalogram emotion features;

[0219] an expected emotion recognition module for inputting the electroencephalogram emotion features into the emotion classification head to recognize an expected emotion generated by the user;

[0220] an emotion classifier updating module for updating the emotion classifier according to a loss value between the ideal emotion and the expected emotion;

[0221] a training condition judgment module for judging whether a preset training condition is met if the updating is completed; if yes, a training completion determination module; if no, returning to execute the electroencephalogram emotion feature generation module;

[0222] a completion determination module for determining that the emotion classifier completes the training.

[0223] In an embodiment of the present application, the channel attention module comprises two fully connected layers;

[0224] The electroencephalogram emotion feature generation module comprises:

[0225] a pooling module for performing an average pooling operation on the electroencephalogram segment signal to obtain first candidate electroencephalogram features;

[0226] a first fully connected module for inputting the first candidate electroencephalogram features into the first layer of the fully connected layer to map them into second channel features;

[0227] a first activation module for using a hyperbolic tangent function Tanh to activate the second channel features to obtain third channel features;

[0228] a second fully connected module for inputting the third channel features into the second layer of the fully connected layer to map them into fourth channel features;

[0229] a second activation module for using a flexible maximum transmission function Softmax to activate the fourth channel features to obtain the attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0230] a fusion module configured to fuse the EEG segment signals and the attention weights into EEG emotion features.

[0231] The cross-subject five-channel emotion modeling device provided by the embodiment of the present application can execute the cross-subject five-channel emotion modeling method provided by any embodiment of the present application, has the corresponding function modules and beneficial effects of executing the cross-subject five-channel emotion modeling method.

[0232] Embodiment five

[0233] Referring to Figure 9 , a structural schematic diagram of an emotion regulation device provided by an embodiment of the present application is shown.

[0234] As Figure 9 shown, the device comprises:

[0235] The EEG signal acquisition module 901 is configured to call the brain-computer interface to collect original EEG signals of a user under O1, O2, C3, C4 and FP2 channels.

[0236] The preprocessing module 902 is configured to preprocess the original EEG signals to obtain target EEG signals.

[0237] The EEG signal segmentation module 903 is configured to segment the target EEG signals into a plurality of EEG segment signals.

[0238] The emotion classifier loading module 904 is configured to load an emotion classifier trained according to the method of embodiment one.

[0239] The desired emotion generation module 905 is configured to input the EEG segment signals into the emotion classifier to identify a desired emotion generated by the user under the attention of the O1, O2, C3, C4 and FP2 channels.

[0240] The business operation execution module 906 is configured to execute a business operation according to the desired emotion to guide the user to regulate the emotion.

[0241] In an embodiment of the present application, further comprising:

[0242] The actual emotion determination module is configured to determine an actual emotion induced by the user in a current time period.

[0243] The emotion classifier updating module is configured to update the emotion classifier in the current time period under the attention of the O1, O2, C3, C4 and FP2 channels, taking the EEG segment signals as samples and the actual emotion as labels.

[0244] an emotion classifier replacement module, configured to replace the current emotion classifier with the updated emotion classifier in a next time period.

[0245] In an embodiment of the present application, the emotion classifier replacement module comprises:

[0246] a performance index calculation module, configured to calculate a first performance index of the updated emotion classifier in identifying emotions of the user and a second performance index of the current emotion classifier in identifying emotions of the user in a next time period;

[0247] an emotion classifier allowed replacement module, configured to allow the updated emotion classifier to replace the current emotion classifier if the first performance index is superior to the second performance index;

[0248] an emotion classifier prohibited replacement module, configured to prohibit the updated emotion classifier from replacing the current emotion classifier if the first performance index is inferior to the second performance index.

[0249] In an embodiment of the present application, the preprocessing module 902 comprises:

[0250] a first filtering module, configured to input the original electroencephalogram signal into a second-order Butterworth high-pass filter to filter out signals less than a first frequency threshold, to obtain a first candidate electroencephalogram signal;

[0251] a second filtering module, configured to input the first candidate electroencephalogram signal into a second-order band-stop Butterworth filter to filter out signals between a second frequency threshold and a third frequency threshold, to obtain a second candidate electroencephalogram signal;

[0252] a third filtering module, configured to input the second candidate electroencephalogram signal into a second-order low-pass Butterworth high filter to filter out signals greater than a fourth frequency threshold, to obtain a target electroencephalogram signal;

[0253] wherein the first frequency threshold is less than the second frequency threshold, the second frequency threshold is less than the third frequency threshold, and the third frequency threshold is less than the fourth frequency threshold.

[0254] In an embodiment of the present application, the service operation execution module 906 comprises:

[0255] an ideal emotion determination module, configured to determine an ideal emotion evoked in the user by the multimedia data;

[0256] an evocation accuracy calculation module, configured to calculate a ratio between a co-occurrence quantity and a total quantity of the ideal emotion, to obtain an accuracy of the user-evoked emotion; the co-occurrence quantity is a quantity of times that the expected emotion is the same as the ideal emotion.

[0257] The adjusting operation execution module is configured to perform a service operation according to the accuracy, so as to guide the user to adjust the emotion.

[0258] In an embodiment of the present application, the EEG signal segmentation module 903 comprises:

[0259] The window adding module is configured to add a window to the target EEG signal.

[0260] The window sliding module is configured to extract a signal in the window as an EEG segment signal each time the window is slid by a preset step length.

[0261] The width of the window is greater than the step length.

[0262] In an embodiment of the present application, the emotion classifier comprises a channel attention module and an emotion classification head; and the expected emotion generation module 905 comprises:

[0263] The EEG emotion feature generation module is configured to input the EEG segment signal into the channel attention module to add attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, so as to obtain an EEG emotion feature.

[0264] The expected emotion recognition module is configured to input the EEG emotion feature into the emotion classification head to recognize an expected emotion generated by the user.

[0265] In an embodiment of the present application, the channel attention module comprises two fully connected layers.

[0266] The EEG emotion feature generation module comprises:

[0267] The pooling module is configured to perform an average pooling operation on the EEG segment signal to obtain a first candidate EEG feature.

[0268] The first fully connected module is configured to input the first candidate EEG feature into the first fully connected layer to map the first candidate EEG feature into a second channel feature.

[0269] The first activation module is configured to use a hyperbolic tangent function Tanh to activate the second channel feature to obtain a third channel feature.

[0270] The second fully connected module is configured to input the third channel feature into the second fully connected layer to map the third channel feature into a fourth channel feature.

[0271] The second activation module is configured to activate the fourth channel feature by using a flexible maximum transfer function Softmax to obtain attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0272] The fusion module is configured to fuse the electroencephalogram segment signal and the attention weights into an electroencephalogram emotion feature.

[0273] The emotion regulation device provided in the embodiments of the present application can execute the emotion regulation method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the emotion regulation method.

[0274] Embodiment six

[0275] Referring to Figure 10 , a structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0276] As Figure 10 shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0277] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0278] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the cross-subject five-channel emotion modeling method or the emotion regulation method.

[0279] In some embodiments, the cross-subject five-channel emotion modeling method or the emotion regulation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the cross-subject five-channel emotion modeling method or the emotion regulation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the cross-subject five-channel emotion modeling method or the emotion regulation method by any other appropriate means, such as by means of firmware.

[0280] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0281] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or entirely on a remote machine or server.

[0282] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0283] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0284] The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0285] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0286] Embodiment seven

[0287] The embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the cross-subject five-channel emotion modeling method or the emotion regulation method provided by any of the embodiments of the present application.

[0288] The computer program product, in the implementation process, can be written in one or more programming languages or combinations thereof to implement the computer program code for performing the operations of the present application, the programming languages including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on a user computer, partially on a user computer, as an independent software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0289] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0290] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for modeling emotions across five channels in subjects, characterized in that, include: When playing multimedia data to a user, the brain-computer interface is invoked to collect the user's raw electroencephalogram (EEG) signals in the O1, O2, C3, C4 and FP2 channels; Determine the ideal emotion induced by the multimedia data for the user; The raw EEG signal is preprocessed to obtain the target EEG signal; The target EEG signal is segmented into multiple EEG segment signals; Under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, an emotion classifier is trained using the EEG segment signals as samples and the ideal emotion as labels; The step of training an emotion classifier using the EEG segment signals as samples and the ideal emotion as labels under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel, and the FP2 channel includes: Load the emotion classifier; the emotion classifier includes a channel attention module and an emotion classification head; The EEG segment signal is input into the channel attention module, and attention weights under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel are added to obtain EEG emotional features; The EEG emotion features are input into the emotion classification head to identify the user's expected emotion; The emotion classifier is updated based on the loss value between the ideal emotion and the expected emotion. If the update is completed, determine whether the preset training conditions are met; if yes, determine that the emotion classifier has completed training; if no, return to the step of adding the attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel to the attention module of the EEG segment signal input channel to obtain the EEG emotion features. The channel attention module includes two fully connected layers; The process involves inputting the EEG segment signal into the channel attention module, adding attention weights for the O1, O2, C3, C4, and FP2 channels to obtain EEG emotional features, including: An average pooling operation is performed on the EEG segment signal to obtain the first candidate EEG feature; The first candidate EEG feature is input into the first fully connected layer and mapped to the second channel feature. The second channel feature is activated using the hyperbolic tangent function Tanh to obtain the third channel feature; The third channel feature is input into the second fully connected layer and mapped to the fourth channel feature. The fourth channel feature is activated using the Softmax function to obtain the attention weights for the O1 channel, the O2 channel, the C3 channel, the C4 channel, and the FP2 channel. The EEG segment signals are fused with the attention weights to form EEG emotion features.

2. The cross-subject five-channel emotion modeling method according to claim 1, characterized in that, The preprocessing of the raw EEG signal to obtain the target EEG signal includes: The original EEG signal is input into a second-order Butterworth high-pass filter to filter out signals that are less than the first frequency threshold, thereby obtaining the first candidate EEG signal. The first candidate EEG signal is input into a second-order band-stop Butterworth filter to filter out signals between the second and third frequency thresholds, thus obtaining the second candidate EEG signal. The second candidate EEG signal is input into a second-order low-pass Butterworth high-frequency filter to filter out signals greater than the fourth frequency threshold, thereby obtaining the target EEG signal. Wherein, the first frequency threshold is less than the second frequency threshold, the second frequency threshold is less than the third frequency threshold, and the third frequency threshold is less than the fourth frequency threshold.

3. The cross-subject five-channel emotion modeling method according to claim 1, characterized in that, The step of segmenting the target EEG signal into multiple EEG segment signals includes: Add a window to the target EEG signal; Each time the window is slid according to a preset step size, the signal located in the window is extracted as an EEG segment signal; The width of the window is greater than the step size.

4. An emotion regulation method, characterized in that, include: When playing multimedia data to a user, the brain-computer interface is invoked to collect the user's raw electroencephalogram (EEG) signals in the O1, O2, C3, C4 and FP2 channels; The raw EEG signal is preprocessed to obtain the target EEG signal; The target EEG signal is segmented into multiple EEG segment signals; Load the emotion classifier trained by the cross-subject five-channel emotion modeling method according to any one of claims 1-3; The EEG segment signal is input into the emotion classifier, and the user's desired emotion is identified under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel; Perform business operations based on the expected emotions to guide the user in regulating their emotions.

5. The emotion regulation method according to claim 4, characterized in that, Also includes: Within the current time period, determine the actual emotions evoked by the user; During the current time period, under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, the emotion classifier is updated using the EEG segment signal as a sample and the actual emotion as a label; In the next time period, the updated emotion classifier will replace the current emotion classifier; The step of performing business operations based on the expected emotion to guide the user in regulating their emotions includes: Determine the ideal emotion induced by the multimedia data for the user; The accuracy of the user-induced emotion is obtained by calculating the ratio between the co-occurrence count and the total number of the ideal emotion; the co-occurrence count is the number of times the expected emotion and the ideal emotion are the same. Perform business operations with the stated accuracy to guide the user in regulating their emotions.

6. The emotion regulation method according to claim 5, characterized in that, The step of replacing the current emotion classifier with the updated one in the next time period includes: In the next time period, a first performance metric for recognizing the user's emotions is calculated for the updated emotion classifier, and a second performance metric for recognizing the user's emotions is calculated for the current emotion classifier. If the first performance metric is better than the second performance metric, then the updated emotion classifier is allowed to replace the current emotion classifier. If the first performance metric is inferior to the second performance metric, then replacing the current emotion classifier with the updated emotion classifier is prohibited.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to implement the cross-subject five-channel emotion modeling method as described in any one of claims 1-3, or to implement the emotion regulation method as described in any one of claims 4-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the cross-subject five-channel emotion modeling method as described in any one of claims 1-3, or implements the emotion regulation method as described in any one of claims 4-6.

Images