Cross-subject five-channel emotion modeling and emotion adjusting method and device and storage medium

Abstract

The invention discloses a cross-subject five-channel emotion modeling and emotion adjusting method and device and a storage medium, and the method comprises the steps: calling a brain-computer interface to collect original electroencephalogram signals of a user under an O1 channel, an O2 channel, a C3 channel, a C4 channel and an FP2 channel when multimedia data is played for the user; determining an actual emotion induced by the multimedia data to the user; preprocessing the original electroencephalogram signal to obtain a target electroencephalogram signal; segmenting the target electroencephalogram signal into a plurality of electroencephalogram fragment signals; under the attention of an O1 channel, an O2 channel, a C3 channel, a C4 channel and an FP2 channel, electroencephalogram fragment signals are used as samples, actual emotions are used as labels, and an emotion classifier is trained. Visual and auditory feedback is provided according to the emotion classification evaluation result, and the user performs emotion adjustment / training according to the feedback. According to the embodiment, the number of channels is greatly reduced, so that the data volume of the electroencephalogram signals is greatly reduced, resources consumed during operation are effectively reduced, time consumed during operation is shortened, and real-time performance is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interfaces, and in particular to a method, device and storage medium for five-channel emotion modeling and emotion regulation across subjects. Background Art

[0002] In business scenarios such as watching movies, playing motion games, and mindfulness meditation, the user's emotions (such as happy, neutral, sad, etc.) can be monitored to assist operations in the business scenarios.

[0003] Furthermore, based on the brain-computer interface, 32-channel or 64-channel EEG signals of the user are collected, time domain or frequency domain features are constructed from the EEG signals, and the features are input into SVM (Support Vector Machine) to infer the user's emotions.

[0004] In this process, the large number of channels of EEG signals results in a large amount of data and complicated feature engineering, such as Fourier transform (FFT), power spectral density (PSD), wavelet transform, etc., which results in high computational time. Summary of the invention

[0005] In view of this, the present invention provides a method, device and storage medium for five-channel emotion modeling and emotion regulation across subjects, so as to reduce the time consumption of detecting user emotions.

[0006] A first aspect of the present invention provides a five-channel emotion modeling method across subjects, comprising:

[0007] When multimedia data is played to a user, the brain-computer interface is called to collect original EEG signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0008] determining an ideal emotion induced by the multimedia data on the user;

[0009] Preprocessing the original EEG signal to obtain a target EEG signal;

[0010] Dividing the target EEG signal into a plurality of EEG segment signals;

[0011] Under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, an emotion classifier is trained with the EEG segment signal as a sample and the ideal emotion as a label.

[0012] A second aspect of the present invention provides an emotion regulation method, comprising:

[0013] When multimedia data is played to a user, the brain-computer interface is called to collect original EEG signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0014] Preprocessing the original EEG signal to obtain a target EEG signal;

[0015] Dividing the target EEG signal into a plurality of EEG segment signals;

[0016] Loading an emotion classifier trained according to any one of the methods of claims 1-5;

[0017] Inputting the EEG segment signal into the emotion classifier, and identifying the desired emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0018] A business operation is performed according to the expected emotion to guide the user to adjust the emotion.

[0019] A third aspect of the present invention provides a cross-subject five-channel emotion modeling device, comprising:

[0020] The EEG signal acquisition module is used to call the brain-computer interface to collect the original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel when playing multimedia data to the user;

[0021] An ideal emotion determination module, used to determine the ideal emotion induced by the multimedia data to the user;

[0022] A preprocessing module, used for preprocessing the original EEG signal to obtain a target EEG signal;

[0023] An EEG signal segmentation module, used for segmenting the target EEG signal into a plurality of EEG segment signals;

[0024] The emotion classifier training module is used to train the emotion classifier with the EEG segment signal as a sample and the ideal emotion as a label under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0025] A fourth aspect of the present invention provides an emotion regulation device, comprising:

[0026] The EEG signal acquisition module is used to call the brain-computer interface to collect the original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel when playing multimedia data to the user;

[0027] A preprocessing module, used for preprocessing the original EEG signal to obtain a target EEG signal;

[0028] An EEG signal segmentation module, used for segmenting the target EEG signal into a plurality of EEG segment signals;

[0029] An emotion classifier loading module, used to load the emotion classifier trained according to the method described in Embodiment 1;

[0030] an expected emotion generation module, used for inputting the EEG segment signal into the emotion classifier, and identifying the expected emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0031] The business operation execution module is used to execute business operations according to the expected emotion to guide the user to adjust the emotion.

[0032] A fifth aspect of the present invention provides an electronic device, the electronic device comprising:

[0033] at least one processor; and

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cross-subject five-channel emotion modeling method as described in the first aspect or the emotion regulation method as described in the second aspect.

[0036] The sixth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the cross-subject five-channel emotion modeling method as described in the first aspect or the emotion regulation method as described in the second aspect.

[0037] The seventh aspect of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the cross-subject five-channel emotion modeling method as described in the first aspect or the emotion regulation method as described in the second aspect.

[0038] In this embodiment, when multimedia data is played to the user, the brain-computer interface is called to collect the original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel; the ideal emotion induced by the multimedia data to the user is determined; the original EEG signal is preprocessed to obtain the target EEG signal; the target EEG signal is divided into multiple EEG segment signals; under the attention of the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel, the emotion classifier is trained with the EEG segment signal as a sample and the ideal emotion as a label. This embodiment focuses on the EEG signals of the five channels of O1, O2, C3, C4 and FP2 for emotion recognition, which not only retains the information related to emotion recognition in the EEG signals and ensures the accuracy of emotion recognition, but also greatly reduces the number of channels, thereby greatly reducing the amount of EEG signal data. On this basis, deep learning can be introduced to simplify feature engineering, which can effectively reduce the resources consumed during calculations and reduce the time consumption of calculations, thereby ensuring real-time performance.

[0039] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a flowchart of a five-channel emotion modeling method across subjects provided in Example 1 of the present invention.

[0042] Figure 2 This is an example diagram of an electronic device provided in Embodiment 1 of the present invention.

[0043] Figure 3 This is an example diagram of a head mounted device provided in Embodiment 1 of the present invention.

[0044] Figure 4 It is a schematic diagram of an emotion classifier provided in the first embodiment of the present invention.

[0045] Figure 5 It is a schematic diagram of a channel attention module provided in Example 1 of the present invention.

[0046] Figure 6 This is a flow chart of an emotion regulation method provided in Embodiment 2 of the present invention.

[0047] Figure 7 This is a flow chart of an emotion regulation method provided in Embodiment 3 of the present invention.

[0048] Figure 8 It is a structural schematic diagram of a five-channel emotion modeling device across subjects provided in Embodiment 4 of the present invention.

[0049] Fig. 9 It is a structural schematic diagram of an emotion regulation device provided in Embodiment 5 of the present invention.

[0050] Fig.10 It is a structural schematic diagram of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can cover sequential implementations other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] Embodiment 1

[0054] See also Figure 1 , showing a flow chart of a cross-subject five-channel emotion modeling method provided in Example 1 of the present invention. The method can be executed by a cross-subject five-channel emotion modeling device. The cross-subject five-channel emotion modeling device can be implemented in the form of hardware and / or software. The cross-subject five-channel emotion modeling device can be configured in an electronic device.

[0055] In one case, the electronic device is an integrated device, especially a head-mounted device, such as a headband, a helmet, glasses, etc., wherein a computing unit and a brain-computer interface are provided in the electronic device, and the computing unit is loaded with an emotion classifier.

[0056] In another case, the electronic device may also be a physically separable or detachable device. Figure 2 As shown, the electronic device includes a head-mounted device 201 and a computing device 202 which are independent of each other. The computing device 202 can be deployed locally or in the cloud and is loaded with an emotion classifier. The head-mounted device 201 and the computing device 202 can be connected to each other by wire or wirelessly via a local area network or the Internet, wherein the head-mounted device 201 is configured with a brain-computer interface.

[0057] like Figure 1 As shown, the method includes:

[0058] Step 101: When playing multimedia data to a user, the brain-computer interface is called to collect original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0059] In actual applications, the 10-20 system includes 19 recording electrodes and 2 reference electrodes.

[0060] Two lines were determined on the scalp surface, one was the front-to-back line from the nasion to the external occipital protuberance, which was 100%, and the other was the left-right line between the two preauricular concavities, which was 100%. The intersection of the two lines on the top of the head was the location of the Cz electrode. FPz (frontal pole midline) was 10% backward from the nasion, and every 20% backward from FPz was the location of an electrode, which were Fz (frontal midline), Cz (central midline), Pz (top midline) and Oz (occipital midline).

[0061] The distance between Oz and the external occipital protuberance is 10%. The position of the T3 (left middle temporal) electrode is 10% from the left preauricular concave on the line connecting the two preauricular concave. After that, an electrode is placed every 20% to the right, which are C3 (left central), Cz, C4 (right central) and T4 (right middle temporal). The distance between T4 and the right preauricular concave is 10%. The line from FPz through T3 to Oz is the left temporal line. From FPz to the left 10% is FP1 (left frontal pole). From FP1 to the back every 20% is an electrode, which are F7 (left anterior temporal), T3 (left middle temporal), T5 (left posterior temporal) and O1 (left occipital). Among them, T3 is the intersection of this line and the line connecting the two preauricular concave. O1 is 10% away from Oz. The right temporal line corresponds to this. From front to back, they are FP2 (right temporal pole), F8 (right anterior temporal), T4 (right middle temporal), T6 (right posterior temporal), and O2 (right occipital).

[0062] A line was made from FP1 to O1 and from FP2 to O2, which were the left and right parasagittal lines. From FP1 and FP2, every 20% backward was an electrode site. The left side was F3 (left frontal), C3 (left central), P3 (left parietal) and O1 (left occipital), and the right side was F4 (right frontal), C4 (right central), P4 (right parietal) and O2 (right occipital). In the 10-20 system, FPz and Oz were not included in the 19 recording sites.

[0063] The process of inducing user emotions is relatively complex, involving the interaction of multiple specific brain regions. This embodiment selects five channels, O1, O2, C3, C4 and FP2, through experiments and other means. These five channels involve multiple brain regions such as the frontal region, central region, and parietal region, and can effectively collect EEG signals from emotion-inducing brain regions. Compared with 32 channels or 64 channels, the number of channels is greatly reduced, thereby greatly reducing the amount of EEG signal data, and maintaining the amount of information in the EEG signal that is highly correlated with emotions, thereby maintaining the accuracy of emotion recognition.

[0064] It should be pointed out that the five channels O1, O2, C3, C4 and FP2 are only used as examples and do not constitute a limitation on the scope of protection of the present invention. In addition to the five channels O1, O2, C3, C4 and FP2, various modifications, combinations, sub-combinations and substitutions can be made to the channels according to design requirements and other factors.

[0065] In training mode, when the user wears the head-mounted device, the electrodes in the brain-computer interface can be attached to the positions corresponding to O1, O2, C3, C4 and FP2 (such as Figure 3 Position 4, position 5, position 2, position 3, position 1) as shown, and use other devices to play multimedia data to the user, and when the user feels the multimedia data, corresponding emotions are induced.

[0066] Among them, multimedia data may include text data, audio data, video data, etc., and its forms are diverse, for example, novels, crosstalk, songs, short videos, etc.

[0067] In order to ensure that the content of multimedia data is focused and to induce clear emotions in users, the duration of multimedia data is usually short, such as 2 minutes, 4 minutes, etc.

[0068] In order to improve the accuracy of the test, the user may be prompted with the emotional state to be maintained in the following period before the multimedia data is played, so that the user can adjust his or her emotional state for a short period of time (eg, 3 to 10 seconds).

[0069] Considering the positions of O1, O2, C3, C4 and FP2, dry electrodes can be used in the brain-computer interface. Compared with wet electrodes, dry electrodes are easier for users to wear.

[0070] At this time, the EEG signals under the five channels (O1, O2, C3, C4 and FP2) collected by the brain-computer interface can be read, and the timestamp of the EEG signal can be aligned with the timestamp of the multimedia data. When the alignment is completed, the EEG signal when playing the multimedia data is cut out and recorded as the original EEG signal.

[0071] Step 102: Determine the ideal emotion induced by the multimedia data in the user.

[0072] In a specific implementation, each multimedia data may be labeled with a corresponding label (ie, emotion) in advance according to the content of each multimedia data. Then, the label labeled on the multimedia data may be queried according to data such as ID as the ideal emotion induced by the user.

[0073] Furthermore, multimedia data may be labeled manually, or may be automatically labeled using multimedia attribute information (such as type, etc.), NLP (Natural language processing), computer vision and other technologies, etc., and this embodiment does not impose any restrictions on this.

[0074] For example, when the type of multimedia data is crosstalk, "happy" may be automatically labeled as crosstalk.

[0075] For another example, when the multimedia data is a short video, the comment information input by a group of users on the short video can be collected, and the emotions expressed by the group of users in the comment information can be analyzed using NLP technology, and the emotions can be marked as labels for the short video.

[0076] In addition, when the user's expected emotion is subsequently identified, feedback can be given to the user visually and auditorily, and the user can judge whether his or her emotional state is consistent with the expected emotion. When it is consistent, the ideal emotion is corrected with the expected emotion. When it is inconsistent, the correct ideal emotion is provided.

[0077] Step 103: pre-process the original EEG signal to obtain a target EEG signal.

[0078] In this embodiment, various preprocessing steps, such as amplification, filtering, normalization, etc., may be performed on the original EEG signal to obtain a target EEG signal, so as to improve the quality of the target EEG signal.

[0079] During filtering, the original EEG signal can be input into a second-order Butterworth high-pass filter to filter out low-frequency signals less than a first frequency threshold (such as 0.1 Hz, etc.) to obtain a first candidate EEG signal.

[0080] The first candidate EEG signal is input into a second-order band-stop Butterworth filter to filter out high-frequency signals between a second frequency domain threshold (such as 48 Hz) and a third frequency threshold (such as 52 Hz), thereby obtaining a second candidate EEG signal.

[0081] The second candidate EEG signal is input into a second-order low-pass Butterworth filter to filter out high-frequency signals greater than a fourth frequency threshold (such as 70 Hz) to obtain a target EEG signal.

[0082] The first frequency threshold is smaller than the second frequency domain threshold, the second frequency domain threshold is smaller than the third frequency threshold, and the third frequency threshold is smaller than the fourth frequency threshold.

[0083] Step 104: Divide the target EEG signal into a plurality of EEG segment signals.

[0084] In this embodiment, an end-to-end emotion classifier can be constructed based on deep learning under the condition that the data volume of the EEG signal is greatly reduced.

[0085] At this time, the target EEG signal can be divided into equal parts to obtain multiple EEG segment signals, which can be used as the input of the emotion classifier, omitting feature engineering, simplifying the processing flow, and reducing computing time.

[0086] In a specific implementation, a sliding window may be added to the target EEG signal starting from the endpoint of the target EEG signal.

[0087] Each time the window is slid according to a preset step length, the signal in the window is extracted as an EEG segment signal.

[0088] The width of the window is greater than the sliding step size. For example, the width of the window is 5 seconds and the sliding step size is 3 seconds.

[0089] Step 105 , under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, the emotion classifier is trained with the EEG segment signals as samples and the ideal emotions as labels.

[0090] In practical applications, the emotion classifier is embedded with a channel-based attention mechanism. Through the attention mechanism, the characteristic information of EEG segment signals in five channels (O1, O2, C3, C4 and FP2) can be explored, and attention between the five channels (O1, O2, C3, C4 and FP2) can be added to the EEG segment signals to enhance the weight of emotion-related signals, in order to increase the stability of the emotion classifier and the accuracy of identifying user emotions. In this case, the emotion classifier can be supervised trained with EEG segment signals of a small number of test users (such as 15 users) as samples and ideal emotions as labels, so that the emotion classifier can realize user emotion recognition within subjects, across subjects and across time.

[0091] Since the data volume of the EEG signals of the five channels (O1, O2, C3, C4 and FP2) is small, if the emotion classifier selects random parameters to start training, the accuracy of identifying user emotions will be low in the initial stage of training. Therefore, this embodiment adopts a pre-training method to obtain a general emotion classifier, which is convenient for the emotion classifier to learn the commonalities of EEG signals between different users under the head-mounted device, so as to improve the accuracy of emotion recognition.

[0092] In one embodiment of the present invention, step 105 may include the following steps:

[0093] Step 1051: Load the emotion classifier.

[0094] In this embodiment, the constructed emotion classifier can be loaded into the memory and run, such as Figure 4 As shown in the figure, the emotion classifier includes a channel attention module and an emotion classification head.

[0095] Among them, the channel attention module Attention Module is used to add weights to the EEG signals based on the attention between the five channels (O1, O2, C3, C4 and FP2).

[0096] The emotion classification head identifies the user's emotions based on the characteristics of EEG signals.

[0097] Furthermore, the structure of the emotion classification head is not limited to artificially designed neural networks. For example, the emotion classification head can reuse previously pre-trained models, such as the electroencephalogram network EEGNet. At this time, the output layer of the pre-trained model is modified according to the classification task of user emotion recognition, such as FC (Fully Connected Layer), etc., to save training costs and maintain high accuracy. The neural network can also be optimized by a model quantization method, and the neural network can be searched for the characteristics of EEG signals by a NAS (Neural Architecture Search) method, etc. This embodiment does not impose any restrictions on this.

[0098] Step 1052: Input the EEG segment signal into the channel attention module, add the attention weights of the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel, and obtain the EEG emotion feature.

[0099] In this embodiment, if Figure 4 As shown, the EEG fragment signals of five channels (O1, O2, C3, C4 and FP2) can be input into the channel attention module, and the attention weights under the five channels (O1, O2, C3, C4 and FP2) can be added to obtain the EEG emotion features.

[0100] For example, Figure 5 As shown, the channel attention module consists of two fully connected layers FC.

[0101] In this example, an average pooling operation Avg Pooling may be performed on the EEG segment signal to obtain a first candidate EEG feature.

[0102] The first candidate EEG feature is input into the first fully connected layer FC and mapped to the second channel feature.

[0103] The hyperbolic tangent function Tanh is used to activate the second channel feature to obtain the third channel feature.

[0104] The third channel feature is input into the second fully connected layer FC and mapped to the fourth channel feature.

[0105] The flexible maximum transfer function Softmax is used to activate the fourth channel feature to obtain the attention weights of the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0106] Use Concat, Add and other functions to fuse EEG fragment signals and attention weights into EEG emotion features.

[0107] Step 1053: The EEG emotion features are input into the emotion classification head to identify the desired emotions generated by the user.

[0108] In this embodiment, if Figure 4 As shown, the EEG emotion features are input into the emotion classification head. The emotion classification head classifies the EEG emotion features according to its structure, identifies the emotions generated by the user (such as happiness, neutrality), and records them as expected emotions.

[0109] Generally, the emotion classification head outputs the probability score of each emotion and selects the emotion with the highest probability score as the expected emotion generated by the user.

[0110] Step 1054: Update the emotion classifier according to the loss value between the ideal emotion and the expected emotion.

[0111] For the same EEG signal, its ideal emotion and expected emotion can be brought into the preset loss function (such as cross entropy, etc.) to calculate the loss value, and the loss value can be substituted into optimization algorithms such as SGD (stochastic gradient descent) and Adam (Adaptive momentum) to calculate the update amplitude of the parameters in the emotion classifier (channel attention module Attention Module and emotion classification head Head), and the parameters in the emotion classifier (channel attention module Attention Module and emotion classification head Head) are updated according to the update amplitude.

[0112] Step 1055: If the update is completed, determine whether the preset training conditions are met; if so, execute step 1056; if not, return to execute step 1051.

[0113] Step 1056: Determine whether the emotion classifier has completed training.

[0114] In this embodiment, training conditions can be set in advance as conditions for stopping training the emotion classifier (channel attention module Attention Module and emotion classification head Head), for example, the number of iterations reaches a certain threshold, the loss value is less than a certain threshold, the change in the loss value in multiple iterative training is less than a certain threshold, and so on.

[0115] In each round of iterative training, if the parameters of the emotion classifier (channel attention module AttentionModule and emotion classification head Head) are updated, it can be determined whether the training conditions are met in this round of iterative training.

[0116] If the training conditions are met, it can be considered that the emotion classifier (channel attention module and emotion classification head) has completed training.

[0117] If the training conditions are not met, the next round of iterative training can be entered, and steps 1051 to 1055 are re-executed, and the iterative training is repeated in this way until the emotion classifier (channel attention module Attention Module and emotion classification head Head) completes the training.

[0118] In this embodiment, when multimedia data is played to the user, the brain-computer interface is called to collect the original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel; the ideal emotion induced by the multimedia data to the user is determined; the original EEG signal is preprocessed to obtain the target EEG signal; the target EEG signal is divided into multiple EEG segment signals; under the attention of the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel, the emotion classifier is trained with the EEG segment signal as a sample and the ideal emotion as a label. This embodiment focuses on the EEG signals of the five channels of O1, O2, C3, C4 and FP2 for emotion recognition, which not only retains the information related to emotion recognition in the EEG signals and ensures the accuracy of emotion recognition, but also greatly reduces the number of channels, thereby greatly reducing the amount of EEG signal data. On this basis, deep learning can be introduced to simplify feature engineering, which can effectively reduce the resources consumed during calculations and reduce the time consumption of calculations, thereby ensuring real-time performance.

[0119] Embodiment 2

[0120] See also Figure 6 , showing a flow chart of an emotion regulation method provided in Embodiment 2 of the present invention, the method can be executed by an emotion regulation device, the cross-subject five-channel emotion modeling and emotion regulation device can be implemented in the form of hardware and / or software, and the emotion regulation device can be configured in an electronic device.

[0121] In one case, the electronic device is an integrated device, especially a head-mounted device, such as a headband, a helmet, glasses, etc., wherein a brain-computer interface is provided in the electronic device.

[0122] In another case, the electronic device may also be a physically separable or detachable device. In this case, the electronic device includes a head-mounted device and a computing device that are independent of each other. The computing device can be deployed locally or in the cloud. The head-mounted device and the computing device can be connected via a local area network or the Internet, wherein a brain-computer interface is configured in the head-mounted device.

[0123] like Figure 6 As shown, the method includes:

[0124] Step 601: When playing multimedia data to the user, the brain-computer interface is called to collect the original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0125] In free mode, users use other devices to play multimedia data to users in business scenarios such as watching movies, playing somatosensory games, and mindfulness meditation. When users feel the multimedia data, corresponding emotions are induced. During this period, when wearing a head-mounted device, the electrodes in the brain-computer interface can be pasted to the positions corresponding to O1, O2, C3, C4 and FP2.

[0126] Generally, multimedia data is customized content in business scenarios, and the length of multimedia data varies, such as 5 minutes, 1 hour, etc., which can induce one or more emotions in users.

[0127] At this time, the EEG signals under the five channels (O1, O2, C3, C4 and FP2) collected by the brain-computer interface can be read, and the timestamp of the EEG signal can be aligned with the timestamp of the multimedia data. When the alignment is completed, the EEG signal when playing the multimedia data is cut out and recorded as the original EEG signal.

[0128] Step 602: pre-process the original EEG signal to obtain a target EEG signal.

[0129] In this embodiment, various preprocessing steps, such as amplification, filtering, normalization, etc., may be performed on the original EEG signal to obtain a target EEG signal, so as to improve the quality of the target EEG signal.

[0130] During filtering, the original EEG signal can be input into a second-order Butterworth high-pass filter to filter out low-frequency signals less than a first frequency threshold (such as 0.1 Hz, etc.) to obtain a first candidate EEG signal.

[0131] The first candidate EEG signal is input into a second-order band-stop Butterworth filter to filter out high-frequency signals between a second frequency domain threshold (such as 48 Hz) and a third frequency threshold (such as 52 Hz), thereby obtaining a second candidate EEG signal.

[0132] Input the second candidate EEG signal into a second-order low-pass Butterworth high frequency filter to filter out high-frequency signals greater than a fourth frequency threshold (such as 70 Hz), thereby obtaining a target EEG signal;

[0133] The first frequency threshold is smaller than the second frequency domain threshold, the second frequency domain threshold is smaller than the third frequency threshold, and the third frequency threshold is smaller than the fourth frequency threshold.

[0134] Step 603: Divide the target EEG signal into multiple EEG segment signals.

[0135] In this embodiment, an end-to-end emotion classifier can be constructed based on deep learning under the condition that the data volume of the EEG signal is greatly reduced.

[0136] At this time, the target EEG signal can be divided into equal parts to obtain multiple EEG segment signals, which can be used as the input of the emotion classifier, omitting feature engineering, simplifying the processing flow, and reducing computing time.

[0137] In a specific implementation, a sliding window may be added to the target EEG signal starting from the endpoint of the target EEG signal.

[0138] Each time the window is slid according to a preset step length, the signal in the window is extracted as an EEG segment signal.

[0139] The width of the window is greater than the sliding step size. For example, the width of the window is 5 seconds and the sliding step size is 3 seconds.

[0140] Step 604: Load the emotion classifier.

[0141] In this embodiment, the emotion classifier constructed and trained in advance based on the method of embodiment 1 can be loaded into the memory for execution.

[0142] Step 605: Input the EEG segment signal into the emotion classifier, and identify the desired emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0143] In practical applications, the emotion classifier is embedded with a channel-based attention mechanism, which can input EEG segment signals into the emotion classifier. Through the attention mechanism, the characteristic information of EEG segment signals in five channels (O1, O2, C3, C4 and FP2) can be explored, and the attention between the five channels (O1, O2, C3, C4 and FP2) can be added to the EEG segment signals to enhance the weight of emotion-related signals, in order to increase the stability of the emotion classifier and the accuracy of identifying user emotions.

[0144] Furthermore, this embodiment provides an emotion classifier across users and across time. When using the emotion classifier, it does not rely on pre-collecting the EEG signal of the current user for training. The current user can directly use the emotion classifier to identify his current emotional state.

[0145] In one embodiment of the present invention, the emotion classifier includes a channel attention module and an emotion classification head.

[0146] Among them, the channel attention module Attention Module is used to add weights to the EEG signals based on the attention between the five channels (O1, O2, C3, C4 and FP2).

[0147] The emotion classification head identifies the user's emotions based on the characteristics of EEG signals.

[0148] Furthermore, the structure of the emotion classification head is not limited to artificially designed neural networks. For example, the emotion classification head can reuse previously pre-trained models, such as the electroencephalogram network EEGNet. At this time, the output layer of the pre-trained model is modified according to the classification task of user emotion recognition, such as FC, to save training costs and maintain high accuracy. The neural network can also be optimized by a model quantization method, a neural network searched for the characteristics of EEG signals by a NAS method, and so on. This embodiment does not impose any restrictions on this.

[0149] In this embodiment, step 605 may include the following steps:

[0150] Step 6051: Input the EEG segment signal into the channel attention module, add the attention weights of the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel, and obtain the EEG emotion feature.

[0151] In this embodiment, the EEG segment signals of five channels (O1, O2, C3, C4 and FP2) can be input into the channel attention module, and the attention weights under the five channels (O1, O2, C3, C4 and FP2) can be added to obtain the EEG emotion features.

[0152] Exemplarily, the channel attention module includes two fully connected layers FC.

[0153] In this example, an average pooling operation Avg Pooling may be performed on the EEG segment signal to obtain a first candidate EEG feature.

[0154] The first candidate EEG feature is input into the first fully connected layer FC and mapped to the second channel feature.

[0155] The hyperbolic tangent function Tanh is used to activate the second channel feature to obtain the third channel feature.

[0156] The third channel feature is input into the second fully connected layer FC and mapped to the fourth channel feature.

[0157] The flexible maximum transfer function Softmax is used to activate the fourth channel feature to obtain the attention weights of the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0158] Use Concat, Add and other functions to fuse EEG fragment signals and attention weights into EEG emotion features.

[0159] Step 6052: The EEG emotion features are input into the emotion classification head to identify the desired emotions generated by the user.

[0160] In this embodiment, the EEG emotion features are input into the emotion classification head. The emotion classification head classifies the EEG emotion features according to its structure, identifies the emotions generated by the user, and records them as expected emotions.

[0161] Generally, the emotion classification head outputs the probability score of each emotion and selects the emotion with the highest probability score as the expected emotion generated by the user.

[0162] It is expected that emotions can provide users with more accurate feedback visually, allowing users to more clearly understand their current emotional state, so that this embodiment can effectively identify emotions for specific users.

[0163] In this embodiment, the brain-computer interface is called to collect the original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel; the original EEG signals are preprocessed to obtain the target EEG signals; the target EEG signals are divided into multiple EEG segment signals; the emotion classifier is loaded; the EEG segment signals are input into the emotion classifier, and the desired emotions generated by the user are identified under the attention of the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0164] Step 606: Execute business operations according to the expected emotions to guide the user to adjust emotions.

[0165] In this embodiment, some business scenarios have emotional requirements, and it is possible to identify whether the user's current expected emotion meets the emotional requirements of the business scenario. When the expected emotion deviates from the emotional requirements of the business scenario, the expected emotion is used in the business scenario to perform corresponding business operations to guide the user to adjust his emotions until they meet the emotional requirements of the business scenario.

[0166] In the specific implementation, the ideal emotion induced by the multimedia data to the user is determined. Then, for the same multimedia data, there are both ideal emotions and expected emotions. The ideal emotion is the emotion induced by the user that can be called by the content of the multimedia data. The expected emotion is the emotion predicted by the emotion classifier for the user based on the EEG signal. When the performance of the emotion classifier meets the requirements, the expected emotion can be considered to be the emotion actually generated by the user.

[0167] At this time, a variable can be generated, recorded as the number of co-occurrences, which is the number of times the expected emotion is the same as the ideal emotion, indicating that the user correctly induces emotions according to the guidance of the multimedia data.

[0168] Generally, the co-occurrence quantity is valid in one business scenario and is released when the business scenario ends.

[0169] In a business scenario, for the same multimedia data, if the ideal emotion is the same as the expected emotion, the co-occurrence number can be accumulated by 1; if the ideal emotion is different from the expected emotion, this operation can be ignored.

[0170] The ratio between the number of co-occurrences and the total number of ideal emotions is calculated to obtain the accuracy of the user's induced emotions, so as to execute the business operations provided by the current business scenario according to the accuracy to guide the user to adjust his emotions.

[0171] For example, on the UI (user interface), the accuracy of the user's induced emotions is displayed in the form of a bar graph, etc., which can intuitively show the user the accuracy of the induced emotions. Technologies such as LLM (Large Language Model) are used to generate guidance information for users to regulate their emotions based on the information of the user's induced emotions (such as the accuracy of induced emotions, the distribution of types of incorrectly induced emotions, etc.), and a virtual digital human is used to play the guidance information for regulating emotions to the user to guide the user to regulate his emotions.

[0172] This embodiment focuses on the EEG signals of five channels, namely O1, O2, C3, C4 and FP2, for emotion recognition. It not only retains the information related to emotion recognition in the EEG signals and ensures the accuracy of emotion recognition, thereby improving the efficiency of emotion regulation, but also greatly reduces the number of channels, thereby greatly reducing the amount of EEG signal data. On this basis, deep learning can be introduced to simplify feature engineering, which can effectively reduce the resources consumed during calculations and reduce the time consumption of calculations, thereby ensuring real-time performance.

[0173] Embodiment 3

[0174] See also Figure 7 , shows a flowchart of a method for five-channel emotion modeling and emotion regulation across subjects provided by the third embodiment of the present invention. This embodiment adds an operation of updating the emotion classifier on the basis of the previous embodiment. Figure 7 As shown, the method includes:

[0175] Step 701: When playing multimedia data to the user, the brain-computer interface is called to collect the original EEG signals of the user under the O1 channel, O2 channel, C3 channel, C4 channel and FP2 channel.

[0176] Step 702: pre-process the original EEG signal to obtain a target EEG signal.

[0177] Step 703: Divide the target EEG signal into multiple EEG segment signals.

[0178] Step 704: Load the emotion classifier.

[0179] Step 705: Input the EEG segment signal into the emotion classifier, and identify the desired emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0180] Step 706: Execute business operations according to the expected emotions to guide the user to adjust emotions.

[0181] Step 707: Determine the actual emotion induced by the user during the current time period.

[0182] In the process of identifying the expected emotion of the user, a plurality of shorter time periods may be divided, such as each 4 minutes as a time period.

[0183] In some cases, the actual emotion evoked by the user can be determined with the help of information in the business scenario.

[0184] Taking multimedia data as an example, we can use the attribute information of multimedia data (such as type, etc.), NLP, computer vision and other technologies, and label each multimedia data with a corresponding label (i.e. emotion) based on the content of each multimedia data as the actual emotion induced by the user.

[0185] For example, when the type of multimedia data is crosstalk, "happy" may be automatically labeled as crosstalk.

[0186] For another example, when the multimedia data is a short video, the comment information input by a group of users on the short video can be collected, and the emotions expressed by the group of users in the comment information can be analyzed using NLP technology, and the emotions can be marked as labels for the short video.

[0187] Step 708: In the current time period, under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, the emotion classifier is updated with the EEG segment signal as a sample and the actual emotion as a label.

[0188] In practical applications, attention can be added between O1, O2, C3, C4 and FP2 to the EEG segment signals, and the weights of emotion-related signals can be enhanced in order to increase the accuracy of identifying user emotions. In this case, the EEG segment signals of the current user in the current time period can be used as samples and the actual emotions as labels, and the emotion classifier can be updated in a supervised manner using techniques such as Fine-tuning, so that the emotion classifier can realize personalized user emotion recognition and improve the accuracy of the emotion classifier in cross-user emotion recognition.

[0189] In general, a lower learning rate can be used for supervised updates of the sentiment classifier, that is, the learning rate for updating the sentiment classifier is lower than the learning rate for training the sentiment classifier.

[0190] Under the condition of greatly reducing the amount of EEG signal data, the resource consumption of updating the emotion classifier can be reduced, thereby realizing online short-term and small-sample updating of the emotion classifier.

[0191] Step 709: In the next time period, the updated emotion classifier replaces the current emotion classifier.

[0192] If the training of the emotion classifier is completed in the current time period, the updated emotion classifier may replace the current emotion classifier in the next time period to provide emotion recognition services for users.

[0193] Further, in the next time period, the updated emotion classifier runs simultaneously with the current emotion classifier.

[0194] On the one hand, for the updated emotion classifier, the expected emotion output by the updated emotion classifier is compared with the actual emotion induced by the user, and a first performance indicator of the updated emotion classifier under the recognition of the user's emotion is calculated.

[0195] On the other hand, for the current emotion classifier, the expected emotion output by the current emotion classifier is compared with the actual emotion induced by the user, and a second performance indicator of the current emotion classifier under the recognition of the user's emotion is calculated.

[0196] The type of the first performance indicator is the same as the type of the second performance indicator, for example, accuracy, precision, etc.

[0197] The first performance indicator is compared to the second performance indicator.

[0198] If the first performance indicator is better than the second performance indicator, it means that the performance of the updated emotion classifier is better than the performance of the current emotion classifier, and the updated emotion classifier is allowed to replace the current emotion classifier.

[0199] If the first performance indicator is worse than the second performance indicator, it means that the performance of the updated emotion classifier is worse than the performance of the current emotion classifier. This situation may be caused by overfitting of training, decreased versatility, etc., then it is prohibited to replace the current emotion classifier with the updated emotion classifier, that is, ignore the updated emotion classifier and keep the current emotion classifier unchanged.

[0200] Embodiment 4

[0201] See also Figure 8 , showing a schematic diagram of the structure of a cross-subject five-channel emotion modeling device provided by the fourth embodiment of the present invention. Figure 8 As shown, the device comprises:

[0202] The EEG signal acquisition module 801 is used to call the brain-computer interface to collect the original EEG signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel when playing multimedia data to the user;

[0203] An ideal emotion determination module 802, configured to determine an ideal emotion induced by the multimedia data to the user;

[0204] A preprocessing module 803 is used to preprocess the original EEG signal to obtain a target EEG signal;

[0205] An EEG signal segmentation module 804 is used to segment the target EEG signal into a plurality of EEG segment signals;

[0206] The emotion classifier training module 805 is used to train the emotion classifier with the EEG segment signals as samples and the ideal emotions as labels under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel.

[0207] In one embodiment of the present invention, the preprocessing module 803 includes:

[0208] A first filtering module is used to input the original EEG signal into a second-order Butterworth high-pass filter to filter out signals less than a first frequency threshold, so as to obtain a first candidate EEG signal;

[0209] A second filtering module is used to input the first candidate EEG signal into a second-order band-stop Butterworth filter to filter out the signal between the second frequency domain threshold and the third frequency threshold, so as to obtain a second candidate EEG signal;

[0210] A third filtering module is used to input the second candidate EEG signal into a second-order low-pass Butterworth high filter to filter out signals greater than a fourth frequency threshold to obtain a target EEG signal;

[0211] The first frequency threshold is smaller than the second frequency domain threshold, the second frequency domain threshold is smaller than the third frequency threshold, and the third frequency threshold is smaller than the fourth frequency threshold.

[0212] In one embodiment of the present invention, the EEG signal segmentation module 804 includes:

[0213] A window adding module, used for adding a window to the target EEG signal;

[0214] A window sliding module, used for extracting the signal in the window as an EEG segment signal each time the window is slid according to a preset step length;

[0215] The width of the window is greater than the step size.

[0216] In one embodiment of the present invention, the emotion classifier training module 805 includes:

[0217] An emotion classifier loading module, used to load an emotion classifier; the emotion classifier includes a channel attention module and an emotion classification head;

[0218] An EEG emotion feature generation module is used to input the EEG segment signal into the channel attention module, add the attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, and obtain the EEG emotion feature;

[0219] An expected emotion recognition module, used for inputting the EEG emotion feature into the emotion classification head to identify the expected emotion generated by the user;

[0220] An emotion classifier updating module, used for updating the emotion classifier according to a loss value between the ideal emotion and the expected emotion;

[0221] A training condition judgment module is used to judge whether the preset training conditions are met if the update is completed; if so, the training completion determination module is executed; if not, the EEG emotion feature generation module is returned to execute;

[0222] The completion determination module is used to determine whether the emotion classifier has completed training.

[0223] In one embodiment of the present invention, the channel attention module includes two fully connected layers;

[0224] The EEG emotion feature generation module includes:

[0225] A pooling module, used for performing an average pooling operation on the EEG segment signal to obtain a first candidate EEG feature;

[0226] A first fully connected module, used for inputting the first candidate EEG feature into the first fully connected layer and mapping it into a second channel feature;

[0227] A first activation module, used to activate the second channel feature using a hyperbolic tangent function Tanh to obtain a third channel feature;

[0228] A second fully connected module, used for inputting the third channel feature into the second fully connected layer and mapping it into a fourth channel feature;

[0229] A second activation module is used to activate the fourth channel feature using a flexible maximum transfer function Softmax to obtain attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0230] A fusion module is used to fuse the EEG segment signal with the attention weight into an EEG emotion feature.

[0231] The cross-subject five-channel emotion modeling device provided in the embodiment of the present invention can execute the cross-subject five-channel emotion modeling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the cross-subject five-channel emotion modeling method.

[0232] Embodiment 5

[0233] See also Fig. 9 , showing a schematic structural diagram of an emotion regulation device provided in Embodiment 5 of the present invention.

[0234] like Fig. 9 As shown, the device comprises:

[0235] The EEG signal acquisition module 901 is used to call the brain-computer interface to collect the original EEG signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0236] A preprocessing module 902 is used to preprocess the original EEG signal to obtain a target EEG signal;

[0237] An EEG signal segmentation module 903 is used to segment the target EEG signal into a plurality of EEG segment signals;

[0238] An emotion classifier loading module 904 is used to load the emotion classifier trained according to the method described in Embodiment 1;

[0239] The expected emotion generation module 905 is used to input the EEG segment signal into the emotion classifier, and identify the expected emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0240] The business operation execution module 906 is used to execute business operations according to the expected emotion to guide the user to adjust his emotion.

[0241] In one embodiment of the present invention, it also includes:

[0242] an actual emotion determination module, for determining the actual emotion induced by the user in a current time period;

[0243] An emotion classifier updating module, used to update the emotion classifier in the current time period, with the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, using the EEG segment signal as a sample and the actual emotion as a label;

[0244] The emotion classifier replacement module is used to replace the current emotion classifier with the updated emotion classifier in the next time period.

[0245] In one embodiment of the present invention, the emotion classifier replacement module includes:

[0246] a performance index calculation module, configured to calculate, in a next time period, a first performance index for the updated emotion classifier under the condition of recognizing the emotion of the user, and a second performance index for the current emotion classifier under the condition of recognizing the emotion of the user;

[0247] A module for allowing a replacement of the emotion classifier, configured to allow the updated emotion classifier to replace the current emotion classifier if the first performance indicator is better than the second performance indicator;

[0248] The emotion classifier replacement prohibition module is used to prohibit replacing the current emotion classifier with the updated emotion classifier if the first performance indicator is inferior to the second performance indicator.

[0249] In one embodiment of the present invention, the preprocessing module 902 includes:

[0250] A first filtering module is used to input the original EEG signal into a second-order Butterworth high-pass filter to filter out signals less than a first frequency threshold, so as to obtain a first candidate EEG signal;

[0251] A second filtering module is used to input the first candidate EEG signal into a second-order band-stop Butterworth filter to filter out the signal between the second frequency domain threshold and the third frequency threshold, so as to obtain a second candidate EEG signal;

[0252] A third filtering module is used to input the second candidate EEG signal into a second-order low-pass Butterworth high filter to filter out signals greater than a fourth frequency threshold to obtain a target EEG signal;

[0253] The first frequency threshold is smaller than the second frequency domain threshold, the second frequency domain threshold is smaller than the third frequency threshold, and the third frequency threshold is smaller than the fourth frequency threshold.

[0254] In one embodiment of the present invention, the business operation execution module 906 includes:

[0255] An ideal emotion determination module, used to determine the ideal emotion induced by the multimedia data to the user;

[0256] an induced accuracy calculation module, used to calculate the ratio between the number of co-occurrences and the total number of the ideal emotions, to obtain the accuracy of the user's induced emotions; the number of co-occurrences is the number when the expected emotion is the same as the ideal emotion;

[0257] The adjustment operation execution module is used to execute the business operation according to the accuracy to guide the user to adjust emotions.

[0258] In one embodiment of the present invention, the EEG signal segmentation module 903 includes:

[0259] A window adding module, used for adding a window to the target EEG signal;

[0260] A window sliding module, used for extracting the signal in the window as an EEG segment signal each time the window is slid according to a preset step length;

[0261] The width of the window is greater than the step size.

[0262] In one embodiment of the present invention, the emotion classifier includes a channel attention module and an emotion classification head; the expected emotion generation module 905 includes:

[0263] An EEG emotion feature generation module is used to input the EEG segment signal into the channel attention module, add the attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, and obtain the EEG emotion feature;

[0264] The expected emotion recognition module is used to input the EEG emotion feature into the emotion classification head to identify the expected emotion generated by the user.

[0265] In one embodiment of the present invention, the channel attention module includes two fully connected layers;

[0266] The EEG emotion feature generation module includes:

[0267] A pooling module, used for performing an average pooling operation on the EEG segment signal to obtain a first candidate EEG feature;

[0268] A first fully connected module, used for inputting the first candidate EEG feature into the first fully connected layer and mapping it into a second channel feature;

[0269] A first activation module, used to activate the second channel feature using a hyperbolic tangent function Tanh to obtain a third channel feature;

[0270] A second fully connected module, used for inputting the third channel feature into the second fully connected layer and mapping it into a fourth channel feature;

[0271] A second activation module is used to activate the fourth channel feature using a flexible maximum transfer function Softmax to obtain attention weights under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel;

[0272] A fusion module is used to fuse the EEG segment signal with the attention weight into an EEG emotion feature.

[0273] The emotion regulation device provided in the embodiment of the present invention can execute the emotion regulation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the emotion regulation method.

[0274] Embodiment 6

[0275] See also Fig.10 , showing a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0276] like Fig.10 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0277] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0278] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the cross-subject five-channel emotion modeling method or the emotion regulation method.

[0279] In some embodiments, the cross-subject five-channel emotion modeling method or emotion regulation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the cross-subject five-channel emotion modeling method or emotion regulation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the cross-subject five-channel emotion modeling method or emotion regulation method in any other appropriate manner (e.g., by means of firmware).

[0280] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0281] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0282] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0283] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0284] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0285] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0286] Embodiment 7

[0287] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the cross-subject five-channel emotion modeling method or emotion regulation method provided in any embodiment of the present invention.

[0288] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0289] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0290] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A five-channel emotion modeling method across subjects, characterized in that: include: When multimedia data is played to a user, the brain-computer interface is called to collect original EEG signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel; determining an ideal emotion induced by the multimedia data on the user; Preprocessing the original EEG signal to obtain a target EEG signal; Dividing the target EEG signal into a plurality of EEG segment signals; Under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, an emotion classifier is trained with the EEG segment signal as a sample and the ideal emotion as a label.

2. The method according to claim 1, characterized in that The preprocessing of the original EEG signal to obtain a target EEG signal includes: Inputting the original EEG signal into a second-order Butterworth high-pass filter to filter out signals less than a first frequency threshold, thereby obtaining a first candidate EEG signal; Inputting the first candidate EEG signal into a second-order band-stop Butterworth filter to filter out signals between a second frequency domain threshold and a third frequency threshold, thereby obtaining a second candidate EEG signal; Inputting the second candidate EEG signal into a second-order low-pass Butterworth high filter to filter out signals greater than a fourth frequency threshold, thereby obtaining a target EEG signal; The first frequency threshold is smaller than the second frequency domain threshold, the second frequency domain threshold is smaller than the third frequency threshold, and the third frequency threshold is smaller than the fourth frequency threshold.

3. The method according to claim 1, characterized in that The step of dividing the target EEG signal into a plurality of EEG segment signals comprises: adding a window to the target EEG signal; Each time the window is slid according to a preset step length, a signal located in the window is extracted as an EEG segment signal; The width of the window is greater than the step size.

4. The method according to any one of claims 1 to 3, characterized in that The method of training an emotion classifier with the EEG segment signal as a sample and the ideal emotion as a label under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel includes: Loading a sentiment classifier; the sentiment classifier includes a channel attention module and a sentiment classification head; Inputting the EEG segment signal into the channel attention module, adding the attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, and obtaining the EEG emotion feature; Inputting the EEG emotion feature into the emotion classification head to identify the desired emotion generated by the user; updating the emotion classifier according to the loss value between the ideal emotion and the expected emotion; If the update is completed, determine whether the preset training conditions are met; if so, determine that the emotion classifier has completed training; if not, return to execute the step of inputting the EEG segment signal into the channel attention module to add the attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel to obtain the EEG emotion features.

5. The method according to claim 4, characterized in that The channel attention module includes two fully connected layers; The step of inputting the EEG segment signal into the channel attention module and adding the attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel to obtain the EEG emotion feature includes: Performing an average pooling operation on the EEG segment signal to obtain a first candidate EEG feature; Inputting the first candidate EEG feature into the first fully connected layer and mapping it into a second channel feature; Using the hyperbolic tangent function Tanh to activate the second channel feature to obtain the third channel feature; Inputting the third channel feature into the second fully connected layer and mapping it into a fourth channel feature; The fourth channel feature is activated using a flexible maximum transfer function Softmax to obtain attention weights of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel; The EEG segment signal and the attention weight are fused into an EEG emotion feature.

6. A method of emotion regulation, characterized in that: include: When multimedia data is played to a user, the brain-computer interface is called to collect original EEG signals of the user under the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel; Preprocessing the original EEG signal to obtain a target EEG signal; Dividing the target EEG signal into a plurality of EEG segment signals; Loading an emotion classifier trained according to any one of the methods of claims 1-5; Inputting the EEG segment signal into the emotion classifier, and identifying the desired emotion generated by the user under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel; A business operation is performed according to the expected emotion to guide the user to adjust the emotion.

7. The method according to claim 6, characterized in that Also includes: determining an actual emotion induced by the user during a current time period; In the current time period, under the attention of the O1 channel, the O2 channel, the C3 channel, the C4 channel and the FP2 channel, the emotion classifier is updated with the EEG segment signal as a sample and the actual emotion as a label; In the next time period, replacing the current emotion classifier with the updated emotion classifier; or The performing of a business operation according to the expected emotion to guide the user to adjust the emotion includes: determining an ideal emotion induced by the multimedia data on the user; Calculating the ratio between the number of co-occurrences and the total number of the ideal emotions to obtain the accuracy of the user-induced emotions; the number of co-occurrences is the number when the expected emotion is the same as the ideal emotion; A business operation is performed according to the accuracy to guide the user to regulate emotions.

8. The method according to claim 7, characterized in that In the next time period, replacing the current emotion classifier with the updated emotion classifier includes: In the next time period, a first performance indicator of the updated emotion classifier under the condition of recognizing the emotion of the user is calculated, and a second performance indicator of the current emotion classifier under the condition of recognizing the emotion of the user is calculated; If the first performance indicator is better than the second performance indicator, allowing the updated emotion classifier to replace the current emotion classifier; If the first performance indicator is inferior to the second performance indicator, it is prohibited to replace the current emotion classifier with the updated emotion classifier.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cross-subject five-channel emotion modeling method as described in any one of claims 1-5, or the emotion regulation method as described in any one of claims 6-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-subject five-channel emotion modeling method according to any one of claims 1 to 5, or the emotion regulation method according to any one of claims 6 to 8 is implemented.

Images