Emotion Monitoring System and Method Based on Deep Learning and Brain Wave Morphological Features

Abstract

The present invention discloses an emotion monitoring system and method based on deep learning and electroencephalogram morphological features, which relates to the technical field of emotion state monitoring. By pre-collecting electroencephalogram training data, emotion label data and behavior type label data of each tester, an emotion classification model, an emotion recognition model and a behavior prediction model are trained. In a real monitoring environment, electroencephalogram monitoring data and behavior monitoring data of the person to be monitored are collected, the predicted emotion category and predicted emotion state of the person to be monitored are obtained, and the predicted behavior of the person to be monitored is output, and each behavior prediction model is fine-tuned; each behavior prediction model is repeatedly fine-tuned until a suitable behavior prediction model is selected; prevention and prevention can be carried out before it is judged that the person to be monitored has an excessive behavior, thereby protecting the safety of the person to be monitored and the surrounding environment.

Description

Technical Field

[0001] The present invention relates to the technical field of emotional state monitoring, and particularly to an emotional monitoring system and method based on deep learning and electroencephalogram morphological features. Background Art

[0002] Nowadays, intelligent systems and human-computer interaction technologies are becoming increasingly popular. However, existing behavior prediction schemes usually ignore a key factor - the emotional state of the user. The emotional state is a reflection of a person's internal mental state and directly affects their decision-making, behavior, and experience. Therefore, in order to more accurately understand and predict user behavior, it is necessary to monitor the user's emotional state in real time;

[0003] Chinese Patent No. CN108073284A discloses a shopping system for identifying emotions based on electroencephalograms, including an electroencephalogram acquisition module and its shopping recommendation system. The electroencephalogram sensor is worn on the user's head, and the shopping recommendation system is deployed in the background of the e-commerce website. The electroencephalogram sensor is used to detect the user's electroencephalogram, analyze and process it, and identify emotions. When a specific emotion is identified, the shopping recommendation system obtains relevant data and accurately recommends products to the user; however, this aspect only identifies the user's emotions and fails to predict the user's behavior;

[0004] Therefore, the present invention proposes an emotional monitoring system and method based on deep learning and electroencephalogram morphological features. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. For this purpose, the present invention proposes an emotional monitoring system and method based on deep learning and electroencephalogram morphological features, which can prevent and stop before it is determined that the person to be monitored shows excessive behavior, thereby protecting the safety of the person to be monitored and the surrounding environment.

[0006] To achieve the above object, an emotional monitoring method based on deep learning and electroencephalogram morphological features is proposed, including the following steps:

[0007] Step 1: Pre-collect electroencephalogram training data, emotion label data, and behavior type label data of each test person in an experimental environment;

[0008] Step 2: Use the electroencephalogram training data as input and the emotion label data as output to train an emotion classification model that outputs predicted emotion categories; and train an emotion recognition model that outputs predicted emotional states based on the electroencephalogram training data; for each test person, use the emotion label data and emotional state as input and the behavior type label data as output to train a behavior prediction model;

[0009] Step 3: In a real monitoring environment, collect electroencephalogram monitoring data and behavior monitoring data of the person to be monitored;

[0010] Step 4: Based on the electroencephalogram monitoring data and the emotion classification model, obtain the predicted emotion category of the person to be monitored; based on the electroencephalogram monitoring data and the emotion recognition model, obtain the predicted emotion state of the person to be monitored;

[0011] Step 5: Based on the predicted emotion category of the person to be monitored, the predicted emotion state, and the behavior prediction model of the tester, output the predicted behavior of the person to be monitored, and fine-tune each behavior prediction model;

[0012] Step 6: Repeat Steps 3 to 5 until a suitable behavior prediction model is selected from all the behavior prediction models;

[0013] The method for pre-collecting the electroencephalogram training data, emotion label data, and behavior type label data of each tester in the experimental environment is as follows:

[0014] During each use of the test means on the tester, the real-time electroencephalogram signals of each waveband of the tester are collected in real time, and the electroencephalogram signals of each waveband are combined into an electroencephalogram signal sequence corresponding to the waveband in chronological order. The electroencephalogram signal sequences of each waveband form an electroencephalogram training feature set; all the electroencephalogram training feature sets of each tester form the electroencephalogram training data of the tester;

[0015] After using the test means on the tester, collect the emotion category labels of the tester through the way of return visit; all the emotion category labels of each tester form the emotion label data of the tester;

[0016] After using the test means on the tester, record the behavior type labels of each tester by manual recording. All the behavior type labels of each tester form the behavior type label data of the tester;

[0017] The method for training the emotion classification model that outputs the predicted emotion category is as follows:

[0018] Use the electroencephalogram training feature set in the electroencephalogram training data as the input of the emotion classification model. The emotion classification model takes the predicted value of the emotion category corresponding to the electroencephalogram training feature set as the output, takes the emotion category label of the tester corresponding to the electroencephalogram training feature set as the prediction target, takes the difference between the predicted value of the emotion category and the emotion category label as the first prediction error, and takes minimizing the sum of the squares of the first prediction error as the training target; train the emotion classification model until the sum of the squares of the first prediction error reaches convergence and then stop training;

[0019] The emotion recognition model that outputs the predicted emotion state includes the following steps:

[0020] Step 11: Preprocess each set of EEG training features in the EEG training data to obtain the corresponding preprocessed EEG training set;

[0021] Step 12: Input each preprocessed EEG training set into the emotion recognition model to obtain the corresponding emotion feature expression set for each preprocessed EEG training set, and output each said emotion feature expression set; the emotion recognition model is an unsupervised learning model;

[0022] The unsupervised learning model is an autoencoder;

[0023] The method of training the behavior prediction model is as follows:

[0024] For each tester:

[0025] Collect the emotion feature expression sets corresponding to each set of EEG training features of the tester;

[0026] Combine the emotion label data and the emotion feature expression set of the tester after using the test means to form a set of behavior prediction feature sets;

[0027] Use each set of behavior prediction feature sets as the input of the behavior prediction model. The behavior prediction model takes the predicted value of the behavior type corresponding to the behavior prediction feature set as the output, takes the behavior type label of the tester corresponding to the behavior prediction feature set as the prediction target, takes the difference between the predicted value of the behavior type and the behavior type label as the second prediction error, and takes minimizing the sum of the squares of the second prediction error as the training target; train the behavior prediction model until the sum of the squares of the second prediction error converges and then stop training;

[0028] The method of collecting the EEG monitoring data and behavior monitoring data of the person to be monitored in the real monitoring environment is as follows:

[0029] The real monitoring environment is an environment where the person to be monitored wears an EEG signal acquisition device and the EEG signal of the person to be monitored is collected in real time through the EEG signal acquisition device;

[0030] During the process of the person to be monitored wearing the EEG signal acquisition device, collect the EEG signals of each band of the person to be monitored in real time, and form the corresponding EEG monitoring signal sequence for each band of EEG signals in chronological order; the length of the EEG monitoring signal sequence is the same as the length of the EEG signal sequence of the tester; the EEG monitoring signal sequence constitutes the EEG monitoring data of the person to be monitored;

[0031] Collect the behavior type of the person to be monitored after collecting the EEG monitoring signal sequence in real time as the behavior monitoring data;

[0032] The way to output the predicted behavior of the person to be monitored is as follows:

[0033] Combine the predicted emotion category and predicted emotion state of the person to be monitored to form a set of monitoring behavior prediction feature sets;

[0034] Input the monitoring behavior prediction feature sets into each behavior prediction model to obtain the predicted values of the behavior types of the person to be monitored output by each behavior prediction model;

[0035] Use the voting method to select the predicted value of the behavior type with the most occurrences from all the predicted values of the behavior types of the person to be monitored as the predicted behavior of the person to be monitored;

[0036] The way to fine-tune each behavior prediction model is as follows:

[0037] For each behavior prediction model:

[0038] Use the monitoring behavior prediction feature sets of the person to be monitored as the input of this behavior prediction model, and use the difference between the predicted value of the behavior type output by this behavior prediction model and the behavior monitoring data corresponding to the behavior prediction feature sets as the third prediction error, and use the sum of the squares of the third prediction errors as the loss function to retrain this behavior prediction model to obtain the fine-tuned behavior prediction model;

[0039] The way to determine whether a suitable behavior prediction model is selected is as follows:

[0040] Select the behavior prediction model with the highest prediction accuracy from all the behavior prediction models as the quasi-suitable behavior prediction model. If the prediction accuracy of the quasi-suitable behavior prediction model reaches the preset accuracy threshold, then use the quasi-suitable behavior prediction model as the suitable behavior prediction model; otherwise, it is considered that no suitable behavior prediction model is selected.

[0041] Propose an emotion monitoring system based on deep learning and brain wave morphological features, including a training data collection module, a model training module, and a behavior prediction module; among them, each module is connected electrically;

[0042] The training data collection module is used to pre-collect the brain wave training data, emotion label data, and behavior type label data of each test person in the experimental environment, and send the brain wave training data, emotion label data, and behavior type label data to the model training module;

[0043] A model training module, which takes electroencephalogram training data as input and emotion label data as output, trains an emotion classification model that outputs predicted emotion categories; and trains an emotion recognition model that outputs predicted emotion states based on the electroencephalogram training data; for each tester, taking emotion label data and emotion states as input and behavior type label data as output, trains a behavior prediction model, and sends the emotion classification model, emotion recognition model and behavior prediction model to the behavior prediction module;

[0044] The behavior prediction module is used to perform the following steps:

[0045] Step 21: In a real monitoring environment, collect the electroencephalogram monitoring data and behavior monitoring data of the person to be monitored;

[0046] Step 22: Based on the electroencephalogram monitoring data and the emotion classification model, obtain the predicted emotion category of the person to be monitored; based on the electroencephalogram monitoring data and the emotion recognition model, obtain the predicted emotion state of the person to be monitored;

[0047] Step 23: Based on the predicted emotion category and predicted emotion state of the person to be monitored and the behavior prediction model of the tester, output the predicted behavior of the person to be monitored, and fine-tune each behavior prediction model;

[0048] Step 24: Repeat steps 21 to 23 until a suitable behavior prediction model is selected from all the behavior prediction models.

[0049] An electronic device is proposed, including: a processor and a memory, wherein, a computer program that can be called by the processor is stored in the memory;

[0050] The processor executes the above-mentioned emotion monitoring method based on deep learning and electroencephalogram morphological features by calling the computer program stored in the memory.

[0051] A computer-readable storage medium is proposed, on which a rewritable computer program is stored;

[0052] When the computer program runs on a computer device, the computer device executes the above-mentioned emotion monitoring method based on deep learning and electroencephalogram morphological features.

[0053] Compared with the prior art, the beneficial effects of the present invention are:

[0054] The present invention pre-collects electroencephalogram training data, emotion label data, and behavior type label data of each tester in an experimental environment in advance. Using the electroencephalogram training data as input and the emotion label data as output, an emotion classification model that outputs predicted emotion categories is trained; and an emotion recognition model that outputs predicted emotion states is trained based on the electroencephalogram training data; for each tester, using the emotion label data and emotion states as input and the behavior type label data as output, a behavior prediction model is trained. In a real monitoring environment, electroencephalogram monitoring data and behavior monitoring data of the person to be monitored are collected. Based on the electroencephalogram monitoring data and the emotion classification model, the predicted emotion category of the person to be monitored is obtained; based on the electroencephalogram monitoring data and the emotion recognition model, the predicted emotion state of the person to be monitored is obtained. Based on the predicted emotion category, predicted emotion state of the person to be monitored, and the behavior prediction model of the tester, the predicted behavior of the person to be monitored is output, and each behavior prediction model is fine-tuned. Repeat the fine-tuning of each behavior prediction model until a suitable behavior prediction model is selected from all the behavior prediction models; by training the emotion classification model and the emotion recognition model, the emotion type of the person to be monitored and the emotional characteristic expression of this emotion type are predicted, and then according to the emotion type and emotional characteristic expression, the behavior prediction model is trained, so that the real-time monitoring of the emotion state of the person to be monitored can be realized, and the future behavior can be predicted, and prevention and prevention can be carried out before it is judged that the person to be monitored shows extreme behavior, so as to protect the safety of the person to be monitored and the surrounding environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of the emotion monitoring method based on deep learning and electroencephalogram morphological features in Embodiment 1 of the present invention;

[0056] Figure 2 is a module connection diagram of the emotion monitoring system based on deep learning and electroencephalogram morphological features in Embodiment 2 of the present invention;

[0057] Figure 3 is a schematic structural diagram of an electronic device in Embodiment 3 of the present invention;

[0058] Figure 4 is a schematic structural diagram of a computer-readable storage medium in Embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0060] Embodiment 1

[0061] As Figure 1 shown, an emotion monitoring method based on deep learning and electroencephalogram morphological features includes the following steps:

[0062] Step 1: Pre-collect electroencephalogram training data, emotion label data, and behavior type label data of each tester in the experimental environment;

[0063] Step 2: Use the electroencephalogram training data as input and the emotion label data as output to train an emotion classification model that outputs predicted emotion categories; and train an emotion recognition model that outputs predicted emotion states based on the electroencephalogram training data; for each tester, use the emotion label data and emotion states as input and the behavior type label data as output to train a behavior prediction model;

[0064] Step 3: In the real monitoring environment, collect electroencephalogram monitoring data and behavior monitoring data of the person to be monitored;

[0065] Step 4: Based on the electroencephalogram monitoring data and the emotion classification model, obtain the predicted emotion category of the person to be monitored; based on the electroencephalogram monitoring data and the emotion recognition model, obtain the predicted emotion state of the person to be monitored;

[0066] Step 5: Based on the predicted emotion category, predicted emotion state of the person to be monitored, and the behavior prediction model of the tester, output the predicted behavior of the person to be monitored, and fine-tune each behavior prediction model;

[0067] Step 6: Repeat Steps 3 to 5 until a suitable behavior prediction model is selected from all the behavior prediction models;

[0068] Among them, in the experimental environment, N testers are pre-recruited, several different testing methods are adopted for the testers, and the electroencephalogram signals (EEG signals) of the testers are collected by using an electroencephalogram signal acquisition device; N is the number of pre-selected testers; the testing methods include but are not limited to watching different types of videos, listening to different types of music, reading different types of books, etc.; the electroencephalogram signal acquisition device includes but is not limited to an electroencephalogram amplifier, etc.;

[0069] Among them, the method of pre-collecting electroencephalogram training data, emotion label data, and behavior type label data of each tester in the experimental environment is:

[0070] During each use of the testing method on the testers, the real-time electroencephalogram (EEG) signals of the testers in each frequency band are collected in real time, and the EEG signals in each frequency band are combined into an EEG signal sequence corresponding to the frequency band in chronological order. The EEG signal sequences in each frequency band form an EEG training feature set; all the EEG training feature sets of each tester form the EEG training data of the tester;

[0071] It should be noted that an example of an EEG training feature set is as follows:

[0072] The real-time EEG data collected is shown in Table 1:

[0073] Table 1 EEG Data Table

[0074] Time (seconds) Delta wave (μV) Theta wave (μV) Alpha wave (μV) Beta wave (μV) Gamma wave (μV) 1 9.8 8.3 11.9 8.2 5.5 2 9.6 8.5 11.7 8.0 5.7 3 9.4 8.8 11.5 7.8 5.9 4 9.2 9.0 11.3 7.6 6.1 5 9.0 9.2 11.1 7.4 6.3

[0075] Then the EEG training feature set is

[0076] [[9.8, 9.6, 9.4, 9.2, 9.0], [8.3, 8.5, 8.8, 9.0, 9.2], [11.9, 11.7, 11.5, 11.3, 11.1], [8.2, 8.0, 7.8, 7.6, 7.4], [5.5, 5.7, 5.9, 6.1, 6.3]], where each sequence corresponds to the EEG signal frequency values of the δ wave (0.5 - 4 Hz), θ wave (4 - 8 Hz), α wave (8 - 13 Hz), β wave (13 - 30 Hz), and γ wave (30 - 100 Hz);

[0077] After using the testing method on the testers, the emotional category labels of the testers are collected through a follow-up visit; all the emotional category labels of each tester form the emotional label data of the tester; the emotional category labels include but are not limited to positive, negative, neutral, excited, depressed, anxious, angry, and fearful, etc.;

[0078] After using the testing method on the testers, the behavior type labels of each tester are recorded through manual recording. All the behavior type labels of each tester form the behavior type label data of the tester; the behavior types include but are not limited to relaxation behaviors: such as resting, closing eyes, taking deep breaths, etc.; positive behaviors: such as smiling, laughing, taking the initiative to communicate, etc.; negative behaviors: such as withdrawing, being blocked, thinking alone, etc.; impulsive behaviors: such as being excited, dancing, engaging in exciting activities, etc.; angry behaviors: such as throwing things, roaring, having intense behaviors, etc.;

[0079] Furthermore, the method of training an emotion classification model that outputs predicted emotion categories with the EEG training data as the input and the emotional label data as the output is:

[0080] Use the brain wave training feature set in the brain wave training data as the input of the emotion classification model. The emotion classification model takes the predicted value of the emotion category corresponding to the brain wave training feature set as the output, uses the emotion category label of the tester corresponding to the brain wave training feature set as the prediction target, takes the difference between the predicted value of the emotion category and the emotion category label as the first prediction error, and uses minimizing the sum of the squares of the first prediction error as the training target; Train the emotion classification model until the sum of the squares of the first prediction error converges and then stop training, and train an emotion classification model that outputs the predicted value of the emotion category according to the brain wave training data; The emotion classification model is any one of the time series prediction models, such as an RNN model or an LSTM model; The sum of the squares of the prediction error is the mean square error.

[0081] Further, the emotion recognition model trained based on the brain wave training data to output the predicted emotion state includes the following steps:

[0082] Step 11: Preprocess each brain wave training feature set in the brain wave training data to obtain the corresponding brain wave training preprocessing set; Specifically, the preprocessing process includes but is not limited to operations such as denoising, filtering, and dimensionality reduction to ensure the quality of the data.

[0083] Step 12: Input each brain wave training preprocessing set into the emotion recognition model to obtain the emotion feature expression set corresponding to each brain wave training preprocessing set, and output each said emotion feature expression set; The emotion recognition model is an unsupervised learning model.

[0084] In a preferred embodiment, the unsupervised learning model is an autoencoder; It should be noted that the autoencoder is an unsupervised learning model, which includes an encoder and a decoder. The encoder maps the input data to the latent space (usually a low-dimensional representation), and the decoder maps the representation in the latent space back to the original data space; In the emotion recognition model training task of this embodiment, the autoencoder can learn the compact representation of each brain wave training feature set, and the brain wave training feature set contains information related to the emotion state.

[0085] An example of the generation of a brain wave training feature set is as follows:

[0086] Input:

[0087] Brain wave training preprocessing set: Each sample is a time series representing the brain wave records under different emotion states. For example, each sample contains the brain wave signals within a period of time, which may include dozens to hundreds of data points.

[0088] Sample 1: [0.2, 0.5, 0.3,..., 0.8] (time series data)

[0089] Sample 2: [0.1, 0.3, 0.4, …, 0.6]

[0090] Output:

[0091] Representation of emotional state: The representation of the emotional state learned by the unsupervised learning model, where the representation of the emotional state is a low-dimensional representation of the preprocessed set of electroencephalogram training data or an encoding in the latent space; the representation of the emotional state captures the emotion-related features in the electroencephalogram data, such as the intensity and pleasantness of emotions;

[0092] Emotional representation of Sample 1: [0.1, -0.3, 0.2] (low-dimensional representation);

[0093] Emotional representation of Sample 2: [0.3, -0.1, -0.5];

[0094] Furthermore, for each tester, the way to train the behavior prediction model with emotional label data and emotional state as input and behavior type label data as output is as follows:

[0095] For each tester:

[0096] Collect the set of emotional feature expressions corresponding to each set of electroencephalogram training features of this tester;

[0097] Combine the emotional label data of this tester after using the test means and the set of emotional feature expressions into a set of behavior prediction features;

[0098] Use each set of behavior prediction features as the input of the behavior prediction model. The behavior prediction model takes the predicted value of the behavior type corresponding to the set of behavior prediction features as the output, takes the behavior type label of this tester corresponding to the set of behavior prediction features as the prediction target, takes the difference between the predicted value of the behavior type and the behavior type label as the second prediction error, and takes minimizing the sum of the squares of the second prediction error as the training objective; train the behavior prediction model until the sum of the squares of the second prediction error converges and then stop training, and train a behavior prediction model that outputs the behavior type that the monitored person may implement due to the emotional state according to the set of behavior prediction features; the behavior prediction model is any one of the classification models, and the classification models include but are not limited to DNN models, SVM models, etc.; the sum of the squares of the prediction error is the mean square error;

[0099] Furthermore, the way to collect the electroencephalogram monitoring data and behavior monitoring data of the person to be monitored in the real monitoring environment is as follows:

[0100] The real monitoring environment is an environment where the person to be monitored wears an electroencephalogram signal acquisition device and the electroencephalogram signal of the person to be monitored is collected in real time through the electroencephalogram signal acquisition device;

[0101] During the process of the person to be monitored wearing the electroencephalogram signal acquisition device, the electroencephalogram signals of each band of the person to be monitored are collected in real time, and the electroencephalogram signals of each band are combined into a corresponding electroencephalogram monitoring signal sequence in chronological order; the length of the electroencephalogram monitoring signal sequence is the same as the length of the electroencephalogram signal sequence of the tester; the electroencephalogram monitoring signal sequence constitutes the electroencephalogram monitoring data of the person to be monitored;

[0102] The behavior type of the person to be monitored after collecting the electroencephalogram monitoring signal sequence is collected in real time as behavior monitoring data; specifically, the behavior monitoring data can be automatically recognized by using an action recognition model, and the action recognition model includes but is not limited to a two-stream CNN model, a TSN model, etc.;

[0103] Further, the method for obtaining the predicted emotion category of the person to be monitored based on the electroencephalogram monitoring data and the emotion classification model is as follows:

[0104] Input the electroencephalogram monitoring data into the emotion classification model to obtain the predicted value of the emotion category output by the emotion classification model as the predicted emotion category;

[0105] The method for obtaining the predicted emotion state of the person to be monitored based on the electroencephalogram monitoring data and the emotion recognition model is as follows:

[0106] Preprocess the electroencephalogram monitoring data and then input it into the emotion recognition model to obtain the emotion feature expression set of the person to be monitored output by the emotion recognition model, and use the emotion feature expression set of the person to be monitored as the predicted emotion state;

[0107] Further, the method for outputting the predicted behavior of the person to be monitored based on the predicted emotion category, predicted emotion state of the person to be monitored and the behavior prediction model of the tester is as follows:

[0108] Combine the predicted emotion category and predicted emotion state of the person to be monitored into a set of monitoring behavior prediction feature sets;

[0109] Input the set of monitoring behavior prediction features into each behavior prediction model to obtain the predicted values of the behavior types of the person to be monitored output by each behavior prediction model;

[0110] Use the voting method to select the predicted value of the behavior type that appears most frequently from all the predicted values of the behavior types of the person to be monitored as the predicted behavior of the person to be monitored;

[0111] It can be understood that the predicted behavior of the person to be monitored is the behavior that the person to be monitored may make in the future, which can be prevented and stopped before it is judged that the person to be monitored shows extreme behavior, so as to protect the safety of the person to be monitored and the surrounding environment;

[0112] Furthermore, due to the different emotional states of each person, when the person to be monitored first uses the behavior prediction model, the results output by different behavior prediction models may be different. Therefore, it is also necessary to fine-tune the behavior prediction model according to the actual behavior of the person to be monitored;

[0113] Furthermore, the method for fine-tuning each behavior prediction model is as follows:

[0114] For each behavior prediction model:

[0115] Use the monitoring behavior prediction feature set of the person to be monitored as the input of this behavior prediction model, and use the difference between the predicted value of the behavior type output by this behavior prediction model and the behavior monitoring data corresponding to the behavior prediction feature set as the third prediction error. Use the sum of the squares of the third prediction errors as the loss function, and retrain this behavior prediction model to obtain a fine-tuned behavior prediction model;

[0116] Furthermore, the method for determining whether a suitable behavior prediction model is selected is as follows:

[0117] Select the behavior prediction model with the highest prediction accuracy from all behavior prediction models as the quasi-suitable behavior prediction model. If the prediction accuracy of the quasi-suitable behavior prediction model reaches the preset accuracy threshold, then use the quasi-suitable behavior prediction model as the suitable behavior prediction model; otherwise, it is considered that no suitable behavior prediction model is selected;

[0118] It can be understood that the suitable behavior prediction model is the behavior prediction model closest to the person to be monitored. After obtaining the suitable behavior prediction model, the suitable behavior prediction model can be directly used to predict the behavior of the person to be monitored, without using all behavior prediction models to predict the person to be monitored.

[0119] Embodiment 2

[0120] As Figure 2 shown, the emotion monitoring system based on deep learning and electroencephalogram morphology features includes a training data collection module, a model training module, and a behavior prediction module; wherein, each module is connected electrically;

[0121] Among them, the training data collection module is mainly used to pre-collect the electroencephalogram training data, emotion label data, and behavior type label data of each test person in the experimental environment, and send the electroencephalogram training data, emotion label data, and behavior type label data to the model training module;

[0122] Among them, the model training module is mainly used to take the electroencephalogram training data as the input and the emotion label data as the output to train an emotion classification model that outputs the predicted emotion category; and train an emotion recognition model that outputs the predicted emotion state based on the electroencephalogram training data; for each tester, take the emotion label data and the emotion state as the input and the behavior type label data as the output to train a behavior prediction model, and send the emotion classification model, the emotion recognition model, and the behavior prediction model to the behavior prediction module;

[0123] Among them, the behavior prediction module is mainly used to perform the following steps:

[0124] Step 21: In the real monitoring environment, collect the electroencephalogram monitoring data and behavior monitoring data of the person to be monitored;

[0125] Step 22: Based on the electroencephalogram monitoring data and the emotion classification model, obtain the predicted emotion category of the person to be monitored; based on the electroencephalogram monitoring data and the emotion recognition model, obtain the predicted emotion state of the person to be monitored;

[0126] Step 23: Based on the predicted emotion category of the person to be monitored, the predicted emotion state, and the behavior prediction model of the tester, output the predicted behavior of the person to be monitored, and fine-tune each behavior prediction model;

[0127] Step 24: Repeat steps 21 to 23 until a suitable behavior prediction model is selected from all the behavior prediction models.

[0128] Embodiment 3

[0129] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 3 shown, according to another aspect of the present application, an electronic device 100 is further provided. The electronic device 100 may include one or more processors and one or more memories. Among them, computer-readable code is stored in the memory, and when the computer-readable code is run by one or more processors, it can implement the emotion monitoring method based on deep learning and electroencephalogram morphological features as described above.

[0130] The method or device according to the embodiment of the present application can also be implemented by means of Figure 3 the architecture of the electronic device shown. As Figure 3As shown in the figure, the electronic device 100 may include a bus 101, one or more CPUs 102, a ROM 103, a RAM 104, a communication port 105 connected to a network, an input / output component 106, a hard disk 107, etc. The storage device in the electronic device 100, such as the ROM 103 or the hard disk 107, may store the implementation of the emotion monitoring method based on deep learning and electroencephalogram morphological features provided in this application. The implementation of the emotion monitoring method based on deep learning and electroencephalogram morphological features may include the following steps: Step 1: Pre-collect the electroencephalogram training data, emotion label data, and behavior type label data of each tester in an experimental environment; Step 2: Use the electroencephalogram training data as the input and the emotion label data as the output to train an emotion classification model that outputs predicted emotion categories; and train an emotion recognition model that outputs predicted emotion states based on the electroencephalogram training data; for each tester, use the emotion label data and emotion states as the input and the behavior type label data as the output to train a behavior prediction model; Step 3: In a real monitoring environment, collect the electroencephalogram monitoring data and behavior monitoring data of the person to be monitored; Step 4: Based on the electroencephalogram monitoring data and the emotion classification model, obtain the predicted emotion category of the person to be monitored; based on the electroencephalogram monitoring data and the emotion recognition model, obtain the predicted emotion state of the person to be monitored; Step 5: Based on the predicted emotion category, predicted emotion state of the person to be monitored, and the behavior prediction model of the tester, output the predicted behavior of the person to be monitored, and fine-tune each behavior prediction model; Step 6: Repeat Steps 3 to 5 until a suitable behavior prediction model is selected from all the behavior prediction models

[0131] Further, the electronic device 100 may further include a user interface 108. Of course, Figure 3 The architecture shown is only exemplary. When implementing different devices, one or more components in the electronic device shown may be omitted according to actual needs Figure 3 as required

[0132] Embodiment 4

[0133] Figure 4 is a schematic diagram of the structure of a computer-readable storage medium provided by an embodiment of this application. As Figure 4 shown, it is a computer-readable storage medium 200 according to an embodiment of this application. Computer-readable instructions are stored on the computer-readable storage medium 200. When the computer-readable instructions are run by a processor, the implementation of the emotion monitoring method based on deep learning and electroencephalogram morphological features according to the embodiment of this application described with reference to the above drawings can be executed. The computer-readable storage medium 200 includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and cache memory, etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, etc

[0134] In addition, according to an embodiment of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the present application provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be run by a processor to execute instructions corresponding to the method steps provided by the present application. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0135] The method, apparatus, and device of the present application can be implemented in many ways. For example, the method, apparatus, and device of the present application can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present application are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, the present application can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers a recording medium storing a program for executing the method according to the present application.

[0136] In addition, parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0137] As described above in the specific embodiments, the purpose, technical solution, and beneficial effects of the present invention are further described in detail. It should be understood that the above is only the specific embodiment of the present invention and is not used to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this embodiment shall be included in the protection scope of this embodiment.

[0138] The above preset parameters or preset thresholds are all set by those skilled in the art according to the actual situation or obtained by simulating a large amount of data.

[0139] The above embodiments are only used to illustrate the technical method of the present invention and not to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An emotion monitoring method based on deep learning and electroencephalogram morphological features, characterized in that, It includes the following steps: Step 1: Pre-collect the electroencephalogram training data, emotion label data, and behavior type label data of each tester in the experimental environment; Step 2: Use the electroencephalogram training data as the input and the emotion label data as the output to train an emotion classification model that outputs the predicted emotion category; And train an emotion recognition model that outputs the predicted emotion state based on the electroencephalogram training data; for each tester, use the emotion label data and emotion state as the input and the behavior type label data as the output to train a behavior prediction model; Step 3: In the real monitoring environment, collect the electroencephalogram monitoring data and behavior monitoring data of the person to be monitored; Step 4: Based on the electroencephalogram monitoring data and the emotion classification model, obtain the predicted emotion category of the person to be monitored; based on the electroencephalogram monitoring data and the emotion recognition model, obtain the predicted emotion state of the person to be monitored; Step 5: Based on the predicted emotion category, predicted emotion state of the person to be monitored, and the behavior prediction model of the tester, output the predicted behavior of the person to be monitored, and fine-tune each behavior prediction model; Step 6: Repeat steps 3 to 5 until a suitable behavior prediction model is selected from all the behavior prediction models; The method of fine-tuning each behavior prediction model is as follows: For each behavior prediction model: Use the monitoring behavior prediction feature set of the person to be monitored as the input of this behavior prediction model, and use the difference between the predicted value of the behavior type output by this behavior prediction model and the behavior monitoring data corresponding to the behavior prediction feature set as the third prediction error. Use the sum of the squares of the third prediction error as the loss function, and retrain this behavior prediction model to obtain the fine-tuned behavior prediction model; The method of determining whether a suitable behavior prediction model is selected is as follows: Select the behavior prediction model with the highest prediction accuracy from all the behavior prediction models as the quasi-suitable behavior prediction model. If the prediction accuracy of the quasi-suitable behavior prediction model reaches the preset accuracy threshold, then use the quasi-suitable behavior prediction model as the suitable behavior prediction model; otherwise, it is considered that no suitable behavior prediction model is selected.

2. The emotion monitoring method based on deep learning and electroencephalogram morphological features according to claim 1, characterized in that The method of pre-collecting the electroencephalogram training data, emotion label data, and behavior type label data of each tester in the experimental environment is as follows: During each use of the test means on the tester, real-time collect the real-time electroencephalogram signals of each band of the tester, and form the electroencephalogram signal sequence of the corresponding band in chronological order. The electroencephalogram signal sequences of each band form the electroencephalogram training feature set; all the electroencephalogram training feature sets of each tester form the electroencephalogram training data of this tester; After using the test means on the tester, collect the emotion category labels of the tester through the return visit method; all the emotion category labels of each tester form the emotion label data of this tester; After using the test means on the tester, record the behavior type label of each tester by manual recording. All the behavior type labels of each tester form the behavior type label data of this tester.

3. The emotion monitoring method based on deep learning and electroencephalogram morphological features according to claim 2, characterized in that, The method for training the emotion classification model that outputs predicted emotion categories is as follows: Use the brain wave training feature set in the brain wave training data as the input of the emotion classification model. The emotion classification model outputs the predicted value of the emotion category corresponding to the brain wave training feature set, uses the emotion category label of the tester corresponding to the brain wave training feature set as the prediction target, uses the difference between the predicted value of the emotion category and the emotion category label as the first prediction error, and uses minimizing the sum of squares of the first prediction error as the training target; Train the emotion classification model until the sum of squares of the first prediction error converges and then stop training.

4. The emotion monitoring method based on deep learning and electroencephalogram morphological features according to claim 3, characterized in that The emotion recognition model that outputs the predicted emotion state includes the following steps: Step 11: Preprocess each brain wave training feature set in the brain wave training data to obtain the corresponding brain wave training preprocessing set; Step 12: Input each brain wave training preprocessing set into the emotion recognition model to obtain the emotion feature expression set corresponding to each brain wave training preprocessing set, and output each emotion feature expression set; the emotion recognition model is an unsupervised learning model.

5. The emotion monitoring method based on deep learning and electroencephalogram morphological features according to claim 4, wherein, The method for training the behavior prediction model is as follows: For each tester: Collect the emotion feature expression set corresponding to each brain wave training feature set of the tester; Combine the emotion label data and the emotion feature expression set of the tester after using the test means to form a set of behavior prediction feature sets; Use each set of behavior prediction feature sets as the input of the behavior prediction model. The behavior prediction model outputs the predicted value of the behavior type corresponding to the behavior prediction feature set, uses the behavior type label of the tester corresponding to the behavior prediction feature set as the prediction target, uses the difference between the predicted value of the behavior type and the behavior type label as the second prediction error, and uses minimizing the sum of squares of the second prediction error as the training target; train the behavior prediction model until the sum of squares of the second prediction error converges and then stop training.

6. The emotion monitoring method based on deep learning and electroencephalogram morphological features according to claim 5, characterized in that The method for collecting the brain wave monitoring data and behavior monitoring data of the person to be monitored in the real monitoring environment is as follows: The real monitoring environment is an environment where the person to be monitored wears a brain wave signal acquisition device and the brain wave signal acquisition device collects the brain wave signals of the person to be monitored in real time; During the process of the person to be monitored wearing the brain wave signal acquisition device, collect the brain wave signals of each band of the person to be monitored in real time, and form the corresponding brain wave monitoring signal sequence in chronological order; the length of the brain wave monitoring signal sequence is the same as the length of the brain wave signal sequence of the tester; the brain wave monitoring signal sequence constitutes the brain wave monitoring data of the person to be monitored; Collect the behavior type of the person to be monitored after collecting the brain wave monitoring signal sequence in real time as the behavior monitoring data.

7. The emotion monitoring method based on deep learning and electroencephalogram morphological features according to claim 6, characterized in that, The method for outputting the predicted behavior of the person to be monitored is as follows: Combine the predicted emotion category and predicted emotion state of the person to be monitored to form a set of monitoring behavior prediction feature sets; Input the monitoring behavior prediction feature sets into each behavior prediction model to obtain the predicted values of the behavior types of the person to be monitored output by each behavior prediction model; Use the voting method to select the predicted value of the behavior type with the most occurrences from the predicted values of all behavior types of the personnel to be monitored as the predicted behavior of the personnel to be monitored.

8. An emotion monitoring system based on deep learning and electroencephalogram morphological features, which is used to implement the emotion monitoring method based on deep learning and electroencephalogram morphological features described in any one of claims 1-7, characterized in that, It includes a training data collection module, a model training module, and a behavior prediction module; among them, each module is connected electrically. The training data collection module is used to pre-collect the electroencephalogram training data, emotion label data, and behavior type label data of each tester in the experimental environment, and send the electroencephalogram training data, emotion label data, and behavior type label data to the model training module. The model training module is used to train an emotion classification model that outputs predicted emotion categories with the electroencephalogram training data as the input and the emotion label data as the output; and train an emotion recognition model that outputs predicted emotion states based on the electroencephalogram training data; for each tester, use the emotion label data and emotion state as the input and the behavior type label data as the output to train a behavior prediction model, and send the emotion classification model, emotion recognition model, and behavior prediction model to the behavior prediction module. The behavior prediction module is used to perform the following steps: Step 21: In the real monitoring environment, collect the electroencephalogram monitoring data and behavior monitoring data of the personnel to be monitored. Step 22: Based on the electroencephalogram monitoring data and the emotion classification model, obtain the predicted emotion category of the personnel to be monitored; based on the electroencephalogram monitoring data and the emotion recognition model, obtain the predicted emotion state of the personnel to be monitored. Step 23: Based on the predicted emotion category, predicted emotion state of the personnel to be monitored, and the behavior prediction model of the tester, output the predicted behavior of the personnel to be monitored, and fine-tune each behavior prediction model. Step 24: Repeat steps 21 to 23 until a suitable behavior prediction model is selected from all behavior prediction models.

9. An electronic device, characterized in that, It includes: A processor and a memory, where The memory stores a computer program that can be called by the processor. The processor, by calling the computer program stored in the memory, executes the emotion monitoring method based on deep learning and electroencephalogram morphological features described in any one of claims 1-7 in the background.

10. A computer-readable storage medium, characterized in that, It stores an erasable computer program on it. When the computer program runs on the computer device, the computer device executes the emotion monitoring method based on deep learning and electroencephalogram morphological features described in any one of claims 1-7 in the background.

Images