A method, device and medium for eloquence communication training based on EEG emotion recognition

Abstract

The present invention discloses an eloquence communication training method, device and medium based on EEG emotion recognition, the method comprising: identifying physiological activity data to obtain the probability distribution of emotions and the probability distribution of emotion classification; mapping the deep features of the physiological activity data to the emotion category to obtain the emotion classification; evaluating the current emotional state of the user according to the calculated data to obtain the user's emotion value; generating the user's eloquence communication training program according to the user's emotion value. The present invention proposes an eloquence communication training method, device and medium based on EEG emotion recognition, which analyzes and calculates the physiological activity data from three different angles to obtain the emotion data, and then calculates the comprehensive emotion value according to these data, and then generates the eloquence communication training program for the user's current condition, which can solve the problem that it is difficult to generate an adaptive eloquence communication training program for the user's emotions according to the user's physiological data such as the EEG signal.

Description

Technical Field

[0001] The present invention relates to the technical field of eloquence training, and in particular to an eloquence communication training method, device and medium based on electroencephalogram emotion recognition. Background Art

[0002] In eloquence training, users often focus on improving skills and ways of expression, but tend to overlook a crucial factor - their own emotions. In fact, emotional state has a profound impact on the performance of eloquence. When emotions are stable, thinking will be clearer, and users can organize language and logic more accurately, making expressions more organized and easy to understand. In addition, since a stable emotional state helps to improve concentration, promote the development of thinking and memory abilities, and improve the feedback effect in training, a stable emotional state can improve the efficiency of eloquence training. Existing methods for eloquence training based on user emotions mainly help users stabilize their emotions and relieve tension to cope with challenges in eloquence expression. The emotion recognition system used relies on facial expressions, sound analysis or text processing.

[0003] However, the traditional method of conducting eloquence training based on user emotions ignores the potential value of physiological data sources such as EEG signals in accurately identifying human emotions, resulting in the inability to fully capture the complex emotional changes caused by the user's inner state; and it is difficult to generate a unique eloquence training plan based on the user's emotional state, which makes it difficult for users to adjust their own expression methods according to their emotional state, inhibiting the improvement of learning efficiency and emotion management. Summary of the invention

[0004] The present invention provides an eloquence communication training method, device and medium based on EEG emotion recognition, so as to solve the problem that it is difficult to generate an adaptive eloquence communication training program for the user's emotions according to the user's physiological data such as EEG signals.

[0005] In order to solve the above problems, the present invention provides an eloquence communication training method based on EEG emotion recognition, comprising:

[0006] Obtain physiological activity data of users during communication;

[0007] Performing internal physiological state recognition and external manifestation recognition on the physiological activity data to obtain a probability distribution of emotions and a probability distribution of emotion classification respectively;

[0008] Mapping the deep features of the physiological activity data to emotion categories to obtain emotion classification;

[0009] Evaluate the current emotional state of the user according to the probability distribution of the emotion, the probability distribution of the emotion classification and the emotion classification to obtain the emotion value of the user;

[0010] An eloquence communication training program for the user is generated according to the emotion value of the user.

[0011] The present invention identifies the internal physiological state of physiological activity data and obtains the probability distribution of emotions, which can intuitively reflect the current user's psychological situation and help understand the user's emotional state from the perspective of internal psychological activities; by identifying the external manifestations of physiological activity data, it can capture the external signs and movement changes related to emotions, and obtain the probability distribution of emotion classification, which helps to understand the user's emotional state from the perspective of external manifestations. Mapping the deep features of physiological activity data to predefined emotion categories is a further analysis of physiological activity data, which can intuitively and effectively obtain the user's emotion category. According to these three types of data, the user's current emotional state can be evaluated accurately by utilizing the deep insight characteristics of the probability distribution of emotions, the intuitive behavior performance of the probability distribution of emotion classification, and the deep analysis characteristics of the emotion classification data, so as to ensure that the emotion value can effectively reflect the current user's learning state, and then generate an adaptive eloquence communication training program, providing users with more suitable and efficient training content, and greatly improving learning effects.

[0012] Compared with the existing technology, the present invention analyzes and calculates physiological activity data from three different angles, obtains the probability distribution of emotions, the probability distribution of emotion classification and emotion classification, and then calculates a comprehensive emotion value based on these data, which can fully display the user's current learning status, and then generate an eloquence and communication training program for the user's current situation, thereby improving the learning effect. Therefore, it can solve the problem of difficulty in generating an adaptive eloquence and communication training program based on the user's emotions based on the user's physiological data such as EEG signals.

[0013] As a preferred solution, the physiological activity data is subjected to internal physiological state recognition to obtain the probability distribution of emotions, specifically:

[0014] Extracting features from the physiological activity data to obtain a feature graph;

[0015] According to the feature graph, the dynamic changes of emotions during communication are captured through a long short-term memory network to obtain a dynamic change value of emotions;

[0016] The dynamic change value of the emotion is mapped to the emotion category to obtain the probability distribution of the emotion.

[0017] This preferred solution extracts features from the data and can integrate the features in the physiological activity data to form a feature map that can describe the changes in physiological activities over time. Since the user's emotions will change dynamically with the content and context of the communication, the long short-term memory network can accurately capture this dynamic emotional change and obtain the dynamic change value of the emotion. According to the dynamic change value of the emotion, it is mapped to the predefined emotion category, and finally the probability distribution of various emotion categories is obtained, which indicates the possibility of the user showing various emotions at different time points, which directly reflects the user's internal physiological state.

[0018] As a preferred solution, feature extraction is performed on the physiological activity data to obtain a feature map, specifically:

[0019] Extracting the rhythm and intensity of the speech and the sound spectrum characteristics from the physiological activity data to obtain a speech rhythm value, a speech intensity value and a sound spectrum characteristic value respectively;

[0020] Calculate a feature vector based on the speech rhythm value, the speech intensity value and the sound spectrum characteristic value;

[0021] According to the feature vector, local spatiotemporal features related to emotions are extracted through a convolutional neural network to obtain the feature map.

[0022] This preferred solution integrates and calculates the extracted speech rhythm value, speech intensity value and sound spectrum characteristic value to form a feature vector, which can comprehensively reflect the user's speech performance state related to emotions in the speech. For continuously changing physiological data, its features often have temporal and spatial correlations. The convolutional neural network can capture the changes in features in time and space, thereby extracting feature maps, providing a strong data basis for subsequent calculations.

[0023] As a preferred solution, the external manifestation form of the physiological activity data is identified to obtain the probability distribution of emotion classification, specifically:

[0024] Extracting features from the physiological activity data to obtain speech features;

[0025] The speech features are converted into emotion representation values ​​by capturing the emotional flow and long-term semantic dependencies between data;

[0026] The emotional state is classified and calculated according to the emotional representation value to obtain the probability distribution of the emotional classification.

[0027] This preferred solution can more deeply understand the emotional content in speech by capturing the emotional flow and long-term semantic dependencies between data, and then more accurately convert speech features into emotional representation values, thereby removing redundant information and highlighting key features related to emotions. The probability distribution of emotion classification is calculated based on the emotional representation value. This probability distribution not only provides the final emotion classification result, but also gives the confidence of each classification result, reflecting the external manifestation of emotion.

[0028] As a preferred solution, feature extraction is performed on the physiological activity data to obtain speech features, specifically:

[0029] Calculating the dynamic range of the speech data in the physiological activity data to obtain the standard deviation of the speech amplitude;

[0030] Calculating the speech speed change of the voice data in the physiological activity data to obtain a rhythm variation value of the voice;

[0031] Calculating the frequency spectrum discreteness of the voice data in the physiological activity data to obtain the sound spectrum entropy;

[0032] The speech feature is composed of the standard deviation of the speech amplitude, the rhythm variation value of the speech and the sound spectrum entropy.

[0033] This preferred solution performs feature extraction calculations on the dynamic range, speech rate change and spectral discreteness of the voice data in the physiological activity data, and obtains three different types of parameters respectively. The standard deviation of the voice amplitude can reflect the degree of emotional fluctuation. Since a larger standard deviation means a larger change in voice intensity, it may reflect emotional fluctuation or excitement. A larger rhythm variation value may indicate emotional instability or change. A larger sound spectrum entropy may mean that the sound quality is more diverse or unstable. Therefore, a comprehensive voice feature is constructed by combining these three feature data, which can intuitively reflect the user's external emotional state and its expression in voice.

[0034] As a preferred solution, the emotional state classification calculation is performed according to the emotional representation value to obtain the probability distribution of the emotional classification, specifically:

[0035] Using a preset activation function, the emotion representation value is converted into an activation value of an output layer in a deep learning model by means of linear transformation and nonlinear activation;

[0036] According to the activation value, the probability distribution of the emotion classification is generated by the Softmax function of the deep learning model.

[0037] This preferred solution introduces a nonlinear activation function to enable the deep learning model to capture the complex patterns and nuances in the emotion representation values, so that the activation values ​​obtained reflect the model's prediction strength for different emotion categories. After using the Softmax function, the deep learning model will generate a probability value for each emotion category. These probability values ​​together constitute the probability distribution of emotion classification, providing information about the uncertainty of the model's prediction, thereby helping to generate a more appropriate eloquence communication training program.

[0038] As a preferred solution, the deep features of the physiological activity data are mapped to the emotion categories to obtain the emotion classification, specifically:

[0039] Calculate a comprehensive speech feature vector according to the physiological activity data;

[0040] Extracting emotion-related feature representations from the speech comprehensive feature vector to obtain deep feature values;

[0041] The deep feature values ​​are mapped to emotion categories using a deep learning model to obtain the emotion classification.

[0042] This preferred solution can capture speech features associated with different emotional states by performing feature extraction in a comprehensive speech feature vector; and uses a deep learning model to map the features to predefined emotion categories, so that the deep learning model can learn the complex relationship between deep feature values ​​and emotion categories, thereby ensuring the accuracy of the final emotion classification result.

[0043] As a preferred solution, according to the emotion value of the user, an eloquence communication training program for the user is generated, specifically:

[0044] Knowing the emotional state of the user according to the emotional value of the user;

[0045] If the emotional state shows that the user is currently in an anxious state, then the preset training difficulty standard value is reduced, and an eloquence communication training program for the user is generated according to the reduced training difficulty standard value;

[0046] If the emotional state shows that the user is currently in a stable state, the training difficulty standard value is increased, and an eloquence communication training program for the user is generated according to the increased training difficulty standard value.

[0047] In this preferred solution, for users who are in an anxious state, reducing the difficulty of training can help relieve their stress and avoid further emotional burden; while for emotionally stable users, increasing the difficulty can challenge their abilities and promote further improvement of their skills. This adaptive eloquence and communication training program can help provide users with more appropriate and efficient training content, thereby improving learning outcomes.

[0048] The present invention also provides an eloquence communication training device based on EEG emotion recognition, comprising a data module, an emotion module, a mapping module, a state module and a task module;

[0049] Wherein, the data module is used to obtain the physiological activity data of the user during communication;

[0050] The emotion module is used to identify the internal physiological state and external manifestation of the physiological activity data, and obtain the probability distribution of emotions and the probability distribution of emotion classification respectively;

[0051] The mapping module is used to map the deep features of the physiological activity data to the emotion category to obtain the emotion classification;

[0052] The state module is used to evaluate the current emotional state of the user according to the probability distribution of the emotion, the probability distribution of the emotion classification and the emotion classification to obtain the emotion value of the user;

[0053] The task module is used to generate an eloquence communication training program for the user according to the emotion value of the user.

[0054] As a preferred solution, the emotion module includes an extraction unit, a capture unit and a mapping unit;

[0055] Wherein, the extraction unit is used to extract features from the physiological activity data to obtain a feature map;

[0056] The capturing unit is used to capture the dynamic changes of emotions during communication through a long short-term memory network according to the feature graph to obtain a dynamic change value of emotions;

[0057] The mapping unit is used to map the emotion dynamic change value to the emotion category to obtain the probability distribution of the emotion.

[0058] As a preferred solution, the extraction unit includes a speech subunit, a vector subunit and a feature map subunit;

[0059] The speech subunit is used to extract the rhythm and intensity of the speech in the physiological activity data, as well as the sound spectrum characteristics, and obtain the speech rhythm value, the speech intensity value and the sound spectrum characteristic value respectively;

[0060] The vector subunit is used to calculate the feature vector according to the speech rhythm value, the speech intensity value and the sound spectrum characteristic value;

[0061] The feature map subunit is used to extract local spatiotemporal features related to emotions through a convolutional neural network according to the feature vector to obtain the feature map.

[0062] As a preferred solution, the emotion module includes an activity unit, a relationship unit and a classification unit;

[0063] Wherein, the activity unit is used to extract features from the physiological activity data to obtain speech features;

[0064] The relationship unit is used to convert the speech features into emotion representation values ​​by capturing the emotion flow and long-term semantic dependency between data;

[0065] The classification unit is used to perform classification calculation of the emotional state according to the emotion representation value to obtain the probability distribution of the emotion classification.

[0066] As a preferred solution, the activity unit includes an amplitude subunit, a variation subunit, a spectrum subunit and a feature subunit;

[0067] Wherein, the amplitude subunit is used to calculate the dynamic range of the voice data in the physiological activity data to obtain the standard deviation of the voice amplitude;

[0068] The variation subunit is used to calculate the speech rate variation of the speech data in the physiological activity data to obtain the rhythm variation value of the speech;

[0069] The spectrum subunit is used to calculate the spectrum discreteness of the voice data in the physiological activity data to obtain the sound spectrum entropy;

[0070] The feature subunit is used to form the speech feature from the standard deviation of the speech amplitude, the rhythm variation value of the speech and the sound spectrum entropy.

[0071] As a preferred solution, the classification unit includes an activation subunit and a classification subunit;

[0072] The activation subunit is used to use a preset activation function to convert the emotion representation value into an activation value of an output layer in a deep learning model by means of linear transformation and nonlinear activation;

[0073] The classification subunit is used to generate the probability distribution of the emotion classification according to the activation value through the Softmax function of the deep learning model.

[0074] As a preferred solution, the task module includes an emotion unit, an anxiety unit and a stability unit;

[0075] Wherein, the emotion unit is used to know the emotional state of the user according to the emotional value of the user;

[0076] The anxiety unit is configured to reduce a preset training difficulty standard value if the emotional state shows that the user is currently in an anxious state, and generate an eloquence communication training program for the user according to the reduced training difficulty standard value;

[0077] The stabilization unit is used to increase the training difficulty standard value if the emotional state shows that the user is currently in a stable state, and generate an eloquence communication training program for the user according to the increased training difficulty standard value.

[0078] As a preferred solution, the mapping module includes a synthesis unit, a depth unit and a model unit;

[0079] Wherein, the synthesis unit is used to calculate the speech synthesis feature vector according to the physiological activity data;

[0080] The depth unit is used to extract the feature representation related to emotion from the comprehensive feature vector of speech to obtain the depth feature value;

[0081] The model unit is used to map the deep feature value to the emotion category using a deep learning model to obtain the emotion classification.

[0082] The present invention also provides a storage medium having a computer program stored thereon. The computer program is called and executed by a computer to implement the above-mentioned method for eloquence communication training based on EEG emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a flowchart of an eloquence communication training method based on EEG emotion recognition provided by an embodiment of the present invention;

[0084] Figure 2 It is a structural schematic diagram of an eloquence communication training device based on EEG emotion recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0086] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "several" means two or more.

[0087] An eloquence communication training method based on EEG emotion recognition provided by an embodiment of the present invention is mainly used in situations where it is necessary to identify the user's current emotions based on the user's brain wave and other physiological activity data, and then generate an adaptive eloquence communication training method to train the user in eloquence communication.

[0088] Embodiment 1:

[0089] See also Figure 1 The embodiment of the present invention provides an eloquence communication training method based on EEG emotion recognition, including S1 to S5, and the specific implementation steps are as follows:

[0090] S1. Obtaining physiological activity data of the user during communication.

[0091] Step S1 of the embodiment of the present invention is specifically:

[0092] When users communicate, an electroencephalogram (EEG) sensor is used to capture the electrical activity of key areas of the brain, such as the frontal lobe, to obtain brain wave data;

[0093] Using facial electromyography (EMG) sensors to monitor muscle activity in the forehead and around the eyes to obtain emotional expression data;

[0094] Use heart rate variability (HRV) and galvanic skin response (GSR) sensors to monitor changes in heart rate and skin to obtain emotional state data;

[0095] Obtain voice data during user communication;

[0096] Physiological activity data is composed of brain wave data, emotion expression data, emotion state data and voice data.

[0097] This embodiment S1 can combine advanced EEG data acquisition technology and multimodal sensors (including bioimpedance, heart rate variability, skin electrical response, etc.) by continuously monitoring and recording the user's EEG activity, capturing subtle muscle changes related to facial expressions through facial EMG, and recording data through HRV and GSR. It can accurately capture subtle physiological changes that are closely related to emotional states, thereby providing a comprehensive physiological-emotional portrait.

[0098] S2. Identify the internal physiological state and external manifestation of the physiological activity data to obtain the probability distribution of emotions and the probability distribution of emotion classification respectively.

[0099] Step S2 of the embodiment of the present invention includes S2.1 to S2.4; wherein S2.1 is a process of constructing a feature map, S2.2 is a process of calculating the probability distribution of emotions, S2.3 is a process of constructing speech features, and S2.4 is a process of calculating the probability distribution of emotion classification, specifically:

[0100] S2.1. Extract the speech rhythm, speech intensity and sound spectrum characteristics from the physiological activity data and obtain the speech rhythm value R s , speech intensity value I s and the sound spectrum characteristic value S f ;

[0101] Among them, speech rhythm is to capture the rhythm characteristics of emotional expression by calculating the changes in the user's speaking speed and the length of pauses;

[0102] Voice intensity is a measure of the intensity of the voice during communication to reflect the intensity of the user's emotions;

[0103] The sound spectrum characteristics include fundamental frequency, formant, etc. These features are related to the user's emotional state and the perceived quality of the sound.

[0104] According to the speech rhythm value R s , speech intensity value I s and the sound spectrum characteristic value S f Calculate the eigenvector v s ;

[0105] According to the feature vector v s , the local spatiotemporal features related to emotions are extracted through convolutional neural networks to obtain the feature map h c ;

[0106] Among them, the eigenvector is:

[0107] v s =[R s ,I s ,S f ] T

[0108] The feature map is:

[0109]

[0110] Among them, T represents the transpose of the vector; i represents the index of the output feature map, indicating that this is the i-th feature after processing; j represents the index of the summation, which is used to traverse the feature vector v sAll elements of , where j is used to index each element in the feature vector and the corresponding convolution W c,i,j The weight of

[0111] W c and b c are the weights and biases of the convolutional layer, and ReLU is the activation function.

[0112] In this embodiment S2.1, the extracted speech rhythm value, speech intensity value and sound spectrum characteristic value are integrated and calculated to form a feature vector, which can comprehensively reflect the user's speech performance state related to emotions in the speech. For continuously changing physiological data, its features often have temporal and spatial correlations. The convolutional neural network can capture the changes of features in time and space, thereby extracting feature maps, providing a strong data basis for subsequent calculations.

[0113] S2.2. According to the feature map h c , the dynamic changes of emotions during communication are captured through the long short-term memory network (LSTM), and the dynamic change value of emotions h is obtained t1 ;

[0114] In the recurrent neural network (RNN), a fully connected layer is used to transform the dynamic change value h t1 Mapped to the emotion category, we get the probability distribution p of the emotion t1 ; Among them, during mapping, the softmax function is used to ensure that the output p t1 All elements of are positive, and the sum of all elements is 1, so that p t1 can be interpreted as a probability distribution;

[0115] Among them, the long short-term memory network is updated to obtain the dynamic change value of emotion h t1 The formula is:

[0116] i t =σ(W i ·[h t-1 ,h c,t ]+b i );

[0117] f t =σ(W f ·[h t-1 ,h c,t ]+b f );

[0118] o t =σ(W o ·[h t-1 ,h c,t ]+b o );

[0119]

[0120] h t1 =o t ⊙tanhc t

[0121] The probability distribution of emotions is:

[0122] p t1 =softmax(W p ·h t1 +b p )

[0123] Among them, i t is the output of the input gate, f t is the output of the forget gate, o t is the output of the output gate, is the temporary cell state, c t Indicates the state of the cell after update, h t1 is the output of the long short-term memory network, σ is the sigmoid function, ⊙ represents the Hadamard product, h t-1 is the output at the previous moment, h c,t is the input at the current moment, W i and b i are the weight and bias of the input gate, W f and b f are the weight and bias of the forget gate, W o and b o are the weight and bias of the output gate, W c and b c are the weight and bias of the temporary cell state, W p and b p They are the weight matrix and bias vector of the fully connected layer respectively;

[0124] In this embodiment S2.2, since the user's emotions will change dynamically with the content and context of the communication, the long short-term memory network can accurately capture this dynamic emotional change and obtain the emotional dynamic change value. According to the emotional dynamic change value, it is mapped to the predefined emotional category, and finally the probability distribution of various emotional categories is obtained, which indicates the possibility of the user showing various emotions at different time points, and directly reflects the user's internal physiological state.

[0125] S2.3. Calculate the dynamic range of speech data in physiological activity data and obtain the standard deviation D of speech amplitude r ;

[0126] The speech rate variation of the speech data in the physiological activity data is measured by calculating the variance of the speech rate in different time windows, and the rhythm variation value R of the speech is obtained. v ;

[0127] Calculate the spectral discreteness of the speech data in the physiological activity data and obtain the sound spectrum entropy S e ;

[0128] The standard deviation of the speech amplitude D r , the rhythm variation value R of speech v and the sound spectrum entropy S e constitutes the characteristics of speech;

[0129] The standard deviation of the speech amplitude is:

[0130]

[0131] The rhythm variation value of speech is:

[0132]

[0133] The sound spectrum entropy is:

[0134]

[0135] Among them, x t is the amplitude value at time point t, is the average amplitude, T is the total number of time frames; s i is the speaking rate in the ith window, s- is the average speaking rate, N is the number of windows, and P(f) is the normalized value of the power spectral density at frequency f.

[0136] In this embodiment S2.3, the voice data in the physiological activity data is subjected to feature extraction and calculation on dynamic range, speech rate variation and spectrum discreteness, and three different types of parameters are obtained respectively. The standard deviation of the voice amplitude can reflect the degree of emotional fluctuation. Since a larger standard deviation means a larger variation in voice intensity, it may reflect emotional fluctuation or excitement. A larger rhythm variation value may indicate emotional instability or change. A larger sound spectrum entropy may mean that the sound quality is more diverse or unstable. Therefore, a comprehensive voice feature is constructed by combining these three feature data, which can intuitively reflect the user's external emotional state and its expression in voice.

[0137] S2.4. Capturing the emotional flow and long-term semantic dependencies between data through a bidirectional LSTM network (Bi-LSTM) to convert speech features into emotional representation values ​​h t2 ;

[0138] Use the preset activation function to transform the emotion representation value h t2It is converted into the activation value o of the output layer in the deep learning model by linear transformation and nonlinear activation. t1 ;

[0139] According to the activation value o t1 , the probability distribution p of emotion classification is generated through the Softmax function of the deep learning model t2 ;

[0140] Among them, the emotion representation value is:

[0141]

[0142] For parameters and have:

[0143]

[0144] The activation values ​​of the output layer and the probability distribution of emotion classification are:

[0145] o t1 =tanh(W h ·h t2 +b h );

[0146] p t2 =softmax(W o o t1 +b o )

[0147] Among them, e t =,D r ,R v ,S e-t , e t is the eigenvector at time step t, e t Used to get h by merging the forward and backward hidden states t , D r is the standard deviation of speech amplitude, R v is the rhythm variation value of speech, S e is the sound spectrum entropy; W represents the weight matrix, and W h is the weight matrix connected to the hidden layer (or middle layer), W o is the weight matrix connected to the output layer (i.e., the layer for sentiment classification), b h and b o is bias.

[0148] By capturing the emotional flow and long-term semantic dependencies between data, this embodiment S2.4 can more deeply understand the emotional content in speech, and then more accurately convert speech features into emotional representation values, thereby removing redundant information and highlighting key features related to emotions. The probability distribution of emotion classification is calculated based on the emotional representation values. This probability distribution not only provides the final emotion classification result, but also gives the confidence of each classification result, reflecting the external manifestation of emotion;

[0149] Moreover, by introducing nonlinear activation functions, the deep learning model can capture the complex patterns and nuances in the emotion representation values, so that the obtained activation values ​​reflect the prediction strength of the model for different emotion categories. After using the Softmax function, the deep learning model generates a probability value for each emotion category. These probability values ​​together constitute the probability distribution of emotion classification, providing information about the uncertainty of the model prediction, which helps to generate more appropriate eloquence communication training programs.

[0150] S3. Map the deep features of the physiological activity data to the emotion categories to obtain the emotion classification.

[0151] Step S3 of the embodiment of the present invention includes S3.1 to S3.2, wherein S3.1 is a process of calculating emotion classification, and S3.2 is a process of constructing a decision tree to explain how to infer emotions from features, specifically:

[0152] S3.1. Calculate the pitch variation (PV) and speech fluency (SF) of speech according to the physiological activity data to obtain the pitch variation value PV and speech fluency value SF of speech respectively;

[0153] The voice tonality change value PV and language fluency value SF are integrated to obtain the voice comprehensive feature vector v int ;

[0154] From the speech synthesis feature vector v int Extract the feature representation related to emotion and obtain the deep feature value h;

[0155] Use a deep feature transformation network (DFTN) to map the deep feature value h to the emotion category to obtain the emotion classification p. DFTN is a deep learning network for static data analysis. It can learn more complex emotion patterns from the dynamic emotion features extracted by LSTM. Moreover, the network focuses on identifying key emotion drivers, such as voice tonality changes and language fluency. For example, a multi-layer perceptron (MLP) is used to extract emotion-related high-level representations from complex features.

[0156] Among them, the tonality change value of the voice is:

[0157]

[0158] The language fluency values ​​are:

[0159]

[0160] The comprehensive feature vector of speech is:

[0161] v int =[R s ,I s ,S f ,PV,SF] T

[0162] The depth feature value is:

[0163] h=DFTN(v int )

[0164] Emotions are categorized as:

[0165] p = softmax(W h + b)

[0166] Where N is the total number of measurement time points, f i represents the fundamental frequency at time i, v i represents the speaking rate at time i, R s ,I s and S f are the speech rhythm value, speech intensity value and sound spectrum characteristic value respectively; W and b are the weight and bias of the fully connected layer, which are used to map the deep features to the emotion categories.

[0167] It should be noted that, in this embodiment, the long short-term memory network (LSTM) is mainly used to process continuous physiological and speech data to identify and track the time-dependent changes of emotional states;

[0168] The Deep Feature Transformation Network (DFTN) further extracts complex patterns related to emotions from the output of LSTM, performs static deep feature analysis, and finally outputs the precise probability distribution of emotion classification;

[0169] For the long short-term memory network (LSTM) and deep feature transformation network (DFTN), this division of labor allows the system to not only capture the dynamic changes of emotions, but also effectively identify key factors affecting emotions, providing an effective scientific basis for formulating training strategies.

[0170] This embodiment S3.1 can capture speech features associated with different emotional states by performing feature extraction in a comprehensive speech feature vector; and uses a deep transformation network (DFTN) to map the features to predefined emotion categories, so that the deep learning model can learn the complex relationship between deep feature values ​​and emotion categories, thereby ensuring the accuracy of the final emotion classification result. In addition, the output of DFTN is the probability distribution of emotion classification, which can provide a data basis for subsequent training adjustments and feedback generation.

[0171] S3.2, extracting more advanced, rich and abstract information carriers from the preset user original emotion features to obtain advanced features h; and establishing emotion category labels Y containing several emotion categories;

[0172] Get the structural model;

[0173] The control structure model is trained by learning the relationship between the high-level feature h and the emotion category label Y, so that the structure model learns a series of "if-then" rules (for example, if the value of a certain feature is greater than a certain threshold, the sample is classified into a specific category), and the decision tree Tree is obtained after the training is completed;

[0174] Control decision tree Tree explains how to infer emotions from features;

[0175] Among them, the construction formula of the decision tree is:

[0176] Tree = buildDecisionTree(h,Y)

[0177] Among them, buildDecisionTree is a function name, which represents the process of building a decision tree. The function of this function is to build a decision tree model based on the given input data.

[0178] In this embodiment S3.2, a decision tree is constructed to explain how to infer emotions from features. Since the decision tree can visualize complex processing processes, users can clearly see the entire process from input data to output emotions. It can not only provide intuitive and visual explanations, but also help people understand the key factors that affect emotions and improve model performance.

[0179] S4. Evaluate the user's current emotional state according to the probability distribution of emotions, the probability distribution of emotion classification, and the emotion classification to obtain the user's emotion value.

[0180] Step S4 of the embodiment of the present invention is specifically as follows:

[0181] By combining the probability distribution p of the emotion t1 , the probability distribution of emotion classification p t2, emotion classification p and preset user historical emotion data, evaluate the user's current emotional state and obtain the user's emotion value.

[0182] S5. Generate an eloquence communication training program for the user based on the user's emotional value.

[0183] The steps of the embodiment of the present invention include S5.1 to S5.2; wherein S5.1 is the process of generating an eloquence communication training program, and S5.2 is the process of performing eloquence communication training on a user, specifically:

[0184] S5.1. Know the user's emotional state based on the user's emotional value, identify the dominant emotion in the emotional state through a preset algorithm and analyze its potential impact;

[0185] If the emotional state shows that the user is currently in an anxious state, the preset training difficulty standard value is reduced through the preset emotional decision model, and more positive incentive feedback is provided, and the user's eloquence communication training plan is generated according to the reduced training difficulty standard value;

[0186] If the emotional state shows that the user is currently in a stable state, the training difficulty standard value is increased through the emotional decision model, and the user's eloquence communication training plan is generated according to the increased training difficulty standard value;

[0187] For example, if the user is in a state of high anxiety, then build some relaxation training or meditation eloquence and communication training programs; if the user is in an emotionally stable state, then build a more challenging eloquence and communication training program.

[0188] Among them, the emotional decision-making model dynamically adjusts the training parameters based on the emotional probability distribution contained in the emotional state and the user's historical training performance; specifically: first, define the influence weights of different emotional states, and then analyze these weights through decision support algorithms such as logistic regression to generate a standard value for training difficulty; then use this standard value to guide the specific adjustment of training difficulty, duration and frequency to ensure that the training plan is both personalized and efficient, thereby generating the user's eloquence and communication training plan.

[0189] The construction process of the emotional decision-making model is as follows:

[0190] 1) Convert sentiment probability into decision-making indicators:

[0191] First, define the training adjustment parameters corresponding to each emotional state. For example, high anxiety may reduce the difficulty, and high pleasure may increase the difficulty;

[0192] Then, a weight is assigned to each emotional state, where the weight is calculated based on the emotional probability distribution and represents the influence of the emotional state on the training program.

[0193] 2) Construction of emotional decision-making model:

[0194] Use decision trees, logistic regression, or neural networks to analyze the weights of various emotions and combine them with the user's historical training data to determine the training strategy that best suits the current emotional state;

[0195] The model calculates a "training intensity index" (i.e., the standard value of training difficulty), which determines the adjustment of training difficulty, duration, frequency, etc.

[0196] According to the training strategy, the initial emotional decision-making model is generated in combination with the training intensity index and the training adjustment parameters.

[0197] 3) Model training and implementation:

[0198] The initial emotional decision-making model is trained through historical data to ensure that the model can understand the most effective training methods under different emotional states and obtain the final emotional decision-making model.

[0199] In this embodiment S5.1, for users in an anxious state, reducing the training difficulty helps to relieve their stress and avoid further emotional burden; while for emotionally stable users, increasing the difficulty can challenge their abilities and promote further improvement of their skills. This adaptive eloquence communication training program helps to provide users with more appropriate and efficient training content, thereby improving learning effects;

[0200] Furthermore, in order to accurately establish an eloquence communication training program, this embodiment introduces an emotional decision model, which dynamically adjusts training parameters based on the emotional probability distribution and the user's historical training performance, thereby ensuring that the training program is both personalized and efficient.

[0201] S5.2. Use the eloquence communication training program to train users in eloquence communication;

[0202] When the user is training, the training intensity and type of the eloquence communication training program are adjusted in real time by continuously monitoring the changes in the user's emotional state, and the user's real-time emotional data is collected;

[0203] After the user completes the training task, the user's profile is updated with the collected real-time emotion data;

[0204] Furthermore, by dynamically adjusting the algorithm, the training content of the eloquence and communication training program is continuously optimized based on the user’s real-time emotional data, ensuring that the training can challenge the user without causing undue stress.

[0205] In this embodiment S5.2, this instant feedback mechanism helps users manage and adjust their emotions more effectively. This cycle ensures that the training content is continuously optimized and personalized over time, gradually improving the user's ability in emotion management and eloquence.

[0206] The embodiment of the present invention also includes S6, visually displaying the real-time emotion data.

[0207] The steps of the embodiment of the present invention include S6.1 to S6.2; wherein S6.1 is a process of establishing visualization data, S6.2 is a process of constructing emotion visualization data, and S6.3 is a process of performing user interaction, specifically:

[0208] S6.1. Decode and standardize the real-time emotion data, synchronize the time and unify the format to obtain the parsed data p t3 ;

[0209] According to the data access frequency and the user's real-time requirements, the parsed data is cached according to the preset cache mechanism. t3 ;

[0210] For the parsed data p t3 Preprocessing is performed, including data cleaning (removing outliers or correcting erroneous data), data aggregation (such as averaging to smooth out emotional changes), etc., to obtain visual data v vis ;

[0211] The visualization data is:

[0212] v vis =aggregate({p t3-k →p t3 +)

[0213] Here, aggregate is a function that aggregates data within a certain time window into a set of average data for visualization, and k is the length or size of the time window.

[0214] S6.2. Control the preset visualization engine to obtain visualization data v in a structured format (e.g., a JSON object containing a timestamp and corresponding sentiment analysis results) vis , obtain the first data Data; wherein the first data Data can be expressed as Data = {'timestamp':t,'emotions':p t3 +, 'timestamp' represents the timestamp of the data item, 'emotions' represents the emotional information of the data item, t represents the timestamp data, p t3 It is to parse the data;

[0215] Remove redundant information in the first data Data, and adjust the format of the first data Data according to visualization requirements (such as converting the emotion probability into the input format required by the visualization library), to obtain second data;

[0216] Generate a number of corresponding graphic representations according to the second data to obtain emotion visualization data; wherein the number of graphic representations include a curve of emotion state change over time, a heat map of emotion intensity, etc.;

[0217] Among them, the drawing operation can be expressed as:

[0218] Graph(time,p tn )=Plot(time series of p tn )

[0219] Among them, time is a sequence representing time points, p tn Plot (time series of p tn ) means to convert p tn Plot the time series data into a graph.

[0220] S6.3. Integrate emotion visualization data into augmented reality or virtual reality (AR / VR) environments;

[0221] Creating user interfaces in AR or VR environments, such as displaying emotion changes through head-mounted displays, allowing users to explore emotion data through gestures or viewing different perspectives, and allowing users to interact with emotion visualization data through touch screens, voice commands, or mobile device operations;

[0222] Adjust visual output based on user interaction, and update the display content of the user interface in real time to respond to user input;

[0223] When the user interface is running, the performance of the visualization engine is monitored to ensure efficient processing and display of real-time data streams, and the response speed and processing power of the user interface are improved by optimizing data processing and visualization algorithms based on performance feedback (for example, through asynchronous data loading, memory management optimization, etc.).

[0224] This embodiment S6.3 uses AR / VR technology to not only provide emotion recognition training, but also create an immersive environment in which users can practice eloquence. This immersive experience can greatly improve user participation and the effect of emotion learning;

[0225] In addition, natural interaction methods such as touch screens can reduce the user's burden on technical operations, allowing them to focus more on training content and ensuring that all users can quickly get started and effectively use the user interaction interface.

[0226] Overall, this embodiment has the following beneficial effects:

[0227] The embodiment of the present invention identifies the internal physiological state of physiological activity data and obtains the probability distribution of emotions, which can intuitively reflect the current user's psychological situation and help understand the user's emotional state from the perspective of internal psychological activities; by identifying the external manifestations of physiological activity data, it can capture the external signs and movement changes related to emotions, and obtain the probability distribution of emotion classification, which helps to understand the user's emotional state from the perspective of external manifestations. Mapping the deep features of physiological activity data to predefined emotion categories is a further analysis of physiological activity data, which can intuitively and effectively obtain the user's emotion category. According to these three types of data, the user's current emotional state can be evaluated accurately by utilizing the deep insight characteristics of the probability distribution of emotions, the intuitive behavior performance of the probability distribution of emotion classification, and the deep analysis characteristics of the emotion classification data, so as to ensure that the emotion value can effectively reflect the current user's learning state, and then generate an adaptive eloquence communication training program, providing users with more suitable and efficient training content, and greatly improving learning effects.

[0228] Furthermore, this embodiment provides comprehensive support for emotion and eloquence training through comprehensive multi-source data analysis, which is not limited to the improvement of eloquence skills, but also includes the cultivation of emotion management ability, which is essential for effective public speaking.

[0229] Embodiment 2:

[0230] See also Figure 2 , an embodiment of the present invention provides an eloquence communication training device based on EEG emotion recognition, comprising a data module 10, an emotion module 20, a mapping module 30, a state module 40 and a task module 50;

[0231] Wherein, the data module 10 is used to obtain the physiological activity data of the user during communication;

[0232] The emotion module 20 is used to identify the internal physiological state and external manifestation of the physiological activity data, and obtain the probability distribution of emotions and the probability distribution of emotion classification respectively;

[0233] A mapping module 30, used to map the deep features of the physiological activity data to the emotion category to obtain the emotion classification;

[0234] A state module 40 is used to evaluate the current emotional state of the user according to the probability distribution of emotions, the probability distribution of emotion classification and the emotion classification to obtain the emotion value of the user;

[0235] The task module 50 is used to generate a user's eloquence communication training program according to the user's emotion value.

[0236] In one embodiment, the data module 10 is specifically:

[0237] When users communicate, an electroencephalogram (EEG) sensor is used to capture the electrical activity of key areas of the brain, such as the frontal lobe, to obtain brain wave data;

[0238] Using facial electromyography (EMG) sensors to monitor muscle activity in the forehead and around the eyes to obtain emotional expression data;

[0239] Use heart rate variability (HRV) and galvanic skin response (GSR) sensors to monitor changes in heart rate and skin to obtain emotional state data;

[0240] Obtain voice data during user communication;

[0241] Physiological activity data is composed of brain wave data, emotion expression data, emotion state data and voice data.

[0242] The data module 10 of this embodiment continuously monitors and records the user's EEG activity, captures subtle muscle changes related to facial expressions through facial EMG, and records data through HRV and GSR. It can combine advanced EEG data acquisition technology and multimodal sensors (including bioimpedance, heart rate variability, skin electrical response, etc.) to accurately capture subtle physiological changes that are closely related to emotional states, thereby providing a comprehensive physiological and emotional portrait.

[0243] In one embodiment, the emotion module 20 includes a speech subunit, a vector subunit, a feature map subunit, a capture unit, a mapping unit, an amplitude subunit, a variation subunit, a spectrum subunit, a feature subunit, a relation unit, an activation subunit, and a classification subunit;

[0244] Among them, the speech subunit, vector subunit and feature map subunit are the process of constructing feature maps, the capture unit and mapping unit are the process of calculating the probability distribution of emotions, the amplitude subunit, change subunit, spectrum subunit and feature subunit are the process of constructing speech features, and the relationship unit, activation subunit and classification subunit are the process of calculating the probability distribution of emotion classification, specifically:

[0245] The speech subunit is used to extract the speech rhythm, speech intensity and sound spectrum characteristics in the physiological activity data, and obtain the speech rhythm value R s , speech intensity value Is and the sound spectrum characteristic value S f ;

[0246] Among them, speech rhythm is to capture the rhythm characteristics of emotional expression by calculating the changes in the user's speaking speed and the length of pauses;

[0247] Voice intensity is a measure of the intensity of the voice during communication to reflect the intensity of the user's emotions;

[0248] The sound spectrum characteristics include fundamental frequency, formant, etc. These features are related to the user's emotional state and the perceived quality of the sound.

[0249] The vector subunit is used to calculate the speech rhythm value R s , speech intensity value I s and the sound spectrum characteristic value S f Calculate the eigenvector v s ;

[0250] The feature map subunit is used to calculate the feature vector v s , the local spatiotemporal features related to emotions are extracted through convolutional neural networks to obtain the feature map h c ;

[0251] Among them, the eigenvector is:

[0252] v s =[R s ,I s ,S f ] T

[0253] The feature map is:

[0254]

[0255] Among them, T represents the transpose of the vector; i represents the index of the output feature map, indicating that this is the i-th feature after processing; j represents the index of the summation, which is used to traverse the feature vector v s All elements of , where j is used to index each element in the feature vector and the corresponding convolution W c,i,j The weight of

[0256] W c and b c are the weights and biases of the convolutional layer, and ReLU is the activation function.

[0257] The speech subunit, vector subunit and feature map subunit of this embodiment integrate and calculate the extracted speech rhythm value, speech intensity value and sound spectrum characteristic value to form a feature vector, which can comprehensively reflect the user's speech performance state related to emotions in the speech. For continuously changing physiological data, its features often have temporal and spatial correlations. The convolutional neural network can capture the changes of features in time and space, thereby extracting feature maps, providing a strong data basis for subsequent calculations.

[0258] Capture unit, used to obtain the feature map h c , the dynamic changes of emotions during communication are captured through the long short-term memory network (LSTM), and the dynamic change value of emotions h is obtained t1 ;

[0259] The mapping unit is used to transform the dynamic change value h of the emotion into the full connection layer in the recurrent neural network (RNN). t1 Mapped to the emotion category, we get the probability distribution p of the emotion t1 ; Among them, during mapping, the softmax function is used to ensure that the output p t1 All elements of are positive, and the sum of all elements is 1, so that p t1 can be interpreted as a probability distribution;

[0260] Among them, the long short-term memory network is updated to obtain the dynamic change value of emotion h t1 The formula is:

[0261] i t =σ(W i ·[h t-1 ,h c,t ]+b i );

[0262] f t =σ(W f ·[h t-1 ,h c,t ]+b f );

[0263] o t =σ(W o ·[h t-1 ,h c,t ]+b o );

[0264]

[0265] h t1 =o t ⊙tanhc t

[0266] The probability distribution of emotions is:

[0267] p t1 =softmax(W p ·h t1 +b p )

[0268] Among them, i t is the output of the input gate, f t is the output of the forget gate, o t is the output of the output gate, is the temporary cell state, c t Indicates the state of the cell after update, h t1 is the output of the long short-term memory network, σ is the sigmoid function, ⊙ represents the Hadamard product, h t-1 is the output at the previous moment, h c,t is the input at the current moment, W i and b i are the weight and bias of the input gate, W f and b f are the weight and bias of the forget gate, W o and b o are the weight and bias of the output gate, W c and b c are the weight and bias of the temporary cell state, W p and b p They are the weight matrix and bias vector of the fully connected layer respectively;

[0269] In the capture unit and mapping unit of this embodiment, since the user's emotions will change dynamically with the content and context of the communication, the long short-term memory network can accurately capture such dynamic emotional changes and obtain the dynamic change value of the emotion. According to the dynamic change value of the emotion, it is mapped to the predefined emotion category, and finally the probability distribution of various emotion categories is obtained, which indicates the possibility of the user showing various emotions at different time points, and directly reflects the user's internal physiological state.

[0270] The amplitude subunit is used to calculate the dynamic range of speech data in physiological activity data and obtain the standard deviation D of speech amplitude. r ;

[0271] The variation subunit is used to measure the speech rate variation of the speech data in the physiological activity data by calculating the variance of the speech rate in different time windows, and obtain the rhythm variation value R of the speech. v ;

[0272] The spectrum subunit is used to calculate the spectrum discreteness of the speech data in the physiological activity data and obtain the sound spectrum entropy S e ;

[0273] The feature subunit is used to represent the standard deviation D of the speech amplitude. r , the rhythm variation value R of speech v and the sound spectrum entropy S e constitutes the characteristics of speech;

[0274] The standard deviation of the speech amplitude is:

[0275]

[0276] The rhythm variation value of speech is:

[0277]

[0278] The sound spectrum entropy is:

[0279]

[0280] Among them, x t is the amplitude value at time point t, is the average amplitude, T is the total number of time frames; s i is the speaking rate in the ith window, is the average speaking rate, N is the number of windows, and P(f) is the normalized value of the power spectral density at frequency f.

[0281] In this embodiment, the amplitude subunit, change subunit, spectrum subunit and feature subunit perform feature extraction calculations on the dynamic range, speech rate change and spectrum discreteness of the voice data in the physiological activity data, and obtain three different types of parameters respectively. The standard deviation of the voice amplitude can reflect the degree of emotional fluctuation. Since a larger standard deviation means a larger change in voice intensity, it may reflect emotional fluctuation or excitement. A larger rhythm variation value may indicate emotional instability or change. A larger sound spectrum entropy may mean that the sound quality is more diverse or unstable. Therefore, a comprehensive voice feature is constructed by combining these three feature data, which can intuitively reflect the user's external emotional state and its expression in the voice.

[0282] The relation unit is used to capture the emotional flow and long-term semantic dependencies between data through a bidirectional LSTM network (Bi-LSTM), and convert speech features into emotional representation values ​​h t2 ;

[0283] The activation subunit is used to use the preset activation function to convert the emotion representation value h t2 It is converted into the activation value o of the output layer in the deep learning model by linear transformation and nonlinear activation. t1 ;

[0284] The classification subunit is used to classify the activation value o t1 , the probability distribution p of emotion classification is generated through the Softmax function of the deep learning model t2 ;

[0285] Among them, the emotion representation value is:

[0286]

[0287] For parameters and have:

[0288]

[0289] The activation values ​​of the output layer and the probability distribution of emotion classification are:

[0290]

[0291] p t2 =softmax(W o o t1 +b o )

[0292] Among them, e t =,D r ,R v ,S e-t , e t is the eigenvector at time step t, e t Used to get h by merging the forward and backward hidden states t , D r is the standard deviation of speech amplitude, R v is the rhythm variation value of speech, S e is the sound spectrum entropy; W represents the weight matrix, and W h is the weight matrix connected to the hidden layer (or middle layer), W o is the weight matrix connected to the output layer (i.e., the layer for sentiment classification), b h and b o is bias.

[0293] The relationship unit, activation subunit, and classification subunit of this embodiment can capture the emotional flow and long-term semantic dependency between data to more deeply understand the emotional content in speech, and then more accurately convert speech features into emotional representation values, thereby removing redundant information and highlighting key features related to emotions. The probability distribution of emotion classification is calculated based on the emotional representation value. This probability distribution not only provides the final emotion classification result, but also gives the confidence of each classification result, reflecting the external manifestation of emotion;

[0294] Moreover, by introducing nonlinear activation functions, the deep learning model can capture the complex patterns and nuances in the emotion representation values, so that the obtained activation values ​​reflect the prediction strength of the model for different emotion categories. After using the Softmax function, the deep learning model generates a probability value for each emotion category. These probability values ​​together constitute the probability distribution of emotion classification, providing information about the uncertainty of the model prediction, which helps to generate more appropriate eloquence communication training programs.

[0295] In one embodiment, the mapping module 30 includes a synthesis unit, a depth unit, a model unit, and a decision tree unit, wherein the synthesis unit, the depth unit, and the model unit are processes for calculating emotion classification, and the decision tree unit is a process for constructing a decision tree to explain how to infer emotions from features, specifically:

[0296] A comprehensive unit is used to calculate the pitch variation (PV) and speech fluency (SF) of speech according to the physiological activity data, and obtain the pitch variation value PV and the speech fluency value SF of speech respectively;

[0297] The comprehensive unit is also used to integrate the voice tonality change value PV and the language fluency value SF to obtain the voice comprehensive feature vector v int ;

[0298] A deep unit for synthesizing feature vector v from speech int Extract the feature representation related to emotion and obtain the deep feature value h;

[0299] A model unit is used to map the deep feature value h to the emotion category using a deep feature transformation network (DFTN) to obtain the emotion classification p; wherein DFTN is a deep learning network for static data analysis, which can learn more complex emotion patterns from the dynamic emotion features extracted by LSTM, and the network focuses on identifying key emotion drivers, such as voice tonality changes, language fluency, etc.; for example, a multi-layer perceptron (MLP) is used to extract emotion-related high-level representations from complex features;

[0300] Among them, the tonality change value of the voice is:

[0301]

[0302] The language fluency values ​​are:

[0303]

[0304] The comprehensive feature vector of speech is:

[0305]

[0306] The depth feature value is:

[0307] h=DFTN(v int )

[0308] Emotions are categorized as:

[0309] p = softmax(W h + b)

[0310] Where N is the total number of measurement time points, f i represents the fundamental frequency at time i, v i represents the speaking rate at time i, R s ,I s and S f are the speech rhythm value, speech intensity value and sound spectrum characteristic value respectively; W and b are the weight and bias of the fully connected layer, which are used to map the deep features to the emotion categories.

[0311] It should be noted that, in this embodiment, the long short-term memory network (LSTM) is mainly used to process continuous physiological and speech data to identify and track the time-dependent changes of emotional states;

[0312] The Deep Feature Transformation Network (DFTN) further extracts complex patterns related to emotions from the output of LSTM, performs static deep feature analysis, and finally outputs the precise probability distribution of emotion classification;

[0313] For the long short-term memory network (LSTM) and deep feature transformation network (DFTN), this division of labor allows the system to not only capture the dynamic changes of emotions, but also effectively identify key factors affecting emotions, providing an effective scientific basis for formulating training strategies.

[0314] This embodiment S3.1 can capture speech features associated with different emotional states by performing feature extraction in a comprehensive speech feature vector; and uses a deep transformation network (DFTN) to map the features to predefined emotion categories, so that the deep learning model can learn the complex relationship between deep feature values ​​and emotion categories, thereby ensuring the accuracy of the final emotion classification result. In addition, the output of DFTN is the probability distribution of emotion classification, which can provide a data basis for subsequent training adjustments and feedback generation.

[0315] The decision tree unit is used to extract more advanced, rich and abstract information carriers from the preset user original emotion features to obtain the advanced features h; and to establish the emotion category label Y containing several emotion categories;

[0316] Decision tree units are also used to obtain structural models;

[0317] The decision tree unit is also used to control the structural model to train the structural model by learning the relationship between the high-level feature h and the emotion category label Y, so that the structural model learns a series of "if-then" rules (for example, if the value of a certain feature is greater than a certain threshold, the sample is classified into a specific category) to obtain the decision tree Tree after training;

[0318] The decision tree unit is also used to control the decision tree Tree to explain how to infer emotions from features;

[0319] Among them, the construction formula of the decision tree is:

[0320] Tree = buildDecisionTree(h,Y)

[0321] Among them, buildDecisionTree is a function name, which represents the process of building a decision tree. The function of this function is to build a decision tree model based on the given input data.

[0322] The decision tree unit of this embodiment explains how to infer emotions from features by constructing a decision tree. Since the decision tree can visualize complex processing processes, users can clearly see the entire process from input data to output emotions. It can not only provide intuitive and visual explanations, but also help people understand the key factors that affect emotions and improve model performance.

[0323] In one embodiment, the state module 40 is specifically:

[0324] By combining the probability distribution p of the emotion t1 , the probability distribution of emotion classification p t2 , emotion classification p and preset user historical emotion data, evaluate the user's current emotional state and obtain the user's emotion value.

[0325] In one embodiment, the task module 50 includes an emotion unit, an anxiety unit, a stability unit and a training unit; wherein the emotion unit, the anxiety unit and the stability unit are the process of generating an eloquence communication training program, and the training unit is the process of conducting eloquence communication training for users, specifically:

[0326] The emotion unit is used to know the user's emotional state based on the user's emotional value, identify the dominant emotion in the emotional state through a preset algorithm, and analyze its potential impact;

[0327] Anxiety unit, used to reduce the preset training difficulty standard value through a preset emotional decision model if the emotional state shows that the user is currently in an anxious state, and provide more positive motivation feedback, and generate an eloquence communication training plan for the user according to the reduced training difficulty standard value;

[0328] The stabilization unit is used to increase the standard value of training difficulty through the emotional decision model if the emotional state shows that the user is currently in a stable state, and generate an eloquence communication training plan for the user according to the increased standard value of training difficulty.

[0329] For example, if the user is in a state of high anxiety, then build some relaxation training or meditation eloquence and communication training programs; if the user is in an emotionally stable state, then build a more challenging eloquence and communication training program.

[0330] Among them, the emotional decision-making model dynamically adjusts the training parameters based on the emotional probability distribution contained in the emotional state and the user's historical training performance; specifically: first, define the influence weights of different emotional states, and then analyze these weights through decision support algorithms such as logistic regression to generate a standard value for training difficulty; then use this standard value to guide the specific adjustment of training difficulty, duration and frequency to ensure that the training plan is both personalized and efficient, thereby generating the user's eloquence and communication training plan.

[0331] The construction process of the emotional decision-making model is as follows:

[0332] 1) Convert sentiment probability into decision-making indicators:

[0333] First, define the training adjustment parameters corresponding to each emotional state. For example, high anxiety may reduce the difficulty, and high pleasure may increase the difficulty;

[0334] Then, a weight is assigned to each emotional state, where the weight is calculated based on the emotional probability distribution and represents the influence of the emotional state on the training program.

[0335] 2) Construction of emotional decision-making model:

[0336] Use decision trees, logistic regression, or neural networks to analyze the weights of various emotions and combine them with the user's historical training data to determine the training strategy that best suits the current emotional state;

[0337] The model calculates a "training intensity index" (i.e., the standard value of training difficulty), which determines the adjustment of training difficulty, duration, frequency, etc.

[0338] According to the training strategy, the initial emotional decision-making model is generated in combination with the training intensity index and the training adjustment parameters.

[0339] 3) Model training and implementation:

[0340] The initial emotional decision-making model is trained through historical data to ensure that the model can understand the most effective training methods under different emotional states and obtain the final emotional decision-making model.

[0341] In the emotion unit, anxiety unit and stability unit of this embodiment, for users in an anxious state, reducing the training difficulty helps to relieve their stress and avoid further emotional burden; while for users with stable emotions, increasing the difficulty can challenge their abilities and promote further improvement of skills. This adaptive eloquence communication training program helps to provide users with more appropriate and efficient training content, thereby improving learning effects;

[0342] Furthermore, in order to accurately establish an eloquence communication training program, this embodiment introduces an emotional decision model, which dynamically adjusts training parameters based on the emotional probability distribution and the user's historical training performance, thereby ensuring that the training program is both personalized and efficient.

[0343] The practical training unit is used to train users in eloquence and communication using the eloquence and communication training program;

[0344] The training unit is also used to adjust the training intensity and type of the eloquence communication training program in real time by continuously monitoring the changes in the user's emotional state during the user's training, and collect the user's real-time emotional data;

[0345] The training unit is also used to update the user's profile through the collected real-time emotion data after the user completes the training task;

[0346] Furthermore, by dynamically adjusting the algorithm, the training content of the eloquence and communication training program is continuously optimized based on the user’s real-time emotional data, ensuring that the training can challenge the user without causing undue stress.

[0347] In the practical training unit of this embodiment, this instant feedback mechanism helps users manage and adjust their emotions more effectively. This cycle ensures that the training content is continuously optimized and personalized over time, gradually improving the user's ability in emotion management and eloquence.

[0348] The embodiment of the present invention also includes S6, visually displaying the real-time emotion data.

[0349] The steps of the embodiment of the present invention include S6.1 to S6.2; wherein the decoding unit is a process of establishing visualization data, the format unit is a process of constructing emotion visualization data, and the interaction unit is a process of performing user interaction, specifically:

[0350] The decoding unit is used to decode and standardize the real-time emotion data, as well as perform time synchronization and format unification processing to obtain the parsed data p t3 ;

[0351] The decoding unit is also used to cache the parsed data according to the preset cache mechanism according to the data access frequency and the real-time requirements of the user. t3 ;

[0352] The decoding unit is also used to parse the data p t3 Preprocessing is performed, including data cleaning (removing outliers or correcting erroneous data), data aggregation (such as averaging to smooth out emotional changes), etc., to obtain visual data v vis ;

[0353] The visualization data is:

[0354] v vis =aggregate({p t3-k →p t3 +)

[0355] Here, aggregate is a function that aggregates data within a certain time window into a set of average data for visualization, and k is the length or size of the time window.

[0356] The format unit is used to control the preset visualization engine to obtain visualization data v in a structured format (such as a JSON object containing a timestamp and the corresponding sentiment analysis results). vis , obtain the first data Data; wherein the first data Data can be expressed as Data = {'timestamp':t,'emotions':p t3 +;

[0357] Among them, 'timestamp' represents the timestamp of the data item, 'emotions' represents the emotional information of the data item, t represents the timestamp data, and p t3 It is parsing data.

[0358] The format unit is further used to remove redundant information in the first data Data, and adjust the format of the first data Data according to visualization requirements (such as converting the emotion probability into an input format required by the visualization library) to obtain second data;

[0359] The format unit is further used to generate a number of corresponding graphic representations according to the second data to obtain emotional visualization data; wherein the number of graphic representations include a curve of emotional state change over time, a heat map of emotional intensity, etc.;

[0360] Among them, the drawing operation can be expressed as:

[0361] Graph(time,p tn )=Plot(time series of p tn )

[0362] Among them, time is a sequence representing time points, p tnPlot (time series of p tn ) means to convert p tn Plot the time series data into a graph.

[0363] An interactive unit for integrating emotion visualization data into augmented reality or virtual reality (AR / VR) environments;

[0364] The interactive unit is also used to create a user interactive interface in an AR or VR environment, such as displaying emotion changes through a head-mounted display, allowing the user to explore the emotion data through gestures or viewing different perspectives, and allowing the user to interact with the emotion visualization data by touching the screen, voice commands, or mobile device operations;

[0365] The interaction unit is further used to adjust the visual output according to the user's interaction and update the display content of the user interaction interface in real time in response to the user input;

[0366] The interaction unit is also used to ensure efficient processing and display of real-time data streams by monitoring the performance of the visualization engine when the user interaction interface is running, and to improve the response speed and processing capability of the user interaction interface by optimizing data processing and visualization algorithms based on performance feedback (for example, through asynchronous data loading, memory management optimization, etc.).

[0367] The interactive unit of this embodiment uses AR / VR technology to not only provide emotion recognition training, but also create an immersive environment in which users can practice eloquence. This immersive experience can greatly improve user participation and the effect of emotion learning;

[0368] In addition, natural interaction methods such as touch screens can reduce the user's burden on technical operations, allowing them to focus more on training content and ensuring that all users can quickly get started and effectively use the user interaction interface.

[0369] Overall, this embodiment has the following beneficial effects:

[0370] The embodiment of the present invention identifies the internal physiological state of physiological activity data and obtains the probability distribution of emotions, which can intuitively reflect the current user's psychological situation and help understand the user's emotional state from the perspective of internal psychological activities; by identifying the external manifestations of physiological activity data, it can capture the external signs and movement changes related to emotions, and obtain the probability distribution of emotion classification, which helps to understand the user's emotional state from the perspective of external manifestations. Mapping the deep features of physiological activity data to predefined emotion categories is a further analysis of physiological activity data, which can intuitively and effectively obtain the user's emotion category. According to these three types of data, the user's current emotional state can be evaluated accurately by utilizing the deep insight characteristics of the probability distribution of emotions, the intuitive behavior performance of the probability distribution of emotion classification, and the deep analysis characteristics of the emotion classification data, so as to ensure that the emotion value can effectively reflect the current user's learning state, and then generate an adaptive eloquence communication training program, providing users with more suitable and efficient training content, and greatly improving learning effects.

[0371] Furthermore, this embodiment provides comprehensive support for emotion and eloquence training through comprehensive multi-source data analysis, which is not limited to the improvement of eloquence skills, but also includes the cultivation of emotion management ability, which is essential for effective public speaking.

[0372] Embodiment three:

[0373] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for eloquence communication training based on EEG emotion recognition;

[0374] Wherein, the eloquence communication training method based on EEG emotion recognition, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0375] The above are preferred embodiments of the present invention. It should be noted that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for training eloquence and communication based on EEG emotion recognition, characterized in that: include: Obtaining physiological activity data of users during communication; Performing internal physiological state recognition and external manifestation recognition on the physiological activity data to obtain a probability distribution of emotions and a probability distribution of emotion classification respectively; Mapping the deep features of the physiological activity data to the emotion category to obtain the emotion classification; specifically: calculating the speech comprehensive feature vector according to the physiological activity data; extracting the feature representation related to the emotion from the speech comprehensive feature vector to obtain the deep feature value; Mapping the deep feature values ​​to emotion categories using a deep learning model to obtain the emotion classification; Evaluate the current emotional state of the user according to the probability distribution of the emotion, the probability distribution of the emotion classification and the emotion classification to obtain the emotion value of the user; Generating an eloquence communication training program for the user according to the emotion value of the user; Wherein, the speech comprehensive feature vector is: v int =[R s ,I s ,S f ,PV,SF] T ; For the parameters PV and SF: Where N is the total number of measurement time points, f i represents the fundamental frequency at time i, v i represents the speaking rate at time i, R s ,I s and S f They are speech rhythm value, speech intensity value and sound spectrum characteristic value, PV is the speech tonality change value, and SF is the language fluency value.

2. The method for eloquence communication training based on EEG emotion recognition as claimed in claim 1, characterized in that: The physiological activity data is subjected to internal physiological state recognition to obtain the probability distribution of emotions, specifically: Extracting features from the physiological activity data to obtain a feature graph; According to the feature graph, the dynamic changes of emotions during communication are captured through a long short-term memory network to obtain a dynamic change value of emotions; The dynamic change value of the emotion is mapped to the emotion category to obtain the probability distribution of the emotion.

3. The method for eloquence communication training based on EEG emotion recognition as claimed in claim 2, characterized in that: Feature extraction is performed on the physiological activity data to obtain a feature map, specifically: Extracting the rhythm and intensity of the speech and the sound spectrum characteristics from the physiological activity data to obtain a speech rhythm value, a speech intensity value and a sound spectrum characteristic value respectively; Calculate a feature vector based on the speech rhythm value, the speech intensity value and the sound spectrum characteristic value; According to the feature vector, local spatiotemporal features related to emotions are extracted through a convolutional neural network to obtain the feature map.

4. The method for eloquence communication training based on EEG emotion recognition as claimed in claim 1, characterized in that: The external manifestation of the physiological activity data is identified to obtain the probability distribution of emotion classification, which is specifically: Extracting features from the physiological activity data to obtain speech features; The speech features are converted into emotion representation values ​​by capturing the emotional flow and long-term semantic dependencies between data; The emotional state is classified and calculated according to the emotional representation value to obtain the probability distribution of the emotional classification.

5. The method for eloquence communication training based on EEG emotion recognition as claimed in claim 4, characterized in that: Feature extraction is performed on the physiological activity data to obtain speech features, specifically: Calculating the dynamic range of the speech data in the physiological activity data to obtain the standard deviation of the speech amplitude; Calculating the speech speed change of the voice data in the physiological activity data to obtain a rhythm variation value of the voice; Calculating the frequency spectrum discreteness of the voice data in the physiological activity data to obtain the sound spectrum entropy; The speech feature is composed of the standard deviation of the speech amplitude, the rhythm variation value of the speech and the sound spectrum entropy.

6. The method for eloquence communication training based on EEG emotion recognition as claimed in claim 4, characterized in that: The emotional state classification calculation is performed according to the emotional representation value to obtain the probability distribution of the emotional classification, which is specifically: Using a preset activation function, the emotion representation value is converted into an activation value of an output layer in a deep learning model by means of linear transformation and nonlinear activation; According to the activation value, the probability distribution of the emotion classification is generated by the Softmax function of the deep learning model.

7. The method for eloquence communication training based on EEG emotion recognition as claimed in claim 1, characterized in that: According to the emotion value of the user, an eloquence communication training program for the user is generated, specifically: Knowing the emotional state of the user according to the emotional value of the user; If the emotional state shows that the user is currently in an anxious state, then the preset training difficulty standard value is reduced, and an eloquence communication training program for the user is generated according to the reduced training difficulty standard value; If the emotional state shows that the user is currently in a stable state, the training difficulty standard value is increased, and an eloquence communication training program for the user is generated according to the increased training difficulty standard value.

8. An eloquence communication training device based on EEG emotion recognition, characterized in that: It includes data module, emotion module, mapping module, state module and task module; Wherein, the data module is used to obtain the physiological activity data of the user during communication; The emotion module is used to identify the internal physiological state and external manifestation of the physiological activity data, and obtain the probability distribution of emotions and the probability distribution of emotion classification respectively; The mapping module is used to map the deep features of the physiological activity data to the emotion category to obtain the emotion classification; specifically: calculate the speech comprehensive feature vector according to the physiological activity data; extract the feature representation related to the emotion from the speech comprehensive feature vector to obtain the deep feature value; use the deep learning model to map the deep feature value to the emotion category to obtain the emotion classification; The state module is used to evaluate the current emotional state of the user according to the probability distribution of the emotion, the probability distribution of the emotion classification and the emotion classification to obtain the emotion value of the user; The task module is used to generate an eloquence communication training program for the user according to the emotion value of the user; Wherein, the speech comprehensive feature vector is: v int =[R s ,I s ,S f ,PV,SF] T ; For the parameters PV and SF: Where N is the total number of measurement time points, f i represents the fundamental frequency at time i, v i represents the speaking rate at time i, R s ,I s and S f They are speech rhythm value, speech intensity value and sound spectrum characteristic value, PV is the speech tonality change value, and SF is the language fluency value.

9. The eloquence communication training device based on EEG emotion recognition as claimed in claim 8, characterized in that: The emotion module includes an extraction unit, a capture unit and a mapping unit; Wherein, the extraction unit is used to extract features from the physiological activity data to obtain a feature map; The capturing unit is used to capture the dynamic changes of emotions during communication through a long short-term memory network according to the feature graph to obtain a dynamic change value of emotions; The mapping unit is used to map the emotion dynamic change value to the emotion category to obtain the probability distribution of the emotion.

10. The eloquence communication training device based on EEG emotion recognition as claimed in claim 9, characterized in that: The extraction unit includes a speech subunit, a vector subunit and a feature map subunit; The speech subunit is used to extract the rhythm and intensity of the speech in the physiological activity data, as well as the sound spectrum characteristics, and obtain the speech rhythm value, the speech intensity value and the sound spectrum characteristic value respectively; The vector subunit is used to calculate the feature vector according to the speech rhythm value, the speech intensity value and the sound spectrum characteristic value; The feature map subunit is used to extract local spatiotemporal features related to emotions through a convolutional neural network according to the feature vector to obtain the feature map.

11. The eloquence communication training device based on EEG emotion recognition according to claim 8, characterized in that: The emotion module includes an activity unit, a relationship unit and a classification unit; Wherein, the activity unit is used to extract features from the physiological activity data to obtain speech features; The relationship unit is used to convert the speech features into emotion representation values ​​by capturing the emotion flow and long-term semantic dependency between data; The classification unit is used to perform classification calculation of the emotional state according to the emotion representation value to obtain the probability distribution of the emotion classification.

12. The eloquence communication training device based on EEG emotion recognition according to claim 11, characterized in that: The activity unit includes an amplitude subunit, a variation subunit, a spectrum subunit and a feature subunit; Wherein, the amplitude subunit is used to calculate the dynamic range of the voice data in the physiological activity data to obtain the standard deviation of the voice amplitude; The variation subunit is used to calculate the speech rate variation of the speech data in the physiological activity data to obtain the rhythm variation value of the speech; The spectrum subunit is used to calculate the spectrum discreteness of the voice data in the physiological activity data to obtain the sound spectrum entropy; The feature subunit is used to form the speech feature from the standard deviation of the speech amplitude, the rhythm variation value of the speech and the sound spectrum entropy.

13. The eloquence communication training device based on EEG emotion recognition according to claim 11, characterized in that: The classification unit includes an activation subunit and a classification subunit; The activation subunit is used to use a preset activation function to convert the emotion representation value into an activation value of an output layer in a deep learning model by means of linear transformation and nonlinear activation; The classification subunit is used to generate the probability distribution of the emotion classification according to the activation value through the Softmax function of the deep learning model.

14. The eloquence communication training device based on EEG emotion recognition according to claim 8, characterized in that: The task module includes an emotion unit, an anxiety unit and a stability unit; Wherein, the emotion unit is used to know the emotional state of the user according to the emotional value of the user; The anxiety unit is configured to reduce a preset training difficulty standard value if the emotional state shows that the user is currently in an anxious state, and generate an eloquence communication training program for the user according to the reduced training difficulty standard value; The stabilization unit is used to increase the training difficulty standard value if the emotional state shows that the user is currently in a stable state, and generate an eloquence communication training program for the user according to the increased training difficulty standard value.

15. A storage medium, characterized in that: The storage medium stores a computer program, which is called and executed by a computer to implement any one of the eloquence communication training methods based on EEG emotion recognition as claimed in claims 1 to 7.

Images