An emotion evaluation method and device based on deep time series modeling

Abstract

This application discloses an emotion assessment method and apparatus based on deep temporal modeling. The method includes: real-time acquisition of speech data and EEG signals of a target object; conversion of the acquired speech data into an audio sequence and the EEG signals into an EEG sequence according to a preset time interval; inputting the audio sequence into a first sub-network of an emotion extractor to obtain speech emotion results; inputting the EEG sequence into a second sub-network of the emotion extractor to obtain brain activity emotion results; evaluating the emotional state of the target object based on the speech emotion results and brain activity emotion results at different times to obtain multiple emotion scores; fusing the multiple emotion scores of the target object within a target time period to obtain a comprehensive emotion score evaluation; and determining the emotional state type of the target object according to a preset emotion score scale.

Description

Technical Field

[0001] This application relates to the field of pattern recognition technology, and more specifically, to a method and apparatus for emotion assessment based on deep temporal modeling. Background Technology

[0002] Negative emotions related to learning, such as anxiety, frustration, and aversion to learning, have a significant negative impact on learners' academic performance and mental health. However, traditional classroom teaching methods often fail to provide timely attention to the emotional state of each learner, resulting in some learners lacking effective guidance and support when facing learning pressure.

[0003] In recent years, with the development of emotion computing technology, it has become possible to identify learners' emotional states by analyzing their physiological signals (such as brainwaves and heart rate) and behavioral signals (such as facial expressions and speech). However, the application of existing emotion computing technology in education still has some limitations. For example, most systems rely on only a single signal source (such as facial expressions or speech), which limits the accuracy and generalization of emotion recognition, hinders the effective fusion of multimodal data, and prevents the construction of efficient and accurate emotion assessment models. Therefore, there is a problem of not being able to obtain highly accurate emotion recognition results, thus preventing the provision of more suitable teaching guidance methods for learners based on the emotion recognition results. Summary of the Invention

[0004] To address at least one deficiency or improvement need in the existing technology, this invention provides an emotion assessment method and apparatus based on deep temporal modeling, which enables learners to receive more relevant teaching guidance based on emotion recognition results, thereby improving teaching efficiency and optimizing the learner's learning experience.

[0005] To achieve the above objectives, according to a first aspect of the present invention, an emotion assessment method based on deep temporal modeling is provided. The method includes: real-time acquisition of speech data and EEG signals of a target object; conversion of the acquired speech data into an audio sequence and the EEG signals into an EEG sequence according to a preset time interval; inputting the audio sequence into a first sub-network of an emotion extractor to obtain speech emotion results; inputting the EEG sequence into a second sub-network of the emotion extractor to obtain brain activity emotion results; evaluating the target object's emotional state based on the speech emotion results and brain activity emotion results at different times to obtain multiple emotion scores; fusing the multiple emotion scores of the target object within a target time period to obtain a comprehensive emotion score evaluation; and determining the target object's emotional state type according to a preset emotion score scale.

[0006] In one exemplary embodiment, the first subnetwork includes an acoustic information processing branch employing a time-aware bidirectional multi-scale network model and a text information processing branch employing a large language model; the second subnetwork employs an ASTR brainwave emotion recognition model.

[0007] In an exemplary embodiment, the step of inputting the audio sequence into the first sub-network of the emotion extractor to obtain the speech emotion result includes: converting the audio sequence into a linear spectrogram through a short-time Fourier transform; applying a Mel filter bank to the linear spectrogram to obtain a spectrogram; converting the frequency axis of the spectrogram into a Mel scale; performing a logarithmic transform on the output of each Mel filter to obtain a Mel spectrogram; and performing a discrete cosine transform on the Mel spectrogram to obtain Mel frequency cepstral coefficient features.

[0008] In an exemplary embodiment, after performing a discrete cosine transform on the Mel frequency cepstral coefficient feature to obtain the Mel frequency cepstral coefficient feature, the method further includes: inputting the Mel frequency cepstral coefficient feature into the acoustic information processing branch of the first sub-network to capture deep temporal information and obtain deep temporal features; applying a Sigmoid function to the deep temporal features to learn a time attention map; and performing element-wise multiplication between the time attention map and the input feature map to generate time-aware features.

[0009] In an exemplary embodiment, after generating the time-aware features, the method further includes: fusing the feature outputs of feedforward and feedback time attention blocks located at the same timestamp to obtain a representation vector of the target receptive field; using weighted summation to fuse features from time attention blocks at different timestamps, and outputting a predicted classification result by passing the weighted output feature sequence through a fully connected layer and a softmax function; inputting the transcribed text into the large language model of the first sub-network, and outputting the speech emotion result for the speech text according to a preset evaluation template.

[0010] In an exemplary embodiment, the step of inputting the EEG sequence into the second sub-network of the emotion extractor to obtain the emotional results of brain activity includes: mapping the distribution of EEG signals to a spatial matrix according to the spatial distribution of electrodes; and extracting multi-granularity spatial features of the EEG signals through a cascaded attention mechanism.

[0011] In an exemplary embodiment, after extracting multi-granular spatial features from the EEG signal through the cascaded attention mechanism, the method further includes: inputting the spatial features of the time segment into a unidirectional self-attention module, extracting time change features, and generating a time-space representation vector; and passing the time-space representation vector to a classifier to obtain a brain activity emotion score.

[0012] According to a second aspect of the present invention, an emotion assessment device based on deep temporal modeling is also provided, comprising: a acquisition unit for real-time acquisition of speech data and EEG signals of a target object, converting the acquired speech data into an audio sequence and the EEG signals into an EEG sequence according to a preset time interval; an acquisition unit for inputting the audio sequence into a first subnetwork of an emotion extractor to acquire speech emotion results, and inputting the EEG sequence into a second subnetwork of an emotion extractor to acquire brain activity emotion results; an evaluation unit for evaluating the emotional state of the target object based on the speech emotion results and brain activity emotion results of the target object at different times, and obtaining multiple emotion scores; and a determination unit for fusing the multiple emotion scores of the target object within a target time period to obtain a comprehensive emotion score evaluation, and determining the emotional state type of the target object according to a preset emotion score scale.

[0013] According to a third aspect of the invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the above-described emotion assessment method based on deep temporal modeling at runtime.

[0014] According to a fourth aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the aforementioned emotion assessment method based on deep temporal modeling through the computer program.

[0015] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0016] This invention provides an emotion assessment method based on deep temporal modeling. By integrating multiple data sources such as audio signals and EEG signals, and combining the emotion recognition and text generation capabilities of a large language model, it achieves real-time and accurate assessment and judgment of learners' negative emotional states. Based on the emotion recognition results, it provides learners with more suitable teaching guidance methods, which solves the shortcomings of existing technologies in emotion computing and psychological intervention in the field of education, improves teaching efficiency, optimizes the educational process, and enhances learners' learning experience and emotional development. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1A flowchart illustrating an optional emotion assessment method based on deep temporal modeling provided in this application embodiment;

[0019] Figure 2 A flowchart illustrating another optional emotion assessment method based on deep temporal modeling provided for embodiments of this application;

[0020] Figure 3 A schematic diagram illustrating an optional application scenario of a voice sensor and EEG acquisition device provided in this application embodiment;

[0021] Figure 4 A schematic diagram of the network structure of an optional emotion extractor provided in an embodiment of this application;

[0022] Figure 5 A schematic diagram of the structure of an optional emotion assessment device based on deep temporal modeling provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of an optional electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0026] According to one aspect of the embodiments of this application, a sentiment assessment method based on deep temporal modeling is provided. The following is in conjunction with... Figure 1 This application describes a sentiment assessment method based on deep temporal modeling provided in its embodiments.

[0027] Figure 1 This is a flowchart illustrating an optional emotion assessment method based on deep temporal modeling provided in an embodiment of this application, as shown below. Figure 1As shown, the process of this method may include the following steps:

[0028] S102, real-time acquisition of speech data and EEG signals of the target object, and conversion of the acquired speech data into audio sequences and EEG signals into EEG sequences according to preset time intervals;

[0029] S104, input the audio sequence into the first sub-network of the emotion extractor to obtain the speech emotion result, and input the EEG sequence into the second sub-network of the emotion extractor to obtain the brain activity emotion result;

[0030] S106: The target's emotional state is evaluated based on the target's voice emotion results and brain activity emotion results at different times, and multiple emotion scores are obtained.

[0031] S108 integrates multiple emotion scores of the target object within the target time period to obtain a comprehensive emotion score assessment, and determines the emotional state type of the target object based on a preset emotion score scale.

[0032] This application provides a deep temporal modeling-based emotion assessment method applicable to educational scenarios. Through intelligent emotion analysis, it optimizes learners' learning experience and emotional development.

[0033] For example, a high-sensitivity voice sensor located on a desk can be used to acquire voice data resources of a target object, such as a learner during class or group discussion, in real time, while the EpocX portable wireless EEG device can be used to collect the learner's EEG signals in real time.

[0034] Specifically, in combination Figure 2 and Figure 3 As shown, audio data and EEG signal data are captured during class time using a voice sensor fixed to the front of the desk and the EpocX portable wireless EEG acquisition device worn on the learner's head.

[0035] The learner's speech signal data and EEG signal data were preprocessed separately. The collected data were then converted into audio sequences according to preset time intervals. and EEG sequences The input learner audio data is pre-emphasized to attenuate the low-frequency components of the signal while preserving or amplifying the high-frequency components. The specific calculation is as follows:

[0036]

[0037] in, This is the pre-emphasized audio signal. The original audio signal. It is the pre-emphasis coefficient, with a value between 0.9 and 1.

[0038] The pre-emphasized audio signal is divided into frames, and the frames are extracted. Long voice signals, each time moving The length continues until the entire speech signal is segmented.

[0039] After framing, a windowing operation is introduced, and the Hamming window is chosen as the window function, which can be calculated as follows:

[0040]

[0041]

[0042] in, The length of the window, This is a windowed audio signal.

[0043] The learner's audio data is transcribed into text using the SenseVoice-Small automatic speech recognition model, and the transcribed text is compiled. The learner's EEG signal data is denoised and despoofed using a bandpass filter with a preset bandpass frequency range of 0.5-70Hz and channel normalization, resulting in denoised EEG signal data with the signal limited to between 0 and 1.

[0044] The baseline removal algorithm is used to eliminate irrelevant environmental components in the EEG signal and divide the signal into baseline time segments. and emotional time segments ,in It is the number of baseline segments. It is the total number of segments. It refers to the number of channels in the electroencephalogram (EEG) signal. This refers to the length of the segment. Since the baseline segment and the emotional segment have the same time segment format, baseline removal is accomplished by subtracting the average baseline signal from the emotional signal, calculated as follows:

[0045]

[0046] in, This indicates the signal after removing the baseline;

[0047] The preprocessed audio data and multi-channel EEG signals were divided into time segments using a fixed segmentation interval. Specifically, for multichannel EEG signals, each time segment contains One channel and Each sampling point is represented as a matrix. The average value of the signal for each channel is calculated to form a channel mean vector for the time segment. .

[0048] The preprocessed learner multimodal signal data is input into the trained negative emotion extractor. Audio sequences are input into the first sub-network of the negative emotion extractor: the Bi-directional Temporal Adaptive Mixture Network (Bi-TAMN), which combines acoustic processing and text processing branches to obtain the learned speech emotion results. EEG sequences are input into the second sub-network of the negative emotion extractor: the Adaptive Spatio-Temporal Representation Learning Model (ASTR), an EEG emotion recognition model to obtain the learner's brain activity emotion results.

[0049] The learners' negative emotional states during learning are assessed in stages based on the recognition results of their speech and EEG signals at different times. The negative emotional scores from these stages are integrated to obtain a comprehensive assessment of the negative emotional state throughout the entire classroom process. The learners' emotional state types are determined based on a pre-designed scale of negative emotion scores.

[0050] Through steps S102 to S108, real-time acquisition of the target object's speech data and EEG signals is performed. Based on preset time intervals, the acquired speech data is converted into audio sequences, and the EEG signals are converted into EEG sequences. The audio sequences are input into the first sub-network of the emotion extractor to obtain speech emotion results, and the EEG sequences are input into the second sub-network of the emotion extractor to obtain brain activity emotion results. The target object's emotional state is evaluated based on the speech emotion results and brain activity emotion results at different times, resulting in multiple emotion scores. These multiple emotion scores within the target time period are then integrated to obtain a comprehensive emotion score evaluation. The target object's emotional state type is determined based on a preset emotion score scale. This process solves the problem of not being able to obtain highly accurate emotion recognition results, thus hindering the provision of more suitable teaching guidance methods for learners based on emotion recognition results. This improves teaching efficiency and optimizes the learner's learning experience.

[0051] In one exemplary embodiment, the first subnetwork includes an acoustic information processing branch employing a time-aware bidirectional multi-scale network model and a text information processing branch employing a large language model; the second subnetwork employs an ASTR brainwave emotion recognition model.

[0052] In one exemplary embodiment, the step of inputting the audio sequence into the first sub-network of the emotion extractor to obtain the speech emotion result includes:

[0053] S11, convert the audio sequence into a linear spectrogram through a short-time Fourier transform, apply a Mel filter bank to the linear spectrogram to obtain a spectrum, and convert the frequency axis of the spectrum into a Mel scale;

[0054] S12, perform a logarithmic transformation on the output of each Mel filter to obtain the Mel spectrum;

[0055] S13, Perform discrete cosine transform on the Mel frequency spectrum to obtain the Mel frequency cepstral coefficient characteristics.

[0056] like Figure 4 As shown, the audio sequence The spectrum is converted to a linear spectrogram via Short-Time Fourier Transform (STFT), and the calculation method is as follows:

[0057]

[0058] in, This indicates the time offset, where ω is the angular frequency index.

[0059] Applying a Mel filter bank to the linear spectrogram converts the frequency axis of the spectrogram to a Mel scale. The filter calculation expression is as follows:

[0060]

[0061] in, Represented as a certain sampling point The frequency of, among which Step size, The frame rate of the audio frame.

[0062] Then, a logarithmic transform is performed on the output of each Mel filter to obtain the Mel spectrum. A discrete cosine transform (DCT) is then performed on the obtained Mel spectrum to obtain the MFCC features. The specific calculation process is as follows:

[0063]

[0064] in, It is the logarithmic power value of the Mel band.

[0065] In an exemplary embodiment, after performing a discrete cosine transform on the Mel frequency cepstral coefficients to obtain the Mel frequency cepstral coefficient features, the method further includes:

[0066] S21, the Mel frequency cepstral coefficient features are input into the acoustic information processing branch of the first sub-network for deep temporal information capture to obtain deep temporal features;

[0067] S22, apply the Sigmoid function to deep temporal features to learn a temporal attention map;

[0068] S23, perform element-wise multiplication between the time attention map and the input feature map to generate time-aware features.

[0069] In this embodiment, the extracted 39-dimensional MFCC features The input is fed into the temporal awareness module (TAB) in the acoustic feature extraction branch BI-TAMN of the speech emotion recognition module to capture deep temporal information of the speech. The specific steps are as follows:

[0070] For feature maps Perform dilated causal convolution operation, with the dilation rate increasing exponentially. The formula is as follows:

[0071]

[0072] in, For the first The output feature map of the dilated convolution in each tab To expand the causal convolution kernel, The kernel size is [size]. For expansion rate, and These are the indices of the convolutional kernel in width and height, respectively;

[0073] For feature maps Perform batch normalization operations. The companies are as follows:

[0074]

[0075] in, and These are the batch mean and variance. and These are learnable parameters.

[0076] By introducing the ReLU activation function and performing spatial dropout, the speech deep temporal features are obtained. ;

[0077] Features Applying the Sigmoid function Learning time attention map , means as follows:

[0078]

[0079] Next, the attention map With input feature map Perform element-wise multiplication to generate time-aware features. Specifically:

[0080]

[0081] in, No. Input feature map of TAB.

[0082] In one exemplary embodiment, after generating the time-aware features, the method further includes:

[0083] S31, the feature outputs of the feedforward and feedback time attention blocks located at the same time stamp are fused to obtain the representation vector of the target receptive field;

[0084] S32 uses weighted summation to fuse features from attention blocks at different timestamps. The weighted output feature sequence is then passed through a fully connected layer and a softmax function to output the predicted classification result.

[0085] S33: Input the transcribed text into the large language model of the first sub-network, and output the speech emotion result for the speech text according to the preset evaluation template.

[0086] In this embodiment, the feature outputs of the feedforward and feedback TAB blocks located at the same timestamp are fused for feature fusion, as follows:

[0087]

[0088] in, This indicates the characteristics of a feedforward network. Representing the characteristics of a feedforward network, global time pooling operation Taking the average over the time dimension, the final result is from the [number]th [year]. A representation vector for a specific receptive field is obtained from each Temporal Attention Block (TAB);

[0089] A weighted summation operation is used to fuse features from different timestamp tabs. The weighted summation operation uses a dynamic receptive field (DRF) to fuse the weights. The definition of DRF fusion is as follows:

[0090]

[0091] in, These are trainable parameters.

[0092] The weighted output feature sequence The predicted sentiment probability is calculated using a fully connected layer, and this process can be represented as:

[0093]

[0094] in, This represents a fully connected layer. The output of the final fully connected layer is then subjected to a softmax operation to predict the sentiment. The specific operation process is as follows:

[0095]

[0096] in, This is represented as the prediction result being the first... The probability of each category is used to select the category with the highest predicted probability as the final classification result.

[0097] The transcribed text generated from learner audio data using speech recognition technology is input into the text feature processing branch of the speech emotion recognition module. The transcribed text is then input into the GPT4 large language model, and evaluated according to a pre-set evaluation template: {'Topic Relevance': 'Whether the evaluation content closely relates to the learning state'; 'Logicality': 'Whether the evaluation content is logically coherent'; 'Interactivity': 'Whether the learner participates in asking questions, discussions, and providing feedback'; 'Accuracy of Expression': 'Whether the evaluation content is accurately expressed and whether the word choice is appropriate'}}, outputting the evaluation results for the speech text.

[0098] In one exemplary embodiment, inputting the EEG sequence into the second subnetwork of the emotion extractor to obtain the emotional results of brain activity includes:

[0099] S41, based on the spatial distribution of the electrodes, the distribution of EEG signals is mapped to a spatial matrix;

[0100] S42 extracts multi-granularity spatial features from EEG signals through a cascaded attention mechanism.

[0101] In this embodiment, the preprocessed learner EEG sequence The data is input into the second subnetwork of the negative emotion extractor, the ASTR EEG emotion recognition model, to obtain the learner's brain activity and emotional results during the classroom:

[0102] Based on the spatial distribution of the electrodes, the channel averages from the time segments are filled into a 9×9 spatial matrix. The specific construction method is as follows: the top view of the brain is transformed into a matrix. Each position corresponds to the mean value of an electrode signal. The positions where no electrodes are placed are filled with 0s. The expression for the spatial matrix is:

[0103]

[0104] in, This represents the calculation of the mean value of the corresponding electrode signal. This spatial matrix contains the temporal correlation and spatial distribution between electrodes.

[0105] The spatial matrix was divided into six brain regions: the frontal lobe, left temporal lobe, right temporal lobe, parietal lobe, and occipital lobe. For the signal sequence of each brain region, its similarity to the overall brain signal sequence was calculated and quantified as a correlation feature. The Euclidean distance formula was used to quantify the correlation.

[0106]

[0107] in, Indicates brain regions The average signal sequence, This represents the signal sequence of the entire brain;

[0108] By employing channel attention mechanisms, activation weights for brain regions are generated based on the similarity between signals from those regions and the overall brain signal. The calculation formula is as follows:

[0109]

[0110] in, and These are the weights of the fully connected layer. It is a bias term. The hyperbolic tangent activation function is used. for An activation function is used to normalize the weights. Calculate the Euclidean distance between the signal sequences of each brain region and the overall brain signal to generate a similarity vector. This represents the activation weights of brain regions generated by channel attention;

[0111] The information exchange patterns between learners' brain regions are extracted as coarse-grained spatial features to represent the spatial interaction information of neural circuits.

[0112] The amount of information in a brain region at a given moment is calculated based on the signal values ​​of electrodes within that region. Weighted information entropy is defined. Indicates brain regions In time segment The information content is calculated using the following formula:

[0113]

[0114] in, Indicates brain regions The number of sampling electrodes in the middle, Indicates time segment Central Channel The signal value, This represents the amount of information contained in a brain region, and is constructed as a brain region information sequence by weighting the information entropy values ​​of all time segments. :

[0115]

[0116] The Dynamic Time Warping (DTW) algorithm is used to quantify the information exchange patterns between brain regions. For the temporal information sequences of two brain regions... and Calculate its DTW distance and define the adjacency matrix. The formula for calculating the intensity of information exchange between brain regions is as follows:

[0117]

[0118] in, It is an adjacency matrix The Line number The column elements represent the degree of information exchange between corresponding brain regions. Calculate the DTW distance.

[0119] The information interaction patterns are weighted using a channel attention mechanism, and the calculation formula is as follows:

[0120]

[0121] Multi-granular spatial weights are generated by using weighted information entropy, DTW to calculate weights, and cross-attention weights, and then combined with the original signal matrix. Calculate the multidimensional spatial activity matrix of neural circuits The formula is as follows:

[0122]

[0123] In one exemplary embodiment, after extracting multi-granularity spatial features from the EEG signal via a cascaded attention mechanism, the method further includes:

[0124] S51: Input the spatial features of the time segment into the unidirectional self-attention module, extract the temporal change features, and generate a time-space representation vector;

[0125] S52 passes the time-space representation vector to the classifier to obtain the brain activity emotion score.

[0126] A one-way Transformer is used to introduce temporal order into spatial features and extract temporal variation features from neural activity. The one-way Transformer encoder specifically quantifies the temporal dependencies between time segments, calculating only the correlation between the current time segment and previous time segments, and masking inverse dependencies.

[0127] Based on the similarity score, the intermediate sequence of each event segment is calculated, specifically as follows:

[0128]

[0129] Through the above steps, temporal and spatial features are fused to generate a temporal-spatial representation vector. The spatial activity matrix of each time segment is flattened into a vector, which is used as input to generate a short-term representation vector containing emotion-related temporal-spatial neural activity features.

[0130] The final output of the last stage is passed to a classifier, which consists of two linear projections with an activation function in between, producing a final brain activity emotional state score. Based on the recognition results of learners' speech and EEG signals at different times, a phased assessment of learners' negative emotional states is conducted. By integrating the multimodal phased negative emotion classification results and brain activity scores, a comprehensive assessment of negative emotions throughout the entire classroom process is obtained.

[0131] For example, the calculation of the emotional state score of brain activity throughout the process:

[0132] By time interval The emotional scores for each segment of the learning state were obtained, and the calculation process is as follows:

[0133]

[0134] in, The emotional score representing the learning state of a segment. Valence of emotion It indicates the level of emotional arousal.

[0135] Throughout the entire learning process, every Calculate an emotional score for a learning state. All The emotional score of the entire learning state is obtained by summing the results according to certain weights. The calculation process is as follows:

[0136]

[0137] Finally obtained This refers to the brain activity and emotional state score throughout the entire learning process.

[0138] Based on the pre-defined negative emotion scale, the learners' types of negative emotional states were determined, as shown in Table 1:

[0139] Table 1

[0140]

[0141] Furthermore, based on emotion recognition results, more tailored teaching guidance can be provided to learners. For example, for those prone to distraction, diversified teaching methods can be implemented, with interactive sessions every 10-15 minutes; the learning environment can be optimized by arranging seating appropriately, placing easily distracted learners near the podium for easy teacher attention and reminders; attention training, such as meditation, can be conducted to improve students' concentration. For those experiencing anxiety and tension, psychological counseling and relaxation training can be provided; the difficulty and goals of learning tasks can be adjusted to reduce stress; opportunities for success can be created, encouraging learners to start with simple tasks to enhance their self-confidence.

[0142] For learners who dislike learning, we should explore their interests and connect the learning content with things they enjoy; set personalized goals, break down tasks, and provide rewards; pay attention to their learning process, and provide timely guidance and assistance to prevent them from developing a dislike for learning due to persistent difficulties. For learners who are passive thinkers, we should create an inclusive atmosphere, encourage expression and positive feedback; organize group cooperative learning so that passive thinkers can form groups with other students to complete learning tasks together; and provide personalized attention and individual communication after class.

[0143] By providing learners with more relevant teaching guidance based on emotion recognition results, teaching efficiency is improved while optimizing the learner's learning experience.

[0144] According to another aspect of the embodiments of this application, an evaluation apparatus for implementing the above-described emotion evaluation method based on deep temporal modeling is also provided. Figure 5 This is a schematic diagram of the structure of an optional emotion assessment device based on deep temporal modeling according to an embodiment of this application, as shown below. Figure 5 As shown, the device may include:

[0145] The acquisition unit 502 is used to acquire the speech data and EEG signals of the target object in real time, and convert the acquired speech data into audio sequences and the EEG signals into EEG sequences according to a preset time interval.

[0146] The acquisition unit 504 is used to input the audio sequence into the first sub-network of the emotion extractor to obtain the speech emotion result, and input the EEG sequence into the second sub-network of the emotion extractor to obtain the brain activity emotion result;

[0147] Evaluation unit 506 is used to evaluate the emotional state of the target object based on the voice emotion results and brain activity emotion results of the target object at different times, and obtain multiple emotion scores;

[0148] Unit 508 is used to integrate multiple emotion scores of the target object within the target time period to obtain a comprehensive emotion score evaluation, and to determine the emotion state type of the target object according to the preset emotion score table.

[0149] It should be noted that the acquisition unit 502 in this embodiment can be used to perform the above step S102, the acquisition unit 504 in this embodiment can be used to perform the above step S104, the evaluation unit 506 in this embodiment can be used to perform the above step S106, and the determination unit 508 in this embodiment can be used to perform the above step S108.

[0150] Through the aforementioned modules, real-time acquisition of speech data and EEG signals from the target subject is performed. Based on preset time intervals, the acquired speech data is converted into audio sequences, and the EEG signals are converted into EEG sequences. The audio sequences are input into the first sub-network of the emotion extractor to obtain speech emotion results, and the EEG sequences are input into the second sub-network to obtain brain activity emotion results. The target subject's emotional state is evaluated based on the speech and brain activity emotion results at different times, resulting in multiple emotion scores. These multiple emotion scores are then integrated within the target time period to obtain a comprehensive emotion score evaluation. The target subject's emotional state type is determined based on a preset emotion score scale. This approach solves the problem of obtaining highly accurate emotion recognition results, which hinders the provision of more suitable teaching guidance methods for learners. This improves teaching efficiency and optimizes the learner's learning experience.

[0151] In one exemplary embodiment, the first subnetwork includes an acoustic information processing branch employing a time-aware bidirectional multi-scale network model and a text information processing branch employing a large language model; the second subnetwork employs an ASTR brainwave emotion recognition model.

[0152] In one exemplary embodiment, the acquisition unit includes:

[0153] The conversion module is used to convert an audio sequence into a linear spectrogram through a short-time Fourier transform, apply a Mel filter bank to the linear spectrogram to obtain a spectrum, and convert the frequency axis of the spectrum into a Mel scale.

[0154] The logarithmic transform module is used to perform a logarithmic transform on the output of each Mel filter to obtain the Mel spectrum.

[0155] The discrete cosine transform module is used to perform a discrete cosine transform on the Mel spectrum to obtain the Mel frequency cepstral coefficient characteristics.

[0156] In one exemplary embodiment, the apparatus further includes:

[0157] The information capture unit is used to input the Mel frequency cepstral coefficient features into the acoustic information processing branch of the first sub-network to capture deep temporal information and obtain deep temporal features;

[0158] The learning unit is used to apply the Sigmoid function to deep temporal features and learn a temporal attention map;

[0159] The first generation unit is used to perform element-wise multiplication between the temporal attention map and the input feature map to generate time-aware features.

[0160] In one exemplary embodiment, the apparatus further includes:

[0161] The feature fusion unit is used to fuse the feature outputs of the feedforward and feedback time attention blocks located at the same time stamp to obtain the representation vector of the target receptive field.

[0162] The prediction unit is used to fuse features from attention blocks at different timestamps using weighted summation. The weighted feature sequence is then passed through a fully connected layer and a softmax function to output the prediction classification result.

[0163] The output unit is used to input the transcribed text into the large language model of the first sub-network and output the speech emotion result for the speech text according to the preset evaluation template.

[0164] In one exemplary embodiment, the acquisition unit includes:

[0165] The mapping module is used to map the distribution of EEG signals to a spatial matrix based on the spatial distribution of the electrodes;

[0166] The extraction module is used to extract multi-granularity spatial features from EEG signals through a cascaded attention mechanism.

[0167] In one exemplary embodiment, the apparatus further includes:

[0168] The second generation unit is used to input the spatial features of the time segment into the unidirectional self-attention module, extract the temporal change features, and generate a time-space representation vector.

[0169] The transmission unit is used to pass the time-space representation vector to the classifier to obtain the brain activity emotion score. It should be noted that the examples and scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in a hardware environment, and can be implemented in software or hardware, wherein the hardware environment includes a network environment.

[0170] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above-described deep temporal modeling-based emotion assessment methods in the embodiments of this application.

[0171] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0172] S1, real-time acquisition of speech data and EEG signals of the target object, and conversion of the acquired speech data into audio sequences and EEG signals into EEG sequences according to preset time intervals;

[0173] S2, input the audio sequence into the first sub-network of the emotion extractor to obtain the speech emotion result, and input the EEG sequence into the second sub-network of the emotion extractor to obtain the brain activity emotion result;

[0174] S3 assesses the target's emotional state based on the target's vocal emotion results and brain activity emotion results at different times, and obtains multiple emotion scores;

[0175] S4 integrates multiple emotion scores of the target object within the target time period to obtain a comprehensive emotion score assessment, and determines the emotional state type of the target object based on a preset emotion score scale.

[0176] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0177] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0178] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described emotion assessment method based on deep temporal modeling is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0179] Figure 6 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application, such as... Figure 6 As shown, it includes a processor 602, a communication interface 604, a memory 606, and a communication bus 608. The processor 602, communication interface 604, and memory 606 communicate with each other via the communication bus 608.

[0180] Memory 606 is used to store computer programs;

[0181] When processor 602 executes a computer program stored in memory 606, it performs the following steps:

[0182] S1, real-time acquisition of speech data and EEG signals of the target object, and conversion of the acquired speech data into audio sequences and EEG signals into EEG sequences according to preset time intervals;

[0183] S2, input the audio sequence into the first sub-network of the emotion extractor to obtain the speech emotion result, and input the EEG sequence into the second sub-network of the emotion extractor to obtain the brain activity emotion result;

[0184] S3 assesses the target's emotional state based on the target's vocal emotion results and brain activity emotion results at different times, and obtains multiple emotion scores;

[0185] S4 integrates multiple emotion scores of the target object within the target time period to obtain a comprehensive emotion score assessment, and determines the emotional state type of the target object based on a preset emotion score scale.

[0186] Optionally, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0187] Memory may include RAM or non-volatile memory. Volatile memory, for example, at least one disk storage device. Alternatively, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0188] As an example, the memory 606 described above may include, but is not limited to, the acquisition unit 502, the evaluation unit 504, the assessment unit 506, and the determination unit 508 in the deep temporal modeling-based emotion assessment device. Furthermore, it may include, but is not limited to, other module units in the deep temporal modeling-based emotion assessment device, which will not be elaborated upon in this example.

[0189] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0190] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0191] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0192] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0194] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0196] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0197] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0198] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0199] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for emotion assessment based on deep temporal modeling, characterized in that, include: Real-time acquisition of speech data and EEG signals of target objects; according to preset time intervals, the acquired speech data is converted into audio sequences and the EEG signals are converted into EEG sequences. The audio sequence is input into the first sub-network of the emotion extractor to obtain the speech emotion result, and the EEG sequence is input into the second sub-network of the emotion extractor to obtain the brain activity emotion result; wherein, the first sub-network includes an acoustic information processing branch based on a time-aware bidirectional multi-scale network model and a text information processing branch based on a large language model. The target's emotional state is evaluated based on the target's vocal emotion results and brain activity emotion results at different times, resulting in multiple emotion scores; By integrating multiple emotion scores of the target object within the target time period, a comprehensive emotion score assessment is obtained, and the emotion state type of the target object is determined according to a preset emotion score scale. The method further includes: The Mel frequency cepstral coefficient features are input into the acoustic information processing branch of the first sub-network to capture deep temporal information and obtain deep temporal features. Apply the Sigmoid function to deep temporal features to learn a temporal attention map; The time-aware feature is generated by performing element-wise multiplication between the time attention map and the input feature map. The feature outputs of the feedforward and feedback time attention blocks located at the same time stamp are fused to obtain the representation vector of the target receptive field; We employ a weighted summation method to fuse features from attention blocks at different timestamps. The weighted feature sequence is then passed through a fully connected layer and a softmax function to output the predicted classification result. The transcribed text is input into the large language model of the first sub-network, and the voice emotion result of the speech text is output according to the preset evaluation template. The voice emotion result of the target object is formed based on the prediction classification result output by the acoustic information processing branch and the voice emotion result of the speech text output by the text information processing branch. The process of inputting the EEG sequence into the second subnetwork of the emotion extractor to obtain brain activity and emotional results includes: Based on the spatial distribution of the electrodes, the distribution of EEG signals is mapped to a spatial matrix; Multi-granular spatial feature extraction of EEG signals using a cascaded attention mechanism; By calculating weights using weighted information entropy, dynamic time warping, and cross-attention weights, multi-granular spatial weights are generated. Combined with the original signal matrix, a multi-dimensional spatial activity matrix of the neural circuit is calculated. 2.The emotion evaluation method based on deep time-series modeling of claim 1, wherein, The second sub-network uses an adaptive spatiotemporal representation learning ASTR EEG emotion recognition model. 3.The emotion evaluation method based on deep time-series modeling of claim 1, wherein, The step of inputting the audio sequence into the first sub-network of the emotion extractor to obtain the speech emotion result includes: The audio sequence is converted into a linear spectrogram through a short-time Fourier transform. A Mel filter bank is applied to the linear spectrogram to obtain a spectrum. The frequency axis of the spectrum is then converted into a Mel scale. Perform a logarithmic transformation on the output of each Mel filter to obtain the Mel spectrum. The Mel frequency cepstral coefficient characteristics are obtained by performing a discrete cosine transform on the Mel spectrum. 4.The emotion evaluation method based on deep time-series modeling of claim 1, wherein, After extracting multi-granularity spatial features from EEG signals using a cascaded attention mechanism, the method further includes: The spatial features of time segments are input into a one-way self-attention module to extract time change features and generate a time-space representation vector. The time-space representation vector is passed to the classifier to obtain the brain activity emotion score.

5. A deep temporal modeling-based emotion assessment device, used to perform the deep temporal modeling-based emotion assessment method as described in any one of claims 1 to 4, characterized in that, include: The acquisition unit is used to acquire the voice data and EEG signals of the target object in real time, and convert the acquired voice data into audio sequences and the EEG signals into EEG sequences according to a preset time interval. The acquisition unit is used to input audio sequences into the first subnetwork of the emotion extractor to acquire speech emotion results, and input EEG sequences into the second subnetwork of the emotion extractor to acquire brain activity emotion results. The evaluation unit is used to evaluate the emotional state of the target object based on the voice emotion results and brain activity emotion results at different times, and obtain multiple emotion scores; The determination unit is used to integrate multiple emotion scores of the target object within the target time period to obtain a comprehensive emotion score assessment, and to determine the emotion state type of the target object based on a preset emotion score scale.

6. A computer readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 4. 7.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 4 through the computer program.

Images