Teaching speech emotion recognition method based on dynamic time sequence modeling and multi-scale fusion

Abstract

The invention discloses a teaching speech emotion recognition method based on dynamic time sequence modeling and multi-scale fusion. The method comprises the steps of speech data set preprocessing, speech feature extraction, speech emotion classification network construction, speech emotion classification network training, inputting a test set into the trained speech emotion classification network, and outputting the probability of each type of emotion. According to the method, the direction of the information flow is dynamically adjusted through the adaptive time displacement module, the features of different time scales are extracted by using the multi-scale convolution branch, the modeling capability of the model for a complex time sequence structure and variation data is improved, the feature expression is enhanced, the time sequence features are extracted by using the Wav2Vec2.0 pre-training model, and the time sequence features are extracted by using the Wav2Vec2.0 pre-training model. A speech emotion classification network comprising an AdaShiftFormer learning module and a multi-scale time sequence fusion module is constructed, training is carried out in combination with classification loss and comparison loss, and classification performance is optimized. The method is superior to the prior art in emotion classification accuracy and feature expression ability, and can assist in teacher speech behavior analysis and classroom interaction optimization in an intelligent education scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech emotion recognition, and specifically relates to a teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion. Background Art

[0002] Classroom teaching is not only a basic field for imparting knowledge, but also an important space for emotional communication. The teacher's verbal behavior in the classroom not only undertakes the task of cognitive guidance, but the emotions and attitudes it conveys also directly affect students' learning motivation, emotional state and classroom atmosphere. Studies have shown that positive emotions (such as happiness) expressed by teachers in the classroom can help stimulate students' interest in learning and create an open and inclusive teaching environment; neutral emotional expression can help maintain good classroom order and academic atmosphere; and if teachers express negative emotions (such as anger), it will often inhibit students' active participation and even damage their learning confidence and classroom experience. Therefore, accurately identifying the emotional state of teachers in the teaching process is a key technical support for improving the quality of classroom teaching and promoting effective interaction between teachers and students.

[0003] In recent years, speech emotion recognition, as a key area of ​​convergence between artificial intelligence and education, has made initial progress. However, current mainstream methods rely on acoustic features extracted from pre-trained models, stacked with simple fully connected layers or recurrent networks. This makes it difficult to effectively model the short-term dynamics and long-term dependencies of emotional expression in teachers' speech. Furthermore, classroom speech is characterized by natural rhythm, subtle emotions, and variability. Existing models are significantly inadequate in processing multi-scale temporal structural information, failing to fully capture the emotional layers and evolutionary processes inherent in teachers' speech. Furthermore, to address the limited number of real-world teaching speech data samples, some studies have attempted to introduce data augmentation methods such as SpecAugment or to implement transfer learning optimization on specific tasks through fine-tuning strategies. However, these strategies still have limitations in terms of feature utilization efficiency and structural expression capabilities, making it difficult to achieve high-performance emotion recognition with small sample sizes.

[0004] Therefore, there is an urgent need for a lightweight model architecture with dynamic time series modeling capabilities and multi-scale time series information fusion mechanism, so as to more deeply explore the fine-grained emotional change characteristics in speech while ensuring parameter efficiency, and improve the model's ability to analyze emotional information in the teaching context and overall generalization performance, thereby assisting in the analysis of teachers' speech behavior and optimization of classroom interactions in intelligent education scenarios. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion with high emotion classification accuracy, strong feature expression ability, and strong generalization ability under small sample conditions.

[0006] The technical solution adopted to solve the above technical problems is: a teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion, including the following steps:

[0007] Step 1. Divide the speech emotion dataset

[0008] The teaching speech dataset is used to select one session as the test set using the five-fold cross-validation method, and the remaining four sessions are combined into the training set, and 10% of them are divided as the validation set;

[0009] Step 2. Extract speech features

[0010] The speech data is uniformly resampled to 16kHz and input into the Wav2Vec2.0 pre-trained model. The CNN feature extractor of the Wav2Vec2.0 pre-trained model extracts frame-level acoustic features. The multi-layer Transformer encoder then aggregates contextual information to output a temporal feature vector.

[0011] Step 3. Build a speech emotion classification network

[0012] The speech emotion classification network consists of a data enhancement module, a first AdaShiftFormer learning module, a second AdaShiftFormer learning module, a multi-scale temporal fusion module, and a global average pooling module connected in sequence. The output of the global average pooling module is connected to a fully connected classification module and a contrastive learning module in parallel.

[0013] The data enhancement module is used to enhance the time domain and frequency domain of the time series feature vector at the model input end to improve the robustness to noise and different speaking speeds and accents, effectively suppress overfitting, and output the enhanced time series feature vector;

[0014] The first AdaShiftFormer learning module is used to shift the temporal feature vector output by the data augmentation module forward and backward according to a number of learnable shift steps. While capturing local short-term emotional fluctuations, it also provides multi-perspective temporal dependency information for downstream convolution operations, thereby enhancing the model's sensitivity to rapid emotional changes.

[0015] The second AdaShiftFormer learning module is used to further perform adaptive time-shift learning on the feature sequence output by the first stage over a longer time range. It models the global emotion evolution trend through multi-step shift fusion, improving the model's ability to express long-term dependencies.

[0016] The multi-scale temporal fusion module is used to perform depth-separable convolution operations on the feature sequences output by the first AdaShiftFormer learning module and the second AdaShiftFormer learning module in parallel at multiple convolution kernel scales, calculate adaptive fusion weights for each scale through a fully connected network, and finally perform weighted summation of the features at each scale according to these weights, thereby taking into account both local and global temporal information and achieving coordinated fusion of fine-grained and macro features;

[0017] The global average pooling module is used to perform an average pooling operation on the fused time series feature sequence along the time dimension, compressing the time series information of variable length into a global representation vector of fixed size;

[0018] The fully connected classification module is used to receive the global representation vector from the global average pooling module, generate the class raw score of each emotion class, and complete the supervised classification through the cross entropy loss;

[0019] The contrastive learning module is used to project the global representation into the contrast space, construct the positive and negative sample contrast loss, and optimize the discriminability and structure of the features;

[0020] Step 4. Train the speech emotion classification network

[0021] Step 4.1. Construct the loss function L according to the following formula total ,

[0022] L total =L CE +α·L SupCon ,

[0023]

[0024] Where y i Represents the true emotional category label value of the i-th category, represents the label value of the i-th emotion category predicted by the speech emotion classification network, C is the number of emotion categories, P(i) is the set of positive samples of the i-th category, |P(i)| is the number of positive samples of the i-th category, A(i) is the set of comparison samples of the i-th category, z i is the feature vector of the i-th anchor sample, z p is the eigenvector of the positive sample, z a is the feature vector of the contrast sample, α is the weight coefficient for controlling the influence of contrastive learning on the total loss, τ is the temperature coefficient, and N is the number of samples;

[0025] Step 4.2. Train the speech emotion classification network

[0026] Set the training parameters, input the training set into the speech emotion classification network for training, use the cosine annealing strategy to adjust the learning rate, and use the Adam method to train to the loss function L total Convergence: The Wav2Vec2.0 pre-trained model is trained together with the speech emotion classification network. The training parameters are consistent with the downstream network, and all parameters are updated through backpropagation.

[0027] Step 4.3. Save the optimal weight file after each fold training is completed;

[0028] Step 5. Input the test set into the trained speech emotion classification network and output the probability of each emotion.

[0029] As a preferred technical solution, the first AdaShiftFormer learning module is composed of an adaptive time shift module, an MLP module and a layer normalization module connected in series;

[0030] The adaptive time shift module is used to adaptively shift features at different moments forward and backward based on the global features of the input time series feature vector, fusing information at multiple time granularities through learnable shift weights to enhance the ability to capture short-term sudden changes and long-term dependencies in speech.

[0031] The MLP module is used to perform two-layer full connection and nonlinear activation mapping on the temporal features after adaptive time shift, reorganize and expand the feature dimension, thereby enhancing the model's ability to express and distinguish high-order semantic relationships;

[0032] The layer normalization is used to perform channel normalization processing on features that have undergone time displacement and nonlinear mapping, balance the numerical distribution of each dimension, stabilize model training and accelerate gradient convergence;

[0033] The second AdaShiftFormer learning module has the same structure as the first AdaShiftFormer learning module.

[0034] As a preferred technical solution, the adaptive time displacement module is as follows: the channel displacement layer and the displacement weight generation layer are output in parallel to the dynamic fusion controller layer, and the dynamic fusion controller layer, the feature projection layer, and the residual connection layer are connected in series in sequence;

[0035] The channel shift layer is used to divide the input features into a forward offset group, a backward offset group, and a static group in the channel dimension, and to controllably scroll the first two groups along the time dimension to capture temporal dependency information of different durations;

[0036] The displacement weight generation layer is composed of a first fully connected layer, a GELU activation layer, a second fully connected layer, and a Dropout layer in series. It is used to output the initial fusion weight corresponding to each offset step size based on the global statistical characteristics of the input sequence through two layers of full connection and activation mapping, providing a measurement basis for subsequent adaptive fusion;

[0037] The dynamic fusion controller layer is used to apply the Gumbel-Softmax operation to the output of the displacement weight generation layer to approximately discretize multiple continuous displacement weights, so that the model can learn end-to-end and select the most appropriate time offset mode for parallel fusion;

[0038] The feature projection layer is used to perform dimension transformation and distribution adjustment on the fused time series features through linear mapping, nonlinear activation and layer normalization operations, thereby enhancing feature expression capabilities and ensuring numerical stability;

[0039] The residual connection layer is used to add the output of the feature projection layer to the original input features, retain the original timing information and promote gradient flow, so as to stabilize training and improve the model convergence speed.

[0040] As a preferred technical solution, the processing method of the channel displacement layer is:

[0041] The input feature vector is divided into three parts in the channel dimension, namely the forward displacement channel part Backward displacement channel part Static channel part Among them, B is the batch size, T is the time step, and C is the number of channels;

[0042] According to the formula, the forward displacement channel F forward Perform negative displacement, and the backward displacement channel F backward Perform a positive displacement:

[0043] Shift forward (F,d)=ROLL(F,-d,dim=1);

[0044] Shift backward (F,d)=ROLL(F,d,dim=1);

[0045] Where F is the input tensor, d is the shift length, dim=1 is the operation in the time dimension, and ROLL(·) is the cyclic shift operation in the time dimension;

[0046] The static channel is reconnected with the forward displacement channel and the backward displacement channel after displacement processing to obtain the displacement enhancement feature F shifted :

[0047] F shifted=CONCAT(Shift forward (F forward ,d),Shift backward (F backward ,d),F static , dim=2), dim=2 is operated in the channel dimension.

[0048] As a preferred technical solution, the multi-scale time series fusion module is as follows: a multi-scale feature extraction layer and a weight network generation layer are connected in parallel and output to a weighted fusion layer, and the weighted fusion layer, the output projection layer, and the residual connection layer are connected in series in sequence;

[0049] The multi-scale feature extraction layer is used to perform depth-separable convolution and nonlinear mapping on the input temporal features at multiple convolution kernel scales in parallel, capturing local, mid-range and global temporal dependency information respectively;

[0050] The weight network generation layer is used to globally pool the input features, output the fusion weight of each scale branch through two layers of full connection and activation mapping, and perform Softmax normalization to indicate the importance of each scale;

[0051] The weighted fusion layer is used to sum the output features of each scale branch according to the adaptive weights generated by the weight network to achieve collaborative fusion and feature integration of multi-scale information;

[0052] The output projection layer is used to perform linear mapping on the fused multi-scale features and reproject the channel dimension to a feature dimension consistent with the input so that subsequent modules can continue processing;

[0053] The residual connection layer is used to add the result of the output projection layer to the original input features, retaining the input information and stabilizing the network gradient flow, thereby accelerating convergence and improving model performance;

[0054] As a preferred technical solution, the processing method of the multi-scale feature extraction layer is as follows:

[0055]

[0056] Where, is the output feature of the k-th scale branch in the multi-scale feature extraction layer, k is the convolution kernel of different scales, DepthwiseConv1D k (x) is a depth-wise separable one-dimensional convolution, GELU(·) is a smooth activation function based on the Gaussian error function, P(·) is a dimension permutation operation, LayerNorm(·) is a layer normalization operation, and P -1 (·) is the inverse permutation operation.

[0057] As a preferred technical solution, the convolution kernel k of different scales is 3, 7 or 15.

[0058] As a preferred technical solution, the contrastive learning module is composed of a first linear layer, a ReLU activation layer, and a second linear layer connected in series.

[0059] The beneficial effects of the present invention are as follows:

[0060] The present invention dynamically adjusts the direction of information flow through an adaptive time displacement module, which can effectively capture the short-term dynamic changes of speech emotional characteristics and accurately model the complex and changeable emotional expressions of teachers in the classroom, such as sudden changes in teachers' tone, thereby more accurately identifying the emotional categories in speech, such as happiness, anger, sadness, neutrality, etc.

[0061] The present invention extracts features of different time scales through multi-scale convolution branches, fully considering the natural rhythm of classroom speech and the delicate and changeable emotions, enabling the model to analyze and integrate speech emotion features from multiple dimensions, more comprehensively depicting the evolution process of speech emotion, and thus improving the accuracy of emotion classification.

[0062] The powerful dynamic time series modeling capability of the present invention can process the complex emotional time series structure in speech signals and adapt to various changes and fluctuations in speech emotions over time. It can also effectively model and analyze speech segments with more obscure emotional expressions or more complex emotional transitions, thereby enhancing the robustness of the model.

[0063] The combination of the adaptive time shift module and the multi-scale convolution branch of the present invention can mine richer and deeper emotional features in speech, forming a feature representation with stronger discrimination, which helps to better distinguish different emotion categories and provide stronger support for subsequent emotion classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic flow diagram of the present invention.

[0065] Figure 2 It is a structural diagram of the speech emotion classification network of the present invention.

[0066] Figure 3 This is a schematic diagram of the structure of the first AdaShiftFormer module.

[0067] Figure 4 yes Figure 3 Schematic diagram of the structure of the adaptive time displacement module.

[0068] Figure 5 yes Figure 4 Schematic diagram of the structure of the displacement weight generation layer.

[0069] Figure 6 It is a structural diagram of the multi-scale time series fusion module. DETAILED DESCRIPTION

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0071] Example 1

[0072] exist Figure 1 In this embodiment, the teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion includes the following steps:

[0073] Step 1. Divide the speech emotion dataset

[0074] The teaching speech dataset is used to select one session as the test set using the five-fold cross-validation method, and the remaining four sessions are combined into the training set, and 10% of them are divided as the validation set;

[0075] Step 2. Extract speech features

[0076] The speech data is uniformly resampled to 16kHz and input into the Wav2Vec2.0 pre-trained model. The CNN feature extractor of the Wav2Vec2.0 pre-trained model extracts frame-level acoustic features. The multi-layer Transformer encoder then aggregates contextual information to output a temporal feature vector.

[0077] Step 3. Build a speech emotion classification network

[0078] like Figure 2 The speech emotion classification network is composed of a data enhancement module, a first AdaShiftFormer learning module, a second AdaShiftFormer learning module, a multi-scale time series fusion module, and a global average pooling module connected in sequence. The output of the global average pooling module is connected to a fully connected classification module and a contrastive learning module in parallel.

[0079] The data augmentation module is used to perform various time-domain and frequency-domain perturbations on the time series feature vector at the model input to improve robustness to noise, different speaking speeds, and accents, effectively suppress overfitting, and output the enhanced time series feature vector.

[0080] like Figure 3 The first AdaShiftFormer learning module is used to shift the input time series feature vector forward and backward according to a number of learnable shift steps. While capturing local short-term emotional fluctuations, it also provides multi-perspective temporal dependency information for downstream convolution operations to enhance the model's sensitivity to rapid emotional changes.

[0081] The first AdaShiftFormer learning module consists of an adaptive time shift module, an MLP module, and layer normalization in series; the adaptive time shift module is used to adaptively shift features at different moments forward and backward according to the global features of the input time series feature vector, and fuse information of multiple time granularities through learnable shift weights to enhance the ability to capture short-term sudden changes and long-term dependencies in speech; the MLP module is used to perform two-layer full connection and nonlinear activation mapping on the time series features after adaptive time shift, reorganize and expand the feature dimensions, thereby enhancing the model's ability to express and distinguish high-order semantic relationships; layer normalization is used to perform channel normalization processing on features that have undergone time shift and nonlinear mapping, balance the numerical distribution of each dimension, stabilize model training, and accelerate gradient convergence;

[0082] like Figure 4 ,The structure of the adaptive time displacement module is as follows: the channel displacement layer and the displacement weight generation layer are output in parallel to the dynamic fusion controller layer, and the dynamic fusion controller layer, the feature projection layer, and the residual connection layer are connected in series in sequence; the channel displacement layer is used to divide the input features into forward offset group, backward offset group and static group in the channel dimension, and controllably scroll the first two groups along the time dimension to capture temporal dependency information of different durations; the displacement weight generation layer is composed of the first fully connected layer, the GELU activation layer, the second fully connected layer, and the Dropout layer in series, as shown in Figure 2. Figure 5 , which is used to output the initial fusion weight corresponding to each offset step size through two layers of full connection and activation mapping based on the global statistical features of the input sequence, providing a measurement basis for subsequent adaptive fusion; the dynamic fusion controller layer is used to apply the Gumbel-Softmax operation to the output of the displacement weight generation layer, and approximately discretize multiple continuous offset weights, so that the model can learn end-to-end and select the most appropriate time offset mode for parallel fusion; the feature projection layer is used to perform dimensionality transformation and distribution adjustment on the fused time series features through linear mapping, nonlinear activation and layer normalization operations, thereby enhancing feature expression capabilities and ensuring numerical stability; the residual connection layer is used to add the output of the feature projection layer to the original input features, retaining the original time series information and promoting gradient flow, so as to stabilize training and improve the model convergence speed.

[0083] Among them, the processing method of the channel displacement layer is:

[0084] The input feature vector is divided into three parts in the channel dimension, namely the forward displacement channel part Backward displacement channel part Static channel part Among them, B is the batch size, T is the time step, and C is the number of channels;

[0085] According to the formula, the forward displacement channel F forwardPerform negative displacement, and the backward displacement channel F backward Perform a positive displacement:

[0086] Shift forward (F,d)=ROLL(F,-d,dim=1);

[0087] Shift backward (F,d)=ROLL(F,d,dim=1);

[0088] Where F is the input tensor, d is the shift length, dim=1 is the operation in the time dimension, and ROLL(·) is the cyclic shift operation in the time dimension;

[0089] The static channel is reconnected with the forward displacement channel and the backward displacement channel after displacement processing to obtain the displacement enhancement feature F shifted :

[0090] F shifted =CONCAT(Shift forward (F forward ,d),Shift backward (F backward ,d),F static , dim=2), dim=2 is operated in the channel dimension;

[0091] The second AdaShiftFormer learning module is used to further perform adaptive time-shift learning on the feature sequence output by the first stage over a longer time range, model the global emotional evolution trend through multi-step shift fusion, and improve the model's ability to express long-term dependencies; the second AdaShiftFormer learning module has the same structure as the first AdaShiftFormer learning module.

[0092] The multi-scale temporal fusion module performs depthwise separable convolution operations on the feature sequences output by the first and second AdaShiftFormer at multiple convolution kernel scales in parallel. It calculates adaptive fusion weights for each scale through a fully connected network and finally sums the features at each scale according to these weights, thereby taking into account both local and global temporal information and achieving the coordinated fusion of fine-grained and macro features.

[0093] like Figure 6The structure of the multi-scale time series fusion module is as follows: the multi-scale feature extraction layer and the weight network generation layer are output in parallel to the weighted fusion layer, and the weighted fusion layer, output projection layer, and residual connection layer are connected in series in sequence; the multi-scale feature extraction layer is used to perform depth-separable convolution and nonlinear mapping on the input time series features in parallel at multiple convolution kernel scales to capture local, medium-range and global time-dependent information respectively; the weight network generation layer is used to globally pool the input features and output the fusion weight of each scale branch through two layers of full connection and activation mapping, and is normalized by Softmax to indicate the importance of each scale; the weighted fusion layer is used to weight the output features of each scale branch according to the adaptive weights generated by the weight network to achieve collaborative fusion and feature integration of multi-scale information; the output projection layer is used to perform linear mapping on the fused multi-scale features and re-project the channel dimension to the feature dimension consistent with the input so that subsequent modules can continue to process; the residual connection layer is used to add the results of the output projection layer to the original input features, retaining the input information and stabilizing the network gradient flow, thereby accelerating convergence and improving model performance.

[0094] Among them, the processing method of the multi-scale feature extraction layer is as follows:

[0095]

[0096] Where, is the output feature of the k-th scale branch in the multi-scale feature extraction layer, k is the convolution kernel of different scales, and the value of k is 3, 7 or 15; DepthwiseConv1D k (x) is a depth-separable one-dimensional convolution operation on the input x, GELU(·) is a smooth activation function based on the Gaussian error function, P(·) is a dimension permutation operation, LayerNorm(·) is a layer normalization operation, and P -1 (·) is the inverse permutation operation.

[0097] The global average pooling module is used to perform an average pooling operation on the fused time series feature sequence along the time dimension, compressing the time series information of variable length into a global representation vector of fixed size;

[0098] The fully connected classification module receives the global representation vector from the global average pooling module, generates the class raw scores of each emotion class, and performs supervised classification through cross entropy loss;

[0099] The contrastive learning module consists of a first linear layer, a ReLU activation layer, and a second linear layer in series. It is used to project the global representation into the contrast space, construct the positive and negative sample contrast loss, and optimize the discriminability and structure of the features.

[0100] Step 4. Train the speech emotion classification network

[0101] Step 4.1. Construct the loss function L according to the following formula total ,

[0102] L total =L CE +α·L SupCon ,

[0103]

[0104]

[0105] Where y i Represents the true emotional category label value of the i-th category, represents the emotion category label value predicted by the speech emotion classification network of the i-th category, C is the number of emotion categories, P(i) is the set of positive samples of the i-th category, |P(i)| is the number of positive samples of the i-th category, A(i) is the set of comparison samples of the i-th category, z i is the feature vector of the i-th anchor sample, z p is the eigenvector of the positive sample, z a is the feature vector of the contrast sample, α is the weight coefficient for controlling the influence of contrastive learning on the total loss, α = 0.1, τ is the temperature coefficient, τ = 0.07, and N is the number of samples;

[0106] Step 4.2. Train the speech emotion classification network

[0107] Set the training parameters: the server graphics card for training is RTX3090, the data batch size is 32, and the learning rate is 5×10 -4 , training for 100 rounds, with a learning rate decay of 5×10 -6 ;

[0108] The training set is input into the speech emotion classification network for training, the cosine annealing strategy is used to adjust the learning rate, and the Adam method is used to train the loss function L total Convergence: The Wav2Vec2.0 pre-trained model is trained together with the speech emotion classification network. The training parameters are consistent with the downstream network, and all parameters are updated through backpropagation.

[0109] Step 4.3. Save the optimal weight file after each fold training is completed;

[0110] Step 5. Input the test set into the trained speech emotion classification network and output the probability of each emotion.

[0111] In order to verify the beneficial effects of the present invention, the inventors conducted computer comparative simulation experiments using the speech emotion recognition method based on adaptive temporal shift and multi-scale fusion in Example 1 of the present invention (hereinafter referred to as Example 1) and "S.Shen, F.Liu and A.Zhou," Minglingor Misalignment? Temporal Shift for Speech Emotion Recognition with Pre-Trained Representations," ICASSP 2023-2023IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5, doi: 10.1109 / ICASSP49357.2023.10095193. (hereinafter referred to as Comparative Experiment 1) and the traditional CNN-BiLSTM speech classification method (hereinafter referred to as Comparative Experiment 2). The accuracy UA was verified according to the following formula:

[0112]

[0113] Among them, C represents the total number of emotion categories, TP c N is the number of samples correctly predicted by the model as category c, c is the total number of true samples belonging to category c.

[0114] The experimental results are shown in Table 1.

[0115] Table 1. Experimental results of Example 1 and comparative experiments

[0116] Experimental group Verification accuracy UA (%) Example 1 74.40 Comparative Experiment 1 73.92 Comparative Experiment 2 66.9

[0117] As can be seen from Table 1, the classification accuracy of the method in Example 1 of the present invention is 74.40%, which is higher than the speech classification methods in Comparative Experiment 1 and Comparative Experiment 2, and can better judge the speaker's emotion.

Claims

1. A teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion, characterized by: The following steps are involved: Step 1. Divide the speech emotion dataset The teaching speech dataset is used to select one session as the test set using the five-fold cross-validation method, and the remaining four sessions are combined into the training set, and 10% of them are divided as the validation set; Step 2. Extract speech features The speech data is uniformly resampled to 16kHz and input into the Wav2Vec2.0 pre-trained model. The CNN feature extractor of the Wav2Vec2.0 pre-trained model extracts frame-level acoustic features. The multi-layer Transformer encoder then aggregates contextual information to output a temporal feature vector. Step 3. Build a speech emotion classification network The speech emotion classification network consists of a data enhancement module, a first AdaShiftFormer learning module, a second AdaShiftFormer learning module, a multi-scale temporal fusion module, and a global average pooling module connected in sequence. The output of the global average pooling module is connected to a fully connected classification module and a contrastive learning module in parallel. The data enhancement module is used to enhance the time domain and frequency domain of the time series feature vector at the model input end to improve the robustness to noise and different speaking speeds and accents, effectively suppress overfitting, and output the enhanced time series feature vector; The first AdaShiftFormer learning module is used to shift the temporal feature vector output by the data augmentation module forward and backward according to a number of learnable shift steps. While capturing local short-term emotional fluctuations, it also provides multi-perspective temporal dependency information for downstream convolution operations, thereby enhancing the model's sensitivity to rapid emotional changes. The second AdaShiftFormer learning module is used to further perform adaptive time-shift learning on the feature sequence output by the first stage over a longer time range. It models the global emotion evolution trend through multi-step shift fusion, improving the model's ability to express long-term dependencies. The multi-scale temporal fusion module is used to perform depth-separable convolution operations on the feature sequences output by the first AdaShiftFormer learning module and the second AdaShiftFormer learning module in parallel at multiple convolution kernel scales, calculate adaptive fusion weights for each scale through a fully connected network, and finally perform weighted summation of the features at each scale according to these weights, thereby taking into account both local and global temporal information and achieving coordinated fusion of fine-grained and macro features; The global average pooling module is used to perform an average pooling operation on the fused time series feature sequence along the time dimension, compressing the time series information of variable length into a global representation vector of fixed size; The fully connected classification module is used to receive the global representation vector from the global average pooling module, generate the class raw score of each emotion class, and complete the supervised classification through the cross entropy loss; The contrastive learning module is used to project the global representation into the contrast space, construct the positive and negative sample contrast loss, and optimize the discriminability and structure of the features; Step 4. Train the speech emotion classification network Step 4.

1. Construct the loss function L according to the following formula total , L total =L CE +α·L SupCon , Where y i Represents the true emotional category label value of the i-th category, represents the label value of the i-th emotion category predicted by the speech emotion classification network, C is the number of emotion categories, P(i) is the set of positive samples of the i-th category, |P(i)| is the number of positive samples of the i-th category, A(i) is the set of comparison samples of the i-th category, z i is the feature vector of the i-th anchor sample, z p is the eigenvector of the positive sample, z a is the feature vector of the contrast sample, α is the weight coefficient for controlling the influence of contrastive learning on the total loss, τ is the temperature coefficient, and N is the number of samples; Step 4.

2. Train the speech emotion classification network Set the training parameters, input the training set into the speech emotion classification network for training, use the cosine annealing strategy to adjust the learning rate, and use the Adam method to train to the loss function L total Convergence: The Wav2Vec2.0 pre-trained model is trained together with the speech emotion classification network. The training parameters are consistent with the downstream network, and all parameters are updated through backpropagation. Step 4.

3. Save the optimal weight file after each fold training is completed; Step 5. Input the test set into the trained speech emotion classification network and output the probability of each emotion.

2. The teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion according to claim 1 is characterized in that: The first AdaShiftFormer learning module consists of an adaptive time shift module, an MLP module, and a layer normalization module in series; The adaptive time shift module is used to adaptively shift features at different moments forward and backward based on the global features of the input time series feature vector, fusing information at multiple time granularities through learnable shift weights to enhance the ability to capture short-term sudden changes and long-term dependencies in speech. The MLP module is used to perform two-layer full connection and nonlinear activation mapping on the temporal features after adaptive time shift, reorganize and expand the feature dimension, thereby enhancing the model's ability to express and distinguish high-order semantic relationships; The layer normalization is used to perform channel normalization processing on features that have undergone time displacement and nonlinear mapping, balance the numerical distribution of each dimension, stabilize model training and accelerate gradient convergence; The second AdaShiftFormer learning module has the same structure as the first AdaShiftFormer learning module.

3. The teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion according to claim 2 is characterized in that: The adaptive time displacement module comprises: a channel displacement layer and a displacement weight generation layer connected in parallel and outputted to a dynamic fusion controller layer, and the dynamic fusion controller layer, a feature projection layer, and a residual connection layer connected in series in sequence; The channel shift layer is used to divide the input features into a forward offset group, a backward offset group, and a static group in the channel dimension, and controllably scroll the first two groups along the time dimension to capture temporal dependency information of different durations; The displacement weight generation layer is composed of a first fully connected layer, a GELU activation layer, a second fully connected layer, and a Dropout layer in series. It is used to output the initial fusion weight corresponding to each offset step size based on the global statistical characteristics of the input sequence through two layers of full connection and activation mapping, providing a measurement basis for subsequent adaptive fusion; The dynamic fusion controller layer is used to apply the Gumbel-Softmax operation to the output of the displacement weight generation layer to approximately discretize multiple continuous displacement weights, so that the model can learn end-to-end and select the most appropriate time offset mode for parallel fusion; The feature projection layer is used to perform dimension transformation and distribution adjustment on the fused time series features through linear mapping, nonlinear activation and layer normalization operations, thereby enhancing feature expression capabilities and ensuring numerical stability; The residual connection layer is used to add the output of the feature projection layer to the original input features, retain the original timing information and promote gradient flow, so as to stabilize training and improve the model convergence speed.

4. The teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion according to claim 3 is characterized in that: The processing method of the channel displacement layer is: The input feature vector is divided into three parts in the channel dimension, namely the forward displacement channel part Backward displacement channel part Static channel part Among them, B is the batch size, T is the time step, and C is the number of channels; According to the formula, the forward displacement channel F forward Perform negative displacement, and the backward displacement channel F backward Perform a positive displacement: Shift forward (F,d)=ROLL(F,-d,dim=1); Shift backward (F,d)=ROLL(F,d,dim=1); Where F is the input tensor, d is the shift length, dim=1 is the operation in the time dimension, and ROLL(·) is the cyclic shift operation in the time dimension; The static channel is reconnected with the forward displacement channel and the backward displacement channel after displacement processing to obtain the displacement enhancement feature F shifted : F shifted =CONCAT(Shift forward (F forward ,d),Shift backward (F backward ,d),F static , dim=2), dim=2 is operated in the channel dimension.

5. The teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion according to claim 1 is characterized in that: The multi-scale temporal fusion module comprises: a multi-scale feature extraction layer and a weight network generation layer connected in parallel and outputted to a weighted fusion layer, and the weighted fusion layer, an output projection layer, and a residual connection layer connected in series in sequence; The multi-scale feature extraction layer is used to perform depth-separable convolution and nonlinear mapping on the input temporal features at multiple convolution kernel scales in parallel, capturing local, mid-range and global temporal dependency information respectively; The weight network generation layer is used to perform global pooling on the input features, output the fusion weight of each scale branch through two layers of full connection and activation mapping, and perform Softmax normalization to indicate the importance of each scale; The weighted fusion layer is used to sum the output features of each scale branch according to the adaptive weights generated by the weight network to achieve collaborative fusion and feature integration of multi-scale information; The output projection layer is used to perform linear mapping on the fused multi-scale features and reproject the channel dimension to a feature dimension consistent with the input so that subsequent modules can continue processing; The residual connection layer is used to add the result of the output projection layer to the original input features, retaining the input information and stabilizing the network gradient flow, thereby accelerating convergence and improving model performance.

6. The teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion according to claim 5 is characterized in that: The processing method of the multi-scale feature extraction layer is as follows: Where, is the output feature of the k-th scale branch in the multi-scale feature extraction layer, k is the convolution kernel of different scales, DepthwiseConv1D k (x) is a depth-wise separable one-dimensional convolution, GELU(·) is a smooth activation function based on the Gaussian error function, P(·) is a dimension permutation operation, LayerNorm(·) is a layer normalization operation, and P -1 (·) is the inverse permutation operation.

7. The teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion according to claim 6 is characterized in that: The convolution kernel k of different scales is 3, 7 or 15.

8. The teaching speech emotion recognition method based on dynamic time series modeling and multi-scale fusion according to claim 1 is characterized in that: The contrastive learning module is composed of a first linear layer, a ReLU activation layer, and a second linear layer connected in series.

Images