Speech emotion recognition method based on multi-feature fusion

Abstract

The application discloses a speech emotion recognition method based on multi-feature fusion and specifically relates to the technical field of speech emotion recognition; a speech signal is preprocessed, and multi-scale low-level acoustic features are extracted; context information is modeled based on a gated recurrent unit to generate emotion representation features; semantic features are extracted in parallel, including emotion dictionary matching, semantic embedding and dependency syntax analysis; through an adaptive attention mechanism, acoustic and semantic features are cross-modally aligned and fused to form high-order emotion fusion features; then, a residual network and a GRU combined structure are used for classification to output fine-grained emotion labels; when the confidence is lower than a threshold, multi-round fusion judgment is performed in combination with context semantics and historical emotion trends; the method has high recognition accuracy and robustness and is suitable for various complex speech interaction scenes.

Description

Technical Field

[0001] This invention relates to the field of speech emotion recognition technology, and more specifically to a speech emotion recognition method based on multi-feature fusion. Background Technology

[0002] With the development of intelligent voice interaction technology, speech emotion recognition (SER) has been widely used in various scenarios such as customer service assistance, smart homes, and medical care. However, existing speech emotion recognition methods generally suffer from the following technical problems: Most methods can only identify basic emotion categories such as "happy," "angry," and "sad," and struggle to recognize complex, fine-grained emotions such as "suppressed anger" and "resentful sadness." This results in the system failing to accurately reflect the user's inner state, making it prone to misjudgment, especially in scenarios like psychological counseling or crisis intervention. Furthermore, when non-emotional interference such as slight coughing, pauses, or sudden changes in speech rate are present in the speech, traditional methods struggle to distinguish them from genuine emotional signals, significantly reducing the accuracy of emotion recognition. Most existing methods rely solely on single or linear combinations of features such as pitch, intensity, or Mel-frequency cepstral coefficients (MFCCs), making it difficult to comprehensively capture emotional cues across multiple dimensions of speech, including time, frequency, and semantics. Summary of the Invention

[0003] The purpose of this invention is to provide a speech emotion recognition method based on multi-feature fusion to address the shortcomings of the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a speech emotion recognition method based on multi-feature fusion, comprising: The original speech signal is preprocessed to remove silent segments and background noise, and then windowed in frames to obtain the processed speech signal sequence. A multi-scale low-level acoustic feature set is extracted from the processed speech signal sequence, including Mel frequency cepstral coefficients, spectrogram, zero-crossing rate, speech energy and its first and second order difference features, denoted as feature set A1; Based on the feature set A1, a temporal dependency model is constructed, and context-related feature sequences are extracted to obtain the sentiment representation feature set A2 after temporal modeling. Semantic features are extracted in parallel, including sentiment tendency features extracted based on sentiment dictionary and semantic embedding model after the speech signal is transcribed into text, as well as semantic dependency relationship features, which constitute the semantic feature set A3; Cross-modal alignment and fusion of feature sets A2 and A3 are performed, and weight relationships between multiple features are established through an adaptive attention mechanism to generate a unified high-order sentiment fusion feature set A4. The fused high-order sentiment feature set A4 is input into the sentiment recognition and classification model. A multi-layer residual network and gated recurrent unit architecture are used to distinguish sentiment categories and output fine-grained sentiment labels. If the confidence level of the emotion recognition classification model is lower than the set threshold, then further fusion judgment is performed by combining contextual semantics and historical emotion trend features.

[0005] Preferably, the preprocessing of the original speech signal includes: Endpoint detection is performed on the original speech signal. An improved short-time energy and zero-crossing rate joint decision method is used to locate the speech start and end points, remove silence segments, and extract effective speech segments. A composite noise reduction algorithm combining spectral subtraction and Wiener filtering is applied to the intercepted effective speech signal to suppress background noise while preserving the frequency characteristics of the speech signal and improving the signal-to-noise ratio. The noise-reduced speech signal is pre-emphasized, and a first-order high-pass filter is used to enhance the high-frequency components in order to compensate for the high-frequency attenuation during the transmission of the vocal tract. The pre-emphasized speech signal is divided into frames with a frame length of 25ms and a frame shift of 10ms. A Hamming window function is applied to each frame to smooth the frame boundaries and suppress spectral leakage, resulting in the processed speech signal frame sequence.

[0006] Preferably, the step of extracting a multi-scale low-level acoustic feature set from the processed speech signal sequence includes: Multi-scale frame windows were constructed on the processed speech signal frame sequence at three time scales of 20 milliseconds, 40 milliseconds and 80 milliseconds to obtain the speech dynamic characteristics at different time resolutions. At each time scale, Mel frequency cepstral coefficients are extracted, including the first to 13th order coefficients and their first and second order difference features. The Mel frequency cepstral coefficient MFCC calculation process includes: performing fast Fourier transform on the windowed frame, bandpass filtering through the Mel filter bank, taking the logarithmic energy and then performing discrete cosine transform to obtain the nonlinear compressed spectrum features of the speech. Based on the same frame window, the short-time energy and zero-crossing rate features of each frame are extracted. The short-time energy is obtained by calculating the sum of squares of the amplitudes of all sampling points within the frame, and the zero-crossing rate is obtained by dividing the number of statistical symbol changes by the number of sampling points, which are used to characterize the trend of speech intensity and frequency changes. The MFCC, first-order difference, second-order difference, short-time energy and zero-crossing rate features extracted at all scales are normalized and cascaded according to time scale to form a unified low-level acoustic feature set A1, which is used to reflect the emotional changes of speech signals at different time scales.

[0007] Preferably, the obtained emotional representation feature set A2 after time-series modeling includes: The feature set A1 is constructed into a time series input tensor according to the frame order of the speech signal, while maintaining the time alignment order of the original frame order. This tensor is used as the modeling input, and its dimension is T×D, where T represents the number of frames and D represents the feature dimension of each frame. A bidirectional gated recurrent unit network (BiGRU) is constructed, which includes two parallel channels: forward GRU and backward GRU. These channels model the feature sequence from front to back and from back to front, respectively, to capture the contextual information. The GRU unit controls the state memory and forgetting mechanism through reset gate and update gate. The forward and backward hidden states of the BiGRU network are concatenated to form the context-sensitive feature representation of each frame in the global context, resulting in the intermediate sequence feature set A2'. A temporal attention mechanism is applied to the intermediate sequence feature set A2' to calculate the importance weight of each frame to the overall speech segment emotion. The features are then weighted and summed according to the attention weights to form a fixed-dimensional temporal context emotion representation feature set A2.

[0008] Preferably, the parallel extraction of semantic layer features includes: The processed speech signal is subjected to automatic speech recognition processing. An end-to-end speech recognition algorithm based on the fusion of deep acoustic model and language model is used to transcribe the speech signal into corresponding text information. The recognition model adopts the jointly trained CTC-Transformer architecture. The transcribed text sequence is matched with a sentiment dictionary. A multidimensional sentiment dictionary is used to segment and annotate the text, and the word frequency distribution, sentiment polarity distribution and word class intensity statistical features are extracted to form a preliminary sentiment tendency vector. The text is input into a pre-trained semantic embedding model, and a Transformer-based contextual language model is used to generate semantic embedding representations for each word. The contextual sentiment semantic vector of the whole sentence is extracted by average pooling and max pooling methods. Dependency syntactic analysis is performed on the text to construct a semantic dependency graph. Based on syntactic dependency relations, the dependency paths of subject-verb-object structure, modification structure, and sentiment modifiers and core words are extracted, and structured semantic dependency features are generated. Finally, these features are fused with sentiment vectors and semantic embedding vectors to form a semantic feature set A3.

[0009] Preferably, the cross-modal alignment and fusion of the A2 and A3 feature sets includes: The emotional representation feature set A2 and the semantic feature set A3 are subjected to dimensionality standardization. A fully connected network is used to linearly map each feature to unify their feature dimensions to the preset fusion space dimension d. The dimensionally standardized A2 and A3 are input into a multi-head adaptive attention mechanism to construct a cross-modal fusion representation, where A2 is used as the query vector and A3 is used as the key vector and value vector. The attention mechanism is used to calculate the attention distribution of acoustic context features to semantic features and weighted aggregation to obtain the preliminary fusion feature representation A4′. The preliminary fused feature A4′ is residually connected with the original features A2 and A3, and then the feature is reconstructed through a feedforward fully connected network with ReLU activation function to generate a fused feature vector sequence. The fused sequence is compressed using a pooling mechanism, and then a combination of adaptive average pooling and max pooling is used to obtain a fixed-length unified high-order sentiment fusion feature set A4.

[0010] Preferably, the step of inputting the fused high-order emotion feature set A4 into the emotion recognition and classification model includes: The high-order emotion fusion feature set A4 is input into a multi-layer residual fully connected network. The residual network consists of several linear transformation layers. Each layer includes a fully connected layer with an activation function and a skip connection structure. The output of the residual layer is a feature vector sequence of dimension d. The feature vector sequence output by the residual network is input into a single-layer or multi-layer gated recurrent unit (GRU) to further model the long-distance dependencies between features using its temporal modeling capability. The GRU unit adopts a bidirectional structure with a hidden state dimension of 128 and outputs context-dependent feature representation vectors. The GRU output vector is mapped to the same output dimension as the preset number of sentiment categories through a fully connected layer, and then normalized using the Softmax function to obtain the predicted probability distribution of each sentiment category; The final sentiment recognition result is output based on the category label corresponding to the maximum value in the sentiment prediction probability.

[0011] Preferably, the multi-round fusion judgment includes: A confidence threshold θ is set. When the highest category probability output by the emotion recognition classification model is lower than the confidence threshold θ, a supplementary discrimination process is triggered. Extract the contextual information of the current speech segment in the semantic sequence, including the semantic embedding representation of the two sentences before and after it and the sentiment word tendency features. Encode it based on BERT or contextual language model to form a contextual semantic feature vector, which is used to supplement the semantic missing parts in the current judgment. Construct a historical emotion trend sequence, statistically analyze the emotion prediction labels and confidence levels of the current speaker in the previous rounds of voice interaction, calculate its emotion change curve and fluctuation frequency, and extract the emotion stability index, trend directionality and maximum emotion value as historical feature vectors; The high-order fusion feature A4 of the current segment, the contextual semantic feature vector, and the historical sentiment trend vector are concatenated and input into a lightweight two-layer fully connected network for further judgment. The updated sentiment category and confidence level are output. If the value is still lower than the confidence level threshold θ, it is marked as "uncertain sentiment" or handed over to human confirmation.

[0012] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. The speech emotion recognition method based on multi-feature fusion provided by this invention fully combines acoustic features, semantic features, and historical interaction information. Through multi-scale feature extraction, bidirectional temporal modeling, cross-modal attention alignment and fusion, and a classification architecture combining residual structure and GRU, it achieves high-precision discrimination of fine-grained emotions. Especially in situations with ambiguous emotional expression, strong semantic dependence, and complex backgrounds, it can still maintain good robustness and recognition accuracy, effectively solving the problems of weak feature representation ability, insufficient contextual understanding, and difficulty in processing low-confidence outputs in traditional methods.

[0013] 2. This invention innovatively introduces a confidence-driven multi-round fusion judgment mechanism. When the model output is uncertain, it automatically fuses contextual semantics and historical sentiment trends for secondary discrimination, enhancing the system's stability in judging fuzzy sentiment states and its ability to cope with complex emotional changes. The overall solution has advantages such as clear structure, adjustable parameters, and flexible deployment, making it suitable for various practical application scenarios, such as intelligent customer service, voice interaction devices, psychological counseling systems, and human-computer collaboration platforms, and possessing significant engineering application value. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] For examples, please refer to Figure 1As shown in this embodiment, the speech emotion recognition method based on multi-feature fusion includes: The original speech signal is preprocessed to remove silent segments and background noise, and then windowed in frames to obtain the processed speech signal sequence. A multi-scale low-level acoustic feature set is extracted from the processed speech signal sequence, including Mel frequency cepstral coefficients, spectrogram, zero-crossing rate, speech energy and its first and second order difference features, denoted as feature set A1; Based on the feature set A1, a temporal dependency model is constructed, and context-related feature sequences are extracted to obtain the sentiment representation feature set A2 after temporal modeling. Semantic features are extracted in parallel, including sentiment tendency features extracted based on sentiment dictionary and semantic embedding model after the speech signal is transcribed into text, as well as semantic dependency relationship features, which constitute the semantic feature set A3; Cross-modal alignment and fusion of feature sets A2 and A3 are performed, and weight relationships between multiple features are established through an adaptive attention mechanism to generate a unified high-order sentiment fusion feature set A4. The fused high-order sentiment feature set A4 is input into the sentiment recognition and classification model. A multi-layer residual network and gated recurrent unit architecture are used to distinguish sentiment categories and output fine-grained sentiment labels. If the confidence level of the emotion recognition classification model is lower than the set threshold, then further fusion judgment is performed by combining contextual semantics and historical emotion trend features.

[0018] Speech endpoint detection aims to identify the start and end points in the original speech signal, thereby accurately removing silence segments and background segments, retaining only the time interval containing valid speech. This invention employs an improved joint decision method based on short-time energy and zero-crossing rate to enhance detection accuracy in non-ideal environments.

[0019] Short-time energy calculation: The original speech signal is divided into consecutive short-time frames, typically with a frame length of 20 milliseconds and a frame shift of 10 milliseconds. For each frame, the corresponding short-time energy is obtained by calculating the sum of the squared amplitude values.

[0020] Zero-crossing rate calculation: This involves counting the number of signal sign changes within each frame, i.e., the number of positive and negative sign switching events between adjacent sampling points. The zero-crossing rate is used to distinguish between silent segments and high-frequency noise segments.

[0021] Joint decision mechanism: Short-time energy and zero-crossing rate are simultaneously compared with adaptive thresholds. The short-time energy threshold is defined as the mean of the energy of the entire speech segment plus twice the standard deviation, and the zero-crossing rate threshold is defined as the mean of the zero-crossing rates of the entire speech segment plus the standard deviation. If the short-time energy of a frame is higher than the energy threshold and the zero-crossing rate is lower than the zero-crossing rate threshold, then the frame is determined to be a speech frame.

[0022] Continuous frame connection mechanism: To prevent misjudgment of the start or end point of speech, a contextual continuous frame judgment mechanism is adopted. That is, if more than 5 consecutive frames meet the speech judgment condition, the speech is considered to have started; otherwise, if more than 5 consecutive frames are below the judgment condition, the speech is considered to have ended.

[0023] The above steps can effectively remove silent segments and environmental noise segments, ensuring that subsequent processing focuses on the actual speech content.

[0024] To enhance the expressive power of speech signals in noisy environments, this invention employs a composite noise reduction technique combining spectral subtraction and Wiener filtering. This technique integrates the advantages of frequency domain noise estimation and time domain smoothing.

[0025] Spectral subtraction noise reduction: First, a short-time Fourier transform is performed on the effective speech segments after endpoint detection to transform the speech signal from the time domain to the frequency domain; Then, the initial noise power spectrum is estimated based on the first 20 frames of silent segments; Next, the estimated noise spectrum is subtracted from the power spectrum of the current frame speech, and the result is forcibly truncated to a non-negative value to obtain the noise-reduced spectrum.

[0026] Wiener filtering enhancement: The spectral subtraction signal is then enhanced by Wiener filtering, and the filter coefficients are calculated by the ratio of the speech signal power spectrum to the noise power spectrum. In this invention, the Wiener filter gain function is defined as the signal-to-noise ratio divided by the reciprocal of the signal-to-noise ratio plus 1. This function can retain more information in the high-energy frequency band on the spectrum while suppressing the low-energy region.

[0027] Amplitude spectrum reconstruction and inverse transform: Phase compensation is performed on the denoised spectrum and inverse Fourier transform is executed to reconstruct the time-domain speech signal, thus completing the denoising process.

[0028] This composite noise reduction technology can effectively suppress the interference of background noise on high-frequency information and phoneme boundaries while preserving speech features.

[0029] To compensate for the natural attenuation of high-frequency energy during transmission and enhance the high-frequency components in the speech signal to make it more distinguishable in subsequent feature extraction, this invention pre-emphasizes the denoised speech signal.

[0030] The specific method is as follows: Perform a difference operation between each sampling point and its previous sampling point, with the difference coefficient set to 0.97; The processing formula is: the current output sample equals the current input sample minus 0.97 times the previous input sample; This processing is essentially a first-order high-pass filter, whose frequency response can significantly enhance the energy in the frequency band above 2000 Hz, while suppressing low-frequency redundant information.

[0031] Pre-emphasis processing not only enhances the clarity of speech boundaries but also improves the recognizability of high-frequency emotional features.

[0032] Speech signals are non-stationary signals, and can only be considered as quasi-stationary processes for a short period of time. Therefore, framing is necessary for subsequent feature extraction. This invention employs an overlapping framing method with a frame length of 25 milliseconds and a frame shift of 10 milliseconds, ensuring that each frame contains sufficient frequency resolution and guarantees the continuity of inter-frame transitions.

[0033] Framing steps: The sampling rate is set to 16 kHz, so each frame contains 400 sampling points, and the frame shift is 160 sampling points; the original speech sequence is extracted by a sliding window to form multiple adjacent and partially overlapping frame sequences.

[0034] Windowing: To reduce spectral leakage caused by discontinuities at frame endpoints, a Hamming window function is used to weight each frame; The Hamming window function is defined as follows: the window value is equal to 0.54 minus 0.46 multiplied by the cosine function, where the variable is the sample point index divided by the frame length and then multiplied by twice the value of pi; Windowing can effectively smooth frame boundaries and enhance spectral stability.

[0035] Finally, after frame segmentation and windowing, a set of processed speech frame sequences is obtained, and each frame can be regarded as a basic analysis unit containing stable emotional features.

[0036] Since the dynamic changes of speech signals vary significantly at different time resolutions, a single frame length may not be able to simultaneously capture both instantaneous emotional fluctuations and long-term trends. Therefore, this invention first constructs windows for the processed speech signal frame sequence at multiple time scales to capture multi-granular temporal features.

[0037] Specifically, the speech signal frames are constructed into frame windows according to the following three time scales: 20-millisecond frame length: used to capture fine-grained short-term changes, such as speech jitter and instantaneous tone changes; 40-millisecond frame length: reflects phoneme level changes and intonation shifts over a medium time range; 80-millisecond frame length: Covers a wider range of emotional profile trends, such as the smoothness of tone or the rise in mood between segments.

[0038] The frame shift was uniformly set to 10 milliseconds to ensure aligned timelines across the three scales, facilitating subsequent feature alignment and fusion. Hamming window weighting was used to reduce the impact of boundary discontinuities on spectral calculations.

[0039] At the three time scales mentioned above, Mel-frequency cepstral coefficients (MFCCs) of the speech signal are extracted. These coefficients are classic acoustic features that simulate the frequency characteristics perceived by the human ear. The extraction steps include: Fast Fourier Transform (FFT): Transforms each frame of signal from the time domain to the frequency domain to obtain the power spectral density of the frame; Mel filter bank processing: The spectrum is weighted by several Mel-scale bandpass filters. The Mel scale is a non-linear frequency division that mimics human hearing. In this invention, the number of filters is set to 26, covering the range of 300 Hz to 8000 Hz. Logarithmic transformation: Taking the natural logarithm of the filter output energy transforms the multiplicative relationship into an additive characteristic, enhancing stability; Discrete Cosine Transform (DCT): Performs a discrete cosine transform on a logarithmic energy sequence to extract a compressed representation of the energy.

[0040] Finally, the first 13 MFCC coefficients are extracted at each time scale, and their first-order difference (Δ) and second-order difference (ΔΔ) are further calculated to describe the changing trend and dynamic change capability of the features, resulting in a 39-dimensional feature vector (13-dimensional MFCC + 13-dimensional Δ + 13-dimensional ΔΔ).

[0041] In addition to MFCC, this invention also extracts short-time energy and zero-crossing rate features from the same frame window to further enrich the dimensions of acoustic features: The short-time energy is calculated by summing the squares of all samples within each frame, reflecting the energy intensity within the frame. This feature helps identify stress, pauses, and high emotional peaks in speech. The zero-crossing rate is calculated by counting the number of symbol changes between adjacent sampling points in each frame and dividing by the total number of sampling points, reflecting the activity of high-frequency signals. A high zero-crossing rate is usually associated with fast speech and intense emotions, while a low zero-crossing rate corresponds to a calm tone or a sad state.

[0042] These two features are also calculated at three time scales to enhance the expressive power of speech dynamic features at different frequency levels.

[0043] Because different scales and feature dimensions have different dimensions and inconsistent value ranges, this invention performs normalization on all features after feature extraction to improve subsequent recognition accuracy. The normalization method adopts Z-score standardization, which involves subtracting the mean of each feature value from the mean of the entire sample and then dividing by the standard deviation, resulting in a zero-mean, unit-variance form.

[0044] The normalized features are concatenated according to the time scale, specifically as follows: First, concatenate the feature vectors (including MFCC, Δ, ΔΔ, energy, and zero-crossing rate) at the 20-millisecond scale frame by frame; Next, the features at the 40-millisecond and 80-millisecond scales are connected; The final result is a unified multi-scale low-level acoustic feature set A1, with 117 dimensions (39 dimensions × 3 per scale) plus 6 dimensions of energy and zero-crossing rate per scale, for a total of 123 dimensions.

[0045] This set contains complete basic speech features at different time resolutions, has high expressive power, and can provide a high-quality input feature source for subsequent context modeling and emotion recognition.

[0046] The temporal structure input tensor is constructed based on the feature set A1 extracted in the previous steps. A1 is a low-level acoustic feature set at the frame level, containing feature information such as MFCC coefficients at multiple scales, their first and second order differences, short-time energy, and zero-crossing rate.

[0047] The frame-level features in A1 are organized according to their original order in the speech signal to form a time-series input tensor, denoted as X. The dimension of the input tensor is T × D, where: T represents the number of frames in the speech signal, i.e., the number of time steps; D represents the feature dimension of each frame (which can be 123 dimensions after multi-scale fusion as mentioned above).

[0048] When this tensor is used as input to the neural network, the frame order remains unchanged, ensuring that temporal dependencies are not compromised. This input provides complete contextual information about speech emotion along the time axis for subsequent temporal modeling.

[0049] To model the temporal context of the speech signal, this invention employs a Bidirectional Gated Recurrent Unit (BiGRU) structure. This structure consists of two independent GRUs, which model the input tensor in temporal order (from front to back) and in reverse temporal order (from back to front), respectively, resulting in two sets of hidden states: Forward GRU: Processes frames sequentially from frame 1 to frame T, capturing emotional changes in speech from beginning to end; Reverse GRU: It processes data from frame T back to frame 1, models the impact of future speech information on the semantics of the current frame, and makes up for the shortcomings of traditional unidirectional structures that cannot utilize future context.

[0050] Each GRU unit contains two core control gates: a reset gate and an update gate. The reset gate controls the degree to which the current input depends on the hidden state of the previous moment; Update the door control to retain how much historical information is currently hidden.

[0051] Let the input vector be The hidden state at the previous moment was ,but: The output of the reset gate is a vector between 0 and 1, which is calculated by adding a bias to the linear combination of the input and the previous state, followed by the sigmoid activation function. The update gate has the same structure as its predecessor, adjusting the information update frequency; The current hidden state is a non-linear combination of the previous hidden state and the current input. Specifically, a gating mechanism is used for weighted fusion to ensure that long-distance dependencies are not forgotten.

[0052] In this invention, the GRU hidden layer dimension is set to 256, that is, the output in each direction is T × 256, and after bidirectional concatenation, an intermediate feature representation sequence of T × 512 is formed, denoted as A2'.

[0053] To integrate bidirectional information, this invention concatenates the hidden states of each frame output by the forward GRU and the reverse GRU along the feature dimension, that is, it connects the forward state and the reverse state of the t-th frame to form the representation of the t-th frame in the global context.

[0054] This splicing operation significantly enhances the integrity of semantic expression, especially when recognizing complex emotions that require contextual support, such as "hesitant anger" or "suppressed sadness."

[0055] The resulting intermediate sequence feature set A2' is a T × 512 temporal feature tensor that maintains the temporal structure corresponding to the original frame sequence, which can be further processed by subsequent attention mechanisms.

[0056] To further extract the emotional expression information carried by keyframes in speech and form an emotional representation of the entire speech, this invention introduces a temporal attention mechanism to compress and aggregate features of A2'.

[0057] The attention mechanism calculates a weight value for each frame, representing the importance of that frame for emotion recognition within the entire speech segment. The specific steps are as follows: Weight calculation: A linear transformation is performed on the context representation vector of each frame (i.e., the weight matrix is ​​multiplied by the feature vector of that frame and a bias term is added), then processed by the tanh activation function, and finally normalized by the softmax function to obtain the attention weights for all frames. The sum of the weight values ​​is 1; Weighted summation: The representation vectors of all frames are summed according to their attention weights to obtain a global feature vector of fixed dimension, namely the sentiment representation feature set A2, which has a dimension of 1 × 512.

[0058] This step can not only automatically identify emotional keyframes, but also filter out frames with weak or interfering emotional expression, thereby improving the overall discrimination performance of the model.

[0059] The processed speech signal is transcribed into high-quality natural language text, which serves as the input for semantic layer analysis. To achieve accurate speech-to-text mapping, this invention employs an end-to-end speech recognition model based on joint training, which integrates the advantages of the CTC (Connectionist Temporal Classification) decoding mechanism and the Transformer structure, featuring strong modeling capabilities and high training efficiency.

[0060] Acoustic encoder: The input is a sequence of processed speech signal frames (such as Mel spectrograms or A1 features), and features are extracted from it using multi-layer one-dimensional convolution and position encoder. The output is a sequence of hidden state vectors.

[0061] CTC branch: Calculates time-aligned label probabilities through a CTC path, which is used to constrain the output sequence and prevent the prediction results from losing inter-frame synchronization.

[0062] The Transformer decoder branch employs a self-attention mechanism to encode speech features and prefix character prediction results, and models the output semantic space based on a cross-attention layer to achieve high-quality character prediction.

[0063] Joint loss training: The final model is jointly trained using a weighted average of CTC loss and Transformer cross-entropy loss, with the CTC loss weight set to 0.3 and the Transformer loss weight set to 0.7, to balance frame alignment accuracy and contextual consistency of output semantics.

[0064] This recognition model can provide robust text transcription output in noisy or complex speech scenarios, providing accurate input for subsequent semantic feature extraction.

[0065] Performing sentiment lexicon analysis on the transcribed text is the first layer of semantic understanding in the semantic feature modeling of this invention. This step aims to extract explicit emotional expression signals from the text and construct a sentiment tendency vector.

[0066] The sentiment lexicon is constructed as follows: It comprises four main categories: positive words, negative words, neutral words, and ambiguous words, totaling approximately 5,000 entries. Each entry includes metadata such as sentiment polarity value (ranging from -1 to 1), sentiment intensity (ranging from 0 to 1), and contextual dependency weight.

[0067] Word segmentation and matching: The text is segmented into words using Jieba or a Transformer-based word segmenter, and then matched word by word with dictionary entries, with polarity and intensity information annotated.

[0068] Statistical vector generation: Analyze the frequency distribution of sentiment-related words, such as the proportion of positive words and the proportion of negative words; Calculate the weighted average of emotional intensity; Extract features such as the magnitude of emotion changes (the difference between the maximum and minimum values); All statistics constitute a preliminary sentiment vector with a dimension of 10.

[0069] This sentiment vector reflects the distribution and intensity of explicit emotional words in a sentence, and is a quantitative description of the overall emotional tone of the sentence.

[0070] To address the numerous implicit emotions, context-dependent expressions, and non-emotional word-triggered emotional cues present in emotional expression, this invention employs a deep semantic embedding model to construct contextual emotion vectors, thereby supplementing the limitations of dictionary analysis.

[0071] Model selection: Use a pre-trained Chinese semantic embedding model, such as BERT or RoBERTa, with no less than 100 million parameters and deep context modeling capabilities.

[0072] Embedding generation: Input the entire text into the model and generate a 768-dimensional semantic vector for each word. The overall output dimension is N×768, where N is the number of words.

[0073] Context aggregation methods: Average pooling: averaging all word vectors position by position to obtain the overall semantic representation of the sentence; Max pooling: Selects the maximum activation value in each dimension to enhance the significant semantic features in the sentence; By concatenating or weighting average pooling and max pooling, a contextual semantic sentiment vector with a dimension of 768 × 2 is formed.

[0074] This embedding vector can identify complex emotions expressed in context in texts such as "I can still hold on, but I'm really tired," providing strong support for deep sentiment classification.

[0075] To further enhance the understanding of the relationship between syntactic structure and emotional modification, this invention extracts semantic dependency relationships at the text level and extracts deep-level sentence structure features by constructing a semantic dependency graph.

[0076] Dependency parsing tools: Use open-source semantic dependency parsing tools (such as LTP or Stanford CoreNLP Chinese dependency parser) to identify subject-verb-object and modification dependency relations in the transcribed text.

[0077] Dependency graph construction: Construct sentences into dependency graphs, where nodes represent words, edges represent dependency relationships, and the relationship types are labeled (such as subject-verb relationship, adverbial modification, adjective modification, etc.).

[0078] Feature extraction: Extract the sentiment word paths between the subject, predicate, and object; Determine whether the emotion modifier is located near the core verb; Mark whether there are mood change points in the grammatical structure (such as mood reversal structures introduced by conjunctions such as "but" or "yet").

[0079] Graph vector representation: The dependency structure is encoded as a graph structure vector with a dimension of 64, including structural features such as dependency path depth, shortest path length from key sentiment words to the beginning / end of the sentence, and modification relation density.

[0080] The vectors generated from the three semantic information sources are fused together to form the final semantic feature set A3, where: Sentiment vector dimension: 10; Semantic embedding vector dimension: 1536 (768 average pooling + 768 max pooling); Dependency structure vector dimension: 64; The total dimensions are 1610.

[0081] The fusion method employs vector concatenation to preserve the independence and integrity of each information source. The semantic feature set A3 can be used in parallel with the acoustic feature A2 for feature fusion, providing multi-dimensional and context-rich input support for subsequent emotion recognition models.

[0082] Since the acoustic feature set A2 and the semantic feature set A3 originate from different sources, their original dimensions are usually inconsistent, and their distribution spaces differ. Direct fusion would lead to feature conflicts and dimensional mismatches. Therefore, it is necessary to map the two feature sets to a unified fusion space dimension.

[0083] Input definition: Feature set A2 ∈ T is the number of frames, and D1 is the acoustic feature dimension; Feature set A3 ∈ D2 is the semantic feature dimension, which is usually a sentence-level vector.

[0084] Mapping method: Construct two linear transformation networks (i.e., fully connected networks) respectively, and perform dimensional transformations on A2 and A3 respectively; The mapped features are denoted as A2′ and A3′, respectively, and their target dimension is uniformly d (set to 512 in this invention).

[0085] Network structure: Each mapping network contains one fully connected layer with weight matrices of sizes D1×d and D2×d, and the activation function is ReLU; The mapping function takes the form of: the output equals the input multiplied by the weight matrix plus the bias, and then processed by the ReLU activation function.

[0086] After this step, the acoustic and semantic modalities are projected into the same feature space, providing a structural basis for subsequent attention alignment.

[0087] To capture the explicit correspondence between acoustic and semantic features, as well as sentiment relevance, this invention employs a multi-head attention mechanism for modal alignment. Specifically, acoustic feature A2′ is used as the query vector, and semantic feature A3′ is used as the key and value vectors to construct an adaptive attention matrix, learning the dynamic weight distribution among cross-modal features.

[0088] Attention core mechanism: query vector for each frame (From frame t in A2′), calculate its attention weight with the key vector k (from A3′). The calculation method is as follows: First, calculate the dot product similarity ( (inner product with k). Then divide by the scaling factor (√d) to prevent gradient explosion; Finally, the attention distribution α is obtained by Softmax normalization.

[0089] Weighted summation generates the fused representation: The weighted semantic representation of each frame is as follows: ; where v is a value vector, i.e., A3′.

[0090] Multi-head extension: To enhance the model's ability to capture various semantic-acoustic dependencies, a multi-head attention mechanism is used to construct multiple subspaces in parallel. Each subspace independently learns attention weights. Finally, the outputs of all subspaces are concatenated and restored to dimension d through a linear transformation, forming a preliminary fused feature set A4′ ∈ .

[0091] In order to preserve the structural information of the original modalities and improve the expressive power of the network during the fusion process, this invention introduces residual connections and nonlinear reconstruction mechanisms to perform deep processing on the fusion feature A4′.

[0092] Residual connection: The preliminary fused feature A4′ is concatenated with the mapped A2′ and A3′ respectively or the elements are weighted and merged to achieve feature information compensation and original modality preservation.

[0093] Reconstruction of feedforward neural networks: A two-layer feedforward neural network is used to nonlinearly reconstruct the fused features; The first layer maps the input to a high dimension (e.g., 1024 dimensions) and uses ReLU activation; The second layer compresses it back to the target dimension d (i.e., 512 dimensions), forming the final reconstructed feature sequence.

[0094] This process enhances nonlinear modeling capabilities, enabling fused features to possess stronger contextual abstraction and emotional expression capabilities.

[0095] In order to convert the temporal fusion feature sequence A4′ into a fixed-length sentence-level vector for subsequent sentiment classification, this invention designs an adaptive pooling mechanism to compress the sequence.

[0096] Pooling methods: Adaptive average pooling: averaging each dimension across all time steps to capture the overall feature distribution trend; Adaptive max pooling: Extracts the maximum value of each dimension in the time series to highlight key frame information.

[0097] Vector concatenation: The average pooling result is concatenated with the max pooling result to obtain the final high-order fusion vector A4∈ of dimension 2d. .

[0098] The higher-order emotion fusion feature set A4 is a context-aware feature representation obtained by aligning and fusing acoustic feature A2 and semantic feature A3 through a cross-modal attention mechanism. Its dimension is 1×1024, meaning that a single sample contains a 1024-dimensional fusion feature vector.

[0099] Residual network structure: To enhance the modeling capability for nonlinear feature distributions, this invention designs a residual network structure consisting of three fully connected layers, each of which contains the following structure: Linear mapping layer: Maps the input features to the same dimension d (set to 512 in this embodiment) through the weight matrix and the bias term. Activation function: ReLU (rectified linear unit) is used as the nonlinear activation function to enhance the model's expressive power; Residual connection mechanism: Skip connection performs element-wise addition on the input vector and the activated output to prevent gradient vanishing and degradation problems.

[0100] Explanation of residual formula: Let the input of a residual unit be vector x, the mapping weight be W1, and the bias be b1, then the output is: Residual output = ReLU(W1·x + b1) + x; Each residual block output is normalized to maintain numerical stability. The final residual network output is a sequence of intermediate feature vectors of dimension d, denoted as A4′∈ .

[0101] In actual speech emotion expression, even if A4 is a fused single-sentence level feature, it may still retain information from multiple frame-level time segments. To model potential time dependencies and improve the sensitivity of recognition to emotion development trends, gated recurrent units (GRUs) are further used to model the feature sequence.

[0102] GRU structure parameter settings: The GRU unit employs a bidirectional GRU structure to capture the semantic dependencies of information in both forward and backward time. The hidden state dimension of the one-way GRU is set to 128, and the output dimension of the two-way GRU is 256 after merging. If the input is in sequence form (such as multiple A4′ sub-vectors generated by sliding window or segmented concatenation), then the input shape is T×512, where T is the number of time steps.

[0103] GRU information processing flow: The GRU unit processes the input at each time step as follows: Reset Gate: Controls the interaction strength between the current input information and the previous hidden state; Update Gate: Controls the fusion ratio of previous and new information in the current hidden state; Output hidden state: After fusing the input and state, the hidden state of the current time step is generated, which represents the semantic context state.

[0104] Contextual Output Vector: The GRU structure transforms the input sequence into a context-sensitive feature vector, denoted as z∈ This is used for subsequent classification operations.

[0105] Output layer construction: Input the context feature vector output by GRU into a fully connected classification layer, and the output dimension is equal to the preset number of sentiment categories K.

[0106] In this invention, K is set to 7, representing 7 fine-grained emotion categories: neutral, joy, anger, sadness, surprise, anxiety, and indifference.

[0107] Probability Normalization: After classification, the output is normalized using the Softmax function, mapping the original scores to probability distributions across sentiment categories. The Softmax function works by taking the exponent of the score for each category and then dividing by the sum of the exponents of all categories to obtain the normalized probability. The expression is: , where i is the current category, and j iterates through all categories.

[0108] Predicted output: Select the category with the highest predicted probability as the final sentiment label; if the highest probability is lower than the set confidence threshold θ (θ = 0.35 in this invention), an auxiliary processing mechanism is triggered, such as a secondary model judgment or output of an "uncertain" label, to improve system stability and fault tolerance.

[0109] Emotional label output: Based on the Softmax output results above, the model can accurately predict the emotional category of the input speech and output labels such as "pleasure", "anger", "sadness" etc.

[0110] This model architecture supports flexible expansion of the number of emotion tags. It can recognize a variety of fine-grained emotional states (such as "restrained anger" and "guilty sadness") simply by adjusting the number of output layer nodes and the labeling of training samples.

[0111] To determine the reliability of the classifier's output, a fixed confidence threshold θ is first set to filter the output probabilities. In this invention, θ is a floating-point number between 0 and 1, used to measure the classifier's confidence in its prediction results.

[0112] Definition: Let the output of the model classifier be a probability vector P ∈ , where K represents the total number of emotion categories; Take the maximum value ; If p max If the confidence level is less than θ, the current judgment is considered insufficient, and a supplementary judgment process needs to be initiated.

[0113] Typical threshold: In this invention, θ = 0.35 is set by default. After extensive experimental verification, this value has achieved a good balance between precision and recall. This threshold can be flexibly adjusted according to the scenario. For example, it can be reduced to 0.30 for psychological counseling tasks to improve sensitivity, and increased to 0.45 for business interaction tasks to improve decision conservatism.

[0114] To compensate for the lack of expressive power in current speech segments due to the absence of a complete semantic structure, this invention introduces a contextual semantic modeling method. By combining the content of the sentences before and after the current sentence to construct a complete context, the accuracy of judgment is improved.

[0115] Context statement selection: In the text transcription sequence, select the sentence preceding the current speech sentence. With the next sentence ; If the current segment is the first or last sentence, only one available direction will be selected.

[0116] Embedded representation generation: using a Transformer-class contextual language model consistent with the main model (such as BERT-base) Current sentence , Perform separate encoding; The [CLS] bit output is used as the sentence-level semantic representation, with an embedding dimension of 768. The embedding results of the three sentences are concatenated or weighted to form a complete contextual semantic feature vector.

[0117] Increased sentimentality: right and The sentence is matched with a sentiment dictionary, and the proportion and polarity of positive and negative sentiment words are statistically analyzed. The statistical results are concatenated with the sentence embedding vector to form a contextual semantic enhancement vector, with a final dimension of 768 + N, where N is the dictionary statistical dimension (e.g., 10 dimensions).

[0118] To capture the trajectory of the speaker's emotional changes during multiple rounds of voice interaction, this invention constructs a historical emotional trend sequence and analyzes the evolution of emotions to assist in current judgment.

[0119] History window settings: Define a fixed window length H (e.g., H = 5) to represent a maximum of 5 rounds of voice interaction that can be traced back. For each round of interaction, obtain the sentiment prediction label and confidence level output by the main model.

[0120] Trend feature construction: Emotional directionality: Statistically measure the direction of emotional change (e.g., from negative to positive), quantified by +1, 0, and -1; Confidence fluctuation frequency: This measures the frequency of switching between high and low confidence levels to assess emotional stability. Extreme value extraction: Extract the emotion category with the highest confidence in historical emotions as the emotional peak; Trend curve encoding: Embed the label sequence into a trend vector and extract the temporal pattern through one-dimensional convolution.

[0121] The final output vector is a fixed-length vector with a dimension of 64, which, together with the context semantic vector, serves as a supplementary feature input to the subsequent discriminative network.

[0122] To avoid introducing excessive model computation, this invention uses a lightweight fully connected network to perform a second fusion and judgment of multimodal features, thereby improving real-time performance and deployment efficiency.

[0123] Input concatenation: The following three types of features are concatenated to form the fused input vector Z: The high-order fusion feature A4 (1024 dimensions) of the current segment; Contextual semantic feature vector (768–778 dimensions); Historical sentiment trend characteristics (64 dimensions); The final input vector dimension is approximately 1856–1866.

[0124] Secondary judgment network structure: Includes two fully connected layers: First layer: Input dimension is approximately 1856, output dimension is 512, ReLU activation is used; The second layer maps the 512 dimensions to a K-dimensional output layer, representing the scores for each sentiment category; Normalization is performed using Softmax to output a new sentiment probability distribution.

[0125] Final judgment output: If the new maximum probability is still lower than θ (i.e., p) max If < θ), the current speech segment is marked as "uncertain emotion"; subsequent processing strategies can be configured, such as transferring to manual review, generating confidence prompts, or retaining emotion-neutral processing, etc. If p max If the value is greater than or equal to θ, then the new prediction result is accepted as the final output sentiment label.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A speech emotion recognition method based on multi-feature fusion, characterized in that: include: The original speech signal is preprocessed to remove silent segments and background noise, and then windowed in frames to obtain the processed speech signal sequence. A multi-scale low-level acoustic feature set is extracted from the processed speech signal sequence, including Mel frequency cepstral coefficients, spectrogram, zero-crossing rate, speech energy and its first and second order difference features, denoted as feature set A1; Based on the feature set A1, a temporal dependency model is constructed, and context-related feature sequences are extracted to obtain the sentiment representation feature set A2 after temporal modeling. Semantic features are extracted in parallel, including sentiment tendency features extracted based on sentiment dictionary and semantic embedding model after the speech signal is transcribed into text, as well as semantic dependency relationship features, which constitute the semantic feature set A3; Cross-modal alignment and fusion of feature sets A2 and A3 are performed, and weight relationships between multiple features are established through an adaptive attention mechanism to generate a unified high-order sentiment fusion feature set A4. The fused high-order sentiment feature set A4 is input into the sentiment recognition and classification model. A multi-layer residual network and gated recurrent unit architecture are used to distinguish sentiment categories and output fine-grained sentiment labels. If the confidence level of the emotion recognition classification model is lower than the set threshold, then further fusion judgment is performed by combining contextual semantics and historical emotion trend features.

2. The speech emotion recognition method based on multi-feature fusion according to claim 1, characterized in that: in, Preprocessing of the raw speech signal includes: Endpoint detection is performed on the original speech signal. An improved short-time energy and zero-crossing rate joint decision method is used to locate the speech start and end points, remove silence segments, and extract effective speech segments. A composite noise reduction algorithm combining spectral subtraction and Wiener filtering is applied to the intercepted effective speech signal to suppress background noise while preserving the frequency characteristics of the speech signal and improving the signal-to-noise ratio. The noise-reduced speech signal is pre-emphasized, and a first-order high-pass filter is used to enhance the high-frequency components in order to compensate for the high-frequency attenuation during the transmission of the vocal tract. The pre-emphasized speech signal is divided into frames with a frame length of 25ms and a frame shift of 10ms. A Hamming window function is applied to each frame to smooth the frame boundaries and suppress spectral leakage, resulting in the processed speech signal frame sequence.

3. The speech emotion recognition method based on multi-feature fusion according to claim 2, characterized in that: in, The extraction of multi-scale low-level acoustic feature sets from the processed speech signal sequence includes: Multi-scale frame windows were constructed on the processed speech signal frame sequence at three time scales of 20 milliseconds, 40 milliseconds and 80 milliseconds to obtain the speech dynamic characteristics at different time resolutions. At each time scale, Mel frequency cepstral coefficients are extracted, including the first to 13th order coefficients and their first and second order difference features. The Mel frequency cepstral coefficient MFCC calculation process includes: performing fast Fourier transform on the windowed frame, bandpass filtering through the Mel filter bank, taking the logarithmic energy and then performing discrete cosine transform to obtain the nonlinear compressed spectrum features of the speech. Based on the same frame window, the short-time energy and zero-crossing rate features of each frame are extracted. The short-time energy is obtained by calculating the sum of squares of the amplitudes of all sampling points within the frame, and the zero-crossing rate is obtained by dividing the number of statistical symbol changes by the number of sampling points, which are used to characterize the trend of speech intensity and frequency changes. The MFCC, first-order difference, second-order difference, short-time energy and zero-crossing rate features extracted at all scales are normalized and cascaded according to time scale to form a unified low-level acoustic feature set A1, which is used to reflect the emotional changes of speech signals at different time scales.

4. The speech emotion recognition method based on multi-feature fusion according to claim 3, characterized in that: in, The obtained sentiment representation feature set A2 after time-series modeling includes: The feature set A1 is constructed into a time series input tensor according to the frame order of the speech signal, while maintaining the time alignment order of the original frame order. This tensor is used as the modeling input, and its dimension is T×D, where T represents the number of frames and D represents the feature dimension of each frame. A bidirectional gated recurrent unit network (BiGRU) is constructed, which includes two parallel channels: forward GRU and backward GRU. These channels model the feature sequence from front to back and from back to front, respectively, to capture the contextual information. The GRU unit controls the state memory and forgetting mechanism through reset gate and update gate. The forward and backward hidden states of the BiGRU network are concatenated to form the context-sensitive feature representation of each frame in the global context, resulting in the intermediate sequence feature set A2'. A temporal attention mechanism is applied to the intermediate sequence feature set A2' to calculate the importance weight of each frame to the overall speech segment emotion. The features are then weighted and summed according to the attention weights to form a fixed-dimensional temporal context emotion representation feature set A2.

5. The speech emotion recognition method based on multi-feature fusion according to claim 4, characterized in that: in, The parallel extraction of semantic layer features includes: The processed speech signal is subjected to automatic speech recognition processing. An end-to-end speech recognition algorithm based on the fusion of deep acoustic model and language model is used to transcribe the speech signal into corresponding text information. The recognition model adopts the jointly trained CTC-Transformer architecture. The transcribed text sequence is matched with a sentiment dictionary. A multidimensional sentiment dictionary is used to segment and annotate the text, and the word frequency distribution, sentiment polarity distribution and word class intensity statistical features are extracted to form a preliminary sentiment tendency vector. The text is input into a pre-trained semantic embedding model, and a Transformer-based contextual language model is used to generate semantic embedding representations for each word. The contextual sentiment semantic vector of the whole sentence is extracted by average pooling and max pooling methods. Dependency syntactic analysis is performed on the text to construct a semantic dependency graph. Based on syntactic dependency relations, the dependency paths of subject-verb-object structure, modification structure, and sentiment modifiers and core words are extracted, and structured semantic dependency features are generated. Finally, these features are fused with sentiment vectors and semantic embedding vectors to form a semantic feature set A3.

6. The speech emotion recognition method based on multi-feature fusion according to claim 5, characterized in that: in, The cross-modal alignment and fusion of the A2 and A3 feature sets includes: The emotional representation feature set A2 and the semantic feature set A3 are subjected to dimensionality standardization. A fully connected network is used to linearly map each feature to unify their feature dimensions to the preset fusion space dimension d. The dimensionally standardized A2 and A3 are input into a multi-head adaptive attention mechanism to construct a cross-modal fusion representation, where A2 is used as the query vector and A3 is used as the key vector and value vector. The attention mechanism is used to calculate the attention distribution of acoustic context features to semantic features and weighted aggregation to obtain the preliminary fusion feature representation A4′. The preliminary fused feature A4′ is residually connected with the original features A2 and A3, and then the feature is reconstructed through a feedforward fully connected network with ReLU activation function to generate a fused feature vector sequence. The fused sequence is compressed using a pooling mechanism, and then a combination of adaptive average pooling and max pooling is used to obtain a fixed-length unified high-order sentiment fusion feature set A4.

7. The speech emotion recognition method based on multi-feature fusion according to claim 6, characterized in that: in, The step of inputting the fused high-order sentiment feature set A4 into the sentiment recognition and classification model includes: The high-order emotion fusion feature set A4 is input into a multi-layer residual fully connected network. The residual network consists of several linear transformation layers. Each layer includes a fully connected layer with an activation function and a skip connection structure. The output of the residual layer is a feature vector sequence of dimension d. The feature vector sequence output by the residual network is input into a single-layer or multi-layer gated recurrent unit (GRU) to further model the long-distance dependencies between features using its temporal modeling capability. The GRU unit adopts a bidirectional structure with a hidden state dimension of 128 and outputs context-dependent feature representation vectors. The GRU output vector is mapped to the same output dimension as the preset number of sentiment categories through a fully connected layer, and then normalized using the Softmax function to obtain the predicted probability distribution of each sentiment category; The final sentiment recognition result is output based on the category label corresponding to the maximum value in the sentiment prediction probability.

8. The speech emotion recognition method based on multi-feature fusion according to claim 7, characterized in that: in, The multi-round fusion judgment includes: A confidence threshold θ is set. When the highest category probability output by the emotion recognition classification model is lower than the confidence threshold θ, a supplementary discrimination process is triggered. Extract the contextual information of the current speech segment in the semantic sequence, including the semantic embedding representation of the two sentences before and after it and the sentiment word tendency features. Encode it based on BERT or contextual language model to form a contextual semantic feature vector, which is used to supplement the semantic missing parts in the current judgment. Construct a historical emotion trend sequence, statistically analyze the emotion prediction labels and confidence levels of the current speaker in the previous rounds of voice interaction, calculate its emotion change curve and fluctuation frequency, and extract the emotion stability index, trend directionality and maximum emotion value as historical feature vectors; The high-order fusion feature A4 of the current segment, the contextual semantic feature vector, and the historical sentiment trend vector are concatenated and input into a lightweight two-layer fully connected network for further judgment. The updated sentiment category and confidence level are output. If they are still lower than the confidence level threshold θ, they are marked as "uncertain sentiment" or submitted to human confirmation.

Images