Speech emotion analysis method and system

Abstract

The invention discloses a voice emotion analysis method and system, and the method comprises the steps: 1, collecting training voice data, and carrying out the feature extraction of the training voice data, and obtaining a training comprehensive feature matrix; 2, inputting the training comprehensive feature matrix into a preset attention enhancement residual learning model to obtain prediction probability distribution of emotion categories, and obtaining predicted emotion categories according to the prediction probability distribution of the emotion categories; 3, training a preset attention enhancement residual learning model according to the classification cross entropy loss function and a label smoothing technology to obtain an optimal attention enhancement residual learning model; and 4, performing feature extraction on the target voice data to obtain a target comprehensive feature matrix, and inputting the target comprehensive feature matrix into the optimal attention enhancement residual learning model to obtain an emotion category. According to the method, the complex mode of the emotional voice can be captured, and the accuracy and robustness of voice emotion recognition are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech emotion recognition, and particularly relates to a speech emotion analysis method and system. BACKGROUND

[0002] As a highly potential research field, speech emotion analysis has important application value in many fields such as human-computer interaction, intelligent customer service, and mental health monitoring. In recent years, significant progress has been made in related work.

[0003] In the aspect of feature extraction, researchers are constantly exploring new feature types to improve the ability to capture speech emotion information. In addition to traditional acoustic features such as MFCCs, some time-frequency features and prosodic features are also widely used. For example, time-frequency features extracted by wavelet transform can more meticulously depict the changes of speech signals at different times and frequencies, providing more abundant information for emotion recognition. At the same time, some researches begin to focus on the nonlinear features of speech signals, such as fractal dimension, to better reflect the complexity and irregularity of speech signals.

[0004] In the aspect of model construction, the development of deep learning technology brings new breakthroughs to speech emotion analysis. Convolutional Neural Networks (CNN) are widely used in speech emotion feature extraction and classification due to their excellent performance in image and speech processing. Recurrent Neural Networks (RNN) and its variants, such as Long Short-Term Memory Networks (LSTM) and Gated Recurrent Units (GRU), can effectively process sequence data and capture the timing information in speech signals, achieving good results in speech emotion analysis. In addition, the introduction of attention mechanism has also become a research hotspot, which can help the model automatically focus on the key parts of the speech signal and improve the accuracy of emotion recognition.

[0005] Despite the above progress, the current technology still has many limitations. Traditional speech emotion analysis methods rely on limited traditional acoustic features such as Mel Frequency Cepstral Coefficients (MFCCs) in feature extraction, which is difficult to fully capture the complex patterns and emotion information in speech signals. In model construction, some models lack effective attention mechanisms to focus on key features, leading to insufficient mining of emotion features, and the recognition accuracy and generalization ability need to be improved. In addition, some models are prone to overfitting problems and perform poorly when processing new speech data. These problems limit the further development and wide application of speech emotion analysis technology, therefore, an effective speech emotion analysis method and system are needed to improve the recognition performance and reliability. SUMMARY

[0006] In order to solve the problems that the existing voice emotion technology is difficult to capture the complex mode of voice emotion and the model is insufficient in feature representation, generalization ability and overfitting, the present application provides a voice emotion analysis method and system, which can capture the complex mode of emotional voice, effectively improve the accuracy and robustness of voice emotion recognition, enhance the generalization ability of the model and alleviate the overfitting phenomenon by introducing Katz fractal dimension for complex feature extraction, constructing an attention-enhanced residual learning model and using a smooth loss function, thereby providing strong support for the wide application of voice emotion analysis in human-computer interaction, medical treatment, entertainment and other fields.

[0007] In order to achieve the above-mentioned purpose, the technical scheme of the present application is:

[0008] The present application provides a voice emotion analysis method, which comprises the following steps:

[0009] Step 1: Collecting training voice data, performing feature extraction on the training voice data to obtain a training comprehensive feature matrix, so as to comprehensively capture the complex mode and emotional information in the training voice data;

[0010] Step 2: Inputting the training comprehensive feature matrix into a preset attention-enhanced residual learning model to obtain a prediction probability distribution of the emotional category, and obtaining a predicted emotional category according to the prediction probability distribution of the emotional category; wherein the attention-enhanced residual learning model comprises an information encoding module, which is used to convert the input comprehensive feature matrix into a high-level representation form;

[0011] Step 3: Training the preset attention-enhanced residual learning model according to a classification cross-entropy loss function and a label smoothing technique to obtain an optimal attention-enhanced residual learning model, so as to improve the accuracy of the model prediction result;

[0012] Step 4: Performing feature extraction on target voice data to obtain a target comprehensive feature matrix, inputting the target comprehensive feature matrix into the optimal attention-enhanced residual learning model to obtain an emotional category.

[0013] Further, the step 1 specifically comprises:

[0014] Performing feature extraction on the training voice data to obtain complex features and traditional acoustic features, and constructing a comprehensive feature matrix according to the complex features and the traditional acoustic features; the complex features comprise Katz fractal dimension, and the traditional acoustic features comprise mel-frequency cepstral coefficient, first-order difference and second-order difference of mel-frequency cepstral coefficient, zero-crossing rate, chroma feature, spectral contrast, spectral centroid, spectral roll-off, tonal feature and spectral bandwidth, so as to comprehensively capture the complex mode and emotional information in the training voice data.

[0015] Furthermore, the Katz fractal dimension is calculated according to the following formula:

[0016]

[0017] d=max(|x i -x0|)

[0018] Where D is the Katz fractal dimension, L is the total length of the signal path, and d is x i The maximum distance from x0, n is the number of points in the signal, x i is the i-th point in the signal, x0 is the starting point of the signal, Δx i is the distance between two adjacent points in the signal.

[0019] Furthermore, the attention-enhanced residual learning model also includes a multi-head attention module, a residual module and an output layer;

[0020] The multi-head attention module is used to focus on different positions of the input features and capture diverse contextual relationships;

[0021] The residual module is used to capture local patterns and enhance feature propagation;

[0022] The output layer is used to convert the processed features into emotion category probabilities.

[0023] Furthermore, the information encoding module includes a fully connected layer, a batch normalization, a leakage rectified linear unit activation function, and a random dropout function connected in sequence;

[0024] The leakage-corrected linear unit activation function is used to introduce nonlinear features into the information encoding module, so that the model can learn more complex feature relationships.

[0025] Furthermore, the residual module includes multiple convolutional layers, an adjustment convolutional layer, and a residual output layer, so that the model can more efficiently capture the complex features in the speech signal and enhance the feature propagation effect;

[0026] The multiple convolutional layers have the same structure, and the convolutional layers are expressed by the following formula:

[0027]

[0028] Among them, H i+1 is the output of the i+1th convolutional layer, * is the convolution operation, σ is the ReLU activation function, and are the weights and biases of the convolutional layer respectively;

[0029] The adjusted convolution layer is expressed as follows:

[0030]

[0031] wherein H adj is the output of the adjusted convolutional layer, adj is the weight of the adjusted convolutional layer, is the input of the residual module, b adj is the bias of the adjusted convolutional layer.

[0032] The residual output layer is expressed by the following formula:

[0033] H residual = H7+ H adj

[0034] wherein H residual is the output of the residual output layer, H7 is the output of the last convolutional layer, and H adj is the output of the adjusted convolutional layer.

[0035] Further, the output layer is expressed by the following formula:

[0036] H flatten = Flatten(H residual )

[0037] Y = Softmax(W out H flatten +b out )

[0038] wherein H flatten is the flattened feature, Flatten is the flattening process, H residual is the output of the residual output layer, Y is the predicted probability distribution of the emotion category, W out and b out are the weight and bias of the output layer, respectively.

[0039] The second aspect of the present application provides a speech emotion analysis system, comprising:

[0040] a feature extraction module, configured to collect training speech data, perform feature extraction on the training speech data to obtain a training comprehensive feature matrix, and facilitate comprehensive capture of complex patterns and emotion information in the training speech data;

[0041] a prediction module, configured to input the training comprehensive feature matrix into a preset attention-enhanced residual learning model to obtain a predicted probability distribution of an emotion category, and obtain a predicted emotion category according to the predicted probability distribution of the emotion category; wherein the attention-enhanced residual learning model comprises an information encoding module, and the information encoding module is configured to convert the input comprehensive feature matrix into a high-level representation form.

[0042] The training module is used for training the preset attention-enhanced residual learning model according to a classification cross-entropy loss function and a label smoothing technique, so as to obtain an optimal attention-enhanced residual learning model, and facilitate to improve the accuracy of the prediction result of the model.

[0043] The classification module is used for performing feature extraction on the target voice data to obtain a target comprehensive feature matrix, inputting the target comprehensive feature matrix into the optimal attention-enhanced residual learning model, and obtaining the emotion category.

[0044] The third aspect of the present application provides an electronic device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the voice emotion analysis method according to the first aspect when executing the computer program.

[0045] The fourth aspect of the present application provides a computer readable storage medium, comprising a stored computer program, wherein the device where the storage medium is located executes the voice emotion analysis method according to the first aspect when the computer program runs.

[0046] The beneficial effects of the present application are as follows:

[0047] The present application introduces Katz fractal dimension to extract complex features, which facilitates to obtain complex features in the voice. Furthermore, the attention-enhanced residual learning model is constructed to accurately recognize the voice emotion, and finally the smoothing loss function is used to obtain more accurate results. The present application effectively improves the accuracy and robustness of voice emotion recognition, enhances the generalization ability of the model, and alleviates the overfitting phenomenon. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A flow chart of a voice emotion analysis method provided by the embodiment of the present application.

[0049] Figure 2 A schematic diagram of a voice emotion analysis method provided by the embodiment of the present application.

[0050] Figure 3 A schematic diagram of a feature extraction process provided by the embodiment of the present application.

[0051] Figure 4 A schematic diagram of an attention-enhanced residual learning model provided by the embodiment of the present application.

[0052] Figure 5 A schematic diagram of a hardware building operation step provided by the embodiment of the present application.

[0053] Figure 6This is an architectural diagram of a speech emotion analysis system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0055] Example 1

[0056] like Figure 1 and Figure 2 As shown, a speech emotion analysis method includes:

[0057] S101: Collect training speech data, and perform feature extraction on the training speech data to obtain a training comprehensive feature matrix.

[0058] Specifically, Katz fractal dimension (KFD) is used for complex feature extraction and combined with traditional acoustic features to obtain a comprehensive feature matrix. Traditional acoustic features include Mel-frequency cepstral coefficients (MFCCs), first-order differences and second-order differences of MFCCs, zero-crossing rate (ZCR), chromaticity features, spectral contrast, spectral centroid, spectral roll-off, modal features, and spectral bandwidth.

[0059] Katz fractal dimension is calculated as follows:

[0060]

[0061] d=max(|x i -x0|)

[0062] Where D is the Katz fractal dimension, L is the total length of the signal path, and d is x i The maximum distance from x0, n is the number of points in the signal, x i is the i-th point in the signal, x0 is the starting point of the signal, Δx i is the distance between two adjacent points in the signal.

[0063] Preferably, the traditional acoustic features can be calculated and extracted according to existing technologies.

[0064] S102: input the training comprehensive feature matrix into a preset attention-enhanced residual learning model to obtain a prediction probability distribution of the sentiment category, and obtain a predicted sentiment category according to the prediction probability distribution of the sentiment category; wherein the attention-enhanced residual learning model comprises an information encoding module, and the information encoding module is configured to convert the input comprehensive feature matrix into a high-level representation form.

[0065] Specifically, the attention-enhanced residual learning model comprises an information encoding module, a multi-head attention module, a residual module and an output layer.

[0066] The information encoding module is configured to convert the input comprehensive feature matrix into a high-level representation form. The multi-head attention module is configured to focus on different positions of the input features and capture diversified context relationships. The residual module is configured to capture local patterns and enhance feature propagation. The output layer is configured to convert the processed features into a sentiment category probability.

[0067] S103: training the preset attention-enhanced residual learning model according to a classification cross-entropy loss function and a label smoothing technique to obtain an optimal attention-enhanced residual learning model.

[0068] Specifically, a smoothed classification cross-entropy loss function is adopted, and a label smoothing technique is used to adjust the real label, and the original one-hot encoding label y i,j is modified to enhance the generalization ability of the model and alleviate the overfitting problem. The label smoothing technique is used to adjust the real label, and the original one-hot encoding label y i,j is modified, and is specifically represented by the following formula:

[0069]

[0070] wherein α is a smoothing parameter (usually set to 0.1), C represents the total number of sentiment categories, y′ i,j is the modified y i,j , y i,j is the real label (one-hot encoding form) of the i th sample belonging to the j th category (if the i th sample belongs to the j th category, then 1, otherwise 0).

[0071] Preferably, the application also uses an Adam optimizer to adjust the learning rate based on the estimation of the gradient matrix, which is defined as follows:

[0072]

[0073] wherein, is the classification cross-entropy loss, N is the total number of samples, is the prediction probability of the i th sample belonging to the j th category, η is the learning rate, m t and v trespectively, are the estimated values of the first and second moments of the gradient, and t+1 is the gradient information at the t+1 time, and t is the gradient information at the t time.

[0074] In the above formula, the logarithmic operation is used to measure the logarithmic difference between the true label and the predicted value, while ensuring numerical stability.

[0075] S104: Feature extraction is performed on the target voice data to obtain a target comprehensive feature matrix, the target comprehensive feature matrix is input into the optimal attention-enhanced residual learning model, and a sentiment category is obtained.

[0076] Specifically, feature extraction is performed on the target voice data to obtain a target comprehensive feature matrix, the target comprehensive feature matrix is input into the optimal attention-enhanced residual learning model, a prediction probability distribution of the sentiment category corresponding to the target voice data is obtained, and the sentiment category is obtained according to the prediction probability distribution of the sentiment category corresponding to the target voice data.

[0077] The present application combines complex features and traditional acoustic features by constructing a comprehensive feature matrix, which facilitates comprehensive capture of complex patterns and emotional information in voice data. An attention-enhanced residual learning model is constructed, and the residual module and information encoding module enhance the utilization of context information. Through the multi-head attention module, the model can effectively focus on key features, improving the recognition accuracy and generalization ability of the model.

[0078] The present application brings new breakthroughs in the field of voice emotion recognition, and has broad application prospects in many fields such as human-computer interaction, intelligent customer service, psychological health monitoring, entertainment, etc., and can promote the development and innovation of related fields. Specifically, for example: in the field of human-computer interaction, the present application can make intelligent devices more accurately understand user emotions, such as intelligent customer service can adjust response strategies according to user emotions, intelligent speakers and virtual assistants can provide appropriate services according to user emotional states, greatly improving the interactive experience; in the field of psychological health monitoring and medical treatment, the evaluation and monitoring of patients' psychological health status can be realized by analyzing voice emotions, providing a reference for doctors to adjust treatment plans and improving the accuracy of remote medical diagnosis; in the entertainment industry, games can adjust the plot and difficulty according to player voice emotional feedback, film and television creation can optimize works according to audience voice comments, and VR and AR experiences can also be optimized in real time; in the field of education, teachers can understand students' emotions and adjust teaching methods in a timely manner, online education platforms can provide students with guidance that meets their emotional needs, and special education teachers can also provide more targeted support.

[0079] Example 2

[0080] On the basis of the above-mentioned embodiments, the structure of the attention-enhanced residual learning model is proposed, which specifically includes:

[0081] The attention-augmented residual learning (AARL) model consists of an information coding (IC) module, a multi-head attention (MHA) module, a residual module, and an output layer.

[0082] The information encoding module includes a fully connected layer, batch normalization, leakage rectified linear unit activation function, and random dropout function connected in sequence. Specifically, the input comprehensive feature matrix is ​​processed by the fully connected layer H. The specific process is expressed as follows:

[0083] H1=W1X+b1

[0084] Where W1∈R D×F and b1∈R D are the weights and bias terms of the fully connected layer, D is the number of neurons, F is the dimension of X, H1 is the output of the first fully connected layer, and X is the comprehensive feature matrix of the input.

[0085] Then batch normalization (BN) is applied to normalize the data, and the LeakyReLU activation function is used to introduce nonlinear characteristics. Finally, Dropout regularization is applied to prevent overfitting.

[0086] Preferably, the number of information encoding modules is two, the two information encoding modules have the same structure and are connected in series, and the information encoding module is expressed by the following formula:

[0087]

[0088] Among them, μ and σ 2 is the mini-batch mean and variance, γ and β are learnable parameters, ∈ is a small constant; α = 0.01, p is the random inactivation rate, the value is 0.55, is the output after batch normalization of H1, H1 is the output of the fully connected layer, BN is batch normalization, For The output after activation function processing, LeakyReLU is the activation function with leakage rectified linear unit, For The output after random inactivation, Dropout is a random inactivation function.

[0089] The Multi-Head Attention (MHA) module can focus on different positions of the input sequence at the same time and capture diverse contextual relationships. i The attention output is calculated through a specific attention mechanism Attention:

[0090]

[0091] Among them, head iis the output of the i-th attention head, Attention is the attention mechanism, Q is the query vector, K is the key vector, and V is the value vector, are the projection matrices of the query vector, key vector, and value vector, respectively, d k is the dimension of the key vector, and softmax is the activation function.

[0092] The outputs of all attention heads are concatenated and then projected to form the final attention output, allowing the model to pay more attention to key information in the speech signal. This is represented by the following formula:

[0093] MHA(Q, K, V) = Concat(head1,..., head h )W o

[0094] where Concat is the concatenation, MHA(Q, K, V) is the output of the multi-head attention module, W o is the output projection matrix, and h is the total number of attention heads.

[0095] The residual module H i+1 is used to capture local patterns and enhance feature propagation. Each convolutional layer is responsible for extracting hierarchical features. To facilitate residual connections, an adjusted convolutional layer is applied, and the final residual output is obtained, which helps the model learn more complex feature patterns. The convolutional layer is represented by the following formula:

[0096]

[0097] where H i+1 is the output of the i+1-th convolutional layer, * is the convolution operation, σ is the ReLU activation function, and are the weights and biases of the convolutional layer, respectively.

[0098] To facilitate residual connections, the adjusted convolutional layer is represented by the following formula:

[0099]

[0100] where H adj is the output of the adjusted convolutional layer, W adj is the weight of the adjusted convolutional layer, is the input of the residual module, and b adj is the bias of the adjusted convolutional layer.

[0101] The residual output layer is represented by the following formula:

[0102] H residual = H7+ H adj

[0103] wherein H residual is the output of the residual output layer, H7 is the output of the last convolutional layer, H adj is the output of the adjustment convolutional layer.

[0104] The output layer first performs a flattening process on the feature map, and then inputs it into a fully connected layer equipped with a Softmax(·) activation function, which is specifically expressed according to the following formula:

[0105] H flatten = Flatten(H residual )

[0106] Y=Softmax(W out H flatten +b out )

[0107] wherein H flatten is the flattened feature, Flatten is the flattening process, H residual is the output of the residual output layer, Y is the predicted probability distribution of the emotion category, W out and b out are the weight and bias of the output layer, respectively.

[0108] According to the predicted probability distribution of the emotion category, the classification and recognition of the speech emotion are realized.

[0109] Embodiment 3

[0110] On the basis of the above-mentioned embodiments, an embodiment is proposed, which specifically comprises:

[0111] The input part selected by the present application mainly selects four data sets including BodEMODB, EMODB, CASIA and SAVEE.

[0112] BodEMODB is a Tibetan corpus, which contains 3000 utterances of 10 gender-balanced speakers, covering 6 emotion categories of anger (A), fear (F), happiness (H), sadness (Sa), surprise (Su) and neutral (N).

[0113] EMODB is a German data set, which has 535 German utterances of 10 speakers, representing 7 emotion states of anger (A), boredom (B), fear (F), disgust (D), happiness (H), sadness (Sa) and neutral (N).

[0114] CASIA is a Mandarin Chinese data set, which is composed of 1200 utterances of 4 speakers, with 6 emotions of anger (A), fear (F), happiness (H), neutral (N), sadness (Sa) and surprise (Su) evenly distributed.

[0115] SAVEE is an English dataset containing 480 English utterances of 4 male speakers, covering 7 emotions: anger (A), disgust (D), fear (F), happiness (H), sadness (Sa), surprise (Su) and neutral (N).

[0116] Different speech data and application scenarios may have different representations in acoustic features. The use of multiple comprehensive emotional speech datasets can better adapt to various situations. The combination of multiple speech sources can more flexibly adapt to changes in data and improve the universality and adaptability of the model on speech data of different languages, different age groups or different emotional expression styles. Whether it is to analyze Mandarin Chinese speech or English, German and other language speech, the present application can provide effective emotional analysis support.

[0117] As shown in Figure 3 The present application adopts a comprehensive feature extraction process, and the traditional acoustic features involved include mel-frequency cepstral coefficients (MFCCs), first-order and second-order differences of MFCCs, zero-crossing rate (ZCR), chroma, spectral contrast, spectral centroid, spectral roll-off, mode features and spectral bandwidth. These features provide rich feature support for subsequent speech emotion analysis. For example, mel-frequency cepstral coefficients (MFCCs) can effectively reflect the spectral envelope features of speech and capture the formant structure of speech, which plays an important role in distinguishing different speech content and emotion-related speech changes; the zero-crossing rate (ZCR) can measure the number of times the speech signal crosses the zero level in unit time, which can distinguish between unvoiced and voiced sounds, and further reflect some prosodic and emotional features of speech. The combination of multiple traditional acoustic features can comprehensively describe the speech signal from multiple aspects such as spectrum, prosody and harmonics, and more completely capture the emotional information contained in the speech; in addition, comprehensive feature extraction can improve the recognition accuracy of the model and enhance its robustness; a single acoustic feature can only reflect certain aspects of speech characteristics, making it difficult to fully represent emotions. The combination of multiple traditional acoustic features can provide more rich input information for the model and enhance the model's ability to distinguish different emotional states. In practical applications, speech signals will be disturbed by various factors such as environmental noise and pronunciation differences of different speakers. The use of multiple traditional acoustic features can enhance the robustness of the model, so that it can work more stably under different conditions. Even if some features are disturbed in a particular environment, other features can still provide effective emotional information to ensure that the model can continue to accurately analyze speech emotions. In a noisy environment, although features such as spectral bandwidth may be affected by noise, features such as MFCCs can still provide relatively stable speech features for the model to maintain the performance of the model.

[0118] Specifically, the traditional acoustic features include low-level descriptor features and high-level statistical function features, and the low-level descriptor features are as follows:

[0119] Mel Frequency Cepstral Coefficients (MFCCs): Preprocess the speech signal with operations like framing, windowing, etc. Then convert the time-domain signal to the frequency domain using the Fast Fourier Transform. Next, filter the frequency-domain signal with a Mel filter bank to obtain the Mel spectrum. Take the log of the Mel spectrum and perform a Discrete Cosine Transform to finally obtain the Mel Frequency Cepstral Coefficients.

[0120] Zero Crossing Rate (ZCR): The number of times an audio signal changes from positive to negative or negative to positive within a certain time period.

[0121] Chroma Features: In speech emotion analysis, the harmonic structure of speech varies under different emotional states, and chroma features can capture these changes.

[0122] Spectral Contrast: The contrast of different frequency components in the spectrum. For example, in an audio signal, the difference in energy between certain frequency ranges compared to other frequency ranges can be measured by spectral contrast.

[0123] Spectral Centroid: Reflects the center of gravity of a signal in the frequency spectrum. It can be used to describe the brightness, timbre characteristics of sound, etc.

[0124] Spectral Roll-off: The characteristic of the signal's spectral energy gradually decreasing with increasing frequency.

[0125] Tonnetz Features: Characteristics of the mode in audio, mainly derived from the fundamental frequency variation pattern of speech signals. These features carry a wealth of information and play a key role in tasks such as speech recognition, emotion analysis, language and speaker recognition.

[0126] Spectral Bandwidth: The width of the frequency range occupied by the signal in the frequency spectrum. It reflects the distribution range of the frequency components of the speech signal and provides important information for emotion analysis.

[0127] Advanced statistical function features are as follows:

[0128] First-order and second-order differences of MFCCs: The first-order difference of MFCCs reflects the rate of change of MFCC coefficients between adjacent frames, which can be used to capture the dynamic characteristics of speech signals. The second-order difference further reflects the changes in the first-order difference, more deeply describing the dynamic trend of speech signals.

[0129] Chroma feature: Chroma itself is a basic feature to describe the harmonic structure in audio, which can be directly used as a low-level descriptor to reflect the basic spectral characteristics of speech signals, and is a kind of characterization of the original information of speech. At the same time, based on the chroma feature, the first-order difference, second-order difference and other statistical quantities can be calculated to analyze the changes in the time sequence and dig out more potential information. At this time, chroma becomes a basic element to construct high-level statistical function features.

[0130] Low-level descriptor features such as mel-frequency cepstral coefficients and zero-crossing rate are basic acoustic features directly extracted from speech signals. The extraction process is relatively simple, and it reflects the basic properties of speech signals such as frequency and amplitude changes. High-level statistical function features are obtained by calculating first-order difference, second-order difference and other statistical operations based on low-level descriptor features. It involves more complex mathematical transformations, can further dig out the relationship and trend between features, and reflects more abstract and high-level characteristics of speech signals.

[0131] The fractal-inspired measurement method is to capture the complexity and self-similarity of speech signals through Katz fractal dimension (KFD). KFD innovatively provides a unique feature extraction dimension for speech emotion analysis from the perspective of fractal geometry. Traditional acoustic features mainly focus on the time or frequency characteristics of speech signals, while speech signals have complex nonlinear and self-similar structures. KFD can effectively capture the characteristics that traditional features cannot describe. In actual speech emotion analysis, KFD can accurately quantify the complexity and self-similarity of speech signals at different time scales. For example, in angry or excited speech, the complexity of the signal changes, and KFD can sensitively capture this change, providing new key information for emotion recognition, making up for the shortcomings of traditional features, and enriching the expression ability of the feature vector.

[0132] As shown in Figure 4 , the attention-enhanced residual learning model is divided into four main components: information encoding module, multi-head attention module, residual module and output layer.

[0133] The information coding (IC) module is a key preprocessing step for the attention-enhanced residual learning model. It receives the input speech feature matrix, first performs a preliminary transformation on the data through a fully connected layer to adjust the data dimensions and feature combinations, then uses batch normalization processing to distribute the data within a suitable numerical interval, enhancing the stability of model training. Subsequently, it introduces non-linear characteristics by using a leaky rectified linear unit (LeakyReLU) activation function, allowing the model to learn more complex feature relationships. Finally, it applies a dropout regularization technique to prevent model overfitting. This series of operations progressively converts the input feature matrix into a more expressive high-level representation, providing high-quality data for in-depth analysis by subsequent modules. This process can be repeated through additional fully connected layers to further extract abstract information from the data and improve feature representation capabilities.

[0134] The multi-head attention (MHA) module is one of the core components of the model for implementing precise sentiment analysis. It draws on human attention mechanisms to allow the model to focus on different positions in the speech sequence simultaneously, capturing diverse contextual relationships. When processing speech features, the MHA module projects the input feature matrix into multiple different subspaces, forming multiple attention heads. Each attention head independently calculates the attention output, focusing on different key parts of the input sequence through specific attention mechanisms to extract subtle emotional cues. The outputs of all attention heads are concatenated and then projected to form the final attention output. This allows the model to comprehensively focus on key information in the speech signal from multiple angles, effectively improving the ability to capture complex emotional features and thus enhancing the accuracy of speech sentiment analysis.

[0135] The residual module plays an important role in strengthening feature learning and propagation in the attention-enhanced residual learning model. It consists of multiple convolutional layers, each responsible for extracting hierarchical patterns and local details from speech features. In traditional convolutional processes, as the network depth increases, the gradient vanishing problem easily occurs, affecting the model's ability to learn deep features. To address this issue, the residual module introduces residual connections, allowing the model to directly learn the difference between the input and output, i.e., the residual information, through adjusted convolutional layers. This design enables the model to more efficiently capture complex features in the speech signal, enhances feature propagation effects, avoids performance degradation due to increased network depth, and improves the model's ability to learn speech emotional features, thereby enhancing the accuracy and reliability of speech sentiment analysis.

[0136] The output layer is a key part of the attention-enhanced residual learning model for outputting the final emotion analysis result. It first performs flattening processing on the feature map processed by the previous module, converts the multi-dimensional feature map into a one-dimensional vector, and facilitates subsequent classification operations. Then, the flattened vector is input into a fully connected layer equipped with a Softmax activation function. The Softmax function converts the input vector into a probability distribution over multiple emotion categories, and the model determines the emotion category of the speech according to the probability size. Finally, the output layer outputs the prediction result of the speech emotion, realizing the conversion from speech features to emotion classification and providing an intuitive and understandable output for the speech emotion analysis task.

[0137] Meanwhile, the attention-enhanced residual learning model is optimized using a loss function and an Adam optimizer.

[0138] In order to enhance the generalization ability of the model and alleviate the overfitting problem, a smooth classification cross-entropy loss function is used, which quantifies the gap between the model prediction result and the true label. In speech emotion analysis, the model will output the prediction probability of each emotion category after processing the input speech signal, while the true label represents the actual emotion contained in the speech. By calculating the classification cross-entropy loss, the accuracy of the model prediction can be intuitively reflected. If the model prediction is significantly different from the true label, the loss value is high; otherwise, the loss value is low, providing a clear optimization direction for the model training. Adjusting the true label using the label smoothing technique is a major feature of this loss function. Traditional hard labels may cause the model to overfit during training, while label smoothing converts hard labels into soft labels with some uncertainty by smoothing the true label. This makes the model not too dependent on a certain label during training, and can learn more general features, thereby enhancing the generalization ability of the model, reducing the overfitting phenomenon, and enabling the model to maintain good performance when facing new, unseen speech data.

[0139] An important feature of the Adam optimizer is its ability to adaptively adjust the learning rate. During the model training process, the learning rate determines the step size of parameter updates. If the learning rate is too large, the model may skip the optimal solution during training, resulting in failure to converge; if the learning rate is too small, the training speed will be very slow, consuming a lot of time. The Adam optimizer dynamically adjusts the learning rate based on the first and second moment estimates of the gradient. In the early stages of training, it can quickly bring the model close to the optimal solution; in the later stages of training, it can automatically reduce the learning rate to avoid missing the optimal solution and ensure the efficiency and stability of model training. The Adam optimizer also combines the advantages of the momentum method and the RMSProp algorithm, updating parameters by calculating the exponential moving average of the gradient. This method can effectively utilize historical gradient information, reduce parameter update fluctuations, and make the model converge to the optimal solution more quickly. In the training of attention-enhanced residual learning models, the Adam optimizer can bring the model to a good performance state in a shorter time, improving training efficiency and reducing training time, and still ensuring good optimization results in complex model structures and large-scale data training scenarios.

[0140] Embodiment 4

[0141] Based on the above embodiments, as shown in Figure 5 the hardware building operation steps when the application is applied, specifically include:

[0142] S401: Build a speech emotion analysis system.

[0143] Specifically, select a suitable programming language (such as Python) and related tool library (such as librosa for audio processing, TensorFlow or PyTorch for model building), according to the principles and requirements of speech emotion analysis, build a system architecture (i.e. attention-enhanced residual learning model) containing feature extraction module, prediction module, training module and classification module.

[0144] S402: Initialize the system environment.

[0145] Specifically, configure the hardware resources (such as CPU, GPU, etc.) required for system operation, install and import the tool library used to build the system, and initialize the settings of parameters, variables, etc. in the system to ensure that the system is in a runnable state.

[0146] S403: Accept the voice file.

[0147] Specifically, through file reading interface, script or related input device, import the voice file to be analyzed into the speech emotion analysis system. It is worth noting that before this, it is necessary to ensure that the format of the voice file meets the system requirements.

[0148] S404: Perform voice emotion analysis.

[0149] Specifically, the system preprocesses the imported voice file (such as noise reduction, normalization, etc.), extracts voice features (such as MFCCs, etc.), inputs the features into the trained attention-enhanced residual learning model, and obtains the emotion analysis result.

[0150] Finally, the trained model can be applied to practical scenarios, such as human-computer interaction, so that the system can make more appropriate responses according to the emotions in the user's voice, and in mental health monitoring, assist doctors in assessing the emotional state of patients, etc.

[0151] Embodiment 5

[0152] Based on the above embodiments, as shown in Figure 6 The present application provides a voice emotion analysis system, comprising:

[0153] The feature extraction module is configured to collect training voice data and extract features from the training voice data to obtain a training comprehensive feature matrix.

[0154] The prediction module is configured to input the training comprehensive feature matrix into a preset attention-enhanced residual learning model to obtain a prediction probability distribution of the emotion category, and obtain a predicted emotion category according to the prediction probability distribution of the emotion category; wherein the attention-enhanced residual learning model comprises an information encoding module configured to convert the input comprehensive feature matrix into a high-level representation form.

[0155] The training module is configured to train the preset attention-enhanced residual learning model according to a classification cross-entropy loss function and a label smoothing technique to obtain an optimal attention-enhanced residual learning model.

[0156] The classification module is configured to extract features from target voice data to obtain a target comprehensive feature matrix, input the target comprehensive feature matrix into the optimal attention-enhanced residual learning model, and obtain an emotion category.

[0157] It should be noted that the voice emotion analysis system provided by the embodiments of the present application is to realize the voice emotion analysis method described above, and the functions thereof can be referred to the above method embodiments, which will not be repeated here.

[0158] Embodiment 6

[0159] Based on the above embodiments, the present application provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the voice emotion analysis method according to Embodiment 1.

[0160] The application provides a computer readable storage medium, the storage medium comprising a stored computer program, wherein the computer program controls a device where the storage medium is located to execute a voice emotion analysis method as described in the above embodiment 1 when the computer program is running.

[0161] To sum up, the application extracts complex features in the voice by introducing Katz fractal dimension, which is convenient for obtaining complex features in the voice. An attention-enhanced residual learning model is constructed to accurately recognize voice emotion, and a smoothing loss function is used for the attention-enhanced residual learning model to obtain more accurate results. The application effectively improves the accuracy and robustness of voice emotion recognition, enhances the generalization ability of the model, and relieves the overfitting phenomenon.

[0162] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A speech emotion analysis method, characterized in that: include: Step 1: Collect training speech data and extract features from the training speech data to obtain a training comprehensive feature matrix; Step 2: Inputting the training comprehensive feature matrix into a preset attention-enhanced residual learning model to obtain a predicted probability distribution of the emotion category, and obtaining a predicted emotion category based on the predicted probability distribution of the emotion category; wherein the attention-enhanced residual learning model includes an information encoding module, and the information encoding module is used to convert the input comprehensive feature matrix into a high-level representation; Step 3: Train the preset attention-enhanced residual learning model based on the classification cross-entropy loss function and label smoothing technology to obtain the optimal attention-enhanced residual learning model; Step 4: Extract features from the target speech data to obtain a target comprehensive feature matrix, and input the target comprehensive feature matrix into the optimal attention-enhanced residual learning model to obtain the emotion category.

2. A speech emotion analysis method according to claim 1, characterized in that: The step 1 specifically includes: Feature extraction is performed on the training speech data to obtain complex features and traditional acoustic features, and a comprehensive feature matrix is ​​constructed based on the complex features and traditional acoustic features; the complex features include Katz fractal dimensions, and the traditional acoustic features include Mel-frequency cepstral coefficients, first-order differences and second-order differences of Mel-frequency cepstral coefficients, zero-crossing rate, chromaticity features, spectral contrast, spectral centroid, spectral roll-off, mode features, and spectral bandwidth.

3. A speech emotion analysis method according to claim 2, characterized in that: The Katz fractal dimension is calculated according to the following formula: d=max(|x i -x0|) Where D is the Katz fractal dimension, L is the total length of the signal path, and d is x i The maximum distance from x0, n is the number of points in the signal, x i is the i-th point in the signal, x0 is the starting point of the signal, Δx i is the distance between two adjacent points in the signal.

4. A speech emotion analysis method according to claim 1, characterized in that: The attention-enhanced residual learning model also includes a multi-head attention module, a residual module and an output layer; The multi-head attention module is used to focus on different positions of the input features and capture diverse contextual relationships; The residual module is used to capture local patterns and enhance feature propagation; The output layer is used to convert the processed features into emotion category probabilities.

5. A speech emotion analysis method according to claim 1, characterized in that: The information encoding module includes a fully connected layer, a batch normalization layer, a leaky rectified linear unit activation function, and a random dropout function connected in sequence; The leakage-rectified linear unit activation function is used to introduce nonlinear features into the information encoding module.

6. A speech emotion analysis method according to claim 4, characterized in that: The residual module includes multiple convolutional layers, an adjustment convolutional layer and a residual output layer; The multiple convolutional layers have the same structure, and the convolutional layers are expressed by the following formula: Among them, H i+1 is the output of the i+1th convolutional layer, * is the convolution operation, σ is the ReLU activation function, and are the weights and biases of the convolutional layer respectively; The adjusted convolution layer is expressed as follows: Among them, H adj To adjust the convolutional layer output, W adj To adjust the weights of the convolutional layer, is the input of the residual module, b adj To adjust the bias of the convolutional layer; The residual output layer is expressed as follows: H residual =H7+H adj Among them, H residual is the output of the residual output layer, H7 is the output of the last convolutional layer, and H adj To adjust the convolutional layer output.

7. A speech emotion analysis method according to claim 6, characterized in that: The output layer is expressed as follows: H flatten =Flatten(H residual ) Y =Softmax(W out H flatten +b out ) Among them, H flatten Flatten is the feature after flattening, H residual is the output of the residual output layer, Y is the predicted probability distribution of the emotion category, W out and b out are the weight and bias of the output layer, respectively.

8. A speech emotion analysis system, characterized in that: include: The feature extraction module is used to collect training speech data and extract features from the training speech data to obtain a training comprehensive feature matrix; A prediction module is used to input the training comprehensive feature matrix into a preset attention-enhanced residual learning model to obtain a predicted probability distribution of the emotion category, and obtain a predicted emotion category based on the predicted probability distribution of the emotion category; wherein the attention-enhanced residual learning model includes an information encoding module, and the information encoding module is used to convert the input comprehensive feature matrix into a high-level representation; The training module is used to train the preset attention-enhanced residual learning model based on the classification cross-entropy loss function and label smoothing technology to obtain the optimal attention-enhanced residual learning model; The classification module is used to extract features from the target speech data to obtain a target comprehensive feature matrix, and then input the target comprehensive feature matrix into the optimal attention-enhanced residual learning model to obtain the emotion category.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for speech emotion analysis according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a speech emotion analysis method according to any one of claims 1 to 7.

Images