Explanatable speech emotion recognition algorithm based on multilevel feature deentanglement

Abstract

The embodiment of the invention relates to the field of speech emotion recognition, in particular to an interpretable speech emotion recognition method based on multilevel feature deentanglement. A specific embodiment of the method comprises the following steps: acquiring a voice training sample set; constructing an initial interpretable speech emotion recognition model; for a target voice training sample, executing the following training steps: inputting sample voice information into the initial feature extraction network model; inputting the initial general emotion feature information into the initial fusion multi-level emotion feature hybrid self-encoding network model; inputting the initial multi-level emotion feature information into an initial multi-level perception classification model; determining a difference value between the initial emotion label and a sample emotion label included in the target training sample; determining the initial interpretable speech emotion recognition model as a trained interpretable speech emotion recognition model; and inputting the target voice information into the interpretable voice emotion recognition model. According to the embodiment, the accuracy of speech emotion recognition can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech emotion recognition, and specifically to an interpretable speech emotion recognition algorithm based on multi-level feature disentanglement. Background Art

[0002] As the core medium of human communication, speech not only carries rich semantic content, but also contains deep emotional colors. In the rapidly developing human society, emotional information is becoming more and more important. Human-computer interaction systems with emotion perception capabilities help build a more harmonious human-computer environment. Therefore, speech emotion recognition has received more and more attention from the industry and academia, and has been widely used in smart cars, mental health, distance education and other fields.

[0003] Given the uniqueness of individual voice differences and expression habits, speaker information inevitably leaves a unique mark in emotional expression. These personalized features often interfere with the accurate extraction of emotional representation, thereby restricting the effectiveness of speech emotion recognition. To break through this bottleneck, researchers have explored a variety of strategies: on the one hand, by incorporating speaker recognition tasks to fine-tune the emotion classification model to weaken the influence of speaker characteristics; on the other hand, adversarial domain adaptation technology is used to directly weaken the interference of non-emotional factors on emotion classification.

[0004] However, although existing methods have made some progress, they are still insufficient in dealing with the complex and changeable emotional features in speech. In particular, although some advanced decoupling models can effectively separate the multiple features in speech, they often focus on the stable static feature decoupling, while ignoring the continuous dynamic features that are crucial to emotional expression and the effective fusion mechanism between these features, which is crucial for fully capturing subtle changes in emotions. In addition, the widely used Softmax (normalized) classifier is difficult to meet the high-precision and high-robustness recognition requirements when dealing with highly entangled nonlinear data such as speakers and emotional features. Therefore, it is urgent to explore more advanced methods to overcome these challenges and achieve more accurate and comprehensive analysis of speech emotions. Summary of the invention

[0005] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present application propose an interpretable speech emotion recognition algorithm, a computer device and a computer-readable storage medium based on multi-level feature disentanglement to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present application provide an interpretable speech emotion recognition algorithm based on multi-level feature disentanglement, the algorithm comprising: obtaining a speech training sample set, wherein the speech training samples in the speech training sample set include: sample speech information and sample emotion labels; constructing an initial interpretable speech emotion recognition model, wherein the initial interpretable speech emotion recognition model includes: an initial feature extraction network model, an initial fusion multi-level emotion feature hybrid autoencoder network model, and an initial multi-layer perceptual classification model; selecting a target speech training sample from the speech training sample set; for the target speech training sample, performing the following training steps: inputting the sample speech information included in the target training sample into the initial feature extraction network model; The method comprises the following steps: obtaining initial universal emotion feature information in the model; inputting the initial universal emotion feature information into the initial fused multi-level emotion feature hybrid autoencoder network model to obtain initial multi-level emotion feature information; inputting the initial multi-level emotion feature information into the initial multi-layer perception classification model to obtain initial emotion labels; determining the difference between the initial emotion labels and the sample emotion labels included in the target training samples based on a preset loss function; in response to determining that the difference value is less than the preset difference value, determining the initial interpretable speech emotion recognition model as the trained interpretable speech emotion recognition model; in response to receiving the target speech information, inputting the above target speech information into the above interpretable speech emotion recognition model to obtain the emotion label.

[0008] In a second aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the method described in any implementation of the first aspect is implemented.

[0009] In a third aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0010] The above-mentioned various embodiments of the present application have the following beneficial effects: through the interpretable speech emotion recognition algorithm based on multi-level feature disentanglement of some embodiments of the present application, first, a speech training sample set is obtained, wherein the speech training samples in the speech training sample set include: sample speech information and sample emotion labels. Secondly, an initial interpretable speech emotion recognition model is constructed, wherein the initial interpretable speech emotion recognition model includes: an initial feature extraction network model, an initial fusion multi-level emotion feature hybrid autoencoder network model, and an initial multi-layer perception classification model. Then, a target speech training sample is selected from the speech training sample set. Then, for the target speech training sample, the following training steps are performed: the sample speech information included in the target training sample is input into the initial feature extraction network model to obtain the initial general emotion feature information; the initial general emotion feature information is input into the initial fusion multi-level emotion feature hybrid autoencoder network model to obtain the initial multi-level emotion feature information; the initial multi-level emotion feature information is input into the initial multi-layer perception classification model to obtain the initial emotion label; based on the preset loss function, the difference value between the initial emotion label and the sample emotion label included in the target training sample is determined; in response to determining that the difference value is less than the preset difference value, the initial interpretable speech emotion recognition model is determined as the trained interpretable speech emotion recognition model. Finally, in response to receiving the target speech information, the above target speech information is input into the above interpretable speech emotion recognition model to obtain the emotion label. Therefore, the present application proposes an interpretable speech emotion recognition algorithm based on multi-level feature disentanglement, and constructs a fusion multi-level emotion feature hybrid autoencoder network, which can comprehensively capture the multi-level emotion information in the speech and enhance the efficiency and accuracy of the autoencoder in feature disentanglement. In addition, this application also constructs a multi-layer perception classification model based on KAN (Kolmogorov-Arnold Networks, a new neural network architecture), which can help the model capture complex nonlinear relationships in the data and further improve the accuracy of classification. Thus, the accuracy of speech emotion recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a flowchart of some embodiments of an interpretable speech emotion recognition algorithm based on multi-level feature disentanglement according to the present application;

[0013] Figure 2 It is a schematic diagram of the structure of a computer device suitable for implementing some embodiments of the present application. DETAILED DESCRIPTION

[0014] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0015] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0016] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0017] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0018] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0019] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0020] Figure 1 The flowchart 100 of some embodiments of the interpretable speech emotion recognition algorithm based on multi-level feature disentanglement according to the present application is shown. The interpretable speech emotion recognition algorithm based on multi-level feature disentanglement includes the following steps:

[0021] Step 101: Obtain a speech training sample set.

[0022] In some embodiments, the execution subject of the interpretable speech emotion recognition algorithm based on multi-level feature disentanglement can obtain a speech training sample set from a speech database by wired connection or wireless connection. Among them, the speech training samples in the speech training sample set may include: sample speech information and sample emotion tags. Here, the speech database may be an IEMOCAP (Interactive Emotional Dyadic Motion Capture Database, multimodal emotion database) database. For example, the data collection in the speech database may be conducted in the form of a two-person dialogue from five professional actors and five professional actresses. Here, the dialogue is divided into two parts, one part is a fixed script, and the other part is a free play under a given theme scenario (for example, the given theme scenario may be: campus scene, office scene). For example, the speech database includes 4787 impromptu dialogues and 5255 script dialogues. Each sentence is marked with a specific emotion tag, which is divided into ten categories including anger, happiness, sadness, neutrality, etc. Here, the sample speech information included in the speech training samples in the speech training sample set can characterize the dialogue in the speech database. The sample emotion tags included in the speech training samples in the speech training sample set may be emotion tags in the speech database. For example, a five-fold cross-validation strategy (Leave-One-Session-Out, LOSO) was used for experiments.

[0023] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0024] Step 102, constructing an initial interpretable speech emotion recognition model.

[0025] In some embodiments, the execution subject may construct an initial interpretable speech emotion recognition model, wherein the initial interpretable speech emotion recognition model includes: an initial feature extraction network model, an initial fusion multi-level emotion feature hybrid autoencoder network model, and an initial multi-layer perception classification model.

[0026] In practice, the above execution entity can build an initial interpretable speech emotion recognition model through the following steps:

[0027] The first step is to construct an initial feature extraction network model. The initial feature extraction network model can be an untrained model that takes sample speech information as input and initial general emotional feature information as output. In practice, the above-mentioned execution entity can determine the speech emotion base model as the initial feature extraction network model. For example, the speech emotion base model can be an emotion2vec (speech emotion base model) model.

[0028] Therefore, the emotion2vec network operates under the data2vec (self-supervised framework), using online distillation strategies to promote continuous interaction and iterative optimization between the teacher network and the student network, and continuously enhance the ability to understand and capture emotional features. At the same time, by combining the training strategy of sentence-level loss and frame-level loss, it can not only capture the overall global emotion, but also more accurately understand the local or frame-level changes of emotion, thereby extracting more comprehensive and accurate emotional features.

[0029] The second step is to construct an initial fused multi-level emotional feature hybrid autoencoder network model. Among them, the initial fused multi-level emotional feature hybrid autoencoder network model can be an untrained model with initial general emotional feature information as input and initial multi-level emotional feature information as output. In practice, the core structure of the initial fused multi-level emotional feature hybrid autoencoder network model can include an emotional feature disentanglement module (including autoencoder network, LSTM network), an emotional feature integration module and a decoder. The decoder can be an untrained model with initial reshaped emotional feature information as input and initial multi-level emotional feature information as output. The decoder can be composed of a fully connected layer. For example, the decoder can refer to the following formula:

[0030] x′=o(W′·z+b′).

[0031] Where x′ represents the initial multi-level emotional feature information. z represents the initial reshaped emotional feature information. W′ and b′ represent the weight matrix and bias in the decoding process. σ represents the activation function. For example, the activation function can be ReLU (Linear rectification function).

[0032] The emotional feature disentanglement module may include: a static emotional feature disentanglement submodule and a dynamic emotional feature disentanglement submodule. Among them, the static emotional feature disentanglement submodule may be an untrained model with initial general emotional feature information as input and initial static emotional feature information as output. The dynamic emotional feature disentanglement submodule may be an untrained model with initial general emotional feature information as input and initial dynamic emotional feature information as output. For example, the static emotional feature disentanglement submodule may be an Auto-Encoder model. The dynamic emotional feature disentanglement submodule may be an LSTM (Long Short-Term Memory) model. For example, the static emotional feature disentanglement submodule may refer to the following formula:

[0033] v=o(W·x+b).

[0034] Wherein, v represents the initial static emotional feature information. x represents the initial general emotional feature information. W and b represent the weight matrix and bias in the encoding process. σ represents the activation function. For example, the activation function can be ReLU (Linear Rectification function).

[0035] Therefore, the fully connected neural network integrated inside the autoencoder network gradually maps the original features into a low-dimensional but information-dense latent space. In this process, stable speech features that fluctuate less over time are enhanced, while information such as noise and interference are effectively suppressed, thereby extracting stable static speech features.

[0036] For example, the dynamic emotion feature disentanglement submodule can refer to the following formula:

[0037] u,(h n ,c n )=LSTM(x,(h0,c0))

[0038] Among them, h n and c n They represent hidden vectors and memory cells, respectively, and their dimensions are the same. h0 and c0 are their initial states. u represents the initial dynamic emotional feature information, and LSTM represents the dynamic emotional feature disentanglement submodule.

[0039] Therefore, LSTM processes the data of each time step in the sequence step by step through its cyclic structure, and uses memory units to retain and transmit the dynamic information obtained from the previous time step, thereby effectively capturing the dynamic characteristics of emotion changing over time. This paper uses a two-layer LSTM architecture to deeply mine the dynamic features in the sequence in a hierarchical manner to make up for the dynamic information ignored by the autoencoder.

[0040] The emotion feature integration module may include: an emotion feature fusion unit and an emotion feature reshaping unit. The emotion feature fusion unit may be an untrained model that takes initial static emotion feature information and initial dynamic emotion feature information as input and initial comprehensive emotion feature information as output. The emotion feature reshaping unit may be an untrained model that takes initial comprehensive emotion feature information as input and initial reshaped emotion feature information as output. For example, the emotion feature fusion unit may refer to the following formula:

[0041] x fused =Concat(v,u).

[0042] Among them, x fused Represents the initial comprehensive emotional feature information. Concat is used to connect v and u.

[0043] Therefore, one-dimensional splicing is used to fuse static features and dynamic features in the feature dimension to obtain initial comprehensive emotional feature information.

[0044] For example, the sentiment feature reshaping unit can refer to the following formula:

[0045]

[0046] Among them, h represents the initial reshaped emotional feature information. μ(x fused ) represents the mean vector of the initial comprehensive emotional feature information. σ(x fused ) represents the variance vector of the initial comprehensive emotional feature information. ε∈N(0,I) represents the random noise vector sampled from the standard normal distribution. N stands for "normal", which is the most common continuous random variable distribution in probability theory. I represents the variance of the standard normal distribution, that is, the case where the mean is 0 and the variance is 1. Represents element-wise multiplication.

[0047] The third step is to construct an initial multi-layer perception classification model, where the initial multi-layer perception classification model can be an untrained model that takes initial multi-level emotional feature information as input and takes initial emotional labels as output.

[0048] In practice, the above execution entity can build an initial multi-layer perception classification model through the following sub-steps:

[0049] The first sub-step is to determine a perceptual classification model. For example, the perceptual classification model may be a multi-layer perceptual classifier based on KAN (Kolmogorov-Arnold Networks, a novel neural network architecture).

[0050] In the second sub-step, the Kaiming initialization strategy is used to adjust the weights of each network layer for the above-mentioned perceptual classification model. Thus, the Kaiming initialization strategy can be used to adjust the weights of the basic linear layer and the spline interpolation.

[0051] The third sub-step is to perform noise injection processing on the adjusted perceptual classification model to adjust the scaling factor of the spline function in the model. In practice, the above-mentioned execution subject can inject random noise into the adjusted perceptual classification model through a preset noise algorithm. For example, the preset noise algorithm can be a Gaussian noise algorithm. Thus, by injecting random noise into the spline interpolation and enabling the independent scaling mechanism, the model can adaptively adjust the scaling factor of the spline interpolation.

[0052] The fourth sub-step is to perform regularization processing on the adjusted perceptual classification model to adjust the parameter weights in the model. In practice, the above-mentioned execution subject can perform regularization processing on the adjusted perceptual classification model to adjust the parameter weights in the model by using a preset regularization algorithm. For example, the preset regularization algorithm can be an L1 regularization (L1regularization, L1 norm regularization) algorithm. Thus, by performing L1 regularization on the parameter weights, the occurrence of overfitting can be effectively prevented and the generalization ability of the model can be improved.

[0053] The fifth sub-step is to perform activation adjustment processing on the adjusted perceptual classification model to generate an initial multi-layer perceptual classification model. In practice, the above-mentioned execution subject can determine the preset activation function as the activation function in the adjusted perceptual classification model to obtain the initial multi-layer perceptual classification model. For example, the preset activation function can refer to the following formula:

[0054] φ(x)=w·(b(x)+spline(x)).

[0055] Where φ(x) represents the preset activation function. b(x) represents ReLU (Linear rectification function). spline(x) = 0 represents the spline function. Here, i represents the serial number. i represents the i-th weight coefficient. i (x) is the i-th basis function, and the B-spline basis function is used here. Each activation function is initialized to spline(x)=0.

[0056] In the fourth step, the initial feature extraction network model, the initial fusion multi-level emotion feature hybrid autoencoder network model, and the initial multi-layer perception classification model are fused to generate an initial interpretable speech emotion recognition model. Here, the initial interpretable speech emotion recognition model may include a learning rate. For example, the learning rate may be 0.01.

[0057] Step 103: Select a target speech training sample from the speech training sample set.

[0058] In some embodiments, the execution entity may select a target speech training sample from a speech training sample set.

[0059] Step 104: for the target speech training sample, perform the following training steps:

[0060] Step 1041, inputting the sample speech information included in the target training sample into the initial feature extraction network model to obtain initial general emotion feature information.

[0061] In some embodiments, the execution subject may input the sample speech information included in the target training sample into the initial feature extraction network model to obtain the initial universal emotion feature information. For example, the initial universal emotion feature information may be "[batch_size, feature_dim]". batch_size = 128. feature_dim = 768.

[0062] Step 1042, inputting the initial universal emotion feature information into the initial fused multi-level emotion feature hybrid autoencoder network model to obtain the initial multi-level emotion feature information.

[0063] In some embodiments, the above-mentioned execution subject can input the initial general emotional feature information into the initial fused multi-level emotional feature hybrid autoencoder network model to obtain the initial multi-level emotional feature information. Among them, the above-mentioned initial fused multi-level emotional feature hybrid autoencoder network model may include: an emotional feature disentanglement module, an emotional feature integration module, and a decoder. Among them, the emotional feature disentanglement module may include: a static emotional feature disentanglement submodule and a dynamic emotional feature disentanglement submodule. The emotional feature integration module may include: an emotional feature fusion unit and an emotional feature reshaping unit.

[0064] In practice, the execution subject can input the initial general emotion feature information into the initial fusion multi-level emotion feature hybrid autoencoder network model through the following steps to obtain the initial multi-level emotion feature information:

[0065] In the first step, the initial general emotional feature information is input into the static emotional feature disentanglement submodule to obtain the initial static emotional feature information.

[0066] In the second step, the initial general emotional feature information is input into the dynamic emotional feature disentanglement submodule to obtain the initial dynamic emotional feature information.

[0067] The third step is to input the initial static emotion feature information and the initial dynamic emotion feature information into the emotion feature fusion unit to obtain the initial comprehensive emotion feature information. The initial comprehensive emotion feature information includes multi-dimensional emotion feature information. Here, the multi-dimensional emotion feature information may include the initial static emotion feature information and the initial dynamic emotion feature information.

[0068] The fourth step is to input the initial comprehensive emotional feature information into the emotional feature reconstruction unit to obtain the initial reconstructed emotional feature information.

[0069] The fifth step is to input the initial reshaped emotional feature information into the decoder to obtain the initial multi-level emotional feature information.

[0070] Step 1043, input the initial multi-level emotional feature information into the initial multi-layer perception classification model to obtain an initial emotional label.

[0071] In some embodiments, the execution entity may input the initial multi-level emotion feature information into an initial multi-layer perception classification model to obtain an initial emotion label.

[0072] Step 1044: Determine the difference between the initial emotion label and the sample emotion label included in the target training sample based on a preset loss function.

[0073] In some embodiments, the execution subject may determine the difference between the initial emotion label and the sample emotion label included in the target training sample based on a preset loss function, wherein the preset loss function may include: a classification loss function and a reconstruction loss function.

[0074] In practice, the execution subject may determine the difference between the initial emotion label and the sample emotion label included in the target training sample based on the preset loss function through the following steps:

[0075] In the first step, based on the classification loss function, a first difference value between the initial emotion label and the sample emotion label included in the target training sample is determined. For example, the classification loss function can refer to the following formula:

[0076]

[0077] Where L1 represents the classification loss function. i represents the sequence number. n represents the number of speech training samples in the speech training sample set. i Represents the sentiment label of the i-th sample.

[0078] In the second step, based on the reconstruction loss function, a second difference value between the initial universal emotion feature information and the initial multi-level emotion feature information is determined. For example, the reconstruction loss function can refer to the following formula:

[0079]

[0080] Wherein, L2 represents the reconstruction loss function. i represents the sequence number. n represents the number of speech training samples in the speech training sample set. x represents the initial general emotion feature information. x′ represents the initial multi-level emotion feature information.

[0081] In the third step, the sum of the first difference value and the second difference value is determined as the difference value.

[0082] Step 1045, in response to determining that the difference value is less than the preset difference value, determining the initial interpretable speech emotion recognition model as the trained interpretable speech emotion recognition model.

[0083] In some embodiments, the above-mentioned execution subject can determine the initial interpretable speech emotion recognition model as the trained interpretable speech emotion recognition model in response to determining that the difference value is less than a preset difference value.

[0084] Optionally, in response to determining that the difference value is greater than or equal to the preset difference value, the execution subject may perform the following adjustment steps:

[0085] The first step is to adjust the parameters of the initial interpretable speech emotion recognition model, and determine the adjusted model as the optimized version. In practice, first, the above-mentioned execution subject can adjust the learning rate included in the initial interpretable speech emotion recognition model through a first preset adjustment algorithm. Then, based on the adjusted learning rate, the above-mentioned execution subject can adjust the network parameters of the initial interpretable speech emotion recognition model through a second preset adjustment algorithm. Then, the above-mentioned execution subject can determine the adjusted initial interpretable speech emotion recognition model as the initial interpretable speech emotion recognition model. For example, the first preset adjustment algorithm can be: CyclicLR (Cyclical Learning Rates for Training Neural Networks, a learning rate adjustment algorithm based on a deep learning model) algorithm. The second preset adjustment algorithm can be: RMSprop (Root Mean Square Propagation, root mean square directional propagation) algorithm.

[0086] The second step is to exclude the used samples from the speech training sample set and reselect the target speech training sample from the remaining samples. In practice, first, the execution subject can remove the selected speech training samples from the speech training sample set to obtain the removed speech training sample set. Then, the execution subject can select the target speech training sample from the removed speech training sample set.

[0087] The third step is to perform further training steps to optimize the model performance. In practice, the above execution subject can perform the above training steps again.

[0088] Step 105, in response to receiving the target voice information, inputting the target voice information into the interpretable voice emotion recognition model to obtain an emotion label.

[0089] In some embodiments, the execution subject may, in response to receiving the target voice information, input the target voice information into the interpretable voice emotion recognition model to obtain an emotion label, wherein the target voice information may be an unlabeled conversational voice.

[0090] The above-mentioned various embodiments of the present application have the following beneficial effects: through the interpretable speech emotion recognition algorithm based on multi-level feature disentanglement of some embodiments of the present application, first, a speech training sample set is obtained, wherein the speech training samples in the speech training sample set include: sample speech information and sample emotion labels. Secondly, an initial interpretable speech emotion recognition model is constructed, wherein the initial interpretable speech emotion recognition model includes: an initial feature extraction network model, an initial fusion multi-level emotion feature hybrid autoencoder network model, and an initial multi-layer perception classification model. Then, a target speech training sample is selected from the speech training sample set. Then, for the target speech training sample, the following training steps are performed: the sample speech information included in the target training sample is input into the initial feature extraction network model to obtain the initial general emotion feature information; the initial general emotion feature information is input into the initial fusion multi-level emotion feature hybrid autoencoder network model to obtain the initial multi-level emotion feature information; the initial multi-level emotion feature information is input into the initial multi-layer perception classification model to obtain the initial emotion label; based on the preset loss function, the difference value between the initial emotion label and the sample emotion label included in the target training sample is determined; in response to determining that the difference value is less than the preset difference value, the initial interpretable speech emotion recognition model is determined as the trained interpretable speech emotion recognition model. Finally, in response to receiving the target speech information, the above target speech information is input into the above interpretable speech emotion recognition model to obtain the emotion label. Therefore, the present application proposes an interpretable speech emotion recognition algorithm based on multi-level feature disentanglement, and constructs a fusion multi-level emotion feature hybrid autoencoder network, which can comprehensively capture the multi-level emotion information in the speech and enhance the efficiency and accuracy of the autoencoder in feature disentanglement. In addition, this application also constructs a multi-layer perception classification model based on KAN (Kolmogorov-Arnold Networks, a new neural network architecture), which can help the model capture complex nonlinear relationships in the data and further improve the accuracy of classification. Thus, the accuracy of speech emotion recognition can be improved.

[0091] The present application also provides a computer device 200. Figure 2As shown, the computer device 200 includes: a bus 201, a processor 202, a memory 203 and a communication interface 204. The processor 202, the memory 203 and the communication interface 204 communicate with each other through the bus 201. The computer device 200 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computer device 200.

[0092] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 The bus 201 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 201 may include a path for transmitting information between various components of the computer device 200 (for example, the memory 203, the processor 202, and the communication interface 204).

[0093] The processor 202 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0094] The memory 203 may include a volatile memory, such as a random access memory (RAM). The memory 203 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0095] The memory 203 stores executable program codes, and the processor 202 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module, sampling module, determination module, and mixing module, thereby implementing the above-mentioned interpretable speech emotion recognition algorithm based on multi-level feature disentanglement. That is, the memory 203 stores instructions for executing the above-mentioned interpretable speech emotion recognition based on multi-level feature disentanglement.

[0096] The communication interface 204 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computer device 200 and other devices or a communication network.

[0097] An embodiment of the present application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute the above-mentioned interpretable speech emotion recognition algorithm based on multi-level feature disentanglement.

[0098] The embodiment of the present application also provides a computer-readable storage medium. The above-mentioned computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The above-mentioned available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state hard disk), etc. The computer-readable storage medium includes instructions, and the above-mentioned instructions instruct the computing device to execute the above-mentioned interpretable speech emotion recognition algorithm based on multi-level feature disentanglement.

[0099] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An interpretable speech emotion recognition method based on multi-level feature disentanglement, comprising: Acquire a speech training sample set, wherein the speech training samples in the speech training sample set include: sample speech information and sample emotion labels; Constructing an initial interpretable speech emotion recognition model, wherein the initial interpretable speech emotion recognition model includes: an initial feature extraction network model, an initial fusion multi-level emotion feature hybrid autoencoder network model, and an initial multi-layer perception classification model; Selecting a target speech training sample from a speech training sample set; For the target speech training sample, perform the following training steps: Inputting sample speech information included in the target training sample into the initial feature extraction network model to obtain initial general emotion feature information; Inputting the initial universal emotion feature information into the initial fusion multi-level emotion feature hybrid autoencoder network model to obtain the initial multi-level emotion feature information; Input the initial multi-level emotional feature information into the initial multi-layer perception classification model to obtain the initial emotional label; Based on a preset loss function, determining a difference value between an initial emotion label and a sample emotion label included in a target training sample; In response to determining that the difference value is less than a preset difference value, determining the initial interpretable speech emotion recognition model as the trained interpretable speech emotion recognition model; In response to receiving the target speech information, the target speech information is input into the interpretable speech emotion recognition model to obtain an emotion label.

2. The interpretable speech emotion recognition method based on multi-level feature disentanglement according to claim 1, wherein: The method further comprises: In response to determining that the difference value is greater than or equal to the preset difference value, performing the following adjustment steps: Adjusting parameters of the initial interpretable speech emotion recognition model, and determining the adjusted model as an optimized version; Excluding the used samples from the speech training sample set, and reselecting the target speech training samples from the remaining samples; Further training steps are performed to optimize the model performance.

3. The interpretable speech emotion recognition method based on multi-level feature disentanglement according to claim 1, wherein: The constructing of an initial interpretable speech emotion recognition model comprises: Construct an initial feature extraction network model; Construct an initial fusion multi-level emotional feature hybrid autoencoder network model; Build an initial multi-layer perception classification model; The initial feature extraction network model, the initial fusion multi-level emotion feature hybrid autoencoder network model, and the initial multi-layer perception classification model are fused to generate an initial interpretable speech emotion recognition model.

4. The interpretable speech emotion recognition method based on multi-level feature disentanglement according to claim 1, wherein: The initial fusion multi-level emotional feature hybrid autoencoder network model includes: an emotional feature disentanglement module, an emotional feature integration module, and a decoder, wherein the emotional feature disentanglement module includes: a static emotional feature disentanglement submodule and a dynamic emotional feature disentanglement submodule, and the emotional feature integration module includes: an emotional feature fusion unit and an emotional feature reshaping unit; And the initial universal emotion feature information is input into the initial fusion multi-level emotion feature hybrid autoencoder network model to obtain the initial multi-level emotion feature information, including: The initial universal emotion feature information is input into the static emotion feature disentanglement submodule to obtain the initial static emotion feature information, wherein the static emotion feature disentanglement submodule refers to the following formula: v=σ(W·x+b), Among them, v represents the initial static emotional feature information, x represents the initial general emotional feature information, W and b represent the weight matrix and bias in the encoding process, and σ represents the activation function; The initial general emotional feature information is input into the dynamic emotional feature disentanglement submodule to obtain the initial dynamic emotional feature information, wherein the dynamic emotional feature disentanglement submodule refers to the following formula: u,(h n ,c n )=LSTM(x,(h0,c0)), Among them, h n and c n They represent hidden vectors and memory cells respectively, and their dimensions are the same. h0 and c0 are their initial states, u represents the initial dynamic emotional feature information, and LSTM represents the dynamic emotional feature disentanglement submodule. The initial static emotional feature information and the initial dynamic emotional feature information are input into the emotional feature fusion unit to obtain the initial comprehensive emotional feature information, wherein the initial comprehensive emotional feature information includes multi-dimensional emotional feature information. The emotional feature fusion unit refers to the following formula: x fuse =Concat(v,u), Among them, x fused Represents the initial comprehensive emotional feature information, and Concat is used to connect v and u; The initial comprehensive emotional feature information is input into the emotional feature reconstruction unit to obtain the initial reshaped emotional feature information, wherein the emotional feature reconstruction unit refers to the following formula: Among them, h represents the initial reshaped emotional feature information, μ(x fused ) represents the mean vector of the initial comprehensive emotional feature information, σ(x fused ) represents the variance vector of the initial comprehensive emotional feature information, ε∈N(0,I) represents the random noise vector sampled from the standard normal distribution, N represents "normal", which is the most common continuous random variable distribution in probability theory, and I represents the variance of the standard normal distribution, that is, the case where the mean is 0 and the variance is 1. represents element-wise multiplication; The initial reshaped emotional feature information is input into the decoder to obtain the initial multi-level emotional feature information, wherein the decoder refers to the following formula: x′=σ(W′·h+b′), Among them, x′ represents the initial multi-level emotional feature information, h represents the initial reshaped emotional feature information, W and b represent the weight matrix and bias in the decoding process, and σ represents the activation function.

5. The interpretable speech emotion recognition method based on multi-level feature disentanglement according to claim 3, wherein: The constructing of the initial multi-layer perception classification model comprises: Determine the perceptual classification model; The Kaiming initialization strategy is used to adjust the weights of each network layer of the perception classification model; Performing noise injection processing on the adjusted perceptual classification model to adjust the scaling coefficient of the spline function in the model; Regularization is performed on the adjusted perceptual classification model to adjust parameter weights in the model; The adjusted perceptual classification model is subjected to activation adjustment processing to generate an initial multi-layer perceptual classification model.

6. The interpretable speech emotion recognition method based on multi-level feature disentanglement according to claim 1, wherein: The loss functions include: classification loss function and reconstruction loss function; And the step of determining the difference value between the initial emotion label and the sample emotion label included in the target training sample based on the loss function includes: Based on the classification loss function, determining a first difference value between the initial emotion label and the sample emotion label included in the target training sample; Determining a second difference value between the initial universal emotion feature information and the initial multi-level emotion feature information based on the reconstruction loss function; A sum of the first difference value and the second difference value is determined as the difference value.

Images