Speech emotion recognition method and device based on adjacent frame similarity fusion

Abstract

The application discloses a speech emotion recognition method and device based on adjacent frame similarity fusion. The method comprises the following steps: acquiring speech data, and obtaining preliminary features after feature extraction; performing feature slicing on the preliminary features, and then performing adjacent frame similarity fusion to obtain fusion features; making the fusion features interact in a frequency domain space, and obtaining final features after residual connection of the obtained features; and recognizing speech emotion according to the final features. The application fully considers the time sequence information of the speech signal, enhances the expression of the hierarchical relationship of the emotion data, and can realize better classification effect in emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and more specifically to a method and apparatus for speech emotion recognition based on adjacent frame similarity fusion. Background Technology

[0002] The task of speech emotion recognition aims to identify and understand the speaker's emotional state by analyzing emotional information in speech signals. However, due to the ambiguity, subjectivity, complexity, and time-varying nature of emotions, extracting and recognizing emotional information from speech signals is extremely complex.

[0003] In previous studies, many methods relied on artificially designed traditional acoustic features, such as fundamental frequency and energy. The extraction of these features not only depends on professional knowledge and experience, but also leads to information loss. In addition, speech signals are highly correlated with temporal information. Emotions in speech are not only static features, but also include changes and interactions in the time series. However, in many traditional emotion recognition methods, this temporal interaction characteristic is often ignored. Summary of the Invention

[0004] In view of this, in order to at least partially solve the above-mentioned technical problems, the present invention provides a speech emotion recognition method and apparatus based on adjacent frame similarity fusion. By measuring the degree of similarity between adjacent frames, the method captures the temporal relationship in the speech signal, enhances the communication and fusion of features in the time dimension, and enables the features to better express the dynamic changes of emotions in the time dimension.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A speech emotion recognition method based on adjacent frame similarity fusion includes the following steps:

[0007] Acquire speech data and extract features to obtain preliminary features;

[0008] After segmenting the initial features, the similarity of adjacent frames is fused to obtain the fused features;

[0009] The fused features interact in the frequency domain space, and the resulting features are obtained by residual connection to obtain the final features.

[0010] Voice emotion is identified based on the final features.

[0011] According to an embodiment of the present invention, a pre-trained WavLM model is used for feature extraction, and the output of the last output layer of the model is used as the preliminary extracted features; preferably, the features are saved after format conversion.

[0012] According to an embodiment of the present invention, feature segmentation of the initial features includes:

[0013] Perform a convolution operation on the initial features to obtain convolutional features;

[0014] Positional information is added after padding the convolutional features in the time dimension.

[0015] According to an embodiment of the present invention, the step of adjacent frame similarity fusion includes:

[0016] The features after feature segmentation are divided equally along the time dimension;

[0017] Calculate the inter-frame cosine similarity matrix for each group of features;

[0018] The cosine similarity matrix is ​​weighted onto the features within the corresponding groups, and then the features from each group are concatenated to obtain the fused features.

[0019] According to an embodiment of the present invention, enabling fused features to interact in the frequency domain space includes:

[0020] After transforming the fused features to the frequency domain, the global relationships of the features are captured through a learnable weight matrix, and then transformed back to the spatial domain.

[0021] According to an embodiment of the present invention, the obtained features are processed through residual connections to obtain the final features, including:

[0022] The obtained features are then processed through normalization and MLP modules, and then residually connected with the fused features.

[0023] According to an embodiment of the present invention, recognizing speech emotion based on the final feature includes:

[0024] The final features are input into a four-class classifier for emotion recognition, with emotion categories including happiness, anger, sadness, and neutrality.

[0025] According to an embodiment of the present invention, the recognition process is trained based on the following loss function:

[0026]

[0027] In the formula, α is The weight, The cross-entropy loss is in Euclidean space, where β is... The weight, For distance loss in hyperbolic space; where, and The expression is:

[0028]

[0029]

[0030] Among them, y i,euc , These are the i-th elements of the label vector and output vector in Euclidean space, respectively, where N is the number of categories; y hy , These are the mappings of the label vector and the output vector onto hyperbolic space, respectively.

[0031] In addition, this application also discloses a speech emotion recognition device based on adjacent frame similarity fusion. This device applies a speech emotion recognition method based on adjacent frame similarity fusion as described above, including:

[0032] The feature extraction unit is used to acquire speech data and extract features to obtain preliminary features.

[0033] The feature fusion unit is used to perform feature segmentation on the preliminary features and then fuse the similarity between adjacent frames to obtain fused features.

[0034] The feature generation unit is used to enable the fused features to interact in the frequency domain space, and the resulting features are obtained by residual connection to obtain the final features.

[0035] The emotion recognition unit identifies the emotion in the speech based on the final features.

[0036] As can be seen from the above technical solutions, the present invention discloses a speech emotion recognition method and device based on adjacent frame similarity fusion. Compared with the prior art, the present invention, by constructing feature segments, fusing adjacent frame similarity and extracting frequency domain features, not only avoids the information loss problem caused by manually designing acoustic features in traditional methods, but also fully considers the temporal information of speech signals.

[0037] In particular, in the adjacent frame similarity fusion module, the interaction between features in the time dimension is enhanced by calculating the cosine similarity matrix and weighted fusion, which makes up for the shortcomings of traditional methods in terms of temporal interaction characteristics.

[0038] Furthermore, this application combines the complementarity between Euclidean space and hyperbolic space to design a loss function, representing and understanding the relationship between the output and label vectors from different representation spaces. This loss function guides the model's learning, enhances the expression of hierarchical relationships in sentiment data, and achieves better classification results in sentiment recognition. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a flowchart of the speech emotion recognition method based on adjacent frame similarity fusion of the present invention;

[0041] Figure 2 This is an example flowchart of the speech emotion recognition method based on adjacent frame similarity fusion according to the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] This invention addresses the limitations of existing methods, such as the performance constraints of manually designed and extracted features and the neglect of temporal contextual interactions in speech features. It proposes a speech emotion recognition method based on adjacent frame similarity fusion, such as… Figure 1 As shown, the steps include:

[0044] S1. Acquire speech data and extract features to obtain preliminary features;

[0045] S2. After segmenting the initial features, perform similarity fusion between adjacent frames to obtain fused features;

[0046] S3. The fused features interact in the frequency domain space, and the resulting features are obtained by residual connection to obtain the final features.

[0047] S4. Recognize the emotion in the speech based on the final features.

[0048] In one exemplary embodiment, such as Figure 2 As shown,

[0049] Step S1: Acquire speech data, perform feature extraction to obtain preliminary features;

[0050] First, this embodiment uses the IEMOCAP dataset as the original speech signal; and uses the pre-trained WavLM model for feature extraction, that is, loading the parameters of the pre-trained model WavLM, traversing the session folder of the IEMOCAP dataset, reading and loading the speech file, and inputting the file into the pre-trained model for preliminary feature extraction.

[0051] In one embodiment, the sampling rate is 16kHz, and the output of the last output layer of the model is used as the initial feature. The feature is saved in npy format so that the feature is persistent and can be reloaded when needed.

[0052] Step S2 involves segmenting the initial features and then fusing adjacent frames based on their similarity to obtain fused features. In this embodiment, this specifically includes:

[0053] S21. Perform feature segmentation on the initial features, including:

[0054] Perform a convolution operation on the initial features to obtain convolutional features;

[0055] The convolutional features are padded along the time dimension to maintain their original size before and after convolution; then positional information is added, resulting in:

[0056] X = Conv(x) + P

[0057] The initial features of the input are: The defined positional encoding.

[0058] S22, The steps of adjacent frame similarity fusion include:

[0059] The features after feature segmentation are divided equally along the time dimension, i.e.

[0060]

[0061] Where G is the number of groups into which the feature is equally divided, and n f The number of feature frames in each group is T, and the time dimension is T.

[0062] S23. Calculate the cosine similarity matrix S between frames in each feature group; for each feature frame in each group, calculate the cosine similarity between frames to obtain a cosine similarity matrix, which represents the similarity between adjacent frames:

[0063]

[0064] Among them, X i X j express The feature vectors of the i-th and j-th frames.

[0065] S24. The cosine similarity matrix is ​​weighted onto the corresponding intra-group features, and then the features from each group are concatenated to obtain the fused features. This involves multiplying the similarity matrix by the intra-group features to leverage the temporal similarity between adjacent frames and enhance the inter-channel interaction of features in the temporal dimension. The formula is expressed as:

[0066]

[0067] S G This represents the cosine similarity matrix of the Gth group, with a size of n. f ×n f , This represents the k-th feature obtained by dividing the data equally. This represents the features of the Gth group after weighted processing.

[0068] Then feature concatenation is performed to obtain X. (2) :

[0069]

[0070] Step S3: The fused features interact in the frequency domain space, and the resulting features are then connected by residuals to obtain the final features.

[0071] In this embodiment, the fused features are made to interact in the frequency domain space through a global filter layer. That is, a learnable weight matrix with the same size as the feature is first defined to cover the entire feature.

[0072] The features are transformed from the spatial domain to the frequency domain using a two-dimensional discrete Fourier transform.

[0073] Then, a global filter is used to capture the global, long-range relationships of features in the frequency domain; that is, the weight matrix is ​​multiplied element-wise with the transformed features.

[0074] Finally, the inverse Fourier transform is applied to transform the features from the frequency domain back to the spatial domain, obtaining feature X. (3) .

[0075] Furthermore, make feature X (3) After passing through normalization and MLP modules, the features are residually connected to the fused features to obtain the final feature representation:

[0076] X (4) =X (2) +MLP(LN(X (3) ))

[0077] Where MLP(·) represents a multilayer perceptron, the activation function is GELU, LN(·) represents layer normalization, and X (4) This is the final emotional characteristic obtained.

[0078] Step S4, recognizing voice emotion based on the final features; including:

[0079] The final features are input into a four-class classifier for emotion recognition, with emotion categories including happiness / excitement, anger, sadness, and neutrality.

[0080] In this embodiment, the initial learning rate is set to 3×10. -4 The algorithm is optimized using stochastic gradient descent. The rating metrics include weighted accuracy and unweighted accuracy.

[0081] To further optimize the above technical solution, the emotion recognition process needs to be pre-trained.

[0082] First, the filenames and corresponding label information of the preliminary features are read, and their size is fixed at 324×1024. Insufficient data is padded with zeros, and excess data is truncated. This invention uses leave-one-out cross-validation to divide the processed feature data into training and validation sets for five sessions in the IEMOCAP dataset.

[0083] Because Euclidean space effectively describes the planar structure and linear correlation between data, while hyperbolic space, being a non-Euclidean geometric structure, better reflects the non-linear hierarchical structure between data, this application considers the complementarity of the mappings of output and label vectors in Euclidean and non-Euclidean spaces from the perspective of representation space. It calculates the cross-entropy loss in Euclidean space and the distance between the output and label vectors in hyperbolic space. The loss function designed in this embodiment is as follows:

[0084]

[0085] In the formula, α is The weight, The cross-entropy loss is in Euclidean space, where β is... The weight, The distance loss is in hyperbolic space; the loss function combination takes into account the complementarity of Euclidean and non-Euclidean spaces.

[0086] in, and The expression is:

[0087]

[0088] Among them, y i,euc , These are the i-th elements of the label vector and output vector in Euclidean space, respectively, where N is the number of categories; y hy , These are the mappings of the label vector and the output vector onto hyperbolic space, respectively.

[0089] In this embodiment, the distance in hyperbolic space is calculated by first mapping the vector to hyperbolic geometric space using the Poincaré disk model, as shown in the following formula:

[0090]

[0091] Where tanh(·) represents the hyperbolic tangent function, c represents the negative curvature of the sphere, and x represents the label vector or output vector. hy This represents the corresponding vector in hyperbolic space.

[0092] The formula for calculating hyperbolic distance is as follows:

[0093]

[0094] Where, d hy Represents vector y hy and hyperbolic distance, y hy and These represent the output vector and label vector mapped to hyperbolic space, respectively; Represents operators in hyperbolic space.

[0095] In another embodiment, this application also discloses a speech emotion recognition device based on adjacent frame similarity fusion. This device applies a speech emotion recognition method based on adjacent frame similarity fusion as described above, including:

[0096] The feature extraction unit is used to acquire speech data and extract features to obtain preliminary features.

[0097] The feature fusion unit is used to perform feature segmentation on the preliminary features and then fuse the similarity between adjacent frames to obtain fused features.

[0098] The feature generation unit is used to enable the fused features to interact in the frequency domain space, and the resulting features are obtained by residual connection to obtain the final features.

[0099] The emotion recognition unit identifies the emotion in the speech based on the final features.

[0100] Since the content of this device is consistent with the speech emotion recognition method based on adjacent frame similarity fusion, it will not be described again here.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0102] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech emotion recognition method based on adjacent frame similarity fusion, characterized in that, obtaining speech data and performing feature extraction to obtain preliminary features; performing feature slicing and adjacent frame similarity fusion on the preliminary features in turn to obtain fused features; performing feature slicing on the initial features, including: performing convolution operation on the initial features to obtain convolution features; adding position information after padding the convolution features in the time dimension; the step of adjacent frame similarity fusion includes: equally dividing the sliced features from the time dimension; calculating the inter-frame cosine similarity matrix in each group of features; weighting the cosine similarity matrix to the corresponding group of features, and splicing each group of features to obtain fused features; making the fused features interact in the frequency domain space, and obtaining the final features after residual connection of the obtained features; recognizing speech emotion according to the final features.

2. The speech emotion recognition method based on adjacent frame similarity fusion according to claim 1, characterized in that, A pre-trained WavLM model is used for feature extraction, and the output of the model is used as the preliminary features. 3.The speech emotion recognition method based on adjacent frame similarity fusion according to claim 1, characterized in that, Making the fused features interact in the frequency domain space includes: After transforming the fused features to the frequency domain, the global relationship of the features is captured through a learnable weight matrix, and then transformed back to the spatial domain.

4. The speech emotion recognition method based on adjacent frame similarity fusion according to claim 1, characterized in that, The final features obtained after residual connection of the obtained features include: The obtained features are sequentially normalized and passed through an MLP module, and then residual connected with the fused features.

5. The speech emotion recognition method based on adjacent frame similarity fusion according to claim 1, characterized in that, Recognizing speech emotion according to the final features includes: inputting the final features into a four-class classifier for emotion recognition, and the emotion categories include happy, angry, sad, and neutral.

6. The speech emotion recognition method based on adjacent frame similarity fusion according to claim 1, characterized in that, The recognition process is trained according to the following loss function, In the formula, a is the weight of is the cross-entropy loss in Euclidean space, and β is the weight of is the distance loss in hyperbolic space; wherein, and the expression of where y i,euc , are the i-th element of the label vector, output vector in Euclidean space, respectively, and N is the number of classes; y hy , are the mapping of the label vector, output vector in hyperbolic space, respectively.

7. A speech emotion recognition device based on adjacent frame similarity fusion, characterized in that, Applying a speech emotion recognition method based on adjacent frame similarity fusion according to any one of claims 1-6, including: a feature extraction unit for obtaining speech data and performing feature extraction to obtain preliminary features; a feature fusion unit for performing feature slicing and adjacent frame similarity fusion on the preliminary features in turn to obtain fused features; a feature generation unit for making the fused features interact in the frequency domain space, and obtaining the final features after residual connection of the obtained features; an emotion recognition unit for recognizing speech emotion according to the final features.

Images