Speech emotion recognition method and device based on adjacent frame similarity fusion

Abstract

The invention discloses a speech emotion recognition method and device based on adjacent frame similarity fusion. The method comprises the following steps: acquiring voice data, and performing feature extraction to obtain initial features; carrying out feature fragmentation on the preliminary features, and then carrying out adjacent frame similarity fusion to obtain fused features; performing interaction on the fusion features in a frequency domain space, and performing residual connection on the obtained features to obtain final features; and recognizing the speech emotion according to the final feature. According to the method, the time sequence information of the voice signal is fully considered, the expression of the hierarchical relationship of the emotion data is enhanced, and a better classification effect can be realized in emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and more specifically, to a speech emotion recognition method and device based on adjacent frame similarity fusion. Background Art

[0002] The task of speech emotion recognition aims to identify and understand the emotional state of the speaker by analyzing the emotional information in the speech signal. However, due to the characteristics of emotions such as ambiguity, subjectivity, complexity, and time-variability, it is very complex to extract and recognize emotional information from speech signals.

[0003] In past research, many methods relied on traditional acoustic features designed manually, such as fundamental frequency, energy, etc. The extraction of these features not only depends on professional knowledge and experience, but also leads to information loss; in addition, the speech signal is highly correlated with temporal information, and the emotion in speech includes not only static features, but also changes and interactions in the time series; however, in many traditional emotion recognition methods, this characteristic of temporal interaction is often ignored. Summary of the Invention

[0004] In view of this, in order to at least partially solve the above technical problems, the present invention provides a speech emotion recognition method and device based on adjacent frame similarity fusion, which captures the temporal relationship in the speech signal by measuring the similarity between adjacent frames, enhances the forward and backward communication and fusion of features in the time dimension, and enables the features to better express the dynamic changes of emotions in the time dimension.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A speech emotion recognition method based on adjacent frame similarity fusion, the steps include:

[0007] Obtain speech data and perform feature extraction to obtain preliminary features;

[0008] After performing feature slicing on the preliminary features, perform adjacent frame similarity fusion to obtain fused features;

[0009] Enable the fused features to interact in the frequency domain space, and the obtained features are obtained through residual connection to obtain the final features;

[0010] Recognize the speech emotion according to the final features.

[0011] According to an embodiment of the present invention, a pre-trained WavLM model is used for feature extraction, and the output of the last output layer of the model is used as the extracted preliminary features; preferably, after format conversion, it is saved.

[0012] According to an embodiment of the present invention, feature slicing of the initial features includes:

[0013] Perform a convolution operation on the initial features to obtain convolution features;

[0014] Add positional information after padding the convolution features in the time dimension.

[0015] According to an embodiment of the present invention, the steps of adjacent frame similarity fusion include:

[0016] Equally divide the features after feature sharding in the time dimension;

[0017] Calculate the inter-frame cosine similarity matrix for each group of features;

[0018] Weight the cosine similarity matrix to the corresponding features within each group respectively, and then splice the features of each group to obtain fused features.

[0019] According to an embodiment of the present invention, enabling the fused features to interact in the frequency domain space includes:

[0020] After transforming the fused features into the frequency domain, capture the global relationship of the features through a learnable weight matrix, and then transform back to the spatial domain.

[0021] According to an embodiment of the present invention, the obtained features obtain the final features after residual connection, including:

[0022] The obtained features are sequentially passed through a normalization and MLP module and then subjected to residual connection with the fused features.

[0023] According to an embodiment of the present invention, identifying the speech emotion based on the final features includes:

[0024] Input the final features into a four-class classifier for emotion recognition, and the emotion categories include happy, angry, sad, and neutral.

[0025] According to an embodiment of the present invention, training the recognition process according to the following loss function

[0026]

[0027] In the formula, α is the weight of is the cross-entropy loss in Euclidean space, β is the weight of is the distance loss in hyperbolic space; wherein, and The expressions of are:

[0028]

[0029]

[0030] where y i,euc and are the i-th elements of the label vector and the output vector in the Euclidean space respectively, and N is the number of categories; y hy and are the mappings of the label vector and the output vector in the hyperbolic space respectively.

[0031] In addition, the present application also discloses a voice emotion recognition device based on adjacent frame similarity fusion. This device applies a voice emotion recognition method based on adjacent frame similarity fusion as described above, and includes:

[0032] A feature extraction unit, configured to obtain voice data and perform feature extraction to obtain preliminary features;

[0033] A feature fusion unit, configured to perform adjacent frame similarity fusion after feature sharding on the preliminary features to obtain fused features;

[0034] A feature generation unit, configured to enable the fused features to interact in the frequency domain space, and the obtained features obtain final features after residual connection;

[0035] An emotion recognition unit, configured to recognize the voice emotion according to the final features.

[0036] It can be seen from the above technical solutions that the present invention discloses a voice emotion recognition method and device based on adjacent frame similarity fusion. Compared with the prior art, the present invention constructs feature sharding, adjacent frame similarity fusion and frequency domain feature extraction, which not only avoids the problem of information loss caused by artificially designed acoustic features in traditional methods, but also fully considers the temporal information of voice signals.

[0037] Especially in the adjacent frame similarity fusion module, by calculating the cosine similarity matrix and weighted fusion, the channel interaction between features in the time dimension is enhanced, making up for the deficiency of traditional methods in temporal interaction characteristics.

[0038] In addition, the present application designs a loss function in combination with the complementarity between the Euclidean space and the hyperbolic space, represents and understands the relationship between the output and the label vector from different representation spaces, guides the model to learn through this loss function, enhances the expression of the hierarchical relationship of emotion data, and achieves a better classification effect in emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0040] Figure 1 This is the flowchart of the speech emotion recognition method based on adjacent frame similarity fusion of the present invention;

[0041] Figure 2 This is an example flowchart of the speech emotion recognition method based on adjacent frame similarity fusion of the present invention. Detailed implementation manners

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0043] Aiming at the limitations of the features extracted by manual design in the existing methods and the neglect of the temporal communication of speech features in the prior art, the present invention proposes a speech emotion recognition method based on adjacent frame similarity fusion, as Figure 1 shown, the steps include:

[0044] S1. Obtain speech data and perform feature extraction to obtain preliminary features;

[0045] S2. After performing feature slicing on the preliminary features, perform adjacent frame similarity fusion to obtain fused features;

[0046] S3. Make the fused features interact in the frequency domain space, and the obtained features are obtained as final features after residual connection;

[0047] S4. Recognize the speech emotion according to the final features.

[0048] In an exemplary embodiment, as Figure 2 shown,

[0049] Step S1, obtain speech data, and obtain preliminary features after feature extraction;

[0050] First of all, in this embodiment, the IEMOCAP dataset is used as the original speech signal; and a pre-trained WavLM model is used for feature extraction, that is, the parameters of the pre-trained model WavLM are loaded, the session folders of the IEMOCAP dataset are traversed, the speech files are read and loaded, and the files are input into the pre-trained model for preliminary feature extraction;

[0051] In one embodiment, the sampling rate is 16KHz, and the output of the last output layer of the model is used as the preliminary feature. At the same time, the feature is saved in the npy format to make the feature persistently saved and can be reloaded when needed.

[0052] In step S2, after performing feature sharding on the preliminary features, adjacent-frame similarity fusion is carried out to obtain fused features. In this embodiment, it specifically includes:

[0053] S21. Perform feature sharding on the initial features, including:

[0054] Perform a convolution operation on the initial features to obtain convolution features;

[0055] Pad the convolution features in the time dimension to keep the size of the features unchanged before and after convolution; then add position information to obtain:

[0056] X = Conv(x) + P

[0057] where the input initial feature is is the defined position encoding.

[0058] S22. The steps of adjacent-frame similarity fusion include:

[0059] Divide the features after feature sharding equally in the time dimension, that is

[0060]

[0061] where G is the number of groups for equal division of the features, n f is the number of feature frames in each group, and T is the size of the time dimension.

[0062] S23. Calculate the inter-frame cosine similarity matrix S for each group of features; for the feature frames within each group, calculate the cosine similarity between frame and frame to obtain a cosine similarity matrix, which characterizes the similarity between adjacent frames:

[0063]

[0064] where X i 、X j represents the feature vectors of the i-th frame and the j-th frame in

[0065] S24. Weight the cosine similarity matrix to the corresponding features within each group respectively, and then splice the features of each group to obtain the fused features. That is, multiply the similarity matrix by the features within the group to utilize the similarity between adjacent frames in time series and enhance the inter-channel interaction of the features in the time dimension. The formula is expressed as:

[0066]

[0067] S G represents the cosine similarity matrix of the G-th group, with a size of n f ×n f , Represents the k-th group of features obtained by equal division, represents the features after the G-th group of weighted processing.

[0068] Then perform feature concatenation to obtain X (2) :

[0069]

[0070] Step S3, enable the fused features to interact in the frequency domain space, and the obtained features are connected through residuals to obtain the final features;

[0071] In this embodiment, the fused features are enabled to perform information interaction in the frequency domain space through a global filter layer, that is, first define a learnable weight matrix with the same size as the features to cover the entire features;

[0072] And use the two-dimensional discrete Fourier transform to transform the features, transforming the features from the spatial domain to the frequency domain;

[0073] Then use a global filter to capture the global and long-distance relationships of the features in the frequency domain; that is, multiply the weight matrix and the transformed features element-wise;

[0074] Finally, apply the inverse Fourier transform to transform the features from the frequency domain back to the spatial domain to obtain the features X (3) .

[0075] Furthermore, enable the features X (3) to pass through normalization and the MLP module in sequence and then perform residual connection with the fused features to obtain the final feature representation:

[0076] X (4) = X (2) + MLP(LN(X (3) ))

[0077] where MLP(·) represents a multi-layer perceptron, the activation function is GELU, LN(·) represents layer normalization, and X (4) is the finally obtained sentiment feature.

[0078] Step S4, perform speech emotion recognition according to the final features; including:

[0079] Input the final features into a four-class classifier for emotion recognition, and the emotion categories include happy / excited, angry, sad, and neutral.

[0080] In this embodiment, the initial learning rate is set to 3×10 -4 , and the random gradient descent algorithm is used for optimization. The rating metrics include weighted accuracy and unweighted accuracy.

[0081] To further optimize the above technical solution, it is necessary to pre-train the above emotion recognition process in advance;

[0082] First, read the file names corresponding to the preliminary features and the corresponding label information, and fix their size to 324×1024. The insufficient part is filled with zeros, and the excess part is truncated. In the present invention, for five sessions in the IEMOCAP dataset, the processed feature data is divided into a training set and a validation set by using the leave-one-out cross-validation method.

[0083] Since the Euclidean space can effectively describe the planar structure and linear correlation between data; the hyperbolic space is a non-Euclidean geometric structure that better reflects the non-linear hierarchical structure between data. Therefore, from the perspective of the representation space, this application considers the complementarity of the mapping of the output and label vectors in the Euclidean space and the non-Euclidean space, and calculates the cross-entropy loss in the Euclidean space and the distance between the output and label vectors in the hyperbolic space; the loss function designed in this embodiment is as follows:

[0084]

[0085] In the formula, α is the weight of, is the cross-entropy loss in the Euclidean space, β is the weight of, is the distance loss in the hyperbolic space; the combined loss function considers the complementarity of the Euclidean space and the non-Euclidean space.

[0086] Among them, and The expressions of are:

[0087]

[0088] Among them, y i,euc , are the i-th elements of the label vector and the output vector in the Euclidean space respectively, and N is the number of categories; y hy , are the mappings of the label vector and the output vector in the hyperbolic space respectively.

[0089] In this embodiment, to calculate the distance in the hyperbolic space, first map the vector to the hyperbolic geometric space through the Poincaré disk model, and the formula is as follows:

[0090]

[0091] Among them, tanh(·) represents the hyperbolic tangent function, c represents the spherical negative curvature, x represents the label vector or the output vector, and x hy represents the corresponding vector in the hyperbolic space.

[0092] For the calculation of the hyperbolic distance, the formula is as follows:

[0093]

[0094] where d hy represents the hyperbolic distance between the vectors y hy and , and y hy and represent the output vector and the label vector mapped into the hyperbolic space respectively; represents an operator in the hyperbolic space.

[0095] In another embodiment, the present application also discloses a speech emotion recognition device based on adjacent frame similarity fusion. The device applies a speech emotion recognition method based on adjacent frame similarity fusion as described above, and includes:

[0096] A feature extraction unit, configured to obtain speech data and perform feature extraction to obtain preliminary features;

[0097] A feature fusion unit, configured to perform adjacent frame similarity fusion after feature sharding on the preliminary features to obtain fused features;

[0098] A feature generation unit, configured to enable the fused features to interact in the frequency domain space, and the obtained features obtain final features after residual connection;

[0099] An emotion recognition unit, configured to recognize the speech emotion according to the final features.

[0100] Since the content of this device is the same as that of the speech emotion recognition method based on adjacent frame similarity fusion, it will not be repeated here.

[0101] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0102] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech emotion recognition method based on adjacent frame similarity fusion, characterized in that: Acquire speech data and perform feature extraction to obtain preliminary features; The preliminary features are sequentially segmented and the adjacent frames are fused similarly to obtain fused features; The fused features are made to interact in the frequency domain, and the obtained features are connected by residuals to obtain the final features; The speech emotion is recognized according to the final features.

2. A speech emotion recognition method based on adjacent frame similarity fusion according to claim 1, characterized in that: The pre-trained WavLM model is used for feature extraction, and the output of the model is used as the preliminary features.

3. The method for speech emotion recognition based on adjacent frame similarity fusion according to claim 1, characterized in that: Perform feature sharding on the initial features, including: Perform convolution operation on the initial features to obtain convolution features; The convolutional features are padded in the time dimension and then the position information is added.

4. The method for speech emotion recognition based on adjacent frame similarity fusion according to claim 1, characterized in that: The steps of similarity fusion of adjacent frames include: Divide the sharded features equally from the time dimension; Calculate the cosine similarity matrix between frames in each set of features; The cosine similarity matrix is ​​weighted to the features in the corresponding groups respectively, and the features of each group are concatenated to obtain fused features.

5. The method for speech emotion recognition based on adjacent frame similarity fusion according to claim 1, characterized in that: Make the fusion features interact in the frequency domain, including: After transforming the fusion features into the frequency domain, the global relationship of the features is captured through the learnable weight matrix. Then transform back to the spatial domain.

6. The method for speech emotion recognition based on adjacent frame similarity fusion according to claim 1, characterized in that: The obtained features are connected through residual connection to obtain the final features, including: The obtained features are normalized and passed through the MLP module in turn, and then residually connected with the fused features.

7. The method for speech emotion recognition based on adjacent frame similarity fusion according to claim 1, characterized in that: The speech emotion is recognized according to the final feature, including: The final features are input into a four-category classifier for emotion recognition, and the emotion categories include happiness, anger, sadness, and neutral.

8. The method for speech emotion recognition based on adjacent frame similarity fusion according to claim 1, characterized in that: The recognition process is trained according to the following loss function: In the formula, α is The weight of is the cross entropy loss in Euclidean space, and β is The weight of is the distance loss in hyperbolic space; where and The expression is: Among them, y i,euc , are the i-th element of the label vector and the output vector in the Euclidean space, respectively, and N is the number of categories; y hy , They are respectively the mapping of label vector and output vector on hyperbolic space.

9. A speech emotion recognition device based on adjacent frame similarity fusion, characterized in that: A method for speech emotion recognition based on adjacent frame similarity fusion according to any one of claims 1 to 8 is applied, comprising: A feature extraction unit, used to acquire speech data and perform feature extraction to obtain preliminary features; A feature fusion unit is used to perform feature segmentation on the preliminary features and similarity fusion of adjacent frames in sequence to obtain fused features; The feature generation unit is used to make the fusion features interact in the frequency domain space, and the obtained features are connected by residual to obtain the final features; The emotion recognition unit recognizes the speech emotion according to the final feature.

Images