Speech emotion recognition method based on self-supervised learning

Abstract

The present invention discloses a method for speech emotion recognition based on self-supervised learning, comprising obtaining existing speech data set samples; sampling and preprocessing the sample data to obtain a labeled speech data set; performing data enhancement on the labeled speech data set to obtain a labeled mixed speech data set; processing the labeled mixed speech data set using a convolutional neural network to obtain high-level semantic feature information of the speech data; processing the high-level semantic feature information using an attention mechanism to obtain global feature information of the speech data; processing the global feature information using an insertion query and an attention mechanism to obtain local feature information of the speech data; and processing the local feature information to obtain a final speech emotion recognition result. The present invention can extract global feature information of speech data, solving the problem of poor model performance and low generalization caused by the scarcity of labeled data in the current field of speech emotion recognition. The method has high reliability, good accuracy, good performance, and a wide range of applications.

Description

Technical Field

[0001] The present invention belongs to the field of digital signal processing, and in particular relates to a speech emotion recognition method based on self-supervised learning. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, digital voice technology has been widely used in people's production and life, bringing endless convenience to people's production and life.

[0003] Speech emotion recognition technology is a crucial component of digital speech technology. Current speech emotion recognition methods are primarily based on deep learning. This method uses error backpropagation to learn the target function, embedding speech data into a high-dimensional feature space, extracting and analyzing features, and ultimately mining the implicit emotional information in the speech data.

[0004] However, existing speech emotion recognition methods find it difficult to capture the global feature information of speech data, and it is also difficult to learn the potential relationship between features. At the same time, existing methods are limited by the scarcity of labeled data, which leads to problems of overfitting and low generalization in existing methods and models, making it difficult to meet the requirements of today's speech emotion recognition. Summary of the Invention

[0005] The purpose of the present invention is to provide a speech emotion recognition method based on self-supervised learning with high reliability, good accuracy and good performance.

[0006] The speech emotion recognition method based on self-supervised learning provided by the present invention comprises the following steps:

[0007] S1. Obtain existing speech dataset samples;

[0008] S2. Sampling and preprocessing the sample data obtained in step S1 to obtain a labeled speech dataset;

[0009] S3. Perform data enhancement on the labeled speech dataset obtained in step S2 to obtain a labeled mixed speech dataset;

[0010] S4 uses a convolutional neural network to process the labeled mixed speech data set obtained in step S3 to obtain high-level semantic feature information of the speech data;

[0011] S5. Process the high-level semantic feature information obtained in step S4 using an attention mechanism to obtain global feature information of the speech data;

[0012] S6. Process the global feature information obtained in step S5 using an insertion query and attention mechanism to obtain local feature information of the speech data;

[0013] S7. Process the local feature information obtained in step S6 to obtain the final speech emotion recognition result.

[0014] Step S2, in which the sample data obtained in step S1 is sampled and preprocessed to obtain a labeled speech dataset, specifically includes the following steps:

[0015] Each data sample in the sample data obtained in step S1 is uniformly sampled to convert the data sample into time series data with the same number of frames and the same dimension.

[0016] The emotional information corresponding to each data sample in the sample data obtained in step S1 is converted into a one-hot label to establish a labeled speech dataset.

[0017] Step S3 performs data enhancement on the labeled speech dataset obtained in step S2 to obtain a labeled mixed speech dataset, which specifically includes the following steps:

[0018] In the labeled speech dataset obtained in step S2, two speech data of different categories are arbitrarily selected, and several mixed speech samples are constructed through the VMixup technology, so as to expand the labeled speech dataset and obtain a labeled mixed speech dataset.

[0019] The construction of several mixed voice samples by VMixup technology specifically includes the following steps:

[0020] A. From the labeled speech dataset, randomly select two speech data samples x1 and x2 of different categories, the lengths of the speech data samples are |x1| and |x2|, |x1|=|x2| and are equal to the number of sampling frames; at the same time, obtain the emotion category label y1 corresponding to the speech data sample x1 and the emotion category label y2 corresponding to the speech data sample x2; set the hyperparameter α∈{α1,α2,...,α n} and randomly select hyperparameters β1∈[0,β 1max ] and β2∈[0,β 2max ], where α1~α n are all less than 1, β 1max <1, β 2max <1;

[0021] B. Use the following formula to perform the fusion operation on the speech data samples x1 and x2:

[0022]

[0023]

[0024] Wherein, αx1 represents the speech feature segment from the β1*|x1|th frame to the (α+β1)*|x1|-1th frame in the speech sample x1, and the formula is expressed as αx1=x1[β1*|x1|:(α+β1)*|x1|-1]; (1-α)x2 represents the speech segment after deleting the speech feature segment from the β2*|x2|th frame to the (α+β2)*|x2|-1th frame in the speech sample x2, and the formula is expressed as (1-α)x2=x2[1:β2*|x2|-1]+x2[(α+β2)*|x2|:|x | |]; is the fused mixed speech sample, which means that the speech feature segment αx1 is inserted between the speech segments x2[1:β2*|x2|-1] and x2[(α+β2)*|x2|:|x2|]; αy1 represents α times the emotion category label y1 corresponding to the speech sample x1; (1-α)y2 represents 1-α times the emotion category label y2 corresponding to the speech sample x2; Mixed speech samples The corresponding emotion category label;

[0025] C. The obtained mixed speech sample Corresponding emotion category label Together with the additional information of the mixed speech sample (β2, α+β2), it is added to the labeled speech dataset;

[0026] D. Repeat steps B and C for n times, using a different value of α for each repetition.

[0027] E. Add an additional information (0,0) to each original speech sample in the labeled speech dataset and use the additional information as the insert position of the speech sample;

[0028] F. Finally, a labeled mixed speech dataset is obtained.

[0029] Step S4 uses a convolutional neural network to process the labeled mixed speech data set obtained in step S3 to obtain high-level semantic feature information of the speech data, which specifically includes the following steps:

[0030] Construct a convolutional neural network model and input samples from the labeled mixed speech dataset into the constructed convolutional neural network model to extract high-level semantic feature information of the speech data;

[0031] The convolutional neural network model is constructed using the following formula:

[0032] h l =r l +F(r l ,W l )

[0033] r l+1 =ReLU(h l )

[0034] Where r l r is the feature input of the lth residual unit of the ResNet model for the speech data; l+1 is the feature output of the lth residual unit of the ResNet model after the speech data passes through; W l is the lth residual module function F(r l ,W l ) training parameters; F(r l ,W l ) is the learned residual information; h l is the sum of the residual module function input and the residual module function output; ReLU() is the ReLU activation function, and the expression is ReLU(x)=max(0,x).

[0035] Step S5 uses the attention mechanism to process the high-level semantic feature information obtained in step S4 to obtain global feature information of the speech data, which specifically includes the following steps:

[0036] Through the Encoder module in the Transformer architecture, the high-level semantic features obtained in step S4 are used to extract the global feature information of the speech data through the attention mechanism;

[0037] a. Convert the high-level semantic feature information obtained in step S4 into a two-dimensional vector, then add it to the positional encoding and use it as the initial input e0 of the first Encoder module in the Transformer architecture;

[0038] b. Input e of the lth Encoder module l Input to the multi-head attention layer and Add&Norm layer to extract features:

[0039] e l '=LayerNorm(e l +MultiHeadAttention(e l ))

[0040] Where e l 'for e l After the output of the multi-head attention layer and Add&Norm layer in the lth Encoder module; LayerNorm() is the Layer Normalization layer; the Add&Norm layer includes the Add layer and the Layer Normalization layer, where the Add layer refers to el +MultiHeadAttention(e l ), which is a residual connection; the Layer Normalization layer normalizes the hidden layer in the neural network to a standard normal distribution; MultiHeadAttention (e l ) is a multi-head attention layer, including several self-attention Self-Attention (e l ), and Self-Attention(e l )for:

[0041] Q e,l =e l *W e,l,Q

[0042] K e,l =e l *W e,l,K

[0043] V e,l =e l *W e,l,V

[0044]

[0045] Where W e,l,Q 、W e,l,K and W e,l,V are the three trainable feature matrices in the lth Encoder module, used to train the feature e l Perform linear mapping operation; d is the matrix Q e,l and K e,l The number of columns; Q e,l For feature e l After the feature matrix W e,l,Q The result of linear mapping; K e,l For feature e l After the feature matrix W e,l,K The result of linear mapping; V e,l For feature e l After the feature matrix W e,l,V The result of linear mapping; Attention(Q e,l ,K e,l ,V e,l ) is the attention coefficient between each frame feature of the speech sample and other frame features; Softmax() is the Softmax function, which normalizes the attention coefficient; MultiHeadAttention(e l ) is used to allocate all self-attention l) are spliced ​​together, where each self-attention Self-Attention(e l ) corresponds to the linear mapping matrix W e,l,Q 、W e,l,K and W e,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives;

[0046] c. The result obtained in step b is e l 'Input to the Feed Forward layer and Add&Norm layer to further extract features:

[0047] e l+1 =LayerNorm(e l ′+FeedForward(e l ′))

[0048] Where e l+1 for e l 'The output of the Feed Forward layer and Add&Norm layer in the lth Encoder module; FeedForward() is a two-layer fully connected layer, where the activation function of the first layer is the ReLU function, and the second layer does not use an activation function;

[0049] d. Repeat steps b and c for a set number of times to obtain global feature information of the speech data.

[0050] Step S6 uses the insertion query and attention mechanism to process the global feature information obtained in step S5 to obtain local feature information of the speech data, which specifically includes the following steps:

[0051] (1) Set the features output by the last Encoder module in the Transformer architecture and the insert query to be learned to d0, and use them together as the initial input of the first Decoder module in the Transformer architecture;

[0052] (2) d l Input to the multi-head attention layer and Add&Norm layer to extract features:

[0053] d l ′=LayerNorm(d l +MultiHeadAttention(d l ))

[0054] Where d lis the input of the lth Decoder module; MultiHeadAttention() is the multi-head attention layer, LayerNorm() is the Layer Normalization layer; the Add&Norm layer includes the Add and Layer Normalization layers; among them, the operation corresponding to Add is d l +MultiHeadAttention(d l ), which is a residual connection; the LayerNorm() layer normalizes the hidden layer in the neural network to a standard normal distribution; the MultiHeadAttention (d l ) includes several self-attention Self-Attention (d l ), Self-Attention(d l ) is calculated as:

[0055] Q d,l =d l *W d,l,Q

[0056] K d,l =d l *W d,l,K

[0057] V d,l =d l *W d,l,V

[0058]

[0059] Where W d,l,Q 、W d,l,K and W d,l,V are the three feature matrices to be trained in the lth Decoder module, used to train the feature d l Perform linear mapping operation; Q d,l , K d,l and V d,l is the feature d l The results of mapping from different angles, Q d,l is the query of the first attention mechanism, K d,l is the key value of the first attention mechanism, V d,l is the value of the first attention mechanism; d is the matrix Q e,l and K e,l The number of columns; MultiHeadAttention layer MultiHeadAttention(d l ) all self-attention Self-Attention(d l) are spliced ​​together, where each self-attention Self-Attention (d l ) corresponds to the characteristic matrix W d,l,Q 、W d,l,K and W d,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives;

[0060] (3) d l ’ is input to the multi-head attention layer and Add&Norm layer to extract features:

[0061] d″ l =LayerNorm(d′ l +MultiHeadAttention(e6,d′ l ))

[0062] Where MultiHeadAttention() is the multi-head attention layer; LayerNorm() is the LayerNormalization layer; the Add&Norm layer includes the Add and Layer Normalization layers; where the operation corresponding to Add is d′ l +MultiHeadAttention(e6,d′ l ), which is a residual connection; the LayerNorm() layer normalizes the hidden layer in the neural network to a standard normal distribution; the multi-head attention layer MultiHeadAttention(e6,d′ l ) includes several self-attention Self-Attention(e6,d′ l ), Self-Attention(e6,d′ l ) is calculated as:

[0063] Q′ d,l =d′ l *W′ d,l,Q

[0064] K′ d,l =e6*W′ d,l,K

[0065] V′ d，l =e6*W′ d，l，V

[0066]

[0067] Where W′ d,l,Q , W′ d,l,K and W′ d,l,V are the three feature matrices to be trained in the lth Decoder module, W′d,l,Q For the feature d′ l Perform linear mapping operation, W′ d,l,K and W′ d,l,V Used to perform linear mapping operation on feature e6; Q′ d,l is the query of the second attention mechanism, K′ d,l is the key value of the second attention mechanism, V′ d,l is the value of the second attention mechanism; d is the matrix Q′ d,l and K′ d,l The number of columns; MultiHeadAttention layer MultiHeadAttention(e6,d l ′) all self-attention Self-Attention(e6,d l ′) are concatenated together, where each self-attention Self-Attention(e6,d l ′) corresponds to the characteristic matrix W d ' ,l,Q 、W d ' ,l,K and W d ' ,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives;

[0068] (4) d l Input to the Feed Forward layer and Add&Norm layer to further extract features:

[0069] d l+1 =LayerNorm(d l ″+FeedForward(d l ″))

[0070] FeedForward() is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function.

[0071] (5) Repeat steps (2) to (4) a set number of times to obtain local feature information of the speech data.

[0072] Step S7 of processing the local feature information obtained in step S6 to obtain the final speech emotion recognition result specifically includes the following steps:

[0073] Two output heads are established, where the first output head includes a fully connected layer and a Softmax layer, and the second output head includes two fully connected layers; the first output head is used to predict the emotion category of the speech data, and the second output head is used to predict the insert position of the additional information of the speech data;

[0074] The feature dimension output by the first output head is the emotion category; the feature output by the second output head is the dimension of the additional vector insert position, which represents the mixed sample Insert x1 into the position of x2;

[0075] y′=Softmax(max(0,d6W 11 +b 11 ))

[0076] ip′=max(0,max(0,d6W 21 +b 21 )W 22 +b 22 )

[0077] Where y′ is the emotion category of the speech sample x predicted by the model; ip′ is the insert position vector of the speech sample x predicted by the model; Softmax() is the Softmax function; d6 is the local feature information output by the Decoder module in the Transformer architecture; W 11 、W 21 and W 22 is the feature matrix to be trained, which linearly maps the local feature information; b 11 、b 21 and b 22 is a trainable bias term;

[0078] The sentiment category error loss L between the true label y and the predicted label y' is constructed through the cross entropy loss function label (y,y′), construct the insert position vector error loss L between the real insert position vector ip and the predicted insert position vector ip′ through the mean square error loss function insert (ip,ip′), the final total loss function Loss is:

[0079] Loss = L label (y,y′)+λL insert (ip,ip′)

[0080] Where L label (y,y′) is the sentiment category error loss between the true label y and the predicted label y′ constructed using the cross entropy loss function; L insert(ip,ip′) is the insert position vector error loss between the true insert position vector ip and the predicted insert position vector ip′ constructed using the mean square error loss function; λ is a hyperparameter set between 0 and 1.

[0081] The speech emotion recognition method based on self-supervised learning provided by the present invention can extract global feature information of speech data, and at the same time proposes a new data enhancement method to increase the amount of labeled data, and focuses on local features of speech data through the insertquery method, thereby solving the problems of poor model performance and low generalization caused by the scarcity of labeled data in the current field of speech emotion recognition. The global feature information of speech data is obtained through the attention mechanism, and the local features of speech data are analyzed through the insertquery method. The method of the present invention proposes a new speech emotion recognition method with high reliability, good accuracy, good performance, and a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 Schematic diagram of the process of the present invention.

[0083] Figure 2 Schematic diagram of the data enhancement technology of the present invention.

[0084] Figure 3 Schematic diagram of the residual module of the ResNet network model of the present invention. DETAILED DESCRIPTION

[0085] like Figure 1 The method flow chart of the present invention is shown as follows: The speech emotion recognition method based on self-supervised learning provided by the present invention comprises the following steps:

[0086] S1. Obtain existing speech dataset samples;

[0087] S2. Sampling and preprocessing the sample data obtained in step S1 to obtain a labeled speech dataset; specifically, the steps include:

[0088] Each data sample in the sample data obtained in step S1 is uniformly sampled to convert the data sample into time series data with the same number of frames and the same dimension.

[0089] Convert the emotion information corresponding to each data sample in the sample data obtained in step S1 into a one-hot label to establish a labeled speech dataset;

[0090] In specific implementation, each data sample is framed, windowed, Fourier transformed, and filtered to obtain the filter bank output as the feature representation of each original speech data. The filter bank output of each data sample is then uniformly sampled to convert it into time series data with the same number of frames and dimensions. The emotional information already in the original speech dataset is converted into one-hot labels to create a labeled speech dataset.

[0091] Among them, uniform sampling is to sample the filter bank output at the same interval frame number, where the interval frame number is the total number of filter bank output frames divided by the number of same frames; same frame number and same dimension means that all samples have the same number of frames and the same dimension of each frame feature;

[0092] S3. Perform data enhancement on the labeled speech dataset obtained in step S2 to obtain a labeled mixed speech dataset; specifically comprising the following steps:

[0093] In the labeled speech dataset obtained in step S2, two speech data of different categories are randomly selected, and several mixed speech samples are constructed using the VMixup technology, thereby expanding the labeled speech dataset to obtain a labeled mixed speech dataset;

[0094] In the specific implementation, several mixed voice samples are constructed through VMixup technology, including the following steps:

[0095] A. From the labeled speech dataset, randomly select two speech data samples x1 and x2 of different categories, the lengths of the speech data samples are |x1| and |x2|, |x1|=|x2| and are equal to the number of sampling frames; at the same time, obtain the emotion category label y1 corresponding to the speech data sample x1 and the emotion category label y2 corresponding to the speech data sample x2; set the hyperparameter α∈{α1,α2,...,α n} and randomly select hyperparameters β1∈[0,β 1max ] and β2∈[0,β 2max ], where α1~α n are all less than 1, β 1max <1, β 2max <1;

[0096] In the specific implementation, the hyperparameter α is set to α∈{0.1,0.2,0.3,0.4,0.5}, β1∈[0,0.5], β2∈[0,0.5];

[0097] B. Use the following formula to perform the fusion operation on the speech data samples x1 and x2:

[0098]

[0099]

[0100] Wherein, αx1 represents the speech feature segment from the β1*|x1|th frame to the (α+β1)*|x1|-1th frame in the speech sample x1, and the formula is expressed as αx1=x1[β1*|x1|:(α+β1)*|x1|-1]; (1-α)x2 represents the speech segment after deleting the speech feature segment from the β2*|x2|th frame to the (α+β2)*|x2|-1th frame in the speech sample x2, and the formula is expressed as (1-α)x2=x2[1:β2*|x2|-1]+x2[(α+β2)*|x2|:|x2|]; is the fused mixed speech sample, which means that the speech feature segment αx1 is inserted between the speech segments x2[1:β2*|x2|-1] and x2[(α+β2)*|x2|:|x2|]; αy1 represents α times the emotion category label y1 corresponding to the speech sample x1; (1-α)y2 represents 1-α times the emotion category label y2 corresponding to the speech sample x2; Mixed speech samples The corresponding emotion category label;

[0101] C. The obtained mixed speech sample Corresponding emotion category label Together with the additional information of the mixed speech sample (β2, α+β2), it is added to the labeled speech dataset;

[0102] D. Repeat steps B and C for n times, using a different value of α for each repetition.

[0103] E. Add an additional information (0,0) to each original speech sample in the labeled speech dataset and use the additional information as the insert position of the speech sample;

[0104] F. Finally, a labeled mixed speech dataset is obtained;

[0105] Figure 2Schematic diagram of the VMixup data enhancement technology based on speech; the parameters of this example are α=0.4,β1=0.45,β2=0.25,αx1 means that 40% of the speech segments of voice sample voice 1 are cut off from the 45% part, that is, the speech segments x1[9:16] from the 9th to the 16th frames, and (1-α)x2 means that 40% of the speech segments of voice sample voice 2 are deleted from the 25% part, and the remaining two speech feature segments in voice sample voice 2 are retained, that is, the speech segments x2[1:4] from the 1st to the 4th frames and the speech segments x2[13:20] from the 13th to the 20th frames. Indicates that the speech feature segment x1[9:16] is inserted between x2[1:4] and x2[13:20] to form a mixed speech sample. Here we assume that the label of voice 1 is y1 = [0, 0, 1, 0, 0, 0], and the label of voice 2 is y2 = [0, 0, 0, 0, 1, 0, 0], then the mixed voice sample The label is Represents a mixed speech sample 40% of the emotional components belong to voice sample 1, and 60% of the emotional components belong to voice sample 2;

[0106] S4. Processing the labeled mixed speech dataset obtained in step S3 using a convolutional neural network to obtain high-level semantic feature information of the speech data; specifically comprising the following steps:

[0107] Construct a convolutional neural network model and input samples from the labeled mixed speech dataset into the constructed convolutional neural network model to extract high-level semantic feature information of the speech data;

[0108] The convolutional neural network model is constructed using the following formula:

[0109] h l =r l +F(r l ,W l )

[0110] r l+1 =ReLU(h l )

[0111] Where r l r is the feature input of the lth residual unit of the ResNet model for the speech data; l+1 is the feature output of the lth residual unit of the ResNet model after the speech data passes through; W l is the lth residual module function F(r l ,Wl ) training parameters; F(r l ,W l ) is the learned residual information; h l is the sum of the residual module function input and the residual module function output; ReLU() is the ReLU activation function, and the expression is ReLU(x)=max(0,x); Figure 3 Schematic diagram of the residual module of the ResNet network model;

[0112] S5. Processing the high-level semantic feature information obtained in step S4 using an attention mechanism to obtain global feature information of the speech data; specifically comprising the following steps:

[0113] Through the Encoder module in the Transformer architecture, the high-level semantic features obtained in step S4 are used to extract the global feature information of the speech data through the attention mechanism;

[0114] a. Convert the high-level semantic feature information obtained in step S4 into a two-dimensional vector (using the numpy.ndarray.flatten() function), then add it to the positional encoding and use it as the initial input e0 of the first Encoder module in the Transformer architecture.

[0115] b. Input e of the lth Encoder module l Input to the multi-head attention layer and Add&Norm layer to extract features:

[0116] e l '=LayerNorm(e l +MultiHeadAttention(e l ))

[0117] Where e l 'for e l After the output of the multi-head attention layer and Add&Norm layer in the lth Encoder module; LayerNorm() is the Layer Normalization layer; the Add&Norm layer includes the Add layer and the Layer Normalization layer, where the Add layer refers to e l +MultiHeadAttention(e l ), which is a residual connection; the Layer Normalization layer normalizes the hidden layer in the neural network to a standard normal distribution; MultiHeadAttention (e l) is a multi-head attention layer, including several self-attention Self-Attention (e l ), and Self-Attention(e l )for:

[0118] Q e,l =e l *W e,l,Q

[0119] K e,l =e l *W e,l,K

[0120] V e,l =e l *W e,l,V

[0121]

[0122] Where W e,l,Q 、W e,l,K and W e,l,V are the three trainable feature matrices in the lth Encoder module, used to train the feature e l Perform linear mapping operation; d is the matrix Q e,l and K e,l The number of columns; Q e,l For feature e l After the feature matrix W e,l,Q The result of linear mapping; K e,l For feature e l After the feature matrix W e,l,K The result of linear mapping; V e,l For feature e l After the feature matrix W e,l,V The result of linear mapping; Attention(Q e,l ,K e,l ,V e,l ) is the attention coefficient between each frame feature of the speech sample and other frame features; Softmax() is the Softmax function, which normalizes the attention coefficient; MultiHeadAttention(e l ) is used to allocate all self-attention l ) are spliced ​​together, where each self-attention Self-Attention(e l ) corresponds to the linear mapping matrix W e,l,Q 、W e,l,K and W e,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives;

[0123] c. The result obtained in step b is e l 'Input to the Feed Forward layer and Add&Norm layer to further extract features:

[0124] e l+1 =LayerNorm(e l ′+FeedForward(e l ′))

[0125] Where e l+1 for e l 'The output of the Feed Forward layer and Add&Norm layer in the lth Encoder module; FeedForward() is a two-layer fully connected layer, where the activation function of the first layer is the ReLU function, and the second layer does not use an activation function;

[0126] d. Repeat steps b and c a total of the set number of times to obtain global feature information of the voice data;

[0127] S6. Process the global feature information obtained in step S5 using an insertion query and attention mechanism to obtain local feature information of the speech data; specifically comprising the following steps:

[0128] (1) Set the features output by the last Encoder module in the Transformer architecture and the insert query to be learned to d0, and use them together as the initial input of the first Decoder module in the Transformer architecture;

[0129] (2) d l Input to the multi-head attention layer and Add&Norm layer to extract features:

[0130] d l ′=LayerNorm(d l +MultiHeadAttention(d l ))

[0131] Where d l is the input of the lth Decoder module; MultiHeadAttention() is the multi-head attention layer, LayerNorm() is the Layer Normalization layer; the Add&Norm layer includes the Add and Layer Normalization layers; among them, the operation corresponding to Add is d l +MultiHeadAttention(d l), which is a residual connection; the LayerNorm() layer normalizes the hidden layer in the neural network to a standard normal distribution; the MultiHeadAttention (d l ) includes several self-attention Self-Attention (d l ), Self-Attention(d l ) is calculated as:

[0132] Q d,l =d l *W d,l,Q

[0133] K d,l =d l *W d,l,K

[0134] V d,l =d l *W d,l,V

[0135]

[0136] Where W d,l,Q 、W d,l,K and W d,l,V are the three feature matrices to be trained in the lth Decoder module, used to train the feature d l Perform linear mapping operation; Q d,l , K d,l and V d,l is the feature d l The results of mapping from different angles, Q d,l is the query of the first attention mechanism, K d,l is the key value of the first attention mechanism, V d,l is the value of the first attention mechanism; d is the matrix Q e,l and K e,l The number of columns; MultiHeadAttention layer MultiHeadAttention(d l ) all self-attention Self-Attention(d l ) are spliced ​​together, where each self-attention Self-Attention (d l ) corresponds to the characteristic matrix W d,l,Q 、W d,l,K and W d,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives;

[0137] (3) d l ’ is input to the multi-head attention layer and Add&Norm layer to extract features:

[0138] d l =LayerNorm(d l ′+MultiHeadAttention(e6,d l ′))

[0139] Where MultiHeadAttention() is the multi-head attention layer; LayerNorm() is the LayerNormalization layer; the Add&Norm layer includes the Add and Layer Normalization layers; where the operation corresponding to Add is d′ l +MultiHeadAttention(e6,d′ l ), which is a residual connection; the LayerNorm() layer normalizes the hidden layer in the neural network to a standard normal distribution; the multi-head attention layer MultiHeadAttention(e6,d′ l ) includes several self-attention Self-Attention(e6,d′ l ), Self-Attention(e6,d′ l ) is calculated as:

[0140] Q′ d,l =d′ l *W′ d,l,Q

[0141] K′ d,l =e6*W′ d,l,K

[0142] V′ d,l =e6*W′ d,l,V

[0143]

[0144] Where W′ d,l,Q , W′ d,l,K and W′ d,l,V are the three feature matrices to be trained in the lth Decoder module, W′ d,l,Q For the feature d′ l Perform linear mapping operation, W′ d,l,K and W′ d,l,V Used to perform linear mapping operation on feature e6; Q′ d,l is the query of the second attention mechanism, K′ d,l is the key value of the second attention mechanism, V′ d,l is the value of the second attention mechanism; d is the matrix Q′ d,l and K′ d,lThe number of columns; MultiHeadAttention(e6,d′ l ) all self-attention Self-Attention(e6,d′ l ) are concatenated together, where each self-attention Self-Attention(e6,d′ l ) corresponds to the characteristic matrix W′ d,l,Q , W′ d,l,K and W′ d,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives;

[0145] (4) d″ l Input to the Feed Forward layer and Add&Norm layer to further extract features:

[0146] d l+1 =LayerNorm(d″) l +FeedForward(d″ l ))

[0147] FeedForward() is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function.

[0148] (5) Repeating steps (2) to (4) a set number of times, thereby obtaining local feature information of the speech data;

[0149] S7. Processing the local feature information obtained in step S6 to obtain the final speech emotion recognition result; specifically comprising the following steps:

[0150] Two output heads are established, where the first output head includes a fully connected layer and a Softmax layer, and the second output head includes two fully connected layers; the first output head is used to predict the emotion category of the speech data, and the second output head is used to predict the insert position of the additional information of the speech data;

[0151] The feature dimension output by the first output head is the emotion category; the feature output by the second output head is the dimension of the additional vector insert position, which represents the mixed sample Insert x1 into the position of x2;

[0152] y′=Softmax(max(0,d6W 11 +b 11 ))

[0153] ip′=max(0,max(0,d6W 21 +b 21 )W22 +b 22 )

[0154] Where y′ is the emotion category of the speech sample x predicted by the model; ip′ is the insert position vector of the speech sample x predicted by the model; Softmax() is the Softmax function; d6 is the local feature information output by the Decoder module in the Transformer architecture; W 11 、W 21 and W 22 is the feature matrix to be trained, which linearly maps the local feature information; b 11 、b 21 and b 22 is a trainable bias term;

[0155] The sentiment category error loss L between the true label y and the predicted label y' is constructed through the cross entropy loss function label (y,y′), construct the insert position vector error loss L between the real insert position vector ip and the predicted insert position vector ip′ through the mean square error loss function insert (ip,ip′), the final total loss function Loss is:

[0156] Loss = L label (y,y′)+λL insert (ip,ip′)

[0157] Where L label (y,y′) is the sentiment category error loss between the true label y and the predicted label y′ constructed using the cross entropy loss function; L insert (ip,ip′) is the insert position vector error loss between the true insert position vector ip and the predicted insert position vector ip′ constructed using the mean square error loss function; λ is a hyperparameter set between 0 and 1.

[0158] Based on deep learning, the present invention utilizes the attention mechanism and VMixup technology to mine the emotional information of speech data. The method establishes a labeled mixed speech dataset by preprocessing and VMixup operations on the original speech dataset samples; secondly, the high-level semantic feature information of the speech data is extracted through a convolutional neural network model; then, based on the high-level semantic features, the global feature information and local feature information of the speech data are respectively extracted through the Encoder module and the Decoder module in the Transformer architecture; finally, the emotional category of the speech data is determined by analyzing the feature information. Compared with existing speech emotion methods, the present invention can improve the classification ability of speech emotion recognition and eliminate the overfitting of the model. In addition, the present invention uses the attention mechanism to extract the global feature information of speech data and the insert query method to focus on the local features of speech data; and proposes a new data enhancement method VMixup to increase the amount of labeled data, which solves the problem of poor model performance and low generalization due to the scarcity of labeled data in the current field of speech emotion recognition.

Claims

1. A speech emotion recognition method based on self-supervised learning, comprising the following steps: S1. Obtain existing speech dataset samples; S2. Sampling and preprocessing the sample data obtained in step S1 to obtain a labeled speech dataset; S3. Perform data enhancement on the labeled speech dataset obtained in step S2 to obtain a labeled mixed speech dataset; specifically comprising the following steps: In the labeled speech dataset obtained in step S2, two speech data of different categories are randomly selected, and several mixed speech samples are constructed using the VMixup technology, thereby expanding the labeled speech dataset to obtain a labeled mixed speech dataset; in, The construction of several mixed voice samples by VMixup technology specifically includes the following steps: A. From the labeled speech dataset, arbitrarily select two speech data samples x1 and x2 of different categories, where the lengths of the speech data samples are |x1| and |x2|, respectively, and |x1| = |x2| and are equal to the number of sampling frames; at the same time, obtain the emotion category label y1 corresponding to the speech data sample x1 and the emotion category label y2 corresponding to the speech data sample x2; Set the hyperparameters α∈{α1,α2,...,α n } and randomly select hyperparameters β1∈[0,β 1max ] and β2∈[0,β 2max ], where α1~α n are all less than 1, β 1max <1, β 2max <1; B. Use the following formula to perform the fusion operation on the speech data samples x1 and x2: Wherein, αx1 represents the speech feature segment from the β1*|x1|th frame to the (α+β1)*|x1|-1th frame in the speech sample x1, and the formula is expressed as αx1=x1[β1*|x1|:(α+β1)*|x1|-1]; (1-α)x2 represents the speech segment after deleting the speech feature segment from the β2*|x2|th frame to the (α+β2)*|x2|-1th frame in the speech sample x2, and the formula is expressed as (1-α)x2=x2[1:β2*|x2|-1]+x2[(α+β2)*|x2|:|x2|]; is the fused mixed speech sample, which means that the speech feature segment αx1 is inserted between the speech segments x2[1:β2*|x2|-1] and x2[(α+β2)*|x2|:|x2|]; αy1 represents α times the emotion category label y1 corresponding to the speech sample x1; (1-α)y2 represents 1-α times the emotion category label y2 corresponding to the speech sample x2; Mixed speech samples The corresponding emotion category label; C. The obtained mixed speech sample Corresponding emotion category label Together with the additional information of the mixed speech sample (β2, α+β2), it is added to the labeled speech dataset; D. Repeat steps B and C for n times, using a different value of α for each repetition. E. Add an additional information (0,0) to each original speech sample in the labeled speech dataset and use the additional information as the insert position of the speech sample; F. Finally, a labeled mixed speech dataset is obtained; S4 uses a convolutional neural network to process the labeled mixed speech data set obtained in step S3 to obtain high-level semantic feature information of the speech data; S5. Process the high-level semantic feature information obtained in step S4 using an attention mechanism to obtain global feature information of the speech data; S6. Process the global feature information obtained in step S5 using an insertion query and attention mechanism to obtain local feature information of the speech data; S7. Process the local feature information obtained in step S6 to obtain the final speech emotion recognition result.

2. The speech emotion recognition method based on self-supervised learning according to claim 1 is characterized in that Step S2, in which the sample data obtained in step S1 is sampled and preprocessed to obtain a labeled speech dataset, specifically includes the following steps: Each data sample in the sample data obtained in step S1 is uniformly sampled to convert the data sample into time series data with the same number of frames and the same dimension. The emotional information corresponding to each data sample in the sample data obtained in step S1 is converted into a one-hot label to establish a labeled speech dataset.

3. The speech emotion recognition method based on self-supervised learning according to claim 2 is characterized in that Step S4 uses a convolutional neural network to process the labeled mixed speech data set obtained in step S3 to obtain high-level semantic feature information of the speech data, which specifically includes the following steps: Construct a convolutional neural network model and input samples from the labeled mixed speech dataset into the constructed convolutional neural network model to extract high-level semantic feature information of the speech data; The convolutional neural network model is constructed using the following formula: h l =r l +F(r l ,W l ) r l+1 =ReLU(h l ) Where r l The feature input of the speech data passing through the lth residual unit of the ResNet model; r l+1 is the feature output of the lth residual unit of the ResNet model after the speech data passes through; W l is the lth residual module function F(r l ,W l ) training parameters; F(r l ,W l ) is the learned residual information; h l is the sum of the residual module function input and the residual module function output; ReLU() is the ReLU activation function, and the expression is ReLU(x)=max(0,x).

4. The speech emotion recognition method based on self-supervised learning according to claim 3 is characterized in that Step S5 uses the attention mechanism to process the high-level semantic feature information obtained in step S4 to obtain global feature information of the speech data, which specifically includes the following steps: Through the Encoder module in the Transformer architecture, the high-level semantic features obtained in step S4 are used to extract the global feature information of the speech data through the attention mechanism; a. Convert the high-level semantic feature information obtained in step S4 into a two-dimensional vector, then add it to the positional encoding information as the initial input e0 of the first Encoder module in the Transformer architecture; b. Input e of the lth Encoder module l Input to the multi-head attention layer and Add&Norm layer to extract features: in l '=LayerNorm(e l +MultiHeadAttention(e l )) Where e l 'for e l After the output of the multi-head attention layer and Add&Norm layer in the lth Encoder module; LayerNorm() is the Layer Normalization layer; the Add&Norm layer includes the Add layer and the Layer Normalization layer, where the Add layer refers to e l +MultiHeadAttention(e l ), which is a residual connection; the Layer Normalization layer normalizes the hidden layer in the neural network to a standard normal distribution; MultiHeadAttention (e l ) is a multi-head attention layer, including several self-attention Self-Attention (e l ), and Self-Attention(e l )for: Q e,l =e l *W e,l,Q K e,l =e l *IN e,l,K V e,l s.e l *W e,l,V Where W e,l,Q 、W e,l,K and W e,l,V are the three trainable feature matrices in the lth Encoder module, used to train the feature e l Perform linear mapping operation; d is the matrix Q e,l and K e,l The number of columns; Q e,l For feature e l After the feature matrix W e,l,Q The result of linear mapping; K e,l For feature e l After the feature matrix W e,l,K The result of linear mapping; V e,l For feature e l After the feature matrix W e,l,V The result of linear mapping; Attention(Q e,l ,K e,l ,V e,l ) is the attention coefficient between each frame feature of the speech sample and other frame features; Softmax() is the Softmax function, which normalizes the attention coefficient; MultiHeadAttention(e l ) is used to allocate all self-attention l ) are spliced ​​together, where each self-attention Self-Attention(e l ) corresponds to the linear mapping matrix W e,l,Q 、W e,l,K and W e,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives; c. The result obtained in step b is e. l 'Input to the Feed Forward layer and Add&Norm layer to further extract features: in l+1 =LayerNorm(e l ′+FeedForward(e l ′)) Where e l+1 for e l 'The output of the Feed Forward layer and Add&Norm layer in the lth Encoder module; FeedForward() is a two-layer fully connected layer, where the activation function of the first layer is the ReLU function, and the second layer does not use an activation function; d. Repeat steps b and c for a set number of times to obtain global feature information of the speech data.

5. The speech emotion recognition method based on self-supervised learning according to claim 4 is characterized in that Step S6 uses the insertion query and attention mechanism to process the global feature information obtained in step S5 to obtain local feature information of the speech data, which specifically includes the following steps: (1) The features output by the last Encoder module in the Transformer architecture and the insert query to be learned are set to d0, and together serve as the initial input of the first Decoder module in the Transformer architecture; (2) d l Input to the multi-head attention layer and Add&Norm layer to extract features: d l ′=LayerNorm(d l +MultiHeadAttention(d l )) Where d l is the input of the lth Decoder module; MultiHeadAttention() is the multi-head attention layer, LayerNorm() is the Layer Normalization layer; the Add&Norm layer includes the Add and Layer Normalization layers; among them, the operation corresponding to Add is d l +MultiHeadAttention(d l ), which is a residual connection; the LayerNorm() layer normalizes the hidden layer in the neural network to a standard normal distribution; the MultiHeadAttention (d l ) includes several self-attention Self-Attention (d l ), Self-Attention(d l ) is calculated as: Q d,l =d l *W d,l,Q K d,l =d l *W d,l,K V d,l =d l *W d,l,V Where W d,l,Q 、W d,l,K and W d,l,V are the three feature matrices to be trained in the lth Decoder module, used to train the feature d l Perform linear mapping operation; Q d,l is the query of the first attention mechanism, K d,l is the key value of the first attention mechanism, V d,l is the value of the first attention mechanism; d is the matrix Q e,l and K e,l The number of columns; MultiHeadAttention layer MultiHeadAttention(d l ) all self-attention Self-Attention(d l ) are spliced ​​together, where each self-attention Self-Attention (d l ) corresponds to the characteristic matrix W d,l,Q 、W d,l,K and W d,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives; (3) d′ l Input to the multi-head attention layer and Add&Norm layer to extract features: d″ l =LayerNorm(d′ l +MultiHeadAttention(e6,d′ l )) Where MultiHeadAttention() is the multi-head attention layer; LayerNorm() is the Layer Normalization layer; the Add&Norm layer includes the Add and Layer Normalization layers; where the operation corresponding to Add is d′ l +MultiHeadAttention(e6,d′ l ), which is a residual connection; the LayerNorm() layer normalizes the hidden layer in the neural network to a standard normal distribution; the multi-head attention layer MultiHeadAttention(e6,d′ l ) includes several self-attention Self-Attention(e6,d′ l ), Self-Attention(e6,d′ l ) is calculated as: Q′ d,l =d′ l *W′ d,l,Q K′ d ,l=e6*W′ d,l,K V′ d,l =e6*W′ d,l,V Where W′ d,l, Q, W′ d,l,K and W′ d,l,V are the three feature matrices to be trained in the lth Decoder module, W′ d,l,Q For the feature d′ l Perform linear mapping operation, W′ d,l,K and W′ d,l,V Used to perform linear mapping operation on feature e6; Q′ d,l is the query of the second attention mechanism, K′ d,l is the key value of the second attention mechanism, V′ d,l is the value of the second attention mechanism; d is the matrix Q′ d,l and K′ d,l The number of columns; MultiHeadAttention(e6,d′ l ) all self-attention Self-Attention(e6,d′ l ) are concatenated together, where each self-attention Self-Attention(e6,d′ l ) corresponds to the characteristic matrix W′ d,l,Q , W′ d,l,K and W′ d,l,V Different,indicates that the model judges the degree of correlation between speech features from,different perspectives; (4) Change d″ l Input to the Feed Forward layer and Add&Norm layer to further extract features: <h2 style=";text-align:left;direction:ltr">d<h2 style=";text-align:left;direction:ltr"> l+1 <h2 style=";text-align:left;direction:ltr"> =LayerNorm(d″<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> +FeedForward(d″<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> )) FeedForward() is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. (5) Repeat steps (2) to (4) a set number of times to obtain local feature information of the speech data.

6. The speech emotion recognition method based on self-supervised learning according to claim 5 is characterized in that Step S7 of processing the local feature information obtained in step S6 to obtain the final speech emotion recognition result specifically includes the following steps: Two output heads are established, where the first output head includes a fully connected layer and a Softmax layer, and the second output head includes two fully connected layers; the first output head is used to predict the emotion category of the speech data, and the second output head is used to predict the insert position of the additional information of the speech data; The feature dimension output by the first output head is the emotion category; the feature output by the second output head is the dimension of the additional vector insert position, which represents the mixed sample Insert x1 into the position of x2; y′=Softmax(max(0,d6W 11 +b 11 )) ip′=max(0,max(0,d6W 21 +b 21 )W 22 +b 22 ) Where y′ is the emotion category of the speech sample x predicted by the model; ip′ is the insertposition vector of the speech sample x predicted by the model; Softmax() is the Softmax function; d6 is the local feature information output by the Decoder module in the Transformer architecture; W 11 、W 21 and W 22 is the feature matrix to be trained, which linearly maps the local feature information; b 11 、b 21 and b 22 is a trainable bias term; The sentiment category error loss L between the true label y and the predicted label y' is constructed through the cross entropy loss function label (y,y′), construct the insert position vector error loss L between the real insert position vector ip and the predicted insert position vector ip′ through the mean square error loss function insert (ip,ip′), the final total loss function Loss is: Loss=L label (y,y′)+λL insert (ip,ip′) Where L label (y,y′) is the sentiment category error loss between the true label y and the predicted label y′ constructed using the cross entropy loss function; L insert (ip,ip′) is the insert position vector error loss between the true insert position vector ip and the predicted insert position vector ip′ constructed using the mean square error loss function; λ is a hyperparameter set between 0 and 1.

Images