An emotion intelligent recognition method based on data enhancement and cross-modal feature fusion

Abstract

The application discloses an emotion intelligent recognition method based on data enhancement and cross-modal feature fusion, comprising the following steps: multi-modal data acquisition; text data is subjected to data enhancement, and the enhanced text data is used to construct text sentiment features; 3D spectrogram features are extracted from voice data, and the voice sentiment features are obtained by enhancement; cross-modal feature information fusion is performed on the text sentiment features and the voice sentiment features by using a cross-modal Transformer model to obtain fused multi-modal sentiment features; a classification model based on BiGRU and a double pooling layer is used for feature learning on the fused multi-modal sentiment features, and a cross-entropy loss function is used for training; and the trained classification model is used for emotion recognition. The method disclosed by the application can effectively utilize the sentiment information contained in various modes, capture complementary information between different modes, and thus improve the recognition ability and generalization ability of the emotion recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal emotion intelligent recognition, and particularly relates to an emotion intelligent recognition method based on data enhancement and cross-modal feature fusion. BACKGROUND

[0002] Sentiment Analysis is a process of analyzing, processing and extracting subjective data with emotional color by using natural language processing and data mining technology. The sentiment analysis task can be divided into two sub-tasks, Aspect Extraction (aspect extraction) and Sentimental Classification (sentiment classification). The sentiment classification task, also known as polarity analysis, is to divide the text into two or several types according to the meaning and emotional information expressed by the data, that is, to divide the author's tendency, view and attitude. Generally, there are two classifications (positive and negative) or multi-classifications (happy, angry, angry) and the like. The multi-modal of sentiment analysis mainly refers to the diversity of data sources used for emotion recognition, such as text, voice, facial expression, physiological signal and the like.

[0003] Although the research on emotion recognition has made certain achievements in single-modal emotion recognition, the accuracy is still not high. On the one hand, the emotional information expressed by different modalities is slightly different, for example, facial expression and voice tone. If the emotional state of a person is directly determined from only one expression, the result of the determination may be one-sided. From the perspective of people themselves, it is difficult to judge the emotional state of a person by using single voice or text information, and it is even more difficult for a computer. On the other hand, from the perspective of the collected data set, the single-modal data information is not comprehensive enough, and the information collected is easily affected by external environment, and the data obtained may have some deviations. For example, when collecting voice information, it is easily affected by signal strength and external environmental noise. SUMMARY

[0004] To solve the above technical problems, the present application provides an emotion intelligent recognition method based on data enhancement and cross-modal feature fusion, so as to effectively utilize the emotional information contained in multiple modalities, capture the complementary information between different modalities, and improve the recognition ability and generalization ability of the emotion recognition model.

[0005] To achieve the above purpose, the technical scheme of the present application is as follows:

[0006] An emotion intelligent recognition method based on data enhancement and cross-modal feature fusion, comprising the following steps:

[0007] (1) Multi-modal data acquisition;

[0008] (2) Data augmentation is performed on the text data, and text sentiment features are constructed using the augmented text data; 3D spectrogram features are extracted from the speech data, and the 3D spectrogram features are augmented to obtain speech sentiment features;

[0009] (3) Cross-modal feature information fusion is performed on the text sentiment features and the speech sentiment features using a cross-modal Transformer model to obtain fused multi-modal sentiment features;

[0010] (4) Feature learning is performed on the fused multi-modal sentiment features using a classification model based on BiGRU and a double-pooling layer, and training is performed using a cross-entropy loss function;

[0011] (5) Emotion recognition is performed using the trained classification model to obtain an emotion recognition result.

[0012] In the above scheme, in step (2), the text data is augmented by <pad>The four ways of replacement, synonym replacement, back translation and pseudo label prediction are used to complete data enhancement.

[0013] In the further technical solution, the <pad>Replace with: Text data with a probability of 0.3, replacing words in the input sentence with words used to complete the BERT pre-training model sentence <pad>The synonym replacement is: the text data is first segmented, all segmented words are matched in a Chinese synonym table, if a hit is obtained, synonym replacement is performed, and a new sample data is obtained.

[0014] The synonym replacement is: the text data is first segmented, all segmented words are matched in a Chinese synonym table, if a hit is obtained, synonym replacement is performed, and a new sample data is obtained.

[0015] The back-translation method is: the text data is translated in Baidu translation according to the translation order of Chinese, English, Japanese and Chinese, and the obtained data is a new sample data.

[0016] The pseudo-label prediction is: the original text data is trained in a BERT model to obtain a pre-training model, unlabeled audio data is crawled from the network, the text is separated from the audio, and the pseudo-label of the sample data is obtained by the pre-training model after the text is predicted, as a new sample data.

[0017] In the above scheme, in step (2), the method for constructing text sentiment features is as follows:

[0018] First, based on the Word2Vec training word vector matrix, the text data is cleaned and segmented, and the 300-dimensional text semantic word vector obtained by multiplying the word vector matrix and the one-hot vector of the sample segmentation is obtained.

[0019] Secondly, according to the continuous emotion word dictionary, the text emotion word with emotional tendency in the text is mapped to the emotion space to obtain a mapping matrix, and then the bottleneck feature vector of the mapping matrix, i.e. the text emotion word vector, is obtained through an automatic encoder.

[0020] Then, the text semantic word vector and the text emotion word vector are fused to obtain the constructed text sentiment features.

[0021] In the further technical scheme, the continuous emotion word dictionary contains 1653 emotion words labeled with valence values and arousal values, all emotion words in the continuous emotion word dictionary are converted into word vector form, and are expressed as E=(e1, e2,..., en), wherein n=1653, en represents the n-th emotion word vector. n n

[0022] Each emotion word corresponds to a set of VA values, expressed as VA=((ev1, ea1), (ev2, ea2),..., (evn, ean)), wherein evn represents the valence value of the n-th emotion word in the dictionary, ean represents the arousal value of the n-th emotion word in the dictionary. n n n n

[0023] ​​​​​​The sentiment word mapping in the text data is obtained by acquiring the i-th word segment s of the sample. i word vector g i Then, calculate the word vector and the word vector e in E. j The cosine similarity is calculated using the following formula:

[0024]

[0025] Where j = 1, 2, 3, ..., n, using cosθ ij With e j g can be obtained by multiplying the corresponding VA values. i The mapped VA value is expressed by the following formula:

[0026] ev j→i =cos ij θ·ev j

[0027] ea j→i =cos ij θ·ea j

[0028] After the word segmentation is mapped, a mapping matrix of size n×2 can be obtained, which is represented as follows:

[0029] SVA i =[[ev 1→i ,ea 1→i ],[ev 2→i ,ea 2→i ],...,[ev n→i ,ea n→i ]]

[0030] The autoencoder consists of two fully connected Dense layers. The first fully connected layer has an output dimension of 2^12 and uses ReLU as the activation function. The second fully connected layer has an output dimension of n and uses the sigmoid function as the activation function. The optimizer is the Adam optimizer, and the loss function is the mean squared error function.

[0031] In the above scheme, in step (2), the speech sample is amplified with a filter to amplify the high frequency. Then, the speech signal is extracted as a frame through a sliding window with a window length of 25ms and a step size of 10ms. The obtained frame signal is subjected to short-time Fourier transform. After taking the logarithm, the spectrogram of the speech signal is obtained. Then, the first-order difference and second-order difference of the spectrogram feature are calculated. The three are stacked to obtain the 3D spectrogram feature. The 3D spectrogram feature is enhanced by masking the frequency domain channel and masking the time domain channel. Then, the convolutional downsampling module is used to convert the 3D spectrogram feature matrix into a 2D feature matrix.

[0032] In a further technical solution, the filter used first for 3D spectrogram feature extraction is a Pre-Emphasis filter, and the filter formula is as follows:

[0033] x(n)' = x(n) - λ * x(n-1)

[0034] wherein x(n) is the value of the n th sampling point of the speech signal, x(n)' is the value of the n th sampling point of the speech signal after emphasis, λ is an emphasis coefficient, and is set to 0.97;

[0035] The short-time Fourier transform formula is as follows:

[0036]

[0037] wherein STFT(t,f) is the frequency spectrum of the speech signal at t time, τ is the offset value of the window function from the origin; w(τ-t) is an analysis window function centered on τ, and the formula is as follows:

[0038]

[0039] wherein N is the window length, and is set to 400;

[0040] The signal after the short-time Fourier transform is converted into a frequency spectrum formula as follows:

[0041]

[0042] wherein NFFT is set to 512;

[0043] The spectrogram is subjected to logarithmic operation to obtain the spectrogram feature, and the formula is as follows:

[0044] X log = log(X)

[0045] According to the spectrogram feature, the first-order difference and the second-order difference are calculated, and the formula is as follows:

[0046] Y(k) = X log (k+1) - X log (k)

[0047] Z(k) = Y(k+1) - Y(k)

[0048] wherein Y(k) is the first-order difference of the speech frame sequence at k, and Z(k) is the second-order difference of the speech frame sequence at k;

[0049] When the 3D spectrogram feature is subjected to frequency domain channel masking and time domain channel masking for speech data enhancement, a frequency band with a random width and a time band with a random width are set, wherein the maximum width of the frequency band is 27, and the maximum width of the time band is 70.

[0050] The convolutional downsampling module consists of two 2D convolutional layers, one linear layer, one dropout layer, and one merging layer. The two 2D convolutional layers have 512 kernels, a kernel size of (3,3), a stride of (2,2), same padding, and ReLU activation. The linear layer has an output dimension of 512. The dropout rate of the dropout layer is 0.5. The merging layer combines the last two dimensions of the three-dimensional feature matrix into one dimension.

[0051] In the above scheme, in step (3), during the fusion process, when the speech emotion features receive the text emotion features, the text emotion features are input into the attention mechanism to obtain their query vector matrix; the speech emotion features are input into the attention mechanism to obtain the key vector matrix and the value vector matrix; the query vector matrix and the key vector matrix are scaled after dot product and passed through Softmax to obtain the corresponding weight matrix of the value vector matrix, and the weight matrix is ​​multiplied point by point with the value vector matrix; then, after inputting into the LayerNorm and Feed-forward network layers, it becomes the output of the cross-modal Tansformer;

[0052] When receiving speech emotion features from text emotion features, the attention mechanism calculation process differs from the above steps in that the positions of text emotion features and speech emotion features are interchanged.

[0053] The matrices obtained by fusing the two cross-modal sentiment features are concatenated and passed through a linear layer to obtain the fused multimodal sentiment feature information.

[0054] In a further technical solution, the cross-modal Transformer consists of layer normalization, a cross-modal attention mechanism, a feedforward neural network, and a residual structure; the cross-modal attention mechanism accepts three 512-dimensional inputs: Q, K, and V, where the Q input is feature information of one modality, and the K and V inputs are feature information of another modality.

[0055] When text sentiment features are received from speech sentiment features, the text sentiment feature X T In the input attention mechanism, the query vector matrix Q is obtained. T Voice emotion features X S The key vector matrix K is obtained in the input attention mechanism. S Sum value vector matrix V S The formula is as follows:

[0056] Q T =W q X T

[0057] K S =W k X S

[0058] V S =W v X S

[0059] wherein, W q ,W k ,W v are weight matrix of query vector, key vector, value vector respectively; initialization mode is glorot_uniform;

[0060] The attention calculation formula is as follows:

[0061] Y att =V S ·softmax(s(K S ,Q T ))

[0062] wherein, s(K S ,Q T ) is a scoring formula, and its formula is as follows:

[0063]

[0064] wherein, d k is a scaling coefficient, and is set to 32;

[0065] The feedforward neural network is composed of one LayerNorm layer, two linear layers, two Dropout layers, one activation layer with a relu activation function, and a residual structure; wherein the output dimensions of the two linear layers are 256 and 512 respectively; the first layer has a relu activation function, and the second layer has no activation function; the dropout rate of the two Dropout layers is 0.5; the residual structure superimposes the input of the feedforward neural network and the output of the activation layer, and is represented as:

[0066] Y ffm =alpha*X ffm +(1-alpha)*Y relu

[0067] wherein, X ffm , Y relu represent the input of the feedforward neural network and the output of the activation layer respectively, and alpha is a superposition coefficient, which is set to 0.5.

[0068] In the above scheme, in step (4), the multi-modal sentiment feature is first input into the BiGRU, and its formula is as follows:

[0069]

[0070]

[0071]

[0072] wherein x t is the input at the current time t, denotes the forward GRU output vector at t, denotes the backward GRU output vector at t, GRU(·) denotes a unidirectional GRU module, denotes bit-wise addition;

[0073] The double-pooling layer is divided into two paths after output of BiGRU feature learning, one path of features is averaged-pooled, one path of features is max-pooled, the data after two-path pooling is spliced together, and a classification result Y is obtained through a linear layer Linear and a Softmax, and the formula is as follows:

[0074]

[0075]

[0076] Y = Softmax(Linear(Concat(p mean ,p max )))

[0077] wherein K denotes the number of features in the pooling range, and the output dimension of the linear layer linear is 6.

[0078] Through the above technical scheme, the emotion intelligent recognition method based on data enhancement and cross-modal feature fusion provided by the application has the following beneficial effects:

[0079] (1) The text data is enhanced by four methods, which are back-translation method, <pad>Replacement, synonym replacement, pseudo label prediction; two ways are used to enhance the voice data, which are to cover the frequency domain channel and to cover the time domain channel; the number of training samples is increased through the above methods, and the generalization and robustness of the model are improved. Meanwhile, the invention trains various data according to a certain proportion, solving the class imbalance problem of various classes.

[0080] (2) The present application adopts the form of text sentiment word vector and text semantic word vector weighting to construct text sentiment features, uses semantic word vector to solve the problem of sparse text feature representation and no correlation between semantics; at the same time, the prior knowledge in the CVAW sentiment dictionary is used to construct the text sentiment word vector, so that the sentiment tendency in the text is more easily identified.

[0081] (3) The present application adopts a cross-modal Transformer model to realize cross-modal feature information fusion, so that the information between multi-modal data is complementary, and at the same time solves the problem of data misalignment caused by different lengths of multi-modal data, and the long-term relationship dependence problem between multi-modal data fusion, greatly improving the overall recognition effect of the model.

[0082] (4) The present application adopts BiGRU and double pooling layer to realize the representation learning of sentiment features, BiGRU can effectively capture the long-distance dependency relationship; at the same time, the bidirectional model structure can make the model use the future hidden layer state information of the current input; the average pooling layer used in the double pooling layer effectively alleviates the problem of increased estimation value variance caused by limited neighborhood size, and the maximum pooling layer alleviates the problem of estimation mean deviation caused by convolution layer parameter error. BRIEF DESCRIPTION OF DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description.

[0084] Figure 1 The flowchart of the method disclosed by the present application is shown in the figure;

[0085] Figure 2 The text sentiment feature vector construction diagram is shown in the figure;

[0086] Figure 3 The 3D spectrogram frequency masking and time masking diagram is shown in the figure;

[0087] Figure 4 The convolution down-sampling module diagram is shown in the figure;

[0088] Figure 5 The cross-modal Transformer model diagram is shown in the figure;

[0089] Figure 6 The Feed-forward module structure diagram is shown in the figure;

[0090] Figure 7 A classification model structure based on BiGRU and double pooling layers is shown in the figure. DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

[0092] The present application provides an emotion intelligent recognition method based on data enhancement and cross-modal feature fusion, as shown in the figure, the specific implementation is as follows: Figure 1

[0093] I. Multimodal data acquisition

[0094] Open source multi-modal data set is acquired, only text data and voice data in the data set are used.

[0095] II. Data processing

[0096] 1. Text data enhancement

[0097] Four methods are used for text data enhancement, <pad>Replacement, synonym replacement, back-translation, pseudo-label prediction.

[0098] <pad>Replacement is to replace the words in the input sentence with a certain probability, and to replace the words in the input sentence with the words used to complete the BERT pre-training model sentence <pad>Symbol, get a new sample data, we set the probability to 0.3.

[0099] Synonym replacement method is to divide the text data into words, match all the words in the Chinese synonym table, if hit, perform synonym replacement, obtain a new sample data.

[0100] Back translation method is to translate the text data in Baidu translation according to the translation order of Chinese, English, Japanese and Chinese, and the obtained data is a new sample data.

[0101] Pseudo-label prediction method is to train a pre-training sentiment classification model by BERT pre-training model, and crawl unlabeled audio data from the network. After separating the text and speech, the pseudo-label of the sample data is obtained by the pre-training model, which is used as a new sample data.

[0102] Table 1 is the result of a sample after text data enhancement.

[0103] Table 1

[0104] Original sentence Although it was a long time ago, it still sends shivers down my spine when I think about it. <pad>substitutions < / pad> Although it was a long time ago, but one <pad>The rise is still chilling < / pad> Synonym replacement Although it was a long time ago, it still sends shivers down my spine when I think about it. Back-translation Although it was a long time ago, it still sends shivers down my spine when I think about it.

[0105] After completing the four types of data enhancement, different types of sample data are added to the training set. Ensure that the number of sample data of each type is the same.

[0106] 2, text sentiment feature vector construction

[0107] (1) The text sentiment feature vector constructed by the present application is obtained by splicing the text semantic word vector and the text sentiment word vector, as shown in the formula (1). Figure 2 First, the word vector matrix is trained by using skip-gram in Word2Vec, and the general Chinese text database Sogou corpus complete version is selected as the data set. Compared with the multi-modal sentiment data set, the database corpus is more and easier to train the appropriate word vector matrix. The database text data is first divided by jieba, and added to the dictionary to form the word dictionary, which has a size of N. Each word is encoded into an ont-hot vector by one-hot encoding. The Gensim library is called to train the ont-hot vector, and the number of hidden layers is set to 300. After training, the word vector matrix W 300×N .

[0108] (2) The text data in the multi-modal data set is cleaned, and the punctuation marks are removed and divided into words. Let the current sample be S=(s1,s2,...,si,...,sl), where si represents the i-th word in the sample, and i≤l. The one-hot encoding of the word obtains the vector o l . i i ​This vector and the word vector matrix W 300×N Multiplying them together yields a 300-dimensional text semantic word vector g. i The semantic word vector matrix G of the sample can be obtained by multiplying all the word segments in sequence. l×300 Its formula is expressed as:

[0109] g i =W 300×N ·o i T

[0110] G l×300 =[g1,g2,...,g l ]

[0111] Secondly, this invention uses the Chinese Valence-Arousal Words (CVAW) dictionary to obtain text sentiment word vectors. This dictionary contains 1653 sentiment words labeled with valence and arousal values. The valence value represents the pleasantness level of the sentiment word; a higher value indicates a greater pleasantness. The arousal value represents the excitement level of the sentiment word; a higher value indicates a greater excitement, and vice versa. All sentiment words in the valence dictionary are converted into word vector form, represented as E = (e1, e2, ..., e...). n ), where n = 1653, e n This represents the vector of the nth sentiment word. Each sentiment word corresponds to a set of VA values, represented as VA = ((ev1, ea1), (ev2, ea2), ..., (ev...). n ,ea n )), where ev n ea represents the valence value of the nth sentiment word in the dictionary. n The arousal value represents the nth sentiment word in the dictionary;

[0112] Get the i-th word s of the sample i word vector g i Then, calculate the word vector and the word vector e in E. j The cosine similarity is calculated using the following formula:

[0113]

[0114] Where j = 1, 2, 3, ..., n, the cosine similarity represents the degree of similarity between two vectors, ranging from -1 to 1, with a larger value indicating greater similarity. Let cosθ ij With e j g can be obtained by multiplying the corresponding VA values. i The mapped VA value is expressed by the formula:

[0115] ev j→i = cos ij θ·ev j

[0116] ea j→i = cos ij θ·ea j

[0117] After mapping the segmented words, a mapping matrix of size n x 2 is obtained, which is expressed as follows:

[0118] SVA i = [[ev 1→i , ea 1→i ], [ev 2→i , ea 2→i ],..., [ev n→i , ea n→i ]]

[0119] In order to compress the dimension of the matrix, the automatic encoder is used for dimension reduction, which is composed of only two fully connected layers Dense, the output dimension of the first fully connected layer is 212, and the activation function is relu. The output dimension of the second fully connected layer is n, and the activation function is sigmoid function, the optimizer is Adam optimizer, and the loss function is mean square error function (MSE). We take the 212-dimensional bottleneck feature as the sentiment word feature vector s VAi of the segmented word.

[0120] For the whole sample, a bottleneck feature matrix S VA of size l x 212 is obtained, which is the sentiment word feature matrix, and the formula is:

[0121] S VA = [s VA1 , s VA2 ,..., s VAl ]

[0122] The semantic word feature matrix and the sentiment word feature matrix are spliced to obtain the constructed text sentiment feature vector, which is expressed by the formula:

[0123] T l×512 = Concat(G l×300 , S VA l×212 )

[0124] 3, frame the voice data and extract 3D spectrogram features

[0125] First, the audio signal is sampled at a sampling rate of 16000Hz. Let the sampled input audio signal be x(n), the signal is pre-emphasized, and the filter used is a Pre-Emphasis filter, the filter formula is as follows:

[0126] x(n)' = x(n) - λ * x(n-1)

[0127] Where x(n) is the value of the nth sampling point of the speech signal, x(n)' is the value of the nth sampling point of the speech signal after emphasis, and λ is the emphasis coefficient, which is set to 0.97.

[0128] Subsequently, the speech signal is intercepted as a frame through a sliding window with a window length of 25ms and a step length of 10ms, and then the signal is subjected to short-time Fourier transform (STFT), the correlation between the signals in each dimension is removed, and the signal is mapped to a low-dimensional space, the formula is as follows:

[0129]

[0130] Where STFT(t,f) is the frequency spectrum of the speech signal at time t, w(τ-t) is the analysis window function centered at τ, and the formula is as follows:

[0131]

[0132] Where N is the window length, set to 400;

[0133] The signal subjected to short-time Fourier transform is converted into a frequency spectrum graph, and the formula is as follows:

[0134]

[0135] Where NFFT is set to 512.

[0136] The frequency spectrum graph is subjected to logarithmic operation to obtain the spectrogram feature, and the formula is as follows:

[0137] X log = log(X)

[0138] According to the spectrogram feature, the first-order difference and the second-order difference are calculated, and the formula is as follows:

[0139] Y(k) = X log (k+1) - X log (k)

[0140] Z(k) = Y(k+1) - Y(k)

[0141] where Y(k) is the first-order difference of the speech frame sequence at k, the physical meaning is the relationship between the current speech frame and the previous frame, and embodies the relationship between the adjacent two frames; Z(k) is the second-order difference of the speech frame sequence at k, the physical meaning is the relationship between the previous first-order difference and the next first-order difference, and embodies the dynamic relationship between the adjacent three frames.

[0142] 4. Speech data enhancement

[0143] Two ways of masking frequency domain channels and masking time domain channels are used for speech data enhancement on 3D spectrogram features. For each 3D spectrogram, the features stacked in each dimension are randomly masked with a width-random frequency band and a width-random time band, and the values in the range are set to 0. As shown in Figure 3 , we set the maximum width of the frequency band to 27 and the maximum width of the time band to 70.

[0144] The 3D spectrogram features after data enhancement need to be down-sampled by convolution before feature fusion, and the structure diagram is as shown in Figure 4 . The module is composed of two 2D convolution layers, one linear layer, one Dropout layer, and one merging layer. The number of convolution kernels of the two 2D convolution layers is 512, the size of the convolution kernel is (3, 3), the step is (2, 2), the padding mode is same, and the activation function is relu. The output dimension of the linear layer is 512. The dropout rate of the Dropout layer is 0.5. The merging layer merges the last two dimensions of the three-dimensional feature matrix into one dimension. The feature dimension after merging is (n f , 512), where n f is the number of frames that a speech sample can be divided into.

[0145] III. Cross-modal Transformer model realizes multi-modal feature information fusion

[0146] The feature fusion of the text emotion feature and the speech emotion feature of two modalities is realized by a cross-modal Transformer model, and the schematic diagram is as shown in Figure 5 . The cross-modal Transformer model is composed of layer normalization (LayerNorm), cross-modal attention mechanism (Crossmodel Attention), feed-forward neural network (Feed-forward module), and residual structure. The input feature vector is first normalized in each batch, and then input into the cross-modal attention module (Crossmodel Attention) in Q, K, and V three ways. When the speech emotion feature accepts the text emotion feature information, the text emotion feature X T inputs the Q road of the attention mechanism, and obtains the query vector matrix Q T ; voice emotion feature X S Input attention mechanism to get the key vector matrix K S And the value vector matrix V S The formula is as follows:

[0147] Q T = W q X T

[0148] K S = W k X S

[0149] V S = W v X S

[0150] Where, W q , W k , W v are the weight matrices of the query vector, the key vector and the value vector respectively. The initialization method is glorot_uniform.

[0151] The query vector matrix Q T and the key vector matrix K S After dot product, scaling and Softmax, the corresponding weight matrix of the value vector matrix is obtained, and the weight matrix is multiplied point by point with the value vector matrix K S , the formula is as follows:

[0152] Y S→T = V S ·softmax(s(K S ,Q T ))

[0153] Where, s(K S ,Q T ) is the scoring formula, the formula is as follows:

[0154]

[0155] Where, d k is the scaling coefficient, set to 32.

[0156] Then input into LayerNorm and Feed-forward module two modules, get the fusion feature matrix. Feed-forward module is composed of a LayerNorm layer, two linear layers, two Dropout layers, an activation layer with relu activation function, residual structure, etc. As shown in Figure 6 As shown. The output dimensions of the two linear layers are 256 and 512, respectively. The first layer uses ReLU activation, while the second layer has no activation function. The dropout rate of the two Dropout layers is 0.5. The residual structure superimposes the input of the Feed-forward module with the output of the activation layer, and is represented as:

[0157] Y ffm =α·X ffm +(1-α)·Y relu

[0158] Among them, X ffm Y relu These represent the input and activation layer output of the Feed-forward module, respectively; α is the superposition coefficient, set to 0.5.

[0159] When receiving speech emotion feature information from text emotion features, speech emotion feature X S The Q-path of the input attention mechanism is used to obtain its query vector matrix Q. T Text sentiment features X T The key vector matrix K is obtained in the input attention mechanism. S Sum value vector matrix V S The remaining calculation steps are the same as those described above.

[0160] The matrices obtained by fusing the two cross-modal sentiment features are concatenated. To enhance the model's linear representation capability, the output is then passed through a linear layer (Linear) with an output dimension of 512 and a ReLU activation function. This yields the fused multimodal sentiment features. The formula for the cross-modal Transformer model can be expressed as:

[0161] Y (1) =LN(X T )+α·CA(X T ,X S )

[0162]

[0163]

[0164] Where LN(·), CA(·), and FF(·) represent LayerNorm, Crossmodel Attention, and Feed-forward, respectively. The cross-modal Transformer model is composed of multiple layers of the above structure stacked together. This represents the model output of the Nth layer, where N is 16. α is the stacking coefficient of the residual structure, set to 0.5.

[0165] IV. Based on BiGRU and double pooling layer, the multi-modal fusion emotion feature data is subjected to feature learning, and cross-entropy loss function is used for training

[0166] The multi-modal fusion emotion feature is subjected to representation learning through BiGRU and double pooling layer, as shown in FIG. 2. Figure 7 The spliced multi-modal emotion feature is first input into BiGRU. The network is composed of two layers of GRU with opposite directions. GRU has strong ability to model time series data and capture time domain correlation in data. It can be regarded as a memory structure composed of update gate and reset gate. The gate can allow or block information transmission along the sequence, thereby capturing long-term dependencies. Let the current input x t , and its formula is as follows:

[0167] z = σ (x t U z +s t-1 W z )

[0168] r = σ (x t U r +s t-1 W r )

[0169] h t = tanh (x t U h + (s t-1 ⊙r) W h )

[0170] s t = (1-z) ⊙ h t +z ⊙ s t-1

[0171] Wherein, s t-1 represents the hidden layer output of the current GRU module t-1, z is the output of the update gate of the network, which helps the model to decide how much past information to pass to the future. r is the reset gate of the network, which decides how much past information needs to be forgotten. h t is the output of the GRU network. ⊙ represents Hadamard product.

[0172] The BiGRU adopted in the application is to extract features by integrating the outputs of the current forward GRU and reverse GRU, and its formula is as follows:

[0173]

[0174]

[0175]

[0176] wherein, represents a forward GRU output vector at t, represents a backward GRU output vector at t. GRU(·) represents a unidirectional GRU module. represents a bitwise addition.

[0177] After BiGRU feature learning, the output is divided into two paths, one path of features is averaged pooled, and one path of features is maximum pooled. The two paths of pooled data are spliced together, and the classification result Y is obtained through a linear layer Linear and a Softmax, and the formula is as follows:

[0178]

[0179]

[0180] Y = Softmax(Linear(Concat(p mean ,p max )))

[0181] wherein, K represents the number of features in the pooling range, the output dimension of the linear layer linear is 6, and Y obtains the probability of each classification after passing through the Softmax. The maximum value of the maximum value corresponds to the emotion, that is, the classification result.

[0182] The optimizer adopts the Adam optimizer, the loss function is the cross-entropy loss function, and the learning rate strategy is the warm-up strategy, and the formula is as follows:

[0183]

[0184] wherein, d model is the output dimension of the cross-modal Transformer, which is set to 512. step warm-up is the warm-up step, which is set to 0.2 times of the total step. step represents the iteration number, which starts from 1.

[0185] Five, using the trained classification model to perform emotion recognition, and obtaining an emotion recognition result.

[0186] The above description of the disclosed embodiments enables one skilled in the art to make or use the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to these embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.< / pad> < / pad> < / pad> ​< / pad> < / pad> < / pad> < / pad>

Claims

1. A method for intelligent emotion recognition based on data augmentation and cross-modal feature fusion, characterized in that, Includes the following steps: (1) Multimodal data acquisition; (2) Data augmentation is performed on the text data, and the enhanced text data is used to generate text semantic word vectors and text sentiment word vectors, thereby constructing text sentiment features; the speech data is processed to obtain the speech signal spectrogram, and then the first-order difference and second-order difference of the spectrogram features are calculated. The three are stacked to obtain 3D spectrogram features. The 3D spectrogram features are enhanced by masking the frequency domain channel and masking the time domain channel. Then, the 3D spectrogram feature matrix is ​​converted into a 2D feature matrix using the convolutional downsampling module to obtain the speech sentiment features; (3) Use the cross-modal Transformer model to fuse text sentiment features and speech sentiment features to obtain fused multimodal sentiment features; specifically including: When text sentiment features are received from speech sentiment features, the text sentiment features are input into the attention mechanism to obtain their query vector matrix; the speech sentiment features are input into the attention mechanism to obtain the key vector matrix and the value vector matrix; the query vector matrix and the key vector matrix are multiplied by a dot product, scaled, and then passed through Softmax to obtain the corresponding weight matrix of the value vector matrix, and this weight matrix is ​​multiplied point by point with the value vector matrix; then, after being input into the LayerNorm and Feed-forward network layers, it becomes the output of the cross-modal Tansformer. When receiving speech emotion features from text emotion features, the attention mechanism calculation process differs from the above steps in that the positions of text emotion features and speech emotion features are interchanged. The matrices obtained by fusing the two cross-modal sentiment features are concatenated and passed through a linear layer to obtain the fused multimodal sentiment feature information. (4) A classification model based on BiGRU and dual pooling layers is used to learn features from the fused multimodal sentiment features, and the model is trained using the cross-entropy loss function; specifically including: Multimodal sentiment features are first input into BiGRU, and the formula is as follows: ； ； ； in, This is the input at the current time t. This represents the forward GRU output vector at time t. This represents the backward GRU output vector at time t. Indicates a unidirectional GRU module. Indicates bitwise addition; The dual-pooling layer divides the output after BiGRU feature learning into two paths: one path performs average pooling, and the other performs max pooling. The data from both pooling paths are concatenated and then passed through a linear layer and a softmax layer to obtain the classification result. The formula is as follows: ； ； ； Where K represents the number of features within the pooling range, and the linear layer has an output dimension of 6; (5) Use the trained classification model to perform emotion recognition and obtain the emotion recognition results.

2. The emotion intelligent recognition method based on data augmentation and cross-modal feature fusion according to claim 1, characterized in that, In step (2), the text data is transferred through... <pad> Data augmentation is accomplished using four methods: replacement, synonym replacement, back translation, and pseudo-label prediction. After data augmentation, sample data from different categories are added to the training set to ensure that the proportion of samples from each category in the training set is the same.< / pad> 3. The emotion intelligent recognition method based on data augmentation and cross-modal feature fusion according to claim 2, characterized in that, The <pad>Replace with: Text data, with a probability of 0.3, replaces words in the input sentence with words used to complete the sentence of the BERT pre-trained model. <pad> The symbol is used to obtain a new sample data;< / pad> < / pad> The synonym replacement is as follows: the text data is first segmented into words, and all segments are matched against the Chinese synonym list. If a match is found, a synonym replacement is performed to obtain a new sample data. The back-translation method is as follows: the text data is translated in Baidu Translate in the order of Chinese, English, Japanese and Chinese, and the resulting data is a new sample data. The pseudo-label prediction is as follows: the original text data is trained on the BERT model to obtain a pre-trained model, unlabeled audio data is crawled from the network, separated into text and speech, and the pseudo-label of the sample data is predicted by the pre-trained model and used as a new sample data.

4. The emotion intelligent recognition method based on data augmentation and cross-modal feature fusion according to claim 1, characterized in that, In step (2), the method for constructing text sentiment features is as follows: First, a word vector matrix is ​​trained based on Word2Vec. The text data is cleaned and segmented. The word vector matrix is ​​multiplied by the one-hot vector of the sample word segmentation to obtain a 300-dimensional text semantic word vector. Secondly, based on the continuous sentiment word dictionary, the sentiment words with emotional tendencies in the text are mapped to the sentiment space to obtain the mapping matrix. Then, the bottleneck feature vector of the mapping matrix, i.e. the text sentiment word vector, is obtained through an autoencoder. Then, the semantic word vectors and sentiment word vectors of the text are fused to obtain the constructed text sentiment features.

5. The emotion intelligent recognition method based on data augmentation and cross-modal feature fusion according to claim 4, characterized in that, The continuous sentiment lexicon contains 1653 sentiment words labeled with valence and arousal values. All sentiment words in the continuous sentiment lexicon are converted into word vectors, represented as follows: Where n=1653, This represents the vector of the nth sentiment word; Each sentiment word corresponds to a set of VA values, represented as follows: ,in, This represents the valence value of the nth sentiment word in the dictionary. The arousal value represents the nth sentiment word in the dictionary; The sentiment word mapping in the text data is obtained by acquiring the i-th word segment of the sample. word vectors Then, calculate the word vector and... Chinese word vectors The cosine similarity is calculated using the following formula: ； in, ,use and The corresponding VA values ​​can be multiplied to obtain the result. The mapped VA value is expressed by the following formula: ； ； After the word segmentation is mapped, a value of [size missing] can be obtained. The mapping matrix is ​​represented as follows: ； The autoencoder consists of two fully connected Dense layers. The first fully connected layer has an output dimension of 2^12 and uses ReLU as the activation function. The second fully connected layer has an output dimension of n and uses the sigmoid function as the activation function. The optimizer is the Adam optimizer, and the loss function is the mean squared error function.

6. The emotion intelligent recognition method based on data augmentation and cross-modal feature fusion according to claim 1, characterized in that, In step (2), the speech sample is amplified at high frequencies using a filter, and then the speech signal is extracted as a frame through a sliding window with a window length of 25ms and a step size of 10ms. The obtained frame signal is subjected to short-time Fourier transform, and the spectrogram of the speech signal is obtained after taking the logarithm.

7. The emotion intelligent recognition method based on data augmentation and cross-modal feature fusion according to claim 6, characterized in that, The first filter used for 3D spectrogram feature extraction is the Pre-Emphasis filter, and the filter formula is as follows: ； in, The value of the nth sampling point of the speech signal. This is the value of the speech signal after emphasis at the nth sampling point. The aggravation factor is set to 0.97; The short-time Fourier transform formula is as follows: ； in, Let be the spectrum of the speech signal at time t. This is the offset of the window function from the origin; Therefore The analysis window function centered on is given by the following formula: ； in, Set the window length to 400. The formula for converting a signal after short-time Fourier transform into a spectrum is as follows: ； The NFFT value is set to 512. The spectrogram features can be obtained by performing a logarithmic operation on the spectrogram, and the formula is as follows: ； The first-order difference and second-order difference are calculated based on the spectrogram characteristics, and the formulas are as follows: ； ； in, As a speech frame sequence The first difference at the point, As a speech frame sequence The second difference at the point; When enhancing speech data by masking the frequency domain channel and the time domain channel of 3D spectrogram features, a frequency band with a random width and a time band with a random width are set, where the maximum width of the frequency band is 27 and the maximum width of the time band is 70. The convolutional downsampling module consists of two 2D convolutional layers, one linear layer, one dropout layer, and one merging layer. The two 2D convolutional layers have 512 kernels, a kernel size of (3,3), a stride of (2,2), same padding, and ReLU activation. The linear layer has an output dimension of 512. The dropout rate of the dropout layer is 0.

5. The merging layer combines the last two dimensions of the three-dimensional feature matrix into one dimension.

8. The emotion intelligent recognition method based on data augmentation and cross-modal feature fusion according to claim 1, characterized in that, The cross-modal Transformer consists of layer normalization, a cross-modal attention mechanism, a feedforward neural network, and a residual structure. The cross-modal attention mechanism accepts three 512-dimensional inputs: Q, K, and V. The Q input is the feature information of one modality, and the K and V inputs are the feature information of another modality. When text sentiment features are received from speech sentiment features, the text sentiment features In the input attention mechanism, the query vector matrix is ​​obtained. Voice emotional features The key vector matrix is ​​obtained in the input attention mechanism. Sum value vector matrix The formula is as follows: ； ； ； in, , , These are the weight matrices for the query vector, key vector, and value vector, respectively; initialized using glorot_uniform. The formula for calculating attention is as follows: ； in, The scoring formula is as follows: ； in, The scaling factor is set to 32; The feedforward neural network consists of one LayerNorm layer, two linear layers, two Dropout layers, one ReLU activation layer, and a residual structure. The two linear layers have output dimensions of 256 and 512, respectively. The first layer uses ReLU activation, while the second layer has no activation function. The dropout rate of the two Dropout layers is 0.

5. The residual structure superimposes the input of the feedforward neural network with the output of the activation layer, and is represented as follows: ； in, , These represent the input and activation layer output of the feedforward neural network, respectively. The superposition coefficient is set to 0.5.

Images