Emotion recognition method, device, computer equipment and storage medium

Abstract

The present application relates to an emotion recognition method, including: obtaining a first text feature and a corresponding first frequency cepstrum feature and voiceprint feature in speech data; cross-attentionally fusing the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature; encoding the second frequency cepstrum feature and the voiceprint feature to obtain a first audio coding feature and a second audio coding feature; encoding the first text feature and the second text feature to obtain a first text coding feature and a second text coding feature; fusing the first audio coding feature and the first text coding feature with multiple attentions to obtain a first fused feature; obtaining a second fused feature from the first fused feature according to a dynamic routing algorithm; and obtaining an emotion recognition result according to the second fused feature, the second text coding feature, and the second audio coding feature. The method can make full use of the text features and speech features corresponding to the speech data to achieve high-precision emotion recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of artificial intelligence and the help of deep learning, the field of human-computer interaction has attracted more and more attention from researchers. As an important branch of human-computer interaction, emotion recognition has also become a hot research direction. At present, most of the research on emotion recognition focuses on single-modal fields such as speech, facial expressions, and text. As the most direct means of communication for people, speech contains rich emotional information, and changes in people's emotions can be reflected through speech features. Speech emotion recognition converts the input speech signal containing emotional information into readable physical features, extracts speech features related to emotional expression, constructs an emotion recognition classifier for testing and training, and finally outputs the emotion recognition classification results. However, when traditional technologies fuse speech and text, the fusion method is relatively simple, resulting in low emotion recognition accuracy. Summary of the invention

[0003] Based on this, it is necessary to provide an emotion recognition method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the accuracy of emotion recognition in response to the above technical problems.

[0004] In a first aspect, the present application provides an emotion recognition method. The method comprises:

[0005] Acquire voice data, and extract a first text feature and a corresponding voice feature from the voice data, wherein the voice feature includes a first frequency cepstrum feature and a voiceprint feature;

[0006] Cross-attentionally fuse the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature;

[0007] Encoding the second frequency cepstrum feature and the voiceprint feature to obtain a first audio coding feature and a second audio coding feature;

[0008] Encoding the first text feature and the second text feature to obtain a first text encoding feature and a second text encoding feature;

[0009] Performing multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fused feature;

[0010] The first fused features are fused according to a dynamic routing algorithm to obtain a second fused feature;

[0011] The second fusion feature, the second text encoding feature and the second audio encoding feature are spliced ​​to obtain a splicing result, and the splicing result is classified and recognized to obtain an emotion recognition result.

[0012] In one embodiment, cross-attention fusion of the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature includes:

[0013] Calculating a first attention weight of the first text feature relative to the first frequency cepstrum feature, and a second attention weight of the first frequency cepstrum feature relative to the first text feature;

[0014] Determining the second text feature according to the first attention weight and the first frequency cepstrum feature;

[0015] The second frequency cepstrum feature is determined according to the second attention weight and the first text feature.

[0016] In one embodiment, determining the second text feature according to the first attention weight and the first frequency cepstrum feature includes:

[0017] taking the product of the first attention weight and the first frequency cepstrum feature as the second text feature;

[0018] Determining the second frequency cepstrum feature according to the second attention weight and the first text feature includes:

[0019] The product of the second attention weight and the first text feature is used as the second frequency cepstrum feature.

[0020] In one embodiment, encoding the second frequency cepstrum feature and the voiceprint feature to obtain a first audio coding feature and a second audio coding feature includes:

[0021] The second frequency cepstrum feature is subjected to convolution and pooling processing to obtain a first result;

[0022] The standard deviation, maximum value and average value of the first result are concatenated with the voiceprint feature to obtain a second result, and the second result is pooled to obtain the first audio coding feature;

[0023] Perform dimensionality reduction processing on the first audio coding feature to obtain the second audio coding feature.

[0024] In one embodiment, encoding the first text feature and the second text feature to obtain a first text encoding feature and a second text encoding feature includes:

[0025] The second text feature is subjected to convolution and pooling processing to obtain a third result;

[0026] The standard deviation, maximum value and average value of the third result are concatenated with the first text feature to obtain a fourth result, and the fourth result is pooled to obtain the first text encoding feature;

[0027] Perform dimensionality reduction processing on the first text encoding feature to obtain the second text encoding feature.

[0028] In one embodiment, fusing the first fused features according to a dynamic routing algorithm to obtain a second fused feature includes:

[0029] Using the dynamic routing algorithm to fuse the first fused features to obtain an intermediate fused feature;

[0030] The intermediate fusion features are fused using the dynamic routing algorithm to obtain the second fusion features.

[0031] In a second aspect, the present application further provides an emotion recognition device, the device comprising:

[0032] A feature extraction module, used to obtain voice data, extract a first text feature and a corresponding voice feature in the voice data, wherein the voice feature includes a first frequency cepstrum feature and a voiceprint feature;

[0033] A first fusion module, used for cross-attentionally fusing the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature;

[0034] A first encoding module, used for encoding the second frequency cepstrum feature and the voiceprint feature to obtain a first audio encoding feature and a second audio encoding feature;

[0035] A second encoding module, used for encoding the first text feature and the second text feature to obtain a first text encoding feature and a second text encoding feature;

[0036] A second fusion module, configured to perform multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fused feature;

[0037] A third fusion module, configured to fuse the first fusion features according to a dynamic routing algorithm to obtain a second fusion feature;

[0038] The emotion recognition module is used to splice the second fusion feature, the second text encoding feature and the second audio encoding feature to obtain a splicing result, and classify and identify the splicing result to obtain an emotion recognition result.

[0039] In one embodiment, the first fusion module is further used to:

[0040] Calculating a first attention weight of the first text feature relative to the first frequency cepstrum feature, and a second attention weight of the first frequency cepstrum feature relative to the first text feature;

[0041] Determining the second text feature according to the first attention weight and the first frequency cepstrum feature;

[0042] The second frequency cepstrum feature is determined according to the second attention weight and the first text feature.

[0043] In one embodiment, the first fusion module is further used to:

[0044] taking the product of the first attention weight and the first frequency cepstrum feature as the second text feature;

[0045] Determining the second frequency cepstrum feature according to the second attention weight and the first text feature includes:

[0046] The product of the second attention weight and the first text feature is used as the second frequency cepstrum feature.

[0047] In one embodiment, the first encoding module is further used to:

[0048] The second frequency cepstrum feature is subjected to convolution and pooling processing to obtain a first result;

[0049] The standard deviation, maximum value and average value of the first result are concatenated with the voiceprint feature to obtain a second result, and the second result is pooled to obtain the first audio coding feature;

[0050] Perform dimensionality reduction processing on the first audio coding feature to obtain the second audio coding feature.

[0051] In one embodiment, the second encoding module is further used to:

[0052] The second text feature is subjected to convolution and pooling processing to obtain a third result;

[0053] The standard deviation, maximum value and average value of the third result are concatenated with the first text feature to obtain a fourth result, and the fourth result is pooled to obtain the first text encoding feature;

[0054] Perform dimensionality reduction processing on the first text encoding feature to obtain the second text encoding feature.

[0055] In one embodiment, the third fusion module is further used for:

[0056] Using the dynamic routing algorithm to fuse the first fused features to obtain an intermediate fused feature;

[0057] The intermediate fusion features are fused using the dynamic routing algorithm to obtain the second fusion features.

[0058] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0059] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0060] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0061] The above-mentioned emotion recognition method, device, computer equipment, storage medium and computer program product, by acquiring voice data, extracting the first text feature and the corresponding voice feature in the voice data, the voice feature including the first frequency cepstrum feature and the voiceprint feature; cross-attention fusion of the first text feature and the first frequency cepstrum feature to obtain the second text feature and the second frequency cepstrum feature; encoding the second frequency cepstrum feature and the voiceprint feature to obtain the first audio coding feature and the second audio coding feature; encoding the first text feature and the second text feature to obtain the first text coding feature and the second text coding feature; multi-head attention fusion of the first audio coding feature and the first text coding feature to obtain the first fusion feature; fusion of the first fusion feature according to the dynamic routing algorithm to obtain the second fusion feature; splicing the second fusion feature, the second text coding feature and the second audio coding feature to obtain the splicing result, and classifying and identifying the splicing result to obtain the emotion recognition result. The present application can make full use of the text features and voice features corresponding to the voice data and realize high-precision emotion recognition by encoding and deeply fusing the text features and voice features corresponding to the voice data, and then classifying and emotion recognition of the encoding and fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is an application environment diagram of an emotion recognition method in an embodiment;

[0063] Figure 2 is a flowchart of an emotion recognition method in one embodiment;

[0064] Figure 3 is a flow chart of step 204 in one embodiment;

[0065] Figure 4 is a flow chart of step 206 in one embodiment;

[0066] Figure 5 is a flow chart of step 208 in one embodiment;

[0067] Figure 6 is a flow chart of step 212 in one embodiment;

[0068] Figure 7 is a flowchart of an emotion recognition method in another embodiment;

[0069] Figure 8 is a flowchart of an emotion recognition method in another embodiment;

[0070] Fig. 9 is a flow chart of step 706 in one embodiment;

[0071] Fig.10 is a structural block diagram of an emotion recognition device in one embodiment;

[0072] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0074] The emotion recognition method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the voice data sent by the terminal 102, extracts the first text feature and the corresponding voice feature in the voice data, wherein the voice feature includes a first frequency cepstrum feature and a voiceprint feature; the first text feature and the first frequency cepstrum feature are cross-attentively fused to obtain a second text feature and a second frequency cepstrum feature; the second frequency cepstrum feature and the voiceprint feature are encoded to obtain a first audio coding feature and a second audio coding feature; the first text feature and the second text feature are encoded to obtain a first text coding feature and a second text coding feature; the first audio coding feature and the first text coding feature are multi-headedly fused to obtain a first fusion feature; the first fusion feature is fused according to the dynamic routing algorithm to obtain a second fusion feature; the second fusion feature, the second text coding feature and the second audio coding feature are spliced ​​to obtain a splicing result, and the splicing result is classified and recognized to obtain an emotion recognition result.

[0075] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0076] It should be noted here that any step in the emotion recognition method disclosed in the embodiment of the present application can be implemented by the terminal 102 and the server 104 in an interactive manner, or by the server 104 alone, or by the terminal 102 alone, and there is no limitation here.

[0077] In one embodiment, Figure 2 As shown, an emotion recognition method is provided, which is applied to Figure 1 The server in is taken as an example to illustrate, including the following steps 202 to 214.

[0078] Step 202: Acquire speech data, extract first text features and corresponding speech features in the speech data, wherein the speech features include first frequency cepstrum features and voiceprint features.

[0079] The server can receive the voice data sent by the terminal, or directly obtain the voice data through the interface. The voice data can be directly recorded voice data, or voice data extracted from other forms of data, for example, voice data extracted from multimedia data. The server extracts the first text feature and the corresponding voice feature in the voice data, wherein the first text feature can be understood as a text feature in the voice data that can be converted into text form, and the voice feature includes a first frequency cepstrum feature and a voiceprint feature. The first frequency cepstrum feature can be, for example, an MFCC (Mel-scale Frequency Cepstral Coefficients) feature, which represents the distribution of the energy of the signal spectrum in different frequency intervals. The voiceprint feature can be, for example, an ivert (Identity-Vector) feature, which represents a feature with obvious distinguishing features such as the timbre of the corresponding voice data.

[0080] The first text feature converts each word in the corresponding text into a corresponding vector representation to obtain the word embedding matrix corresponding to the entire text. The first frequency cepstrum feature can convert the frequency cepstrum image into a corresponding matrix format. After the voiceprint feature is converted into a matrix or vector format, subsequent processing is performed.

[0081] Step 204 , cross-attentionally fuse the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature.

[0082] The server cross-attentionally fuses the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature. Specifically, the first text feature and the first frequency cepstrum feature are cross-fused through an attention mechanism to obtain a second text feature and a second frequency cepstrum feature. For example, the second text feature and the second frequency cepstrum feature are obtained based on an attention weight between the first text feature and the first frequency cepstrum feature.

[0083] Step 206: Encode the second frequency cepstrum feature and the voiceprint feature to obtain a first audio encoding feature and a second audio encoding feature.

[0084] The server encodes the second frequency cepstrum feature and the voiceprint feature to obtain the first audio encoding feature and the second audio encoding feature. Specifically, the encoding process is equivalent to the process of data processing the second frequency cepstrum feature and the voiceprint feature, for example, performing processing processes such as pooling, convolution, and dimensional alignment.

[0085] Step 208: Encode the first text feature and the second text feature to obtain a first text encoding feature and a second text encoding feature.

[0086] The server encodes the first text feature and the second text feature to obtain the first text encoding feature and the second text encoding feature. The encoding process in this embodiment can be the same as the encoding process in step 206, except that the encoding object is replaced by the first text feature and the second text feature, and the encoding result is the first text encoding feature and the second text encoding feature. That is, after the first text feature and the second text feature are subjected to pooling, convolution, dimension alignment and other processing processes, the first text encoding feature and the second text encoding feature are obtained.

[0087] Step 210, performing multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fused feature.

[0088] The server performs multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fused feature. Multi-head attention uses multiple queries to calculate and select multiple information from the input information in parallel, and each attention focuses on a different part of the input information.

[0089] In one possible implementation, after aligning the dimensions of the first audio coding feature and the first text coding feature, multi-head attention fusion is performed. Dimension alignment is to make the dimensions of the two consistent, which is convenient for subsequent calculation and processing. When performing multi-head attention fusion, the first audio coding feature and the first text coding feature are respectively split into multiple sub-features to obtain multiple first audio coding sub-features and multiple first text coding sub-features, and the multiple first audio coding sub-features and the multiple first text coding sub-features are subjected to multi-head attention fusion to obtain multiple first fusion features.

[0090] Step 212: The first fused features are fused according to a dynamic routing algorithm to obtain a second fused feature.

[0091] The server fuses the first fused features according to the dynamic routing algorithm to obtain the second fused features. The dynamic routing algorithm is an automatic update algorithm applied in the capsule network, which fuses the features according to the correlation between the features. The capsule network is composed of multiple capsules instead of neurons. A capsule includes a group of neurons, which represent the instantiation parameters of a specific type of entity, for example, the direction, possibility, size, etc. of an object or part of an object.

[0092] In a specific application scenario, multiple fusions can be performed continuously according to the dynamic routing algorithm to finally obtain a second fusion feature. For example, the first fusion feature can be fused according to the dynamic routing algorithm to obtain an intermediate fusion feature, and then the intermediate fusion feature can be fused using the dynamic routing algorithm to obtain a second fusion feature.

[0093] Step 214, concatenate the second fusion feature, the second text encoding feature, and the second audio encoding feature to obtain a concatenation result, and classify and identify the concatenation result to obtain an emotion recognition result.

[0094] The server concatenates the second fusion feature, the second text encoding feature, and the second audio encoding feature to obtain a concatenation result, and classifies and identifies the concatenation result to obtain an emotion recognition result. When classifying and identifying the concatenation result, the classification layer of the model may be used for classification and identification, or other classification algorithms may be used for classification and identification. The emotion recognition result is usually in the form of a vector, and each numerical value in the vector is used to represent different emotions. For example, the numerical value in the vector is a numerical value between 0 and 1. If the numerical value is 0.9, it indicates a positive emotion; the numerical value is 0.5, it indicates a neutral emotion; and the numerical value is 0.2, it indicates a negative emotion. In practical applications, the correspondence between the numerical values ​​in the vector and the emotion category can be set according to the specific situation, and no further limitation is made here.

[0095] In the above-mentioned emotion recognition method, by acquiring speech data, extracting the first text feature and the corresponding speech feature in the speech data, the speech feature includes a first frequency cepstrum feature and a voiceprint feature; cross-attention fusion of the first text feature and the first frequency cepstrum feature is performed to obtain a second text feature and a second frequency cepstrum feature; the second frequency cepstrum feature and the voiceprint feature are encoded to obtain a first audio coding feature and a second audio coding feature; the first text feature and the second text feature are encoded to obtain a first text coding feature and a second text coding feature; the first audio coding feature and the first text coding feature are multi-headed attention fused to obtain a first fusion feature; the first fusion feature is fused according to a dynamic routing algorithm to obtain a second fusion feature; the second fusion feature, the second text coding feature and the second audio coding feature are spliced ​​to obtain a splicing result, and the splicing result is classified and identified to obtain an emotion recognition result. The embodiment of the present application can make full use of the text features and speech features corresponding to the speech data by encoding and deeply fusing the text features and speech features corresponding to the speech data, and then classifying and emotion identifying the encoding and fusion results, so as to achieve high-precision emotion recognition.

[0096] In one embodiment, Figure 3 As shown, the first text feature and the first frequency cepstrum feature are cross-attended and fused to obtain the second text feature and the second frequency cepstrum feature in step 204, including the following steps 302 to 306:

[0097] Step 302: Calculate a first attention weight of the first text feature relative to the first frequency cepstrum feature, and a second attention weight of the first frequency cepstrum feature relative to the first text feature.

[0098] The first attention weight of the first text feature relative to the first frequency cepstrum feature refers to the degree of "attention" of the first frequency cepstrum feature to the first text feature. Usually, the first text feature includes multiple first text sub-features, and the first frequency cepstrum feature includes multiple first frequency cepstrum sub-features. When calculating the first attention weight of the first text feature relative to the first frequency cepstrum feature, the attention weight of each first frequency cepstrum feature to each first text feature is calculated separately to obtain the first attention weight. Similarly, when calculating the second attention weight of the first frequency cepstrum feature relative to the first text feature, the attention weight of each first text feature to each first frequency cepstrum feature is calculated separately to obtain the second attention weight.

[0099] In a specific example, assuming that the first text feature is A = [a1 a2 a3 a4 a5], and the first frequency cepstrum feature is B = [b1 b2 b3 b4 b5], then the first attention weight is C1, and the second attention weight is C2, then:

[0100]

[0101] Among them, a i b j Indicates a i Relative to b j The attention weights are, where i = 1, ..., 5; j = 1, ..., 5; b m a n Indicates b m Relative to a n Attention weights, where m = 1,…, 5; n = 1,…, 5,.

[0102] In one possible implementation, the first attention weight and the second attention weight are determined according to the first text feature and the first frequency cepstrum feature. Specifically, the product of the first text feature and the transposition of the first frequency cepstrum feature can be used as the first attention weight, and the product of the first frequency cepstrum feature and the transposition of the first text feature can be used as the second attention weight. Exemplarily, assuming that the first text feature is A and the first frequency cepstrum feature is B, then the first attention weight is C1=AB T , the second attention weight is C2 = BA T Optionally, the first attention weight and the second attention weight are normalized, such as by a softmax function, and converted into values ​​between 0 and 1 before participating in subsequent calculations.

[0103] Step 304: Determine a second text feature according to the first attention weight and the first frequency cepstrum feature.

[0104] In this embodiment, the server determines the second text feature according to the first attention weight and the first frequency cepstrum feature. The second text feature can be determined by the product of the first attention weight and the first frequency cepstrum feature, or by the dot product of the first attention weight and the first frequency cepstrum feature.

[0105] Step 306: Determine a second frequency cepstrum feature according to the second attention weight and the first text feature.

[0106] In this embodiment, the server determines the second frequency cepstrum feature according to the second attention weight and the first text feature. The second frequency cepstrum feature can be determined by the product of the second attention weight and the first text feature, or by the dot product of the second attention weight and the first text feature.

[0107] In one embodiment, step 304 of determining the second text feature according to the first attention weight and the first frequency cepstrum feature includes the following steps.

[0108] The product of the first attention weight and the first frequency cepstrum feature is used as the second text feature.

[0109] The step 306 of determining a second frequency cepstrum feature according to the second attention weight and the first text feature includes:

[0110] The product of the second attention weight and the first text feature is taken as the second frequency cepstrum feature.

[0111] In this embodiment, the product of the first attention weight and the first frequency cepstrum feature is used as the second text feature, and the product of the second attention weight and the first text feature is used as the second frequency cepstrum feature. It is understandable that on the basis of the product of the first attention weight and the first frequency cepstrum feature, or on the basis of the product of the second attention weight and the first text feature, multiplying or dividing by the corresponding coefficient, or adding a constant, to obtain the corresponding second text feature or second frequency cepstrum feature should be regarded as within the protection scope of the embodiment of the present application.

[0112] In one embodiment, Figure 4 As shown, the step 206 of encoding the second frequency cepstrum feature and the voiceprint feature to obtain the first audio coding feature and the second audio coding feature includes the following steps 402 to 406.

[0113] Step 402: The second frequency cepstrum feature is subjected to convolution and pooling processing to obtain a first result.

[0114] In this embodiment, the second frequency cepstrum feature is subjected to convolution and pooling processing, which may be first subjected to convolution processing and then subjected to pooling processing, or first subjected to pooling processing and then subjected to convolution processing. At the same time, the convolution or pooling process may be performed multiple times. In one example, the second frequency cepstrum feature is subjected to a convolution processing and then a pooling processing, and this process is regarded as a group of convolution and pooling processing. After multiple groups of convolution and pooling processing are repeated, for example, repeated 5 times, a first result is obtained.

[0115] Step 404, concatenate the standard deviation, maximum value and average value of the first result with the voiceprint feature to obtain a second result, and perform pooling processing on the second result to obtain the first audio coding feature.

[0116] In this embodiment, after obtaining the first result, the standard deviation, maximum value and average value of the first result are calculated respectively, and the standard deviation, maximum value and average value of the first result are concatenated with the voiceprint feature to obtain the second result, and the second result is further pooled to obtain the first audio coding feature.

[0117] Step 406: Perform dimensionality reduction processing on the first audio coding feature to obtain a second audio coding feature.

[0118] In this embodiment, the first audio coding feature can be processed by dimension reduction according to the first preset dimension, or it can be processed by dimension reduction with reference to the minimum dimension of other features that are subsequently processed together to obtain the second audio coding feature, which is convenient for subsequent calculation processing.

[0119] In one embodiment, Figure 5 As shown, the step 208 of encoding the first text feature and the second text feature to obtain the first text encoding feature and the second text encoding feature includes the following steps 502 to 506.

[0120] Step 502: The second text feature is subjected to convolution and pooling processing to obtain a third result.

[0121] In this embodiment, the second text feature is subjected to convolution and pooling processing, which may be first subjected to convolution processing and then subjected to pooling processing, or first subjected to pooling processing and then subjected to convolution processing. At the same time, the convolution or pooling process may be performed multiple times. In one example, the second text feature is subjected to a convolution processing once and then subjected to a pooling processing once. This process is regarded as a set of convolution and pooling processing. After multiple sets of convolution and pooling processing are repeated, for example, repeated 5 times, a third result is obtained.

[0122] Step 504: concatenate the standard deviation, maximum value and average value of the third result with the first text feature to obtain a fourth result, and perform pooling processing on the fourth result to obtain the first text encoding feature.

[0123] In this embodiment, after obtaining the third result, the standard deviation, maximum value and average value of the third result are calculated respectively, and the standard deviation, maximum value and average value of the third result are concatenated with the voiceprint feature to obtain the fourth result, and the fourth result is further pooled to obtain the first text encoding feature.

[0124] Step 506: Perform dimensionality reduction processing on the first text encoding feature to obtain a second text encoding feature.

[0125] In this embodiment, the first audio coding feature can be processed for dimensionality reduction according to the second preset dimension, or it can be processed for dimensionality reduction with reference to the minimum dimension of other features that are subsequently processed together, to obtain the second text coding feature, which is convenient for subsequent calculation processing.

[0126] In one embodiment, Figure 6 As shown, the step 212 of fusing the first fusion features according to the dynamic routing algorithm to obtain the second fusion features includes the following steps 602 to 604.

[0127] Step 602: Use a dynamic routing algorithm to fuse the first fusion features to obtain intermediate fusion features.

[0128] In this embodiment, there are multiple first fusion features, each of which is split into multiple first fusion sub-features, and based on the correlation between the multiple first fusion sub-features, the multiple first fusion sub-features are fused according to a dynamic routing algorithm to obtain an intermediate fusion feature.

[0129] Step 604: Use a dynamic routing algorithm to fuse the intermediate fusion features to obtain a second fusion feature.

[0130] In this embodiment, based on the correlation between the multiple first fusion features, the intermediate fusion features corresponding to the multiple first fusion features are fused according to the dynamic routing algorithm to obtain the second fusion feature.

[0131] In this embodiment, the first fusion feature is fused by using the dynamic routing algorithm twice to obtain the second fusion feature. The first fusion using the dynamic routing algorithm is equivalent to fusing the first fusion sub-feature of the first fusion feature, that is, fusing the local features of the first fusion feature. The second fusion using the dynamic routing algorithm is equivalent to fusing the overall features of the first fusion feature. The two fusions can make the data fusion more complete, realize the deep fusion of text features and voice features, and further improve the recognition accuracy of emotion recognition.

[0132] In one embodiment, Figure 7 As shown, an emotion recognition method includes the following steps 702 to 714.

[0133] Step 702: Acquire speech data, extract first text features and corresponding speech features in the speech data, wherein the speech features include first frequency cepstrum features and voiceprint features.

[0134] The voice data may be directly recorded voice data, or voice data extracted from other forms of data, for example, voice data extracted from multimedia data. The server extracts the first text feature and the corresponding voice feature from the voice data, wherein the first text feature can be understood as a text feature in the voice data that can be converted into text form, and the voice feature includes a first frequency cepstrum feature and a voiceprint feature. Figure 8 In the example shown, speech data is acquired, and features such as Emb, MFCC, and ivictor are extracted from the speech data, where Emb is the first text feature, MFCC is the first frequency cepstrum feature, and ivictor is the voiceprint feature.

[0135] Step 704 , cross-attentionally fuse the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature.

[0136] In this embodiment, the second text feature and the second frequency cepstrum feature can be obtained according to the attention weight between the first text feature and the first frequency cepstrum feature. The first attention weight of the first text feature relative to the first frequency cepstrum feature and the second attention weight of the first frequency cepstrum feature relative to the first text feature are calculated, and the product of the first attention weight and the first frequency cepstrum feature is used as the second text feature, and the product of the second attention weight and the first text feature is used as the second frequency cepstrum feature. Figure 8 As shown in , the Emb feature and the MFCC feature are cross-attention fused to obtain the Emb_2 feature and the MFCC_2 feature, the Emb_2 feature is the second text feature, and the MFCC_2 feature is the second frequency cepstrum feature.

[0137] Step 706: Encode the second frequency cepstrum feature and the voiceprint feature to obtain a first audio encoding feature and a second audio encoding feature.

[0138] In this embodiment, the second frequency cepstrum feature is subjected to convolution and pooling processing to obtain a first result; the standard deviation, maximum value and average value of the first result are concatenated with the voiceprint feature to obtain a second result, and the second result is subjected to pooling processing to obtain a first audio coding feature; the first audio coding feature is subjected to dimensionality reduction processing to obtain a second audio coding feature. Figure 8 In the example shown, the MFCC_2 feature and the iverson feature are input together into the speech encoder for encoding to obtain the Audio_p feature and the Audio_h feature, wherein the Audio_p feature is the first audio coding feature and the Audio_h feature is the second audio coding feature.

[0139] In one possible implementation, Fig. 9 As shown, the MFCC_2 feature is input into the convolution pooling module for processing to obtain a first result, the standard deviation, maximum value and average value of the first result are concatenated with the ivert feature to obtain a second result, the second result is input into the pooling module for pooling processing to obtain the Audio_p feature, and the Audio_p feature is input into the Dense module for dimensionality reduction processing to obtain the Audio_h feature.

[0140] Step 708: Encode the first text feature and the second text feature to obtain a first text encoding feature and a second text encoding feature.

[0141] In this embodiment, the second text feature is subjected to convolution and pooling processing to obtain a third result, the standard deviation, maximum value and average value of the third result are concatenated with the first text feature to obtain a fourth result, the fourth result is subjected to pooling processing to obtain the first text encoding feature, and the first text encoding feature is subjected to dimensionality reduction processing to obtain the second text encoding feature. Figure 8 In the example shown, the Emb feature and the Emb_2 feature are input into the text encoder for encoding to obtain the text_h feature and the text_p feature, where the text_p feature is the first text encoding feature and the text_h feature is the second text encoding feature.

[0142] Step 710: Perform multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fused feature.

[0143] In this embodiment, the first audio coding feature and the first text coding feature can be split into multiple sub-features to obtain multiple first audio coding sub-features and multiple first text coding sub-features, and the multiple first audio coding sub-features and the multiple first text coding sub-features are fused by multi-head attention to obtain multiple first fused features. Figure 8 In the example shown, the text_p feature and the Audio_p feature are fused by multi-head attention to obtain the Audio_text feature, which is the first fused feature. Obviously, the first fused feature is the result of deep fusion of the first text feature and the speech feature of the speech data.

[0144] Step 712: Use a dynamic routing algorithm to fuse the first fused features to obtain an intermediate fused feature; and use a dynamic routing algorithm to fuse the intermediate fused features to obtain a second fused feature.

[0145] In this embodiment, a dynamic routing algorithm is used to fuse the first fusion feature twice to obtain a second fusion feature. There are multiple first fusion features, and each first fusion feature is split into multiple first fusion sub-features. Based on the correlation between the multiple first fusion sub-features, the multiple first fusion sub-features are fused according to the dynamic routing algorithm to obtain an intermediate fusion feature. Based on the correlation between the multiple first fusion features, the intermediate fusion features corresponding to the multiple first fusion features are fused according to the dynamic routing algorithm to obtain the second fusion feature. Figure 8 In the example shown, the Audio_text feature is fused twice according to the dynamic routing algorithm to obtain the A_T feature, which is the second fused feature.

[0146] Step 714: concatenate the second fusion feature, the second text encoding feature, and the second audio encoding feature to obtain a concatenation result, and classify and identify the concatenation result to obtain an emotion recognition result.

[0147] exist Figure 8 In the example shown, the text_h feature, the Audio_h feature, and the A_T feature are concatenated to obtain a concatenation result, and the concatenation result is input into a classifier for classification and recognition to obtain an emotion recognition result.

[0148] The above-mentioned emotion recognition method can fully utilize the text features and speech features corresponding to the speech data to achieve high-precision emotion recognition by performing multi-level data fusion such as cross-attention fusion, encoding, multi-head attention fusion and two dynamic routing algorithm fusion on the text features and speech features corresponding to the speech data, and then classifying and identifying the emotions on the fused results.

[0149] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0150] Based on the same inventive concept, the embodiment of the present application also provides an emotion recognition device for implementing the emotion recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more emotion recognition device embodiments provided below can refer to the limitations of the emotion recognition method above, and will not be repeated here.

[0151] In one embodiment, Fig.10 As shown, an emotion recognition device is provided, including: a feature extraction module 1002, a first fusion module 1004, a first encoding module 1006, a second encoding module 1008, a second fusion module 1010, a third fusion module 1012 and an emotion recognition module 1014, wherein:

[0152] A feature extraction module 1002 is used to obtain voice data and extract a first text feature and a corresponding voice feature from the voice data, wherein the voice feature includes a first frequency cepstrum feature and a voiceprint feature;

[0153] A first fusion module 1004 is used to perform cross-attention fusion on the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature;

[0154] A first encoding module 1006, configured to encode the second frequency cepstrum feature and the voiceprint feature to obtain a first audio encoding feature and a second audio encoding feature;

[0155] A second encoding module 1008, configured to encode the first text feature and the second text feature to obtain a first text encoding feature and a second text encoding feature;

[0156] A second fusion module 1010 is used to perform multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fusion feature;

[0157] A third fusion module 1012 is used to fuse the first fusion features according to a dynamic routing algorithm to obtain a second fusion feature;

[0158] The emotion recognition module 1014 is used to concatenate the second fusion feature, the second text encoding feature and the second audio encoding feature to obtain a concatenation result, and classify and recognize the concatenation result to obtain an emotion recognition result.

[0159] In one embodiment, the first fusion module 1004 is further configured to:

[0160] Calculating a first attention weight of the first text feature relative to the first frequency cepstrum feature, and a second attention weight of the first frequency cepstrum feature relative to the first text feature;

[0161] Determining the second text feature according to the first attention weight and the first frequency cepstrum feature;

[0162] The second frequency cepstrum feature is determined according to the second attention weight and the first text feature.

[0163] In one embodiment, the first fusion module 1004 is further configured to:

[0164] taking the product of the first attention weight and the first frequency cepstrum feature as the second text feature;

[0165] Determining the second frequency cepstrum feature according to the second attention weight and the first text feature includes:

[0166] The product of the second attention weight and the first text feature is used as the second frequency cepstrum feature.

[0167] In one embodiment, the first encoding module 1006 is further configured to:

[0168] The second frequency cepstrum feature is subjected to convolution and pooling processing to obtain a first result;

[0169] The standard deviation, maximum value and average value of the first result are concatenated with the voiceprint feature to obtain a second result, and the second result is pooled to obtain the first audio coding feature;

[0170] Perform dimensionality reduction processing on the first audio coding feature to obtain the second audio coding feature.

[0171] In one embodiment, the second encoding module 1008 is further configured to:

[0172] The second text feature is subjected to convolution and pooling processing to obtain a third result;

[0173] The standard deviation, maximum value and average value of the third result are concatenated with the first text feature to obtain a fourth result, and the fourth result is pooled to obtain the first text encoding feature;

[0174] Perform dimensionality reduction processing on the first text encoding feature to obtain the second text encoding feature.

[0175] In one embodiment, the third fusion module 1012 is further used for:

[0176] Using the dynamic routing algorithm to fuse the first fused features to obtain an intermediate fused feature;

[0177] The intermediate fusion features are fused using the dynamic routing algorithm to obtain the second fusion features.

[0178] Each module in the above-mentioned emotion recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0179] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store text feature and voice feature data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an emotion recognition method is implemented.

[0180] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the emotion recognition method in the above embodiment are implemented.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the emotion recognition method in the above embodiment are implemented.

[0183] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the emotion recognition method in the above embodiment when executed by a processor.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0185] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An emotion recognition method, characterized in that: The method comprises: Acquire voice data, extract a first text feature and a corresponding voice feature from the voice data, wherein the voice feature includes a first frequency cepstrum feature and a voiceprint feature; Cross-attentionally fusing the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature; Encoding the second frequency cepstrum feature and the voiceprint feature to obtain a first audio coding feature; performing dimensionality reduction processing on the first audio coding feature to obtain a second audio coding feature; Encoding the first text feature and the second text feature to obtain a first text encoding feature; performing dimensionality reduction processing on the first text encoding feature to obtain a second text encoding feature; Performing multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fused feature; The first fused features are fused according to a dynamic routing algorithm to obtain a second fused feature; The second fusion feature, the second text encoding feature and the second audio encoding feature are spliced ​​to obtain a splicing result, and the splicing result is classified and recognized to obtain an emotion recognition result.

2. The method according to claim 1, characterized in that The cross-attention fusion of the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature comprises: Calculating a first attention weight of the first text feature relative to the first frequency cepstrum feature, and a second attention weight of the first frequency cepstrum feature relative to the first text feature; Determining the second text feature according to the first attention weight and the first frequency cepstrum feature; The second frequency cepstrum feature is determined according to the second attention weight and the first text feature.

3. The method according to claim 2, characterized in that According to the first attention weight and the first frequency cepstrum feature, Determining the second text feature includes: Taking the product of the first attention weight and the first frequency cepstrum feature as the second text feature; The determining the second frequency cepstrum feature according to the second attention weight and the first text feature includes: The product of the second attention weight and the first text feature is used as the second frequency cepstrum feature.

4. The method according to claim 1, characterized in that: The step of encoding the second frequency cepstrum feature and the voiceprint feature to obtain a first audio coding feature includes: The second frequency cepstrum feature is subjected to convolution and pooling processing to obtain a first result; The standard deviation, maximum value and average value of the first result are concatenated with the voiceprint feature to obtain a second result, and the second result is pooled to obtain the first audio coding feature.

5. The method according to claim 1, characterized in that The step of encoding the first text feature and the second text feature to obtain a first text encoding feature includes: The second text feature is subjected to convolution and pooling processing to obtain a third result; The standard deviation, maximum value and average value of the third result are concatenated with the first text feature to obtain a fourth result, and the fourth result is pooled to obtain the first text encoding feature.

6. The method according to claim 1, characterized in that The step of fusing the first fused features according to the dynamic routing algorithm to obtain the second fused features includes: Using the dynamic routing algorithm to fuse the first fused features to obtain an intermediate fused feature; The intermediate fusion features are fused using the dynamic routing algorithm to obtain the second fusion features.

7. An emotion recognition device, characterized in that: The device comprises: A feature extraction module, used to obtain voice data, extract a first text feature and a corresponding voice feature in the voice data, wherein the voice feature includes a first frequency cepstrum feature and a voiceprint feature; A first fusion module, used for cross-attentionally fusing the first text feature and the first frequency cepstrum feature to obtain a second text feature and a second frequency cepstrum feature; a first encoding module, configured to encode the second frequency cepstrum feature and the voiceprint feature to obtain a first audio encoding feature; and perform dimensionality reduction processing on the first audio encoding feature to obtain a second audio encoding feature; A second encoding module is used to encode the first text feature and the second text feature to obtain a first text encoding feature; and perform dimensionality reduction processing on the first text encoding feature to obtain a second text encoding feature; A second fusion module, configured to perform multi-head attention fusion on the first audio coding feature and the first text coding feature to obtain a first fused feature; A third fusion module, configured to fuse the first fusion features according to a dynamic routing algorithm to obtain a second fusion feature; The emotion recognition module is used to splice the second fusion feature, the second text encoding feature and the second audio encoding feature to obtain a splicing result, and classify and identify the splicing result to obtain an emotion recognition result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images