Voice emotion recognition method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a speech emotion recognition method and device, electronic equipment and storage medium, the method comprising: obtaining audio data to be recognized and text data corresponding to the audio data; obtaining audio features corresponding to each audio frame in the audio data, and text features corresponding to each word in the text data; determining an alignment matrix corresponding to the audio data according to the audio data and the text data; determining an audio representation matrix corresponding to the audio data based on the alignment matrix and the audio features corresponding to each audio frame; determining a multi-modal representation vector corresponding to the audio data according to an audio representation matrix and a text representation matrix formed by the text features; and performing emotion recognition based on the multi-modal representation vector to obtain an emotion recognition result. This scheme realizes the interactive fusion of audio and text modalities, thereby improving the accuracy of speech emotion recognition.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for voice emotion recognition. Background Technology

[0002] With the development of artificial intelligence technology, emotion recognition has been widely applied in various fields. Currently, the common method for emotion recognition is to perform speech recognition on audio to obtain the corresponding transcribed text, and then identify the emotional information in the audio based on the textual features of the transcribed text.

[0003] It is evident that existing emotion recognition methods only consider information from the text content modality, resulting in low accuracy of emotion recognition results. Summary of the Invention

[0004] To address or at least partially address the aforementioned technical problems, this disclosure provides a voice emotion recognition method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of this disclosure, a voice emotion recognition method is provided, comprising:

[0006] Obtain the audio data to be identified and the corresponding text data;

[0007] Obtain the audio features corresponding to each audio frame in the audio data, and the text features corresponding to each character in the text data;

[0008] Based on the audio data and the text data, an alignment matrix corresponding to the audio data is determined, wherein an element with a value of 1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame belongs to the j-th word, i takes the value from 1 to n, j takes the value from 1 to m, n represents the total number of audio frames, and m represents the total number of words contained in the audio data.

[0009] Based on the alignment matrix and the audio features corresponding to each audio frame, the audio representation matrix corresponding to the audio data is determined;

[0010] Based on the audio representation matrix and the text representation matrix composed of the text features, determine the multimodal representation vector corresponding to the audio data;

[0011] Emotion recognition is performed based on the multimodal representation vectors to obtain the emotion recognition results.

[0012] According to another aspect of this disclosure, a voice emotion recognition device is provided, comprising:

[0013] The first acquisition module is used to acquire the audio data to be identified and the text data corresponding to the audio data;

[0014] The second acquisition module is used to acquire the audio features corresponding to each audio frame in the audio data, and the text features corresponding to each character in the text data;

[0015] The first determining module is used to determine the alignment matrix corresponding to the audio data based on the audio data and the text data, wherein the element with a value of 1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame belongs to the j-th word, i takes the value from 1 to n, j takes the value from 1 to m, n represents the total number of audio frames, and m represents the total number of words contained in the audio data.

[0016] The second determining module is used to determine the audio representation matrix corresponding to the audio data based on the alignment matrix and the audio features corresponding to each audio frame.

[0017] The third determining module is used to determine the multimodal representation vector corresponding to the audio data based on the audio representation matrix and the text representation matrix composed of the text features.

[0018] The recognition module is used to perform emotion recognition based on the multimodal representation vector to obtain the emotion recognition result.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0020] Processor; and

[0021] Stored program memory,

[0022] The program includes instructions that, when executed by the processor, cause the processor to perform the voice emotion recognition method according to the foregoing aspect.

[0023] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the speech emotion recognition method according to the foregoing aspect.

[0024] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the speech emotion recognition method described in the foregoing aspect.

[0025] One or more technical solutions provided in this disclosure acquire audio data to be identified and corresponding text data, and acquire audio features corresponding to each audio frame in the audio data and text features corresponding to each character in the text data. Based on the audio data and text data, an alignment matrix corresponding to the audio data is determined. Then, based on the alignment matrix and the audio features corresponding to each audio frame, an audio representation matrix corresponding to the audio data is determined. Finally, based on the text representation matrix composed of the audio representation matrix and text features, a multimodal representation vector corresponding to the audio data is determined. Emotion recognition is then performed based on the multimodal representation vector to obtain the emotion recognition result. By using the scheme of this disclosure to extract audio features corresponding to each character using the alignment matrix, fine-grained feature recognition at the character level is achieved. Furthermore, the audio representation and text representation are fused to determine the multimodal representation vector corresponding to the audio data for emotion recognition, realizing the interactive fusion of audio and text modalities, thereby improving the accuracy of speech emotion recognition. Attached Figure Description

[0026] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart of a speech emotion recognition method according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 2 A flowchart of a speech emotion recognition method according to another exemplary embodiment of the present disclosure is shown;

[0029] Figure 3 A flowchart of a speech emotion recognition method according to yet another exemplary embodiment of the present disclosure is shown;

[0030] Figure 4 A schematic diagram of the network structure of a speech emotion recognition model according to an exemplary embodiment of the present disclosure is shown;

[0031] Figure 5 A schematic block diagram of a voice emotion recognition device according to an exemplary embodiment of the present disclosure is shown;

[0032] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0033] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0034] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0035] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] The following description, with reference to the accompanying drawings, outlines the speech emotion recognition method, apparatus, electronic device, and storage medium provided in this disclosure.

[0039] Currently, in related technologies, when performing emotion recognition, the audio is usually first transcribed into text through speech recognition, and then emotion recognition is performed based on the textual features of the transcribed text to obtain the emotional information contained in the audio.

[0040] However, emotions are multimodal; both sound and language content can express emotions. For example, even the same content conveyed with different tones will evoke different emotions. Therefore, considering only information from a single text modality is insufficient for emotion recognition, leading to inaccurate results. Furthermore, existing emotion recognition methods typically focus only on the overall information of the sentence, neglecting fine-grained information.

[0041] To address the aforementioned problems, this disclosure provides a speech emotion recognition method. It acquires the audio data to be recognized and the corresponding text data, extracts the audio features corresponding to each audio frame in the audio data, and the text features corresponding to each character in the text data. Based on the audio and text data, an alignment matrix corresponding to the audio data is determined. Then, based on the alignment matrix and the audio features corresponding to each audio frame, an audio representation matrix corresponding to the audio data is determined. Finally, based on the text representation matrix formed by the audio representation matrix and the text features, a multimodal representation vector corresponding to the audio data is determined. Emotion recognition is then performed based on the multimodal representation vector to obtain the emotion recognition result. The scheme of this disclosure utilizes the alignment matrix to extract the audio features corresponding to each character, achieving fine-grained feature recognition at the character level. By fusing audio and text representations to determine the multimodal representation vector corresponding to the audio data for emotion recognition, it achieves interactive fusion of audio and text modalities, thereby improving the accuracy of speech emotion recognition. The scheme provided by this disclosure can be applied to scenarios such as classroom speech emotion recognition and chatbots.

[0042] Figure 1 A flowchart of a speech emotion recognition method according to an exemplary embodiment of the present disclosure is shown. The method can be executed by a speech emotion recognition device provided in the embodiments of the present disclosure, wherein the device can be implemented in software and / or hardware, and can generally be integrated in an electronic device, such as a computer or a server.

[0043] like Figure 1 As shown, the speech emotion recognition method may include the following steps:

[0044] Step 101: Obtain the audio data to be identified and the text data corresponding to the audio data.

[0045] The audio data to be recognized can be a sentence. The corresponding text data can be obtained by transcribing the audio data using a speech recognition tool, which will not be explained in detail here. In online education scenarios, for a single lesson, the audio data of a sentence can be obtained by segmenting it according to a fixed length or using Voice Activity Detection (VAD) technology.

[0046] Step 102: Obtain the audio features corresponding to each audio frame in the audio data, and the text features corresponding to each character in the text data.

[0047] In this embodiment of the disclosure, feature extraction can be performed on the acquired audio data and its corresponding text data to obtain the audio features corresponding to each frame in the audio data and the text features corresponding to each character in the text data.

[0048] For example, when acquiring audio features, the audio data can first be segmented into frames. The audio data is cut into segments of a preset length (e.g., 25 milliseconds) according to a preset time interval (e.g., 10 milliseconds), and one segment is one audio frame. Next, feature extraction is performed on each audio frame to obtain the audio features corresponding to each audio frame in the audio data. In the feature extraction, common acoustic features can be extracted as audio features. For example, traditional methods can be used to statistically analyze features such as sound volume and frequency as audio features. Alternatively, audio feature extraction models such as Large Scale Pretrained Audio Neural Networks (PANNs) trained on the audio set can be used to extract the audio features of each audio frame.

[0049] For example, when obtaining the text features corresponding to each character, the text data can be segmented into characters to obtain multiple characters in the text data. Then, the corresponding text features can be obtained for each character. For example, pre-trained language representation models such as BERT (Bidirectional Encoder Representation from Transformers) and word vector models can be used to obtain the text features of each character in the text data.

[0050] Step 103: Based on the audio data and the text data, determine the alignment matrix corresponding to the audio data, wherein an element with a value of 1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame belongs to the j-th character, i takes the value from 1 to n, j takes the value from 1 to m, n represents the total number of audio frames, and m represents the total number of characters contained in the audio data.

[0051] In this embodiment of the disclosure, in order to capture the subtle emotional expressions implied in sound, fine-grained emotion recognition is performed by extracting the potential acoustic representation corresponding to each word. To accurately identify the acoustic representation corresponding to each word, this disclosure uses an alignment matrix to represent the correspondence between each word and each audio frame. The alignment matrix can be determined based on the audio data and text data.

[0052] For example, the audio text alignment tool MFA (Montreal Forced Aligner) can be used to extract single-word-level alignment matrices. The number of rows in the alignment matrix is ​​the total number of audio frames segmented from the audio data (denoted as n), and the list of the alignment matrix is ​​the total number of characters contained in the text data (denoted as m). An element with a value of 1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame belongs to the j-th character, and an element with a value of non-1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame does not belong to the j-th character. The value of i ranges from 1 to n, and the value of j ranges from 1 to m.

[0053] Step 104: Based on the alignment matrix and the audio features corresponding to each audio frame, determine the audio representation matrix corresponding to the audio data.

[0054] In this embodiment of the disclosure, after determining the alignment matrix, the audio representation matrix corresponding to the audio data can be determined based on the alignment matrix and the audio features corresponding to each audio frame.

[0055] For example, for each character in the audio data, the target audio features corresponding to each character can be determined based on the correspondence between each character and each audio frame represented in the alignment matrix. Then, the audio representation corresponding to each character can be determined based on the target audio features. Following the order of the characters in the audio data, the audio representations corresponding to each character are aggregated to obtain the audio representation matrix corresponding to the audio data. Specifically, when determining the audio representation corresponding to each character based on the target audio features, the average of all target audio features corresponding to the same character can be calculated. The mean of the data in the same dimension among all target audio features is used as the value of the corresponding dimension in the audio representation. For example, the mean of the data in the first dimension among all target audio features is used as the value of the first dimension in the audio representation, and so on, until the audio representation corresponding to that character is finally obtained. Of course, when determining the audio representation corresponding to each character, the value of the corresponding dimension in the audio representation can also be determined by finding the maximum value or summing the data in the same dimension; this disclosure does not impose any limitations on this.

[0056] In this embodiment of the disclosure, in the audio representation matrix corresponding to the audio data, one column of the representation matrix is ​​the audio representation corresponding to a word, that is, the number of columns of the representation matrix is ​​consistent with the total number of words contained in the audio data.

[0057] Step 105: Determine the multimodal representation vector corresponding to the audio data based on the audio representation matrix and the text representation matrix composed of the text features.

[0058] In this embodiment of the disclosure, text features can form a text representation matrix. For example, after obtaining the text features corresponding to each character, the text features corresponding to each word can be transformed through a multi-layer fully connected layer to obtain the text representation matrix corresponding to the audio data. Each column of the text representation matrix represents the text feature corresponding to a single character, and the number of columns in the text representation matrix is ​​consistent with the total number of characters contained in the text data.

[0059] Next, after determining the audio representation matrix and text representation matrix corresponding to the audio data, the multimodal representation vector corresponding to the audio data can be determined based on the audio representation matrix and text representation matrix.

[0060] For example, when determining the multimodal representation vector corresponding to the audio data, the corresponding columns of the audio representation matrix and the text representation matrix can be concatenated first. The first column of the text representation matrix is ​​concatenated after the first column of the audio representation matrix, the second column of the text representation matrix is ​​concatenated after the second column of the audio representation matrix, and so on, to obtain the multimodal representation matrix corresponding to the audio data. Then, the value of each row can be determined as the value of the corresponding row in the multimodal representation vector by taking the maximum value, average value, or summation of the elements in the same row of the multimodal representation matrix.

[0061] For example, when determining the multimodal representation vector corresponding to audio data, the multimodal representation matrix corresponding to the audio data can be obtained by fusing the corresponding data of the audio representation matrix and the text representation matrix through summation, averaging, etc. Then, the value of each row can be determined as the value of the corresponding row in the multimodal representation vector by taking the maximum value, averaging, summing, etc. of the elements in the same row of the multimodal representation matrix.

[0062] It should be noted that in the embodiments of this disclosure and subsequent embodiments, the dimensions of each feature and each matrix are not limited, and can be flexibly set according to actual needs and calculation requirements, as long as the calculation between features and matrices can be completed smoothly.

[0063] Step 106: Perform emotion recognition based on the multimodal representation vector to obtain the emotion recognition result.

[0064] In this embodiment of the disclosure, after determining the multimodal representation vector corresponding to the audio data, emotion recognition can be performed based on the multimodal representation vector to obtain the emotion recognition result. Depending on the actual needs, the emotion recognition result can be a label of "emotional" or "no emotion," or a specific emotion label, such as "angry," "happy," or "sad." In other words, the emotion recognition task of this disclosure can be binary classification or multi-class classification, and the specific settings can be configured according to actual needs; this disclosure does not impose any limitations on this.

[0065] It is understood that the above steps in the embodiments of this disclosure can be implemented by different network layers of the trained emotion recognition model. The determined multimodal representation vector is input to the multi-layer fully connected layers of the emotion recognition model, and the final predicted probability distribution of each emotion label is obtained using the softmax function. For example, if the emotion recognition model is defined as a binary classification model, i.e., with emotion and without emotion, and assuming the output predicted probability distribution is [0.1, 0.9], where 0.9 represents the probability of having emotion, then the emotion recognition result corresponding to the audio data is having emotion.

[0066] The speech emotion recognition method of this disclosure acquires audio data to be recognized and corresponding text data, obtains audio features corresponding to each audio frame in the audio data, and text features corresponding to each character in the text data, and determines an alignment matrix corresponding to the audio data based on the audio data and text data. Then, based on the alignment matrix and the audio features corresponding to each audio frame, an audio representation matrix corresponding to the audio data is determined, and based on the text representation matrix composed of the audio representation matrix and text features, a multimodal representation vector corresponding to the audio data is determined. Emotion recognition is then performed based on the multimodal representation vector to obtain the emotion recognition result. By using the scheme of this disclosure to extract audio features corresponding to each character using the alignment matrix, fine-grained feature recognition at the character level is achieved. Furthermore, the audio representation and text representation are fused to determine the multimodal representation vector corresponding to the audio data for emotion recognition, realizing the interactive fusion of audio and text modalities, thereby improving the accuracy of speech emotion recognition.

[0067] In one alternative embodiment of this disclosure, such as Figure 2 As shown, in Figure 1 Based on the illustrated embodiment, step 104 may include the following sub-steps:

[0068] Step 201: Iterate through each word in the audio data.

[0069] Step 202: Based on the alignment matrix, determine the target audio frame belonging to the currently traversed word.

[0070] In this embodiment, each character in the audio data can be traversed. Starting from the first character, the first character is taken as the currently traversed character. Based on the alignment matrix, the row numbers with an element value of 1 in the first column of the alignment matrix are found. The audio frames corresponding to these rows are the target audio frames belonging to the first character (the currently traversed character). That is, for the first character in the audio data, the value of i in the i-th row and first column of the alignment matrix is ​​found to be 1, and the i-th audio frame is the target audio frame belonging to the first character. This process is repeated for each character in the audio data to find the target audio frame belonging to each character. For whitespace characters, the element value in the alignment matrix is ​​0.

[0071] Step 203: Determine the fused audio features corresponding to the currently traversed character based on the target audio features corresponding to the target audio frame.

[0072] In this embodiment of the disclosure, for the currently traversed character, the fused audio features corresponding to the currently traversed character can be determined based on the audio features (referred to as target audio features) corresponding to the target audio frame belonging to the character.

[0073] As one possible implementation, if there is only one target audio frame belonging to the currently traversed character, then the target audio feature corresponding to that target audio frame is determined as the fused audio feature corresponding to the currently traversed character; if there are multiple target audio frames belonging to the currently traversed character (at least two), then the target audio features corresponding to the multiple target audio frames (i.e., multiple target audio features) can be fused to obtain the fused audio feature corresponding to the currently traversed character.

[0074] For example, when determining the fused audio feature corresponding to the currently traversed character based on multiple target audio features, the target audio features can be summed, and the sum can be used as the fused audio feature corresponding to the currently traversed character. It is understood that summing the target audio features means summing the data at the same position in each target audio feature to obtain the value at the corresponding position in the fused audio feature. For example, a target audio feature can be treated as a column, and the data in the same row can be calculated as the value of the corresponding row in the fused audio feature. Alternatively, a target audio feature can be treated as a row, and the data in the same column can be calculated as the value of the corresponding column in the fused audio feature; this disclosure does not limit the specific calculation method.

[0075] For example, when determining the fused audio feature corresponding to the currently traversed word based on multiple target audio features, the average audio feature and the maximum audio feature of these multiple target audio features can be determined. Determining the average audio feature and the maximum audio feature involves averaging and maximizing the data at the same position in each target audio feature, obtaining the value at the corresponding position in the average audio feature and the value at the corresponding position in the maximum audio feature, respectively. For example, a target audio feature can be treated as a column, and the target audio features can be arranged sequentially. The average value of the data in the same row can be calculated as the value of the corresponding row in the average audio feature, and the maximum value from the data in the same row can be determined as the value of the corresponding row in the maximum audio feature. Alternatively, a target audio feature can be treated as a row, and the average and maximum values ​​of the data in the same column can be calculated to determine the average audio feature and the maximum audio feature; this disclosure does not limit this approach. Then, the obtained average audio feature and maximum audio feature are concatenated to obtain the fused audio feature corresponding to the currently traversed word. In this process, the splicing can be performed by splicing the largest audio feature after the average audio feature, or by splicing the average audio feature after the largest audio feature. This disclosure does not limit the specific splicing method.

[0076] Step 204: In response to the end of the traversal, generate an audio representation matrix corresponding to the audio data based on the fused audio features corresponding to each character.

[0077] In this embodiment of the disclosure, after traversing each character in the audio data, the fused audio features corresponding to each character are obtained, and an audio representation matrix corresponding to the audio data can be generated based on the fused audio features corresponding to each character.

[0078] For example, the fused audio features corresponding to each character can be arranged sequentially according to the order of the characters in the audio data to obtain an audio representation matrix. In this matrix, each column represents the fused audio feature corresponding to a single character.

[0079] The speech emotion recognition method of this disclosure traverses each character in the audio data and determines the target audio frame belonging to the currently traversed character based on the alignment matrix. According to the target audio features corresponding to the target audio frame, the fused audio features corresponding to the currently traversed character are determined. Then, in response to the end of traversal, an audio representation matrix corresponding to the audio data is generated according to the fused audio features corresponding to each character. Thus, the audio features of each character in the audio data are extracted, achieving the purpose of fine-grained recognition.

[0080] In one alternative embodiment of this disclosure, such as Figure 3 As shown, in Figure 1 Based on the illustrated embodiment, step 105 may include the following sub-steps:

[0081] Step 301: Obtain the audio weight matrix corresponding to the audio representation matrix and the text weight matrix corresponding to the text representation matrix composed of the text features.

[0082] The audio weight matrix and text weight matrix can be preset or calculated. The audio weight matrix can be calculated based on the text representation matrix, and vice versa, to further improve the fusion of the audio and text modalities. It is understandable that the dimensions of the audio and text weight matrices are related to the total number of characters in the audio data.

[0083] As an optional implementation, the audio weight matrix can be determined based on the product of a preset initial audio weight matrix and a text representation matrix composed of text features, and the text weight matrix can be determined based on the product of a preset initial text weight matrix and an audio representation matrix. This allows for the adjustment of weights through a combination of text and audio, improving the fusion degree of text and audio modalities.

[0084] The initial audio weight matrix and initial text weight matrix are preset. The dimension of the initial audio weight matrix is ​​related to the dimension of the text representation matrix, and the dimension of the initial text weight matrix is ​​related to the dimension of the audio representation matrix. For example, assuming the dimension of the text representation matrix is ​​t*m, where t represents the number of rows (which can be a preset value) and m represents the number of columns (the total number of characters in the audio data), then the dimension of the initial audio weight matrix can be set to q*t, and the audio weight matrix = initial audio weight matrix * text representation matrix, so the dimension of the audio weight matrix is ​​q*m. Similarly, assuming the dimension of the audio representation matrix is ​​p*m, where p represents the number of rows (which can be a preset value) and m represents the number of columns (the total number of characters in the audio data), then the dimension of the initial text weight matrix can be set to q*p, and the text weight matrix = initial text weight matrix * audio representation matrix, so the dimension of the text weight matrix is ​​q*m. Here, t, m, p, and q are all positive integers. Each element in the audio weight matrix and text weight matrix belongs to [0,1], representing the weight of each feature.

[0085] Step 302: Based on the audio weight matrix and the audio representation matrix, determine the weighted audio representation matrix corresponding to the audio data.

[0086] In this embodiment of the disclosure, after determining the audio weight matrix, the weighted audio representation matrix corresponding to the audio data can be determined based on the audio weight matrix and the audio representation matrix.

[0087] For example, the audio weight matrix has a dimension of 1*m and the audio representation matrix has a dimension of t*m. The audio weight matrix can be multiplied digit by digit by each row of the audio representation matrix to obtain the weighted audio representation matrix.

[0088] For example, both the audio weight matrix and the audio representation matrix have a dimension of t*m. The audio weight matrix and the audio representation matrix can be multiplied digit by digit to obtain the weighted audio representation matrix.

[0089] Step 303: Based on the text weight matrix and the text representation matrix, determine the weighted text representation matrix corresponding to the audio data.

[0090] In this embodiment of the disclosure, after determining the text weight matrix, the weighted text representation matrix corresponding to the audio data can be determined based on the text weight matrix and the text representation matrix.

[0091] For example, the text weight matrix has a dimension of 1*m and the text representation matrix has a dimension of t*m. The text weight matrix can be multiplied digit by digit by each row of the text representation matrix to obtain the weighted text representation matrix.

[0092] For example, both the text weight matrix and the text representation matrix have a dimension of t*m. The text weight matrix and the text representation matrix can be multiplied digit by digit to obtain the weighted text representation matrix.

[0093] Step 304: Determine the multimodal representation vector corresponding to the audio data based on the weighted audio representation matrix and the weighted text representation matrix.

[0094] In this embodiment of the disclosure, after determining the weighted audio representation matrix and the weighted text representation matrix corresponding to the audio data, the multimodal representation vector corresponding to the audio data can be determined based on the weighted audio representation matrix and the weighted text representation matrix.

[0095] For example, when determining the multimodal representation vector, the weighted audio representation vector corresponding to the audio data can be determined first based on the weighted audio representation matrix, and the weighted text representation vector corresponding to the audio data can be determined based on the weighted text representation matrix. For instance, the maximum or mean value of the same row in the weighted audio representation matrix can be used as the element value of the corresponding row in the weighted audio representation vector, and the maximum or mean value of the same row in the weighted text representation matrix can be used as the element value of the corresponding row in the weighted text representation vector, thus obtaining the weighted audio representation vector and the weighted text representation vector. Next, the weighted audio representation vector and the weighted text representation vector can be concatenated to obtain the multimodal representation vector corresponding to the audio data; alternatively, the weighted audio representation vector and the weighted text representation vector can be added bit by bit to obtain the multimodal representation vector corresponding to the audio data.

[0096] For example, when determining the multimodal representation vector, the weighted audio representation matrix and the weighted text representation matrix can be concatenated first to obtain the multimodal representation matrix corresponding to the audio data. Each column vector in the multimodal representation matrix is ​​obtained by concatenating the column vectors of the corresponding columns in the weighted audio representation matrix and the weighted text representation matrix. Then, feature fusion is performed on the multimodal representation matrix based on a preset sequence model to obtain a fused multimodal representation matrix. Finally, the multimodal representation vector corresponding to the audio data is determined based on the fused multimodal representation matrix.

[0097] It is understandable that each column vector of the weighted audio representation matrix represents the weighted audio feature corresponding to a single character in the audio data, and each column vector of the weighted text representation matrix represents the weighted text feature corresponding to a single character in the audio data. Therefore, each column vector of the multimodal representation matrix obtained by concatenating the weighted audio and weighted text representation matrices represents the multimodal feature corresponding to a single character in the audio data. When performing feature fusion on the multimodal representation matrix using a pre-defined sequence model, each column of the multimodal features in the multimodal representation matrix is ​​fused separately to obtain a fused multimodal representation matrix. Each column of the fused multimodal representation matrix represents the sequence-enhanced multimodal feature corresponding to a single character in the audio data.

[0098] The preset sequence model can be, but is not limited to, any of the following models: Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Recurrent Neural Network (RNN).

[0099] Taking the bidirectional LSTM sequence model as an example, the bidirectional LSTM can be used to fuse each column of multimodal features in the multimodal representation matrix multiple times, and the two outputs of the bidirectional LSTM can be concatenated to obtain the sequence-enhanced multimodal features corresponding to each column of multimodal features in the multimodal representation matrix. Each column of sequence-enhanced multimodal features constitutes the fused multimodal representation matrix corresponding to the audio data.

[0100] Different methods can be used to determine the multimodal representation vector corresponding to audio data based on the fused multimodal representation matrix.

[0101] As an optional implementation, each row of the fused multimodal representation matrix can be traversed, and max pooling can be performed on the currently traversed row to obtain the maximum value of the currently traversed row. Upon completion of the traversal, the multimodal representation vector corresponding to the audio data is obtained, where the multimodal representation vector is composed of the maximum value of each determined row. In other words, the maximum value among all data in each row of the fused multimodal representation matrix is ​​determined as the element value of the corresponding row in the multimodal representation vector. After traversing all rows of the fused multimodal representation matrix, the multimodal representation vector corresponding to the audio data is obtained.

[0102] As an optional implementation, each row in the multimodal representation matrix can be traversed, and mean pooling can be performed on the currently traversed row to obtain the average value of the currently traversed row. In response to the end of the traversal, the multimodal representation vector corresponding to the audio data can be obtained, wherein the multimodal representation vector is composed of the average value of each determined row.

[0103] It is understood that the methods for determining the multimodal representation vector corresponding to audio data are not limited to the two implementation methods described in the above examples, and other methods that can determine the multimodal representation vector should also be included in this disclosure.

[0104] The speech emotion recognition method of this disclosure obtains the audio weight matrix corresponding to the audio representation matrix and the text weight matrix corresponding to the text representation matrix composed of text features. Based on the audio weight matrix and the audio representation matrix, it determines the weighted audio representation matrix corresponding to the audio data, and based on the text weight matrix and the text representation matrix, it determines the weighted text representation matrix corresponding to the audio data. Then, based on the weighted audio representation matrix and the weighted text representation matrix, it determines the multimodal representation vector corresponding to the audio data. Thus, it realizes the fusion of features from both audio and text modalities to represent audio data, which is beneficial to improving the accuracy of emotion recognition.

[0105] Figure 4 A schematic diagram of the network structure of a speech emotion recognition model according to an exemplary embodiment of the present disclosure is shown. This speech emotion recognition model is used to implement the speech emotion recognition method described in the above embodiments. Figure 4 As shown, the speech emotion recognition model includes an audio coding network, a text coding network, an alignment network, a modality excitation network, and a prediction network. The audio coding network includes an audio feature extraction subunit and an audio feature fusion subunit. The audio feature extraction subunit extracts features from each audio frame to obtain the corresponding audio features. The audio feature fusion subunit can be a fully connected layer, used to perform simple fusion of the audio features extracted by the audio feature extraction subunit to obtain the original audio representation X = [x1, x2, ..., x...]. n ], where n represents the audio data being divided into n audio frames, xi (i = 1 to n) represents the audio features of the i-th audio frame. The text encoding network includes a text encoding subunit and a text feature fusion subunit. The text encoding subunit converts each character in the text data "how / are / you / today" into vector form, obtaining the text features corresponding to each character. The text feature fusion subunit can be multiple fully connected layers used to fuse each text feature, obtaining the text representation matrix Z = [Z1, Z2, ..., Zn]. m ], where m represents the number of characters in the text data, Z j (j=1~m) represents the text features of the j-th character.

[0106] Figure 4 In the alignment network, there is an audio-text alignment subunit, which is used to determine the alignment matrix A∈R based on the audio data and text data. n*m In the alignment matrix A, if the value in the i-th row and j-th column is 1, it indicates that the i-th audio frame belongs to the j-th word; otherwise, it does not. The alignment network is also used to determine the audio representation matrix Y = [Y1, Y2, ..., Y] corresponding to the audio data based on the alignment matrix and the original audio representation. m The process for determining the audio representation matrix can be found in the relevant description of the foregoing embodiments. The modal excitation network is used to apply the formula Ey = δ(Wy*Z), where Ey ∈ R. q*m The audio weight matrix Ey is calculated using the text representation matrix. Each element of the audio weight matrix Ey is... j (j=1~m) belongs to [0,1], and using the formula Ez=δ(Wz*Z), Ez∈R q*m The text weight matrix Ez is calculated using the audio representation matrix. Each element Ez in the text weight matrix Ez... j (j=1~m) belongs to [0,1], where Wy and Wz are the initial audio weight matrix and the initial text weight matrix, respectively. Although Figure 4 Although not shown in the figure, in this embodiment of the present disclosure, after determining the audio weight matrix Ey and the text weight matrix Ez, the audio weight matrix Ey is multiplied digit by digit by the audio representation matrix to obtain the weighted audio representation matrix, and the text weight matrix Ez is multiplied digit by digit by the text representation matrix to obtain the weighted text representation matrix.

[0107] from Figure 4As can be seen, in the prediction network part, the weighted audio representation matrix and the weighted text representation matrix obtained from the modality excitation network are first concatenated. During concatenation, the concatenation is performed according to corresponding characters. That is, the first column of the weighted audio representation matrix is ​​concatenated with the first column of the weighted text representation matrix, and the second column of the weighted audio representation matrix is ​​concatenated with the second column of the weighted text representation matrix. This yields the multimodal representation matrix G = [g1, g2, ..., g] composed of the multimodal features corresponding to each character. m ], where g j (j = 1 to m) is obtained by concatenating the j-th column of the weighted audio representation matrix and the j-th column of the weighted text representation matrix. Then, for g... j The algorithm utilizes a bidirectional LSTM model for multiple fusions, concatenating the two outputs to obtain sequence-enhanced multimodal features. The sequence-enhanced multimodal features corresponding to each word constitute the fused multimodal representation matrix of the audio data. The prediction network's max-pooling layer performs max-pooling on the fused multimodal representation matrix to obtain the multimodal representation vector corresponding to the audio data. Then, the fully connected layers of the prediction network perform sentiment recognition based on the multimodal representation vector, outputting the final sentiment prediction probability distribution.

[0108] This exemplary embodiment also provides a voice emotion recognition device. Figure 5 A schematic block diagram of a voice emotion recognition device according to an exemplary embodiment of the present disclosure is shown, such as Figure 5 As shown, the voice emotion recognition device 50 includes: a first acquisition module 510, a second acquisition module 520, a first determination module 530, a second determination module 540, a third determination module 550, and a recognition module 560.

[0109] The first acquisition module 510 is used to acquire the audio data to be identified and the text data corresponding to the audio data.

[0110] The second acquisition module 550 is used to acquire the audio features corresponding to each audio frame in the audio data, and the text features corresponding to each character in the text data;

[0111] The first determining module 530 is used to determine the alignment matrix corresponding to the audio data based on the audio data and the text data, wherein the element with a value of 1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame belongs to the j-th word, i takes the value from 1 to n, j takes the value from 1 to m, n represents the total number of audio frames, and m represents the total number of words contained in the audio data.

[0112] The second determining module 540 is used to determine the audio representation matrix corresponding to the audio data based on the alignment matrix and the audio features corresponding to each audio frame;

[0113] The third determining module 550 is used to determine the multimodal representation vector corresponding to the audio data based on the audio representation matrix and the text representation matrix composed of the text features.

[0114] The recognition module 560 is used to perform emotion recognition based on the multimodal representation vector to obtain the emotion recognition result.

[0115] Optionally, the second determining module 540 includes:

[0116] A traversal unit is used to traverse each word in the audio data;

[0117] The first determining unit is used to determine the target audio frame belonging to the currently traversed word based on the alignment matrix;

[0118] The second determining unit is used to determine the fused audio features corresponding to the currently traversed character based on the target audio features corresponding to the target audio frame.

[0119] The generation unit is used to generate an audio representation matrix corresponding to the audio data based on the fused audio features corresponding to each word in response to the end of the traversal.

[0120] Optionally, the second determining unit is further configured to:

[0121] When there is only one target audio frame, the target audio feature corresponding to the target audio frame is determined as the fused audio feature corresponding to the currently traversed word;

[0122] When there are multiple target audio frames, the average audio feature and the maximum audio feature of the multiple target audio features are determined based on the target audio features corresponding to the multiple target audio frames respectively.

[0123] The average audio feature and the maximum audio feature are concatenated to obtain the fused audio feature corresponding to the currently traversed character.

[0124] Optionally, the third determining module 550 includes:

[0125] The acquisition unit is used to acquire the audio weight matrix corresponding to the audio representation matrix and the text weight matrix corresponding to the text representation matrix composed of the text features.

[0126] The first weighted calculation unit is used to determine the weighted audio representation matrix corresponding to the audio data based on the audio weight matrix and the audio representation matrix;

[0127] The second weighted calculation unit is used to determine the weighted text representation matrix corresponding to the audio data based on the text weight matrix and the text representation matrix.

[0128] The sentence representation determination unit is used to determine the multimodal representation vector corresponding to the audio data based on the weighted audio representation matrix and the weighted text representation matrix.

[0129] Optionally, the acquisition unit is further configured to:

[0130] The audio weight matrix is ​​determined by multiplying the preset initial audio weight matrix and the text representation matrix composed of the text features.

[0131] The text weight matrix is ​​determined based on the product of a preset initial text weight matrix and the audio representation matrix.

[0132] Optionally, the sentence, representing a defining unit, is further used for:

[0133] The weighted audio representation matrix and the weighted text representation matrix are concatenated to obtain the multimodal representation matrix corresponding to the audio data. Each column vector in the multimodal representation matrix is ​​obtained by concatenating the column vectors of the corresponding columns in the weighted audio representation matrix and the weighted text representation matrix.

[0134] Based on a preset sequence model, feature fusion is performed on the multimodal representation matrix to obtain a fused multimodal representation matrix;

[0135] Based on the fused multimodal representation matrix, the multimodal representation vector corresponding to the audio data is determined.

[0136] Optionally, the sentence, representing a defining unit, is further used for:

[0137] Iterate through each row of the fused multimodal representation matrix;

[0138] Perform max pooling on the currently traversed row to obtain the maximum value of the currently traversed row;

[0139] In response to the end of the traversal, a multimodal representation vector corresponding to the audio data is obtained, wherein the multimodal representation vector is composed of the maximum value of each determined row.

[0140] The voice emotion recognition device provided in this disclosure can execute any voice emotion recognition method applicable to electronic devices provided in this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in the device embodiments of this disclosure can be referred to the description in any method embodiment of this disclosure.

[0141] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program, when executed by the at least one processor, causing the electronic device to perform a voice emotion recognition method according to embodiments of this disclosure.

[0142] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a speech emotion recognition method according to embodiments of this disclosure.

[0143] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a voice emotion recognition method according to embodiments of this disclosure.

[0144] refer to Figure 6 The present invention describes a structural block diagram of an electronic device 1100 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0145] like Figure 6 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0146] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, output unit 1107, storage unit 1108, and communication unit 1109. Input unit 1106 can be any type of device capable of inputting information to electronic device 1100. Input unit 1106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 1107 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1108 may include, but is not limited to, disk and optical disk. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0147] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the voice emotion recognition method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the voice emotion recognition method by any other suitable means (e.g., by means of firmware).

[0148] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0150] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0152] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0153] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A speech emotion recognition method, wherein, The method includes: Obtain the audio data to be identified and the corresponding text data; Obtain the audio features corresponding to each audio frame in the audio data, and the text features corresponding to each character in the text data; Based on the audio data and the text data, an alignment matrix corresponding to the audio data is determined. The alignment matrix is ​​used to represent the correspondence between each character and each audio frame. The element with a value of 1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame belongs to the j-th character. The value of i is 1 to n, the value of j is 1 to m, n represents the total number of audio frames, and m represents the total number of characters contained in the audio data. Based on the alignment matrix and the audio features corresponding to each audio frame, the audio representation matrix corresponding to the audio data is determined; Based on the audio representation matrix and the text representation matrix composed of the text features, the multimodal representation vector corresponding to the audio data is determined, and one column of the text representation matrix is ​​the text feature corresponding to a character; Emotion recognition is performed based on the multimodal representation vectors to obtain the emotion recognition results.

2. The speech emotion recognition method as described in claim 1, wherein, The step of determining the audio representation matrix corresponding to the audio data based on the alignment matrix and the audio features corresponding to each audio frame includes: Iterate through each word in the audio data; Based on the alignment matrix, the target audio frame belonging to the currently traversed word is determined; Based on the target audio features corresponding to the target audio frame, determine the fused audio features corresponding to the currently traversed character; In response to the end of the traversal, an audio representation matrix corresponding to the audio data is generated based on the fused audio features corresponding to each character.

3. The speech emotion recognition method as described in claim 2, wherein, The step of determining the fused audio features corresponding to the currently traversed character based on the target audio features corresponding to the target audio frame includes: When there is only one target audio frame, the target audio feature corresponding to the target audio frame is determined as the fused audio feature corresponding to the currently traversed word; When there are multiple target audio frames, the average audio feature and the maximum audio feature of the multiple target audio features are determined based on the target audio features corresponding to the multiple target audio frames respectively. The average audio feature and the maximum audio feature are concatenated to obtain the fused audio feature corresponding to the currently traversed character.

4. The speech emotion recognition method as described in claim 1, wherein, The step of determining the multimodal representation vector corresponding to the audio data based on the audio representation matrix and the text representation matrix composed of the text features includes: Obtain the audio weight matrix corresponding to the audio representation matrix, and the text weight matrix corresponding to the text representation matrix composed of the text features; Based on the audio weight matrix and the audio representation matrix, determine the weighted audio representation matrix corresponding to the audio data; Based on the text weight matrix and the text representation matrix, determine the weighted text representation matrix corresponding to the audio data; Based on the weighted audio representation matrix and the weighted text representation matrix, the multimodal representation vector corresponding to the audio data is determined.

5. The speech emotion recognition method as described in claim 4, wherein, The step of obtaining the audio weight matrix corresponding to the audio representation matrix and the text weight matrix corresponding to the text representation matrix composed of the text features includes: The audio weight matrix is ​​determined by multiplying the preset initial audio weight matrix and the text representation matrix composed of the text features. The text weight matrix is ​​determined based on the product of a preset initial text weight matrix and the audio representation matrix.

6. The speech emotion recognition method as described in claim 4, wherein, The step of determining the multimodal representation vector corresponding to the audio data based on the weighted audio representation matrix and the weighted text representation matrix includes: The weighted audio representation matrix and the weighted text representation matrix are concatenated to obtain the multimodal representation matrix corresponding to the audio data. Each column vector in the multimodal representation matrix is ​​obtained by concatenating the column vectors of the corresponding columns in the weighted audio representation matrix and the weighted text representation matrix. Based on a preset sequence model, feature fusion is performed on the multimodal representation matrix to obtain a fused multimodal representation matrix; Based on the fused multimodal representation matrix, the multimodal representation vector corresponding to the audio data is determined.

7. The speech emotion recognition method as described in claim 6, wherein, The step of determining the multimodal representation vector corresponding to the audio data based on the fused multimodal representation matrix includes: Iterate through each row of the fused multimodal representation matrix; Perform max pooling on the currently traversed row to obtain the maximum value of the currently traversed row; In response to the end of the traversal, a multimodal representation vector corresponding to the audio data is obtained, wherein the multimodal representation vector is composed of the maximum value of each determined row.

8. A voice emotion recognition device, wherein, The device includes: The first acquisition module is used to acquire the audio data to be identified and the text data corresponding to the audio data; The second acquisition module is used to acquire the audio features corresponding to each audio frame in the audio data, and the text features corresponding to each character in the text data; The first determining module is used to determine the alignment matrix corresponding to the audio data based on the audio data and the text data. The alignment matrix is ​​used to represent the correspondence between each character and each audio frame. The element with a value of 1 in the i-th row and j-th column of the alignment matrix indicates that the i-th audio frame belongs to the j-th character. The value of i is 1 to n, the value of j is 1 to m, n represents the total number of audio frames, and m represents the total number of characters contained in the audio data. The second determining module is used to determine the audio representation matrix corresponding to the audio data based on the alignment matrix and the audio features corresponding to each audio frame. The third determining module is used to determine the multimodal representation vector corresponding to the audio data based on the audio representation matrix and the text representation matrix composed of the text features, wherein one column of the text representation matrix is ​​the text feature corresponding to a character. The recognition module is used to perform emotion recognition based on the multimodal representation vector to obtain the emotion recognition result.

9. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the speech emotion recognition method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the speech emotion recognition method according to any one of claims 1-7.

Images