Emotion recognition model training method and emotion recognition method, device, and storage medium

Abstract

The specification discloses an emotion recognition model training method and an emotion recognition method and device, and a storage medium. The first importance weight, the second importance weight and the fusion weight are used to determine a sample emotion result corresponding to a sample voice. A loss function is constructed by using a standard emotion result of the sample voice and the sample emotion result, and an emotion recognition model is trained by using the loss function, so that the emotion recognition model adjusts parameters until the emotion recognition model converges. The converged emotion recognition model accurately fuses the features of the audio, the features of the text and the features of the interaction between the audio and the text.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to an emotion recognition model training method, emotion recognition method, device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, accurately identifying customer emotions during communication is crucial for continuously optimizing customer experience across various transactions and improving transaction success rates. In various scenarios, such as ironic or noisy situations, relying solely on text or audio information to identify customer emotions is insufficient to accurately capture their true feelings, thus impacting the customer experience. Summary of the Invention

[0003] This specification provides an emotion recognition method, device, storage medium, and terminal, which can solve the technical problem that relying on unimodal information to recognize customer emotions may lead to inaccurate emotion recognition during the process of recognizing customer emotions. The technical solution is as follows:

[0004] Firstly, this specification provides a method for training an emotion recognition model, the method comprising:

[0005] Input the sample speech into the emotion recognition model to determine at least one audio feature vector and at least one text feature vector corresponding to the sample speech;

[0006] Based on the emotion recognition model, the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector are determined.

[0007] The sample emotion result corresponding to the sample speech is determined based on the first importance weight, the second importance weight, and the fusion weight. The emotion recognition model is trained based on the standard emotion result of the sample speech and the loss function constructed from the sample emotion result until the emotion recognition model converges.

[0008] Secondly, this specification provides an emotion recognition method, the method comprising:

[0009] Acquire the target speech and input the target speech into the emotion recognition model;

[0010] Based on the target emotion result of the target speech determined by the emotion recognition model, the emotion recognition result corresponding to the target speech is determined; wherein, the emotion recognition model is the emotion recognition model described in any of the above embodiments.

[0011] Thirdly, this specification provides an emotion recognition model training device, the device comprising:

[0012] The feature vector determination module is suitable for inputting sample speech into the emotion recognition model and determining at least one audio feature vector and at least one text feature vector corresponding to the sample speech.

[0013] The weight determination module is adapted to determine the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector based on the emotion recognition model.

[0014] The convergence module is adapted to determine the sample emotion result corresponding to the sample speech based on the first importance weight, the second importance weight and the fusion weight, and to train the emotion recognition model based on the standard emotion result of the sample speech and the loss function constructed by the sample emotion result, until the emotion recognition model converges.

[0015] Fourthly, this specification provides an emotion recognition device, the device comprising:

[0016] An acquisition module is adapted to acquire target speech and input the target speech into an emotion recognition model.

[0017] The recognition module is adapted to determine the emotion recognition result corresponding to the target speech based on the target emotion result of the target speech determined by the emotion recognition model; wherein the emotion recognition model is the emotion recognition model according to any one of claims 1 to 9.

[0018] Fifthly, this specification provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0019] Sixthly, a computer program product storing at least one instruction, said at least one instruction being loaded by a processor and executing the above-described method steps.

[0020] In a seventh aspect, this specification provides a terminal that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0021] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following: Based on an emotion recognition model, the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector are determined. Based on the first importance weight, the second importance weight, and the fusion weight, the sample emotion result corresponding to the sample speech is determined. Then, the emotion recognition model is trained using a loss function constructed from the standard emotion result of the sample speech and the sample emotion result, so that the emotion recognition model adjusts its parameters until it converges. The converged emotion recognition model accurately fuses the features of audio, text, and the features of audio and text interaction, enabling accurate identification of customer emotions. This solves the technical problem that relying on single-modal information to identify customer emotions may lead to inaccurate emotion recognition during the process of customer emotion recognition. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 An exemplary system architecture diagram of an emotion recognition model training method provided in the embodiments of this specification;

[0024] Figure 2 A flowchart illustrating an emotion recognition model training method provided in an embodiment of this specification;

[0025] Figure 3 A flowchart illustrating a method for determining feature vectors provided in an embodiment of this specification;

[0026] Figure 4 A schematic diagram of a process for determining an audio feature vector provided in an embodiment of this specification;

[0027] Figure 5 A flowchart illustrating a method for determining text feature vectors, provided for embodiments of this specification;

[0028] Figure 6 A schematic diagram illustrating a process for determining weights, provided as an embodiment of this specification;

[0029] Figure 7 A flowchart illustrating a method for determining the sentiment outcome of a sample, provided as an embodiment of this specification;

[0030] Figure 8A schematic diagram of another process for determining the sentiment result of a sample, provided in an embodiment of this specification;

[0031] Figure 9 A flowchart illustrating an emotion recognition method provided in an embodiment of this specification;

[0032] Figure 10 This is a schematic diagram illustrating the process of an emotion recognition model for recognizing target speech, as provided in an embodiment of this specification.

[0033] Figure 11 This is a schematic diagram of the structure of an emotion recognition model training device provided in the embodiments of this specification;

[0034] Figure 12 This is a schematic diagram of the structure of an emotion recognition device provided in an embodiment of this specification;

[0035] Figure 13 This is a schematic diagram of the structure of a terminal provided in an embodiment of this specification. Detailed Implementation

[0036] To make the features and advantages of the embodiments of this specification more apparent and understandable, the technical solutions of the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the embodiments of this specification.

[0037] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those described in this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments described in this specification as detailed in the appended claims.

[0038] With the rapid development of artificial intelligence, using AI to identify customer emotions to improve transaction success rates is a commonly used technique. In various scenarios, such as human-computer interaction, machines can identify user emotions to adjust communication strategies, improving user experience and thus increasing transaction success rates. Similarly, in financial management, accumulated text or audio communication data can be used to identify users' historical emotions, allowing for timely adjustments to communication strategies, enhancing service experience, and ultimately increasing transaction success rates.

[0039] In various scenarios, when using text information to identify customer emotions, it's difficult to recognize changes in tone of voice. This is because the same text information paired with different tones corresponds to different emotions, making accurate identification of customer emotions challenging. Similarly, when using audio information to identify customer emotions, it's easily affected by noise, leading to inaccurate identification. Relying solely on text or audio information for customer emotion identification fails to accurately capture the customer's true emotions, thus impacting customer experience.

[0040] Therefore, this specification provides an emotion recognition model training method and an emotion recognition method. First, the emotion recognition model is trained. Then, the emotion recognition model that has converged after training is used to recognize the emotion of the target speech, thereby determining the emotion recognition result of the target speech.

[0041] When training the emotion recognition model, sample speech is input into the model. The first importance weight for each audio feature vector, the second importance weight for each text feature vector, and the fusion weight for each audio and text feature vector are determined. Then, the first importance weight, second importance weight, and fusion weight are used to determine the sample emotion result. The emotion recognition model is then trained using a loss function constructed from the standard emotion result and the sample emotion result, allowing the model to adjust its parameters until convergence. The converged emotion recognition model accurately fuses audio features, text features, and features from audio and text interaction, enabling accurate identification of customer emotions. This solves the technical problem that relying on single-modal information for customer emotion recognition may lead to inaccurate emotion recognition.

[0042] Please see Figure 1 , Figure 1 This is an exemplary system architecture diagram of an emotion recognition model training method provided in the embodiments of this specification.

[0043] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0044] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartwatches, smartphones, tablets, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0045] In the embodiments of this specification, firstly, terminal 101 can input sample speech into the emotion recognition model to determine at least one audio feature vector and at least one text feature vector corresponding to the sample speech; then, terminal 101 determines the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector based on the emotion recognition model; finally, terminal 101 determines the sample emotion result corresponding to the sample speech based on the first importance weight, the second importance weight, and the fusion weight, and trains the emotion recognition model using a loss function constructed based on the standard emotion result of the sample speech and the sample emotion result, until the emotion recognition model converges.

[0046] Server 103 can be an integrated server providing various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0047] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification can be applied to a system structure that only includes terminal 101. The embodiments of this specification do not limit this.

[0048] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.

[0049] Please see Figure 2 , Figure 2This is a flowchart illustrating an emotion recognition model training method provided in an embodiment of this specification. The execution entity in this embodiment can be a terminal executing the emotion recognition model training method, a processor within the terminal executing the emotion recognition model training method, or an emotion recognition model training service within the terminal executing the emotion recognition model training method. For ease of description, the following example uses a processor within a terminal as the execution entity to describe the specific execution process of the emotion recognition model training method.

[0050] like Figure 2 As shown, the training methods for emotion recognition models can include at least:

[0051] S202: Input the sample speech into the emotion recognition model and determine at least one audio feature vector and at least one text feature vector corresponding to the sample speech.

[0052] Sample speech can be obtained from a sample speech database, and each sample speech has its corresponding standard emotion result. To improve the accuracy of the emotion recognition model in identifying user emotions, the model is first trained using sample speech. The sample speech is input into the emotion recognition model, which processes it to extract audio data and corresponding text data; the text data can be obtained by recognizing the sample speech. Then, based on the audio and text data, at least one audio feature vector and at least one text feature vector corresponding to the sample speech are determined.

[0053] Here, an audio feature vector can be a carrier of audio features corresponding to a character, word, or sentence; similarly, a text feature vector can be a carrier of text features corresponding to a character, word, or sentence. Audio features can include at least temporal and frequency domain information. The duration of the sample speech can vary. Each character, word, or sentence can correspond to one audio feature, and each character, word, or sentence can also correspond to one text feature; this is not limited here. Text features can include at least semantic information. Therefore, there can be one or more audio feature vectors corresponding to a sample speech, and there can also be one or more text feature vectors corresponding to a sample speech.

[0054] In some feasible implementations, sample speech and its corresponding text can be input into the emotion recognition model, avoiding the emotion recognition model from performing text recognition on the sample speech, which can effectively shorten the training cycle of the emotion recognition model.

[0055] Please see Figure 3 , Figure 3 This is a flowchart illustrating a method for determining feature vectors provided in an embodiment of this specification. Figure 3As shown, in one embodiment provided in this specification, step S202, determining at least one audio feature vector and at least one text feature vector corresponding to the sample speech, includes:

[0056] S302: Preprocess the sample speech to determine at least one audio preprocessing feature vector and at least one text preprocessing feature vector corresponding to the sample speech.

[0057] The preprocessing of the sample speech includes audio preprocessing and text preprocessing; audio preprocessing is used to extract the feature representation of the audio corresponding to the sample speech, and text preprocessing is used to extract the feature representation of the text corresponding to the sample speech. The feature representation of the text is generally a dynamic feature representation.

[0058] In audio preprocessing, the audio corresponding to the sample speech is first segmented. Segmentation methods include, but are not limited to, equal-length segmentation, segmentation of the audio corresponding to each sentence, or segmentation using a segmentation model. A Hanning window is applied to each segmented audio file, and a Fourier transform is performed to obtain the spectrogram of each segment. Then, the spectrogram of each segment is input into a Convolutional Neural Network (CNN) to extract the audio preprocessing feature vector for each segment. Each segment can have one or more audio preprocessing feature vectors. Here, the CNN has the ability to learn representations. Furthermore, the CNN can be replaced by a convolutional recurrent neural network structure.

[0059] In text preprocessing, after obtaining the text corresponding to the sample speech, the text is input into the language representation model within the emotion recognition model to extract the text's feature representation, thereby obtaining at least one preprocessed text feature vector. This feature representation includes at least the semantic features of the text. The language representation model can be a BERT (Bidirectional Encoder Representations from Transformer) module. Alternatively, it can be other language representation models, such as an emotion dictionary, a large pre-trained model, or an LSTM (Long Short-Term Memory) network.

[0060] S304: The context information extraction module determines the audio feature vectors corresponding to each audio preprocessing feature vector and the context information extraction module determines the text feature vectors corresponding to each text preprocessing feature vector.

[0061] The context information extraction module can incorporate corresponding contextual audio information into each audio preprocessing feature vector. The same audio can have different meanings and represent different emotions in different contexts. For example, taking the audio corresponding to the exclamation word "ah" as an example, "ah" expresses a complex tone, which can convey praise, affirmation, reprimand, urging, questioning, etc., and also represents different emotions. Therefore, the contextual audio information corresponding to this audio can be incorporated into the audio preprocessing feature vector corresponding to this audio to further determine the emotion type.

[0062] Similarly, the context information extraction module can also incorporate corresponding contextual text information into each text preprocessing feature vector. The same sentence, word, or character can have different meanings and express different emotions in different contexts. Therefore, by combining its corresponding contextual text information, the meaning can be accurately determined, thereby accurately identifying the emotion. For example, in communication with a client, if the client indicates that the solution is "acceptable," and the contextual text shows a predominance of agreement, it can be determined that the client's true meaning is affirmative; if the contextual text shows a predominance of negativity, it can be determined that the client's true meaning is at least disinterested. Therefore, by incorporating corresponding contextual text information into each text preprocessing feature vector through the context information extraction module, a text feature vector representing the client's true meaning can be obtained.

[0063] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating a process for determining audio feature vectors, provided as an embodiment of this specification. Figure 4 As shown, in one embodiment provided in this specification, step S304, which involves determining the audio feature vectors corresponding to each audio preprocessing feature vector based on the context information extraction module, includes:

[0064] S402: Determine the first target time corresponding to each audio preprocessing feature vector, sort each audio preprocessing feature vector based on each first target time, and obtain each audio preprocessing feature vector sorted according to the first sorting.

[0065] Each audio preprocessing feature vector can be a feature representation of the corresponding character, word, or sentence audio. In the sample speech, each character, word, or sentence audio has its own corresponding occurrence time. The occurrence time of the corresponding character, word, or sentence audio can be used as the first target time of the corresponding audio preprocessing feature vector. Then, the audio preprocessing feature vectors are sorted according to their respective first target times. This can be done in a forward order, for example, 1 minute 11 seconds, 1 minute 13 seconds, and 1 minute 15 seconds; or in a reverse order, for example, 1 minute 15 seconds, 1 minute 13 seconds, and 1 minute 11 seconds. Generally, for ease of subsequent processing, the audio preprocessing feature vectors can be sorted in a forward order according to their respective first target times, resulting in audio preprocessing feature vectors sorted according to the first order.

[0066] S404: The first forward propagation submodule and the first backward propagation submodule in the context information extraction module each process the audio preprocessing feature vectors sorted according to the first sorting, and determine the forward audio feature vector and the backward audio feature vector corresponding to each audio preprocessing feature vector.

[0067] The context information extraction module contains multiple first forward pass sub-modules. When the audio preprocessing feature vectors, sorted according to a first order, are input into the context information extraction module, each audio preprocessing feature vector is assigned to the sequentially connected first forward pass sub-modules according to the first order. Each first forward pass sub-module passes the hidden layer output features to the next first forward pass sub-module in the same direction as the first order of its respective audio preprocessing feature vectors. The hidden layer output features represent the contextual audio information of the audio preprocessing feature vectors in the next first forward pass sub-module. Simultaneously, each first forward pass sub-module receives the hidden layer output features passed from the previous first forward pass sub-module and outputs a positive audio feature vector corresponding to the audio feature representation that incorporates the contextual audio information. It is easy to understand that the first first forward pass sub-module has no previous first forward pass sub-module, and the last first forward pass sub-module has no next first forward pass sub-module.

[0068] Similarly, the context information extraction module has multiple first backward propagation sub-modules. When the audio preprocessing feature vectors, sorted according to the first order, are input into the context information extraction module, each audio preprocessing feature vector is assigned to the sequentially connected first backward propagation sub-modules in the reverse order of the first order. Each first backward propagation sub-module passes the hidden layer output features to the next first backward propagation sub-module in the opposite direction to the first order of its respective audio preprocessing feature vectors. The hidden layer output features represent the contextual audio information of the audio preprocessing feature vectors in the next first backward propagation sub-module. Simultaneously, each first backward propagation sub-module receives the hidden layer output features passed from the previous first backward propagation sub-module and outputs the inverse audio feature vector corresponding to the audio feature representation fused with the contextual audio information. Here, the first first backward propagation sub-module has no previous first backward propagation sub-module, and the last first backward propagation sub-module has no next first backward propagation sub-module.

[0069] S406: Extract the context features of each audio preprocessing feature vector based on the forward and reverse audio feature vectors corresponding to each audio preprocessing feature vector to determine the audio feature vector corresponding to each audio preprocessing feature vector.

[0070] The forward audio feature vector corresponding to each audio preprocessing feature vector is fused with the preceding audio information, and the reverse audio feature vector corresponding to each audio preprocessing feature vector is fused with the following audio information. Based on the forward and reverse audio feature vectors corresponding to each audio preprocessing feature vector, the audio feature representation is fused with the corresponding preceding and following audio information to extract the context features of each audio preprocessing feature vector, thereby determining the audio feature vector corresponding to each audio preprocessing feature vector.

[0071] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating a process for determining text feature vectors, provided as an embodiment of this specification. Figure 5 As shown, in one embodiment provided in this specification, step S304, which determines the text feature vectors corresponding to each text preprocessing feature vector based on the context information extraction module, includes:

[0072] S502: Determine the second target time corresponding to each text preprocessing feature vector, sort each audio preprocessing feature vector based on each second target time, and obtain each text preprocessing feature vector sorted according to the second sorting.

[0073] Each text preprocessing feature vector can be a feature representation of the corresponding character, word, or sentence text. In the sample speech, the speech corresponding to each character, word, or sentence text has its own corresponding occurrence time. The occurrence time of the speech corresponding to the corresponding character, word, or sentence text can be used as the second target time of the corresponding text preprocessing feature vector. It should be noted that there is no strict correspondence between each text preprocessing feature vector and each audio preprocessing feature vector. That is, the two processes of obtaining the text preprocessing feature vector and each audio preprocessing feature vector are independent of each other to obtain accurate single-modal features.

[0074] The preprocessed feature vectors of each text are sorted according to their corresponding second target time. The sorting can be done in forward order (e.g., 2 minutes 13 seconds, 2 minutes 17 seconds, and 2 minutes 19 seconds) or in reverse order (e.g., 2 minutes 19 seconds, 2 minutes 17 seconds, and 2 minutes 13 seconds). Generally, for ease of subsequent processing, the preprocessed feature vectors can be sorted in forward order according to their corresponding second target time, resulting in preprocessed feature vectors sorted according to the second target time.

[0075] S504: The second forward propagation submodule and the second backward propagation submodule in the context information extraction module each process the text preprocessing feature vectors sorted according to the second sorting, and determine the positive text feature vector and the negative text feature vector corresponding to each text preprocessing feature vector.

[0076] The context information extraction module has multiple second forward propagation sub-modules. When each text preprocessing feature vector, which is sorted according to the second sort, is input into the context information extraction module, each text preprocessing feature vector is assigned to the sequentially connected second forward propagation sub-modules according to the second sort.

[0077] Each second forward pass submodule passes the hidden layer output features to the next second forward pass submodule in the same direction as the second sorting direction of its respective text preprocessing feature vectors. The hidden layer output features represent the preceding text information of the text preprocessing feature vectors in the next second forward pass submodule.

[0078] Simultaneously, each second forward propagation submodule receives the features output from the hidden layer passed by the previous second forward propagation submodule and outputs a positive text feature vector corresponding to the text feature representation that incorporates the preceding text information. The first second forward propagation submodule has no preceding second forward propagation submodule, and the last second forward propagation submodule has no following second forward propagation submodule.

[0079] Similarly, the number of second backward propagation sub-modules in the context information extraction module is multiple. When each text preprocessing feature vector, which is sorted according to the second sort, is input into the context information extraction module, each text preprocessing feature vector is assigned to the sequentially connected second backward propagation sub-modules in the reverse order of the second sort.

[0080] Each second backward propagation submodule passes the hidden layer output features to the next second backward propagation submodule in the opposite direction to the second sorting direction of its respective text preprocessing feature vectors. The hidden layer output features represent the audio information following the text preprocessing feature vectors in the next second backward propagation submodule.

[0081] Simultaneously, each second backward propagation submodule receives the features output from the hidden layer passed by the previous second backward propagation submodule and outputs the inverse text feature vector corresponding to the audio feature representation that incorporates the following text information. Here, the first second backward propagation submodule has no previous second backward propagation submodule, and the last second backward propagation submodule has no next second backward propagation submodule.

[0082] S506: Extract the context features of each text preprocessing feature vector based on the forward and reverse text feature vectors corresponding to each text preprocessing feature vector to determine the text feature vector corresponding to each text preprocessing feature vector.

[0083] Each text preprocessing feature vector's forward text feature vector incorporates its preceding text information, while each text preprocessing feature vector's reverse text feature vector incorporates its following text information. Based on these forward and reverse text feature vectors, each text feature representation can be fused with its corresponding preceding and following text information, thereby extracting the context features of each text preprocessing feature vector to determine the corresponding text feature vector.

[0084] S204: Based on the emotion recognition model, determine the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector.

[0085] To fuse the various audio feature vectors, their importance can be considered. Therefore, the first importance weight for each audio feature vector can be determined based on the emotion recognition model. Similarly, to fuse the various text feature vectors, their importance can be considered. Furthermore, to fuse both audio and text feature vectors, fusion weights can be determined for each audio and text feature vector based on their complementary relationships. Specifically, the sum of the first importance weights for each audio feature vector can be 1; the sum of the second importance weights for each text feature vector can be 1; and the sum of the fusion weights for both audio and text feature vectors can be 1.

[0086] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating a process for determining weights, provided as an embodiment of this specification. Figure 6 As shown, step S204, based on the emotion recognition model, determines the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector, including:

[0087] S602: Determine the first importance level of each audio feature vector based on the attention module in the emotion recognition model, and determine the first importance weight of each audio feature vector based on the first importance level of each audio feature vector.

[0088] Each audio feature vector represents at least the temporal and frequency domain information of the corresponding audio, and can also represent the contextual features of the corresponding audio. Therefore, the first importance level of each audio feature vector can be determined based on its temporal, frequency, and contextual information. This first importance level can be obtained through calculation, based on a mapping table, or through other feasible methods. After obtaining the first importance level of each audio feature vector, its corresponding weight can be determined. Then, the weights can be normalized to obtain the first importance weights for each audio feature vector. In one feasible implementation, the first importance weights can be determined based on the first multi-head attention model network in the attention module.

[0089] S604: Determine the second importance level corresponding to each text feature vector based on the attention module in the emotion recognition model, and determine the second importance weight corresponding to each text feature vector based on the second importance level corresponding to each text feature vector.

[0090] Each text feature vector can represent the semantic information of the corresponding text, as well as its contextual information. Therefore, the second importance level of each text feature vector can be determined based on its semantic and contextual information. This second importance level can be obtained through calculation, based on a mapping table, or through other feasible methods. Similarly, after obtaining the second importance level of each text feature vector, its corresponding weight can be determined, and then normalized to obtain the second importance weight for each text feature vector. In one feasible implementation, the second importance weights can be determined based on the second multi-head attention model network in the attention module.

[0091] S606: Determine the fusion importance level of each audio feature vector and each text feature vector based on the attention module in the emotion recognition model, and determine the fusion weight corresponding to each audio feature vector and each text feature vector based on the fusion importance level of each audio feature vector and each text feature vector.

[0092] Based on the interactions between audio feature vectors, text feature vectors, and the interaction between audio and text feature vectors, the fusion importance level of each audio and text feature vector is determined, thereby determining the corresponding fusion weights. The method for determining the fusion weights can refer to the first importance weight or second importance weight, which will not be elaborated here. In one feasible implementation, the fusion weights of each audio and text feature vector can be determined based on a multimodal interactive attention mechanism network.

[0093] S206: Determine the sample emotion result corresponding to the sample speech based on the first importance weight, the second importance weight, and the fusion weight, and train the emotion recognition model with the loss function constructed based on the standard emotion result of the sample speech and the sample emotion result until the emotion recognition model converges.

[0094] Specifically, the audio modal feature vector of the sample speech is determined based on each audio feature vector and its corresponding first importance weight; the text modal feature vector of the sample speech is determined based on each text feature vector and its corresponding second importance weight; and the fusion feature vector of the sample speech is determined based on each audio feature vector, each text feature vector, and their corresponding fusion weights. Then, the sample emotion result corresponding to the sample speech is determined based on the audio modal feature vector, the text modal feature vector, and the fusion feature vector. Finally, the loss function constructed using the standard emotion result and the sample emotion result is used for training and updating the emotion recognition model until the emotion recognition model converges.

[0095] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating a process for determining the sentiment outcome of a sample, provided as an embodiment of this specification. Figure 7 As shown, step S206 determines the sample emotion result corresponding to the sample speech based on the first importance weight, the second importance weight, and the fusion weight, including:

[0096] S702: Determine the audio modal feature vector of the sample speech based on the weighted average of each audio feature vector and its corresponding first importance weight.

[0097] One approach is to multiply each audio feature vector by its corresponding first importance weight and sum the results. Then, divide the resulting vector by the sum of the first importance weights. In one feasible implementation, when the sum of the first importance weights is 1, a weighted sum can be directly performed based on each audio feature vector and its corresponding first importance weight. Another feasible implementation is to determine the audio modality feature vector based on a first multi-head attention model network.

[0098] S704: Determine the text modal feature vector of the sample speech based on the weighted average of each text feature vector and its corresponding second importance weight.

[0099] One approach is to multiply each text feature vector by its corresponding second importance weight, sum the results, and then divide the resulting vector by the sum of the second importance weights. In one feasible implementation, when the sum of the second importance weights is 1, a weighted sum can be directly performed based on each text feature vector and its corresponding second importance weight. Another feasible implementation is to determine the text modality feature vector based on a second multi-head attention model network.

[0100] S706: Determine the fusion feature vector of the sample speech based on the weighted average of each audio feature vector and each text feature vector with the corresponding fusion weight.

[0101] One approach involves multiplying each audio feature vector by its corresponding fusion weight and each text feature vector by its corresponding fusion weight, summing the results, and then dividing the resulting vector by the sum of the fusion weights. In one feasible implementation, when the sum of the fusion weights is 1, a weighted sum can be directly performed based on each audio and text feature vector with its corresponding fusion weight. Another feasible implementation uses a multimodal interactive attention mechanism network to determine the fusion feature vector.

[0102] S708: Determine the sample emotion result corresponding to the sample speech based at least on the audio modality feature vector, the text modality feature vector, and the fused feature vector.

[0103] At least the audio modality feature vector, text modality feature vector, and fused feature vector are fused to obtain fused features. Finally, the sample emotion result corresponding to the sample speech is determined based on the fused features. The sample emotion result corresponding to the sample speech can be determined by judging the probability values ​​of various emotions corresponding to the sample speech. It is easy to understand that when the probability value of one emotion is the highest, that emotion can be regarded as the customer's true emotion corresponding to the sample speech.

[0104] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating another process for determining the sentiment outcome of a sample, provided as an embodiment of this specification. For example... Figure 8 As shown, step S708 determines the sample emotion result corresponding to the sample speech based at least on the audio modality feature vector, the text modality feature vector, and the fused feature vector, including:

[0105] S802: At least the audio modality feature vector, text modality feature vector, and fused feature vector are concatenated to obtain a concatenated feature vector.

[0106] Determine if the dimensions of the audio modality feature vector, text modality feature vector, and fused feature vector are the same. If they are different, a linear transformation can be used to convert them to the same dimension. Then, concatenate the audio modality feature vector, text modality feature vector, and fused feature vector that are in the same dimension to obtain the concatenated feature vector.

[0107] By concatenating audio modality feature vectors, text modality feature vectors, and fused feature vectors, the accuracy of speech emotion recognition is improved by fusing features corresponding to audio, features corresponding to text, and interactive features between audio and text.

[0108] In one embodiment provided in this specification, step S802 involves concatenating at least the audio modality feature vector, the text modality feature vector, and the fused feature vector to obtain a concatenated feature vector, including:

[0109] The audio modality feature vector, text modality feature vector, fused feature vector, each audio feature vector, and each text feature vector are concatenated to obtain the concatenated feature vector.

[0110] S804: Determine the fusion vector corresponding to the spliced ​​feature vector based on the fully connected fusion module in the emotion recognition model.

[0111] The concatenated feature vector is processed by the fully connected fusion module in the emotion recognition model to extract the bimodal interaction fusion information of audio and text, as well as the unimodal internal information of audio and text, thereby obtaining the fusion vector.

[0112] Based on the fusion vector, the audio feature vectors and text feature vectors fused with contextual audio information and contextual text information are learned again. This strengthens the correlation between important features within the audio and weakens the correlation between secondary features within the audio. Similarly, it strengthens the correlation between important features within the text and weakens the correlation between secondary features within the text. Furthermore, it also strengthens the correlation between important features between audio and text.

[0113] S806: Determine the sample emotion result corresponding to the fusion vector based on the emotion recognition module in the emotion recognition model.

[0114] The emotion recognition module in the emotion recognition model normalizes the fusion vector to determine the sample emotion result corresponding to the fusion vector. The emotion recognition module includes a normalization exponential function.

[0115] In one embodiment provided in this specification, step S206, which trains an emotion recognition model based on a loss function constructed from the standard emotion results and sample emotion results of the sample speech, until the emotion recognition model converges, includes:

[0116] The emotion recognition model is trained using a loss function constructed from standard sample emotion results and sample emotion results of sample speech. At least the first importance weight, the second importance weight, and the fusion weight are updated until the emotion recognition model converges.

[0117] A loss function is constructed using the standard sample emotion results and sample emotion results of the sample speech. The parameters in the emotion recognition model are updated using the results corresponding to the loss function, so as to update at least each first importance weight, each second importance weight, and each fusion weight, until the emotion recognition model converges.

[0118] This specification provides an embodiment of a method for training an emotion recognition model. Sample speech is input into the emotion recognition model. The first importance weight for each audio feature vector, the second importance weight for each text feature vector, and the fusion weight for each audio and text feature vector are determined. Then, the first importance weight, the second importance weight, and the fusion weight are used to determine the emotion result of the sample. The emotion recognition model is then trained using a loss function constructed from the standard emotion result of the sample speech and the sample emotion result, allowing the model to adjust its parameters until convergence. The converged emotion recognition model accurately fuses the unimodal features of audio and text, as well as the bimodal features of audio and text interaction, enabling accurate identification of customer emotions. This solves the technical problem that relying on unimodal information to identify customer emotions may lead to inaccurate emotion recognition.

[0119] Please see Figure 9 , Figure 9 This is a flowchart illustrating an emotion recognition method provided in an embodiment of this specification. Figure 9 As shown, emotion recognition methods can include at least:

[0120] S902: Acquire the target speech and input the target speech into the emotion recognition model.

[0121] Optionally, the target speech can be synchronous speech, that is, speech acquired synchronously from the customer. Optionally, the target speech can also be historically accumulated speech. The emotion recognition model used is the emotion recognition model in any embodiment of this specification.

[0122] S904: Based on the target emotion result of the target speech determined by the emotion recognition model, determine the emotion recognition result corresponding to the target speech.

[0123] Optionally, after inputting the target speech into the emotion recognition model, the model, based on the knowledge learned during training to determine the emotion corresponding to the speech, extracts unimodal features corresponding to both audio and text, as well as bimodal features of the audio-text interaction, and then fuses the unimodal features of audio and text with the bimodal features of the audio-text interaction to determine the target emotion result. The emotion recognition result corresponding to the target speech is then determined based on the probability values ​​corresponding to each emotion in the target emotion result.

[0124] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating the process of an emotion recognition model for recognizing target speech, as provided in an embodiment of this specification.

[0125] like Figure 10As shown, the emotion recognition model first determines the corresponding text data and audio data based on the target speech. The text data is input into the BERT model in the emotion recognition model to determine at least one text preprocessing feature vector corresponding to the text data, and the audio data is input into the CNN model in the emotion recognition model to determine at least one audio preprocessing feature vector corresponding to the audio data.

[0126] Subsequently, the first forward propagation submodule (module A in the figure) and the first backward propagation submodule (module B in the figure) in the context information extraction module each process the audio preprocessing feature vectors sorted according to the first sorting, and determine the forward audio feature vector and the backward audio feature vector corresponding to each audio preprocessing feature vector; based on the forward audio feature vector and the backward audio feature vector corresponding to each audio preprocessing feature vector, the context features of each audio preprocessing feature vector are extracted to determine the audio feature vector corresponding to each audio preprocessing feature vector.

[0127] Furthermore, the second forward propagation submodule (module C in the figure) and the second backward propagation submodule (module D in the figure) in the context information extraction module each process the text preprocessing feature vectors sorted according to the second sorting, and determine the positive text feature vector and the negative text feature vector corresponding to each text preprocessing feature vector; based on the positive text feature vector and the negative text feature vector corresponding to each text preprocessing feature vector, the context features of each text preprocessing feature vector are extracted to determine the text feature vector corresponding to each text preprocessing feature vector. The first forward propagation submodule, the first backward propagation submodule, the second forward propagation submodule, and the second backward propagation submodule can all include GRU (Gate Recurrent Unit) units.

[0128] Then, the first multi-head attention model network ( Figure 10 The Multi-Head Attention module determines the audio modality feature vector, based on the second multi-head attention model network ( Figure 10 The Multi-Head Attention module determines the text modal feature vectors, and the multimodal interaction attention mechanism network ( Figure 10 The Bi-Attention module in the emotion recognition model determines the fused feature vector. Finally, the audio modality feature vector, text modality feature vector, fused feature vector, individual audio feature vectors, and individual text feature vectors are concatenated to obtain the concatenated feature vector. This is then applied to the fully connected fusion module in the emotion recognition model. Figure 10 The Fully Connected module determines the fusion vector corresponding to the concatenated feature vectors, and the emotion recognition module in the emotion recognition model ( Figure 10The Softmax module determines the sentiment result corresponding to the fusion vector. Figure 10 a ai Representing the weights of each primary importance, a tai Represents the fusion weights and a ti This represents the weight of each of the second most important factors.

[0129] In the embodiments of this specification, an emotion recognition method is provided. By fusing the unimodal features corresponding to audio and text respectively, as well as the bimodal features of audio and text interaction, an emotion recognition model can be used to accurately identify customer emotions. This solves the technical problem that relying on unimodal information to identify customer emotions may lead to inaccurate emotion recognition.

[0130] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an emotion recognition model training device provided in an embodiment of this specification. Figure 11 As shown, the emotion recognition model training device 1100 includes:

[0131] The feature vector determination module 1110 is adapted to input sample speech into the emotion recognition model and determine at least one audio feature vector and at least one text feature vector corresponding to the sample speech.

[0132] The weight determination module 1120 is suitable for determining the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector based on the emotion recognition model.

[0133] The convergence module 1130 is suitable for determining the sample emotion result corresponding to the sample speech based on the first importance weight, the second importance weight and the fusion weight, and training the emotion recognition model with the loss function constructed based on the standard emotion result of the sample speech and the sample emotion result until the emotion recognition model converges.

[0134] Optionally, the feature vector determination module 1110 is also adapted to preprocess the sample speech to determine at least one audio preprocessing feature vector and at least one text preprocessing feature vector corresponding to the sample speech.

[0135] The context information extraction module determines the audio feature vectors corresponding to each audio preprocessing feature vector and the context information extraction module determines the text feature vectors corresponding to each text preprocessing feature vector.

[0136] Optionally, the feature vector determination module 1110 is further adapted to determine the first target time corresponding to each audio preprocessing feature vector, sort each audio preprocessing feature vector based on each first target time, and obtain each audio preprocessing feature vector sorted according to the first sorting.

[0137] Based on the first forward propagation submodule and the first backward propagation submodule in the context information extraction module, each of the audio preprocessing feature vectors sorted according to the first sorting is processed to determine the positive audio feature vector and the negative audio feature vector corresponding to each audio preprocessing feature vector.

[0138] The context features of each audio preprocessing feature vector are extracted based on the forward and reverse audio feature vectors corresponding to each audio preprocessing feature vector to determine the audio feature vector corresponding to each audio preprocessing feature vector.

[0139] Optionally, the feature vector determination module 1110 is also adapted to determine the second target time corresponding to each text preprocessing feature vector, and sort each audio preprocessing feature vector based on each second target time to obtain each text preprocessing feature vector sorted according to the second sorting.

[0140] Based on the second forward propagation submodule and the second backward propagation submodule in the context information extraction module, each preprocessed feature vector of the text sorted according to the second sorting is processed to determine the positive text feature vector and the negative text feature vector corresponding to each preprocessed feature vector of the text.

[0141] Based on the positive and negative text feature vectors corresponding to each text preprocessing feature vector, the context features of each text preprocessing feature vector are extracted to determine the text feature vector corresponding to each text preprocessing feature vector.

[0142] Optionally, the weight determination module 1120 is also adapted to determine the first importance level corresponding to each audio feature vector based on the attention module in the emotion recognition model, and to determine the first importance weight corresponding to each audio feature vector based on the first importance level corresponding to each audio feature vector.

[0143] The second importance level of each text feature vector is determined based on the attention module in the emotion recognition model, and the second importance weight of each text feature vector is determined based on the second importance level of each text feature vector.

[0144] The importance level of each audio feature vector and each text feature vector is determined by the attention module in the emotion recognition model, and the fusion weights corresponding to each audio feature vector and each text feature vector are determined based on the importance level of each audio feature vector and each text feature vector.

[0145] Optionally, the convergence module 1130 is also adapted to determine the audio modal feature vector of the sample speech based on the weighted average of each audio feature vector and its corresponding first importance weight.

[0146] The text modal feature vector of the sample speech is determined by the weighted average of each text feature vector and its corresponding second importance weight.

[0147] The fusion feature vector of the sample speech is determined by the weighted average of each audio feature vector and each text feature vector with the corresponding fusion weight.

[0148] The sample emotion result corresponding to the sample speech is determined based at least on the audio modality feature vector, the text modality feature vector, and the fused feature vector.

[0149] Optionally, the convergence module 1130 is also adapted to concatenate at least the audio modality feature vector, the text modality feature vector, and the fused feature vector to obtain a concatenated feature vector;

[0150] The fusion vector corresponding to the spliced ​​feature vector is determined based on the fully connected fusion module in the emotion recognition model;

[0151] The emotion result of the sample corresponding to the fusion vector is determined based on the emotion recognition module in the emotion recognition model.

[0152] Optionally, the convergence module 1130 is also adapted to concatenate the audio modality feature vector, the text modality feature vector, the fused feature vector, each audio feature vector, and each text feature vector to obtain a concatenated feature vector.

[0153] Optionally, the convergence module 1130 is also adapted to train the emotion recognition model based on the standard sample emotion results and the loss function constructed from the sample speech, and to update at least each first importance weight, each second importance weight, and each fusion weight until the emotion recognition model converges.

[0154] In the embodiments of this specification, an emotion recognition model training device is provided, wherein: a feature vector determination module is adapted to input sample speech into the emotion recognition model and determine at least one audio feature vector and at least one text feature vector corresponding to the sample speech; a weight determination module is adapted to determine a first importance weight corresponding to each audio feature vector, a second importance weight corresponding to each text feature vector, and a fusion weight corresponding to each audio feature vector and each text feature vector based on the emotion recognition model; and a convergence module is adapted to determine the sample emotion result corresponding to the sample speech based on the first importance weight, the second importance weight, and the fusion weight, and to train the emotion recognition model based on the standard emotion result of the sample speech and the loss function constructed from the sample emotion result, until the emotion recognition model converges.

[0155] The emotion recognition model training device provided in this specification uses a first importance weight, a second importance weight, and a fusion weight to determine the sample emotion result corresponding to the sample speech. Then, it trains the emotion recognition model using a loss function constructed from the standard emotion result of the sample speech and the sample emotion result, allowing the emotion recognition model to adjust its parameters until it converges. This enables the converged emotion recognition model to accurately fuse features from audio, text, and audio-text interaction, thereby achieving accurate identification of customer emotions. This solves the technical problem that relying on single-modal information to identify customer emotions can lead to inaccurate emotion recognition.

[0156] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an emotion recognition device provided in an embodiment of this specification. Figure 12 As shown, the emotion recognition device 1200 includes:

[0157] The acquisition module 1210 is adapted to acquire the target speech and input the target speech into the emotion recognition model;

[0158] The recognition module 1220 is adapted to determine the emotion recognition result corresponding to the target speech based on the target emotion result of the target speech determined by the emotion recognition model;

[0159] The emotion recognition model is any of the emotion recognition models in the above embodiments.

[0160] This specification provides an emotion recognition device, comprising an acquisition module adapted to acquire target speech and input the target speech into an emotion recognition model; and a recognition module adapted to determine the emotion recognition result corresponding to the target speech based on the target emotion result determined by the emotion recognition model. The emotion recognition model is any of the emotion recognition models described in the above embodiments. After training, the emotion recognition model can more accurately integrate the features of audio, text, and audio-text interaction, thereby achieving accurate recognition of customer emotions. This solves the technical problem that relying on single-modal information to recognize customer emotions may lead to inaccurate emotion recognition.

[0161] This specification provides a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of any of the methods described above.

[0162] This specification also provides a computer storage medium that can store multiple instructions adapted for loading by a processor and executing the steps of any of the methods described in the above embodiments.

[0163] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a terminal provided in an embodiment of this specification. Figure 13 As shown, terminal 1300 may include: at least one terminal processor 1301, at least one network interface 1304, user interface 1303, memory 1305, and at least one communication bus 1302.

[0164] The communication bus 1302 is used to realize the connection and communication between these components.

[0165] The user interface 1303 may include a display screen and a camera. Optionally, the user interface 1303 may also include a standard wired interface and a wireless interface.

[0166] The network interface 1304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0167] The terminal processor 1301 may include one or more processing cores. The terminal processor 1301 connects to various parts within the terminal 1300 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1305, and by calling data stored in the memory 1305. Optionally, the terminal processor 1301 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The terminal processor 1301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the terminal processor 1301.

[0168] The memory 1305 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1305 may include a non-transitory computer-readable storage medium. The memory 1305 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1305 may also be at least one storage device located remotely from the aforementioned terminal processor 1301. Figure 13 As shown, the memory 1305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an emotion recognition model training program and / or an emotion recognition program.

[0169] exist Figure 13 In the terminal 1300 shown, the user interface 1303 is mainly used to provide an input interface for the user and to obtain the user's input data; while the terminal processor 1301 can be used to call the emotion recognition model training program stored in the memory 1305 and specifically perform the following operations:

[0170] Input the sample speech into the emotion recognition model to determine at least one audio feature vector and at least one text feature vector corresponding to the sample speech;

[0171] Based on the emotion recognition model, the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector are determined.

[0172] The emotion result corresponding to the sample speech is determined based on the first importance weight, the second importance weight, and the fusion weight. The emotion recognition model is trained by constructing a loss function based on the standard emotion result of the sample speech and the sample emotion result until the emotion recognition model converges.

[0173] In some embodiments, when the terminal processor 1301 determines at least one audio feature vector and at least one text feature vector corresponding to the sample speech, it specifically performs the following steps:

[0174] Preprocess the sample speech to determine at least one audio preprocessing feature vector and at least one text preprocessing feature vector corresponding to the sample speech;

[0175] The context information extraction module determines the audio feature vectors corresponding to each audio preprocessing feature vector and the context information extraction module determines the text feature vectors corresponding to each text preprocessing feature vector.

[0176] In some embodiments, when the terminal processor 1301 executes the context information extraction module to determine the audio feature vectors corresponding to each audio preprocessing feature vector and the text feature vectors corresponding to each text preprocessing feature vector, it specifically performs the following steps:

[0177] Determine the first target time corresponding to each audio preprocessing feature vector, sort each audio preprocessing feature vector based on each first target time, and obtain each audio preprocessing feature vector sorted according to the first sorting.

[0178] Based on the first forward propagation submodule and the first backward propagation submodule in the context information extraction module, each of the audio preprocessing feature vectors sorted according to the first sorting is processed to determine the positive audio feature vector and the negative audio feature vector corresponding to each audio preprocessing feature vector.

[0179] The context features of each audio preprocessing feature vector are extracted based on the forward and reverse audio feature vectors corresponding to each audio preprocessing feature vector to determine the audio feature vector corresponding to each audio preprocessing feature vector.

[0180] In some embodiments, when the terminal processor 1301 determines the text feature vectors corresponding to each text preprocessing feature vector using the context information extraction module, it specifically performs the following steps:

[0181] Determine the second target time corresponding to each text preprocessing feature vector, sort each audio preprocessing feature vector based on each second target time, and obtain each text preprocessing feature vector sorted according to the second sorting.

[0182] Based on the second forward propagation submodule and the second backward propagation submodule in the context information extraction module, each preprocessed feature vector of the text sorted according to the second sorting is processed to determine the positive text feature vector and the negative text feature vector corresponding to each preprocessed feature vector of the text.

[0183] Based on the positive and negative text feature vectors corresponding to each text preprocessing feature vector, the context features of each text preprocessing feature vector are extracted to determine the text feature vector corresponding to each text preprocessing feature vector.

[0184] In some embodiments, when the terminal processor 1301 performs the following steps when determining the first importance weights corresponding to each audio feature vector, the second importance weights corresponding to each text feature vector, and the fusion weights corresponding to each audio feature vector and each text feature vector based on the emotion recognition model:

[0185] The first importance level of each audio feature vector is determined based on the attention module in the emotion recognition model, and the first importance weight of each audio feature vector is determined based on the first importance level of each audio feature vector.

[0186] The second importance level of each text feature vector is determined based on the attention module in the emotion recognition model, and the second importance weight of each text feature vector is determined based on the second importance level of each text feature vector.

[0187] The importance level of each audio feature vector and each text feature vector is determined by the attention module in the emotion recognition model, and the fusion weights corresponding to each audio feature vector and each text feature vector are determined based on the importance level of each audio feature vector and each text feature vector.

[0188] In some embodiments, when the terminal processor 1301 determines the sample emotion result corresponding to the sample speech based on the first importance weight, the second importance weight, and the fusion weight, it specifically performs the following steps:

[0189] The audio modal feature vector of the sample speech is determined by the weighted average of each audio feature vector and its corresponding first importance weight.

[0190] The text modal feature vector of the sample speech is determined by the weighted average of each text feature vector and its corresponding second importance weight.

[0191] The fusion feature vector of the sample speech is determined by the weighted average of each audio feature vector and each text feature vector with the corresponding fusion weight.

[0192] The sample emotion result corresponding to the sample speech is determined based at least on the audio modality feature vector, the text modality feature vector, and the fused feature vector.

[0193] In some embodiments, when the terminal processor 1301 determines the sample emotion result corresponding to the sample speech based at least on the audio modality feature vector, the text modality feature vector, and the fused feature vector, it specifically performs the following steps:

[0194] At least the audio modality feature vector, text modality feature vector, and fused feature vector are concatenated to obtain the concatenated feature vector;

[0195] The fusion vector corresponding to the spliced ​​feature vector is determined based on the fully connected fusion module in the emotion recognition model;

[0196] The emotion result of the sample corresponding to the fusion vector is determined based on the emotion recognition module in the emotion recognition model.

[0197] In some embodiments, when the terminal processor 1301 concatenates at least the audio modality feature vector, the text modality feature vector, and the fused feature vector to obtain a concatenated feature vector, it specifically performs the following steps:

[0198] The audio modality feature vector, text modality feature vector, fused feature vector, each audio feature vector, and each text feature vector are concatenated to obtain the concatenated feature vector.

[0199] In some embodiments, when the terminal processor 1301 trains an emotion recognition model using a loss function constructed from standard sample emotion results and sample emotion results based on sample speech, and the emotion recognition model converges, the following steps are specifically performed:

[0200] The emotion recognition model is trained using a loss function constructed from standard sample emotion results and sample emotion results of sample speech. At least the first importance weight, the second importance weight, and the fusion weight are updated until the emotion recognition model converges.

[0201] exist Figure 13 In the terminal 1300 shown, the user interface 1303 is mainly used to provide an input interface for the user and to obtain the user's input data; while the terminal processor 1301 can also be used to call the emotion recognition program stored in the memory 1305 and specifically perform the following operations:

[0202] Acquire the target speech and input the target speech into the emotion recognition model;

[0203] Based on the target emotion result of the target speech determined by the emotion recognition model, the corresponding emotion recognition result of the target speech is determined.

[0204] The emotion recognition model is any of the emotion recognition models included in the above embodiments.

[0205] In the several embodiments provided in this specification, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0206] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0207] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0208] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0209] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0210] The above is a description of an emotion recognition model training method, emotion recognition method, device, and storage medium provided in the embodiments of this specification. For those skilled in the art, based on the ideas of the embodiments of this specification, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this specification.

Claims

1. A method for training an emotion recognition model, the method comprising: Input the sample speech into the emotion recognition model to determine at least one audio feature vector and at least one text feature vector corresponding to the sample speech; Based on the emotion recognition model, the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector are determined. The audio modal feature vector of the sample speech is determined by a weighted average of each audio feature vector and its corresponding first importance weight. The text modal feature vector of the sample speech is determined by a weighted average of each text feature vector and its corresponding second importance weight. The fusion feature vector of the sample speech is determined by a weighted average of each audio feature vector and each text feature vector with the corresponding fusion weight. The sample emotion result corresponding to the sample speech is determined based at least on the audio modality feature vector, the text modality feature vector, and the fused feature vector; as well as The emotion recognition model is trained based on the standard emotion results of the sample speech and the loss function constructed from the sample emotion results until the emotion recognition model converges. The step of determining the sample emotion result corresponding to the sample speech based at least on the audio modality feature vector, the text modality feature vector, and the fused feature vector includes: At least the audio modality feature vector, the text modality feature vector, and the fused feature vector are concatenated to obtain a concatenated feature vector; The fusion vector corresponding to the spliced ​​feature vector is determined based on the fully connected fusion module in the emotion recognition model. The emotion result of the sample corresponding to the fusion vector is determined based on the emotion recognition module in the emotion recognition model.

2. The method as described in claim 1, wherein determining at least one audio feature vector and at least one text feature vector corresponding to the sample speech includes: The sample speech is preprocessed to determine at least one audio preprocessing feature vector and at least one text preprocessing feature vector corresponding to the sample speech; The context information extraction module determines the audio feature vectors corresponding to each audio preprocessing feature vector and the context information extraction module determines the text feature vectors corresponding to each text preprocessing feature vector.

3. The method as described in claim 2, wherein the context information extraction module determines the audio feature vector corresponding to each audio preprocessing feature vector and the text feature vector corresponding to each text preprocessing feature vector, comprising: Determine the first target time corresponding to each audio preprocessing feature vector, sort each audio preprocessing feature vector based on each first target time, and obtain each audio preprocessing feature vector sorted according to the first sorting. Based on the first forward propagation submodule and the first backward propagation submodule in the context information extraction module, each of the audio preprocessing feature vectors sorted according to the first sorting is processed to determine the positive audio feature vector and the negative audio feature vector corresponding to each audio preprocessing feature vector. The context features of each audio preprocessing feature vector are extracted based on the forward and reverse audio feature vectors corresponding to each audio preprocessing feature vector to determine the audio feature vector corresponding to each audio preprocessing feature vector.

4. The method as described in claim 3, wherein the step of determining the text feature vector corresponding to each text preprocessing feature vector based on the context information extraction module includes: Determine the second target time corresponding to each text preprocessing feature vector, sort each audio preprocessing feature vector based on each second target time, and obtain each text preprocessing feature vector sorted according to the second sorting. Based on the second forward propagation submodule and the second backward propagation submodule in the context information extraction module, each preprocessed text feature vector sorted according to the second sorting is processed to determine the positive text feature vector and the negative text feature vector corresponding to each preprocessed text feature vector. Based on the positive and negative text feature vectors corresponding to each text preprocessing feature vector, the context features of each text preprocessing feature vector are extracted to determine the text feature vector corresponding to each text preprocessing feature vector.

5. The method as described in claim 1, wherein determining the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector based on the emotion recognition model includes: The first importance level of each audio feature vector is determined based on the attention module in the emotion recognition model, and the first importance weight of each audio feature vector is determined based on the first importance level of each audio feature vector. The second importance level of each text feature vector is determined based on the attention module in the emotion recognition model, and the second importance weight of each text feature vector is determined based on the second importance level of each text feature vector. The importance level of each audio feature vector and each text feature vector is determined based on the attention module in the emotion recognition model, and the fusion weight corresponding to each audio feature vector and each text feature vector is determined based on the importance level of each audio feature vector and each text feature vector.

6. The method of claim 1, wherein concatenating at least the audio modality feature vector, the text modality feature vector, and the fused feature vector to obtain a concatenated feature vector comprises: The audio modality feature vector, the text modality feature vector, the fused feature vector, each audio feature vector, and each text feature vector are concatenated to obtain the concatenated feature vector.

7. The method of claim 1, wherein training the emotion recognition model using a loss function constructed based on the standard sample emotion results and the sample emotion results of the sample speech until the emotion recognition model converges includes: The emotion recognition model is trained using a loss function constructed based on the standard sample emotion results and sample emotion results of the sample speech, updating at least each first importance weight, each second importance weight, and each fusion weight, until the emotion recognition model converges.

8. An emotion recognition method, the method comprising: Acquire the target speech and input the target speech into the emotion recognition model; Based on the target emotion result of the target speech determined by the emotion recognition model, the emotion recognition result corresponding to the target speech is determined; wherein, the emotion recognition model is the emotion recognition model according to any one of claims 1 to 7.

9. An emotion recognition model training device, the device comprising: The feature vector determination module is suitable for inputting sample speech into the emotion recognition model and determining at least one audio feature vector and at least one text feature vector corresponding to the sample speech. The weight determination module is adapted to determine the first importance weight corresponding to each audio feature vector, the second importance weight corresponding to each text feature vector, and the fusion weight corresponding to each audio feature vector and each text feature vector based on the emotion recognition model. The convergence module is adapted to: determine the audio modal feature vector of the sample speech based on the weighted average of each audio feature vector and its corresponding first importance weight; determine the text modal feature vector of the sample speech based on the weighted average of each text feature vector and its corresponding second importance weight; determine the fusion feature vector of the sample speech based on the weighted average of each audio feature vector and each text feature vector and their corresponding fusion weights; determine the sample emotion result corresponding to the sample speech based at least on the audio modal feature vector, the text modal feature vector, and the fusion feature vector; and train the emotion recognition model based on the standard emotion result of the sample speech and the loss function constructed from the sample emotion result until the emotion recognition model converges. Specifically, the convergence module is adapted to: concatenate at least the audio modality feature vector, the text modality feature vector, and the fused feature vector to obtain a concatenated feature vector; determine the fusion vector corresponding to the concatenated feature vector based on the fully connected fusion module in the emotion recognition model; and determine the sample emotion result corresponding to the fusion vector based on the emotion recognition module in the emotion recognition model.

10. An emotion recognition device, the device comprising: An acquisition module is adapted to acquire target speech and input the target speech into an emotion recognition model. The recognition module is adapted to determine the emotion recognition result corresponding to the target speech based on the target emotion result of the target speech determined by the emotion recognition model; wherein the emotion recognition model is the emotion recognition model according to any one of claims 1 to 7.

11. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps of any one of claims 1 to 8.

12. A computer program product storing at least one instruction, said at least one instruction being loaded by a processor and executing the method steps of any one of claims 1 to 8.

13. A terminal, comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 8.

Images