Method and device for determining audio emotion and computer equipment

Abstract

The application discloses a kind of audio emotion determination method, device and computer equipment, belong to artificial intelligence technical field.The method comprises: obtaining the audio representation and first text representation of audio, first text representation is the text representation of audio text, and audio text is obtained by audio recognition to audio;Audio recognition error detection is carried out based on audio representation and first text representation, to obtain predicted error probability, and predicted error probability indicates the identification error probability of audio recognition to audio;First text representation is weighted based on predicted error probability, to obtain weighted text representation, and weighted processing is used to set the confidence of first text representation in audio emotion classification process;Audio emotion classification is carried out based on weighted text representation and audio representation, to obtain the predicted audio emotion of audio.The method can improve the robustness of audio emotion classification.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, and computer device for determining audio emotion. Background Technology

[0002] With the popularization of intelligent devices, more and more devices can provide users with more intelligent services through voice interaction. For example, voice interaction devices can recommend suitable songs to users by recognizing the emotions in their voices.

[0003] In related technologies, speech recognition can be performed on the collected user speech to generate speech text, and then speech emotion recognition can be performed based on the speech text. For example, speech emotion recognition can be performed based on the tone words or emotion-related words contained in the speech text.

[0004] Clearly, the voice emotion recognition in related technologies depends on the accuracy of voice recognition. If the accuracy of voice recognition is low, it will affect the accuracy of subsequent voice emotion recognition. Summary of the Invention

[0005] This application provides a method, apparatus, and computer device for determining audio emotion, which can improve the accuracy of audio emotion determination. The technical solution is as follows:

[0006] According to one aspect of this application, a method for determining audio emotion is provided, the method comprising:

[0007] The audio representation and the first text representation of the audio are obtained, wherein the first text representation is the text representation of the audio text, and the audio text is obtained by performing audio recognition on the audio.

[0008] Based on the audio representation and the first text representation, audio recognition error detection is performed to obtain the predicted error probability, where the predicted error probability refers to the recognition error probability of performing audio recognition on the audio.

[0009] The first text representation is weighted based on the predicted error probability to obtain a weighted text representation. The weighting process is used to set the confidence level of the first text representation in the audio sentiment classification process.

[0010] Based on the weighted text representation and the audio representation, audio sentiment classification is performed to obtain the predicted audio sentiment of the audio.

[0011] According to another aspect of this application, a method for determining audio emotion is provided, the method comprising:

[0012] Obtain the sample audio representation and the first sample text representation of the sample audio, wherein the first sample text representation is the text representation of the sample audio text, and the sample audio text is obtained by performing audio recognition on the sample audio;

[0013] By using an error detection network, audio recognition errors are detected based on the sample audio representation and the first sample text representation, and a first sample probability is obtained from the error detection network output. The first sample probability refers to the probability of an audio recognition error when performing audio recognition on the sample audio.

[0014] The first sample text representation is weighted based on the first sample probability to obtain a weighted text representation. The weighting process is used to set the confidence level of the first sample text representation in the audio emotion classification process.

[0015] Audio sentiment classification is performed based on the weighted text representation of the samples and the audio representation of the samples to obtain the sample audio sentiment of the samples.

[0016] The error detection network is trained based on the first sample probability and the recognition error labels of the sample audio.

[0017] According to another aspect of this application, an audio emotion determination device is provided, the device comprising:

[0018] The acquisition module is used to acquire the audio representation and the first text representation of the audio, wherein the first text representation is the text representation of the audio text, and the audio text is obtained by performing audio recognition on the audio.

[0019] The error detection module is used to perform audio recognition error detection based on the audio representation and the first text representation to obtain the predicted error probability, wherein the predicted error probability refers to the recognition error probability of performing audio recognition on the audio.

[0020] The processing module is used to perform weighted processing on the first text representation based on the prediction error probability to obtain a weighted text representation. The weighted processing is used to set the confidence level of the first text representation in the audio sentiment classification process.

[0021] The sentiment classification module is used to perform audio sentiment classification based on the weighted text representation and the audio representation to obtain the predicted audio sentiment of the audio.

[0022] According to another aspect of this application, an audio emotion determination device is provided, the device comprising:

[0023] The acquisition module is used to acquire the sample audio representation and the first sample text representation of the sample audio, wherein the first sample text representation is the text representation of the sample audio text, and the sample audio text is obtained by performing audio recognition on the sample audio.

[0024] The error detection module is used to perform audio recognition error detection based on the sample audio representation and the first sample text representation through the error detection network, and obtain the first sample probability output by the error detection network. The first sample probability refers to the recognition error probability of performing audio recognition on the sample audio.

[0025] The processing module is used to perform weighted processing on the first sample text representation based on the first sample probability to obtain a sample weighted text representation. The weighted processing is used to set the confidence level of the first sample text representation in the audio emotion classification process.

[0026] The sentiment classification module is used to perform audio sentiment classification based on the sample weighted text representation and the sample audio representation to obtain the sample audio sentiment of the sample audio.

[0027] The training module is used to train the error detection network based on the first sample probability and the recognition error labels of the sample audio.

[0028] According to one aspect of this application, a computer device is provided, comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described method for determining audio emotion.

[0029] According to another aspect of this application, a computer-readable storage medium is provided, the storage medium storing a computer program, the computer program being loaded and executed by a processor to implement the method for determining audio emotion as described above.

[0030] According to another aspect of this application, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned method for determining audio emotion.

[0031] The beneficial effects of the technical solutions provided in this application include at least the following:

[0032] The error probability of audio recognition is predicted by using the audio representation and the first text representation. The predicted error probability is then used to weight the first text representation to set its confidence level in the subsequent audio sentiment classification process. This allows the subsequent audio sentiment classification process to focus more on text features with high confidence, reducing the impact of incorrectly predicted text features on the subsequent audio sentiment classification results, thereby improving the accuracy of audio sentiment determination. Furthermore, while increasing the error probability of audio recognition, the rate of decline in the accuracy of sentiment classification results can be reduced, thus improving the robustness of audio sentiment classification. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a computer system illustrated in an exemplary embodiment of this application;

[0035] Figure 2 A flowchart illustrating a method for determining audio emotion provided in an exemplary embodiment of this application is shown;

[0036] Figure 3 This is a schematic diagram illustrating the audio emotion classification process in an exemplary embodiment of this application;

[0037] Figure 4 A flowchart illustrating a method for determining audio emotion provided by another exemplary embodiment of this application is shown;

[0038] Figure 5 This is a schematic diagram illustrating the audio emotion classification process in another exemplary embodiment of this application;

[0039] Figure 6 A flowchart illustrating a method for determining audio emotion provided by another exemplary embodiment of this application is shown;

[0040] Figure 7 A flowchart illustrating a method for determining audio emotion provided by another exemplary embodiment of this application is shown;

[0041] Figure 8 This is a schematic diagram illustrating the training process of an error detection network in an exemplary embodiment of this application;

[0042] Figure 9 A flowchart illustrating a method for determining audio emotion provided by another exemplary embodiment of this application is shown;

[0043] Figure 10 This is a schematic diagram illustrating the training process of an error detection network, as shown in another exemplary embodiment of this application;

[0044] Figure 11 This is a structural block diagram of an audio emotion determination device provided in an exemplary embodiment of this application;

[0045] Figure 12 This is a structural block diagram of an audio emotion determination device provided in another exemplary embodiment of this application;

[0046] Figure 13 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0048] Figure 1 This is a schematic diagram of a computer system illustrating an exemplary embodiment of this application. Figure 1 As shown, the computer system includes a first device 110 and a second device 120.

[0049] The first device 110 is a training device for training an ASR-robust multimodal sentiment model. After the ASR-robust multimodal sentiment model is trained, the first device 110 can send the trained ASR-robust multimodal sentiment model to the second device 120 for deployment. The second device 120 is a device for performing audio sentiment prediction using the ASR-robust multimodal sentiment model. Optionally, the ASR-robust multimodal sentiment model mainly includes an error detection network and a sentiment classification network.

[0050] Optionally, the first device 110 and the second device 120 mentioned above can be computer devices with machine learning capabilities, such as terminals or servers.

[0051] Optionally, the first device 110 and the second device 120 may be the same computer device, or they may be different computer devices. Furthermore, when the first device 110 and the second device 120 are different devices, they may be of the same type, such as both being servers; or they may be of different types. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smartphone, in-vehicle terminal, smart TV, wearable device, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server may be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0052] Please refer to Figure 2 It illustrates a flowchart of a method for determining audio emotion provided in an exemplary embodiment of this application, in which the method is applied... Figure 1 The method, illustrated by the second device 120 shown, includes:

[0053] Step 201: Obtain the audio representation and the first text representation of the audio. The first text representation is the text representation of the audio text, which is obtained by performing audio recognition on the audio.

[0054] The audio can be a user's voice recorded in real time, a song, or a short video with sound. Any sound with content can be used as the audio for audio emotion classification. This application does not limit the audio.

[0055] Optionally, the audio emotion determination method of this application can be applied to recommendation scenarios, intelligent dialogue scenarios, emotion discrimination scenarios, etc. Taking the recommendation scenario as an example, the intelligent device can identify the current mood or emotion of the user based on the user's voice, and thus recommend content that matches the user's current emotion in a more intelligent way, such as recommending songs, videos, novels, etc. Taking the intelligent dialogue scenario as an example, the intelligent device can identify the current mood or emotion of the user based on the user's voice, and thus combine the user's current emotion to have a more intelligent dialogue with the user, etc.

[0056] In one possible implementation, when a user interacts with a smart device via voice, the smart device captures the user's voice through a microphone to obtain the user's audio. In order for the smart device to understand the meaning represented by the user's voice, the smart device is equipped with an Automatic Speech Recognition (ASR) module. This ASR module can perform audio recognition on the audio, convert the audio into audio text, and obtain the audio text of the audio. The recognized audio text is then used for subsequent interaction processes.

[0057] In audio sentiment classification, multimodal information from the audio is often used, including both acoustic and textual features. To achieve this, the audio and its text must first be encoded using acoustic and textual features, respectively. The audio is input into an acoustic encoder, which encodes its features to obtain an audio representation. The audio text is then input into a text encoder, which encodes its features to obtain a first text representation. This first text representation is then used for subsequent sentiment classification based on both the audio and textual representations.

[0058] Optionally, the acoustic encoder can use the wav2vec 2.0 pre-trained acoustic model. wav2vec 2.0 is a model that is self-supervised pre-trained based on a large amount of unlabeled data. It consists of multiple convolutional feature encoding layers and multiple transformers. By inputting the original audio signal into the audio encoder, the audio representation of each frame can be obtained. The text encoder can use the BERT model. BERT is also a model that is self-supervised trained based on a large amount of unlabeled data. By inputting each word corresponding to a sentence into the text encoder, the word representation of each word can be obtained.

[0059] Optionally, the audio representation is a set of audio representations of multiple frames of audio, and the first text representation is a set of representations of multiple words.

[0060] Step 202: Based on the audio representation and the first text representation, perform audio recognition error detection to obtain the predicted error probability. The predicted error probability refers to the probability of recognition error when performing audio recognition.

[0061] Because audio recognition using the ASR module may encounter errors, such as missing words, redundant words, or incorrect word recognition, and subsequent sentiment classification relies on the accuracy of the audio text recognition, errors in audio text recognition can affect the accuracy of subsequent sentiment classification. Therefore, to improve the accuracy of subsequent audio sentiment classification, one possible implementation involves detecting audio recognition errors based on the audio representation and the first text representation to obtain a predicted error probability. This predictive error probability determines the probability of recognition errors during the audio recognition process, i.e., it determines the credibility of the first text representation (audio text). This allows for the selective selection of credible text features during subsequent audio sentiment classification based on this predicted error probability.

[0062] Step 203: The first text representation is weighted based on the prediction error probability to obtain a weighted text representation. The weighting is used to set the confidence level of the first text representation in the audio sentiment classification process.

[0063] To improve the accuracy of subsequent audio sentiment classification, if the prediction error probability of a certain text feature in the first text representation is high, then that text feature should be given as little attention as possible during the audio sentiment classification process; conversely, if the prediction error probability of a certain text feature in the first text representation is low, then that text feature should be given as much attention as possible during the audio sentiment classification process. Therefore, in one possible implementation, the first text representation needs to be weighted using the prediction error probability to set the confidence level of the first text representation in the audio sentiment classification process (higher prediction error probability equals lower confidence level, and lower prediction error rate equals higher confidence level), resulting in a weighted text representation. In the weighted text representation, text features with higher prediction error probabilities have lower weights, and text features with lower prediction error probabilities have higher weights.

[0064] Step 204: Perform audio sentiment classification based on weighted text representation and audio representation to obtain the predicted audio sentiment.

[0065] In one possible implementation, after obtaining the weighted text representation, multimodal audio sentiment classification can be performed based on the weighted text representation and the audio representation. Credible text features are extracted from the weighted text representation, and audio features are extracted from the audio representation. Thus, the text features and audio features are combined to perform audio sentiment classification and obtain the predicted audio sentiment.

[0066] like Figure 3The diagram illustrates an exemplary embodiment of the audio sentiment classification process of this application. Terminal 320 collects the voice of user 310 via microphone to obtain audio 321, and inputs audio 321 into ASR model 322 for audio recognition to obtain audio text 323. Further, the terminal performs audio sentiment classification based on audio 321 and audio text 323. Audio 321 and audio text 323 are then input into an ASR robust multimodal speech sentiment model 324, which performs audio recognition error prediction and audio sentiment classification to obtain sentiment classification result 325.

[0067] In summary, in this embodiment, the error probability of audio recognition is predicted using the audio representation and the first text representation. The predicted error probability is then used to weight the first text representation to set its confidence level in the subsequent audio sentiment classification process. This allows the subsequent audio sentiment classification process to focus more on text features with high confidence, reducing the impact of incorrectly predicted text features on the subsequent audio sentiment classification results, thereby improving the accuracy of audio sentiment determination. Furthermore, while increasing the error probability of audio recognition, the rate of decline in the accuracy of sentiment classification results can be reduced, improving the robustness of audio sentiment classification.

[0068] When the prediction error rate of a certain text feature is high, in order to compensate for the lack of the text feature, in one possible implementation, the text feature can be compensated by the corresponding audio feature to further improve the accuracy of the text feature.

[0069] Please refer to Figure 4 It illustrates a flowchart of a method for determining audio emotion provided in another exemplary embodiment of this application, in which the method is applied... Figure 1 The method, illustrated by the second device 120 shown, includes:

[0070] Step 401: Obtain the audio representation and the first text representation of the audio. The first text representation is the text representation of the audio text, which is obtained by performing audio recognition on the audio.

[0071] The implementation method of step 401 can refer to step 201, and will not be repeated here in this embodiment.

[0072] Step 402: Align the audio representation and the first text representation to determine the second text representation, which refers to the audio representation of the audio text.

[0073] In order to determine whether each word in the audio text is accurately recognized, in one possible implementation, after obtaining the audio representation and the first text representation, the text representation is aligned with the corresponding part of the audio to obtain the audio representation corresponding to each text feature, and the second text representation is determined.

[0074] Optionally, when the audio text consists of N words, and the first text representation corresponding to the audio text also consists of word representations corresponding to N words, determining whether the audio text is accurately recognized requires determining the prediction error probability for each word, which corresponds to determining the audio representation of each word from the audio representation. In an exemplary example, step 402 may include steps 402A and 402B.

[0075] Step 402A: Input the audio representation and the first text representation into the attention mechanism layer to obtain N sub-audio representations output by the attention mechanism layer. The attention mechanism layer is used to determine the sub-audio representation of each word from the audio representation.

[0076] In one possible implementation, the attention mechanism layer aligns each word representation with the corresponding audio representation. Specifically, the audio representation and the first text representation are input into the attention mechanism layer, which then performs attention interaction between the word representation of each word and the audio representation of each frame in the audio representation, thereby determining the sub-audio representation corresponding to each word, which is to obtain N sub-audio representations output by the attention mechanism layer.

[0077] In an exemplary example, the attention mechanism layer can be defined as shown in Equation (1).

[0078]

[0079] Where Q represents the word representation of each word (the feature representation output by the text encoder), and K and V represent the audio representation of each frame of audio (the audio representation output by the acoustic encoder).

[0080] After the attention mechanism principle shown in formula (1) is applied to the word representation and audio representation of each word, the sub-audio representation of each word can be obtained. The sub-audio representation of each word can be shown in formula (2).

[0081]

[0082] in, This represents the word representation of the j-th word in the audio text (the feature representation output by the text encoder). This represents the audio representation of the j-th word in the audio text. This means that the audio representation of the j-th word is obtained by interacting with the word representation and the audio representation through an attention mechanism.

[0083] Step 402B: Generate a second text representation based on N sub-audio representations.

[0084] Corresponding to the first text representation, which consists of word representations of N words, the first text representation should also consist of audio representations of N words. Illustratively, the first text representation can be represented as: Where n represents the number of words in the audio text. The word representation represents the nth word in the audio text; the second text representation can be expressed as: in, This represents the audio representation of the nth word in the audio text.

[0085] Step 403: Input the first text representation and the second text representation into the error detection network to obtain the predicted error probability output by the error detection network.

[0086] To predict whether an audio text has been misrecognized, it is necessary to determine whether the audio text has been misrecognized based on the audio representation corresponding to the first text representation. In one possible implementation, the first text representation and the second text representation (the audio representation corresponding to the aligned text representation) are input into an error detection network for classification to determine whether each word in the audio text has been misrecognized, thereby outputting the prediction error probability.

[0087] Optionally, before inputting the first text representation and the second text representation into the error detection network, the first text representation and the second text representation can be summed to obtain a feature representation for audio recognition error detection. This feature representation can then be input into the error detection network for classification to obtain the predicted error probability output by the error detection network.

[0088] In an exemplary example, the feature representation used for audio recognition error detection can be as shown in Equation (3).

[0089]

[0090] in, Let represent the feature representation of the j-th word in the audio text used for audio recognition error detection. This represents the audio representation (second text representation) of the j-th word in the audio text. This represents the word representation (first text representation) of the j-th word in the audio text.

[0091] Optionally, since the audio text corresponds to N words, the error detection network predicts the prediction error probability for each word, thus obtaining N prediction error probabilities.

[0092] Optionally, the training process of the prediction error network can be referred to in the following embodiment, which will not be repeated here.

[0093] Step 404: The first text representation and the second text representation are weighted based on the prediction error probability to obtain a weighted text representation.

[0094] To compensate for the deficiencies of the first text representation, for example, if a word has a high prediction error probability, its corresponding audio representation can be used to compensate for it, thereby optimizing the text features of that word. In one possible implementation, the first and second text representations can be weighted based on the prediction error probability to obtain a weighted text representation. In the weighted text representation, the text representation with a high prediction error probability can be compensated for by its corresponding audio representation (the second text representation), thereby improving the accuracy of the text features.

[0095] To achieve the effect of having a low weight for the word representation and a high weight for the audio representation of a word with a high prediction error probability in the weighted text representation, it is necessary to weight the audio representation (second text representation) using the prediction error probability and weight the word representation (first text representation) using the probability obtained by subtracting the prediction error probability from the prediction error probability. In an exemplary example, step 404 may include steps 404A to 404D.

[0096] Step 404A: Determine the probability of a correct prediction. The sum of the probability of a correct prediction and the probability of a wrong prediction is 1.

[0097] Optionally, the correct prediction probability of a word can be obtained by subtracting the prediction error probability from 1; that is, the sum of the correct prediction probability and the prediction error probability is 1. For example, if the prediction error probability of the j-th word is 0.9, then the correct prediction probability of the j-th word is only 0.1.

[0098] Step 404B: The second text representation is weighted based on the prediction error probability to obtain the first weighted representation.

[0099] When the prediction error probability of a word is high, the weight of the word representation should be set to low and the weight of the audio representation should be set to high when generating the weighted text representation. Based on the relationship between the weight and the prediction error probability, in one possible implementation, the computer device uses the prediction error probability to weight the second text representation (audio representation) to obtain the first weighted representation.

[0100] Step 404C: The first text representation is weighted based on the predicted correct probability to obtain the second weighted representation.

[0101] Correspondingly, the computer device uses the predicted correct probability to weight the first text representation (the set of word representations) to obtain the second weighted representation.

[0102] Step 404D: Summing the first weighted representation and the second weighted representation to obtain the weighted text representation.

[0103] Optionally, the first weighted representation and the second weighted representation can be summed to obtain a weighted text representation after audio representation compensation. In this weighted text representation, the audio representation (audio feature) of words with high prediction error probability has a larger proportion, while the word representation (text feature) has a smaller proportion; the audio representation (audio feature) of words with low prediction error probability has a smaller proportion, while the word representation (text feature) has a larger proportion.

[0104] In an exemplary example, the formula for determining the weighted text representation can be as shown in formula (4).

[0105]

[0106] in, α represents the weighted text representation of the j-th word in the audio text. j Let (1-α) represent the prediction error probability of the j-th word in the audio text. j () represents the probability of correctly predicting the j-th word in the audio text. The second text representation (audio representation) of the j-th word in the audio text. Let J represent the first text representation (word representation) of the j-th word in the audio text. As can be seen from formula (4), if the prediction error probability of the j-th word in the audio text is high, then the proportion of the audio representation of the word in the weighted text representation is high, and the proportion of the word representation (text representation) is low. Conversely, if the prediction error probability of the j-th word in the audio text is low, then the proportion of the audio representation of the word in the weighted text representation is low, and the proportion of the word representation (text representation) is high. This achieves the purpose of using the audio representation of the word to make up for the lack of text when the prediction error probability is high.

[0107] Step 405: Concatenate the weighted text representation after pooling operation with the audio representation after pooling operation to obtain the audio feature representation.

[0108] In one possible implementation, after obtaining the weighted text representation and the audio representation, the weighted text representation can be input into the pooling layer for average pooling to obtain the weighted text representation after pooling, and the audio representation can be input into the pooling layer for average pooling to obtain the audio representation after pooling. Further, the weighted text representation and the audio representation after pooling are concatenated to obtain the audio feature representation for audio sentiment classification.

[0109] Optionally, since the weighted text representation already includes text features (first text representation) and audio features (second text representation), in other possible implementations, audio sentiment classification can also be performed directly based on the weighted text representation to obtain predicted audio sentiment.

[0110] Step 406: Input the audio feature representation into the sentiment classification network to obtain the predicted audio sentiment output by the sentiment classification network.

[0111] Optionally, the device is equipped with an emotion classification network. The audio feature representation obtained after splicing is input into the emotion classification network to predict the audio emotion, and the probability of belonging to each preset audio emotion is obtained. The audio emotion with the highest probability is then determined as the predicted audio emotion.

[0112] like Figure 5 As shown, it is a schematic diagram illustrating the process of audio emotion classification in another exemplary embodiment of this application. In the feature encoding stage, the audio is input into the audio encoder 501 for feature encoding to obtain the audio representation of each frame, and the audio text is input into the text encoder 502 for feature encoding to obtain the first text representation of each word in the audio text. In the text representation processing stage, the audio representation and the first text representation are input into the attention mechanism layer 503 for interaction to obtain the second text representation (audio representation) of each word in the audio text. Then, the first text representation and the second text representation are summed and input into the error detection network 504 for audio recognition error detection to obtain the prediction error probability of each word in the audio text. Further, the first text representation and the second text representation are weighted based on the prediction error probability to obtain the weighted text representation. In the audio sentiment classification stage: the audio representation is input into the first pooling layer 505 for average pooling to obtain the fused audio representation after pooling, and the weighted text representation is input into the second pooling layer 506 for average pooling to obtain the fused text representation after pooling. The fused audio representation and the fused text representation are concatenated and input into the sentiment classification network 507 to obtain the sentiment classification result (predicted audio sentiment) output by the sentiment classification network 507.

[0113] In this embodiment, by aligning the audio representation and the first text representation, the audio representation (second text representation) corresponding to each word is determined from the audio representation. This allows the first text representation and the second text representation to be weighted according to the prediction error probability, thereby using the audio representation to compensate for the defects of the text representation, improving the accuracy of the text representation, and further improving the accuracy of subsequent audio sentiment classification.

[0114] In other possible implementations, since the audio representation is based on audio encoding and contains more complete audio features, in order to further improve the accuracy of audio sentiment classification, when performing audio sentiment classification based on weighted text representation and audio representation, the audio representation and weighted text representation can also be further processed based on the predicted error probability, thereby further reducing the attention to erroneous text features in the subsequent audio sentiment classification process.

[0115] exist Figure 2 On the basis of, such as Figure 6 As shown, step 202 can be replaced by step 601, and step 204 can be replaced by steps 602 and 603.

[0116] Step 601: Perform audio recognition error detection based on the audio representation and the first text representation to obtain N prediction error probabilities, where each word corresponds to one prediction error probability.

[0117] Since the first text representation consists of word representations of N words, when performing audio recognition error detection based on the audio representation and the first text representation, N prediction error probabilities can be obtained, with one prediction error probability corresponding to each word. For example, if the audio text consists of 5 words, the detection result of audio recognition error detection can be: P = [0.1, 0.3, 0.9, 0.4, 0.5], indicating that the prediction error probability of the 1st word is 0.1, the prediction error probability of the 2nd word is 0.3, the prediction error probability of the 3rd word is 0.9, the prediction error probability of the 4th word is 0.4, and the prediction error probability of the 5th word is 0.5.

[0118] Step 602: Average the N prediction error probabilities to obtain the confidence score of the audio text.

[0119] Optionally, in this embodiment, not only can the first and second text representations be weighted based on the prediction error probability to obtain a weighted text representation, but the prediction error probability is also applied to the concatenation process of the audio representation and the weighted text representation. When the prediction error probability is high, this further reduces the proportion of text features in the concatenated audio feature representation and increases the proportion of audio features. In one possible implementation, the N prediction error probabilities can be averaged to obtain the average prediction error probability of the audio text, further determining the confidence score of the audio text.

[0120] In an exemplary example, the formula for determining the confidence score can be as shown in formula (5).

[0121]

[0122] Where D represents the confidence score, N represents the number of words in the audio text, and α j This represents the prediction error probability of the j-th word in the audio text.

[0123] Step 603: Perform audio sentiment classification based on confidence scores, weighted text representation, and audio representation to obtain the predicted audio sentiment.

[0124] A lower confidence score indicates a lower probability of prediction error for the audio text. Therefore, it's desirable to focus less on text features and more on audio features during audio sentiment classification to improve accuracy. In one possible implementation, the weighted text and audio representations can be further weighted based on the confidence score. The weighted result can then be used for subsequent audio sentiment classification to obtain the predicted audio sentiment.

[0125] In this embodiment, when performing audio sentiment classification based on weighted text representation and audio representation, the confidence score of the weighted text representation is obtained based on the prediction error probability. The audio representation and weighted text representation are then further processed according to the confidence score, thereby further reducing the attention to erroneous text features in the subsequent audio sentiment classification process. This improves the robustness of audio sentiment classification when the audio recognition error is high.

[0126] The audio sentiment classification process in the above embodiments involves an error detection network. The accuracy of the error detection network in predicting the error probability has a significant impact on the subsequent audio sentiment classification process. Therefore, how to train the error detection network to enable it to predict the error probability is what the following embodiments will explain.

[0127] Please refer to Figure 7It illustrates a flowchart of a method for determining audio emotion provided in another exemplary embodiment of this application, in which the method is applied... Figure 1 The method, illustrated by the first device 110 shown, includes:

[0128] Step 701: Obtain the sample audio representation and the first sample text representation of the sample audio. The first sample text representation is the text representation of the sample audio text, which is obtained by performing audio recognition on the sample audio.

[0129] When training an error detection network, it is necessary to supervise the training with the identification error labels of sample audio. The training sample set can include several sample pairs, each containing sample audio and its corresponding identification error label.

[0130] Optionally, when obtaining the identification error label, it is necessary to compare the actual sample text and the sample audio text of the sample audio. In this case, the sample audio needs to be input into the ASR module for audio recognition to obtain the sample audio text corresponding to each sample audio.

[0131] In one possible implementation, after obtaining the sample audio text of the sample audio, the sample audio can be input into an acoustic encoder for audio feature encoding to obtain the sample audio representation output by the acoustic encoder, and the sample audio text can be input into a text encoder for text feature encoding to obtain the first sample text representation output by the text encoder.

[0132] Optionally, the acoustic encoder can use the wav2vec2.0 model, while the text encoder can use the BERT model.

[0133] Step 702: Using an error detection network, audio recognition error detection is performed based on the sample audio representation and the first sample text representation to obtain the first sample probability output by the error detection network. The first sample probability refers to the recognition error probability of the sample audio.

[0134] To predict the probability of audio recognition errors, in one possible implementation, an error detection network is deployed. The error detection network performs audio recognition error detection based on the text representation of the audio text and its corresponding audio representation to predict the probability of audio recognition errors for sample audio, and obtains the first sample probability output by the error detection network.

[0135] Step 703: The first sample text representation is weighted based on the first sample probability to obtain the sample weighted text representation. The weighting is used to set the confidence level of the first sample text representation in the audio emotion classification process.

[0136] If the prediction error rate of a certain sample word in the sample audio text is high, then the audio sentiment classification process should focus as little as possible on the text features of that sample word; conversely, if the prediction error rate of a certain sample word in the sample audio text is low, then the audio sentiment classification process should focus as much as possible on the text features of that sample word. Therefore, in one possible implementation, it is necessary to use the predicted first sample probability to weight the first sample text representation in order to set the confidence level of the first sample text representation in the audio sentiment classification process, thus obtaining the weighted sample text representation.

[0137] Step 704: Perform audio sentiment classification based on the sample weighted text representation and sample audio representation to obtain the sample audio sentiment.

[0138] In one possible implementation, after obtaining the sample weighted text representation, multimodal audio sentiment classification can be performed based on the sample weighted text representation and the sample audio. Credible text features are extracted from the sample weighted text representation, and audio features are extracted from the sample audio representation. Thus, the text features and audio features are combined to perform audio sentiment classification and obtain the sample audio sentiment of the sample audio.

[0139] Step 705: Train an error detection network based on the probability of the first sample and the recognition error labels of the sample audio.

[0140] To enable the error detection network to predict the probability of audio recognition errors, in one possible implementation, a supervisory information—recognition error labels—is provided, allowing the error detection network to be trained based on the loss between the first sample probability and the recognition error labels. Multiple rounds of training can then be used to improve the probability prediction accuracy of the error detection network.

[0141] like Figure 8 The diagram illustrates the training process of an error detection network in an exemplary embodiment of this application. Sample audio is input into an audio encoder 801 for feature encoding to obtain a sample audio representation 802; sample audio text is input into a text encoder 803 to obtain a first sample text representation 804; an error detection network 805 performs audio recognition error detection based on the first sample text representation 804 and the sample audio representation to obtain a first sample probability 806; the error detection network 805 is trained based on the first sample probability 806 and the error detection label 807. Optionally, the first sample text representation 804 is weighted using the first sample probability 806 to obtain a weighted sample text representation 808, enabling subsequent audio sentiment classification based on the weighted sample text representation 808 and the sample audio representation 802 to obtain the sentiment classification result of the sample audio.

[0142] In summary, in this embodiment, by using identified error labels as supervisory information to train the error detection network, the error detection network is equipped with the function of predicting the probability of audio recognition errors. Thus, during model application, the error probability of audio recognition can be predicted using the audio representation and the first text representation. The predicted error probability is then used to weight the first text representation to set its confidence level in subsequent audio sentiment classification. This allows subsequent audio sentiment classification to focus more on text features with high confidence, reducing the impact of incorrectly predicted text features on the subsequent audio sentiment classification results, thereby improving the accuracy of audio sentiment determination. Furthermore, while increasing the audio recognition error probability, it can reduce the rate of decline in the accuracy of sentiment classification results, improving the robustness of audio sentiment classification.

[0143] To further improve model performance, this embodiment sets up multi-task training. Since the prediction accuracy of the error detection network affects the audio sentiment classification process, in addition to using error identification labels as supervision information for training, sample sentiment labels are also used as supervision information for training during model training. Multi-task training is used to improve model performance.

[0144] like Figure 9 As shown, it illustrates a flowchart of a method for determining audio emotion provided in another exemplary embodiment of this application, in which the method is applied... Figure 1 The method, illustrated by the first device 110 shown, includes:

[0145] Step 901: Obtain the sample audio representation and the first sample text representation of the sample audio. The first sample text representation is the text representation of the sample audio text, which is obtained by performing audio recognition on the sample audio.

[0146] The implementation method of step 901 can be referred to the above embodiment, and will not be repeated here.

[0147] Step 902: Align the sample audio representation and the first sample text representation to determine the second sample text representation, which refers to the sample audio representation of the sample audio text.

[0148] In order to determine whether each sample word in the sample audio text is accurately recognized, in one possible implementation, after obtaining the sample audio representation and the first sample text representation, it is necessary to align the first sample text representation with the part corresponding to the audio to obtain the audio representation corresponding to the sample audio text, that is, the second sample text representation.

[0149] Optionally, when the sample audio text consists of M sample words, and the first sample text representation corresponding to the sample audio text also consists of the sample word representations corresponding to the M sample words, determining whether the sample audio text is accurately recognized requires predicting the first sample probability for each sample word, which corresponds to determining the sample audio representation of each sample word from the sample audio representation. In an exemplary example, step 902 may include steps 902A and 902B.

[0150] Step 902A: Input the sample audio representation and the first sample text representation into the attention mechanism layer to obtain M sub-sample audio representations output by the attention mechanism layer. The attention mechanism layer is used to determine the sub-sample audio representation of each sample word from the sample audio representation.

[0151] Optionally, an attention mechanism layer is used to generate a second sample text representation. As can be seen from the definition of the attention mechanism layer shown in formula (1), when determining the second sample text representation, the sample audio representation and the first sample text representation can be input into the attention mechanism layer. The attention mechanism layer performs attention interaction between the sample word representation of each sample word and the sample audio representation of each frame in the sample audio representation, thereby determining the sub-sample audio representation corresponding to each sample word, that is, obtaining the M sub-sample audio representations output by the attention mechanism layer.

[0152] Step 902B: Generate a second sample text representation based on the M sub-sample audio representations.

[0153] When the sample audio text contains M sample words, the first sample text representation is composed of the sample word representations of the M sample words. The attention mechanism layer can determine the sub-sample audio representation corresponding to each sample word, so the second sample text representation should also be composed of the M sub-sample audio representations.

[0154] Step 903: Input the first sample text representation and the second sample text representation into the error detection network to obtain the first sample probability output by the error detection network.

[0155] To predict whether a sample audio text has been misrecognized, it is necessary to determine whether the audio text has been misrecognized based on the sample audio corresponding to the sample audio text. This requires determining whether the audio text has been misrecognized based on the sample audio representation (second sample text representation) corresponding to the first sample text representation. In one possible implementation, the first sample text representation and the second sample text representation (the sample audio representation corresponding to the aligned sample text representation) are input into an error detection network for classification to determine whether each sample word in the sample audio text has been misrecognized, thereby outputting M first sample probabilities, with each sample word having its corresponding first sample probability.

[0156] Optionally, before inputting the text representation of the first sample and the text representation of the second sample, the summation process can be performed to obtain the sample feature representation for audio recognition error detection. This sample feature representation can then be input into the error detection network for classification to obtain the first sample probability output by the error detection network.

[0157] Step 904: Based on the probability of the first sample, the text representation of the first sample and the text representation of the second sample are weighted to obtain the weighted text representation of the sample.

[0158] To compensate for the deficiencies of the first sample text representation, for example, if a sample word has a high first sample probability, it can be compensated for by the corresponding sample audio representation to optimize the text features of that sample word. In one possible implementation, the first sample text representation and the second sample text representation can be weighted based on the first sample probability to obtain a weighted sample text representation. In the weighted sample text representation, the sample text representation with a high first sample probability can be compensated for by its corresponding sample audio representation (the second sample text representation), thereby improving the accuracy of the sample text features.

[0159] To achieve the effect that in the weighted text representation of a sample word with a higher first sample probability, the weight of that sample word's word representation is low, while the weight of its audio representation is high, it is necessary to use the first sample probability to weight the audio representation (second sample text representation), and use the probability obtained by subtracting the first sample probability from the first sample probability to weight the word representation (first sample text representation). In an exemplary example, step 904 may include steps 904A to 904D.

[0160] Step 904A: Determine the probability of the second sample. The sum of the probability of the second sample and the probability of the first sample is 1.

[0161] Optionally, since the first sample probability is the probability of misidentifying a sample word, the correct prediction probability (second sample probability) of that sample word can be obtained by subtracting the first sample probability from 1. That is, the sum of the first sample probability and the second sample probability is 1. For example, if the first sample probability of the j-th word is 0.3, then the second sample probability of the j-th word is 0.7.

[0162] Step 904B: The text representation of the second sample is weighted based on the probability of the first sample to obtain the weighted representation of the first sample.

[0163] When the first sample probability (prediction error probability) of a sample word is high, the weight of the sample word representation of that sample word should be set to low and the weight of the sample audio representation of that sample word should be set to high when generating the sample weighted text representation. Based on the relationship between the weight and the first sample probability, in one possible implementation, the computer device uses the first sample probability to weight the second sample text representation (sample audio representation) to obtain the first sample weighted representation.

[0164] Step 904C: The text representation of the first sample is weighted based on the probability of the second sample to obtain the weighted representation of the second sample.

[0165] Correspondingly, the computer device uses the second sample probability (the probability of correct prediction) to weight the first sample text representation (the set of sample word representations) to obtain the second sample weighted representation.

[0166] Step 904D: Summing the weighted representation of the first sample and the weighted representation of the second sample yields the weighted text representation of the sample.

[0167] Optionally, by performing a weighted summation of the first sample text representation and the second sample text representation according to formula (4), a weighted sample text representation can be obtained after the sample audio representation is sufficient. In this weighted sample text representation, the sample audio representation (audio feature) of the sample word with high first sample probability has a larger proportion, while the sample word representation (text feature) has a smaller proportion; the sample audio representation (audio feature) of the sample word with low first sample probability has a smaller proportion, while the sample word representation (text feature) has a larger proportion.

[0168] Step 905: Concatenate the weighted text representation of the sample after pooling operation with the audio representation of the sample after pooling operation to obtain the audio feature representation of the sample.

[0169] In one possible implementation, after obtaining the sample weighted text representation and the sample audio representation, the sample weighted text representation can be input into a pooling layer for average pooling to obtain the sample weighted text representation after pooling, and the sample audio representation can be input into a pooling layer for average pooling to obtain the sample audio representation after pooling. Further, the sample weighted text representation and the sample audio representation after pooling are concatenated to obtain the sample audio feature representation for audio sentiment classification.

[0170] Step 906: Input the sample audio feature representation into the sentiment classification network to obtain the sample audio sentiment output by the sentiment classification network.

[0171] Optionally, since the prediction accuracy of the error detection network also affects the audio sentiment classification task, a loss for the sentiment classification task is introduced in addition to the loss for the error detection task during model training. In one possible implementation, the sample audio features are input into the sentiment classification network to classify the audio sentiment, obtain the probability of belonging to each preset audio sentiment, and determine the audio sentiment with the highest probability as the sample audio sentiment.

[0172] Step 907: Calculate the error detection loss based on the probability of the first sample and the misidentified label.

[0173] Optionally, the identification of mislabels is based on the alignment and comparison of the actual text of the sample audio and the ASR recognition result (sample audio text). In an exemplary example, the process of determining the identification of mislabels may include the following steps:

[0174] 1. Obtain the actual text of the sample audio.

[0175] 2. Align the actual sample text and the sample audio text.

[0176] Once the actual text of the sample audio is obtained, the sample audio text and the actual text can be aligned to compare whether each sample word in the sample audio is the same as the actual text, and then the mislabel of each sample word can be determined based on the comparison results.

[0177] Third, if the j-th sample word is the same as the actual sample text, determine the j-th misidentification label of the j-th sample word as the first misidentification label, where j is a positive integer less than or equal to M.

[0178] Optionally, when the j-th sample word is the same as the actual sample text, it means that the j-th sample word is correctly identified. Then, the identification error label of the j-th sample word can be determined as 1 (first error label), that is, 1 is determined as the j-th identification error label of the j-th sample word.

[0179] Fourth, if the j-th sample word is different from the actual sample text, determine the j-th misidentified label of the j-th sample word as the second misidentified label.

[0180] Conversely, if the j-th sample word is different from the actual sample text, it means that the j-th sample word is incorrectly identified. In order to distinguish between the two cases of incorrect and correct identification, the identification error label of the j-th sample word can be determined as 0 (second error label), that is, 0 is determined as the j-th identification error label of the j-th sample word.

[0181] Taking the first error label as 1 and the second error label as 0 as an example, the process of determining the error label of the sample audio can be shown in Table 1.

[0182] Table 1

[0183] Actual text He is a - gradparent Identify error labels 1 1 1 0 0

[0184] As shown in Table 1, the audio recognition result of the fourth sample word is different from the actual text, so the recognition error label of the fourth sample word is 0; the audio recognition result of the second sample word is the same as the actual text, so the recognition error label of the second sample word is 1.

[0185] In an exemplary example, the prediction error loss of an error detection network can be expressed as Equation (6).

[0186]

[0187] Among them, L errdect This represents the error detection loss of the error detection network. This indicates whether the j-th word in the k-th sentence is incorrect, and it identifies error labels. This represents the probability (first sample probability) of predicting whether the j-th word in the k-th sentence is incorrect.

[0188] Optionally, the first sample probability and the misidentified label corresponding to each sample word can be substituted into formula (6) to calculate the error detection loss of the error detection network.

[0189] Step 908: Calculate the emotion classification loss based on the sample audio emotion and sample emotion labels.

[0190] This embodiment optimizes the model through a multi-task loss function, combining error detection loss and sentiment classification loss. Correspondingly, it also requires calculating the sentiment classification loss of the sentiment classification network based on the sentiment of the sample audio output by the sentiment classification network and the corresponding sentiment label of the sample audio.

[0191] Step 909: Train the error detection network and the sentiment classification network based on the error detection loss and the sentiment classification loss.

[0192] In an exemplary example, the total loss of the model can be represented as Equation (7).

[0193] L total =L emo +β×L errdect (7)

[0194] Among them, L total L represents the total loss of the model. emo L represents the sentiment classification loss. errdectThe error detection loss, sentiment classification loss, and error detection loss are balanced by the parameter β, which can take any value between 0 and 1 and is set manually. Optionally, in each training round, the sum of the error detection loss and sentiment classification loss can be used to train the error detection network and the sentiment classification network to improve the model's predictive performance.

[0195] like Figure 10 The diagram shown is a schematic representation of the training process of an error detection network according to another exemplary embodiment of this application. In the feature encoding stage, sample audio is input into audio encoder 1001 for feature encoding to obtain a sample audio representation for each frame. Sample audio text is input into text encoder 1002 for feature encoding to obtain a first sample text representation for each sample word in the sample audio text. In the text representation processing stage, the sample audio representation and the first sample text representation are input into attention mechanism layer 1003 for interaction to obtain a second sample text representation (sample audio representation) for each sample word in the sample audio text. The first and second sample text representations are then summed and input into error detection network 1004 for audio recognition error detection to obtain a first sample probability for each sample word in the sample audio text. Further, the first and second sample text representations are weighted based on the first sample probability to obtain... The sample weighted text representation is used in the audio sentiment classification stage. The sample audio representation is input into the first pooling layer 1005 for average pooling to obtain a fused sample audio representation. The sample weighted text representation is input into the second pooling layer 1006 for average pooling to obtain a fused sample text representation. The fused sample audio representation and fused sample text representation are concatenated and input into the sentiment classification network 1007 to obtain the sentiment classification result (predicted audio sentiment) output by the sentiment classification network 1007. In the model training stage, the error detection loss is calculated based on the first sample probability and the error detection label. The sentiment classification loss is calculated based on the sentiment classification result and the sample sentiment label. The error detection loss and the sentiment classification loss are then used to train the error detection network 1004 and the sentiment classification network 1007.

[0196] In this embodiment, the error detection loss is calculated based on the incorrect identification label and the probability of the first sample, and the emotion classification loss is calculated based on the sample audio emotion and the sample emotion label. This allows the model to be trained on multiple tasks using the emotion classification loss and the prediction error loss, which can further improve the robustness of the model.

[0197] Figure 11 This is a structural block diagram of an audio emotion determination device provided in an exemplary embodiment of this application. The device includes:

[0198] The acquisition module 1101 is used to acquire the audio representation and the first text representation of the audio, wherein the first text representation is the text representation of the audio text, and the audio text is obtained by performing audio recognition on the audio.

[0199] Error detection module 1102 is used to perform audio recognition error detection based on the audio representation and the first text representation to obtain a predicted error probability, wherein the predicted error probability refers to the recognition error probability of performing audio recognition on the audio;

[0200] Processing module 1103 is used to perform weighted processing on the first text representation based on the prediction error probability to obtain a weighted text representation, wherein the weighted processing is used to set the confidence level of the first text representation in the audio sentiment classification process;

[0201] The sentiment classification module 1104 is used to perform audio sentiment classification based on the weighted text representation and the audio representation to obtain the predicted audio sentiment of the audio.

[0202] Optionally, the error detection module 1102 is further configured to:

[0203] Align the audio representation and the first text representation to determine the second text representation, whereby the second text representation refers to the audio representation of the audio text;

[0204] The first text representation and the second text representation are input into the error detection network to obtain the predicted error probability output by the error detection network;

[0205] The processing module 1103 is further configured to:

[0206] The first text representation and the second text representation are weighted based on the predicted error probability to obtain the weighted text representation.

[0207] Optionally, the processing module 1103 is further configured to:

[0208] Determine the probability of a correct prediction, wherein the sum of the probability of a correct prediction and the probability of an incorrect prediction is 1;

[0209] The second text representation is weighted based on the predicted error probability to obtain a first weighted representation; the first text representation is weighted based on the predicted correct probability to obtain a second weighted representation.

[0210] The first weighted representation and the second weighted representation are summed to obtain the weighted text representation.

[0211] Optionally, the audio text consists of N words, where N is a positive integer;

[0212] The error detection module 1102 is also used for:

[0213] The audio representation and the first text representation are input into the attention mechanism layer to obtain N sub-audio representations output by the attention mechanism layer. The attention mechanism layer is used to determine the sub-audio representation of each word from the audio representation.

[0214] The second text representation is generated based on the N sub-audio representations.

[0215] Optionally, the emotion classification module 1104 is further configured to:

[0216] The weighted text representation processed by the pooling operation is concatenated with the audio representation processed by the pooling operation to obtain the audio feature representation;

[0217] The audio feature representation is input into the sentiment classification network to obtain the predicted audio sentiment output by the sentiment classification network.

[0218] Optionally, the audio text consists of N words, where N is a positive integer;

[0219] The error detection module 1102 is also used for:

[0220] Based on the audio representation and the first text representation, audio recognition error detection is performed to obtain N prediction error probabilities, where each word corresponds to one prediction error probability;

[0221] The emotion classification module is also used for:

[0222] The confidence score of the audio text is obtained by averaging the N predicted error probabilities.

[0223] Based on the confidence score, the weighted text representation, and the audio representation, audio sentiment classification is performed to obtain the predicted audio sentiment of the audio.

[0224] Figure 12 This is a structural block diagram of an audio emotion determination device provided in another exemplary embodiment of this application, the device comprising:

[0225] The acquisition module 1201 is used to acquire the sample audio representation and the first sample text representation of the sample audio, wherein the first sample text representation is the text representation of the sample audio text, and the sample audio text is obtained by performing audio recognition on the sample audio.

[0226] Error detection module 1202 is used to perform audio recognition error detection based on the sample audio representation and the first sample text representation through an error detection network, and obtain the first sample probability output by the error detection network, wherein the first sample probability refers to the recognition error probability of performing audio recognition on the sample audio;

[0227] Processing module 1203 is used to perform weighted processing on the first sample text representation based on the first sample probability to obtain a sample weighted text representation. The weighted processing is used to set the confidence level of the first sample text representation in the audio emotion classification process.

[0228] The sentiment classification module 1204 is used to perform audio sentiment classification based on the sample weighted text representation and the sample audio representation to obtain the sample audio sentiment of the sample audio.

[0229] Training module 1205 is used to train the error detection network based on the first sample probability and the recognition error label of the sample audio.

[0230] Optionally, the error detection module 1202 is further configured to:

[0231] Align the sample audio representation and the first sample text representation to determine the second sample text representation, whereby the second sample text representation refers to the sample audio representation of the sample audio text;

[0232] The first sample text representation and the second sample text representation are input into the error detection network to obtain the first sample probability output by the error detection network;

[0233] The processing module 1203 is also used for:

[0234] The first sample text representation and the second sample text representation are weighted based on the first sample probability to obtain the sample weighted text representation.

[0235] Optionally, the processing module 1203 is further configured to:

[0236] Determine the probability of the second sample, the sum of the probability of the second sample and the probability of the first sample is 1;

[0237] The second sample text representation is weighted based on the first sample probability to obtain the first sample weighted representation.

[0238] The first sample text representation is weighted based on the second sample probability to obtain the second sample weighted representation.

[0239] The first sample weighted representation and the second sample weighted representation are summed to obtain the sample weighted text representation.

[0240] Optionally, the sample audio text consists of M sample words, where M is a positive integer;

[0241] The error detection module 1202 is also used for:

[0242] The sample audio representation and the first sample text representation are input into the attention mechanism layer to obtain M sub-sample audio representations output by the attention mechanism layer. The attention mechanism layer is used to determine the sub-sample audio representation of each sample word from the sample audio representation.

[0243] Based on the M sub-sample audio representations, the second sample text representation is generated.

[0244] Optionally, the emotion classification module 1204 is further configured to:

[0245] The weighted text representation of the sample after pooling is concatenated with the audio representation of the sample after pooling to obtain the audio feature representation of the sample.

[0246] The sample audio feature representation is input into the sentiment classification network to obtain the sentiment of the sample audio output by the sentiment classification network;

[0247] The training module 1205 is also used for:

[0248] Based on the first sample probability and the identified incorrect label, calculate the error detection loss;

[0249] Based on the sample audio sentiment and sample sentiment labels, calculate the sentiment classification loss;

[0250] The error detection network and the sentiment classification network are trained based on the error detection loss and the sentiment classification loss.

[0251] Optionally, the sample audio text consists of M sample words, where M is a positive integer;

[0252] The device further includes:

[0253] The acquisition module 1201 is also used to acquire the sample actual text of the sample audio;

[0254] An alignment module is used to align the actual sample text and the sample audio text.

[0255] The determination module is used to determine the j-th misidentification label of the j-th sample word as the first misidentification label when the j-th sample word is the same as the actual text of the sample, where j is a positive integer less than or equal to M;

[0256] The determining module is further configured to determine the j-th misidentification label of the j-th sample word as the second misidentification label when the j-th sample word is different from the actual sample text.

[0257] In summary, in this embodiment, the error probability of audio recognition is predicted using the audio representation and the first text representation. The predicted error probability is then used to weight the first text representation to set its confidence level in the subsequent audio sentiment classification process. This allows the subsequent audio sentiment classification process to focus more on text features with high confidence, reducing the impact of incorrectly predicted text features on the subsequent audio sentiment classification results, thereby improving the accuracy of audio sentiment determination. Furthermore, while increasing the error probability of audio recognition, the rate of decline in the accuracy of sentiment classification results can be reduced, improving the robustness of audio sentiment classification.

[0258] Figure 13 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device 1300 may be... Figure 1 The first equipment in the middle is 110, which can also be Figure 1 The second device 120 is included. The computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including random access memory (RAM) 1302 and read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the central processing unit 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 to facilitate the transfer of information between various devices within the computer device, and a mass storage device 1307 for storing the operating system 1313, application programs 1314, and other program modules 1315.

[0259] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 for user input, such as a mouse or keyboard. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include the input / output controller 1310 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0260] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer device-readable media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0261] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1304 and mass storage device 1307 described above can be collectively referred to as memory.

[0262] According to various embodiments of this disclosure, the computer device 1300 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1300 can be connected to the network 1311 via a network interface unit 1312 connected to the system bus 1305, or it can use the network interface unit 1312 to connect to other types of networks or remote computer device systems (not shown).

[0263] The memory also includes one or more programs stored in the memory, and the central processing unit 1301 executes the one or more programs to implement all or part of the steps of the above-mentioned method for determining audio emotion.

[0264] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method for determining audio emotion provided in the above-described method embodiments.

[0265] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the audio emotion determination method provided in the above-described method embodiments.

[0266] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0267] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0268] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining audio emotion, characterized in that, The method includes: The audio representation and the first text representation of the audio are obtained, wherein the first text representation is the text representation of the audio text, and the audio text is obtained by performing audio recognition on the audio. Align the audio representation and the first text representation to determine the second text representation, whereby the second text representation refers to the audio representation of the audio text; The first text representation and the second text representation are input into the error detection network to obtain the predicted error probability output by the error detection network. The predicted error probability refers to the recognition error probability of the audio. Determine the probability of a correct prediction, wherein the sum of the probability of a correct prediction and the probability of an incorrect prediction is 1; The second text representation is weighted based on the predicted error probability to obtain a first weighted representation, wherein the predicted error probability represents the proportion of the audio representation; The first text representation is weighted based on the predicted correct probability to obtain a second weighted representation, where the predicted correct probability represents the proportion of the text representation. Summing the first weighted representation and the second weighted representation yields the weighted text representation; Based on the weighted text representation and the audio representation, audio sentiment classification is performed to obtain the predicted audio sentiment of the audio.

2. The method according to claim 1, characterized in that, The audio text consists of N words, where N is a positive integer; The step of aligning the audio representation and the first text representation to determine the second text representation includes: The audio representation and the first text representation are input into the attention mechanism layer to obtain N sub-audio representations output by the attention mechanism layer. The attention mechanism layer is used to determine the sub-audio representation of each word from the audio representation. The second text representation is generated based on the N sub-audio representations.

3. The method according to claim 1 or 2, characterized in that, The step of performing audio sentiment classification based on the weighted text representation and the audio representation to obtain the predicted audio sentiment includes: The weighted text representation processed by the pooling operation is concatenated with the audio representation processed by the pooling operation to obtain the audio feature representation; The audio feature representation is input into the sentiment classification network to obtain the predicted audio sentiment output by the sentiment classification network.

4. The method according to claim 1 or 2, characterized in that, The audio text consists of N words, and N prediction error probabilities are obtained, where each word corresponds to a prediction error probability, and N is a positive integer; The step of performing audio sentiment classification based on the weighted text representation and the audio representation to obtain the predicted audio sentiment includes: The confidence score of the audio text is obtained by averaging the N predicted error probabilities. Based on the confidence score, the weighted text representation, and the audio representation, audio sentiment classification is performed to obtain the predicted audio sentiment of the audio.

5. A method for determining audio emotion, characterized in that, The method includes: Obtain the sample audio representation and the first sample text representation of the sample audio, wherein the first sample text representation is the text representation of the sample audio text, and the sample audio text is obtained by performing audio recognition on the sample audio; Align the sample audio representation and the first sample text representation to determine the second sample text representation, whereby the second sample text representation refers to the sample audio representation of the sample audio text; The first sample text representation and the second sample text representation are input into the error detection network to obtain the first sample probability output by the error detection network. The first sample probability refers to the recognition error probability of the audio sample. Determine the probability of the second sample, the sum of the probability of the second sample and the probability of the first sample is 1; The second sample text representation is weighted based on the first sample probability to obtain the first sample weighted representation, where the first sample probability represents the proportion of audio representation. The first sample text representation is weighted based on the second sample probability to obtain the second sample weighted representation, where the second sample probability represents the proportion of the text representation. The first sample weighted representation and the second sample weighted representation are summed to obtain the sample weighted text representation; Audio sentiment classification is performed based on the weighted text representation of the samples and the audio representation of the samples to obtain the sample audio sentiment of the samples. The error detection network is trained based on the first sample probability and the recognition error labels of the sample audio.

6. The method according to claim 5, characterized in that, The sample audio text consists of M sample words, where M is a positive integer; The step of aligning the sample audio representation and the first sample text representation to determine the second sample text representation includes: The sample audio representation and the first sample text representation are input into the attention mechanism layer to obtain M sub-sample audio representations output by the attention mechanism layer. The attention mechanism layer is used to determine the sub-sample audio representation of each sample word from the sample audio representation. Based on the M sub-sample audio representations, the second sample text representation is generated.

7. The method according to claim 5 or 6, characterized in that, The step of performing audio sentiment classification based on the sample weighted text representation and the sample audio representation to obtain the sample audio sentiment includes: The weighted text representation of the sample after pooling is concatenated with the audio representation of the sample after pooling to obtain the audio feature representation of the sample. The sample audio feature representation is input into the sentiment classification network to obtain the sentiment of the sample audio output by the sentiment classification network; The step of training the error detection network based on the first sample probability and the recognition error label of the sample audio includes: Based on the first sample probability and the identified incorrect label, calculate the error detection loss; Based on the sample audio sentiment and sample sentiment labels, calculate the sentiment classification loss; The error detection network and the sentiment classification network are trained based on the error detection loss and the sentiment classification loss.

8. The method according to claim 5 or 6, characterized in that, The sample audio text consists of M sample words, where M is a positive integer; The method further includes: Obtain the actual text of the sample audio; Alignment processing is performed on the actual sample text and the sample audio text; If the j-th sample word is the same as the actual text of the sample, the j-th misidentification label of the j-th sample word is determined as the first misidentification label, where j is a positive integer less than or equal to M; If the j-th sample word is different from the actual sample text, the j-th misidentification label of the j-th sample word is determined as the second misidentification label.

9. A device for determining audio emotion, characterized in that, The device includes: The acquisition module is used to acquire the audio representation and the first text representation of the audio, wherein the first text representation is the text representation of the audio text, and the audio text is obtained by performing audio recognition on the audio. An error detection module is used to align the audio representation and the first text representation to determine a second text representation, wherein the second text representation refers to the audio representation of the audio text; The first text representation and the second text representation are input into the error detection network to obtain the predicted error probability output by the error detection network. The predicted error probability refers to the recognition error probability of the audio. The processing module is used to determine the probability of a correct prediction, wherein the sum of the probability of a correct prediction and the probability of an incorrect prediction is 1. The second text representation is weighted based on the predicted error probability to obtain a first weighted representation, wherein the predicted error probability represents the proportion of the audio representation; The first text representation is weighted based on the predicted correct probability to obtain a second weighted representation, where the predicted correct probability represents the proportion of the text representation. Summing the first weighted representation and the second weighted representation yields the weighted text representation; The sentiment classification module is used to perform audio sentiment classification based on the weighted text representation and the audio representation to obtain the predicted audio sentiment of the audio.

10. A device for determining audio emotion, characterized in that, The device includes: The acquisition module is used to acquire the sample audio representation and the first sample text representation of the sample audio, wherein the first sample text representation is the text representation of the sample audio text, and the sample audio text is obtained by performing audio recognition on the sample audio. An error detection module is used to align the sample audio representation and the first sample text representation to determine a second sample text representation, which refers to the sample audio representation of the sample audio text; and to input the first sample text representation and the second sample text representation into an error detection network to obtain a first sample probability output by the error detection network, which refers to the recognition error probability of performing audio recognition on the sample audio. The processing module is used to determine the probability of the second sample, wherein the sum of the probability of the second sample and the probability of the first sample is 1; The second sample text representation is weighted based on the first sample probability to obtain the first sample weighted representation, where the first sample probability represents the proportion of audio representation. The first sample text representation is weighted based on the second sample probability to obtain the second sample weighted representation, where the second sample probability represents the proportion of the text representation. The first sample weighted representation and the second sample weighted representation are summed to obtain the sample weighted text representation; The sentiment classification module is used to perform audio sentiment classification based on the sample weighted text representation and the sample audio representation to obtain the sample audio sentiment of the sample audio. The training module is used to train the error detection network based on the first sample probability and the recognition error labels of the sample audio.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the method for determining audio emotion as described in any one of claims 1 to 4, or to implement the method for determining audio emotion as described in any one of claims 5 to 8.

12. A computer program product, characterized in that, The computer program product includes computer instructions, which a processor executes to cause a computer device to perform the method for determining audio emotion as described in any one of claims 1 to 8.

Images