Emotion recognition method and device, electronic equipment and computer program product

Abstract

The invention provides an emotion recognition method and device, electronic equipment and a computer program product, and the method can recognize voice data to obtain text data and object switching points corresponding to the voice data. And then segmentation is performed according to the object switching points to obtain a plurality of voice data units corresponding to the voice data and a plurality of text data units corresponding to the text data, and the voice data units and the text data units identified by the voice data units correspond to the same voice content. And finally, according to the voice data unit and the text data unit corresponding to each voice content, identifying to obtain the emotion of each voice content. Through the setting, segmentation can be performed according to the changed points of the speaking object to obtain the voice data unit and the text data unit corresponding to the single speaking object, and the voice content emotion of the single speaking object is recognized from the two dimensions of voice and text, so that a more accurate recognition result can be obtained.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition technology, and in particular to an emotion recognition method, device, electronic device and computer program product. Background Art

[0002] Perceiving emotions in a conversation not only helps us better understand the speaker's true intentions, but also helps us extract rich emotional information from the conversation, which can be used as an important sample for training large models of speech synthesis bases. Therefore, how to accurately perceive the speaker's emotions in a conversation has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0003] In view of this, the present application proposes an emotion recognition method, device, electronic device and computer program product, which can accurately perceive the emotions of the speaker in a conversation.

[0004] The technical solutions proposed in this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides an emotion recognition method, comprising:

[0006] Recognize the voice data to obtain text data and object switching points corresponding to the voice data; wherein the object switching points include switching points of each speaking object in the voice data;

[0007] Segmenting according to the object switching point to obtain a plurality of voice data units corresponding to the voice data and a plurality of text data units corresponding to the text data; wherein the voice data unit and the text data unit recognized by the voice data unit correspond to the same voice content;

[0008] The emotion of each voice content is identified based on the voice data unit and the text data unit corresponding to each voice content.

[0009] Furthermore, in the above method, the step of recognizing the voice data to obtain the text data and the object switching point corresponding to the voice data includes:

[0010] The speech data is input into a pre-trained speech recognition model so that the speech recognition model recognizes the speech data and obtains text data and object switching points corresponding to the speech data.

[0011] Furthermore, in the above method, the generation process of the plurality of voice data units includes:

[0012] According to the object switching point, the speech features of the speech data are divided into a plurality of speech data units; the speech features are output by the encoder of the speech recognition model when the speech recognition model processes the speech data.

[0013] Furthermore, in the above method, the generation process of the plurality of text data units includes:

[0014] According to the object switching point, the text features of the text data are divided into a plurality of text data units; the text features are output by the decoder of the speech recognition model when the speech recognition model processes the speech data.

[0015] Furthermore, in the above method, the training process of the speech recognition model includes:

[0016] Acquire multiple audio combinations and text combinations having the same voice content as each audio combination; each audio combination includes at least one audio segment, each audio segment includes a speaking object, and when the speaking object in the text combination changes, an object switching point label is set;

[0017] The speech recognition model is trained using the audio combination as a training sample, the text combination corresponding to the audio combination, and the object switching point labels in the text combination corresponding to the audio combination as training labels.

[0018] Furthermore, in the above method, the step of identifying the emotion of each voice content according to the voice data unit and the text data unit corresponding to each voice content includes:

[0019] The speech data units and text data units corresponding to each speech content are input into a pre-trained emotion recognition model, so that the emotion recognition model can recognize the emotion of each speech content according to the speech data units and text data units corresponding to each speech content.

[0020] Furthermore, in the above method, the training process of the emotion recognition model includes:

[0021] Obtain multiple audio clips and text clips with the same speech content as each audio clip; each audio clip contains a speaker, and each audio clip contains an emotion label;

[0022] The emotion recognition model is trained using the audio features of the audio segment and the text features of a text segment having the same voice content as the audio segment as training samples, and the emotion label of the audio segment as a training label.

[0023] Furthermore, the above method further includes:

[0024] The acquired conversation data is preprocessed to obtain the voice data; the preprocessing includes at least one of background music separation, noise reduction, voice segment extraction, adjacent voice segment merging, and feature extraction.

[0025] In a second aspect, an embodiment of the present application provides an emotion recognition device, comprising:

[0026] A first recognition unit is used to recognize the voice data to obtain text data and object switching points corresponding to the voice data; wherein the object switching points include switching points of each speaking object in the voice data;

[0027] A segmentation unit, used for segmenting according to the object switching point to obtain a plurality of voice data units corresponding to the voice data and a plurality of text data units corresponding to the text data; wherein the voice data unit and the text data unit obtained by recognizing the voice data unit correspond to the same voice content;

[0028] The second recognition unit is used to recognize the emotion of each voice content according to the voice data unit and the text data unit corresponding to each voice content.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0030] A memory and a processor; wherein the memory is used to store programs; and the processor is used to implement any of the methods described above by running the programs in the memory.

[0031] In a fourth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the computer program implements any one of the above methods. Optionally, the computer program can be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0032] The emotion recognition method proposed in the present application can recognize speech data and obtain text data and object switching points corresponding to the speech data, wherein the object switching points include the switching points of each speaking object in the speech data. Then, segmentation is performed according to the object switching points to obtain multiple speech data units corresponding to the speech data and multiple text data units corresponding to the text data, wherein the speech data unit and the text data unit recognized by the speech data unit correspond to the same speech content. Finally, the emotion of each speech content is identified based on the speech data unit and the text data unit corresponding to each speech content. With such a setting, segmentation can be performed according to the point where the speaking object changes, and the speech data unit and the text data unit corresponding to a single speaking object can be obtained, and the emotion of the speech content of a single speaking object can be recognized from two dimensions: speech and text, so that a more accurate recognition result can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0034] Figure 1 It is a flow chart of an emotion recognition method provided in an embodiment of the present application.

[0035] Figure 2 It is a structural diagram of the speech recognition model provided in the embodiment of the present application.

[0036] Figure 3 It is a text combination diagram provided in an embodiment of the present application.

[0037] Figure 4 It is a structural diagram of a speech recognition model and an emotion recognition model provided in an embodiment of the present application.

[0038] Figure 5 It is a structural schematic diagram of an emotion recognition device provided in an embodiment of the present application.

[0039] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] Perceiving the emotions in a conversation not only enables us to better understand the speaker's true intentions, but also enables us to extract rich emotional information from the conversation, which can serve as an important sample for training large models of speech synthesis bases.

[0042] For example, the emotional companionship model can select the appropriate voice to reply to the other party according to the speaker's emotions. For example, if the other party is very lost or sad, the emotional companionship model will reply to the other party in a comforting tone.

[0043] For example, in the past, speech synthesis only needed to record the sound library of a fixed speaker, which usually took a few hours to dozens of hours, in order to accurately model the timbre and rhythm of the recorder. However, with the evolution and update of technology, speech synthesis of a specific speaker is no longer a problem in the industry. More often, users record a few sentences and then can reproduce the user's timbre and rhythm. Under this demand, if the traditional recording solution is still used, it is obviously unable to meet user demands. As AI enters the era of large models, speech synthesis large model solutions have emerged. Large models often require a large amount of high-quality data for training. Take the large speech synthesis model as an example. The mainstream industry now has hundreds of thousands of hours of base data to generate a large speech synthesis base model.

[0044] The way to obtain data is often divided into two ways: manual labeling or machine production. The quality of manually labeled data is relatively good, but the output is small, and everyone's standards are not unified; the machine production method is that the output is proportional to the number of machines invested, and it can be mass-produced with relatively unified standards. However, due to the different sources of the original data processed, different modules and processes are often required to process different data. In addition, for some more difficult original data, such as movies and TV series, there are background sounds and the speakers are constantly switching, and the data retention rate is often low. And this type of discarded data, because of continuous conversations in a short period of time, is often a fragment with rich emotions. Most of these emotions are the most difficult to obtain and the most important emotional data in speech synthesis.

[0045] Based on this, the present application proposes an emotion recognition method, device, electronic device and computer program product. The technical solution obtains speech data units and text data units corresponding to a single speaking object by segmenting according to the points where the speaking object changes, and recognizes the emotion of the speech content of a single speaking object from two dimensions: speech and text, to obtain more accurate emotion recognition results.

[0046] Furthermore, the embodiment of the present application proposes an emotion recognition method, which can be executed by an electronic device, which can be any device with data and instruction processing functions, for example, a laptop, a tablet computer, a desktop computer, a mobile device (for example, a mobile phone, a personal digital assistant, a dedicated messaging device) and other types of user terminals or a combination of any two or more of these electronic devices, or a server. Figure 1 As shown, the method includes:

[0047] S101: Recognize voice data to obtain text data and object switching points corresponding to the voice data.

[0048] Voice data refers to data that needs to be subjected to emotion recognition. It should be noted that if the results of emotion recognition are needed to judge the speaker's emotions to determine the content and tone of the response, the voice data is generally the voice content of one or more people speaking; if rich emotional information needs to be extracted from the conversation as an important sample for training the large model of the speech synthesis base, the voice data includes various conversation data, such as conversation data extracted from movies, TV series, and variety shows.

[0049] The text data corresponding to the voice data refers to the data obtained after the voice data is converted into text. The voice data and the text data essentially contain the same voice content.

[0050] The object switching point includes the switching points of each speaking object in the voice data. Specifically, in the voice data, after one speaking object finishes speaking, another speaking object starts speaking, indicating that the speaking object has switched, and the interval between the time when one speaking object finishes speaking and before the time when the other speaking object starts speaking is taken as an object switching point. For example, the voice data includes the voices of two speaking objects A and B, specifically "A: Today's weather is nice; B: Yes", where the interval between the time when A finishes speaking and before B starts speaking is an object switching point.

[0051] In an embodiment of the present application, speech data is recognized to obtain text data corresponding to the speech data and object switching points in the speech data.

[0052] In some embodiments, the speech data is recognized by using a speech recognition model to obtain text data corresponding to the speech data. The speech data can be recognized by using a mature speech recognition model in the prior art, which is not limited in this embodiment.

[0053] In some embodiments, a speech segmentation model is used to identify speech data and determine object switching points in the speech data. Specifically, a large amount of speech data can be obtained as training samples, and the object switching points in each speech data can be marked as training labels. During training, the training samples are input into the speech segmentation model to obtain the prediction results output by the speech segmentation model. By comparing the prediction results output by the speech segmentation model and the training labels, the loss of the speech segmentation model is determined. With the goal of reducing the loss of the speech segmentation model, the parameters of the speech segmentation model are adjusted, and then the above training process is repeated until the speech segmentation model meets the training requirements. The above speech segmentation model can be trained based on any neural network model, or can be trained based on a pre-trained model, such as a pre-trained large model similar to ChatGPT.

[0054] The speech data is input into the trained speech segmentation model to obtain the time at which the object switching point in the speech data is output by the speech segmentation model.

[0055] S102: Segment according to the object switching point to obtain a plurality of voice data units corresponding to the voice data and a plurality of text data units corresponding to the text data.

[0056] The object switching points obtained in the above steps are divided. In some embodiments, the voice data can be divided according to the object switching points to obtain multiple voice data units corresponding to the voice data; according to the distribution of the text data on the time axis of the voice data, the position of the object switching point in the text data is determined, and the text data is divided according to the object switching points to obtain multiple text data units corresponding to the text data.

[0057] It should be noted that the voice data unit and the text data unit recognized by the voice data unit correspond to the same voice content.

[0058] S103: Identify the emotion of each voice content according to the voice data unit and the text data unit corresponding to each voice content.

[0059] Specifically, for a voice data unit and a text data unit containing the same voice content, emotion recognition may be performed on them to obtain the emotion of the voice content.

[0060] In some embodiments, a sentiment analysis model is used to identify voice data units and text data units containing the same voice content to determine the sentiment of the voice content. Specifically, a large number of unit pairs can be obtained as training samples, wherein each unit pair includes a voice data unit and a text data unit containing the same voice content, and the sentiment corresponding to each unit pair is manually annotated as a training label. During training, the training samples are input into the sentiment analysis model to obtain the prediction results output by the sentiment analysis model, and the loss of the sentiment analysis model is determined by comparing the prediction results output by the sentiment analysis model with the training labels. With the goal of reducing the loss of the sentiment analysis model, the parameters of the sentiment analysis model are adjusted, and then the above training process is repeated until the sentiment analysis model meets the training requirements. The above sentiment analysis model can be obtained by training based on any neural network model, or can be obtained by training based on a pre-trained model, such as a pre-trained large model similar to ChatGPT.

[0061] The speech data unit and the text data unit containing the same speech content are input into the trained sentiment analysis model to obtain the sentiment corresponding to the speech content output by the sentiment analysis model.

[0062] In the above embodiment, the voice data can be recognized to obtain text data and object switching points corresponding to the voice data, wherein the object switching points include the switching points of each speaking object in the voice data. Then, segmentation is performed according to the object switching points to obtain multiple voice data units corresponding to the voice data and multiple text data units corresponding to the text data, wherein the voice data unit and the text data unit recognized by the voice data unit correspond to the same voice content. Finally, the emotion of each voice content is identified based on the voice data unit and the text data unit corresponding to each voice content. With such a setting, segmentation can be performed according to the point where the speaking object changes, to obtain the voice data unit and the text data unit corresponding to a single speaking object, and the voice content emotion of a single speaking object can be identified from both voice and text dimensions, so that a more accurate recognition result can be obtained.

[0063] Moreover, for continuous conversations in a short period of time, such as conversations in movies and TV series, it is also possible to use the information in the continuous conversations to predict the object switching points of different speakers, so that they can be segmented later, and then predict emotional labels such as happiness, sadness, etc. for the segmented data of different speakers, to obtain effective emotional data and greatly improve the retention rate of hard-to-obtain voice data processing such as emotional data.

[0064] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment recognize the voice data to obtain the text data and object switching point corresponding to the voice data, which may specifically include the following steps:

[0065] The speech data is input into a pre-trained speech recognition model so that the speech recognition model recognizes the speech data and obtains text data and object switching points corresponding to the speech data.

[0066] In the embodiment of the present application, a speech recognition model is pre-trained, and the speech recognition model can perform the task of identifying text data and object switching points corresponding to the speech data from the speech data.

[0067] In some embodiments, the speech recognition model is an end-to-end speech recognition model pre-trained using common recognition data, wherein the common recognition data includes speech-text pairs. An identifier corresponding to an object switching point is added to the output layer of the speech recognition model, for example, <sc> Because the output nodes of the speech recognition model are fixed, additional output nodes need to be added to ensure that the speech recognition model can predict whether the speaker changes.

[0068] In this embodiment, the speech recognition model includes an encoder (transformer encoder blocks) and a decoder (transformer decoder blocks), the encoder and the decoder are both stacked using transformer blocks, each transformer block includes a multilayer perceptron (Multilayer Perceptron, MLP) and Selfattention, and the encoder and the decoder use Cross attention interaction, as shown in the following example: Figure 2 shown.

[0069] The speech recognition model can be trained by the following steps:

[0070] A plurality of audio combinations and text combinations having the same voice content as the audio combinations are obtained; each audio combination includes at least one audio segment, each audio segment includes a speaking object, and when the speaking object in the text combination changes, an object switching point label is set; the speech recognition model is trained using the audio combination as a training sample, the text combination corresponding to the audio combination, and the object switching point label in the text combination corresponding to the audio combination as training labels.

[0071] Specifically, some recorded audio data with emotions can be selected from a pre-recorded personal sound library. At the same time, in order to ensure the matching of training and testing, audio data with emotions can also be manually cut out from some film and television works. In some embodiments, the audio data can be pre-processed, including background music separation (source separation), noise reduction (denoise), voice segment extraction using voice activity detection (VAD) technology, adjacent voice segment merging (Merge Segments), etc., to remove interference content in the audio data and obtain processed audio data.

[0072] Then, the manually processed audio data is segmented to ensure that each small audio data after segmentation contains only one speaker, and the small audio data containing only one speaker is defined as an audio segment. The audio segments are randomly combined, including single-person combination, two-person combination or multi-person combination, to obtain multiple audio combinations.

[0073] The text content of each audio combination can be recognized manually or by using a speech recognition device to obtain a text combination with the same speech content as each audio combination. In each text combination, if the speaker changes, an object switching point label is set when the speaker changes. The object switching point label can be represented by 〈sc〉. In addition, the corresponding timestamps when each speaker starts speaking and ends speaking can also be added. For example Figure 3 In the illustrated embodiment, text combination 1 is 0s-1.2s, and SpkA says "The weather is nice"; text combination 2 is 0s-1.5s, and SpkA says "I want to eat cookies" and 0s-0.8s, and SpkB says "Isn't it possible"; since the speaking object changes, 〈sc〉 is added to represent the object switching point label; text combination 3 is 0s-1.5s, and SpkA says "I want to eat cookies" and 0s-1.2s, and SpkA says "The weather is nice"; since the speaking object does not change, there is no need to add an object switching point label.

[0074] The audio combination is used as a training sample, the text combination corresponding to the audio combination, and the object switching point label in the text combination corresponding to the audio combination are used as training labels to train the speech recognition model. More specifically, the speech features of the audio combination can be extracted as the training sample, for example, the 40-dimensional fbank features of the audio combination are extracted by sliding the window as the speech features of the audio combination. In this embodiment, the window length is set to 25ms and the frame shift is set to 10ms.

[0075] The audio combination or the speech features of the audio combination are sent to the speech recognition model, and after softmax classification, the recognition result of the speech recognition model is obtained, and the recognition result of the speech recognition model and the training label are compared to determine the loss of the speech recognition model. The parameters of the speech recognition model are adjusted with the goal of reducing the loss of the speech recognition model, and then the above training process is repeated until the speech recognition model meets the requirements. In some embodiments, the loss function used is cross entropy loss (CrossEntry Loss).

[0076] The speech data is input into the trained speech recognition model so that the speech recognition model recognizes the speech data and obtains the text data and object switching points corresponding to the speech data.

[0077] For example, Figure 2 As shown, voice L refers to voice data, and then the voice data features are input into the voice recognition model. The encoder of the voice recognition model encodes it and sends it to the decoder. The decoder uses the input coding features and the previous input "〈s〉I want to go home" to predict and decode the coding features to obtain the text data and object switching points corresponding to the voice data, "0.0〈s〉I want to go home 1.0〈sc〉1.2No 1.5". It should be noted that inputting the previous input at the decoder can enable the decoder to understand the decoded text history, so that it can more accurately predict the next text output.

[0078] In the above embodiment, the speech recognition model can be trained, and the text data and object switching points corresponding to the speech data can be recognized at the same time, with fast recognition speed and high accuracy.

[0079] As an optional implementation, another embodiment of the present application discloses that the generation process of the multiple voice data units in the above embodiment may specifically include the following steps:

[0080] According to the object switching point, the speech features of the speech data are divided into multiple speech data units; the speech features are output by the encoder of the speech recognition model when the speech recognition model processes the speech data.

[0081] Specifically, the encoder of the speech recognition model can output the encoding features of the speech data, that is, the speech features of the speech data in this embodiment. The speech features output by the encoder can be retained, and then the speech features of the speech data can be divided into multiple speech data units according to the object switching point, and the multiple speech data units can be used as the basis for sensing the sensibility of the speaking object.

[0082] In some embodiments, the N-frame*D-dimensional features output by the encoder are temporarily saved for the next emotion prediction, thereby saving the amount of calculation. In this embodiment, D is 512.

[0083] In the above embodiment, the speech features output by the encoder can be saved for subsequent emotion prediction, and feature extraction does not need to be performed again during the emotion prediction process, which effectively saves the amount of calculation.

[0084] As an optional implementation, another embodiment of the present application discloses that the generation process of multiple text data units in the above embodiment may specifically include the following steps:

[0085] According to the object switching point, the text features of the text data are divided into multiple text data units; the text features are output by the decoder of the speech recognition model when the speech recognition model processes the speech data.

[0086] Specifically, the decoder of the speech recognition model can output features obtained by decoding the encoded features of the speech data, that is, the text features of the text data in this embodiment. The text features output by the decoder can be retained, and then the text features of the text data can be divided into multiple text data units according to the object switching point, and the multiple text data units can be used as the basis for sensing the sensibility of the speaking object.

[0087] In some embodiments, the features of the T-frame*D-dimensional output of the encoder are temporarily saved and used for the next emotion prediction, thereby saving the amount of calculation. In this embodiment, D is 512.

[0088] In the above embodiment, the text features output by the decoder can be saved for subsequent sentiment prediction, and feature extraction does not need to be performed again during the sentiment prediction process, which effectively saves the amount of calculation.

[0089] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment identify the emotion of each voice content according to the voice data unit and the text data unit corresponding to each voice content, and specifically may include the following steps:

[0090] The speech data units and text data units corresponding to each speech content are input into a pre-trained emotion recognition model, so that the emotion recognition model can identify the emotion of each speech content based on the speech data units and text data units corresponding to each speech content.

[0091] In the embodiment of the present application, an emotion recognition model is pre-trained, and the emotion recognition model can perform the task of identifying the speech data units and text data units corresponding to the same speech content and determining the emotion of the speech content.

[0092] In some embodiments, the emotion recognition model includes a decoder (transformer decoder blocks), and the decoder is formed by stacking transformer blocks, each transformer block includes MLP and Selfattention, and cross attention is used between each transformer block, such as Figure 4 shown.

[0093] The training process of the emotion recognition model includes:

[0094] Obtain multiple audio clips and text clips with the same voice content as each audio clip; each audio clip contains a speaker, and each audio clip contains an emotion label; use the audio features of the audio clips and the text features of the text clips with the same voice content as the audio clips as training samples, and use the emotion labels of the audio clips as training labels to train the emotion recognition model.

[0095] According to the above embodiment, an audio segment containing only one speaker can be obtained. The text content of each audio segment can be recognized manually or by using a speech recognition device to obtain a text segment with the same speech content as each audio segment, and an emotion tag can be manually added.

[0096] The audio features of the audio clip and the text features of the text clip with the same speech content as the audio clip are extracted as training samples, and the emotion label of the audio clip is used as the training label to train the emotion recognition model.

[0097] The training samples are sent to the emotion recognition model, and after softmax classification, the recognition results of the emotion recognition model are obtained. The recognition results of the emotion recognition model are compared with the training labels to determine the loss of the emotion recognition model. The parameters of the emotion recognition model are adjusted with the goal of reducing the loss of the emotion recognition model, and then the above training process is repeated until the emotion recognition model meets the requirements. In some embodiments, the loss function used is cross-entropy loss (CrossEntry Loss).

[0098] In some embodiments, the speech recognition model and the emotion recognition model can be trained simultaneously. Specifically, multiple audio combinations and text combinations with the same speech content as each audio combination can be obtained according to the steps recorded in the above embodiments, each audio combination includes at least one audio segment, each audio segment includes a speaking object, when the speaking object in the text combination changes, an object switching point label is set, and each audio segment is manually marked with an emotion label.

[0099] First, the audio combination is used as a training sample, and the text combination corresponding to the audio combination and the object switching point label in the text combination corresponding to the audio combination are used as training labels to train the speech recognition model. After the speech recognition model meets the training requirements, the speech recognition model is fixed, and according to the object switching points recognized by the speech recognition model, the speech features output by the speech recognition model encoder are divided into multiple speech data units, and the text features output by the speech recognition model decoder are divided into multiple text data units. The speech data units and text data units with the same speech content are used as training samples, and the corresponding emotion labels of the speech content are used as training labels to train the emotion recognition model until the emotion recognition model meets the training requirements.

[0100] In the above embodiments, the emotion recognition model can be trained to recognize emotions with fast recognition speed and high accuracy.

[0101] As an optional implementation, another embodiment of the present application discloses that the method of the above embodiment further includes, specifically, the following steps:

[0102] The acquired conversation data is preprocessed to obtain voice data; the preprocessing includes at least one of background music separation, noise reduction, voice segment extraction, adjacent voice segment merging, and feature extraction.

[0103] Specifically, after acquiring the conversation data, in order to remove interference such as background sound and noise in the conversation data, the conversation data can be preprocessed in at least one of background music separation, noise reduction, speech segment extraction, adjacent speech segment merging, and feature extraction to obtain speech data.

[0104] Specifically, background music separation refers to separating background music from conversation data; noise reduction refers to reducing or removing noise from conversation data; speech segment extraction refers to using VAD technology to detect voice activity on conversation data and extract speech segments from conversation data; and adjacent speech segment merging refers to merging speech segments with shorter intervals together as a sentence.

[0105] In a specific embodiment, after acquiring the conversation data, the conversation data is first separated from the background music to obtain the first voice without the background music, and then the first voice without the background music is subjected to noise reduction processing to obtain the second voice, and then the second voice is subjected to voice activity detection using VAD technology to extract the voice segments in the second voice, and the voice segments with shorter intervals are merged together as a sentence to obtain multiple third voices. The third voice is then subjected to feature extraction to obtain voice data, for example, a 40-dimensional fbank feature of the third voice is extracted by sliding a window as the feature of the third voice. In this embodiment, the window length is set to 25ms and the frame shift is set to 10ms.

[0106] The speech data is then input into a speech recognition model to obtain text data and object switching points output by the speech recognition model. Then, according to the object switching points recognized by the speech recognition model, the speech features output by the speech recognition model encoder are divided into multiple speech data units, and the text features output by the speech recognition model decoder are divided into multiple text data units. The speech data units and text data units are input into an emotion recognition model to obtain an emotion recognition result obtained by the emotion recognition model.

[0107] Specifically, by segmenting the original conversation data according to the object switching points identified by the speech recognition model, the speech of each speaker can be obtained. By segmenting the text data according to the object switching points identified by the speech recognition model, the speech text and the corresponding timestamp of each speaker can be obtained. According to the results of the emotion recognition model, the emotion of the speech of each speaker and the corresponding speech text can be determined. In this way, rich training data can be obtained as samples for training the large model of the speech synthesis base.

[0108] It should also be noted that in order to obtain more reliable data, the relevant data can be sent to the subsequent speech recognition module and mos scoring module, based on which the relevant data are screened and scored, and the content with high confidence is output as finished product data.

[0109] Corresponding to the above-mentioned emotion recognition method, the present application embodiment also discloses an emotion recognition device, see Figure 5 As shown, the device comprises:

[0110] The first recognition unit 100 is used to recognize the voice data to obtain text data and object switching points corresponding to the voice data; wherein the object switching points include switching points of each speaking object in the voice data;

[0111] The segmentation unit 110 is used to segment the object according to the object switching point to obtain a plurality of voice data units corresponding to the voice data and a plurality of text data units corresponding to the text data; wherein the voice data unit and the text data unit obtained by recognizing the voice data unit correspond to the same voice content;

[0112] The second recognition unit 120 is used to recognize the emotion of each voice content according to the voice data unit and the text data unit corresponding to each voice content.

[0113] As an optional implementation method, another embodiment of the present application discloses that the first recognition unit 100 of the above embodiment is specifically used to input voice data into a pre-trained voice recognition model so that the voice recognition model recognizes the voice data and obtains text data and object switching points corresponding to the voice data.

[0114] As an optional implementation, another embodiment of the present application discloses that the segmentation unit 110 in the above embodiment is specifically used for:

[0115] According to the object switching point, the speech features of the speech data are divided into multiple speech data units; the speech features are output by the encoder of the speech recognition model when the speech recognition model processes the speech data.

[0116] As an optional implementation, another embodiment of the present application discloses that the segmentation unit 110 in the above embodiment is specifically used for:

[0117] According to the object switching point, the text features of the text data are divided into multiple text data units; the text features are output by the decoder of the speech recognition model when the speech recognition model processes the speech data.

[0118] As an optional implementation, another embodiment of the present application discloses that the device in the above embodiment further includes:

[0119] A training unit is used to obtain multiple audio combinations and text combinations with the same voice content as each audio combination; each audio combination includes at least one audio segment, each audio segment contains a speaking object, and when the speaking object in the text combination changes, an object switching point label is set; the audio combination is used as a training sample, and the text combination corresponding to the audio combination and the object switching point label in the text combination corresponding to the audio combination are used as training labels to train the speech recognition model.

[0120] As an optional implementation, another embodiment of the present application discloses that the second identification unit 120 in the above embodiment is specifically used for:

[0121] The speech data units and text data units corresponding to each speech content are input into a pre-trained emotion recognition model, so that the emotion recognition model can identify the emotion of each speech content based on the speech data units and text data units corresponding to each speech content.

[0122] As an optional implementation, another embodiment of the present application discloses that the training unit in the above embodiment is further used for:

[0123] Obtain multiple audio clips and text clips with the same voice content as each audio clip; each audio clip contains a speaker, and each audio clip contains an emotion label; use the audio features of the audio clips and the text features of the text clips with the same voice content as the audio clips as training samples, and use the emotion labels of the audio clips as training labels to train the emotion recognition model.

[0124] As an optional implementation, another embodiment of the present application discloses that the device in the above embodiment further includes:

[0125] The pre-training unit is used to pre-process the acquired dialogue data to obtain voice data; the pre-processing includes at least one of background music separation, noise reduction, voice segment extraction, adjacent voice segment merging, and feature extraction.

[0126] Specifically, the device provided in this embodiment belongs to the same application concept as the method provided in the above embodiment of this application, can execute the method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not fully described in this embodiment, please refer to the specific processing content of the method provided in the above embodiment of this application, which will not be repeated here.

[0127] The functions implemented by the above units can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0128] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0129] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0130] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0131] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0132] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input-output component through the interface circuit of the control device.

[0133] The input-output component specifically refers to a hardware component that enables a user to input information and output information to the user, such as a microphone, keyboard, handwriting tablet, touch screen, display, speaker, printer, etc.

[0134] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0135] The processor in the control device is a circuit with signal processing capability, which obtains a more accurate recognition result by executing any one of the emotion recognition methods introduced in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of this application are not strictly limited.

[0136] When the control device is applied to a device with a human-computer interaction function, the input and output components of the control device may be input components and output components on the device, such as a microphone, a keyboard, a handwriting tablet, a touch screen, a display, an audio player, etc. At the same time, the processor of the control device may be a CPU or GPU, etc. provided by the device, and the interface circuit of the control device may be an interface circuit between the information input component of the device and a processor such as a CPU or GPU.

[0137] Corresponding to the above emotion recognition method, the present application embodiment also discloses an electronic device, see Figure 6 As shown, the electronic device includes:

[0138] Memory 200 and processor 210;

[0139] The memory 200 is connected to the processor 210 and is used to store programs;

[0140] The processor 210 is used to implement the emotion recognition method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0141] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0142] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0143] A bus may include a pathway that transfers information between components of a computer system.

[0144] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0145] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0146] The memory 200 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0147] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0148] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0149] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0150] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the emotion recognition method provided in the above embodiments of the present application.

[0151] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes a computer program. When the computer program is executed by a processor, the emotion recognition method provided by any of the above embodiments of the present application may be executed. Optionally, the computer program may be stored in a readable storage medium or in the cloud of a computer device; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0152] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0154] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the various steps of the emotion recognition method provided in the above embodiment.

[0155] The computer readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0156] Specifically, the specific working contents of each part of the above-mentioned electronic device, computer program product and storage medium, as well as the specific processing contents when the computer program product or the computer program on the above-mentioned storage medium is executed by the processor, can all be found in the contents of the various embodiments of the above-mentioned emotion recognition method, and will not be repeated here.

[0157] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0158] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0159] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0160] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0162] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0164] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0165] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0166] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0167] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An emotion recognition method, characterized in that: include: Recognize the voice data to obtain text data and object switching points corresponding to the voice data; wherein the object switching points include switching points of each speaking object in the voice data; Segmenting according to the object switching point to obtain a plurality of voice data units corresponding to the voice data and a plurality of text data units corresponding to the text data; wherein the voice data unit and the text data unit recognized by the voice data unit correspond to the same voice content; The emotion of each voice content is identified based on the voice data unit and the text data unit corresponding to each voice content.

2. The method according to claim 1, characterized in that The step of recognizing the voice data to obtain text data and object switching points corresponding to the voice data includes: The speech data is input into a pre-trained speech recognition model so that the speech recognition model recognizes the speech data and obtains text data and object switching points corresponding to the speech data.

3. The method according to claim 2, characterized in that The generation process of the plurality of voice data units includes: According to the object switching point, the speech features of the speech data are divided into a plurality of speech data units; the speech features are output by the encoder of the speech recognition model when the speech recognition model processes the speech data.

4. The method according to claim 2, characterized in that: The generation process of the plurality of text data units includes: According to the object switching point, the text features of the text data are divided into a plurality of text data units; the text features are output by the decoder of the speech recognition model when the speech recognition model processes the speech data.

5. The method according to claim 2, characterized in that: The training process of the speech recognition model includes: Acquire multiple audio combinations and text combinations having the same voice content as each audio combination; each audio combination includes at least one audio segment, each audio segment includes a speaking object, and when the speaking object in the text combination changes, an object switching point label is set; The speech recognition model is trained using the audio combination as a training sample, the text combination corresponding to the audio combination, and the object switching point labels in the text combination corresponding to the audio combination as training labels.

6. The method according to claim 1, characterized in that The step of identifying the emotion of each voice content according to the voice data unit and the text data unit corresponding to each voice content comprises: The speech data units and text data units corresponding to each speech content are input into a pre-trained emotion recognition model, so that the emotion recognition model can recognize the emotion of each speech content according to the speech data units and text data units corresponding to each speech content.

7. The method according to claim 6, characterized in that The training process of the emotion recognition model includes: Obtain multiple audio clips and text clips with the same speech content as each audio clip; each audio clip contains a speaker, and each audio clip contains an emotion label; The emotion recognition model is trained using the audio features of the audio segment and the text features of a text segment having the same voice content as the audio segment as training samples, and the emotion label of the audio segment as a training label.

8. The method according to claim 1, characterized in that Also includes: Preprocessing the acquired conversation data to obtain the voice data; The preprocessing includes at least one of background music separation, noise reduction, speech segment extraction, adjacent speech segment merging, and feature extraction.

9. An emotion recognition device, characterized in that: include: A first recognition unit is used to recognize the voice data to obtain text data and object switching points corresponding to the voice data; wherein the object switching points include switching points of each speaking object in the voice data; A segmentation unit, used for segmenting according to the object switching point to obtain a plurality of voice data units corresponding to the voice data and a plurality of text data units corresponding to the text data; wherein the voice data unit and the text data unit obtained by recognizing the voice data unit correspond to the same voice content; The second recognition unit is used to recognize the emotion of each voice content according to the voice data unit and the text data unit corresponding to each voice content.

10. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images