Emotional data expansion method, device, electronic device and storage medium
By fine-tuning the speech synthesis model on the initial emotion dataset and generating expanded training text, the problem of insufficient data in the speech emotion recognition system is solved, multimodal emotion data expansion is achieved, and the generalization and robustness of the model are improved.
Patent Information
- Application Number
- CN202410446716.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-12
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-04-12
AI Technical Summary
Existing speech emotion recognition systems face challenges in labeling emotion data, leading to a decrease in the accuracy of the model in recognizing the emotions of different speakers. Furthermore, speech emotion recognition has a certain degree of subjectivity, making it difficult to expand multimodal emotion data to improve generalization and robustness.
By obtaining an initial sentiment dataset, a speech synthesis model is used to fine-tune it, generating expanded training text and audio that are semantically similar to the initial training text. Sentiment data is then expanded by combining sentiment tags, and a multimodal sentiment recognition model is trained.
It enables the expansion based on a small amount of sentiment data, improving the generalization and robustness of the sentiment recognition model, and can be expanded from single-modal to multi-modal sentiment data.
Smart Images

Figure CN118155664B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for expanding emotion data. Background Art
[0002] Speech is one of the main communication media in daily life. It not only conveys thoughts but also expresses the speaker's emotional state. Speech emotion recognition can detect the user's mental health status and provide corresponding feedback and responses based on the user's emotional changes in different scenarios (for example, digital humans, customer service, etc.). In addition, speech emotion recognition can also help parents understand their children's mental health status.
[0003] However, in emotion recognition systems, labeling emotion data is a challenging task, which makes the emotion recognition model more susceptible to the emotions of the speakers in the training set. In other words, if we use different speakers from those in the training set during the test phase, the system often produces incorrect emotion recognition results. In addition, speech emotion recognition is somewhat subjective because different users may have different emotional understandings of the same sentence. That is, some people may think that a sentence carries a special or strong emotion, while others may think that it expresses a calm emotion. Summary of the invention
[0004] The present disclosure provides a technical solution for expanding emotion data.
[0005] According to one aspect of the present disclosure, a method for expanding sentiment data is provided, comprising:
[0006] Obtaining an initial emotion data set, wherein the initial emotion data set includes multiple items of initial emotion data corresponding to multiple speakers, and any item of initial emotion data includes initial training audio, initial training text, and an emotion label;
[0007] Using the initial emotion dataset, fine-tuning the speech synthesis model;
[0008] Obtaining an expanded training text that is semantically similar to the initial training text;
[0009] Inputting the expanded training text and the initial training audio corresponding to the expanded training text into the fine-tuned speech synthesis model, and obtaining the expanded training audio corresponding to the expanded training text through the fine-tuned speech synthesis model;
[0010] The expanded emotion data corresponding to the initial emotion data is obtained according to the expanded training text, the expanded training audio and the emotion label.
[0011] In a possible implementation, the initial emotion data further includes a speaker label;
[0012] The method of obtaining the expanded emotion data corresponding to the initial emotion data based on the expanded training text, the expanded training audio and the emotion label comprises: obtaining the expanded emotion data corresponding to the initial emotion data based on the expanded training text, the expanded training audio, the speaker label and the emotion label.
[0013] In a possible implementation, the adopting the initial emotion dataset to fine-tune the speech synthesis model includes:
[0014] For any item of initial emotion data, dividing initial training audio in the initial emotion data into a first audio segment and a second audio segment;
[0015] Inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting a synthesized audio corresponding to a second text segment in the initial training text through the speech synthesis model, wherein the second text segment represents a text segment corresponding to the second audio segment in the initial training text;
[0016] The speech synthesis model is fine-tuned according to difference information between the synthesized audio and the second audio segment.
[0017] In a possible implementation, inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting synthesized audio corresponding to the second text segment in the initial training text through the speech synthesis model includes:
[0018] The first audio segment, the initial training text in the initial emotion data, and the speaker information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs the synthesized audio corresponding to the second text segment in the initial training text.
[0019] In a possible implementation, inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting synthesized audio corresponding to the second text segment in the initial training text through the speech synthesis model includes:
[0020] The first audio segment, the initial training text in the initial emotion data, and the emotion information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs a synthesized audio corresponding to a second text segment in the initial training text.
[0021] In a possible implementation, obtaining an expanded training text that is semantically similar to the initial training text includes:
[0022] Acquire a candidate text set, wherein the candidate text set includes a plurality of candidate texts;
[0023] Determining the semantic similarity between the initial training text and the candidate text through a semantic similarity matching model;
[0024] The candidate texts whose semantic similarity with the initial training text is greater than or equal to a preset threshold are determined as the expanded training texts corresponding to the initial training texts.
[0025] In a possible implementation, obtaining an expanded training text that is semantically similar to the initial training text includes:
[0026] The initial training text is transcribed by using a semantic transcription model to obtain an expanded training text corresponding to the initial training text.
[0027] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes:
[0028] For any one of the emotion data in the expanded emotion data set, the training audio in the emotion data is input into the speaker recognition model, and the speaker recognition model outputs the speaker prediction result corresponding to the training audio; wherein the expanded emotion data set includes the initial emotion data and the expanded emotion data; the speaker recognition model includes a speaker encoder, a speaker decoder and a classifier;
[0029] The speaker recognition model is trained according to the speaker prediction result corresponding to the training audio and the speaker label in the emotion data.
[0030] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes:
[0031] The text encoder is trained by using the training text in the sentiment data in the expanded sentiment dataset.
[0032] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes:
[0033] The audio encoder is trained by using the training audio in the emotional data in the extended emotional dataset.
[0034] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes:
[0035] The text emotion recognition model is trained by using the training texts in the expanded emotion data set and the emotion labels corresponding to the training texts, wherein the text emotion recognition model includes a text encoder and a text decoder.
[0036] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes:
[0037] The speech emotion recognition model is trained by using the training audio in the expanded emotion data set and the emotion label corresponding to the training audio, wherein the speech emotion recognition model includes an audio encoder and an audio decoder.
[0038] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes:
[0039] The multimodal emotion recognition model is trained using the training audio in the expanded emotion data set, the training text corresponding to the training audio, and the emotion label corresponding to the training audio.
[0040] In a possible implementation, the multimodal emotion recognition model includes an audio encoder, a text encoder, and a multimodal decoder;
[0041] The method of training the multimodal emotion recognition model by using the training audio in the expanded emotion data set, the training text corresponding to the training audio, and the emotion label corresponding to the training audio includes:
[0042] For any one item of emotion data in the expanded emotion data set, extracting an audio feature vector corresponding to the training audio in the emotion data through the audio encoder, and extracting a text feature vector corresponding to the training text in the emotion data through the text encoder;
[0043] Inputting the audio feature vector and the text feature vector into the multimodal decoder, and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding;
[0044] The multimodal emotion recognition model is trained according to the emotion tags in the emotion data and the emotion category prediction results corresponding to the training audio and the training text.
[0045] In a possible implementation, the multimodal emotion recognition model further includes a speaker encoder;
[0046] The method further comprises: extracting a speaker feature vector corresponding to the training audio through the speaker encoder;
[0047] The step of inputting the audio feature vector and the text feature vector into the multimodal decoder and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding comprises: inputting the speaker feature vector, the audio feature vector and the text feature vector into the multimodal decoder and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding.
[0048] According to one aspect of the present disclosure, there is provided an emotion data expansion device, comprising:
[0049] A first acquisition module is used to obtain an initial emotion data set, wherein the initial emotion data set includes multiple items of initial emotion data corresponding to multiple speakers, and any item of initial emotion data includes initial training audio, initial training text and emotion label;
[0050] A fine-tuning module, used to fine-tune the speech synthesis model using the initial emotion dataset;
[0051] A second acquisition module is used to obtain an expanded training text that is semantically similar to the initial training text;
[0052] A third acquisition module is used to input the expanded training text and the initial training audio corresponding to the expanded training text into the fine-tuned speech synthesis model, and obtain the expanded training audio corresponding to the expanded training text through the fine-tuned speech synthesis model;
[0053] The fourth acquisition module is used to obtain the expanded emotion data corresponding to the initial emotion data according to the expanded training text, the expanded training audio and the emotion label.
[0054] In a possible implementation, the initial emotion data further includes a speaker label;
[0055] The fourth acquisition module is used to obtain the expanded emotion data corresponding to the initial emotion data according to the expanded training text, the expanded training audio, the speaker label and the emotion label.
[0056] In a possible implementation, the fine-tuning module is used to:
[0057] For any item of initial emotion data, dividing initial training audio in the initial emotion data into a first audio segment and a second audio segment;
[0058] Inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting a synthesized audio corresponding to a second text segment in the initial training text through the speech synthesis model, wherein the second text segment represents a text segment corresponding to the second audio segment in the initial training text;
[0059] The speech synthesis model is fine-tuned according to difference information between the synthesized audio and the second audio segment.
[0060] In a possible implementation, the fine-tuning module is used to:
[0061] The first audio segment, the initial training text in the initial emotion data, and the speaker information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs the synthesized audio corresponding to the second text segment in the initial training text.
[0062] In a possible implementation, the fine-tuning module is used to:
[0063] The first audio segment, the initial training text in the initial emotion data, and the emotion information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs a synthesized audio corresponding to a second text segment in the initial training text.
[0064] In a possible implementation manner, the second obtaining module is used to:
[0065] Acquire a candidate text set, wherein the candidate text set includes a plurality of candidate texts;
[0066] Determining the semantic similarity between the initial training text and the candidate text through a semantic similarity matching model;
[0067] The candidate texts whose semantic similarity with the initial training text is greater than or equal to a preset threshold are determined as the expanded training texts corresponding to the initial training texts.
[0068] In a possible implementation manner, the second obtaining module is used to:
[0069] The initial training text is transcribed by a semantic transcription model to obtain an expanded training text corresponding to the initial training text.
[0070] In a possible implementation manner, the device further includes a first training module, which is used to:
[0071] For any one of the emotion data in the expanded emotion data set, the training audio in the emotion data is input into the speaker recognition model, and the speaker recognition model outputs the speaker prediction result corresponding to the training audio; wherein the expanded emotion data set includes the initial emotion data and the expanded emotion data; the speaker recognition model includes a speaker encoder, a speaker decoder and a classifier;
[0072] The speaker recognition model is trained according to the speaker prediction result corresponding to the training audio and the speaker label in the emotion data.
[0073] In a possible implementation manner, the apparatus further includes a second training module, configured to:
[0074] The text encoder is trained by using the training text in the sentiment data in the expanded sentiment dataset.
[0075] In a possible implementation manner, the apparatus further includes a third training module, configured to:
[0076] The audio encoder is trained by using the training audio in the emotional data in the extended emotional dataset.
[0077] In a possible implementation manner, the apparatus further includes a fourth training module, configured to:
[0078] The text emotion recognition model is trained by using the training texts in the expanded emotion data set and the emotion labels corresponding to the training texts, wherein the text emotion recognition model includes a text encoder and a text decoder.
[0079] In a possible implementation manner, the apparatus further includes a fifth training module, configured to:
[0080] The speech emotion recognition model is trained by using the training audio in the expanded emotion data set and the emotion label corresponding to the training audio, wherein the speech emotion recognition model includes an audio encoder and an audio decoder.
[0081] In a possible implementation manner, the apparatus further includes a sixth training module, configured to:
[0082] The multimodal emotion recognition model is trained using the training audio in the expanded emotion data set, the training text corresponding to the training audio, and the emotion label corresponding to the training audio.
[0083] In a possible implementation, the multimodal emotion recognition model includes an audio encoder, a text encoder, and a multimodal decoder;
[0084] The sixth training module is used for:
[0085] For any one item of emotion data in the expanded emotion data set, extracting an audio feature vector corresponding to the training audio in the emotion data through the audio encoder, and extracting a text feature vector corresponding to the training text in the emotion data through the text encoder;
[0086] Inputting the audio feature vector and the text feature vector into the multimodal decoder, and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding;
[0087] The multimodal emotion recognition model is trained according to the emotion tags in the emotion data and the emotion category prediction results corresponding to the training audio and the training text.
[0088] In a possible implementation, the multimodal emotion recognition model further includes a speaker encoder;
[0089] The device further comprises: an extraction module, configured to extract a speaker feature vector corresponding to the training audio through the speaker encoder;
[0090] The sixth training module is used to: input the speaker feature vector, the audio feature vector and the text feature vector into the multimodal decoder, and obtain the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding.
[0091] According to one aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.
[0092] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.
[0093] According to one aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.
[0094] In the disclosed embodiment, an initial emotion data set is obtained, wherein the initial emotion data set includes multiple initial emotion data corresponding to multiple speakers, and any one of the initial emotion data includes initial training audio, initial training text, and emotion tags. The initial emotion data set is used to fine-tune the speech synthesis model to obtain an extended training text with similar semantics to the initial training text. The extended training text and the initial training audio corresponding to the extended training text are input into the fine-tuned speech synthesis model, and the extended training audio corresponding to the extended training text is obtained through the fine-tuned speech synthesis model. The extended emotion data corresponding to the initial emotion data is obtained based on the extended training text, the extended training audio, and the emotion tags. Thus, the extended emotion data can be expanded based on a small amount of emotion data, and can be expanded from single-modal emotion data to multi-modal emotion data. Training the emotion recognition model based on the extended emotion data is conducive to improving the generalization and robustness of the emotion recognition model.
[0095] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
[0096] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0097] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.
[0098] Figure 1 A flow chart of the emotion data expansion method provided by an embodiment of the present disclosure is shown.
[0099] Figure 2 A schematic diagram of a speaker recognition model provided by an embodiment of the present disclosure is shown.
[0100] Figure 3 A schematic diagram of a text emotion recognition model provided by an embodiment of the present disclosure is shown.
[0101] Figure 4 A schematic diagram of a speech emotion recognition model provided by an embodiment of the present disclosure is shown.
[0102] Figure 5 A schematic diagram of a multimodal emotion recognition model provided by an embodiment of the present disclosure is shown.
[0103] Figure 6 Another schematic diagram of the multimodal emotion recognition model provided by an embodiment of the present disclosure is shown.
[0104] Figure 7A block diagram of the emotion data expansion device provided by an embodiment of the present disclosure is shown.
[0105] Figure 8 A block diagram of an electronic device 1900 provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0106] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0107] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0108] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.
[0109] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0110] Since speech emotion data is highly subjective, labeling is very difficult, so the size of currently available speech emotion datasets is relatively small. In order to obtain a large amount of emotion data that can be used to train emotion recognition models of different modalities, expanding emotion data is an important research topic.
[0111] In the related technology, the emotional data is mainly expanded by methods based on self-supervised learning or methods based on data enhancement. Among them, in the method based on self-supervised learning, the effect of the model determines the upper limit of the expansion of emotional data, and it is impossible to obtain multimodal expanded data. In the method based on data enhancement, the enhanced data itself is repeated (for example, the audio is the same, but the speaking speed or signal-to-noise ratio is different). Therefore, this method cannot fully improve the generalization and robustness of the model. Moreover, the method based on data enhancement cannot obtain multimodal expanded data.
[0112] In order to solve the technical problems similar to those described above, the embodiments of the present disclosure provide a method, device, electronic device and storage medium for expanding emotion data, by obtaining an initial emotion data set, wherein the initial emotion data set includes multiple initial emotion data corresponding to multiple speakers, and any initial emotion data includes initial training audio, initial training text and emotion label, using the initial emotion data set to fine-tune the speech synthesis model, obtain an expanded training text with semantics similar to the initial training text, input the expanded training text and the initial training audio corresponding to the expanded training text into the fine-tuned speech synthesis model, obtain the expanded training audio corresponding to the expanded training text through the fine-tuned speech synthesis model, and obtain the expanded emotion data corresponding to the initial emotion data based on the expanded training text, the expanded training audio and the emotion label, thereby being able to expand based on a small amount of emotion data, and being able to expand from single-modal emotion data to multi-modal emotion data. Training the emotion recognition model based on the emotion data thus expanded is conducive to improving the generalization and robustness of the emotion recognition model.
[0113] The emotion data expansion method provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.
[0114] Figure 1 A flow chart of the emotion data expansion method provided by an embodiment of the present disclosure is shown. In one possible implementation, the executor of the emotion data expansion method may be an emotion data expansion device. For example, the emotion data expansion method may be executed by a terminal device or a server or other electronic device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the emotion data expansion method may be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the emotion data expansion method includes steps S11 to S15.
[0115] In step S11, an initial emotion data set is obtained, wherein the initial emotion data set includes multiple items of initial emotion data corresponding to multiple speakers, and any item of initial emotion data includes initial training audio, initial training text and an emotion label.
[0116] In step S12, the initial emotion data set is used to fine-tune the speech synthesis model.
[0117] In step S13, an expanded training text having semantic similarity to the initial training text is obtained.
[0118] In step S14, the expanded training text and the initial training audio corresponding to the expanded training text are input into the fine-tuned speech synthesis model, and the expanded training audio corresponding to the expanded training text is obtained through the fine-tuned speech synthesis model.
[0119] In step S15, the expanded emotion data corresponding to the initial emotion data is obtained according to the expanded training text, the expanded training audio and the emotion tag.
[0120] In the embodiments of the present disclosure, the emotion data may represent data used to train an emotion recognition model.
[0121] In one possible implementation, open-source speech emotion data can be collected, or speech emotion data can be crawled or annotated based on the web. The speech emotion data can be screened to select audio of multiple emotion categories corresponding to multiple speakers. For example, the VAD (Valence-Arousal-Dominance) model can be used to screen the speech emotion data to obtain audio with louder volume and better sound quality, thereby obtaining the initial training audio.
[0122] In a possible implementation, speech recognition may be performed on the initial training audio to obtain an initial training text corresponding to the initial training audio.
[0123] In one example, for a certain speaker SPKi and a certain emotion category EMOj, the initial training audio may include {wav1, wav2, ..., wavN}, and the initial training text may include {text1, text2, ..., textN}, wherein the initial training audio corresponds to the initial training text one by one.
[0124] According to the initial training audio and initial training text of different emotion categories corresponding to different speakers, multiple emotion data corresponding to different speakers can be obtained, so as to obtain an initial emotion data set. Among them, any initial emotion data includes initial training audio, initial training text and emotion label. The emotion label in any initial emotion data can be determined according to the emotion category corresponding to the initial training audio and initial training text in the initial emotion data. In a possible implementation, any initial emotion data also includes a speaker label.
[0125] In the disclosed embodiment, after obtaining the initial emotion data set, the initial emotion data set can be used to fine-tune the speech synthesis model. The speech synthesis model can be an open source TTS (Text-to-Speech) large model, such as VALLE, VALLE 2.0, etc., which is not limited here. In some application scenarios, the speech synthesis model can also be called a sound synthesis model, a text-to-speech model, a text-to-speech model, a speech generation model, etc., which is not limited here.
[0126] In a possible implementation, the label corresponding to a certain timbre TTSk may be a certain speaker and a certain fixed emotion category. For example, the speaker label corresponding to the timbre TTSk is SPKi, the emotion label is EMOj, and the initial emotion data subset corresponding to the timbre TTSk may be {SPKi_EMOj,{wav1,wav2,…,wavN},{text1,text2,…,textN}}. For example, when there are K speakers and M emotion categories, there are K×M timbre types.
[0127] In a possible implementation, the initial emotion data set is used to fine-tune the speech synthesis model, including: for any item of initial emotion data, dividing the initial training audio in the initial emotion data into a first audio segment and a second audio segment; inputting the first audio segment and the initial training text in the initial emotion data into the speech synthesis model, and outputting the synthesized audio corresponding to the second text segment in the initial training text through the speech synthesis model, wherein the second text segment represents the text segment corresponding to the second audio segment in the initial training text; and fine-tuning the speech synthesis model based on the difference information between the synthesized audio and the second audio segment.
[0128] For example, a certain initial emotion data includes an initial training audio wav1 and an initial training text text1. The initial training audio wav1 can be divided into a first audio segment and a second audio segment, wherein the first audio segment can be the first 3 seconds of the initial training audio wav1, and the second audio segment can be the remaining part of the initial training audio wav1. Correspondingly, the initial training text text1 can be divided into a first text segment and a second text segment, wherein the first text segment can represent the text segment corresponding to the first audio segment in the initial training text text1, and the second text segment can represent the text segment corresponding to the second audio segment in the initial training text text1. The first audio segment and the initial training text text1 can be input into a speech synthesis model, and the speech synthesis model outputs a synthesized audio corresponding to the second text segment. The value of the loss function corresponding to the speech synthesis model can be determined based on the difference information between the synthesized audio and the second audio segment, and the speech synthesis model can be fine-tuned based on the value of the loss function corresponding to the speech synthesis model.
[0129] By adopting this implementation method, the speech synthesis model can learn the ability to output audio with the audio style of the timbre in the initial training audio.
[0130] In a possible implementation, the step of inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting the synthesized audio corresponding to the second text segment in the initial training text through the speech synthesis model, includes: inputting the first audio segment, the initial training text in the initial emotion data, and the speaker information corresponding to the initial emotion data into a speech synthesis model, and outputting the synthesized audio corresponding to the second text segment in the initial training text through the speech synthesis model.
[0131] In this implementation, the speaker information corresponding to the initial emotion data can also be input into the speech synthesis model, thereby helping the speech synthesis model to more efficiently learn the ability to distinguish different timbres and output audio with different timbres.
[0132] In a possible implementation, the first audio segment and the initial training text in the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs the synthesized audio corresponding to the second text segment in the initial training text, including: inputting the first audio segment, the initial training text in the initial emotion data, and the emotion information corresponding to the initial emotion data into a speech synthesis model, and the speech synthesis model outputs the synthesized audio corresponding to the second text segment in the initial training text.
[0133] In this implementation, the emotional information corresponding to the initial emotional data can also be input into the speech synthesis model, thereby helping the speech synthesis model to more efficiently learn the ability to distinguish different timbres and output audio with different timbres.
[0134] In a possible implementation, the step of inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting the synthesized audio corresponding to the second text segment in the initial training text through the speech synthesis model, includes: inputting the first audio segment, the initial training text in the initial emotion data, the speaker information corresponding to the initial emotion data, and the emotion information corresponding to the initial emotion data into the speech synthesis model, and outputting the synthesized audio corresponding to the second text segment in the initial training text through the speech synthesis model.
[0135] In a possible implementation, the speech synthesis model is also used to output the emotion category prediction result corresponding to the initial training text; and fine-tuning the speech synthesis model according to the difference information between the synthesized audio and the second audio segment includes: fine-tuning the speech synthesis model according to the difference information between the synthesized audio and the second audio segment, and the difference information between the emotion category prediction result and the emotion label.
[0136] In a possible implementation, the speech synthesis model is also used to output a speaker prediction result corresponding to the initial training text; and fine-tuning the speech synthesis model according to the difference information between the synthesized audio and the second audio segment includes: fine-tuning the speech synthesis model according to the difference information between the synthesized audio and the second audio segment, and the difference information between the speaker prediction result and the speaker label.
[0137] In a possible implementation, the speech synthesis model is also used to output an emotion category prediction result corresponding to the initial training text and a speaker prediction result corresponding to the initial training text; and fine-tuning the speech synthesis model according to the difference information between the synthesized audio and the second audio segment includes: fine-tuning the speech synthesis model according to the difference information between the synthesized audio and the second audio segment, the difference information between the emotion category prediction result and the emotion label, and the difference information between the speaker prediction result and the speaker label.
[0138] In an embodiment of the present disclosure, after the fine-tuning of the speech synthesis model is completed, an expanded training text corresponding to the initial training text can be obtained. Among them, the expanded training text corresponding to any initial training text can represent a training text with similar semantics to the initial training text. In an embodiment of the present disclosure, an expanded training text can be obtained for each initial training text. And, at least one expanded training text can be obtained for each initial training text. For example, multiple expanded training texts can be obtained for each initial training text.
[0139] In a possible implementation, obtaining an expanded training text that is semantically similar to the initial training text includes: obtaining a candidate text set, wherein the candidate text set includes multiple candidate texts; determining the semantic similarity between the initial training text and the candidate text through a semantic similarity matching model; and determining the candidate text whose semantic similarity to the initial training text is greater than or equal to a preset threshold as the expanded training text corresponding to the initial training text.
[0140] In this implementation, a large amount of text can be crawled from the Internet as candidate texts to obtain a candidate text set. The semantic similarity between the initial training text and the candidate text can be determined by a semantic similarity matching model.
[0141] Among them, the semantic similarity matching model can represent a model for measuring the semantic similarity between two texts. The semantic similarity matching model can be based on deep learning technology, extracting semantic information from the text by learning a large amount of text data, and comparing the similarity between two texts. The semantic similarity matching model can be based on the cosine similarity of the bag-of-words model, the cosine similarity of word vectors based on the Word2Vec algorithm, and the sentence-level and paragraph-level semantic matching based on the BERT pre-training model. These semantic similarity matching models calculate the similarity between two texts by converting the text into a vector or matrix and other forms, and using mathematical and machine learning algorithms.
[0142] In this implementation, if the semantic similarity between any candidate text and any initial training text is greater than or equal to a preset threshold, the candidate text may be determined as the expanded training text corresponding to the initial training text.
[0143] In a possible implementation, obtaining the extended training text that is semantically similar to the initial training text includes: transcribing the initial training text through a semantic transcription model to obtain the extended training text corresponding to the initial training text.
[0144] Among them, the semantic transcription model can represent a model that converts natural language text into another form or another language. The semantic transcription model can be based on deep learning technology, and by learning a large amount of text data, it can grasp the semantic and grammatical relationship between languages, thereby realizing automatic translation or conversion of text. Common semantic transcription models include machine translation models based on neural networks, text generation models based on encoder-decoder architecture, etc. These semantic transcription models can convert input text into vectors or matrices, etc., and use neural network algorithms and self-attention mechanisms to generate output text with the same or similar semantics as the input text.
[0145] In this implementation, an extended training text having similar semantics to the initial training text but different content may be generated through a semantic transcription model.
[0146] In the disclosed embodiment, the extended training text is consistent with the speaker style and emotion of the corresponding initial training text and initial training audio. In one example, the extended training text can be represented by {text_fake1, text_fake2, ..., text_fakeK}.
[0147] In the disclosed embodiment, after obtaining the expanded training text, the expanded training text and the initial training audio corresponding to the expanded training text can be input into the fine-tuned speech synthesis model, and the expanded training audio corresponding to the expanded training text can be obtained through the fine-tuned speech synthesis model. In addition, the emotion label corresponding to the initial training text corresponding to the expanded training text can be determined as the emotion label corresponding to the expanded training text.
[0148] In a possible implementation, the initial emotion data also includes a speaker label; and obtaining the expanded emotion data corresponding to the initial emotion data based on the expanded training text, the expanded training audio and the emotion label includes: obtaining the expanded emotion data corresponding to the initial emotion data based on the expanded training text, the expanded training audio, the speaker label and the emotion label.
[0149] In this implementation, any item of extended emotion data may further include a speaker label. The speaker label in the initial emotion data corresponding to the extended emotion data may be determined as the speaker label in the extended emotion data.
[0150] In an example, the extended emotion data subset corresponding to a speaker SPKi can be expressed as {SPKi:{EMO1:{wav1,wav2,…wavN,wav_fake1,wav_fake2,…,wav_fakeM},{text1,text2,…,textN,text_fake1,text_fake2,…,text_fakeM}},{EMO2:…},…,{EMOk:…}}.
[0151] By adopting the embodiments of the present disclosure, it is possible to expand the emotion data of different timbres SPKi_EMOj respectively.
[0152] According to the initial emotion data and the expanded emotion data, an expanded emotion data set may be obtained, wherein the expanded emotion data set may include the initial emotion data and the expanded emotion data.
[0153] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: for any emotion data in the expanded emotion data set, inputting the training audio in the emotion data into a speaker recognition model, and outputting the speaker prediction result corresponding to the training audio through the speaker recognition model; wherein the expanded emotion data set includes the initial emotion data and the expanded emotion data; the speaker recognition model includes a speaker encoder, a speaker decoder and a classifier; and training the speaker recognition model according to the speaker prediction result corresponding to the training audio and the speaker label in the emotion data.
[0154] In this implementation, the emotional data in the extended emotional data set can be initial emotional data or extended emotional data. When the emotional data is initial emotional data, the training audio in the emotional data is initial training audio, and the training text in the emotional data is initial training text. When the emotional data is extended emotional data, the training audio in the emotional data is extended training audio, and the training text in the emotional data is extended training text.
[0155] Figure 2 FIG. 1 is a schematic diagram showing a speaker recognition model provided by an embodiment of the present disclosure. Figure 2 As shown, the speaker recognition model may include a speaker encoder, a speaker decoder and a classifier. The speaker encoder may represent an encoder in the speaker recognition model, and the speaker decoder may represent a decoder in the speaker recognition model. The speaker recognition model may adopt different model structures.
[0156] The output of the speaker encoder can be recorded as a speaker feature vector. In some application scenarios, the speaker encoder can also be called a voiceprint embedding extractor, a speaker feature vector extractor, etc., which is not limited here. After the speaker recognition model training is completed, the speaker encoder can be selected as the module for extracting the speaker feature vector. In an example, the pre-trained speaker encoder can be represented by S_encoder.
[0157] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: using training text in the emotion data in the expanded emotion data set to train a text encoder.
[0158] In this implementation, the training text in the sentiment data in the expanded sentiment data set can be used to pre-train a language model (such as a BERT model). The pre-trained language model can be used as a text encoder. In some application scenarios, the text encoder can also be called a text embedding extractor, a text feature vector extractor, etc., which are not limited here. In this implementation, in the process of pre-training the language model, a part of the text can be masked, and then the masked part can be predicted.
[0159] In this implementation, in the process of pre-training the language model, texts other than the training texts in the expanded sentiment data set may also be used, which is not limited here. For example, a large amount of texts crawled from the web may be used to pre-train the language model.
[0160] In this implementation, the output of the classifier of the language model can be used as a text feature vector. In one example, the text encoder obtained by pre-training can be represented by T_encoder.
[0161] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: using training audio in the emotion data in the expanded emotion data set to train an audio encoder.
[0162] In this implementation, the training audio in the emotion data in the extended emotion data set can be used to pre-train an unsupervised audio coding model (such as Wav2vec2.0 and other models). The pre-trained audio coding model can be used as an audio encoder. In some application scenarios, the audio encoder can also be called an audio embedding extractor, an audio feature vector extractor, etc., which is not limited here.
[0163] In this implementation, during the pre-training of the unsupervised audio coding model, audio other than the training audio in the expanded emotion dataset may be used to improve the pre-training effect of the audio coding model.
[0164] In an example, the audio encoder obtained through pre-training can be represented by W_encoder.
[0165] In a possible implementation, the expanded emotion dataset may be used to train a unimodal emotion recognition model (eg, a text emotion recognition model or a speech emotion recognition model).
[0166] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: using the training text in the expanded emotion data set and the emotion label corresponding to the training text to train a text emotion recognition model, wherein the text emotion recognition model includes a text encoder and a text decoder.
[0167] Figure 3 Schematic diagram of a text emotion recognition model provided by an embodiment of the present disclosure is shown. Figure 3 As shown, the text emotion recognition model includes a text encoder and a text decoder. The text encoder can be a text encoder T_encoder obtained through pre-training. The output of the text decoder can be an emotion category prediction result. In this implementation, the text emotion recognition model can be trained according to the difference information between the emotion category prediction result output by the text emotion recognition model and the emotion label corresponding to the training text.
[0168] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: using the training audio in the expanded emotion data set and the emotion label corresponding to the training audio to train a speech emotion recognition model, wherein the speech emotion recognition model includes an audio encoder and an audio decoder.
[0169] Figure 4 FIG. 1 is a schematic diagram showing a speech emotion recognition model provided by an embodiment of the present disclosure. Figure 4 As shown, the speech emotion recognition model includes an audio encoder and an audio decoder. The audio encoder can be an audio encoder W_encoder obtained through pre-training. The output of the audio decoder can be an emotion category prediction result. In this implementation, the speech emotion recognition model can be trained based on the difference information between the emotion category prediction result output by the speech emotion recognition model and the emotion label corresponding to the training audio.
[0170] In a possible implementation, after obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: training a multimodal emotion recognition model using training audio in the expanded emotion data set, training text corresponding to the training audio, and emotion tags corresponding to the training audio.
[0171] In this implementation, the training audio, training text and emotion labels in the expanded emotion data set can be used to train the multimodal emotion recognition model. In this implementation, the multimodal emotion recognition model can be trained based on the difference information between the emotion category prediction results output by the multimodal emotion recognition model and the emotion labels corresponding to the training text and training audio.
[0172] In a possible implementation, the multimodal emotion recognition model includes an audio encoder, a text encoder and a multimodal decoder; the multimodal emotion recognition model is trained using training audio in an expanded emotion data set, training text corresponding to the training audio and emotion tags corresponding to the training audio, including: for any one emotion data in the expanded emotion data set, extracting an audio feature vector corresponding to the training audio in the emotion data through the audio encoder, and extracting a text feature vector corresponding to the training text in the emotion data through the text encoder; inputting the audio feature vector and the text feature vector into the multimodal decoder, and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding; and training the multimodal emotion recognition model according to the emotion tags in the emotion data and the emotion category prediction results corresponding to the training audio and the training text.
[0173] Figure 5 A schematic diagram of a multimodal emotion recognition model provided by an embodiment of the present disclosure is shown. Figure 5 As shown, the multimodal emotion recognition model may include an audio encoder, a text encoder and a multimodal decoder. The audio encoder may be an audio encoder W_encoder obtained through pre-training, the text encoder may be a text encoder T_encoder obtained through pre-training, and the multimodal decoder may represent a decoder in the multimodal emotion recognition model.
[0174] In a possible implementation, the multimodal emotion recognition model also includes a speaker encoder; the method also includes: extracting a speaker feature vector corresponding to the training audio through the speaker encoder; the inputting the audio feature vector and the text feature vector into the multimodal decoder, and obtaining the emotion category prediction result corresponding to the training audio and the training text through the multimodal decoding, including: inputting the speaker feature vector, the audio feature vector and the text feature vector into the multimodal decoder, and obtaining the emotion category prediction result corresponding to the training audio and the training text through the multimodal decoding.
[0175] Figure 6 Another schematic diagram of the multimodal emotion recognition model provided by the embodiment of the present disclosure is shown. Figure 6 As shown, the multimodal emotion recognition model may include a speaker encoder, an audio encoder, a text encoder and a multimodal decoder. The dimensions of the features output by the speaker encoder, the audio encoder and the text encoder may be the same. The speaker encoder may be a speaker encoder S_encoder obtained through pre-training.
[0176] In this implementation, the expanded emotion data set can be used to implement the training of a multimodal emotion recognition model combined with speaker information. By adopting this implementation, the trained multimodal emotion recognition model can be speaker-adaptive, and the trained multimodal emotion recognition model can be more robust and more generalizable.
[0177] The emotion data expansion method provided in the embodiments of the present disclosure can be applied to technical fields such as artificial intelligence, data expansion, and multimodal emotion recognition, and is not limited here.
[0178] The emotion data expansion method provided by the embodiment of the present disclosure is described below through a specific application scenario. In this application scenario, the emotion data expansion method can be divided into a data expansion phase, a pre-training phase and a training phase.
[0179] 1. Data expansion stage
[0180] 1. You can collect open source speech emotion data, or crawl or annotate speech emotion data based on the Internet.
[0181] 2. The speech emotion data can be screened to obtain the initial training audio. The initial training audio can be subjected to speech recognition to obtain the initial training text corresponding to the initial training audio. Based on the initial training audio and initial training text of different emotion categories corresponding to different speakers, multiple emotion data corresponding to different speakers can be obtained, thereby obtaining the initial emotion data set.
[0182] 3. The initial emotion dataset can be used to fine-tune the speech synthesis model.
[0183] 4. The semantic similarity matching model can be used to determine the expanded training text that is semantically similar to the initial training text, and / or the initial training text can be transcribed using a semantic transcription model to obtain the expanded training text corresponding to the initial training text.
[0184] For any extended training text, the extended training text and the initial training audio corresponding to the extended training text can be input into the fine-tuned speech synthesis model, and the extended training audio corresponding to the extended training text can be obtained through the fine-tuned speech synthesis model.
[0185] The expanded emotion data can be obtained according to the expanded training text, the expanded training audio, the speaker label and the emotion label. The expanded emotion data set can be obtained according to the initial emotion data and the expanded emotion data.
[0186] 2. Pre-training stage
[0187] 1. A speaker recognition model may be pre-trained based on the training audio and speaker labels in the emotion data in the expanded emotion dataset, wherein the speaker recognition model may include a speaker encoder, a speaker decoder, and a classifier.
[0188] 2. A language model can be pre-trained based on the training text in the sentiment data in the extended sentiment dataset. The pre-trained language model can be used as a text encoder.
[0189] 3. An unsupervised audio coding model can be pre-trained based on the training audio in the emotional data in the extended emotional data set. The pre-trained audio coding model can be used as an audio encoder.
[0190] 3. Training Phase
[0191] 1. Based on the expanded emotion dataset, a unimodal emotion recognition model can be trained.
[0192] (1) The training text in the expanded emotion data set and the emotion tags corresponding to the training text can be used to train a text emotion recognition model. The text emotion recognition model can include a text encoder and a text decoder. The text encoder can be a text encoder obtained through pre-training.
[0193] (2) The training audio in the expanded emotion data set and the emotion label corresponding to the training audio can be used to train a speech emotion recognition model. The speech emotion recognition model may include an audio encoder and an audio decoder. The audio encoder may be an audio encoder obtained through pre-training.
[0194] 2. A multimodal emotion recognition model can be jointly trained based on the expanded emotion data set. The multimodal emotion recognition model can include a speaker encoder, an audio encoder, a text encoder, and a multimodal decoder. The speaker encoder can be a speaker encoder obtained through pre-training, the audio encoder can be an audio encoder obtained through pre-training, and the text encoder can be a text encoder obtained through pre-training.
[0195] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.
[0196] In addition, the present disclosure also provides an emotion data expansion device, an electronic device, a computer-readable storage medium, and a computer program product, all of which can be used to implement any emotion data expansion method provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method part and will not be repeated here.
[0197] Figure 7 FIG. 1 is a block diagram of the emotion data expansion device provided by the embodiment of the present disclosure. Figure 7 As shown, the emotion data expansion device includes:
[0198] A first obtaining module 71 is used to obtain an initial emotion data set, wherein the initial emotion data set includes multiple items of initial emotion data corresponding to multiple speakers, and any item of initial emotion data includes initial training audio, initial training text and emotion label;
[0199] A fine-tuning module 72, configured to fine-tune the speech synthesis model using the initial emotion dataset;
[0200] A second obtaining module 73 is used to obtain an expanded training text that is semantically similar to the initial training text;
[0201] A third obtaining module 74 is used to input the expanded training text and the initial training audio corresponding to the expanded training text into the fine-tuned speech synthesis model, and obtain the expanded training audio corresponding to the expanded training text through the fine-tuned speech synthesis model;
[0202] The fourth obtaining module 75 is used to obtain the expanded emotion data corresponding to the initial emotion data according to the expanded training text, the expanded training audio and the emotion tag.
[0203] In a possible implementation, the initial emotion data further includes a speaker label;
[0204] The fourth obtaining module 75 is used to obtain the expanded emotion data corresponding to the initial emotion data according to the expanded training text, the expanded training audio, the speaker label and the emotion label.
[0205] In a possible implementation, the fine-tuning module 72 is used to:
[0206] For any item of initial emotion data, dividing initial training audio in the initial emotion data into a first audio segment and a second audio segment;
[0207] Inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting a synthesized audio corresponding to a second text segment in the initial training text through the speech synthesis model, wherein the second text segment represents a text segment corresponding to the second audio segment in the initial training text;
[0208] The speech synthesis model is fine-tuned according to difference information between the synthesized audio and the second audio segment.
[0209] In a possible implementation, the fine-tuning module 72 is used to:
[0210] The first audio segment, the initial training text in the initial emotion data, and the speaker information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs the synthesized audio corresponding to the second text segment in the initial training text.
[0211] In a possible implementation, the fine-tuning module 72 is used to:
[0212] The first audio segment, the initial training text in the initial emotion data, and the emotion information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs a synthesized audio corresponding to a second text segment in the initial training text.
[0213] In a possible implementation, the second obtaining module 73 is used to:
[0214] Acquire a candidate text set, wherein the candidate text set includes a plurality of candidate texts;
[0215] Determining the semantic similarity between the initial training text and the candidate text through a semantic similarity matching model;
[0216] The candidate texts whose semantic similarity with the initial training text is greater than or equal to a preset threshold are determined as the expanded training texts corresponding to the initial training texts.
[0217] In a possible implementation, the second obtaining module 73 is used to:
[0218] The initial training text is transcribed by a semantic transcription model to obtain an expanded training text corresponding to the initial training text.
[0219] In a possible implementation manner, the device further includes a first training module, which is used to:
[0220] For any one of the emotion data in the expanded emotion data set, the training audio in the emotion data is input into the speaker recognition model, and the speaker recognition model outputs the speaker prediction result corresponding to the training audio; wherein the expanded emotion data set includes the initial emotion data and the expanded emotion data; the speaker recognition model includes a speaker encoder, a speaker decoder and a classifier;
[0221] The speaker recognition model is trained according to the speaker prediction result corresponding to the training audio and the speaker label in the emotion data.
[0222] In a possible implementation manner, the apparatus further includes a second training module, configured to:
[0223] The text encoder is trained by using the training text in the sentiment data in the expanded sentiment dataset.
[0224] In a possible implementation manner, the apparatus further includes a third training module, configured to:
[0225] The audio encoder is trained by using the training audio in the emotional data in the extended emotional dataset.
[0226] In a possible implementation manner, the apparatus further includes a fourth training module, configured to:
[0227] The text emotion recognition model is trained by using the training texts in the expanded emotion data set and the emotion labels corresponding to the training texts, wherein the text emotion recognition model includes a text encoder and a text decoder.
[0228] In a possible implementation manner, the apparatus further includes a fifth training module, configured to:
[0229] The speech emotion recognition model is trained by using the training audio in the expanded emotion data set and the emotion label corresponding to the training audio, wherein the speech emotion recognition model includes an audio encoder and an audio decoder.
[0230] In a possible implementation manner, the apparatus further includes a sixth training module, configured to:
[0231] The multimodal emotion recognition model is trained using the training audio in the expanded emotion data set, the training text corresponding to the training audio, and the emotion label corresponding to the training audio.
[0232] In a possible implementation, the multimodal emotion recognition model includes an audio encoder, a text encoder, and a multimodal decoder;
[0233] The sixth training module is used for:
[0234] For any one item of emotion data in the expanded emotion data set, extracting an audio feature vector corresponding to the training audio in the emotion data through the audio encoder, and extracting a text feature vector corresponding to the training text in the emotion data through the text encoder;
[0235] Inputting the audio feature vector and the text feature vector into the multimodal decoder, and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding;
[0236] The multimodal emotion recognition model is trained according to the emotion tags in the emotion data and the emotion category prediction results corresponding to the training audio and the training text.
[0237] In a possible implementation, the multimodal emotion recognition model further includes a speaker encoder;
[0238] The device further comprises: an extraction module, configured to extract a speaker feature vector corresponding to the training audio through the speaker encoder;
[0239] The sixth training module is used to: input the speaker feature vector, the audio feature vector and the text feature vector into the multimodal decoder, and obtain the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding.
[0240] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.
[0241] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.
[0242] The embodiment of the present disclosure further provides a computer program, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.
[0243] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.
[0244] An embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.
[0245] The electronic device may be provided as a terminal, a server, or a device in other forms.
[0246] Figure 8 1 is a block diagram of an electronic device 1900 provided in an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal. Figure 8 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.
[0247] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (MacOS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.
[0248] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.
[0249] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0250] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0251] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0252] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0253] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0254] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0255] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0256] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0257] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.
[0258] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0259] If the technical solution of the embodiments of the present disclosure involves personal information, the product using the technical solution of the embodiments of the present disclosure has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of the embodiments of the present disclosure involves sensitive personal information, the product using the technical solution of the embodiments of the present disclosure has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to collect his or her personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
[0260] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for expanding emotional data, characterized in that: include: Obtaining an initial emotion data set, wherein the initial emotion data set includes multiple pieces of initial emotion data corresponding to multiple speakers, any one of the initial emotion data includes initial training audio, initial training text and an emotion label, and the emotion label in any one of the initial emotion data is determined according to the emotion category corresponding to the initial training audio and initial training text in the initial emotion data; Using the initial emotion dataset, fine-tuning the speech synthesis model; Obtaining an expanded training text that is semantically similar to the initial training text; Inputting the expanded training text and the initial training audio corresponding to the expanded training text into the fine-tuned speech synthesis model, and obtaining the expanded training audio corresponding to the expanded training text through the fine-tuned speech synthesis model; The expanded emotion data corresponding to the initial emotion data is obtained according to the expanded training text, the expanded training audio and the emotion label.
2. The method according to claim 1, characterized in that: The initial emotion data also includes a speaker label; The method of obtaining the expanded emotion data corresponding to the initial emotion data based on the expanded training text, the expanded training audio and the emotion label comprises: obtaining the expanded emotion data corresponding to the initial emotion data based on the expanded training text, the expanded training audio, the speaker label and the emotion label.
3. The method according to claim 1, characterized in that: The adopting the initial emotion data set to fine-tune the speech synthesis model includes: For any item of initial emotion data, dividing initial training audio in the initial emotion data into a first audio segment and a second audio segment; Inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting a synthesized audio corresponding to a second text segment in the initial training text through the speech synthesis model, wherein the second text segment represents a text segment corresponding to the second audio segment in the initial training text; The speech synthesis model is fine-tuned according to difference information between the synthesized audio and the second audio segment.
4. The method according to claim 3, characterized in that The step of inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting a synthesized audio corresponding to a second text segment in the initial training text through the speech synthesis model includes: The first audio segment, the initial training text in the initial emotion data, and the speaker information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs the synthesized audio corresponding to the second text segment in the initial training text.
5. The method according to claim 3, characterized in that: The step of inputting the first audio segment and the initial training text in the initial emotion data into a speech synthesis model, and outputting a synthesized audio corresponding to a second text segment in the initial training text through the speech synthesis model includes: The first audio segment, the initial training text in the initial emotion data, and the emotion information corresponding to the initial emotion data are input into a speech synthesis model, and the speech synthesis model outputs a synthesized audio corresponding to a second text segment in the initial training text.
6. The method according to any one of claims 1 to 5, characterized in that The step of obtaining an expanded training text that is semantically similar to the initial training text comprises: Acquire a candidate text set, wherein the candidate text set includes a plurality of candidate texts; Determining the semantic similarity between the initial training text and the candidate text through a semantic similarity matching model; The candidate texts whose semantic similarity with the initial training text is greater than or equal to a preset threshold are determined as the expanded training texts corresponding to the initial training texts.
7. The method according to any one of claims 1 to 5, characterized in that The step of obtaining an expanded training text that is semantically similar to the initial training text comprises: The initial training text is transcribed by a semantic transcription model to obtain an expanded training text corresponding to the initial training text.
8. The method according to claim 2, characterized in that: After obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: For any one of the emotion data in the expanded emotion data set, the training audio in the emotion data is input into the speaker recognition model, and the speaker recognition model outputs the speaker prediction result corresponding to the training audio; wherein the expanded emotion data set includes the initial emotion data and the expanded emotion data; the speaker recognition model includes a speaker encoder, a speaker decoder and a classifier; The speaker recognition model is trained according to the speaker prediction result corresponding to the training audio and the speaker label in the emotion data.
9. The method according to any one of claims 1 to 5, characterized in that After obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: The text encoder is trained by using the training text in the sentiment data in the expanded sentiment dataset.
10. The method according to any one of claims 1 to 5, characterized in that After obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: The audio encoder is trained by using the training audio in the emotional data in the extended emotional dataset.
11. The method according to any one of claims 1 to 5, characterized in that After obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: The text emotion recognition model is trained by using the training texts in the expanded emotion data set and the emotion labels corresponding to the training texts, wherein the text emotion recognition model includes a text encoder and a text decoder.
12. The method according to any one of claims 1 to 5, characterized in that After obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: The speech emotion recognition model is trained by using the training audio in the expanded emotion data set and the emotion label corresponding to the training audio, wherein the speech emotion recognition model includes an audio encoder and an audio decoder.
13. The method according to any one of claims 1 to 5, characterized in that After obtaining the expanded emotion data corresponding to the initial emotion data, the method further includes: The multimodal emotion recognition model is trained using the training audio in the expanded emotion data set, the training text corresponding to the training audio, and the emotion label corresponding to the training audio.
14. The method according to claim 13, characterized in that The multimodal emotion recognition model includes an audio encoder, a text encoder and a multimodal decoder; The method of training the multimodal emotion recognition model by using the training audio in the expanded emotion data set, the training text corresponding to the training audio, and the emotion label corresponding to the training audio includes: For any one item of emotion data in the expanded emotion data set, extracting an audio feature vector corresponding to the training audio in the emotion data through the audio encoder, and extracting a text feature vector corresponding to the training text in the emotion data through the text encoder; Inputting the audio feature vector and the text feature vector into the multimodal decoder, and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding; The multimodal emotion recognition model is trained according to the emotion tags in the emotion data and the emotion category prediction results corresponding to the training audio and the training text.
15. The method according to claim 14, characterized in that The multimodal emotion recognition model also includes a speaker encoder; The method further comprises: extracting a speaker feature vector corresponding to the training audio through the speaker encoder; The step of inputting the audio feature vector and the text feature vector into the multimodal decoder and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding includes: inputting the speaker feature vector, the audio feature vector and the text feature vector into the multimodal decoder and obtaining the emotion category prediction results corresponding to the training audio and the training text through the multimodal decoding.
16. An emotional data expansion device, characterized in that: include: A first acquisition module is used to obtain an initial emotion data set, wherein the initial emotion data set includes multiple items of initial emotion data corresponding to multiple speakers, any item of initial emotion data includes initial training audio, initial training text and emotion label, and the emotion label in any item of initial emotion data is determined according to the emotion category corresponding to the initial training audio and initial training text in the initial emotion data; A fine-tuning module, used to fine-tune the speech synthesis model using the initial emotion dataset; A second acquisition module is used to obtain an expanded training text that is semantically similar to the initial training text; A third acquisition module is used to input the expanded training text and the initial training audio corresponding to the expanded training text into the fine-tuned speech synthesis model, and obtain the expanded training audio corresponding to the expanded training text through the fine-tuned speech synthesis model; The fourth acquisition module is used to obtain the expanded emotion data corresponding to the initial emotion data according to the expanded training text, the expanded training audio and the emotion label.
17. An electronic device, characterized in that: include: one or more processors; a memory for storing executable instructions; The one or more processors are configured to call the executable instructions stored in the memory to execute the method according to any one of claims 1 to 15.
18. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.
