Emotion-based audio generation model training method and system

Abstract

The embodiment of the invention provides an emotion-based audio generation model training method and system. The method comprises the following steps: collecting first audio data corresponding to one emotion by one sound event and second audio data corresponding to multiple emotions; determining a corresponding relationship between the emotion and the sound event from the first audio data and the second audio data; obtaining training data with emotion expression based on the corresponding relation; inputting the training data into an audio generation model constructed based on a potential diffusion model, determining a potential representation of a spectrogram in the training data through a variational automatic encoder, and obtaining a reconstructed spectrogram by a decoder according to the potential representation; and determining loss according to the reconstructed spectrogram, and training an audio generation model. According to the embodiment of the invention, the emotional sound data set is constructed to train the audio generation model, so that the model can generate the audio with different emotional tones for the same sound event by specifying different emotions, thereby finely controlling the generated audio.

Description

Technical Field

[0001] The present invention relates to the field of intelligent speech, and in particular to a training method and system for an emotion-based audio generation model. Background Art

[0002] Emotions play a crucial role in human perception and interaction. In the audio field, emotion recognition and generation of speech and music are very useful in human-computer interaction applications and recommender systems. Therefore, significant progress has been made in speech and music emotion recognition and generation. Similar to speech and music, emotion recognition and generation of general sounds (such as natural sounds and sound effects) also have potential applications in intelligent environments to create emotionally consistent soundscapes and enrich user experiences in virtual reality and games. Existing audio generation models mainly focus on generating high-quality audio that meets specific conditions, such as using Diffsound, audio LDM, and Tango.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art: These existing models focus on quality and can receive input text and then generate corresponding audio based on the input text information. However, if the text specifies a specific emotion, it is impossible to output the audio corresponding to the emotion. This is mainly because the training data does not pay attention to the correspondence between emotions and audio. For a specific sound event, it may correspond to only one emotion or multiple emotions. Summary of the invention

[0004] In order to at least solve the problem in the prior art that if the text input to the model specifies a specific emotion, the audio corresponding to the emotion cannot be output.

[0005] In a first aspect, an embodiment of the present invention provides a method for training an emotion-based audio generation model, comprising: Collecting an emotional audio data set, wherein the emotional audio data set includes: first audio data in which one sound event corresponds to one emotion, and second audio data in which one sound event corresponds to multiple emotions; Determining a correspondence between emotions and sound events from the first audio data and the second audio data; Acquiring training data with emotion expressions based on the correspondence between the emotion and the sound event; Inputting the training data into an audio generation model constructed based on a latent diffusion model, wherein the audio generation model comprises: a variational autoencoder and a decoder, wherein the variational autoencoder determines a latent representation of a spectrogram in the training data, and the decoder obtains a reconstructed spectrogram according to the latent representation; The audio generation model is trained according to the loss determined by the reconstructed spectrogram until a preset condition is met.

[0006] In a second aspect, an embodiment of the present invention provides a training system for an emotion-based audio generation model, comprising: A data collection module is used to collect an emotional audio data set, wherein the emotional audio data set includes: first audio data in which one sound event corresponds to one emotion, and second audio data in which one sound event corresponds to multiple emotions; A correspondence determination module, used to determine the correspondence between emotions and sound events from the first audio data and the second audio data; A training data acquisition module, used to acquire training data with emotion expressions based on the correspondence between the emotion and the sound event; An audio generation module, configured to input the training data into an audio generation model constructed based on a latent diffusion model, wherein the audio generation model comprises: a variational autoencoder and a decoder, wherein the variational autoencoder determines a latent representation of a spectrogram in the training data, and the decoder obtains a reconstructed spectrogram according to the latent representation; The training module is used to train the audio generation model according to the loss determined by the reconstructed spectrogram until a preset condition is met.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the training method of the emotion-based audio generation model of any embodiment of the present invention.

[0008] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the program implements the steps of the training method of the emotion-based audio generation model of any embodiment of the present invention.

[0009] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the training method of the emotion-based audio generation model of any embodiment of the present invention are implemented.

[0010] The beneficial effect of the embodiment of the present invention is that the method constructs an emotional sound dataset, which means that each sound event can be further subdivided by emotion. The collected emotional sound dataset is used to train an audio generation model. This enables the model to generate audio with different emotional tones for the same sound event by specifying different emotions, thereby performing more precise control over the generated audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 is a flow chart of a method for training an emotion-based audio generation model provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of an EMOSOUND emotion label subset of a training method for an emotion-based audio generation model provided by an embodiment of the present invention; Figure 3 It is a data ratio diagram of an EmoSound emotion label subset of a training method for an emotion-based audio generation model provided by an embodiment of the present invention; Figure 4 A schematic diagram of generating audio according to emotion prompts in a training method of an emotion-based audio generation model provided by an embodiment of the present invention; Figure 5 is a schematic diagram of objective results of a training method for an emotion-based audio generation model provided by an embodiment of the present invention; Figure 6 is a schematic diagram of subjective results of a method for training an emotion-based audio generation model provided by an embodiment of the present invention; Figure 7 It is a schematic diagram of generating the same sound event with different emotions in a training method of an emotion-based audio generation model provided by an embodiment of the present invention; Figure 8 It is a structural diagram of a training system for an emotion-based audio generation model provided by an embodiment of the present invention; Fig. 9 A schematic structural diagram of an embodiment of an electronic device for training an emotion-based audio generation model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] like Figure 1 FIG. 1 is a flow chart of a method for training an emotion-based audio generation model provided by an embodiment of the present invention, comprising the following steps: S11: Collect an emotional audio data set, wherein the emotional audio data set includes: first audio data in which one sound event corresponds to one emotion, and second audio data in which one sound event corresponds to multiple emotions; S12: determining a correspondence between emotions and sound events from the first audio data and the second audio data; S13: acquiring training data with emotion expressions based on the correspondence between the emotion and the sound event; S14: inputting the training data into an audio generation model constructed based on a latent diffusion model, wherein the audio generation model comprises: a variational autoencoder and a decoder, the variational autoencoder determines a latent representation of a spectrogram in the training data, and the decoder obtains a reconstructed spectrogram according to the latent representation; S15: Training the audio generation model according to the loss determined by the reconstructed spectrogram until a preset condition is met.

[0015] In this embodiment, it is considered that in the prior art, emotions are generally divided into the following six basic emotions: happiness, sadness, anger, fear, surprise and disgust.

[0016] However, the present application found that the emotion of surprise is usually related to the suddenness of the sound rather than the type of the sound itself. On the other hand, sadness is more related to personal emotions and memories. Therefore, in the present method, the present method pays more attention to the following four emotions, namely happiness, fear, anger and disgust, as prompts for the sound event generation of the present method. Based on these four emotions, the present method constructs a sound emotion dataset called EmoSound, which includes various sound events labeled with emotion labels. It is the basis for training and evaluating audio generation models. By integrating sound emotion recognition and using the latent diffusion model for audio generation. This method fills the gap in emotion / emotion-guided sound generation so that sounds can be generated based on specific emotional cues.

[0017] For step S11, an emotional audio dataset is collected. For this dataset, human speech can express various emotions, but the category of emotion is often related to the content of the speech. Some specific speech content corresponds to only one emotion. For example, when a person says "my pet is hurt", this statement is usually related to anger rather than happiness. Similarly, the emotion evoked by a sound is related to the category of the sound. Some sounds do not evoke any emotions in people, some sounds tend to evoke specific emotions, and some sounds may evoke different emotions. Therefore, the present method collects sounds that may evoke emotions to form the emotional audio dataset of the present method, which is called the EmoSound dataset, and is divided into two subsets: a single emotion label subset, in which a sound event only induces one specific emotion; a mixed emotion label subset, in which a sound event can evoke multiple emotions.

[0018] Regarding the first audio data in which a sound event corresponds to an emotion, that is, a single emotion label subset, specifically, "My pet is injured" corresponds to anger, "I won the lottery" corresponds to happiness, "Oh, there is a loud thunder outside" corresponds to fear, and "Take something away quickly, take it away quickly" corresponds to disgust.

[0019] Regarding the second audio data where one sound event corresponds to multiple emotions, that is, the mixed emotion label subset: A single type of sound event does not always correspond to a single emotion. Although there are multiple audio samples of the same sound event, they can evoke different emotions. For example, a cat's meow can trigger a feeling of fear or happiness. For example, when a cat sees its owner, it will make a short "meow (one) meow (one)" with a clip sound. Hearing this sound will trigger a happy emotion in some people who love cats. When a cat is frightened or injured, it will make a sharp and piercing "meow (two) meow "four" meow (three)". For a more concrete example, when a cat asks for food, the meow is usually soft, and people will feel happy when they hear it, while when a cat is angry, the meow is sharp, and people will feel a little disgusted when they hear it. Similarly, the barking of a dog will also evoke different emotions in people. Not only animals, but also human-based sound events will correspond to multiple emotions. For example, "Ah! There are cockroaches on the bed!" may correspond to emotions such as fear, anger, and disgust. In this method, cat meowing and dog barking can be selected as sound events related to multiple emotions.

[0020] For step S12, the correspondence between emotions and sound events is determined from the first audio data and the second audio data. As an implementation method, the correspondence between emotions and sound events in the first audio data is identified based on a language model, wherein the language model includes: Microsoft Copilot; the correspondence between emotions and sound events in the second audio data manually annotated is received. Specifically, the method uses a large language model (which can be: Microsoft Copilot or other large language models) to identify sound events that may evoke individual happiness, fear, anger or disgust. In this way, a preliminary correspondence between emotions and sound events is obtained, and for each emotion, some common sound events are selected for further analysis. For further accuracy, unreliable correspondences can be manually screened out. Large model analysis and manual annotation are also applicable to the second audio data in which one sound event corresponds to multiple emotions. Specifically, 12 people aged between 20 and 25 who are currently undergraduates or graduate students can be asked to participate in identifying the emotions evoked by the sound when listening. The task of the participants is to select the emotion that the sound evokes after auditory exposure. They are prompted to confirm whether the sound event does indeed evoke the specified emotion and provide "yes" or "no" as their response options. Then, we filtered the sound events and selected those that received positive responses from more than 75% of the survey participants. Figure 2 Displays the correspondence between the selected sound events, the corresponding emotions, and the number of samples.

[0021] For step S13, with the correspondence between emotions and sound events, based on the correspondence, the sound event label can be used as a keyword to search for audio featuring a specified sound event from major video websites, such as Freesound, Youtube, etc., and the audio featuring the specified sound event can be used to generate targeted training for the model to recognize the specified emotion. In this way, an audio generation model for training emotion-based audio can be obtained, such as Figure 3 As shown, the training data of the example of this method (sound events, text corresponding to emotions, and corresponding audio (the spectrogram can be directly obtained from the audio)).

[0022] For step S14, through the above steps, training data is obtained, and the training data is input into the audio generation model based on the latent diffusion model, which can be called VAE-LDM (variational autoencoder- latent diffusion model). In the task of generating text to audio corresponding to sound events and emotions, directly generating spectrograms poses a major challenge to training efficiency. VAE (variational auto encoder) is used to compress the spectrogram into the latent space. The VAE encoder converts the spectrogram into Compressed to latent representation , where M, T, D, and c represent the number of Mel-bands, sequence length, potential embedding dimension, and compression ratio, respectively. The potential representation z is divided into and , represents the mean and variance of the VAE latent space. The decoder is based on the distribution The spectrogram is reconstructed from the samples extracted from .

[0023] For step S15, LDM (latent diffusion model) is effective in generating high-quality audio for text-to-audio generation tasks. LDM involves a forward process to transform the latent distribution into a Gaussian distribution through noise injection. This is followed by a back-diffusion process to reconstruct the spectrogram, handling denoising and spectrogram generation. The transition probability of the Markov chain in the forward process is defined as: in, , , Sampling from a normal distribution As n increases, Close to isotropic Gaussian distribution. Given condition X (e.g. emotion and sound event) and weight As a measure of SNR (signal-to-noise ratio), the reverse process learns to estimate the noise: in, represents a noise estimation network that can be used from reconstruction The above-mentioned learning and training process, for example, ends when the error between the reconstructed spectrogram and the spectrogram directly obtained from the audio is less than a preset threshold.

[0024] As an implementation manner, the training of the audio generation model includes: training with classifier guidance and training without classifier.

[0025] The classifier guided training comprises: The audio generation model is trained by gradient-guided diffusion using the loss determined by the classification results of the spectrogram and the reconstructed spectrogram by the classifier.

[0026] The training without classifier includes: Part of the conditional information of the emotion expression in the training data is randomly discarded, an unclassified reconstructed spectrogram is generated by using the audio generation model, and the audio generation model is trained based on the reconstructed spectrogram and the loss determined by the unclassified reconstructed spectrogram.

[0027] To generate audio based on a specific category, the method can use a classifier-guided diffusion model or a classifier-free diffusion model.

[0028] The classifier-guided diffusion model uses the gradient from the classifier to guide the diffusion process to generate samples belonging to a specific class. By combining the gradient of the classifier, calculating the loss with the samples and updating the gradient, the model can adjust the generation process to generate samples that are more likely to belong to the desired class.

[0029] Classifier-free diffusion modeling is an advanced generative modeling technique that does not require an external classifier to guide the generation process. It involves training a single diffusion model to perform both conditional and unconditional generation tasks. During the training phase, the model randomly discards both types of data (emotions and sound events) with a certain probability. This dual training approach enables the model to learn both conditional and unconditional situations.

[0030] During inference, the model combines these estimates to generate a sample: in, is the control text guide for noise estimation The guiding scale of the contribution of is a text input.

[0031] The audio generation model obtained after training in the above way can generate audio by inputting text with emotional expressions or text containing emotions and sound events, such as happy, fear, happy cat meow, fearcat meow, and outputting corresponding audio. The same sound event can be generated with different emotions, such as Figure 4 shown.

[0032] It can be seen from this implementation that the method constructs an emotional sound dataset, which means that each sound event can be further subdivided by emotion. The audio generation model is trained using the collected emotional sound dataset. This enables the model to generate audio with different emotional tones for the same sound event by specifying different emotions, thereby achieving more refined control over the generated audio.

[0033] The method is experimentally illustrated. Based on EmoSound, the method trains a classifier-guided VAE-LDM and a classifier-free VAE-LPM. The performance of the two models is compared by generating audio for each model and selecting the model that produces better results. Subsequently, the audio generated by the selected model is further subjectively evaluated to assess its effectiveness. Generating Multiple Sound Events Using a Single Emotion: Classifier-Guided LDM The method uses the single-label subset of EmoSound to train an unconditional audio VAE-LDM and a multi-task audio classifier. The architecture of the LDM is similar to the Tango model, except that it does not require a text encoder. The classifier has two output layers: one for the emotion classification logic and the other for the sound event classification logic. Due to the small number of audio samples in EmoSound, the method uses it to fine-tune the UNet model of Tango to achieve higher audio quality and faster training speed. The AdamW optimizer is used to fine-tune the UNet in the VAE-LDM for 40 cycles. The learning rate is set to 3×10-5 with a linear decay scheduler. The classifier is also trained on the EmoSound subset. During classifier training, our method extracts spectral features from audio using the same method as VAE-LDM training, encodes them into latent variables using VAE, and applies random noise to these latent variables. This effectively produces an audio latent variable classifier. During inference, our method specifies a category and uses the gradient of the classifier to guide the VAE-LDM to generate the corresponding audio. When our method experiments with this approach, specifying an emotion and using the gradient of the classifier emotion layer, the emotion category of the generated audio is almost random and the desired specific emotion generation cannot be achieved. In contrast, when specifying a sound event and using the gradient of the classifier sound event layer, most of the generated audio matches the specified sound event, with some instances of other sound events. This can be concluded that using the emotion classifier to guide the diffusion model is ineffective.

[0034] Generating a single sound event with multiple emotions: This method uses a subset of mixed emotion labels to fine-tune Tango's checkpoints. The text hint template is "emotion+sound event".

[0035] Regarding the evaluation indicators, sentiment classification accuracy, in order to objectively evaluate whether the sentiment of the generated audio matches the specified sentiment text prompt, this method uses the collected audio sentiment pairs to train a classification model. The classification model adopts a simple CNN structure. Then, the classifier is used to classify the generated audio, using the sentiment text prompt of the generated audio as the basic truth to obtain the Figure 5 The classification accuracy is labeled acc.

[0036] Audio Quality In order to objectively evaluate the quality of generated audio, this method calculates the FAD (Frechet Audio Distance) of generated audio for four different emotions. Based on the distribution of generated samples and reference samples, FAD is a typical evaluation protocol for evaluating the quality of generated audio in text-to-audio generation tasks.

[0037] Subjective Evaluation To further evaluate the effectiveness of the selected model, this method performed MOS (Mean Opinion Score) from two perspectives: emotional accuracy and audio quality.

[0038] For the task of generating multiple sound events with a single emotion, this method randomly selected 10 generated audio samples for each emotion, a total of 40 samples, and recruited 10 evaluators to evaluate the generated audio. The evaluation criteria are as follows: (1) Audio quality: Evaluators rated the naturalness and realism of the audio on a scale of 1 to 5, with 1 indicating very poor quality and 5 indicating very high quality.

[0039] (2) Emotional perception: The raters selected the emotion they felt when listening to the audio from the following options: anger, disgust, fear, happiness.

[0040] For the task of generating a single sound event with multiple emotions, a similar approach was used, except that only two emotions were selected. Ten generated audio samples were randomly selected for each sound event, and 10 evaluators were recruited to evaluate the audio. The evaluation criteria were similar to the previous task, with the only difference being the emotion options of fear and happiness.

[0041] According to the evaluation indicators, the results are divided into two parts: objective and subjective.

[0042] Objective results: Objective results are shown in Figure 5. For the EmoSound single emotion label subset, this method calculated the FAD of the audio generated for each emotion. For the EmoSound mixed emotion label subset, due to the small amount of data for a single emotion, the FAD of the entire subset was directly calculated. Comparing the classifier-guided and classifier-free methods, it is obvious that the metric of the classifier-free method is better than that of the classifier-guided method. This further verifies the conclusion of this method that the classifier-guided method is ineffective for general sound generation guided by emotion.

[0043] For the classifier-free approach, the emotion classifier trained on the original audio can accurately distinguish the emotions of the generated audio, regardless of whether the task involves generating multiple sound events with a single emotion or a single sound event with multiple emotions. This indicates that most of the emotion categories of the generated audio are consistent with the input text prompt. The FAD values ​​for both tasks are relatively low, which objectively indicates that the generated audio is of high quality.

[0044] Subjective results: Based on the objective results, the classifier-free model performs better. Therefore, this method conducts a subjective evaluation of the audio generated by the classifier-free model to further verify its effectiveness. The subjective MOS results are shown in Figure 2. Figure 6 As shown in Figure 2. From the perspective of subjective MOS quality, the generated audio exhibits high quality. Regarding subjective MOS emotion accuracy, although individual perceptions of emotions may vary, the emotions evoked by the generated audio are largely consistent with the given emotional text cues.

[0045] Objective and subjective results show that the model trained on EmoSound by our method can generate high-quality audio corresponding to specific emotions. In addition, for one sound event, it can generate audio with different emotions, such as Figure 7 As shown, a fearful cat's meow has short, rapid bursts, while a happy cat's meow is more melodic.

[0046] In summary, this method constructs an emotional sound dataset named EmoSound, which can be divided into two parts. The first part consists of sound events corresponding to four different emotions, and each sound event is associated with only one emotion. The second part includes sound events corresponding to two emotions, which means that each sound event can be further subdivided by emotion. Using the collected data, this method further trains an audio generation model. The effectiveness of the audio generation model is verified by objective and subjective evaluation of the generated audio. It is beneficial to use emotions as textual cues when training a general audio generation model. This method allows the trained model to generate audio corresponding to a given emotion. In addition, even for the same sound event, specifying different emotions can produce audio with different emotional tones, allowing for more refined control over the generated audio.

[0047] like Figure 8 The figure is a schematic diagram of the structure of a training system for an emotion-based audio generation model provided by an embodiment of the present invention. The system can execute the training method for the emotion-based audio generation model described in any of the above embodiments and be configured in a terminal.

[0048] The present embodiment provides an emotion-based audio generation model training system 10 comprising: a data collection module 11 , a correspondence determination module 12 , a training data acquisition module 13 , an audio generation module 14 and a training module 15 .

[0049] Among them, the data collection module 11 is used to collect an emotional audio data set, wherein the emotional audio data set includes: a first audio data in which one sound event corresponds to one emotion, and a second audio data in which one sound event corresponds to multiple emotions; the correspondence determination module 12 is used to determine the correspondence between the emotion and the sound event from the first audio data and the second audio data; the training data acquisition module 13 is used to obtain training data with emotional expression based on the correspondence between the emotion and the sound event; the audio generation module 14 is used to input the training data into an audio generation model constructed based on a latent diffusion model, wherein the audio generation model includes: a variational autoencoder and a decoder, the variational autoencoder determines the latent representation of the spectrum graph in the training data, and the decoder obtains a reconstructed spectrum graph according to the latent representation; the training module 15 is used to train the audio generation model according to the loss determined by the reconstructed spectrum graph until a preset condition is met.

[0050] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the training method of the emotion-based audio generation model in any of the above method embodiments; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Collecting an emotional audio data set, wherein the emotional audio data set includes: first audio data in which one sound event corresponds to one emotion, and second audio data in which one sound event corresponds to multiple emotions; Determining a correspondence between emotions and sound events from the first audio data and the second audio data; Acquiring training data with emotion expressions based on the correspondence between the emotion and the sound event; Inputting the training data into an audio generation model constructed based on a latent diffusion model, wherein the audio generation model comprises: a variational autoencoder and a decoder, wherein the variational autoencoder determines a latent representation of a spectrogram in the training data, and the decoder obtains a reconstructed spectrogram according to the latent representation; The audio generation model is trained according to the loss determined by the reconstructed spectrogram until a preset condition is met.

[0051] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the training method of the emotion-based audio generation model in any of the above method embodiments is executed.

[0052] Fig. 9 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for a training method of an emotion-based audio generation model provided in another embodiment of the present application, such as Fig. 9 As shown, the device includes: One or more processors 910 and memory 920, Fig. 9 A processor 910 is taken as an example. The device of the training method of the emotion-based audio generation model may also include: an input device 930 and an output device 940.

[0053] The processor 910, the memory 920, the input device 930 and the output device 940 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0054] The memory 920, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the training method for the emotion-based audio generation model in the embodiment of the present application. The processor 910 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 920, that is, the training method for the emotion-based audio generation model in the above method embodiment is implemented.

[0055] The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 920 may optionally include a memory remotely arranged relative to the processor 910, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0056] The input device 930 can receive input digital or character information. The output device 940 can include a display device such as a display screen.

[0057] The one or more modules are stored in the memory 920, and when executed by the one or more processors 910, the training method of the emotion-based audio generation model in any of the above method embodiments is executed.

[0058] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0059] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0060] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the training method of the emotion-based audio generation model of any embodiment of the present invention.

[0061] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0062] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0063] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0064] (4) Other electronic devices with data processing functions.

[0065] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0066] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for an emotion-based audio generation model, comprising: Collecting an emotional audio data set, wherein the emotional audio data set includes: first audio data in which one sound event corresponds to one emotion, and second audio data in which one sound event corresponds to multiple emotions; Determining a correspondence between emotions and sound events from the first audio data and the second audio data; Acquiring training data with emotion expressions based on the correspondence between the emotion and the sound event; Inputting the training data into an audio generation model constructed based on a latent diffusion model, wherein the audio generation model comprises: a variational autoencoder and a decoder, wherein the variational autoencoder determines a latent representation of a spectrogram in the training data, and the decoder obtains a reconstructed spectrogram according to the latent representation; The audio generation model is trained according to the loss determined by the reconstructed spectrogram until a preset condition is met.

2. The method according to claim 1, wherein: The training of the audio generation model includes: training with classifier guidance and training without classifier.

3. The method according to claim 2, wherein: The classifier guided training comprises: The audio generation model is trained by gradient-guided diffusion using the loss determined by the classification results of the spectrogram and the reconstructed spectrogram by the classifier.

4. The method according to claim 2, wherein: The training without classifier includes: Part of the conditional information of the emotion expression in the training data is randomly discarded, an unclassified reconstructed spectrogram is generated by using the audio generation model, and the audio generation model is trained based on the reconstructed spectrogram and the loss determined by the unclassified reconstructed spectrogram.

5. The method according to claim 2, wherein: The audio generation model is dually trained with and without classifier guidance.

6. The method according to claim 1, wherein: Determining the correspondence between emotion and sound event from the first audio data and the second audio data includes: Identifying a correspondence between emotions and sound events in the first audio data based on a language model, wherein the language model includes: Microsoft Copilot; Receive manually annotated correspondences between emotions and sound events in the second audio data.

7. A training system for an emotion-based audio generation model, comprising: A data collection module is used to collect an emotional audio data set, wherein the emotional audio data set includes: first audio data in which one sound event corresponds to one emotion, and second audio data in which one sound event corresponds to multiple emotions; A correspondence determination module, used to determine the correspondence between emotions and sound events from the first audio data and the second audio data; A training data acquisition module, used to acquire training data with emotion expressions based on the correspondence between the emotion and the sound event; An audio generation module, configured to input the training data into an audio generation model constructed based on a latent diffusion model, wherein the audio generation model comprises: a variational autoencoder and a decoder, wherein the variational autoencoder determines a latent representation of a spectrogram in the training data, and the decoder obtains a reconstructed spectrogram according to the latent representation; The training module is used to train the audio generation model according to the loss determined by the reconstructed spectrogram until a preset condition is met.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 6.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images