Voice-to-audio generation system, training method, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a voice-to-audio generation system, a training method, electronic equipment and a storage medium, and the system comprises a voice encoder which is used for extracting features related to a sound event in input voice; the bridging network is used for mapping the features related to the sound events to semantic features, and the semantic features are used for guiding an audio generation module to generate audio; and an audio generation module comprising a potential diffusion model for generating a potential representation of audio from the input speech, a variational auto-encoder for reconstructing a spectrogram based on the potential representation of audio, and a vocoder for converting the spectrogram into a waveform.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech-to-audio generation, and in particular relates to a speech-to-audio generation system, a training method, an electronic device and a storage medium. Background Art

[0002] In related technologies, ASR (Automatic Speech Recognition) systems can convert speech into text, but usually do not involve audio generation. TTA (Text-to-Audio) models can generate corresponding audio based on text descriptions. Cross-modal audio generation models can generate audio based on text, audio or visual information, but usually do not directly process speech input.

[0003] The inventors found that the above-mentioned technologies lack the ability to generate audio directly from speech, and need to go through intermediate steps such as ASR, which increases complexity and delay. In addition, the above-mentioned technologies fail to fully utilize the semantic information in speech to guide audio generation. Summary of the invention

[0004] Embodiments of the present invention provide a speech-to-audio generation system, method, electronic device, and storage medium, which are used to solve at least one of the above-mentioned technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a speech-to-audio generation system, comprising: a speech encoder for extracting features related to sound events in input speech; a bridging network for mapping the features related to sound events to semantic features, wherein the semantic features are used to guide an audio generation module to generate audio; and an audio generation module, comprising a latent diffusion model, a variational autoencoder, and a vocoder, wherein the latent diffusion model is used to generate a latent representation of audio from the input speech, the variational autoencoder is used to reconstruct a spectrogram based on the latent representation of the audio, and the vocoder is used to convert the spectrogram into a waveform.

[0006] In a second aspect, an embodiment of the present invention further provides a training method for a speech-to-audio generation system, which is used for the speech-to-audio generation system described in the first aspect, comprising: a first stage, pre-training the bridging network; a second stage, fine-tuning the bridging network, wherein all other modules are frozen during fine-tuning; and a third stage, fine-tuning the latent diffusion model, wherein all other modules are frozen during fine-tuning.

[0007] In a third aspect, an embodiment of the present invention further provides a computer program product, comprising a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program comprises program instructions, which, when executed by a computer, cause the computer to perform the steps of the training method for a speech-to-audio generation system of any embodiment of the present invention.

[0008] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in the first aspect.

[0009] In a fifth aspect, an embodiment of the present invention further provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0010] In the method of the embodiment of the present application, the quality of the generation from speech to audio is improved, and the naturalness and interactivity of human-computer interaction are enhanced. Further, the development of speech interaction systems is promoted, and a foundation is provided for more complex speech understanding and generation tasks in the future. Further, new possibilities are provided for the automatic generation and editing of audio content, which helps to enrich the creation and distribution of multimedia content. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0012] Figure 1 A flowchart of a method for training a speech-to-audio generation system provided by one embodiment of the present invention; Figure 2 A diagram of a speech-to-audio (STA) generation task provided for an embodiment of the present invention; Figure 3 An overview of the STAR framework and training strategy provided for one embodiment of the present invention; Figure 4 A linear detection classification result of AUDIOCAPS provided by an embodiment of the present invention; Figure 5 Results of different STA systems provided by an embodiment of the present invention; Figure 6A reduction study result on model design provided for an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The inventors found that the defects of the related art are mainly due to the following reasons: lack of an end-to-end model specifically for speech-to-audio generation. The existing model fails to effectively extract and utilize information related to acoustic events in speech.

[0015] When encountering the above problems, the methods usually adopted by those skilled in the art include improving the integration of ASR and TTA models, or developing more complex cross-modal models. It is not easy to think of the solution of the present application because the generation directly from speech to audio requires innovative acoustic event-related representation learning and end-to-end model design.

[0016] The embodiment of the present application proposes a new Speech-to-Audio (STA) generation task, which uses acoustic event-related semantic representation to extract sufficient semantic information from speech to guide audio generation. The introduction of a bridge network and a multi-stage training strategy significantly improves the generation performance.

[0017] Product structure: Input speech: The speech signal input by the user, which can be directly recorded speech or speech data converted by a speech recognition system.

[0018] Speech encoder: Use a self-supervised pre-trained speech encoder (such as HuBERT or WavLM) to extract the acoustic features of the input speech and the semantic information related to the acoustic events.

[0019] Bridge network: maps the features extracted by the speech encoder to semantic features related to the acoustic event. This network can be a multi-layer perceptron (MLP) or Q-Former (a network structure based on the attention mechanism, used in the bridge network), which is used to convert speech features into representations suitable for audio generation tasks.

[0020] Latent Diffusion Model: Using the semantic features output by the bridge network, the Latent Diffusion Model (LDM) generates a latent representation of the audio. This process includes a forward process that gradually adds Gaussian noise to the latent representation, and a reverse process that gradually removes the noise to generate a latent representation of the target audio.

[0021] Variational Autoencoder (VAE): VAE is used to extract latent representations from the spectrograms of audio to reduce computational complexity. The VAE encoder compresses the spectrogram into representations in a latent space, and the VAE decoder then reconstructs the spectrogram based on these latent representations.

[0022] Vocoder: The role of the vocoder is to convert the VAE reconstructed spectrogram back into an audible audio waveform. This step usually involves nonlinear transformations and signal processing techniques to generate high-quality audio output.

[0023] Output audio: The final generated audio is an audio signal corresponding to the acoustic event described in the input speech, which can be played directly to the user.

[0024] Process Solution Overview of the STAR framework and training strategy. For an input speech utterance, a representation related to the sound event is extracted by the speech encoder, which is then mapped to a semantic feature S by the bridge network. The latent diffusion model generates a latent representation, which is then decoded back into a waveform conditioned on S. The training consists of three stages. The bridge network is pre-trained in the first stage and fine-tuned in the second stage. Finally, the diffusion model is fine-tuned in the third stage.

[0025] In the process of implementing this application, the inventors have also adopted the following alternatives: Alternative 1: Use the traditional ASR and TTA cascade system. The advantage is that the technology is mature and easy to implement. The disadvantage is that the generation process is not end-to-end and the efficiency is low. Alternative 2: Directly use the speech encoder to generate audio. The advantage is that the process is simplified. The disadvantage is that there is a lack of in-depth understanding of the semantics of the acoustic event, and the generation quality may not be high.

[0026] Between the project launch and the final product, the inventors also used the following beta versions: Beta 1: A preliminary STA system without a bridge network. The advantage is that the basic STA concept was verified. The disadvantage is that the generated audio quality is not high and there is a lack of accurate mapping of acoustic events. Beta 2: A bridge network was introduced, but the training strategy was not optimized. The advantage is that the audio quality was improved. The disadvantage is that the training process is not efficient enough and needs further optimization.

[0027] The direct effect that can be achieved by the solution of the embodiment of the present application is: improving the quality of the generation from speech to audio, and enhancing the naturalness and interactivity of human-computer interaction. Chain reaction: promoting the development of voice interaction systems, and providing a foundation for more complex speech understanding and generation tasks in the future. Deeper effects: providing new possibilities for the automatic generation and editing of audio content, helping to enrich the creation and distribution of multimedia content.

[0028] An embodiment of the present application provides a speech-to-audio generation system, comprising: a speech encoder, for extracting features related to sound events in input speech; a bridging network, for mapping the features related to sound events to semantic features, wherein the semantic features are used to guide an audio generation module to generate audio; and an audio generation module, comprising a latent diffusion model, a variational autoencoder, and a vocoder, wherein the latent diffusion model is used to generate a latent representation of audio from the input speech, the variational autoencoder is used to reconstruct a spectrogram based on the latent representation of the audio, and the vocoder is used to convert the spectrogram into a waveform.

[0029] In some optional embodiments, the latent diffusion model includes: a forward process to gradually add Gaussian noise to the latent representation; and an inverse process to gradually denoise to generate a latent representation of the target audio.

[0030] In some optional embodiments, the variational autoencoder includes: an encoder for compressing the spectrogram into a latent representation; and a decoder for reconstructing the spectrogram based on the latent representation.

[0031] In some optional embodiments, the speech encoder is HuBERT or WavLM.

[0032] In some optional embodiments, the bridging network is a multi-layer perceptron (MLP) or a Q-Former.

[0033] Please refer to Figure 1 , which shows a flow chart of an embodiment of a training method for a speech-to-audio generation system of the present application. The training method is used for the speech-to-audio generation system of the aforementioned embodiment.

[0034] like Figure 1 As shown, in step 101, in the first stage, the bridge network is pre-trained; In the second stage, the bridge network is fine-tuned, wherein all other modules are frozen during the fine-tuning; In the third stage, the latent diffusion model is fine-tuned, wherein all other modules are frozen during fine-tuning.

[0035] It should be noted that the above method steps are not used to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the opposite order to the specified step order, and this application has no limitation on this.

[0036] The solution of the present application is described below through a specific embodiment so that those skilled in the art can better understand the solution of the present application.

[0037] In recent years, the research on audio generation has experienced rapid development, especially in audio generation based on text, audio, and vision. However, as a natural way of interaction, speech has not received enough attention in audio generation research. This study aims to fill this gap by proposing a speech-to-audio generation system that enables the model to have stronger interaction capabilities. Specifically, our initial step is to show through representation learning experiments that representations related to sound events can be extracted from speech, which can be used to guide audio generation. Exploiting the semantic information contained in these representations, we propose the first end-to-end generative model from speech to audio. In addition, we introduce a bridge network for representation mapping and design a three-stage training strategy to optimize performance. Experimental results show that our proposed model significantly outperforms baseline models that do not exploit semantic representations and performs comparable to multiple cascade systems. The generated samples are available on DemoWeb.

[0038] Figure 2 An illustration of the Speech to Audio (STA) generation task is shown. The system generates audio from a description of the user's speech.

[0039] Audio generation technology has made great progress, and many cross-modal models based on text, audio, and visual modalities have shown excellent generation performance. However, speech, as a natural and widely used human-computer interaction mode, has not received much attention in audio generation. For example, if someone asks "what is the sound of wind blowing", there is currently no system that can directly output the sound effect. To address this gap, we propose the first speech-to-audio (STA) generation system STAR (Speech-To-Audio generation with sound event-related Representations). It aims to understand the sound event information in speech and generate the corresponding audio. Of course, one would think that such functionality can be achieved by a cascade system containing automatic speech recognition (ASR) and text-to-audio (TTA) generation models. However, as an end-to-end system, STAR has the potential to integrate audio generation into human-computer interaction, thereby enhancing the capabilities of speech interaction systems such as SpeechGPT and BLSP-Emo. It demonstrates the feasibility of end-to-end STA, thereby achieving unified training of speech dialogue systems. In contrast, it is challenging to achieve unified training using cascade models. To achieve end-to-end STA, we must evaluate whether we can extract enough information from speech signals to guide audio generation. Specifically, can we extract speech representations that contain semantic information about sound events such as "dog barking" or "ringing"? Such representations are called sound event-related representations. To this end, we conducted a series of representation learning experiments to evaluate the cross-modal information extraction capabilities of different encoders. The results show that self-supervised pre-trained speech encoders, such as HuBERT and WavLM, are able to extract sound event-related representations.

[0040] Extracting relevant representations from speech signals Due to the lack of direct pre-trained models in speech audio research, it is crucial to study methods to connect the two modes. Drawing on the experience of fine-tuning large models, we introduced a bridge network and implemented a three-stage training method to optimize model performance. Specifically, our framework includes a speech encoder for feature extraction, a bridge network for mapping, and a latent diffusion model (LDM) for generation. The model is optimized through a staged training process of pre-training and supervised fine-tuning (SFT). Experimental results show that STAR significantly outperforms baseline models that do not utilize sound event-related representations. It is also comparable to some cascade systems composed of high-performance models. It should be noted that although STAR does not perform as well as the cascade framework with SOTA subsystems in some cases, it is a pioneering exploration and opens up the possibility of working with audio LLM in the future. This demonstrates the effectiveness of sound event-related representations in guiding audio generation. Our contributions are as follows: 1) We propose a speech-to-audio (STA) generation task and explore the feasibility of extracting semantic representations of sound events from speech; 2) We introduce a bridge network and a multi-stage training strategy for STA, which greatly improves the generation performance and achieves a system comparable to the cascaded system.

[0041] II. Expressions related to sound events In text-to-audio generation, the input is text describing the audio content, called subtitles. In speech-to-audio scenarios, the input is the speech waveform that reads out the subtitles, hereafter called speech. In both forms, the generated content is audio.

[0042] To achieve end-to-end STA, sufficient semantic information must be extracted from speech to guide generation, the most important of which is the sound event type. Although there is no direct research on speech-to-audio generation, previous cross-modal work has paved the way for effective learning paradigm generation. Similarly, in our experiments, we combined a speech encoder and a bridge network to (1) extract high-level features from redundant speech signals and (2) map the features to the sound event-related representations that are truly needed. The quality of the representation is evaluated through the downstream task of sound event classification, because the accuracy of sound events is crucial for audio generation.

[0043] Speech Codec Since speech waveforms are the input to STA, a speech encoder must be used to extract speech embeddings for subsequent processing. We investigate the efficacy of different types of encoders: one type of encoders includes HuBERT and WavLM, which are pre-trained with the masked language modeling objective; the other type of encoders includes encoder-decoder models such as Description-Audio-Codec (DAC), which are trained with an autoencoding approach. Intuitively, we believe that the first two encoders are better at capturing features related to sound events than the latter encoder-decoder models, as the former focus on semantic information while the latter emphasize acoustic details.

[0044] Bridged Network In order to embed the speech extracted by the speech encoder To adapt the semantic representation required for audio generation, we use a bridge network to perform the mapping. We study two architectures of the bridge network, namely, a multi-layer perceptron (MLP) and a Q-Former. The MLP directly projects the , while Q-Former uses cross-attention and learnable query embedding to encode.

[0045] STAR: Speech-to-Audio Generator Figure 3The STAR framework and training strategy overview are shown. For the input speech, the speech encoder extracts representations related to sound events, which are then mapped to semantic features by the bridge network. The latent diffusion model generates latent representations, which are then Conditioned, it is decoded back into a waveform. The training consists of three stages. The bridge network is pre-trained in the first stage and fine-tuned in the second stage. Finally, the diffusion model is fine-tuned in the third stage. Among them, audio summary: caption, speech: speech, speech encoding: speech encoder, bridge network: bridge network, linear layer: linear classifier, variational autoencoder-encoder: audio: audio, vocoder: vocoder, VAEencoder, variational autoencoder-decoder: VAE dencoder, diffusion model: diffusion model, SFT: Supervised Finetuning, supervised finetuning. In 2 / 3 stages, pink arrows indicate training, blue arrows indicate inference, snowflake patterns indicate frozen parameters, and fire patterns indicate trainable. Audio summary "a person types on a keyboard": "A person is typing on a keyboard."

[0046] like Figure 3 As shown, STAR consists of an encoder that extracts relevant features from speech (voice encoder), a cross-modal network that connects speech (bridge network), and an audio generation module, which consists of LDM, variational autoencoder (VAE), and a vocoder that converts spectrograms into waveforms.

[0047] Generate Module a) VAE for extracting audio representation: VAE is used to extract audio representation, aiming to reduce computational complexity. The VAE encoder converts the spectrogram Compression into latent features , where T, M, R, and D represent the sequence length, the number of MEL stripes, the compression ratio, and the potential dimension, respectively. Divided into two halves, representing the average values ​​in the latent space and variance The VAE decoder is based on the distribution Reconstructed spectrum of samples extracted from The vocoder will Convert back to waveform.

[0048] b) Diffusion model: Diffusion model is used to predict It includes cumulative The forward process of adding Gaussian noise, followed by the reverse process of gradually denoising. The transition probability of the Markov chain in the forward process is defined as: in, , follow In the last step middle, follows a standard Gaussian distribution. The model is trained to minimize and the reparameterized loss of the weight λn related to the signal-to-noise ratio: Where θ represents the diffusion network. generate , an iterative reverse step will be performed, starting from n changes to 0: in, .

[0049] Training strategy We introduce multi-stage training to optimize model performance. Its advantages are: (1) focusing on the optimization of specific modules in stages, which is conducive to better convergence in complex tasks; (2) gradually increasing the number of parameters to improve efficiency; (3) making each stage easier to evaluate and monitor.

[0050] like Figure 3 As shown, the training is divided into three stages. Each audio-caption pair in the dataset has a corresponding sound event label. For example, the audio with the caption "a man speaks, then a dog barks" is labeled as having two events (the man speaks and the dog barks). Therefore, we add a sound event classifier using the audio label data to pre-train the bridge network. This is the first stage. In the second and third stages, the bridge network and diffusion model are fine-tuned on STA, respectively, while freezing all other modules.

[0051] Experimental setup For the end-to-end STA system, our first step is to create captions in the form of speech, so a text-to-speech system is needed to first synthesize the text into the corresponding speech. We use AudioCaps and VITS to synthesize speech audio data. Since AudioCaps is a subset of the audio tagging dataset AudioSet, the sound event labels are used in the first training phase. In this phase, the bridge network is pre-trained with a learning rate of 3.2×10 −3, up to 500 epochs, early stopping of 20 epochs. The evaluation metric is mean average precision (mAP).

[0052] In the last two training stages, the generative model was trained for 40 epochs using the AdamW optimizer with a learning rate of 3×10 −5 , using a linear decay schedule.

[0053] The baseline model uses the same structure as STAR, the only difference is that the input content is changed from the extracted sound event-related representation to the Mel spectrogram of speech. We also combine the cascade system of ASR and TTA for comparison. On the ASR side, we use WHISPER, and on the TTA side, we try two advanced generative models, AudioLDM and TANGO 2 (note that this is one of the best performing generation models), respectively. All modules are initialized with pre-trained model weights to inherit their capabilities, including Q-Former, VAE, diffusion model, and vocoder.

[0054] We used common objective audio generation metrics, including FAD, FD, and KL divergence. We also conducted subjective evaluations in which the evaluators listened to 10 samples of each model and scored them on two dimensions: overall audio quality (OVL) and relevance to the input (REL). All evaluators were screened to have no hearing loss and were university educated at reputable institutions. They used designated headphones during the evaluation process.

[0055] Speech Expression Assessment We first evaluate the effectiveness of speech representations extracted by different speech encoders and bridge networks. The multi-label classification performance in the first training phase demonstrates their effectiveness. To provide a reference, we also compare the text encoder with the subtitle input. Since the input is the original subtitle, the performance of the text encoder can be used as a baseline. We evaluate two text encoders: (1) CLAP, which extracts sound event-oriented representations through contrastive learning; and (2) Flan-T5, which is a text encoder that extracts general representations by pre-training on large-scale Chain of Thoughts (CoT) and instruction-based datasets. These text encoders have been shown to be effective in previous TTA studies.

[0056] Figure 4 Linear detection classification results for AUDIOCAPS are shown. The performance of audio caption representation is shown as the top line.

[0057] Results Figure 4 As can be seen, the performance of the speech representation is comparable to that of the subtitle representation extracted by the text encoder, which indicates that information related to sound events can be extracted from speech to guide audio generation. Specifically, HuBERT performs slightly better than WavLM, while Q-Former shows good performance across different speech and subtitle representations. Therefore, we adopt Hu-BERT and Q-Former by default in the following experiments.

[0058] Generate performance Figure 5 The resulting performance of different systems is shown, including both objective and subjective metrics. W, A, T denote WHISPER, audiodm and tango 2. ovl = overall quality. REL = correlation with input.

[0059] Figure 5 The results of different STA systems are listed. Both objective and subjective evaluation results show that the performance of the STA system is significantly better than the baseline model, which verifies the effectiveness and necessity of extracting speech representations related to sound events for audio generation. If the input speech embedding is not sufficiently representative of the corresponding sound event, the model will lack sufficient guidance when generating. In addition, compared with the cascade model, the performance of STAR is comparable to or even slightly better than WHISPER+Audi-oLDM. If TANGO 2 is used as the TTA model and the powerful Flan-T5 is used to encode the text prompts, there is still a clear gap. This shows that it is still necessary to further learn speech representations to enhance speech representations so that they can not only recognize sound events but also have a deeper understanding of speech content. In summary, the performance of this system proves the feasibility of end-to-end STA, which can further support more seamless human-computer interaction. Among them, E2E: end-to-end, baseline system: baseline, ours: ours, cascade system: cascade.

[0060] Ablation studies Figure 6 Results of a subtraction study on the model design are shown.

[0061] We from Figure 6 The impact of the two aspects shown on the system design is studied by ablation.

[0062] 1) Training strategy design: We validate our three-stage training strategy by eliminating one of the training stages. Eliminating (w / o) the first stage (pre-training the bridge network) leads to the most significant performance degradation. This shows that current speech coding models (such as HuBERT) can only extract transcription information and lack content understanding capabilities like Flan-T5, so task-specific STA pre-training is still very necessary. Since the model parameters are initialized from different sources, independent SFTs for the bridge network and the UNet backbone are helpful for training. The combination of the three training stages improves the generation quality.

[0063] 2) Bridge network structure: The results of different bridge networks are consistent with our findings: Q-Former outperforms MLP, and both bridge networks are better than directly inputting speech embedding information to the generative model. This shows that even if the speech encoder can provide sufficient semantic information, a suitable bridge network is crucial to adjust different modalities to promote cross-modal learning.

[0064] In this embodiment of the present application, we explored a new task, namely generating audio from speech input. It aims to understand the sound event information described in the speech and then generate the corresponding audio content. We introduced STAR, an end-to-end STA system that uses speech representations related to sound events to enhance human-computer interaction capabilities. Through comprehensive representation learning experiments, we demonstrated that self-supervised pre-trained speech encoders can extract sufficient semantic information for audio generation. Using the Q-Former bridging network and a three-stage training strategy, STAR's performance is significantly better than the baseline system and is comparable to the cascade system. Ablation studies verify the necessity of our model architecture and training strategy design. We hope that STAR can contribute to the design of future end-to-end voice dialogue models and give them more powerful capabilities.

[0065] In some other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium, the computer storage medium storing computer executable instructions, the computer executable instructions can execute the training method of the speech-to-audio generation system in any of the above method embodiments, for the speech-to-audio generation system; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: In contrastive language-audio pre-training, the audio encoder and text encoder are jointly trained to align semantically similar audio-text pairs in a shared embedding space; In the first stage, the bridge network is pre-trained; In the second stage, the bridge network is fine-tuned, wherein all other modules are frozen during the fine-tuning; In the third stage, the latent diffusion model is fine-tuned, wherein all other modules are frozen during fine-tuning.

[0066] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the speech-to-audio generation method and system, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the speech-to-audio generation method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0067] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any of the above-mentioned speech-to-audio generation methods.

[0068] Figure 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the device includes: one or more processors 710 and a memory 720, Figure 7 A processor 710 is used as an example. The device of the speech-to-audio generation method and system may also include: an input device 730 and an output device 740. The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 In the example, the bus connection is used. The memory 720 is the above-mentioned non-volatile computer-readable storage medium. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, the speech to audio generation method of the above-mentioned method embodiment is realized. The input device 730 can receive input digital or character information, and generate key signal input related to user settings and function control of the speech to audio generation device. The output device 740 may include a display device such as a display screen.

[0069] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0070] As an implementation mode, the electronic device is applied to a speech-to-audio generation system, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: In the first stage, the bridge network is pre-trained; In the second stage, the bridge network is fine-tuned, wherein all other modules are frozen during the fine-tuning; In the third stage, the latent diffusion model is fine-tuned, wherein all other modules are frozen during fine-tuning.

[0071] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0072] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0073] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0074] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0075] (5) Other electronic devices with data interaction functions.

[0076] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0077] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech-to-audio generation system, comprising: A speech encoder for extracting features related to sound events in the input speech; A bridging network, used for mapping the features related to the sound event to semantic features, wherein the semantic features are used to guide the audio generation module to generate audio; and An audio generation module includes a latent diffusion model, a variational autoencoder and a vocoder, wherein the latent diffusion model is used to generate a latent representation of the audio from the input speech, the variational autoencoder is used to reconstruct a spectrogram based on the latent representation of the audio, and the vocoder is used to convert the spectrogram into a waveform.

2. The system according to claim 1, wherein: The potential diffusion model includes: a forward process that gradually adds Gaussian noise to the latent representation; and A reverse process that progressively denoises to generate a latent representation of the target audio.

3. The system according to claim 1, wherein: The variational autoencoder comprises: an encoder for compressing the spectrogram into a latent representation; and A decoder is configured to reconstruct a spectrogram based on the latent representation.

4. The system according to claim 1, wherein: The speech encoder is HuBERT or WavLM.

5. The system according to claim 1, wherein: The bridge network is a multi-layer perceptron (MLP) or a Q-Former.

6. A training method for a speech-to-audio generation system, used in the speech-to-audio generation system according to any one of claims 1 to 3, comprising: In the first stage, the bridge network is pre-trained; In the second stage, the bridge network is fine-tuned, wherein all other modules are frozen during the fine-tuning; In the third stage, the latent diffusion model is fine-tuned, wherein all other modules are frozen during fine-tuning.

7. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method of claim 6.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 6 are implemented.

Images