Speech generation method and related device

Abstract

A speech generation method and related device, the method comprising: obtaining first source data (701) input into a speech generation model (100) comprising a plurality of encoders (111, 112) and a decoder (120), wherein the types of input data of the plurality of encoders (111, 112) are different; a first encoder of the plurality of encoders (111, 112) generates first acoustic features (702) according to the first source data; the decoder (120) converts second acoustic features determined according to the first acoustic features into third acoustic features, wherein the third acoustic features are used to generate speech (703) with target sound. The above technical solution can rely on a single model to perform different speech generation tasks.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech technology, and more specifically, to speech generation methods and related devices. Background Technology

[0002] Speech generation is a technique for generating speech from input. It can refer to various forms of speech generation, such as text-to-speech (TTS), audio-visual conversion, or video-to-speech. Different speech generation tasks are typically handled by different frameworks, which limits its practical application. For example, in some scenarios, electronic devices have limited resources for speech generation. However, different frameworks may require significant resources, such as storage and computational resources, even exceeding the limited resources available, thus impacting the application of speech generation. Summary of the Invention

[0003] This application provides a speech generation method and related equipment. The technical solution can rely on a single model to perform different speech generation tasks.

[0004] According to a first aspect, embodiments of this application provide a speech generation method, comprising: acquiring first source data input into a speech generation model including a plurality of encoders and a decoder, wherein the input data of the plurality of encoders are of different types; a first encoder among the plurality of encoders generating a first acoustic feature based on the first source data, wherein the type of the first source data is consistent with the type of the input data of the first encoder; and a decoder converting a second acoustic feature determined based on the first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with a target sound.

[0005] The speech generation model in this application embodiment has multiple encoders and a shared decoder. The multiple encoders can operate on different input domains, enabling the entire model to perform different speech generation tasks. In other words, the solution in this application embodiment can generate speech from different types of input data using a single model.

[0006] In one possible design, the decoder is a diffusion-based decoder, wherein the decoder converts a second acoustic feature determined based on the first acoustic feature into a third acoustic feature by means of a reverse diffusion process.

[0007] The decoder is a diffusion-based decoder that generates speech through a back-diffusion process. In other words, the speech generation model is a DPM (Digital Persistent Model) capable of generating high-quality speech, with fast adaptation speed and low data requirements. Thus, in the model of this application embodiment, the quality of the generated speech can be ensured.

[0008] For example, the first acoustic feature could be a spectrogram-like feature corresponding to the first source data, and the third acoustic feature could be a spectrogram of speech containing the target sound. The spectrogram of speech containing the target sound can be called the target spectrogram. The spectrogram-like feature corresponding to the first source data is any one of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis, or a concatenation of the spectrogram corresponding to the first source data and the acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis.

[0009] For example, the first source data could be source audio data, source text data, or source video data.

[0010] For example, the second acoustic feature can be the first acoustic feature.

[0011] For example, the third acoustic feature could be a target acoustic feature, such as a fine-grained spectrogram.

[0012] In one possible design, the plurality of encoders includes at least two of the following: a video encoder, a speech encoder, or a text encoder, wherein when the first source data is audio data, the first encoder is the speech encoder; when the first source data is text data, the first encoder is the text encoder; or when the first source data is video data, the first encoder is the video encoder.

[0013] In one possible design, the multiple encoders and decoders are trained separately.

[0014] In one possible design, the plurality of encoders includes a speech encoder and a text encoder, wherein when the first source data is audio data, the first encoder is the speech encoder, and when the first source data is text data, the first encoder is the text encoder.

[0015] The model described above, consisting of a speech encoder, a text encoder, and a decoder, can perform sound cloning and sound conversion: the speech encoder combined with the decoder is used to perform sound conversion, while the text encoder combined with the decoder corresponds to the sound cloning task.

[0016] In one possible design, the first acoustic feature is an average spectrum corresponding to the first source data.

[0017] The average spectrogram can be viewed as a speaker-independent speech representation. The first encoder remains speaker-independent, meaning that it does not require fine-tuning in terms of speaker adaptation.

[0018] In one possible design, the speech encoder, the text encoder, and the decoder are trained separately.

[0019] According to the technical solution provided in the embodiments of this application, the two encoders and decoders in the model can be trained separately to avoid the instability caused by joint training. The two encoders can be trained separately with the same target in a supervised manner. This supervision method is more reliable because the outputs of the two encoders have clear interpretations (e.g., average sound spectrograms) and do not belong to the latent space.

[0020] In one possible design, the method further includes: acquiring second source data input into a speech generation model; a second encoder among the plurality of encoders generating a fourth acoustic feature based on the second source data, wherein the type of the second source data is consistent with the type of the input data of the second encoder, and the second acoustic feature is acquired by concatenating the fourth acoustic feature and the first acoustic feature.

[0021] For example, the first encoder can be a voice encoder or a text encoder, and the second encoder can be a video encoder.

[0022] In one possible design, the decoder converts a second acoustic feature determined based on the first acoustic feature into a third acoustic feature through a backdiffusion process, comprising: the decoder converting the second acoustic feature determined based on the first acoustic feature into the third acoustic feature through the backdiffusion process, conditioned on target sound information, wherein the target sound information is generated by a speaker encoder.

[0023] The speech generation model also includes a speaker encoder, which can be used to reproduce the target voice. Thus, even in scenarios where no target voice data is available for training, i.e., in zero-shot scenarios, the speech generation model provided in this application can generate speech with the target voice.

[0024] According to a second aspect, embodiments of this application provide an electronic device having the function of implementing the method described in the first aspect. The function can be implemented in hardware or by hardware executing corresponding software. The hardware of the software includes one or more modules corresponding to the function.

[0025] According to a third aspect, embodiments of this application provide a computer-readable storage medium having instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any possible implementation thereof.

[0026] According to a fourth aspect, an electronic device is provided, including a processor and a memory. The processor is connected to the memory. The memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions stored in the memory, the processor performs the method of the first aspect or any possible implementation thereof.

[0027] According to a fifth aspect, a chip system is provided, the chip system including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program, such that a server on which the chip resides performs the method of the first aspect or any possible implementation thereof.

[0028] According to a sixth aspect, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0029] Figure 1 This is a schematic block diagram of the speech generation model provided in the embodiments of this application.

[0030] Figure 2 This is a flowchart of one embodiment of the sound conversion provided in this application.

[0031] Figure 3 This is a flowchart of one embodiment of the sound cloning provided in this application.

[0032] Figure 4 This is a flowchart of one embodiment of speech generation provided in this application.

[0033] Figure 5 This is a flowchart of another embodiment of speech generation provided in this application.

[0034] Figure 6 This is a flowchart of yet another embodiment of speech generation provided in this application.

[0035] Figure 7 This is a flowchart of an embodiment of a speech generation method.

[0036] Figure 8 This is a schematic block diagram of the electronic device 800 provided in the embodiments of this application.

[0037] Figure 9 This is a schematic block diagram of the electronic device 900 provided in the embodiments of this application. Detailed Implementation

[0038] The technical solution of this application is described below with reference to the accompanying drawings.

[0039] To facilitate understanding of the embodiments of this application, the relevant terms involved in the embodiments of this application are introduced below.

[0040] (1) Sound cloning

[0041] Voice cloning is a task typically designed to add new voices to a TTS system. In other words, voice cloning is essentially a TTS technique that replicates the voice of a target speaker.

[0042] When target speaker data is available, voice cloning can be performed through speaker adaptation. Speaker adaptation typically refers to fine-tuning the TTS system based on a small amount of target speaker data to obtain a TTS system with good performance tailored to the target voice.

[0043] When only one short target sound sample is available, sound cloning is performed using speaker encoding. Speaker encoding typically refers to using pre-trained or learnable speaker representations to help extract speaker identity information, such as timbre and pitch, from reference speech samples.

[0044] (2) Voice conversion

[0045] Voice conversion is the task of copying the voice of a target speaker while preserving the linguistic content of the source speaker's pronunciation.

[0046] Any-to-one (A2O) voice conversion (VC) aims to convert any speaker (including speakers not seen during training) into a fixed target speaker.

[0047] In practice, it is best to have an arbitrary-pair voice conversion model. An arbitrary-pair arbitrary voice conversion model is a model that can reproduce the target voice while preserving the source speech content, even when the source speaker and the target speaker are not necessarily in the training dataset.

[0048] (3) Diffusion probabilistic model (DPM)

[0049] DPM comprises forward diffusion and backward diffusion. Forward diffusion gradually adds Gaussian noise to the data, while backward diffusion attempts to remove this noise. DPM is trained to minimize the distance between the trajectories of the forward and backward diffusion processes. In other words, the goal of DPM training is to find backward diffusion such that the backward diffusion trajectory closely follows the forward diffusion trajectory, but in the reverse temporal order.

[0050] Different speech generation tasks are typically handled using different models. For example, TTS and voice translation are two common speech generation tasks that are usually handled using different models.

[0051] This application provides a speech generation model capable of processing different types of input data to generate speech. In other words, the speech generation model of this application can handle a variety of different speech generation tasks.

[0052] The speech generation model provided in this application includes multiple encoders and a decoder shared by the multiple encoders. The outputs of the multiple encoders can be the inputs of the decoder.

[0053] Figure 1 This is a schematic block diagram of the speech generation model provided in the embodiments of this application. For example... Figure 1 As shown, the speech generation model 100 may include an encoder 111, an encoder 112, and a decoder 120.

[0054] It should be noted that, Figure 1 This is merely a schematic diagram of the speech generation model provided in the embodiments of this application. Figure 1 The number of encoders shown does not constitute any limitation. Figure 1 In this example, the speech generation model 100 includes two encoders; in other cases, the speech generation model may include more encoders.

[0055] Each of a plurality of encoders is used to acquire acoustic features corresponding to its own input data. A decoder is used to acquire target acoustic features conditioned on the target sound, based on the output of at least one encoder. The target acoustic features conditioned on the target sound can be used to generate speech with the target sound. For example, the output domain of the decoder can be a spectrogram of the speech with the target sound. The spectrogram of the speech with the target sound can be called the target spectrogram. The output of the decoder can be converted into a waveform by a vocoder (e.g., a HiFi-GAN vocoder). The vocoder may or may not be part of the speech generation model.

[0056] For example, multiple encoders can be implemented using neural networks.

[0057] The multiple encoders are of different types. The type of input data for the multiple encoders is related to the type of encoder. Accordingly, the input data for the multiple encoders is data of different types. The input data for the multiple encoders can be referred to as source data.

[0058] In one possible implementation, the multiple encoders may include at least two of the following: a speech encoder, a text encoder, or a video encoder.

[0059] The input data for a speech encoder can be acoustic data such as audio, speech, or acoustic features. Acoustic features can be spectrograms, also known as spectral features.

[0060] For example, the spectrogram could be a MEL spectrogram, in which case the speech encoder could also be called a MEL encoder, and the MEL spectrogram could also be called a MEL feature.

[0061] The input data for a text encoder can be text data, such as text, characters, or phoneme embeddings.

[0062] The input data for a video encoder can be video data. For example, a video encoder can be a lip-reading encoder.

[0063] For example, Figure 1 Encoder 111 in the text can be a voice encoder. Figure 1 The encoder 112 in the code can be a text encoder. For example, Figure 1 Encoder 111 in the text can be a voice encoder. Figure 1 The encoder 112 in the code can be a video encoder. For example, Figure 1 Encoder 111 in the code can be a text encoder. Figure 1 The encoder 112 in the text can be a video encoder.

[0064] The output domains of multiple encoders can be the same or different.

[0065] In one possible implementation, the encoder in the speech generation model is used to generate an output similar to a spectrogram.

[0066] For example, an output similar to a spectrogram could be a spectrogram itself. Alternatively, an output similar to a spectrogram could be acoustic features that can be aligned with a spectrogram on the time axis, such as pitch, loudness, and a spectrogram convolved with a filter bank along the frequency axis. Or, an output similar to a spectrogram could be a concatenation of a spectrogram and acoustic features that can be aligned with a spectrogram on the time axis.

[0067] Optionally, in some embodiments, at least one encoder is used to generate a spectrogram in the speech generation model. In this case, the spectrogram generated by the encoder can be viewed as acoustic features corresponding to the input data of the encoder itself.

[0068] Alternatively, in some embodiments, multiple encoders work together to generate a spectrogram in the speech generation model.

[0069] The aforementioned speech encoder, text encoder, or video encoder can be used to generate a spectrogram. Alternatively, at least one of the speech encoder, text encoder, or video encoder can be used to generate a spectrogram.

[0070] It should be noted that the encoder is merely an example. As mentioned above, the output domain of the decoder can be a spectrogram, i.e., the target spectrogram. In this case, the encoder should generate an output that can be aligned with the target spectrogram. Outputs similar to a spectrogram can be aligned with the target spectrogram. Therefore, other encoders capable of generating spectrogram-like outputs can also be used as the encoder in the embodiments of this application.

[0071] Furthermore, the encoder's output can approximate the target spectrogram. For example, the encoder's output can be one of the following: an average spectrogram corresponding to the encoder's input data, a spectrogram of a specific sound, or a low-resolution spectrogram of the target sound.

[0072] For ease of understanding and explanation, the embodiments of this application are illustrated using an average spectrum diagram as an example.

[0073] The average spectrogram can also be called the average sound spectrogram. The average sound refers to the pronunciation of each phoneme, and its characteristics may be the same as the average characteristics across a dataset of multiple speakers. For example, the average sound spectrogram could be an average sound mel spectrogram, which can be referred to as the average phoneme-level mel feature.

[0074] For example, an encoder used to predict the average spectrogram corresponding to the input data can be acquired through training. Specifically, the encoder can be trained with the goal of reducing the difference between the encoder's output and the ground truth average spectrogram corresponding to the training source data. During encoder training, the training source data is the encoder's input data. Methods for acquiring the ground truth average spectrogram can be found in the example in the next section.

[0075] During inference, the output of the encoder trained in the above manner can be regarded as the average spectrum corresponding to the input data of the encoder.

[0076] In the embodiments of this application, the encoder can be used to predict the average spectrogram corresponding to the input data of the encoder.

[0077] For example, a speech encoder can be used to predict the average spectrogram corresponding to the source audio, a text encoder can be used to predict the average spectrogram corresponding to the source text, and a video encoder can be used to predict the average spectrogram corresponding to the source video.

[0078] The average spectrogram is independent of the speaker corresponding to the encoder's input data. The speaker corresponding to the encoder's input data can be called the source speaker. Therefore, the average spectrogram can be regarded as a speech representation independent of the speaker.

[0079] Optionally, in some embodiments, the multiple encoders and decoders in the speech generation model can be trained separately.

[0080] by Figure 1 Taking model 100 as an example, encoder 111, encoder 112, and decoder 120 can be trained separately. In other words, encoder 111, encoder 112, and decoder 120 can be regarded as three independent modules. During the training of one of the three independent modules, the parameters of the other modules are fixed.

[0081] For example, Figure 1 The encoder 111 in the code can be used to predict the average spectrogram corresponding to the input data of the encoder 111. Figure 1 The encoder 112 can be used to predict the average spectrogram corresponding to the input data of encoder 112. The training process of the encoder is explained below using encoder 111 as a mel encoder and encoder 112 as a text encoder as an example.

[0082] MEL encoder After training, the audio data X0 is converted into an average spectrogram corresponding to the audio data X0.

[0083] For example, the mel encoder After training, the output spectrum is... and ground true average spectrum The mean square error (MSE) between the two is minimized, and during training, X0 is the training source audio data.

[0084] The training source audio data can be a training source spectrogram X0. By replacing the features corresponding to each phoneme in the training source spectrogram X0 with features corresponding to that specific phoneme aggregated from a corpus of speech data from multiple speakers, a ground truth average spectrogram can be obtained. The corpus can be an existing corpus or a corpus set up according to needs.

[0085] For example, phoneme A is present in the training source spectrogram X0. The features of phoneme A in the training source spectrogram X0 are replaced with the average features of phoneme A. The average features of phoneme A are obtained by aggregating the features of phoneme A in a corpus of speech data from multiple speakers. The above steps are performed for each phoneme in the training source spectrogram X0 to obtain the ground truth average spectrogram corresponding to the training source spectrogram X0.

[0086] During inference, X0 is the source audio data to be processed, referred to simply as the source audio data. The output of the MEL encoder trained in the above manner... It can be viewed as an average spectrogram corresponding to the source audio data X0.

[0087] For example, a transformer-based architecture can be used as a speech encoder.

[0088] The text encoder ψ is trained to convert the source text data T into an average spectrogram corresponding to the source text data T.

[0089] For example, the text encoder ψ is trained to output a spectrogram. and ground true average spectrum Minimize the MSE between the training data and the target data. During training, T is the source text data for training.

[0090] Obtain the ground true average spectrum The method can be the same as described above. That is, when the language content of the training source text data T and the training source audio data X0 is the same, the ground truth average spectrogram... Alternatively, the same can be true, meaning that during training, the text encoder's objective...

[0091] The target output is the same as the target output of the speech encoder.

[0092] The text encoder can be an existing text encoder or a self-configured text encoder.

[0093] For example, a text encoder can be Figure 3 The encoder shown. A text encoder converts input text into an encoded text sequence, which is then mapped to frame-by-frame features, such as a spectrogram. Figure 3 As shown, convolutional layers (conv) and bidirectional long short-term memory (Bi LSTM) are used to generate the encoded text sequence. A duration predictor generates a monotonic alignment, which indicates how many frames each element of the text input lasts, helping to generate a spectrogram. Upsampling is a process that repeats each output of the Bi-LSTM the number of times the duration predictor predicts it, ensuring that a spectrogram with the correct duration is generated.

[0094] Optionally, in some embodiments, the speech generation model includes a speaker encoder. The speaker encoder provides target sound information to the decoder, in which case the decoder generates acoustic features conditioned on the target sound. The decoder is a speaker-conditioned decoder. For example, the decoder can be used to convert an average spectrogram into a fine-grained spectrogram conditioned on the target sound information.

[0095] For example, the target voice information can be a speaker embedding.

[0096] The speaker encoder can be trained jointly with the decoder, so the speaker encoder can also be considered part of the decoder.

[0097] The speaker encoder can be called a speaker coding network.

[0098] The decoder can be a diffusion-based decoder. The speech generation model in this application embodiment can be viewed as DPM attempting to convert acoustic features extracted from source data by at least one of multiple encoders into target acoustic features by using a speaker-related score matching network (called the decoder).

[0099] Forward diffusion transforms any source data into a normal random variable. Where I is the identity matrix, Predicted by at least one encoder.

[0100] For example, the source data could be a source spectrogram X0. It can be a MEL encoder. The predicted average sound spectrum. Therefore, the prior in this DPM. It is a speech representation independent of the speaker, preserving the linguistic content of the source data.

[0101] The backdiffusion parameterized by the decoder is trained to approximate the forward diffusion trajectory in the time variable t∈[0,1].

[0102] As mentioned above, the decoder and multiple encoders can be trained separately.

[0103] The encoder parameterizes the terminal distribution of forward diffusion (i.e., prior), while the decoder parameterizes the back diffusion.

[0104] For example, once the DPM prior is established... Parameterized MEL encoder After training, the MEL encoder The parameters are fixed, and the decoder corresponding to the backdiffusion begins training.

[0105] As one possible implementation, DPM can be formalized using stochastic differential equations (SDEs).

[0106] Forward X and Backward The diffusion process can be obtained through the following SDE:

[0107]

[0108] Where t∈[0,1], and These are independent forward and reverse standard Brownian motions, respectively. β t It is non-negative noise scheduling. X t These are samples in forward diffusion. It is a sample in the back diffusion.

[0109] Speaker conditionalization in the decoder is handled by the speaker coding network g. t (Y) Enabled.

[0110] By integrating into the score matching network θ Integrating with score matching networks θ Jointly trained speaker coding network g t (Y), the inverse SDE (Formula 1.2) is conditional on the target sound:

[0111]

[0112] The decoder parameters are represented by θ, and Y = {Y} s} s∈[0,1] This is the overall trajectory of the reference spectrum Y0 calculated for the target sound under forward diffusion. In other words, Y = {Y}s} s∈[0,1] It is the overall forward diffusion trajectory starting from Y0. The reference spectrogram Y0 can be the training spectrogram during training, i.e., the training source spectrogram X0. The reference spectrogram Y0 can also be the spectrogram of the target sound during inference.

[0113] A well-trained decoder can learn from prior knowledge. China Generative modeling is achieved by sampling and simulating the parameterized back-diffusion path using the decoder at unit time intervals [0,1]. Samples generated at the initial time point... It is the output of the speech generation task.

[0114] During inference, in each iteration of the backdiffusion process, the speaker embedding is also re-estimated and fed back to the gradient prediction network of the decoder.

[0115] The decoder can be implemented using a neural network. For example, the decoder has a UNet-based architecture.

[0116] Speaker coding network g t (Y) can be composed of 2D convolutions and multilayer perceptrons (MLPs).

[0117] It should be noted that DPM can also be formalized in other ways. For example, DPM can also be formalized using Markov chains, but this embodiment of the application does not limit this approach.

[0118] The speech generation model in this application embodiment has multiple encoders and a shared decoder. The multiple encoders can operate on different input domains, enabling the entire model to perform different speech generation tasks. In other words, the solution in this application embodiment can generate speech from different types of input data using a single model.

[0119] The decoder is a diffusion-based decoder that generates speech through a back-diffusion process. In other words, the speech generation model is a DPM (Digital Persistent Model) capable of generating high-quality speech, with fast adaptation speed and low data requirements. Thus, in the model of this application embodiment, the quality of the generated speech can be ensured.

[0120] According to the technical solution provided in the embodiments of this application, the two encoders and decoders in the model can be trained separately to avoid the instability caused by joint training. The two encoders can be trained separately with the same target in a supervised manner. This supervision method is more reliable because the outputs of the two encoders have explicit interpretations (e.g., average sound spectrograms) and do not belong to the latent space. Regarding speaker adaptation, only the decoder needs fine-tuning, while the two encoders remain speaker-independent.

[0121] Furthermore, the speech generation model also includes a speaker encoder, which can be used to reproduce the target voice. Thus, even in scenarios where no target voice data is available for training, i.e., in zero-shot scenarios, the speech generation model provided in this application embodiment can generate speech with the target voice.

[0122] The model in this application embodiment can be in different modes when performing different speech generation tasks. In other words, the model can perform different speech generation tasks according to different modes. In different modes, the encoders involved in performing the task can be different.

[0123] For example, multiple encoders may include a speech encoder, in which a model can be used to perform voice conversion. When the model is in voice conversion mode, the speech encoder, in conjunction with a decoder, is used to perform voice conversion.

[0124] Figure 2 This is a flowchart of one embodiment of the sound conversion provided in this application.

[0125] The source data is audio data, corresponding to a speech encoder, i.e. Figure 2 The MEL encoder in the code predicts the average spectrogram corresponding to the source speaker audio X0 based on the source speaker audio. The sound in the source speaker audio belongs to speaker A. A diffusion-based decoder, conditioned on the target sound information, generates a fine-grained spectrogram based on the average spectrogram. The target sound information can be obtained by processing the target speaker audio Y0 using the speaker encoder. The sound in the target speaker audio belongs to... Figure 2 Speaker B in the context. The target speaker is... Figure 2 Speaker B in the text. A fine-grained spectrogram can be converted into speech with the target sound (i.e., speaker B's sound). The fine-grained spectrogram can be viewed as the target acoustic features, i.e. Figure 2 Target spectrum diagram

[0126] It should be noted that, although Figure 2 The fact that only one encoder is shown does not mean that the model has only one encoder. Figure 2 The encoder shown is only for illustrating the encoder that performs data processing in sound conversion mode.

[0127] For example, multiple encoders can include text encoders, in which case the model can be used to perform sound cloning. When the model is in sound cloning mode, sound cloning is performed using a text encoder in conjunction with a decoder.

[0128] Figure 3This is a flowchart of one embodiment of the sound cloning provided in this application.

[0129] The source data is text data, corresponding to Figure 3 The text encoder predicts the average spectrogram corresponding to the source text T based on the source text T. A diffusion-based decoder, conditioned on the target sound information, generates a fine-grained spectrogram based on the average spectrogram. The target sound information can be obtained by processing the target speaker audio Y0 using a speaker encoder. The sound in the target speaker audio Y0 belongs to speaker B. The target speaker is... Figure 3 Speaker B in the text. A fine-grained spectrogram can be converted into speech with the target sound (i.e., speaker B's sound). The fine-grained spectrogram can be viewed as the target acoustic features, i.e. Figure 3 Target spectrum diagram

[0130] It should be noted that, although Figure 3 The fact that only one encoder is shown does not mean that the model has only one encoder. Figure 3 The encoder shown is only for illustrating the encoder used for data processing in sound cloning mode.

[0131] For example, multiple encoders can include a speech encoder and a text encoder, in which case the model can be used to generate speech based on input audio data and input text data.

[0132] Figure 4 This is a flowchart of one embodiment of speech generation provided in this application.

[0133] The source data includes audio data and corresponding text data. The audio data and the corresponding text data are respectively... Figure 4 The code uses a MEL encoder and a text encoder. The MEL encoder predicts the average spectrogram corresponding to the source speaker audio X0. The sound in the source speaker audio X0 belongs to speaker A. The text encoder predicts the average spectrogram corresponding to the source text T. A diffusion-based decoder, conditioned on the target sound information, generates a fine-grained spectrogram based on the average spectrogram, which is determined from the outputs of the MEL encoder and the text encoder. For example, the average spectrogram input to the decoder can be either the average spectrogram corresponding to the source speaker audio X0 or the average spectrogram corresponding to the source text T. The target sound information can be obtained by processing the target speaker audio Y0 using a speaker encoder. The sound in the target speaker audio Y0 belongs to speaker B. The target speaker is... Figure 4Speaker B in the text. A fine-grained spectrogram can be converted into speech with the target sound (i.e., speaker B's sound). The fine-grained spectrogram can be viewed as the target acoustic features, i.e. Figure 4 Target spectrum diagram

[0134] It should be noted that, although Figure 4 Only two encoders are shown, but this does not mean that the model has only two encoders.

[0135] For example, multiple encoders may include lip-reading video encoders, in which case the model can be used to generate speech based on the input video.

[0136] Figure 5 This is a flowchart of one embodiment of speech generation provided in this application.

[0137] The source data is video data, corresponding to Figure 5 The lip-reading video encoder predicts the average spectrogram corresponding to the source video. The sound in the source video belongs to speaker A. A diffusion-based decoder, conditioned on the target sound information, generates a fine-grained spectrogram based on the average spectrogram. The target sound information can be obtained by processing the target speaker audio Y0 using a speaker encoder. The sound in the target speaker audio Y0 belongs to speaker B. The target speaker is... Figure 5 Speaker B in the text. A fine-grained spectrogram can be converted into speech with the target sound (i.e., speaker B's sound). The fine-grained spectrogram can be viewed as the target acoustic features, i.e. Figure 5 Target spectrum diagram

[0138] It should be noted that, although Figure 5 The fact that only one encoder is shown does not mean that the model has only one encoder.

[0139] For example, multiple encoders can include video encoders and speech encoders, in which case the model can be used to generate speech based on input video and input audio.

[0140] Figure 6 This is a flowchart of one embodiment of speech generation provided in this application.

[0141] The source data types include video data and corresponding audio data. The video data and the corresponding audio data are respectively... Figure 6The video encoder and MEL encoder are used in the process. The source speaker audio X0 can be extracted from the source video. The MEL encoder predicts the average spectrogram corresponding to the source speaker audio X0. The voice in the source speaker audio X0 belongs to speaker A. The video encoder generates a video embedding based on the source video. For example, video embeddings can be used for emotion recognition. A diffusion-based decoder, conditioned on the target voice information, generates a fine-grained spectrogram based on concatenated features. For example, concatenated features can be obtained by concatenating the average spectrogram and the video embedding. The target voice information can be obtained by processing the target speaker audio Y0 using the speaker encoder. The voice in the target speaker audio Y0 belongs to speaker B. The target speaker is... Figure 6 Speaker B in the text. A fine-grained spectrogram can be converted into speech with the target sound (i.e., speaker B's sound). The fine-grained spectrogram can be viewed as the target acoustic features, i.e. Figure 6 Target spectrum diagram

[0142] It should be noted that, although Figure 6 Only two encoders are shown, but this does not mean that the model has only two encoders.

[0143] Figure 7 This is a flowchart of an embodiment of a speech generation method. Figure 7 The method shown can be performed by a device or a device capable of performing model operations. For example, the device can be a cloud service device or a terminal device, such as a computer, server, or other device with sufficient computing power to perform the data processing method. Alternatively, the device can be a system consisting of a cloud service device and a terminal device.

[0144] Figure 7 The method shown includes the following steps:

[0145] 701, Obtain the first source data input into a speech generation model that includes multiple encoders and decoders, wherein the input data of the multiple encoders are of different types;

[0146] 702, the first encoder among multiple encoders generates a first acoustic feature based on the first source data, wherein the type of the first source data is consistent with the type of the input data of the first encoder;

[0147] 703, the decoder converts the second acoustic feature determined based on the first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with the target sound.

[0148] Optionally, the decoder can be a diffusion-based decoder. The decoder converting a second acoustic feature determined based on a first acoustic feature into a third acoustic feature can include: the decoder converting the second acoustic feature determined based on the first acoustic feature into a third acoustic feature through a reverse diffusion process.

[0149] For example, the first source data could be source audio data, source text data, or source video data.

[0150] Speech generation can be Figure 1 The model in the text.

[0151] Optionally, the multiple encoders include at least two of the following: a video encoder, a speech encoder, or a text encoder, wherein when the first source data is audio data, the first encoder is a speech encoder; when the first source data is text data, the first encoder is a text encoder; or when the first source data is video data, the first encoder is a video encoder.

[0152] For example, the first acoustic feature could be a spectrogram-like feature corresponding to the first source data, and the third acoustic feature could be a spectrogram of speech containing the target sound. The spectrogram of speech containing the target sound can be called the target spectrogram. The spectrogram-like feature corresponding to the first source data could be any of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data that can be aligned with the target spectrogram on the time axis, or a concatenation of a spectrogram corresponding to the first source data and an acoustic feature that can be aligned with the target spectrogram on the time axis.

[0153] For example, the second acoustic feature can be the first acoustic feature.

[0154] For example, the third acoustics can be the target acoustic features, such as a fine-grained spectrogram.

[0155] For example, the third acoustic feature can be converted into speech with the target sound by a vocoder.

[0156] The speech generation model in this application embodiment has multiple encoders and a shared decoder. The multiple encoders can operate on different input domains, enabling the entire model to perform different speech generation tasks. In other words, the solution in this application embodiment can generate speech from different types of input data using a single model.

[0157] The decoder is a diffusion-based decoder that generates speech through a back-diffusion process. In other words, the speech generation model is a DPM (Digital Persistent Model) capable of generating high-quality speech, with fast adaptation speed and low data requirements. Thus, in the model of this application embodiment, the quality of the generated speech can be ensured.

[0158] Optionally, multiple encoders and decoders can be trained separately.

[0159] Optionally, the multiple encoders include a speech encoder and a text encoder. When the first source data is audio data, the first encoder is a speech encoder; when the first source data is text data, the first encoder is a text encoder.

[0160] The model described above, consisting of a speech encoder, a text encoder, and a decoder, can perform sound cloning and sound conversion: the speech encoder combined with the decoder is used to perform sound conversion, while the text encoder combined with the decoder corresponds to the sound cloning task.

[0161] Furthermore, due to the hybrid nature of the speech encoder and text encoder, speaker adaptation can be performed on untranscribed data.

[0162] Optionally, the first acoustic feature is an average spectrum corresponding to the first source data.

[0163] The average spectrogram can be viewed as a speaker-independent speech representation. The first encoder remains speaker-independent, meaning that it does not require fine-tuning for speaker adaptation. If multiple encoders remain speaker-independent, only the decoder needs fine-tuning for speaker adaptation.

[0164] For example, when the first source data is audio data, the speech encoder can generate an average spectrogram corresponding to the audio data. When the first source data is text data, the text encoder can generate an average spectrogram corresponding to the audio data.

[0165] In this way, the model can convert speaker-independent acoustic features (e.g., average spectrograms extracted from text data by a text encoder or average spectrograms extracted from audio data by a speech encoder) into target acoustic features through a decoder.

[0166] Optionally, the speech encoder, text encoder, and decoder are trained separately.

[0167] According to the technical solution provided in the embodiments of this application, the two encoders and decoders in the model can be trained separately to avoid the instability caused by joint training. The two encoders can be trained separately with the same target in a supervised manner. This supervision method is more reliable because the outputs of the two encoders have explicit interpretations (e.g., average sound spectrograms) and do not belong to the latent space. Regarding speaker adaptation, only the decoder needs fine-tuning, while the two encoders remain speaker-independent.

[0168] Optionally, the method may further include the following steps (not shown in the figure):

[0169] 704, Obtain the second source data input into the speech generation model;

[0170] 705, the second encoder among multiple encoders generates a fourth acoustic feature based on the second source data, wherein the type of the second source data is consistent with the type of the input data of the second encoder, and the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature.

[0171] The type of the second source data can be different from the type of the first source data. In this case, the second encoder and the first encoder are different. In other words, different types of input data can be processed by different encoders in the model.

[0172] For example, the first acoustic feature could be an average spectrogram corresponding to the first source data. The second acoustic feature could be a video embedding generated by the video encoder (i.e., the second encoder).

[0173] It should be noted that the step numbers in the above method are for illustrative purposes and do not limit the order in which the steps are executed.

[0174] Optionally, step 703 includes the decoder converting a second acoustic feature determined based on a first acoustic feature into a third acoustic feature through a back-diffusion process, conditioned on the target sound information, wherein the target sound information is generated by a speaker encoder.

[0175] The speaker encoder can be considered part of the decoder because the speaker encoder is trained jointly with the decoder.

[0176] The speech generation model also includes a speaker encoder, which can be used to reproduce the target voice. Thus, even in scenarios where no target voice data is available for training, i.e., in zero-shot scenarios, the speech generation model provided in this application can generate speech with the target voice.

[0177] Figure 8 This is a schematic block diagram of the electronic device 800 provided in an embodiment of this application. Figure 8 As shown, the electronic device 800 includes: a first acquisition module 801, a first generation module 802, and a conversion module 803.

[0178] The first acquisition module 801 is used to acquire first source data input into a speech generation model that includes multiple encoders and decoders, wherein the input data of the multiple encoders are of different types.

[0179] The first generation module 802 is used for: the first encoder among a plurality of encoders generating a first acoustic feature based on the first source data, wherein the type of the first source data is consistent with the type of the input data of the first encoder.

[0180] The conversion module 803 is used for: the decoder to convert a second acoustic feature determined based on a first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with a target sound.

[0181] Optionally, the decoder is a diffusion-based decoder, and the conversion module is specifically used to convert the second acoustic feature determined according to the first acoustic feature into a third acoustic feature through a reverse diffusion process.

[0182] Optionally, the multiple encoders include at least two of the following: a speech encoder, a text encoder, or a video encoder. When the first source data is audio data, the first encoder is a speech encoder; when the first source data is text data, the first encoder is a text encoder; or when the first source data is video data, the first encoder is a video encoder.

[0183] Optionally, the speech encoder, multiple encoders, and decoders are trained separately.

[0184] Optionally, the third acoustic feature is a target spectrogram, and the first acoustic feature is a spectrogram-like feature corresponding to the first source data. The spectrogram-like feature corresponding to the first source data is any one of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis, or a concatenation of a spectrogram corresponding to the first source data and an acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis.

[0185] Optionally, the first acoustic feature is an average spectrum corresponding to the first source data.

[0186] Optionally, the electronic device further includes: a second acquisition module and a second generation module. Figure 8 (Not shown in the image).

[0187] The second acquisition module is used to acquire the second source data input into the speech generation model.

[0188] The second generation module is used for: the second encoder among multiple encoders to generate a fourth acoustic feature based on the second source data, wherein the type of the second source data is consistent with the type of the input data of the second encoder, and the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature.

[0189] Optionally, the conversion module is specifically used for: the decoder converting the second acoustic feature determined based on the first acoustic feature into the third acoustic feature through a back-diffusion process, conditioned on the target sound information, wherein the target sound information is generated by the speaker encoder.

[0190] Figure 9 This is a schematic block diagram of the electronic device 900 provided in the embodiments of this application.

[0191] like Figure 9 As shown, the electronic device 900 may include a transceiver 901, a processor 902, and a memory 903. The memory 903 may be used to store code, instructions, etc., executed by the processor 902.

[0192] It should be understood that the processor 902 can be an integrated circuit chip with signal processing capabilities. In implementation, the various steps of the above method embodiments can be completed by hardware integrated logic circuits in the processor or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. A general-purpose processor can be a microprocessor, or the processor can be any conventional processor, etc. The steps of the methods disclosed in the embodiments of this invention can be directly executed and completed by a hardware decoding processor, or executed and completed using a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and combines it with the hardware in the processor to complete the steps of the above methods.

[0193] It is understood that the memory 903 in the embodiments of the present invention can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0194] It should be noted that the memory in the systems and methods described in this specification includes, but is not limited to, these memories and any other suitable types of memory.

[0195] This application also provides a system-on-a-chip (SoC) comprising an input / output interface, at least one processor, at least one memory, and a bus. The at least one memory stores instructions, and the at least one processor invokes the instructions from the at least one memory to perform the operations described in the above embodiments.

[0196] This application also provides a computer storage medium that can store program instructions to execute any of the above methods.

[0197] Alternatively, the storage medium may specifically be memory 903.

[0198] Those skilled in the art will recognize that the various units and algorithm steps described in conjunction with the embodiments disclosed in this specification can be implemented by electronic hardware or by a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but should not consider such implementation to be beyond the scope of this application.

[0199] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the above-described systems, devices, and units can be referred to the corresponding processes in the above method embodiments. Further details will not be repeated here.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the described apparatus embodiments are merely examples. For example, unit division is only a logical functional division, and other division methods may exist in actual implementation. For example, multiple units or components may be merged or integrated into another system, or some features may be ignored or not performed. Furthermore, the mutual coupling or direct coupling or communication connection shown or described can be implemented through some interfaces. Indirect coupling or communication connection between apparatuses or units can be implemented electronically, mechanically, or otherwise.

[0201] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one location or distributed across multiple network units. Some or all units can be selected according to actual needs to achieve the objectives of the embodiment.

[0202] Furthermore, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0203] When these functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions in this application, either in essence or in part, that contribute to the prior art, can be implemented in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to instruct a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes any medium capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0204] The above description is merely a specific implementation of this application and is not intended to limit the scope of protection of this application. Any variations or substitutions that are readily conceived by those skilled in the art within the scope of the technology disclosed in this application should fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice generation method characterized by, include: First source data and second source data are obtained as input to a speech generation model comprising multiple encoders and decoders, wherein the input data of the multiple encoders are of different types; The first encoder among the plurality of encoders generates a first acoustic feature based on the first source data, wherein the type of the first source data is consistent with the type of the input data of the first encoder; The second encoder in the plurality of encoders generates a fourth acoustic feature based on the second source data, wherein the type of the second source data is consistent with the type of the input data of the second encoder, and the type of the second source data is different from that of the first source data; The decoder converts the second acoustic feature into a third acoustic feature, wherein the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature, and the third acoustic feature is used to generate speech with the target sound.

2. The method of claim 1, wherein, The decoder is a diffusion-based decoder that converts the second acoustic feature into the third acoustic feature, including: The decoder converts the second acoustic feature into the third acoustic feature through a reverse diffusion process.

3. The method according to claim 1 or 2, characterized in that, The plurality of encoders includes at least two of the following: a speech encoder, a text encoder, or a video encoder, wherein when the first source data is audio data, the first encoder is the speech encoder; when the first source data is text data, the first encoder is the text encoder; or when the first source data is video data, the first encoder is the video encoder.

4. The method according to claim 1 or 2, characterized in that, The multiple encoders and the decoder are trained respectively.

5. The method according to claim 1 or 2, characterized in that, The third acoustic feature is a target spectrogram, and the first acoustic feature is a spectrogram-like feature corresponding to the first source data. The spectrogram-like feature corresponding to the first source data is any one of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis, or a concatenation of the spectrogram corresponding to the first source data and the acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis.

6. The method of claim 5, wherein, The first acoustic feature is the average spectrum corresponding to the first source data.

7. The method according to claim 1 or 2, characterized in that, The decoder converts the second acoustic feature into the third acoustic feature, including: The decoder converts the second acoustic feature into the third acoustic feature through a back-diffusion process, conditioned on the target sound information, wherein the target sound information is generated by the speaker encoder.

8. An electronic device, comprising: include: The first acquisition module is used to: acquire first source data and second source data input into a speech generation model including multiple encoders and decoders, wherein the input data of the multiple encoders are of different types; A first generation module is configured to: generate a first acoustic feature from the first encoder among the plurality of encoders based on the first source data, wherein the type of the first source data is consistent with the type of the input data of the first encoder; The second generation module is configured to: generate a fourth acoustic feature from the second encoder among the plurality of encoders based on the second source data, wherein the type of the second source data is consistent with the type of the input data of the second encoder, and the type of the second source data is different from that of the first source data; A conversion module is configured to: convert a second acoustic feature into a third acoustic feature, wherein the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature, and the third acoustic feature is used to generate speech with a target sound.

9. The electronic device of claim 8, wherein, The decoder is a diffusion-based decoder, and the conversion module is specifically used for: The decoder converts the second acoustic feature into the third acoustic feature through a reverse diffusion process.

10. The electronic device of claim 8 or 9, wherein, The plurality of encoders includes at least two of the following: a speech encoder, a text encoder, or a video encoder, wherein when the first source data is audio data, the first encoder is the speech encoder; when the first source data is text data, the first encoder is the text encoder; or when the first source data is video data, the first encoder is the video encoder.

11. The electronic device of claim 8 or 9, wherein, The multiple encoders and the decoder are trained respectively.

12. The electronic device of claim 8 or 9, wherein, The third acoustic feature is a target spectrogram, and the first acoustic feature is a spectrogram-like feature corresponding to the first source data. The spectrogram-like feature corresponding to the first source data is any one of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis, or a concatenation of the spectrogram corresponding to the first source data and the acoustic feature corresponding to the first source data that is aligned with the target spectrogram on the time axis.

13. The electronic device of claim 12, wherein, The first acoustic feature is the average spectrum corresponding to the first source data.

14. The electronic device of claim 8 or 9, wherein, The conversion module is specifically used for: The decoder converts the second acoustic feature into the third acoustic feature through a back-diffusion process, conditioned on the target sound information, wherein the target sound information is generated by the speaker encoder.

15. A computer-readable storage medium having instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method according to any one of claims 1 to 7.

16. An electronic device, comprising: The device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to retrieve and run the computer program from the memory, causing the computer on which the chip resides to perform the method according to any one of claims 1 to 7.

17. A computer program product, characterised in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1 to 7.

Images