Voice generation method and related equipment

Abstract

A speech generation method and related device, the method comprising: acquiring first source data (701) input into a speech generation model (100) comprising a plurality of encoders (111, 112) and a decoder (120), the types of input data of the plurality of encoders (111, 112) being different; a first encoder of the plurality of encoders (111, 112) generates a first acoustic feature (702) from the first source data; the decoder (120) converts a second acoustic feature determined from the first acoustic feature into a third acoustic feature, where the third acoustic feature is used to generate speech with a target sound (703). According to the technical scheme, different voice generation tasks can be executed by means of a single model.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech technology, and more specifically, to a speech generation method and related devices. Background Art

[0002] Speech generation is a technology that generates speech based on input. Speech generation can refer to various speech generation, such as text-to-speech (TTS), sound conversion, or video-to-speech. Different speech generation tasks are usually handled by different frameworks, which limits the application of speech generation in practice. For example, in some scenarios, the resources for speech generation on electronic devices are limited. However, different frameworks may require a lot of resources, such as storage resources and computing resources, or even exceed the limited resources, which affects the application of speech generation. Summary of the invention

[0003] The embodiment of the present application provides a speech generation method and related devices. The technical solution can rely on a single model to perform different speech generation tasks.

[0004] According to a first aspect, an embodiment of the present application provides a speech generation method, comprising: obtaining first source data input into a speech generation model including multiple encoders and decoders, wherein the types of the input data of the multiple encoders are different; a first encoder among the multiple encoders generates a first acoustic feature based on the first source data, wherein the type of the first source data is consistent with the type of the input data of the first encoder; the decoder converts a second acoustic feature determined based on the first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with a target sound.

[0005] The speech generation model of the embodiment of the present application has multiple encoders and a shared decoder, wherein the multiple encoders can be respectively operated on different input domains, so that the entire model performs corresponding different speech generation tasks. In other words, the solution of the embodiment of the present application can generate speech according to different types of input data through one model.

[0006] In one possible design, the decoder is a diffusion-based decoder, and the decoder converts the second acoustic feature determined based on the first acoustic feature into a third acoustic feature, including: the decoder converts the second acoustic feature determined based on the first acoustic feature into the third acoustic feature through a reverse diffusion process.

[0007] The decoder is a diffusion-based decoder that can generate speech through a reverse diffusion process. In other words, the speech generation model is a DPM that can generate high-quality speech, has a fast adaptation speed, and has a small data requirement. In this way, in the model of the embodiment of the present application, the quality of the generated speech can be ensured.

[0008] For example, the first acoustic feature may be a spectrogram-like feature corresponding to the first source data, and the third acoustic feature may be a spectrogram of speech with a target sound. The spectrogram of speech with a target sound may be referred to as a target spectrogram. The spectrogram-like feature corresponding to the first source data is any one of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data aligned with the target spectrogram on the time axis, or a concatenation of the spectrogram corresponding to the first source data and the acoustic feature corresponding to the first source data aligned with the target spectrogram on the time axis.

[0009] For example, the first source data may be source audio data, source text data, or source video data.

[0010] For example, the second acoustic feature may be the first acoustic feature.

[0011] For example, the third acoustic feature may be a target acoustic feature, such as a fine-grained spectrogram.

[0012] In one possible design, the multiple encoders include at least two of the following: a video encoder, a speech encoder, or a text encoder, wherein, when the first source data is audio data, the first encoder is the speech encoder, when the first source data is text data, the first encoder is the text encoder, or when the first source data is video data, the first encoder is the video encoder.

[0013] In one possible design, the multiple encoders and decoders are trained separately.

[0014] In one possible design, the multiple encoders include a speech encoder and a text encoder, wherein, when the first source data is audio data, the first encoder is the speech encoder, and when the first source data is text data, the first encoder is the text encoder.

[0015] The above model consisting of a speech encoder, a text encoder and a decoder can perform voice cloning and voice conversion: the speech encoder combined with the decoder is used to perform voice conversion, while the text encoder combined with the decoder corresponds to the voice cloning task.

[0016] In one possible design, the first acoustic feature is an average frequency spectrogram corresponding to the first source data.

[0017] The average spectrogram can be viewed as a speaker-independent representation of speech. The first encoder remains speaker-independent, which means that it does not need to be fine-tuned for speaker adaptation.

[0018] In one possible design, the speech encoder, the text encoder and the decoder are trained separately.

[0019] According to the technical solution provided in the embodiment of the present application, the two encoders and decoders in the model can be trained separately to avoid the instability caused by joint training. The two encoders can be trained in a supervised manner with the same goal, which is more reliable because the outputs of the two encoders have a clear explanation (e.g., an average sound spectrogram) and do not belong to the latent space.

[0020] In one possible design, the method also includes: obtaining second source data that is input into the speech generation model; a second encoder among the multiple encoders generates a fourth acoustic feature based on the second source data, wherein the type of the second source data is consistent with the type of the input data of the second encoder, and the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature.

[0021] For example, the first encoder may be a speech encoder or a text encoder, and the second encoder may be a video encoder.

[0022] In one possible design, the decoder converts the second acoustic feature determined based on the first acoustic feature into a third acoustic feature through a back diffusion process, including: the decoder converts the second acoustic feature determined based on the first acoustic feature into the third acoustic feature through the back diffusion process based on target sound information, wherein the target sound information is generated by a speaker encoder.

[0023] The speech generation model also includes a speaker encoder, which can be used to replicate the target sound. In this way, even in a scenario where there is no target sound data for training, that is, in a zero-shot scenario, the speech generation model provided in the embodiment of the present application can generate speech with the target sound.

[0024] According to a second aspect, an embodiment of the present application provides an electronic device, which has the function of implementing the method in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware of the software includes one or more modules corresponding to the function.

[0025] According to a third aspect, an embodiment of the present application provides a computer-readable storage medium having instructions, and when the instructions are executed on a computer, the computer executes the method in the first aspect or any possible implementation of the first aspect.

[0026] According to a fourth aspect, an electronic device is provided, comprising a processor and a memory. The processor is connected to the memory. The memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions stored in the memory, the processor executes the method in the first aspect or any possible implementation of the first aspect.

[0027] According to the fifth aspect, a chip system is provided, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program, so that the server where the chip is located executes the method in the first aspect or any possible implementation of the first aspect.

[0028] According to a sixth aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes the method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic block diagram of the speech generation model provided in an embodiment of the present application.

[0030] Figure 2 It is a flowchart of an embodiment of sound conversion provided by an embodiment of the present application.

[0031] Figure 3 It is a flow chart of an embodiment of sound cloning provided by an embodiment of the present application.

[0032] Figure 4 It is a flowchart of an embodiment of speech generation provided by an embodiment of the present application.

[0033] Figure 5 This is a flowchart of another embodiment of speech generation provided by an embodiment of the present application.

[0034] Figure 6 This is a flowchart of another embodiment of speech generation provided in the embodiments of the present application.

[0035] Figure 7 is a flow chart of an embodiment of a speech generation method.

[0036] Figure 8 It is a schematic block diagram of an electronic device 800 provided in an embodiment of the present application.

[0037] Fig. 9 It is a schematic block diagram of an electronic device 900 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solution of the present application is described below in conjunction with the accompanying drawings.

[0039] In order to facilitate understanding of the embodiments of the present application, relevant terms involved in the embodiments of the present application are introduced below.

[0040] (1) Voice cloning

[0041] Voice cloning is a task that is usually formulated as adding a new voice to a TTS system. In other words, voice cloning is essentially a TTS technique that replicates the voice of a target speaker.

[0042] When target speaker data is available, voice cloning can be performed by speaker adaptation. Speaker adaptation usually refers to fine-tuning the TTS system based on a small amount of target speaker data to obtain a well-performing TTS for the target voice.

[0043] When only a short target voice sample is available, voice cloning is performed by speaker encoding, which generally refers to the use of pre-trained or learnable speaker representations to help extract speaker identity information, such as timbre and pitch, from a reference speech sample.

[0044] (2) Sound conversion

[0045] Voice conversion is the task of replicating the voice of a target speaker while preserving the linguistic content of the source speaker's utterance.

[0046] Any-to-one (A2O) voice conversion (VC) aims to convert any speaker (including speakers unseen during training) to a fixed target speaker.

[0047] In practice, it is desirable to have an any-to-any voice conversion model. An any-to-any voice conversion model is one that can replicate the target voice while preserving the source speech content when both the source and target speakers are not necessarily part of the training dataset.

[0048] (3) Diffusion probabilistic model (DPM)

[0049] DPM consists of forward diffusion and backward diffusion. Forward diffusion gradually adds Gaussian noise to the data, while backward diffusion attempts to remove this noise. DPM is trained to minimize the distance between the trajectories of the forward and backward diffusion processes. In other words, the training goal of DPM is to find backward diffusion so that the trajectory of backward diffusion closely follows the trajectory of forward diffusion, but in reverse time order.

[0050] Different speech generation tasks are usually handled by different models. For example, TTS and voice conversion are two common speech generation tasks, which are usually handled by different models.

[0051] The embodiment of the present application provides a speech generation model, which can process different types of input data to generate speech. In other words, the speech generation model of the embodiment of the present application can handle a variety of different speech generation tasks.

[0052] The speech generation model provided in the embodiment of the present application includes multiple encoders and a decoder shared by the multiple encoders. The outputs of the multiple encoders can be the inputs of the decoders.

[0053] Figure 1 is a schematic block diagram of a speech generation model provided in an embodiment of the present application. Figure 1 As shown, the speech generation model 100 may include an encoder 111 , an encoder 112 , and a decoder 120 .

[0054] It should be noted that Figure 1 It is just a schematic diagram of the speech generation model provided in the embodiment of the present application. Figure 1 The number of encoders shown does not constitute any limitation. Figure 1 In the embodiment, the speech generation model 100 includes two encoders. In other cases, the speech generation model may further include more encoders.

[0055] Each of the multiple encoders is used to obtain acoustic features corresponding to its own input data. The decoder is used to obtain target acoustic features conditioned on a target sound according to the output of at least one encoder. The target acoustic features conditioned on the target sound can be used to generate speech with the target sound. For example, the output domain of the decoder can be a spectrogram of speech with the target sound. The spectrogram of speech with the target sound can be called a target spectrogram. The output of the decoder can be converted into a waveform by a vocoder (e.g., a general generative adversarial network for efficient and high fidelity speech synthesis (HiFi-GAN) vocoder for efficient and high fidelity speech synthesis). The vocoder may belong to a speech generation model, or the vocoder may not belong to a speech generation model.

[0056] For example, multiple encoders can be implemented using neural networks.

[0057] The types of the multiple encoders are different. The types of the input data of the multiple encoders are related to the types of the encoders. Accordingly, the input data of the multiple encoders are different types of data. The input data of the multiple encoders can be called source data.

[0058] In a possible implementation manner, the multiple encoders may include at least two of the following: a speech encoder, a text encoder, or a video encoder.

[0059] The input data of the speech encoder can be acoustic data such as audio, speech or acoustic features. The acoustic features can be spectrograms, also known as spectral features.

[0060] For example, the spectrogram may be a mel spectrogram, in which case the speech encoder may also be referred to as a mel encoder, and the mel spectrogram may also be referred to as a mel feature.

[0061] The input data of the text encoder can be text data such as text, character or phoneme embedding.

[0062] The input data of the video encoder may be video data. For example, the video encoder may be a lip reading encoder.

[0063] For example, Figure 1 The encoder 111 in may be a speech encoder, Figure 1 The encoder 112 in can be a text encoder. For example, Figure 1 The encoder 111 in may be a speech encoder, Figure 1 The encoder 112 in may be a video encoder. For example, Figure 1 The encoder 111 in may be a text encoder, Figure 1 The encoder 112 in may be a video encoder.

[0064] The output domains of multiple encoders can be the same or different.

[0065] In one possible implementation, an encoder in a speech generation model is used to generate an output similar to a spectrogram.

[0066] For example, the spectrogram-like output may be a spectrogram. Alternatively, the spectrogram-like output may be an acoustic feature that may be aligned with the spectrogram on a time axis, such as pitch, loudness, and a spectrogram convolved with a filter bank along a frequency axis. Alternatively, the spectrogram-like output may be a concatenation of a spectrogram and an acoustic feature that may be aligned with the spectrogram on a time axis.

[0067] Optionally, in some embodiments, at least one encoder is used to generate a spectrogram in a speech generation model. In this case, the spectrogram generated by the encoder can be regarded as an acoustic feature corresponding to the input data of the encoder itself.

[0068] Alternatively, in some embodiments, multiple encoders work together to generate spectrograms in a speech generation model.

[0069] The above-mentioned speech encoder, text encoder or video encoder can be used to generate the spectrogram. Alternatively, at least one encoder among the speech encoder, text encoder or video encoder can be used to generate the spectrogram.

[0070] It should be noted that the encoder is only an example. As mentioned above, the output domain of the decoder can be a spectrogram, i.e., a target spectrogram. In this case, the encoder should generate an output that can be aligned with the target spectrogram. An output similar to a spectrogram can be aligned with the target spectrogram. Therefore, other encoders that can generate an output similar to a spectrogram can also be used as the encoder in the embodiment of the present application.

[0071] Furthermore, the output of the encoder may approximate the target spectrogram. For example, the output of the encoder may be one of the following: an average spectrogram corresponding to the encoder's input data, a spectrogram of a particular sound, or a low-resolution spectrogram of the target sound.

[0072] For ease of understanding and explanation, the embodiments of the present application are described using an average spectrum diagram as an example.

[0073] The average spectrogram may be referred to as an average sound spectrogram. The average sound refers to the pronunciation of each phoneme, and its features may be the same as the average features in the dataset of multiple speakers. For example, the average sound spectrogram may be an average sound mel spectrogram, and the average sound mel spectrogram may be referred to as an average phoneme-level mel feature.

[0074] For example, an encoder for predicting an average spectrogram corresponding to input data can be obtained by training. Specifically, the encoder can be trained with the goal of reducing the difference between the output of the encoder and the ground truth average spectrogram corresponding to the training source data. During the training process of the encoder, the training source data is the input data of the encoder. The method of obtaining the ground truth average spectrogram can refer to the example in the next section.

[0075] During inference, the output of the encoder trained in the above manner can be viewed as the average spectrogram corresponding to the encoder’s input data.

[0076] In an embodiment of the present application, the encoder may be configured to predict an average frequency spectrogram corresponding to input data of the encoder.

[0077] For example, a speech encoder may be used to predict an average spectrogram corresponding to a source audio, a text encoder may be used to predict an average spectrogram corresponding to a source text, and a video encoder may be used to predict an average spectrogram corresponding to a source video.

[0078] The average spectrogram is independent of the speaker corresponding to the input data of the encoder. The speaker corresponding to the input data of the encoder can be called the source speaker, so the average spectrogram can be regarded as a speech representation independent of the speaker.

[0079] Optionally, in some embodiments, multiple encoders and decoders in the speech generation model can be trained separately.

[0080] by Figure 1 Taking the model 100 in FIG. 1 as an example, the encoder 111, the encoder 112 and the decoder 120 can be trained separately. In other words, the encoder 111, the encoder 112 and the decoder 120 can be regarded as three independent modules. During the training of one module of the three independent modules, the parameters of the other modules are fixed.

[0081] For example, Figure 1 The encoder 111 in may be used to predict an average spectrogram corresponding to input data of the encoder 111, Figure 1 The encoder 112 in the example can be used to predict an average spectrogram corresponding to the input data of the encoder 112. The following takes the encoder 111 as a mel encoder and the encoder 112 as a text encoder as an example to illustrate the training process of the encoder.

[0082] mel encoder After training, the audio data X0 is converted into an average spectrogram corresponding to the audio data X0.

[0083] For example, the mel encoder After training, the output spectrogram and ground truth average spectrogram The mean square error (MSE) between them is minimized, and during training, X0 is the training source audio data.

[0084] The training source audio data may be a training source spectrogram X0. The ground truth average spectrogram may be obtained by replacing the features corresponding to each phoneme in the training source spectrogram X0 with the features corresponding to the particular phoneme aggregated in a corpus of speech data from multiple speakers. The corpus can be an existing corpus, or a corpus set up as needed.

[0085] For example, there is phoneme A in the training source spectrogram X0. The features of phoneme A in the training source spectrogram X0 are replaced with the average features of phoneme A. The average features of phoneme A are obtained by aggregating the features of phoneme A in a corpus of speech data from multiple speakers. The above steps are performed for each phoneme in the training source spectrogram X0 to obtain the ground truth average spectrogram corresponding to the training source spectrogram X0.

[0086] During inference, X0 is the source audio data that needs to be processed, referred to as source audio data. The output of the mel encoder trained in the above way is It can be seen as the average spectrogram corresponding to the source audio data X0.

[0087] For example, transformer-based architectures can be used as speech encoders.

[0088] The text encoder ψ is trained to convert the source text data T into the average spectrogram corresponding to the source text data T.

[0089] For example, the text encoder ψ is trained to output a spectrogram and ground truth average spectrogram During training, T is the training source text data.

[0090] Get the ground truth average spectrogram The method can be the same as above. That is, when the language content of the training source text data T and the training source audio data X0 is the same, the ground truth average spectrogram can also be the same, that is, during training, the text encoder

[0091] The target output is the same as the target output of the speech encoder.

[0092] The text encoder may be a text encoder of an existing structure or a self-configured text encoder.

[0093] For example, a text encoder could be Figure 3 The encoder shown in Figure 1. The text encoder converts the input text into an encoded text sequence, which is then mapped to frame-by-frame features such as spectrograms. Figure 3 As shown, convolutional layers (conv) and bidirectional long short term memory (Bi LSTM) are used to generate the encoded text sequence. The duration predictor generates a monotonic alignment that indicates how many frames each element of the text input lasts, which can help generate a spectrogram. Upsampling is a process that repeats each output of the Bi-LSTM by the number of times predicted by the duration predictor to ensure that a spectrogram with the correct duration can be generated.

[0094] Optionally, in some embodiments, the speech generation model has a speaker encoder. The speaker encoder is used to provide target sound information to the decoder, in which case the decoder generates acoustic features conditioned on the target sound. The decoder is a speaker-conditioned decoder. For example, the decoder can be used to convert an average spectrogram into a fine-grained spectrogram conditioned on the target sound information.

[0095] For example, the target sound information may be a speaker embedding.

[0096] The speaker encoder can be trained jointly with the decoder, so the speaker encoder can also be considered as a part of the decoder.

[0097] The speaker encoder can be called a speaker encoding network.

[0098] The decoder can be a diffusion-based decoder. The speech generation model in the embodiment of the present application can be regarded as a DPM that attempts to convert acoustic features extracted from source data by at least one encoder in multiple encoders into target acoustic features by using a speaker-dependent score matching network (called a decoder).

[0099] Forward diffusion transforms any source data into normal random variables Where I is the identity matrix, Predicted by at least one encoder.

[0100] For example, the source data may be a source spectrogram X0. Can be a mel encoder The predicted average sound spectrogram. Therefore, the prior in this DPM It is a speaker-independent speech representation that preserves the linguistic content of the source data.

[0101] The back-diffusion parameterized by the decoder is trained to inversely approximate the forward diffusion trajectory in the time variable t∈[0,1].

[0102] As mentioned above, the decoder and multiple encoders can be trained separately.

[0103] The encoder parameterizes the terminal distribution for forward diffusion (i.e., the prior), while backward diffusion is parameterized using the decoder.

[0104] For example, once the DPM prior Parameterized mel encoder After training, the mel encoder The parameters of are fixed and the decoder corresponding to back-diffusion is trained.

[0105] As a possible implementation, DPM can be formalized by adopting stochastic differential equation (SDE).

[0106] Forward X and Reverse The diffusion process can be obtained by the following SDE:

[0107]

[0108] Among them, t∈[0,1], and are the forward and reverse standard Brownian motions that are independent of each other. t is a non-negative noise schedule. X t is a sample in forward diffusion, is a sample in back diffusion.

[0109] Speaker conditioning in the decoder is performed by the speaker encoder network g t (Y) Enable.

[0110] By integrating into the fractional matching network s θ and match the score network s θ Jointly trained speaker encoder network g t (Y), the reverse SDE (Formula 1.2) is conditioned on the target sound:

[0111]

[0112] The decoder parameters are represented by θ, Y = {Y s} s∈[0,1] is the overall trajectory of the reference spectrogram Y0 calculated for the target sound under forward diffusion. In other words, Y = {Ys} s∈[0,1] is the overall forward diffusion trajectory starting from Y0. The reference spectrogram Y0 may be a training spectrogram during training, that is, a training source spectrogram X0. The reference spectrogram Y0 may be a spectrogram of a target sound during inference.

[0113] A well-trained decoder can be trained by Middle Pair The sample is taken and the path of back diffusion parameterized by the decoder is simulated at unit time intervals [0,1] to achieve generative modeling. The sample generated at the initial time point is the output of the speech generation task.

[0114] During inference, at each iteration of the back-diffusion process, the speaker embeddings are also re-estimated and fed back to the gradient prediction network of the decoder.

[0115] The decoder can be implemented with a neural network. For example, the decoder has an architecture based on UNet.

[0116] Speaker Encoding Network t (Y) can be composed of 2D convolution and multilayer perceptron (MLP).

[0117] It should be noted that DPM can also be formalized in other ways. For example, DPM can also be formalized by using a Markov chain, which is not limited in the embodiments of the present application.

[0118] The speech generation model of the embodiment of the present application has multiple encoders and a shared decoder, wherein the multiple encoders can be respectively operated on different input domains, so that the entire model performs corresponding different speech generation tasks. In other words, the solution of the embodiment of the present application can generate speech according to different types of input data through one model.

[0119] The decoder is a diffusion-based decoder that can generate speech through a reverse diffusion process. In other words, the speech generation model is a DPM that can generate high-quality speech, has a fast adaptation speed, and has a small data requirement. In this way, in the model of the embodiment of the present application, the quality of the generated speech can be ensured.

[0120] According to the technical solution provided in the embodiment of the present application, the two encoders and decoders in the model can be trained separately to avoid the instability caused by joint training. The two encoders can be trained in a supervised manner with the same target, which is more reliable because the outputs of the two encoders have a clear explanation (e.g., an average sound spectrogram) and do not belong to the latent space. In terms of speaker adaptation, only the decoder needs to be fine-tuned, while the two encoders remain independent of the speaker.

[0121] In addition, the speech generation model also includes a speaker encoder, which can be used to replicate the target sound. In this way, even in a scenario where there is no target sound data for training, that is, in a zero-shot scenario, the speech generation model provided in the embodiment of the present application can generate speech with the target sound.

[0122] The model in the embodiment of the present application can be in different modes when performing different speech generation tasks. In other words, the model can perform different speech generation tasks according to different modes. In different modes, the encoders involved in performing the tasks can be different.

[0123] For example, the plurality of encoders may include a speech encoder, in which the model may be used to perform voice conversion. When the model is in voice conversion mode, the speech encoder is used in conjunction with a decoder to perform voice conversion.

[0124] Figure 2 It is a flowchart of an embodiment of sound conversion provided by an embodiment of the present application.

[0125] The type of source data is audio data, corresponding to the speech encoder, that is, Figure 2 The mel encoder in . The mel encoder predicts the average spectrogram corresponding to the source speaker audio X0 based on the source speaker audio. The sound in the source speaker audio belongs to speaker A. The diffusion-based decoder conditioned on the target sound information generates a fine-grained spectrogram based on the average spectrogram. The target sound information can be obtained by processing the target speaker audio Y0 through the speaker encoder. The sound in the target speaker audio belongs to Figure 2 The target speaker is B. Figure 2 The fine-grained spectrogram can be converted into speech with the target voice (i.e., the voice of speaker B). The fine-grained spectrogram can be regarded as the target acoustic feature, i.e. Figure 2 Target spectrum in

[0126] It should be noted that although Figure 2 Just one encoder is shown, which does not mean that the model has only one encoder. Figure 2 The encoder shown is only for illustrating the encoder for data processing in the voice conversion mode.

[0127] As another example, the multiple encoders may include a text encoder, in which case the model may be used to perform voice cloning. When the model is in voice cloning mode, the text encoder is used in conjunction with the decoder to perform voice cloning.

[0128] Figure 3It is a flow chart of an embodiment of sound cloning provided by an embodiment of the present application.

[0129] The source data type is text data, corresponding to Figure 3 The text encoder in . The text encoder predicts the average spectrogram corresponding to the source text T based on the source text T. The diffusion-based decoder conditioned on the target sound information generates a fine-grained spectrogram based on the average spectrogram. The target sound information can be obtained by processing the target speaker audio Y0 through the speaker encoder. The sound in the target speaker audio Y0 belongs to speaker B. The target speaker is Figure 3 The fine-grained spectrogram can be converted into speech with the target voice (i.e., the voice of speaker B). The fine-grained spectrogram can be regarded as the target acoustic feature, i.e. Figure 3 Target spectrum in

[0130] It should be noted that although Figure 3 Just one encoder is shown, which does not mean that the model has only one encoder. Figure 3 The encoders shown are only for illustration of encoders performing data processing in the voice cloning mode.

[0131] As another example, the multiple encoders may include a speech encoder and a text encoder, in which case the model can be used to generate speech based on input audio data and input text data.

[0132] Figure 4 It is a flowchart of an embodiment of speech generation provided by an embodiment of the present application.

[0133] The source data includes audio data and text data corresponding to the audio data, and the audio data and the text data corresponding to the audio data correspond to Figure 4 The mel encoder and text encoder in . The mel encoder predicts the average spectrogram corresponding to the source speaker audio X0 based on the source speaker audio X0. The sound in the source speaker audio X0 belongs to speaker A. The text encoder predicts the average spectrogram corresponding to the source text T based on the source text T. The diffusion-based decoder conditioned on the target sound information generates a fine-grained spectrogram based on the average spectrogram, and the average spectrogram is determined based on the output of the mel encoder and the output of the text encoder. For example, the average spectrogram used as the input of the decoder can be either the average spectrogram corresponding to the source speaker audio X0 or the average spectrogram corresponding to the source text T. The target sound information can be obtained by processing the target speaker audio Y0 through the speaker encoder. The sound in the target speaker audio Y0 belongs to speaker B. The target speaker is Figure 4The fine-grained spectrogram can be converted into speech with the target voice (i.e., the voice of speaker B). The fine-grained spectrogram can be regarded as the target acoustic feature, i.e. Figure 4 Target spectrum in

[0134] It should be noted that although Figure 4 Just because two encoders are shown does not mean that the model has only two encoders.

[0135] As another example, the multiple encoders may include a lip-reading video encoder, in which case the model can be used to generate speech based on the input video.

[0136] Figure 5 It is a flowchart of an embodiment of speech generation provided by an embodiment of the present application.

[0137] The type of source data is video data, corresponding to Figure 5 The lip-reading video encoder in . The lip-reading video encoder predicts the average spectrogram corresponding to the source video based on the source video. The sound in the source video belongs to speaker A. The diffusion-based decoder conditioned on the target sound information generates a fine-grained spectrogram based on the average spectrogram. The target sound information can be obtained by processing the target speaker audio Y0 through the speaker encoder. The sound in the target speaker audio Y0 belongs to speaker B. The target speaker is Figure 5 The fine-grained spectrogram can be converted into speech with the target voice (i.e., the voice of speaker B). The fine-grained spectrogram can be regarded as the target acoustic feature, i.e. Figure 5 Target spectrum in

[0138] It should be noted that although Figure 5 Just one encoder is shown, which does not mean that the model has only one encoder.

[0139] As another example, the multiple encoders may include a video encoder and a speech encoder, in which case the model can be used to generate speech based on the input video and the input audio.

[0140] Figure 6 It is a flowchart of an embodiment of speech generation provided by an embodiment of the present application.

[0141] The types of source data include video data and audio data corresponding to the video data, and the video data and the audio data corresponding to the video data correspond to Figure 6The video encoder and mel encoder in . The source speaker audio X0 can be extracted from the source video. The mel encoder predicts the average spectrogram corresponding to the source speaker audio X0 based on the source speaker audio X0. The sound in the source speaker audio X0 belongs to speaker A. The video encoder generates a video embedding based on the source video. For example, video embedding can be used for emotion recognition. The diffusion-based decoder conditioned on the target sound information generates a fine-grained spectrogram based on the concatenated features. For example, the concatenated features can be obtained by concatenating the average spectrogram and the video embedding. The target sound information can be obtained by processing the target speaker audio Y0 through the speaker encoder. The sound in the target speaker audio Y0 belongs to speaker B. The target speaker is Figure 6 The fine-grained spectrogram can be converted into speech with the target voice (i.e., the voice of speaker B). The fine-grained spectrogram can be regarded as the target acoustic feature, i.e. Figure 6 Target spectrum in

[0142] It should be noted that although Figure 6 Just because two encoders are shown does not mean that the model has only two encoders.

[0143] Figure 7 is a flow chart of an embodiment of a speech generation method. Figure 7 The method shown can be performed by a device or a device capable of performing model operations. For example, the device can be a cloud service device or a terminal device, such as a computer, a server, or other device with sufficient computing power to perform the data processing method. Alternatively, the device can be a system composed of a cloud service device and a terminal device.

[0144] Figure 7 The method shown comprises the following steps:

[0145] 701, obtaining first source data input into a speech generation model including a plurality of encoders and decoders, wherein the input data of the plurality of encoders are of different types;

[0146] 702, a first encoder among the plurality of encoders generates a first acoustic feature according to first source data, wherein a type of the first source data is consistent with a type of input data of the first encoder;

[0147] 703. The decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with a target sound.

[0148] Optionally, the decoder may be a diffusion-based decoder. The decoder converting the second acoustic feature determined according to the first acoustic feature into the third acoustic feature may include: the decoder converting the second acoustic feature determined according to the first acoustic feature into the third acoustic feature through a reverse diffusion process.

[0149] For example, the first source data may be source audio data, source text data, or source video data.

[0150] Speech generation can be Figure 1 The model in .

[0151] Optionally, the multiple encoders include at least two of the following: a video encoder, a speech encoder, or a text encoder, wherein, when the first source data is audio data, the first encoder is a speech encoder, when the first source data is text data, the first encoder is a text encoder, or when the first source data is video data, the first encoder is a video encoder.

[0152] For example, the first acoustic feature may be a spectrogram-like feature corresponding to the first source data, and the third acoustic feature may be a spectrogram of speech with a target sound. The spectrogram of speech with a target sound may be referred to as a target spectrogram. The spectrogram-like feature corresponding to the first source data may be any of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data that can be aligned with the target spectrogram on the time axis, or a concatenation of a spectrogram corresponding to the first source data and an acoustic feature that can be aligned with the target spectrogram on the time axis.

[0153] For example, the second acoustic feature may be the first acoustic feature.

[0154] For example, the third acoustic may be a target acoustic feature, such as a fine-grained spectrogram.

[0155] For example, the third acoustic feature may be converted into speech having a target sound by a vocoder.

[0156] The speech generation model of the embodiment of the present application has multiple encoders and a shared decoder, wherein the multiple encoders can be respectively operated on different input domains, so that the entire model performs corresponding different speech generation tasks. In other words, the solution of the embodiment of the present application can generate speech according to different types of input data through one model.

[0157] The decoder is a diffusion-based decoder that can generate speech through a reverse diffusion process. In other words, the speech generation model is a DPM that can generate high-quality speech, has a fast adaptation speed, and has a small data requirement. In this way, in the model of the embodiment of the present application, the quality of the generated speech can be ensured.

[0158] Optionally, multiple encoders and decoders are trained separately.

[0159] Optionally, the plurality of encoders include a speech encoder and a text encoder. When the first source data is audio data, the first encoder is a speech encoder, and when the first source data is text data, the first encoder is a text encoder.

[0160] The above model consisting of a speech encoder, a text encoder and a decoder can perform voice cloning and voice conversion: the speech encoder combined with the decoder is used to perform voice conversion, while the text encoder combined with the decoder corresponds to the voice cloning task.

[0161] Furthermore, due to the hybrid nature of the speech encoder and text encoder, speaker adaptation can be performed on untranscribed data.

[0162] Optionally, the first acoustic feature is an average spectrogram corresponding to the first source data.

[0163] The average spectrogram can be viewed as a speaker-independent representation of speech. The first encoder remains speaker-independent, which means that the first encoder does not need to be fine-tuned for speaker adaptation. If the multiple encoders remain speaker-independent, only the decoder needs to be fine-tuned for speaker adaptation.

[0164] For example, when the first source data is audio data, the speech encoder may generate an average spectrogram corresponding to the audio data. When the first source data is text data, the text encoder may generate an average spectrogram corresponding to the audio data.

[0165] In this way, the model can convert speaker-independent acoustic features (e.g., average spectrograms extracted from text data by a text encoder or average spectrograms extracted from audio data by a speech encoder) into target acoustic features through a decoder.

[0166] Optionally, the speech encoder, text encoder and decoder are trained separately.

[0167] According to the technical solution provided in the embodiment of the present application, the two encoders and decoders in the model can be trained separately to avoid the instability caused by joint training. The two encoders can be trained in a supervised manner with the same target, which is more reliable because the outputs of the two encoders have a clear explanation (e.g., an average sound spectrogram) and do not belong to the latent space. In terms of speaker adaptation, only the decoder needs to be fine-tuned, while the two encoders remain independent of the speaker.

[0168] Optionally, the method may further include the following steps (not shown in the figure):

[0169] 704, obtaining second source data input into the speech generation model;

[0170] 705. A second encoder among the multiple encoders generates a fourth acoustic feature based on second source data, wherein a type of the second source data is consistent with a type of input data of the second encoder, and the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature.

[0171] The type of the second source data may be different from the type of the first source data, in which case the second encoder is different from the first encoder. In other words, different types of input data may be processed by different encoders in the model.

[0172] For example, the first acoustic feature may be an average spectrogram corresponding to the first source data. The second acoustic feature may be a video embedding generated by a video encoder (ie, the second encoder).

[0173] It should be noted that the step numbers in the above method are only for illustration and convenience and do not limit the execution order of the steps.

[0174] Optionally, step 703 includes the decoder converting the second acoustic feature determined according to the first acoustic feature into a third acoustic feature through a back diffusion process based on target sound information, wherein the target sound information is generated by a speaker encoder.

[0175] The speaker encoder can be considered as a part of the decoder because the speaker encoder is trained jointly with the decoder.

[0176] The speech generation model also includes a speaker encoder, which can be used to replicate the target sound. In this way, even in a scenario where there is no target sound data for training, that is, in a zero-shot scenario, the speech generation model provided in the embodiment of the present application can generate speech with the target sound.

[0177] Figure 8 8 is a schematic block diagram of an electronic device 800 provided in an embodiment of the present application. Figure 8 As shown, the electronic device 800 includes: a first acquisition module 801 , a first generation module 802 and a conversion module 803 .

[0178] The first acquisition module 801 is used to acquire first source data input into a speech generation model including multiple encoders and decoders, wherein the types of input data of the multiple encoders are different.

[0179] The first generating module 802 is used for: a first encoder among multiple encoders generates a first acoustic feature according to first source data, wherein the type of the first source data is consistent with the type of input data of the first encoder.

[0180] The conversion module 803 is used for: the decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with a target sound.

[0181] Optionally, the decoder is a diffusion-based decoder, and the conversion module is specifically used for: the decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature through a reverse diffusion process.

[0182] Optionally, the plurality of encoders include at least two of the following: a speech encoder, a text encoder, or a video encoder. When the first source data is audio data, the first encoder is a speech encoder, when the first source data is text data, the first encoder is a text encoder, or when the first source data is video data, the first encoder is a video encoder.

[0183] Optionally, the speech encoder, the plurality of encoders and the decoder are trained separately.

[0184] Optionally, the third acoustic feature is a target spectrogram, the first acoustic feature is a spectrogram-like feature corresponding to the first source data, and the spectrogram-like feature corresponding to the first source data is any one of: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data aligned with the target spectrogram on the time axis, or a concatenation of a spectrogram corresponding to the first source data and an acoustic feature corresponding to the first source data aligned with the target spectrogram on the time axis.

[0185] Optionally, the first acoustic feature is an average spectrogram corresponding to the first source data.

[0186] Optionally, the electronic device further includes: a second acquisition module and a second generation module ( Figure 8 not shown).

[0187] The second acquisition module is used to acquire second source data which is input into the speech generation model.

[0188] The second generating module is used for: a second encoder among multiple encoders generates a fourth acoustic feature according to second source data, wherein the type of the second source data is consistent with the type of input data of the second encoder, and the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature.

[0189] Optionally, the conversion module is specifically used for: the decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature through a back diffusion process based on target sound information, wherein the target sound information is generated by a speaker encoder.

[0190] Fig. 9 It is a schematic block diagram of an electronic device 900 provided in an embodiment of the present application.

[0191] like Fig. 9 As shown, the electronic device 900 may include a transceiver 901, a processor 902, and a memory 903. The memory 903 may be used to store codes, instructions, etc. executed by the processor 902.

[0192] It should be understood that the processor 902 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by a hardware integrated logic circuit in the processor or by instructions in the form of software. The processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The processor can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general processor can be a microprocessor, or the processor can be any traditional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present invention can be directly executed and completed by a hardware decoding processor, or executed and completed using a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with the hardware in the processor.

[0193] It is understandable that the memory 903 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM may be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0194] It should be noted that the memories in the systems and methods described in this specification include but are not limited to these memories and any other suitable types of memories.

[0195] The embodiment of the present application also provides a system chip, wherein the system chip includes an input / output interface, at least one processor, at least one memory and a bus. The at least one memory is used to store instructions, and the at least one processor is used to call instructions of the at least one memory to perform the operations in the method in the above embodiment.

[0196] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium can store program instructions to execute any of the above methods.

[0197] Optionally, the storage medium may specifically be the memory 903 .

[0198] Those skilled in the art will appreciate that, in conjunction with the examples described in the embodiments disclosed in this specification, each unit and algorithm step can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions of each specific application, but it should not be considered that the implementation is beyond the scope of this application.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned systems, devices and units can refer to the corresponding processes in the above-mentioned method embodiments, which will not be described in detail here.

[0200] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the described device embodiments are only examples. For example, unit division is only a logical function division, and other division methods can be used in actual implementation. For example, multiple units or components can be merged or integrated into another system, or some features can be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or described can be implemented through some interfaces. The indirect coupling or communication connection between devices or units can be implemented electronically, mechanically or in other ways.

[0201] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one location, or may be distributed over multiple network units. Some or all of the units may be selected based on actual needs to achieve the purpose of the solution of the embodiment.

[0202] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0203] When these functions are implemented in the form of software functional units and sold or used as independent products, these functions can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution in this application can be implemented in the form of a software product in essence or in the part that contributes to the prior art or in the part of the technical solution. The computer software product is stored in a storage medium, including several instructions to instruct a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the application. The above-mentioned storage medium includes any medium that can store program code, such as a USB disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0204] The above description is only a specific implementation of the present application and is not intended to limit the scope of protection of the present application. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should belong to the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A speech generation method, characterized in that: include: Obtaining first source data input into a speech generation model including a plurality of encoders and decoders, wherein the input data of the plurality of encoders are of different types; A first encoder among the plurality of encoders generates a first acoustic feature according to the first source data, wherein a type of the first source data is consistent with a type of input data of the first encoder; The decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with a target sound.

2. The method according to claim 1, characterized in that The decoder is a diffusion-based decoder, and the decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature, including: The decoder converts the second acoustic feature determined according to the first acoustic feature into the third acoustic feature through a back diffusion process.

3. The method according to claim 1 or 2, characterized in that: The multiple encoders include at least two of the following: a speech encoder, a text encoder or a video encoder, wherein, when the first source data is audio data, the first encoder is the speech encoder, when the first source data is text data, the first encoder is the text encoder, or when the first source data is video data, the first encoder is the video encoder.

4. The method according to any one of claims 1 to 3, characterized in that The plurality of encoders and the decoder are trained separately.

5. The method according to any one of claims 1 to 4, characterized in that The third acoustic feature is a target spectrogram, the first acoustic feature is a spectrogram-like feature corresponding to the first source data, and the spectrogram-like feature corresponding to the first source data is any one of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data and aligned with the target spectrogram on the time axis, or a concatenation of the spectrogram corresponding to the first source data and the acoustic feature corresponding to the first source data and aligned with the target spectrogram on the time axis.

6. The method according to claim 5, characterized in that The first acoustic feature is an average spectrogram corresponding to the first source data.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Obtaining second source data input into the speech generation model; A second encoder among the multiple encoders generates a fourth acoustic feature based on the second source data, wherein a type of the second source data is consistent with a type of input data of the second encoder, and the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature.

8. The method according to any one of claims 1 to 7, characterized in that The decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature, including: The decoder converts the second acoustic feature determined according to the first acoustic feature into the third acoustic feature through the back diffusion process under the condition of target sound information, wherein the target sound information is generated by a speaker encoder.

9. An electronic device, characterized in that: include: A first acquisition module is used to: acquire first source data input into a speech generation model including a plurality of encoders and decoders, wherein the input data of the plurality of encoders are of different types; A first generating module, configured to: generate, by a first encoder among the plurality of encoders, a first acoustic feature according to the first source data, wherein a type of the first source data is consistent with a type of input data of the first encoder; A conversion module is used for: the decoder converts the second acoustic feature determined according to the first acoustic feature into a third acoustic feature, wherein the third acoustic feature is used to generate speech with a target sound.

10. The electronic device according to claim 9, characterized in that: The decoder is a diffusion-based decoder, and the conversion module is specifically used for: The decoder converts the second acoustic feature determined according to the first acoustic feature into the third acoustic feature through a back diffusion process.

11. The electronic device according to claim 9 or 10, characterized in that: The multiple encoders include at least two of the following: a speech encoder, a text encoder or a video encoder, wherein, when the first source data is audio data, the first encoder is the speech encoder, when the first source data is text data, the first encoder is the text encoder, or when the first source data is video data, the first encoder is the video encoder.

12. The electronic device according to any one of claims 9 to 11, characterized in that: The plurality of encoders and the decoder are trained separately.

13. The electronic device according to any one of claims 9 to 12, characterized in that: The third acoustic feature is a target spectrogram, the first acoustic feature is a spectrogram-like feature corresponding to the first source data, and the spectrogram-like feature corresponding to the first source data is any one of the following: a spectrogram corresponding to the first source data, an acoustic feature corresponding to the first source data and aligned with the target spectrogram on the time axis, or a concatenation of the spectrogram corresponding to the first source data and the acoustic feature corresponding to the first source data and aligned with the target spectrogram on the time axis.

14. The electronic device according to claim 13, characterized in that: The first acoustic feature is an average spectrogram corresponding to the first source data.

15. The electronic device according to any one of claims 9 to 14, characterized in that: Also includes: A second acquisition module is used to: acquire second source data input into the speech generation model; A second generating module is used for: a second encoder among the multiple encoders generates a fourth acoustic feature according to the second source data, wherein the type of the second source data is consistent with the type of input data of the second encoder, and the second acoustic feature is obtained by concatenating the fourth acoustic feature and the first acoustic feature.

16. The electronic device according to any one of claims 9 to 15, characterized in that: The conversion module is specifically used for: The decoder converts the second acoustic feature determined according to the first acoustic feature into the third acoustic feature through the back diffusion process under the condition of target sound information, wherein the target sound information is generated by a speaker encoder.

17. A computer-readable storage medium having instructions, characterized in that: When the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

18. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program, so that the computer where the chip is located executes the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

Images